«Using complex networks towards information retrieval and diagnostics in multidimensional imaging Soumya Jyoti Banerjee1, Mohammad Azharuddin2, ...»
Using complex networks towards information
retrieval and diagnostics in multidimensional
Soumya Jyoti Banerjee1, Mohammad Azharuddin2, Debanjan Sen3, Smruti Savale3,
Himadri Datta3, Anjan Kr Dasgupta2, and Soumen Roy1,*
1 Bose Institute, 93/1 Acharya PC Roy Road, Kolkata 700 009, India
2 Department of Biochemistry, University of Calcutta, 35 Ballygunge Circular Road, Kolkata 700 019, India
3 Regional Institute of Ophthalmology, Calcutta Medical College and Hospital, Kolkata 700 073, India arXiv:1506.02602v2 [cs.IR] 2 Dec 2015 * firstname.lastname@example.org ABSTRACT We present a fresh and broad yet simple approach towards information retrieval in general and diagnostics in particular by applying the theory of complex networks on multidimensional, dynamic images. We demonstrate a successful use of our method with the time series generated from high content thermal imaging videos of patients suffering from the aqueous deﬁcient dry eye (ADDE) disease. Remarkably, network analyses of thermal imaging time series of contact lens users and patients upon whom Laser-Assisted in situ Keratomileusis (Lasik) surgery has been conducted, exhibit pronounced similarity with results obtained from ADDE patients. We also propose a general framework for the transformation of multidimensional images to networks for futuristic biometry. Our approach is general and scalable to other ﬂuctuation-based devices where network parameters derived from ﬂuctuations, act as effective discriminators and diagnostic markers.
Introduction The ﬁeld of Content Based Image Retrieval (CBIR) started with retrieval of speciﬁc images from a large array of images.
Nowadays, CBIR is more generally referred to as Content Based Multimedia Information Retrieval (CBMIR) or simply MIR.
Information retrieval in general can be conceived of as ﬁnding material of an unstructured nature that satisﬁes an information need from within large collections1. Applications of pictorial search into a database of images already existed in specia
Graph Theory in Computer Vision: A Toplogical Perspective. Many problems in image processing can be naturally mapped to energy minimisation approaches. However, such energy minimisation problems could be highly demanding from the computational point of view, as the general requirement is to minimise a non-convex function in a space with thousands of dimensions. Thankfully, dynamic programming can be used, but, only in a limited number of cases, where the energy functions have special forms4. In absence of such privileges, researchers typically used global optimisation techniques like simulated annealing6 or greedy algorithms7 for image smoothing which would be very slow for obvious reasons.
“Graph cut” approaches have come to be widely used in computer vision especially those that could be formulated in terms of energy minimisation. The essence of such approaches is that the basic technique is to construct a specialized graph on which the energy function to be minimized, such that the minimum cut on the graph in turn minimizes the energy. This follows from from the max-ﬂow min-cut theorem that in a ﬂow network, the amount of maximum ﬂow is equal to capacity of the minimum cut. It was shown that maximising the ﬂow through an image network is associated with the maximum a posteriori estimate of a binary image, introduction of sources and sinks make the problem efﬁciently solvable5. These approaches have been used successfully in a wide variety of vision problems including shape matching8, image restoration9, 10, ﬁngerprint recognition11, surface ﬁtting12, stereo and motion9, 10 and medical imaging13.
There also exists a body of work14, 15 towards applying spectral encoding of a graph for indexing to large database of image features represented as Directed Acyclic Graphs (DAG). Databases of topological signatures can be indexed efﬁciently to retrieve model objects having similar topology. Signiﬁcant research has been conducted on a general class of matching methods, called bipartite matching, to problems in object recognition The time complexity for ﬁnding such a matching in a weighted √ bipartite graph with N vertices was determined as O N 2 NloglogN 16.
Recent researches on image segmentation have used multi-resolution community detection methods in ﬂuorescent lifetime microscopy17, 18. Replica inference approaches have also been used towards unsupervised multiscale image segmentation19.
Herein, we have used graph theory from a different perspective. Instead of object identiﬁcation based on spatial correlations, we have exploited the relational topology of the image objects. This approach adds another angle to image segmentation and object identiﬁcation, two classic problems in image processing.
Time Series to Networks. A large number of approaches to analyze time series have been proposed over time. These range from time-frequency methods, such as Fourier and wavelet transforms20–22, to nonlinear methods, such as phase-space embeddings, Lyapunov exponents, correlation dimensions and entropies23–25. These techniques are helpful for summarizing the characteristics of a time series into compact metrics. Such brevity can be efﬁciently exploited to effectively understand the dynamics or to predict how the system will evolve over time. However, these measures preserve many but not all of the important properties of a given time series. Therefore, there is considerable research toward the identiﬁcation of metrics that can capture the additional information or quantify time series in a completely new ways26–29.
Quite independent of the above, the ﬁeld of complex networks has been extensively studied by itself and successfully applied in manifold instances in science, nature and engineering30, 31. With signiﬁcant advances being reported from various ﬁelds32–43, the importance of converting time series into networks is becoming increasingly clear over the last few years44.
From Videos to Time Series and Thence to Networks. In this work, we furnish a new, simple and general route to information retrieval by combining developments from these disparate ﬁelds and show that such an approach can yield rich dividends for MDI in general and for diagnostics in particular. Indeed, following the broad framework proposed here, it is possible to construct inexpensive devices for non-invasive diagnostics and biometric based applications, which can perform successfully in real-time45. Our method consists of the following steps: (i) conversion of a given video or MDI into time series, (ii) conversion of the time series into a network, and ﬁnally (iii) analysing the network metrics to identify speciﬁc topological metric/(s) which can act as good discriminators for different videos.
Advantages of the Present Approach. The process of conversion of any given video to a time series has been known for some time46. Albeit, to our knowledge, the fullest potential of this conversion in thermal imaging has not yet been exploited. Effective utilization of the vast research in time series analysis and related advances is obviously critical to gain liberal advantage of this transformation in information retrieval.
A network based representation of time series, provides us with an analytical tool that may allow object identiﬁcation, which is not possible in many conventional image processing techniques. The uniqueness of the present identiﬁcation approach is the use of analyses based on temporal instead of spatial distributions. As such network, based insights can be fed back for extraction of hidden image contents which are not evident from the spatial data alone.
Principal component analysis or PCA47 is a potent and widely used linear transform in signal and image processing, more speciﬁcally in image compression, blind signal separation, face and pattern recogntion48–51 etc. Essentially, PCA is a method for transforming a multidimensional dataset to a lower dimension. The basis vectors follow modes of the greatest variance, when the data is represented by PCA in the new coordinate system. However, PCA is also computationally expensive compared to many other processes like Fast Fourier Transformation. Herein, we show that in the present approach, the computationally expensive procedure of PCA adopted for dimensional reduction in conventional image processing can be safely circumvented.
Obviously, dimension reduction achieved by our method would make feature identiﬁcation of complex videos and images computationally far simpler. As detailed below, our approach effectively opens up the avenue of ﬂuctuation based diagnostics in biomedical MDI. Hardware implementations of this method is extremely versatile, as it is smart, fast and portable45.
No established method, to our knowledge, has addressed the problem of the dynamics of thermal behavior from source thermal imaging data. Conventional image processing algorithms typically attempt appropriate segmentation, noise elimination and object identiﬁcation like morphological changes or relative pixel dynamics. For videos, images extensive work has been done on motion tracking and this known to have important implications in contexts like security and surveillance.
3/13 Lastly, The present work of mapping biomedical videos into a time series and thence to a network should be implemented in diagnostic approaches, which need to record biomedical time-series data over a prolonged duration, perhaps without rest.
The inconvenience or pain caused to a patient is imaginable.
Applications to Diagnostics: Dry Eye Disease. While the approach proposed in this paper is very general, herein, we speciﬁcally concentrate on patients with Aqueous Deﬁcient Dry Eye (ADDE) disease, contact lens users and patients who have undergone Lasik operations. We also investigate applications of our work in biometrics. ADDE is a disturbance in tear ﬁlm physiology that leads to various abnormal states of ocular surface cells that elevate the incidence of ocular surface disorders and infection. ADDE represents one of the most common ocular pathologies and is a complex multifactorial disease characterized by an immune and inﬂammatory process that affects the lacrimal glands and ocular surface. Its diagnosis by assessment of the tear ﬁlm has been extensively studied52–54. Most of the diagnostic approaches are based on either osmomolarity or evaporation of the tear ﬁlm. Studies indicate that most ADDE measures do not capture the etiologies for dry eye, such as dysfunctional neurology, hormonal inﬂuences or the inﬂammatory nature of the condition. Another situation that may affect the alteration of the tear dynamics is the use of contact lens. Statistical studies of thermal ﬂuctuation of healthy individuals (control) and ADDE patients where non-invasive TI was used, have been conducted recently55. Signiﬁcant correlation of thermal ﬂuctuations is found between left and right eye of control whereas this property is completely absent in eyes of ADDE patients. However, the problem of classiﬁcation of dry eye either from collective or individual data remains unsolved. Also, parametric classiﬁcation to differentiate or diagnose healthy and dry eye individuals is still unavailable. The mechanism proposed here shows that thermal ﬂuctuation based approaches and a robust parametrization of such ﬂuctuation by network mapping may be a powerful alternative approach to express the etiology of the eye. Throughout this paper, we use the terms dry eye and ADDE interchangeably.
However, it should be especially noted that in medical literature, ADDE denotes only one of the spectra of alteration of ocular surfaces going by the name of the dry eye.
Methods Ethics statement. All experiments analyzed herein were conducted after approval of the Ethics committee of Regional Institute of Opthalmology (RIO), Kolkata and were carried out in accordance with the approved guidelines of RIO. The research adhered to the tenets of the Declaration of Helsinki of the World Medical Association. Informed consent was obtained from all subjects.
Fig. 1 presents a schematic outline of our method. Speciﬁc details of our method are extensively discussed below.
Thermal Imaging Setup. For our experiments, we used a Forward Looking Infra Red (FLIR) thermal camera, Model no.
FLIR SC 305, FLIR Systems AB, Sweden. This camera is equipped with an RJ-45 gigabit Ethernet connection that supplies 16 bit 320 × 240 images at rates as high as 60 Hz along with linear temperature data. The video can be exported to several formats including AVI. In the FLIR SC 305 model, compression is used in the original video image and only the in built frame compression is used. Each frame is then cropped to select a region of interest (say eye, cheek etc). The camera was used with a thermal sensitivity of less than 0.05oC at 30oC, spatial, temporal and image resolution of 1.36 mrad, 9 frames per second and 320 × 240 pixels respectively, with spectral range between 7.5 and 13 mm.
Details of Data Collection and Clinical Background of Subjects. Following are details of patient groups and healthy
individuals for whom the thermal imaging videos were recorded for a duration of about 15 second and subsequently analysed:
(a) 36 Healthy individuals or for 72 eyes. Among them, 20 were female and 16 male, with a mean age of 28.4 years.
(b) 42 ADDE patients or for 84 eyes. Among them, 25 were female and 17 male, with a mean age of 35.2 years.
(c) 32 patients who had Lasik surgery or for 64 eyes. Among them, 18 were male and 14 female, with a mean age of 35.4 years.
(d) 29 Contact lens users or for 58 eyes. Among them, 15 were female and 14 male, with a mean age of 30.6 years.
For (d), videos were separately acquired for every individual when he or she was (i) wearing lens, and, (ii) not wearing lens.
The ocular surface temperature were recorded with eyes open and the subjects were asked not to blink during the recording.
Noise of individual data could come from blinking of eyes if the videos are recorded for a longer duration. Probability of blinking of eyes tends to zero in a small duration like 15 second and therefore noise is negligible for the recorded data.