«Andreas Pommert Karl Heinz H¨ hne Bernhard Pﬂesser o Ernst Richter Martin Riemer Thomas Schiemann Rainer Schubert Udo Schumacher Ulf Tiede ...»
Creating a high-resolution spatial/symbolic model
of the inner organs based on the Visible Human
Andreas Pommert Karl Heinz H¨ hne Bernhard Pﬂesser
Ernst Richter Martin Riemer Thomas Schiemann
Rainer Schubert Udo Schumacher Ulf Tiede
Institute of Mathematics and Computer Science in Medicine (IMDM)
University Hospital Hamburg-Eppendorf, Hamburg, Germany
Dept. of Pediatric Radiology
University Hospital Hamburg-Eppendorf, Hamburg, Germany
Institute of Anatomy
University Hospital Hamburg-Eppendorf, Hamburg, Germany Abstract Computerized three-dimensional models of the human body, based on the Visible Human Project of the National Library of Medicine, so far do not reﬂect the rich anatomical detail of the original cross-sectional images. In this paper, a spatial/symbolic model of the inner organs is developed, which is based on more than 1000 cryosections and congruent fresh and frozen CT images of the male Visible Human. The spatial description is created using color-space segmentation, graphic modeling, and a matched volume visualization with subvoxel resolution. It is linked to a symbolic knowledge base, providing an ontology of anatomical terms. With over 650 three-dimensional anatomical constituents, this model offers an unsurpassed photorealistic presentation and level of detail. A three-dimensional atlas of anatomy and radiology based on this model is available as a PC-based program.
Key words: Visible Human, three-dimensional body model, anatomical atlas, color-space segmentation, volume visualization 1 Introduction While in classical medicine, knowledge about the human body is represented in books and atlases, present-day computer science allows for new, more powerful and Email address: firstname.lastname@example.org (Andreas Pommert).
Article published in Med. Image Anal. 5 (3), 221-228, 2001 versatile computer-based representations of knowledge. Their most simple manifestations are multimedia CD-ROMs containing collections of classical pictures and text, which may be browsed arbitrarily or according to various criteria. Although computerized, such media still follow the old paradigm of text printed on pages accompanied by pictures. This genre includes impressive atlases of crosssectional anatomy, notably from the photographic cross-sections of the Visible Human Project (Ackerman, 1991; Spitzer et al., 1996).
In the past years, however, it has been shown that spatial knowledge, especially about the structure of the human body, may be much more efﬁciently represented by computerized three-dimensional models (H¨ hne et al., 1995). These can be o constructed from cross-sectional images generated by computer tomography (CT), magnetic resonance imaging (MRI), or histologic cryosectioning, as in the case of the Visible Human Project. Such models may be used interactively on a computer screen or in virtual reality environments. If such models are connected to a knowledge base of descriptive information, they can even be interrogated or disassembled by addressing names of organs (H¨ hne et al., 1995; Brinkley et al., 1999; Golland o et al., 1999). They can thus be regarded as a “self-explaining body”.
Until now, the Visible Human Project has not reported three-dimensional models that reﬂect the rich anatomical detail of the original cross-sectional images. This is largely due to the fact that, for the majority of anatomical objects contained in the data, the cross-sectional images could not be converted into a set of coherent realistic surfaces. If we succeed in converting all the detail into a 3D model, we gain an unsurpassed representation of human structure that opens new possibilities for learning anatomy and simulating interventions or radiological examinations.
2 Earlier Work
Building a comprehensive model of the inner organs of the Visible Human requires both a spatial description consisting of three-dimensional objects, which are displayed using methods of volume visualization, as well as a linked symbolic description of relevant anatomical terms and their relations.
In general, volume visualization may or may not include a segmentation step. In volume rendering, transparency values are assigned to the individual voxels according to the intensity values and changes at the object borders (Levoy, 1988). In the case of the Visible Human, this method yields semitransparent views, which are suitable e.g. for visualization of the outer surface and the musculoskeletal system (Stewart et al., 1996; Tsiaras, 1997). This way, impressive animations could be created (Gagvani and Silver, 2000; Tsiaras, 2000). It fails, however, to display internal structures properly. In addition, organ borders are not explicitly indicated, thus making the removal or exclusive display of an organ impossible.
Segmentation, i. e. the exact determination of the surface location of an organ, is therefore crucial for building a realistic model. So far, complete automatic segmentation using methods of computer vision is suitable for very special application areas only, and could not be used to build an extensive model of the human body. The brute force approach to segmentation is manual outlining of objects on the cross-sections (Mullick and Nguyen, 1996; Seymour and Kriebel, 1998). Besides the fact that this procedure is tedious and very time consuming, it is largely observer-dependent and, even more important, does not yield exact and continuous surfaces. Furthermore, despite the high resolution of the dataset, important details such as nerves and small blood vessels cannot be identiﬁed clearly, because their size and contrast is too small.
So far, no symbolic description of the inner organs which is suitable for our purposes is available. A general discussion of the problems arising, focusing on the thorax, may be found elsewhere (Rosse et al., 1998).
3 Methods and Materials We therefore aimed at a method that yields surfaces for the segmentable organs that are as exact as possible and textured with their original color. In order to arrive at a complete model, we decided to model non-segmentable objects like nerves and small blood vessels artiﬁcially on the basis of landmarks present in the image volume. Even though none of the methods presented here is entirely new, building a complex model required a number of substantial improvements.
The original dataset of the male Visible Human consists of 1871 photographic cross-sections with a slice distance of 1 mm and a spatial resolution of 0.33 mm (Figure 1, left). For reasons of data storage and computing capacity, resolution of the cross-sections was reduced to 1 mm by averaging 3¢3 pixels. From 1049 such slices, an image volume of 573¢330¢1049 voxels of 1 mm ¿ was composed, where each voxel is represented by a set of red, green and blue intensities (RGB-tuple).
The Visible Human dataset also includes two sets of computer tomographic images of 1 mm slice distance, one taken from the fresh, the other (like the photographic one) from the frozen cadaver. Both were transformed into an image volume congruent with the photographic one, using an interactive, landmark-based registration (Schiemann et al., 1994). Since the frozen body was cut into four large blocks before image acquisition, all these parts had to be aligned individually, leaving some noticeable gaps in the data volume.
Fig. 1. Left: Photographic cross-section of the abdomen of the male Visible Human. Right:
Parameterized ellipsoids in color-space, used for classiﬁcation of various tissue types in the abdomen. Many objects show similar colors, resulting in overlapping ellipsoids.
3.2 Segmentation The image volume thus created was segmented with an interactive tool, based on classiﬁcation in color-space (Schiemann et al., 1997). It can be summarized as follows: On one or several cross-sections, an expert marks a typical region of the organ under consideration. All voxels in the volume with similar RGB-tuples are then collected by the program and shown as a painted three-dimensional mask. This mask usually needs to be reﬁned by repeating this procedure in order to discriminate the target organ from the surrounding structures more clearly.
A cluster thus deﬁned in color-space usually has an ellipsoidal shape, due to the correlation of the color components. Since a set of tuples is difﬁcult to handle during subsequent visualization, this cluster is approximated by a parameterized ellipsoid, which is described by its center and three axis vectors. In general, there are other regions present in the volume which also match this color-space description. If they are not connected to the target organ, it can be isolated easily by a 3D connected component analysis. If not, borders are manually sculptured using a volume editor.
The result of this procedure is a description of an object in terms of an ellipsoid in color-space and a set of voxels, which are marked by object membership labels. Some of the ellipsoids deﬁned for segmentation of the abdomen are shown in Figure 1 (right). As can be seen, there are anatomical constituents like the intestine which could not be described using one ellipsoid only; in this case, actually seven ellipsoids were required. On the other hand, the same ellipsoid may be valid for (parts of) various anatomical constituents, such as small intestine and colon, or even for hundreds of muscles.
As a general strategy, we applied our segmentation procedure going from simple to difﬁcult tasks. This way, borders already deﬁned could be used to facilitate segmentation of other objects. As a ﬁrst step, several tissue classes such as fat, muscles, cartilage etc. were deﬁned, for which the ellipsoids could be easily determined within a few minutes. For segmentation of bone, it proved easier to use the frozen CT dataset, applying a threshold value.
Since many objects show similar colors, the resulting ellipsoids are often overlapping (Figure 1, right). Therefore, some regions such as the anterior parts of the lung or the pericardium could not be segmented this way. In case of the lung, the missing parts could be determined using the frozen CT dataset and a threshold. For the pericardium and similar cases, the volume editor was used.
3.3 Graphic modeling
For several small constituents such as nerves and blood vessels, which were considered essential for a comprehensive anatomical model, our color-space segmentation proved impossible. As regards nerves, this is mostly due to very low contrast between nervous and fat tissues, while many small arteries are collapsed as a postmortem artifact. Both problems also appear for the full resolution data.
For these cases, we developed a tube editor which allows us to include tube-like structures into the model (Figure 2). Ball-shaped markers of variable diameter are imposed by an expert onto the landmarks still visible on the cross-sections or on the 3D image. These markers are subsequently automatically connected using Overhauser splines (Yamaguchi, 1988). If one of the markers is moved, these splines will cause only local changes, which makes them easy to handle. Unlike the segmented objects, which are represented as sets of voxels, objects modeled with the tube editor are represented as polygon surfaces.
Fig. 2. Small nerves or arteries which could not be segmented were interactively modeled using a tube editor. Tubes are deﬁned by placing spheres of varying diameter into the volume, which are connected by interpolating splines.
3.4 Volume visualization The volume visualization algorithm we developed is characterized by the fact that it renders surfaces from volume data, using a ray casting approach (Tiede et al., 1998). Local surface texture (color) and inclination, as needed for surface shading, are calculated from the RGB-tuples at the segmented border line.
A decisive quality improvement is achieved by determining the surface positions with subvoxel resolution. This is done by considering both the ellipsoids (or thresholds, for CT) and the object membership labels. If a surface was created using labels only, it would appear blocky, especially when zooming into the scene. On the other hand, if only the ellipsoids were used, objects usually could not be identiﬁed without ambiguity.
In order to avoid these problems, ellipsoids and labels are combined using a colordriven algorithm (Schiemann et al., 1997; Tiede et al., 1998). Depending on the RGB-tuple found at a sampling point on a viewing ray, all ellipsoids enclosing this tuple in color-space are collected, deﬁning a set of “object candidates”. In a second step, it is tested whether a matching object label is present in the vicinity of the sampling point. In that case, an object has been found. Its subvoxel surface position is determined by interpolating the color at the sampling point (inside the ellipsoid) and the color at the previous sampling point on the viewing ray (outside the ellipsoid), such that the color at the surface is representing the object border (on the surface of the ellipsoid). Since this approach considers colors (or intensities, for CT) before labels, a smooth, continuous surface is obtained, which is not limited by voxel size.
The objects modeled with the tube editor are visualized with standard computer graphics methods within the context of the segmented objects. The visualization program, an extended version of the VOXEL-MAN system (H¨ hne et al., 1995), o runs on Linux workstations. Because of the size and resolution of the model, computation of a single image may take several minutes, even on a high-end workstation.
3.5 Knowledge modeling
While segmentation and graphic modeling provide a spatial description of anatomical objects, a comprehensive model also requires a linked symbolic description regarding anatomical terms and their relations. For this purpose, we developed a knowledge base system, using a semantic network approach (Pommert et al., 1994;
H¨ hne et al., 1995). Among others, an object is described by o ¯ names (preferred terms, synonyms, colloquial terms) in various languages ¯ pointers to related medical information (texts, histological images, references etc.) ¯ segmentation and visualization parameters (ellipsoid or threshold, object label, shading method, etc.) For choosing anatomical terms, we built on standardized nomenclature wherever available (Federative Committee on Anatomical Terminology, 1998).