3-D/2-D registration of CT and MR to X-ray images

A crucial part of image-guided therapy is registration of preoperative and intraoperative images, by which the precise position and orientation of the patient's anatomy is determined in three dimensions. This paper presents a novel approach to register three-dimensional (3-D) computed tomography (CT) or magnetic resonance (MR) images to one or more two-dimensional (2-D) X-ray images. The registration is based solely on the information present in 2-D and 3-D images. It does not require fiducial markers, intraoperative X-ray image segmentation, or timely construction of digitally reconstructed radiographs. The originality of the approach is in using normals to bone surfaces, preoperatively defined in 3-D MR or CT data, and gradients of intraoperative X-ray images at locations defined by the X-ray source and 3-D surface points. The registration is concerned with finding the rigid transformation of a CT or MR volume, which provides the best match between surface normals and back projected gradients, considering their amplitudes and orientations. We have thoroughly validated our registration method by using MR, CT, and X-ray images of a cadaveric lumbar spine phantom for which "gold standard" registration was established by means of fiducial markers, and its accuracy assessed by target registration error. Volumes of interest, containing single vertebrae L1-L5, were registered to different pairs of X-ray images from different starting positions, chosen randomly and uniformly around the "gold standard" position. CT/X-ray (MR/ X-ray) registration, which is fast, was successful in more than 91% (82% except for L1) of trials if started from the "gold standard" translated or rotated for less than 6 mm or 17 degrees (3 mm or 8.6 degrees), respectively. Root-mean-square target registration errors were below 0.5 mm for the CT to X-ray registration and below 1.4 mm for MR to X-ray registration.     

Standardized evaluation methodology for 2-D-3-D registration

In the past few years, a number of two-dimensional (2-D) to three-dimensional (3-D) (2-D-3-D) registration algorithms have been introduced. However, these methods have been developed and evaluated for specific applications, and have not been directly compared. Understanding and evaluating their performance is therefore an open and important issue. To address this challenge we introduce a standardized evaluation methodology, which can be used for all types of 2-D-3-D registration methods and for different applications and anatomies. Our evaluation methodology uses the calibrated geometry of a 3-D rotational X-ray (3DRX) imaging system (Philips Medical Systems, Best, The Netherlands) in combination with image-based 3-D-3-D registration for attaining a highly accurate gold standard for 2-D X-ray to 3-D MR/CT/3DRX registration. Furthermore, we propose standardized starting positions and failure criteria to allow future researchers to directly compare their methods. As an illustration, the proposed methodology has been used to evaluate the performance of two 2-D-3-D registration techniques, viz. a gradient-based and an intensity-based method, for images of the spine. The data and gold standard transformations are available on the internet (http://www.isi.uu.nl/Research/Databases/).     

Robust Gradient-Based 3-D/2-D Registration of CT and MR to X-Ray Images

One of the most important technical challenges in image-guided intervention is to obtain a precise transformation between the intrainterventional patient's anatomy and corresponding preinterventional 3-D image on which the intervention was planned. This goal can be achieved by acquiring intrainterventional 2-D images and matching them to the preinterventional 3-D image via 3-D/2-D image registration. A novel 3-D/2-D registration method is proposed in this paper. The method is based on robustly matching 3-D preinterventional image gradients and coarsely reconstructed 3-D gradients from the intrainterventional 2-D images. To improve the robustness of finding the correspondences between the two sets of gradients, hypothetical correspondences are searched for along normals to anatomical structures in 3-D images, while the final correspondences are established in an iterative process, combining the robust random sample consensus algorithm (RANSAC) and a special gradient matching criterion function. The proposed method was evaluated using the publicly available standardized evaluation methodology for 3-D/2-D registration, consisting of 3-D rotational X-ray, computed tomography, magnetic resonance (MR), and 2-D X-ray images of two spine segments, and standardized evaluation criteria. In this way, the proposed method could be objectively compared to the intensity, gradient, and reconstruction-based registration methods. The obtained results indicate that the proposed method performs favorably both in terms of registration accuracy and robustness. The method is especially superior when just a few X-ray images and when MR preinterventional images are used for registration, which are important advantages for many clinical applications.      

Registration of 3-D CT and 2-D Flat Images of Mouse via Affine Transformation

It is difficult to directly coregister the 3-D fluorescence molecular tomography (FMT) image of a small tumor in a mouse whose maximal diameter is only a few millimeters with a larger CT image of the entire animal that spans about 10 cm. This paper proposes a new method to register 2-D flat and 3-D CT image first to facilitate the registration between small 3-D FMT images and large 3-D CT images. A novel algorithm combining differential evolution and improved simplex method for the registration between the 2-D flat and 3-D CT images is introduced and validated with simulated images and real images of mice. The visualization of the alignment of the 3-D FMT and CT image through 2-D registration shows promising results.      

Voxel-based 2-D/3-D registration of fluoroscopy images and CT scansfor image-guided surgery

Registration of intraoperative fluoroscopy images with preoperative 3D CT images can he used for several purposes in image-guided surgery. On the one hand, it can be used to display the position of surgical instruments, which are being tracked by a localizer, in the preoperative CT scan. On the other hand, the registration result can be used to project preoperative planning information or important anatomical structures visible in the CT image on to the fluoroscopy image. For this registration task, a novel voxel-based method in combination with a new similarity measure (pattern intensity) has been developed. The basic concept of the method is explained at the example of 2D/3D registration of a vertebra in an X-ray fluoroscopy image with a 3D CT image. The registration method is described, and the results for a spine phantom are presented and discussed. Registration has been carried out repeatedly with different starting estimates to study the capture range. Information about registration accuracy has been obtained by comparing the registration results with a highly accurate “ground-truth” registration, which has been derived from fiducial markers attached to the phantom prior to imaging. In addition, registration results for different vertebrae have been compared. The results show that the rotation parameters and the shifts parallel to the projection plane can accurately be determined from a single projection. Because of the projection geometry, the accuracy of the height above the projection plane is significantly lower     

Multiple-object 2-D-3-D registration for noninvasive pose identification of fracture fragments

This paper presents a multiple-object 2-D-3-D registration technique for noninvasively identifying the poses of fracture fragments in the space of a preoperative treatment plan. The plan is made by manipulating and aligning computer models of individual fracture fragments that are segmented from a diagnostic computed tomography. The registration technique iteratively updates the treatment plan and matches its digitally reconstructed radiographs to a small number of intraoperative fluoroscopic images. The proposed approach combines an image similarity metric that integrates edge information with mutual information, and a global-local optimization scheme, to deal with challenges associated with the registration of multiple small fragments and limited imaging orientations in the operating room. The method is easy to use as minimum user interaction is required. Experiments on simulated fractures and two distal radius fracture phantoms demonstrate clinically acceptable target registration errors with capture range as large as 10 mm.     

Postoperative Calculation of Acetabular Cup Position Using 2-D–3-D Registration

A method to accurately measure the position and orientation of an acetabular cup implant from postoperative X-rays has been designed and validated. The method uses 2-D-3-D registration to align both the prosthesis and the preoperative computed tomography (CT) volume to the X-ray image. This allows the position of the implant to be calculated with respect to a CT-based surgical plan. Experiments have been carried out using ten sets of patient data. A conventional plain-film measurement technique was also investigated. A gold standard implant position and orientation was calculated using postoperative CT. Results show our method to be significantly more accurate than the plain-film method for calculating cup anteversion. Cup orientation and position could be measured to within a mean absolute error of 1.4 mm or degrees.     

Postoperative calculation of acetabular cup position using 2-D-3-D registration.

A method to accurately measure the position and orientation of an acetabular cup implant from postoperative X-rays has been designed and validated. The method uses 2-D-3-D registration to align both the prosthesis and the preoperative computed tomography (CT) volume to the X-ray image. This allows the position of the implant to be calculated with respect to a CT-based surgical plan. Experiments have been carried out using ten sets of patient data. A conventional plain-film measurement technique was also investigated. A gold standard implant position and orientation was calculated using postoperative CT. Results show our method to be significantly more accurate than the plain-film method for calculating cup anteversion. Cup orientation and position could be measured to within a mean absolute error of 1.4 mm or degrees.     

Gradient-based 2-D/3-D rigid registration of fluoroscopic X-ray to CT

We present a gradient-based method for rigid registration of a patient preoperative computed tomography (CT) to its intraoperative situation with a few fluoroscopic X-ray images obtained with a tracked C-arm. The method is noninvasive, anatomy-based, requires simple user interaction, and includes validation. It is generic and easily customizable for a variety of routine clinical uses in orthopaedic surgery. Gradient-based registration consists of three steps: 1) initial pose estimation; 2) coarse geometry-based registration on bone contours, and; 3) fine gradient projection registration (GPR) on edge pixels. It optimizes speed, accuracy, and robustness. Its novelty resides in using volume gradients to eliminate outliers and foreign objects in the fluoroscopic X-ray images, in speeding up computation, and in achieving higher accuracy. It overcomes the drawbacks of intensity-based methods, which are slow and have a limited convergence range, and of geometry-based methods, which depend on the image segmentation quality. Our simulated, in vitro, and cadaver experiments on a human pelvis CT, dry vertebra, dry femur, fresh lamb hip, and human pelvis under realistic conditions show a mean 0.5-1.7 mm (0.5-2.6 mm maximum) target registration accuracy.     

Markerless real-time 3-D target region tracking by motion backprojection from projection images

Accurate and fast localization of a predefined target region inside the patient is an important component of many image-guided therapy procedures. This problem is commonly solved by registration of intraoperative 2-D projection images to 3-D preoperative images. If the patient is not fixed during the intervention, the 2-D image acquisition is repeated several times during the procedure, and the registration problem can be cast instead as a 3-D tracking problem. To solve the 3-D problem, we propose in this paper to apply 2-D region tracking to first recover the components of the transformation that are in-plane to the projections. The 2-D motion estimates of all projections are backprojected into 3-D space, where they are then combined into a consistent estimate of the 3-D motion. We compare this method to intensity-based 2-D to 3-D registration and a combination of 2-D motion backprojection followed by a 2-D to 3-D registration stage. Using clinical data with a fiducial marker-based gold-standard transformation, we show that our method is capable of accurately tracking vertebral targets in 3-D from 2-D motion measured in X-ray projection images. Using a standard tracking algorithm (hyperplane tracking), tracking is achieved at video frame rates but fails relatively often (32% of all frames tracked with target registration error (TRE) better than 1.2 mm, 82% of all frames tracked with TRE better than 2.4 mm). With intensity-based 2-D to 2-D image registration using normalized mutual information (NMI) and pattern intensity (PI), accuracy and robustness are substantially improved. NMI tracked 82% of all frames in our data with TRE better than 1.2 mm and 96% of all frames with TRE better than 2.4 mm. This comes at the cost of a reduced frame rate, 1.7 s average processing time per frame and projection device. Results using PI were slightly more accurate, but required on average 5.4 s time per frame. These results are still substantially faster than 2-D to 3-D registration. We conclude that motion backprojection from 2-D motion tracking is an accurate and efficient method for tracking 3-D target motion, but tracking 2-D motion accurately and robustly remains a challenge.     

Intensity-based 2-D-3-D registration of cerebral angiograms

We propose a new method for aligning three-dimensional (3-D) magnetic resonance angiography (MRA) with 2-D X-ray digital subtraction angiograms (DSA). Our method is developed from our algorithm to register computed tomography volumes to X-ray images based on intensity matching of digitally reconstructed radiographs (DRRs). To make the DSA and DRR more similar, we transform the MRA images to images of the vasculature and set to zero the contralateral side of the MRA to that imaged with DSA. We initialize the search for a match on a user defined circular region of interest. We have tested six similarity measures using both unsegmented MRA and three segmentation variants of the MRA. Registrations were carried out on images of a physical neuro-vascular phantom and images obtained during four neuro-vascular interventions. The most accurate and robust registrations were obtained using the pattern intensity, gradient difference, and gradient correlation similarity measures, when used in conjunction with the most sophisticated MRA segmentations. Using these measures, 95% of the phantom start positions and 82% of the clinical start positions were successfully registered. The lowest root mean square reprojection errors were 1.3 mm (standard deviation 0.6) for the phantom and 1.5 mm (standard deviation 0.9) for the clinical data sets. Finally, we present a novel method for the comparison of similarity measure performance using a technique borrowed from receiver operator characteristic analysis.     

Brain shift estimation in image-guided neurosurgery using 3-D ultrasound

Intraoperative brain deformation is one of the most important causes affecting the overall accuracy of image-guided neurosurgical procedures. One option for correcting for this deformation is to acquire three-dimensional (3-D) ultrasound data during the operation and use this data to update the information provided by the preoperatively acquired MR data. For 12 patients 3-D ultrasound images have been reconstructed from freehand sweeps acquired during neurosurgical procedures. Ultrasound data acquired prior to and after opening the dura, but prior to surgery, have been quantitatively compared to the preoperatively acquired MR data to estimate the rigid component of brain shift at the first stages of surgery. Prior to opening the dura the average brain shift measured was 3.0 mm parallel to the direction of gravity, with a maximum of 7.5 mm, and 3.9 mm perpendicular to the direction of gravity, with a maximum of 8.2 mm. After opening the dura the shift increased on average 0.2 mm parallel to the direction of gravity and 1.4 mm perpendicular to the direction of gravity. Brain shift can be detected by acquiring 3-D ultrasound data during image-guided neurosurgery. Therefore, it can be used as a basis for correcting image data and preoperative planning for intraoperative deformations.     

Statistical distributions of DCT coefficients and their application to an interframe compression algorithm for 3-D medical images

Displacement estimated interframe (DEI) coding, a coding scheme for 3-D medical image data sets such as X-ray computed tomography (CT) or magnetic resonance (MR) images, is presented. To take advantage of the correlation between contiguous slices, a displacement-compensated difference image based on the previous image is encoded. The best fitting distribution functions for the discrete cosine transform (DCT) coefficients obtained from displacement compensated difference images are determined and used in allocating bits and optimizing quantizers for the coefficients. The DEI scheme is compared with 2-D block discrete cosine transform (DCT) as well as a full-frame DCT using the bit allocation technique of S. Lo and H.K. Huang (1985). For X-ray CT head images, the present bit allocation and quantizer design, using an appropriate distribution model, resulted in a 13-dB improvement in the SNR compared to the full-frame DCT using the bit allocation technique. For an image set with 5-mm slice thickness, the DEI method gave about 5% improvement in the compression ratio on average and less blockiness at the same distortion. The performance gain increases to about 10% when the slice thickness decreases to 3 mm.     

A comparison of similarity measures for use in 2-D-3-D medicalimage registration

A comparison of six similarity measures for use in intensity-based two-dimensional-three-dimensional (2-D-3-D) image registration is presented. The accuracy of the similarity measures are compared to a “gold-standard” registration which has been accurately calculated using fiducial markers. The similarity measures are used to register a computed tomography (CT) scan of a spine phantom to a fluoroscopy image of the phantom. The registration is carried out within a region-of-interest in the fluoroscopy image which is user defined to contain a single vertebra. Many of the problems involved in this type of registration are caused by features which were not modeled by a phantom image alone. More realistic “gold-standard” data sets were simulated using the phantom image with clinical image features overlaid. Results show that the introduction of soft-tissue structures and interventional instruments into the phantom image can have a large effect on the performance of some similarity measures previously applied to 2-D-3-D image registration. Two measures were able to register accurately and robustly even when soft-tissue structures and interventional instruments were present as differences between the images. These measures were pattern intensity and gradient difference. Their registration accuracy, for all the rigid-body parameters except for the source to film translation, was within a root-mean-square (rms) error of 0.53 mm or degrees to the “gold-standard” values. No failures occurred while registering using these measures     

Rapid Dynamic Image Registration of the Beating Heart for Diagnosis and Surgical Navigation

Dynamic cardiac magnetic resonance imaging (MR) and computed tomography (CT) provide cardiologists and cardiac surgeons with high-quality 4-D images for diagnosis and therapy, yet the effective use of these high-quality anatomical models remains a challenge. Ultrasound (US) is a flexible imaging tool, but the US images produced are often difficult to interpret unless they are placed within their proper 3-D anatomical context. The ability to correlate real-time 3-D US volumes (RT3D US) with dynamic MR/CT images would offer a significant contribution to improve the quality of cardiac procedures. In this paper, we present a rapid two-step method for registering RT3D US to high-quality dynamic 3-D MR/CT images of the beating heart. This technique overcomes some major limitations of image registration (such as the correct registration result not necessarily occurring at the maximum of the mutual information (MI) metric) using the MI metric. We demonstrate the effectiveness of our method in a dynamic heart phantom (DHP) study and a human subject study. The achieved mean target registration error of CT+US images in the phantom study is 2.59 mm. Validation using human MR/US volumes shows a target registration error of 1.76 mm. We anticipate that this technique will substantially improve the quality of cardiac diagnosis and therapies.      

A comprehensive cardiac motion estimation framework using both untagged and 3-D tagged MR images based on nonrigid registration

In this paper, we present a novel technique based on nonrigid image registration for myocardial motion estimation using both untagged and 3-D tagged MR images. The novel aspect of our technique is its simultaneous usage of complementary information from both untagged and 3-D tagged MR images. To estimate the motion within the myocardium, we register a sequence of tagged and untagged MR images during the cardiac cycle to a set of reference tagged and untagged MR images at end-diastole. The similarity measure is spatially weighted to maximize the utility of information from both images. In addition, the proposed approach integrates a valve plane tracker and adaptive incompressibility into the framework. We have evaluated the proposed approach on 12 subjects. Our results show a clear improvement in terms of accuracy compared to approaches that use either 3-D tagged or untagged MR image information alone. The relative error compared to manually tracked landmarks is less than 15% throughout the cardiac cycle. Finally, we demonstrate the automatic analysis of cardiac function from the myocardial deformation fields.     

Registration and tracking to integrate X-ray and MR images in an XMR facility

We describe a registration and tracking technique to integrate cardiac X-ray images and cardiac magnetic resonance (MR) images acquired from a combined X-ray and MR interventional suite (XMR). Optical tracking is used to determine the transformation matrices relating MR image coordinates and X-ray image coordinates. Calibration of X-ray projection geometry and tracking of the X-ray C-arm and table enable three-dimensional (3-D) reconstruction of vessel centerlines and catheters from bi-plane X-ray views. We can, therefore, combine single X-ray projection images with registered projection MR images from a volume acquisition, and we can also display 3-D reconstructions of catheters within a 3-D or multi-slice MR volume. Registration errors were assessed using phantom experiments. Errors in the combined projection images (two-dimensional target registration error--TRE) were found to be 2.4 to 4.2 mm, and the errors in the integrated volume representation (3-D TRE) were found to be 4.6 to 5.1 mm. These errors are clinically acceptable for alignment of images of the great vessels and the chambers of the heart. Results are shown for two patients. The first involves overlay of a catheter used for invasive pressure measurements on an MR volume that provides anatomical context. The second involves overlay of invasive electrode catheters (including a basket catheter) on a tagged MR volume in order to relate electrophysiology to myocardial motion in a patient with an arrhythmia. Visual assessment of these results suggests the errors were of a similar magnitude to those obtained in the phantom measurements.     

Optimal CT scanning plan for long-bone 3-D reconstruction

Digital computed tomographic (CT) data are widely used in three-dimensional (3-D) construction of bone geometry and density features for 3-D modelling purposes. During in vivo CT data acquisition the number of scans must be limited in order to protect patients from the risks related to X-ray absorption. The aim of this work is to automatically define, given a finite number of CT slices, the scanning plan which returns the optimal 3-D reconstruction of a bone segment from in vivo acquired CT images. An optimization algorithm based on a Discard-Insert-Exchange technique has been developed. In the proposed method the optimal scanning sequence is searched by minimizing the overall reconstruction error of a two-dimensional (2-D) prescanning image: an anterior-posterior (AP) X-ray projection of the bone segment. This approach has been validated in vitro on 3 different femurs. The 3-D reconstruction errors obtained through the optimization of the scanning plan on the 3-D prescanning images and on the corresponding 3-D data sets have been compared. 2-D and 3-D data sets have been reconstructed by linear interpolation along the longitudinal axis. Results show that direct 3-D optimization yields root mean square reconstruction errors which are only 4%-7% lower than the 2-D-optimized plan, thus proving that 2-D-optimization provides a good suboptimal scanning plan for 3-D reconstruction. Further on, 3-D reconstruction errors given by the optimized scanning plan and a standard radiological protocol for long bones have been compared. Results show that the optimized plan yields 20%-50% lower 3-D reconstruction errors     

GPU Accelerated Generation of Digitally Reconstructed Radiographs for 2-D/3-D Image Registration

Recent advances in programming languages for graphics processing units (GPUs) provide developers with a convenient way of implementing applications which can be executed on the CPU and GPU interchangeably. GPUs are becoming relatively cheap, powerful, and widely available hardware components, which can be used to perform intensive calculations. The last decade of hardware performance developments shows that GPU-based computation is progressing significantly faster than CPU-based computation, particularly if one considers the execution of highly parallelisable algorithms. Future predictions illustrate that this trend is likely to continue. In this paper, we introduce a way of accelerating 2-D/3-D image registration by developing a hybrid system which executes on the CPU and utilizes the GPU for parallelizing the generation of digitally reconstructed radiographs (DRRs). Based on the advancements of the GPU over the CPU, it is timely to exploit the benefits of many-core GPU technology by developing algorithms for DRR generation. Although some previous work has investigated the rendering of DRRs using the GPU, this paper investigates approximations which reduce the computational overhead while still maintaining a quality consistent with that needed for 2-D/3-D registration with sufficient accuracy to be clinically acceptable in certain applications of radiation oncology. Furthermore, by comparing implementations of 2-D/3-D registration on the CPU and GPU, we investigate current performance and propose an optimal framework for PC implementations addressing the rigid registration problem. Using this framework, we are able to render DRR images from a 256×256×133 CT volume in ～?24 ms using an NVidia GeForce 8800 GTX and in ～?2 ms using NVidia GeForce GTX 580. In addition to applications requiring fast automatic patient setup, these levels of performance suggest image-guided radiation therapy at video frame rates is technically feasible using relatively low cost PC architecture.     

A new validation method for X-ray mammogram registration algorithms using a projection model of breast X-ray compression

Establishing spatial correspondence between features visible in X-ray mammograms obtained at different times has great potential to aid assessment and quantitation of change in the breast indicative of malignancy. The literature contains numerous nonrigid registration algorithms developed for this purpose, but existing approaches are flawed by the assumption of inappropriate 2-D transformation models and quantitative estimation of registration accuracy is limited. In this paper, we describe a novel validation method which simulates plausible mammographic compressions of the breast using a magnetic resonance imaging (MRI) derived finite element model. By projecting the resulting known 3-D displacements into 2-D and generating pseudo-mammograms from these same compressed magnetic resonance (MR) volumes, we can generate convincing images with known 2-D displacements with which to validate a registration algorithm. We illustrate this approach by computing the accuracy for two conventional nonrigid 2-D registration algorithms applied to mammographic test images generated from three patient MR datasets. We show that the accuracy of these algorithms is close to the best achievable using a 2-D one-to-one correspondence model but that new algorithms incorporating more representative transformation models are required to achieve sufficiently accurate registrations for this application.