Refinement procedure for the image alignment in high-resolution electron tomography

High-resolution electron tomography from a tilt series of transmission electron microscopy images requires an accurate image alignment procedure in order to maximise the resolution of the tomogram. This is the case in particular for ultra-high resolution where even very small misalignments between individual images can dramatically reduce the fidelity of the resultant reconstruction. A tomographic-reconstruction based and marker-free method is proposed, which uses an iterative optimisation of the tomogram resolution. The method utilises a search algorithm that maximises the contrast in tomogram sub-volumes. Unlike conventional cross-correlation analysis it provides the required correlation over a large tilt angle separation and guarantees a consistent alignment of images for the full range of object tilt angles. An assessment based on experimental reconstructions shows that the marker-free procedure is competitive to the reference of marker-based procedures at lower resolution and yields sub-pixel accuracy even for simulated high-resolution data.     

Accurate marker-free alignment with simultaneous geometry determination and reconstruction of tilt series in electron tomography

An image alignment method for electron tomography is presented which is based on cross-correlation techniques and which includes a simultaneous refinement of the tilt geometry. A coarsely aligned tilt series is iteratively refined with a procedure consisting of two steps for each cycle: area matching and subsequent geometry correction. The first step, area matching, brings into register equivalent specimen regions in all images of the tilt series. It determines four parameters of a linear two-dimensional transformation, not just translation and rotation as is done during the preceding coarse alignment with conventional methods. The refinement procedure also differs from earlier methods in that the alignment references are now computed from already aligned images by reprojection of a backprojected volume. The second step, geometry correction, refines the initially inaccurate estimates of the geometrical parameters, including the direction of the tilt axis, a tilt angle offset, and the inclination of the specimen with respect to the support film or specimen holder. The correction values serve as an indicator for the progress of the refinement. For each new iteration, the correction values are used to compute an updated set of geometry parameters by a least squares fit. Model calculations show that it is essential to refine the geometrical parameters as well as the accurate alignment of the images to obtain a faithful map of the original structure.     

Automatic TEM image alignment by trifocal geometry

Here we propose a novel method for automatic, markerless, feature‐based alignment of TEM images suitable for electron tomography. The proposed method, termed trifocal alignment, is more accurate than the previous markerless methods. The key components developed are: (1) a reliable multi‐resolution algorithm for matching feature points between images; (2) a robust, maximum‐likelihood‐based estimator for determining the geometry of three views – the trifocal constraint – required for validating the correctness of the matches; and (3) a robust, large‐scale optimization framework to compute the alignment parameters from hundreds of thousands of feature point measurements from a few hundred images. The ability to utilize such a large number of measurements successfully compensates for point localization errors. The method was experimentally confirmed with electron tomography tilt series of biological and material sciences samples, consisting of from 40 to 150 images. The results show that, with this feature‐based alignment approach, a level of accuracy comparable with fiducial marker alignment can be achieved.     

稀疏表示下的噪声图像超分辨率重构

为了能够完成噪声图像的超分辨率重构,提出了一种基于稀疏表示的噪声图像超分辨率重构方法,可以同时完成图像去噪和超分辨率重构。首先,对样本图像和低分辨率图像进行块划分,建立样本库。其次,建立图像退化模型,采用相似样本加权平均的方式对输出的高分辨率图像块进行表示。根据输入的低分辨率图像块,计算样本块与输出的高分辨率图像块之间的相似性。提出了一种相似性描述方法,能够很好地解决噪声带来的影响。然后,采用相似性对稀疏编码优化模型进行惩罚,提出一种权值求解模型。模型可以自适应的搜索相似样本块而不需要预先设定相似块的个数。最后,求解权值,根据权值和样本块重构高分辨率图像块,并重构高分辨率图像。实验结果表明:所提出的方法较其它常见超分辨率算法的峰值信噪比可提高0.5dB左右,重构的图像细节更丰富,去噪效果更好,更适合实际应用。     

Computed three-dimensional reconstruction of median-eminence capillary modules: Image alignment and correlation

Image alignment is an absolute requirement for three-dimensional (3-D) reconstruction from serial sections, and Fourier correlation is the most powerful way to compute alignments. The rotational and translational components of misalignment can be corrected by an iterative correlation procedure, but for images having significant differences, alignment can fail with a likelihood proportional to the extent of the differences. We found that translational correction was determined much more reliably when low-pass filters were applied to the product transforms from which the correlations were calculated. Rotational corrections based on polar analyses of the auto-correlations of the images instead of on the images directly contributed to more accurate alignments. These methods were used to generate 3-D reconstructions of brain capillary modules from serial-section mosaics of digitized transmission electron micrographs.     

Alignment of AFM images using an iterative mathematical procedure

An iterative mathematical procedure for the alignment of sequentially recorded atomic force microscope images (AFM) is presented. The computer program is able to correct commonly observed drifts in vertical and lateral directions, rotations around a vertical or lateral axis and differences in scale. This method is applied on dissolution experiments of uranium dioxide (UO2) surfaces. Images recorded during in situ experiments, which are shifted probably due to thermal fluctuations, can be aligned with good accurancy. In a further approach the UO2 surface is marked by electron-beam-induced deposition (EBD or EBID) with microstructured reference points. The alignment can be distinctly improved using marked sample surfaces because of the characteristic shape of the markings, which do not change during the experiment. Furthermore, the markings can be used to find again a domain on a sample surface. Therefore, AFM images recorded before and after an ex situ experiment (e.g. treatment in corrosive medium for a longer period of time) can be aligned with a nanometer spatial resolution.     

Noise bias in the refinement of structures derived from single particles

One of the main goals in the determination of three-dimensional macromolecular structures from electron microscope images of individual molecules and complexes (single particles) is a sufficiently high spatial resolution, about 4 A, at which the interpretation with an atomic model becomes possible. To reach high resolution, an iterative refinement procedure using an expectation maximization algorithm is often used that leads to a more accurate alignment of the positional and orientational parameters for each particle. We show here the results of refinement algorithms that use a phase residual, a linear correlation coefficient, or a weighted correlation coefficient to align individual particles. The algorithms were applied to computer-generated data sets that contained projections from model structures, as well as noise. The algorithms show different degrees of over-fitting, especially at high resolution where the signal is weak. We demonstrate that the degree of over-fitting is reduced with a weighting scheme that depends on the signal-to-noise ratio in the data. The weighting also improves the accuracy of resolution measurement by the commonly used Fourier shell correlation. The performance of the refinement algorithms is compared to that using a maximum likelihood approach. The weighted correlation coefficient was implemented in the computer program FREALIGN.     

Circular harmonic averaging of rotary-shadowed and negatively stained creatine kinase macromolecules

The structure of mitochondrial creatine kinase is investigated by high-resolution shadowing at very low temperature and conventional negative staining. The electron microscopic images are analyzed with circular harmonic averaging, a method suited for the processing of single molecules. The rotational alignment and averaging is performed with the circular harmonic components, which allows data compression and several steps of noise reduction to be carried out within the averaging procedure. In addition, the symmetry can be deduced. For the mitochondrial creatine kinase, a fourfold symmetry is found that is compatible with the biochemical and biophysical characterization of the molecule.     

Projected potential profiles across interfaces obtained by reconstructing the exit face wave function from through focal series

An iterative method for reconstructing the exit face wave function from a through focal series of transmission electron microscopy image line profiles across an interface is presented. Apart from high-resolution images recorded with small changes in defocus, this method works also well for a large defocus range as used for Fresnel imaging. Using the phase-object approximation the projected electrostatic as well as the absorptive potential profiles across an interface are determined from this exit face wave function. A new experimental image alignment procedure was developed in order to align images with large relative defocus shift. The performance of this procedure is shown to be superior to other image alignment procedures existing in the literature. The reconstruction method is applied to both simulated and experimental images.     

Measurement of the surrounding liquid motion of a single rising bubble using a Dual-Camera PIV system

A new type of Particle Image Velocimetry technique, called “Dual-Camera PIV System”, was developed in order to achieve higher-accuracy measurement at a high time resolution. It is very difficult to measure precisely a complex flow field such as a gas–liquid two phase flow using PIV, because of the existence of a strong turbulence. In the conventional dynamic PIV, a time interval of two images required for analysis depends basically on a camera frame rate. A time interval of a set of PIV images affects the measurement accuracy significantly. Therefore, it is necessary to shorten the time interval of a set of PIV images as well as to achieve high frame rates. For this specific purpose, we developed a measurement system composed of two high speed cameras. The interval of two images obtained from each camera was controlled arbitrarily. Furthermore, a recursive cross-correlation method was adopted as PIV algorithm in order to achieve high spatial resolution. The interrogation areas were determined from the number density of PIV particles. The developed system was evaluated by cross-correlation coefficient and signal–noise (S/N) ratio. As the demonstration, the surrounding liquid motion in the vicinity of a single rising bubble was measured via this measurement system.     

Practical procedure for coma-free alignment using caustic figure

The practical procedure for coma-free alignment using a single defocused transmission electron microscopy (TEM) image is presented. Caustic figures observed in the defocused TEM image of a focused probe are utilized. Coma-free alignment can be carried out by coinciding a bright-field spot with the center of a caustic curve as observed in an underfocus TEM image. With this method, beam tilt misalignment is reduced to the sub-mrad order (e.g. 0.3mrad for 300kV FEG-TEM). This can be done without intentional beam tilting, an amorphous specimen, high-resolution TEM images, or fast Fourier transform for diffractogram or cross-correlation, which are used in previous methods. Residual coma aberration is detected using the multiple Bragg images of a known crystal. Similarity between the present coma-free alignment and well-known STEM alignment using shadow image is discussed.     

Optimization of EFTEM image acquisition by using elastically filtered images for drift correction

Because of its high spatial resolution, energy-filtering transmission electron microscopy (EFTEM) has become widely used for the analysis of the chemical composition of nanostructures. To obtain the best spatial resolution, the precise correction of instrumental influences and the optimization of the data acquisition procedure are very important. In this publication, we discuss a modified image acquisition procedure that optimizes the acquisition process of the EFTEM images, especially for long exposure times and measurements that are affected by large spatial drift. To alleviate the blurring of the image caused by the spatial drift, we propose to take several EFTEM images with a shorter exposure time (sub-images) and merge these sub-images afterwards. To correct for the drift between these sub-images, elastically filtered images are acquired between two subsequent sub-images. These elastically filtered images are highly suitable for spatial drift correction based on the cross-correlation method. The use of the drift information between two elastically filtered images permits to merge the drift-corrected sub-images automatically and with high accuracy, resulting in sharper edges and an improved signal intensity in the final EFTEM image. Artefacts that are caused by prominent noise-peaks in the dark reference image have been suppressed by calculating the dark reference image from three images. Furthermore, using the information given by the elastically filtered images, it is possible to drift-correct a set of EFTEM images already during the acquisition. This simplifies the post-processing for elemental mapping and offers the possibility for active drift correction using the image shift function of the microscope, leading to an increased field of view.     

基于区域划分的红外超分辨率重建

提出了红外超分辨率重建系统以获取高分辨率红外数据。首先,根据红外图像获取过程建立了数学模型,讨论了降采样、模糊、运动以及高斯噪声对红外系统的影响;在非退化特征提取的基础上提出了基于特征的亚像素配准算法,其根据所得到的非退化特征应用归一化均方根误差来估计两帧之间的亚像素位移。然后,分析了传统全变分因子在高分辨重建时的不足并对其进行改进;利用区域划分将图像划分为平滑区域和细节区域,并根据区域的不同情况自适应全变分因子,从而使细节区域不至于过平滑。最后,利用MM(Majorization Minimixation)算法对合成的低分辨率红外图像和真实红外图像进行了超锐度重建。与同类相关算法的比较实验显示:所提算法亚像素配准最大误差为0.09pixel,重建后的红外图像质量优于其他同类算法。所提算法可以对低分辨红外图像序列进行有效重建,具有配准精度高、重建图像细节丰富等特点,可应用于各种红外成像系统。     

Automated high-throughput electron tomography by pre-calibration of image shifts

Electron tomography is a versatile method for obtaining three‐dimensional (3D) images with transmission electron microscopy. The technique is suitable to investigate cell organelles and tissue sections (100–500 nm thick) with 4–20 nm resolution. 3D reconstructions are obtained by processing a series of images acquired with the samples tilted over different angles. While tilting the sample, image shifts and defocus changes of several µm can occur. The current generation of automated acquisition software detects and corrects for these changes with a procedure that incorporates switching the electron optical magnification. We developed a novel method for data collection based on the measurement of shifts prior to data acquisition, which results in a five‐fold increase in speed, enabling the acquisition of 151 images in less than 20 min. The method will enhance the quality of a tilt series by minimizing the amount of required focus‐change compensation by aligning the optical axis to the tilt axis of the specimen stage. The alignment is achieved by invoking an amount of image shift as deduced from the mathematical model describing the effect of specimen tilt. As examples for application in biological and materials sciences 3D reconstructions of a mitochondrion and a zeolite crystal are presented.     

A comparison of conventional Everhart-Thornley style and in-lens secondary electron detectors: a further variable in scanning electron microscopy

The secondary electron (SE) imaging of several samples across a range of scanning electron microscopes (SEM) and SE detectors under matched operating conditions has generated a highly variable image data set. Using microanalytical conditions (10-15?kV), images from in-column SE detectors reveal the presence of surface films and contaminants that are invisible to conventional Everhart-Thornley SE detectors under the same conditions. Data from studying the effects of working distance, the image resolution derived through contrast transfer function analysis and electrostatic mirror imaging of the SE detectors in operation combine with other studies to suggest that the classically defined SE1 component can be separated from other SE components. SE images obtained by tailored mechanical design and energy-filtering will provide SE images with probe-sized resolution and dominated by surface detail currently only seen in low-voltage SEM, potentially even from thermionic-sourced columns.     

Spatial filtering of electron micrographs of negatively stained alpha-amylase crystals

Images of the alpha-amylase molecule from pig pancreas have been obtained in two projections by the application of spatial filtering techniques to electron micrographs of negatively stained microcrystals of the enzyme. The Fourier filtering was performed on a PPD 11/40 digital computer after microdensitometry on an Optronics P-1000 photoscan system. The set of procedures and programs we employed are described herein. We compared these images with equivalent images obtained by three-dimensional X-ray diffraction analysis using conventional isomorphous replacement at approximately twice the resolution. We find that there is quite good agreement between the two kinds of images, and that a number of gross structural features compare quite well. We conclude that the combination of electron microscopy with digital spatial filtering when applied to protein microcrystals yields very respectable results.     

Invariant classification of molecular views in electron micrographs

Biological macromolecules can exhibit many different orientations in electron microscopical preparations. In particular in vitreous-ice-embedded specimens, the number of different views can be high. Existing techniques of analysis require the alignment of the molecular views relative to one or more reference images with cross-correlation ("matched filtering") techniques and are somewhat unsatisfactory because of the high noise level and the large number of different views in such images. We here propose a method in which first rotation-, translation- and mirror-invariant functions are derived from the large set of input images. These functions are subsequently classified automatically using multivariate statistical classification techniques. The different molecular views in the images can therewith be found without bias, provided that a statistically significant number of copies of the views are present in the data set. The basic ideas are exemplified with realistic model data.     

A novel "ghost"-free tomographic image reconstruction method applicable to rotary-shadowed replica specimens

Electron tomography by conventional filtered back-projection is often seriously impaired by anisotropic resolution due to unavoidable limitation in specimen tilt-angles. We propose a new approach to overcome the problem for thin film-like replica-type specimens in which internal density is supposed as homogenous and contiguously distributed, by imposing a reasonable constraint of density-existing region in the reconstruction procedure. The objects were approximated as a distribution of binary voxels and the intensity of the projected images being proportional to the thickness along the projection ray. The new reconstruction algorithm consists of initial determination of approximate constraint region by a topographic analysis by stereo-photogrammetry, followed by iterative computation to find the unique solution of simultaneous equations, so that all the intensity distribution in tilt-series images are included within pre-determined voxel arrangement. During a trial run with a new methodology, we realized its significantly advantageous feature that much less number of projection images than conventional back-projection is required to perform the reconstruction of almost equivalent quality. Here, we show the performance of this novel algorithm by 3-D reconstruction of quick-freeze deep-etch replica specimens without any trace of spurious ghosting caused by missing-wedge problems.     

Initial resolution measurements of miniaturized electrostatic lenses for LVSEM

Theoretical and experimental investigations on miniaturized electrostatic lenses for high-resolution low-voltage scanning electron microscopical applications are presented. The electron optical column consists of a Schottky emitter including the extraction anode and a miniaturized three electrode lens consisting of conventional (electron microscopical) platinum apertures. The lens has been optimized with respect to resolution and a value of about 5 nm at a working distance of 1 mm and a beam energy of 1 keV is predicted by simulation. Details on the resolution measurements are presented. An initial experimental value amounts to 31 nm. Specific problems encountered during building and alignment of the lens and measuring the resolution are discussed as well.     

Simulated SEM images for resolution measurement

Resolution is a key performance metric, which often defines the quality of a scanning electron microscope (SEM). Traditionally, there is the subjective measurement of the distance between two points on special "resolution" samples and there are several computer-based resolution-calculation methods. These computer-based resolution-calculation methods are much more precise than direct measurement, but none of them can currently be considered an objective way of measuring the resolution. The methods are still under development; therefore, objective testing is necessary. One approach to algorithm testing is to use simulated images. Simulated images are very useful for this purpose because they can be well-defined in all parameters unlike the real SEM images. Simulated images can be generated that closely mimic the gold-on-carbon SEM test sample images that usually consist of bright grains on a dark background. Simulation can account for edge effect, roughness of the substrate, different focusing, drift and vibration, and noise. Shapes, positions, and sizes of the grain structures are random. The simulated images can be then used for testing the resolution-calculation methods, especially for finding how the particular properties of SEM images affect the resultant instrument performance and image resolution. To support this testing, NIST has developed and made available a reference set of simulated SEM images generated using the methods described in this article.