Quantitative Image Processing for Three-Dimensional Episcopic Images of Biological Structures: Current State and Future Direction

1. Introduction

Episcopic microscopy refers to imaging techniques that utilize serial sectioning and block-face imaging to create high-resolution large-volume three-dimensional (3D) datasets. High Resolution Episcopic Microscopy (HREM) and its associated techniques: Optical HREM, Episcopic fluorescent image capture (EFIC) and multi-fluorescent HREM (MF-HREM), require samples to be embedded in a supporting material to allow automated sectioning at the micron scale. After each section is removed, an image is captured of the cut surface, enabling a 3D image stack to be constructed. While this review focuses on optical episcopic imaging, it is worth noting that an equivalent methodology exists in electron microscopy: serial block face scanning electron microscopy [1]. In contrast to serial histology, the image stack produced by episcopic microscopy is inherently aligned and therefore requires no registration step.

Each of the methods above have established experimental protocols across a range of biological samples; and, in the case of optical HREM and EFIC, have been used to test biological and medical hypotheses for over a decade [2]. Alongside the development of experimental protocols, there has also been development of image processing tools to extract quantitative information from HREM images.

Much of the quantitative analysis of HREM images to date has relied upon manual annotation of structures of interest, for example, to extract tissues volumes, surface areas or number of anomalies. As HREM can be a high-throughput imaging modality (particularly for mouse embryo studies), and the datasets produced may be very large, manual image processing can present a bottleneck both in terms of both computational resource and researcher time.

In this review we briefly outline the goals and methods of quantitative 3D image processing and present results from a systematic review into the current practices for image processing approaches in the episcopic imaging literature. Additionally, we highlight obstacles preventing wider use of quantitative image analysis. Finally, we propose avenues for further research and suggest what future developments we may expect.

1.1. Background on Episcopic Imaging Techniques

HREM was originally developed as a high-throughput platform for phenotyping transgenic mouse models [2,3,4] and has since been developed into a commercial system (Optical HREM, Indigo Scientific). For these embryo phenotypic studies eosin was added to the embedding medium, with image contrast resulting from eosin blocking (when bound to eosinophilic proteins, the fluorescence of unbound eosin is inhibited or frequency-shifted). This produces images with an appearance similar to the inverse of a traditional eosin staining in histology. Similarly, imaging of LacZ expression through x-gal staining has also been used in conjunction with eosin-HREM, using a secondary filter set. As with Eosin contrast, x-gal staining blocks the native fluorescence of the resin embedded sample.

Episcopic Fluorescence Image Capture (EFIC) is a variant of HREM in which samples are embedded in non-fluorescent resin and contrast is provided solely by the native autofluorescence of the tissue [5]. More recently MF-HREM was developed to enable imaging of multiple targeted fluorescent stains in a similar manner to other 3D optical imaging modalities such as light sheet or serial two-photon microscopy [6].

There are also variants of episcopic imaging which utilise cryosectioning to circumvent some of the difficulties of standard HREM sample preparation [7,8]. Rather than dehydrating and embedding samples in resin, cryosectioning episcopic microscopy requires that the samples are frozen to harden them and enable accurate sectioning, which is more time efficient and compatible with a greater number of fluorescent stains.

Applications of HREM and its variants have included large-scale embryo phenotyping to build up an atlas of embryonic-lethal gene knockouts (`Deciphering the Mechanisms of Developmental Disorders (DMDD)’ [4], assessing blood vessel morphology in human skin biopsies [9,10,11], in addition to other use cases where a large 3D field of view is essential.

1.2. The importance of quantitative image analysis

Quantitative image analysis is required for statistical analysis of results, allowing biological hypotheses to be tested and results to be reported with confidence. Moreover, quantification can be used to extract more information from image data and, when automated, facilitate high throughput analysis. Well established quantification pipelines allow researchers to standardise their image analysis, and the objectivity of quantitative analysis makes for more robust interpretations of data. Finally, quantised results can be used for further experiments such as in-silico modelling or training of a machine learning model.

Performing quantitative analysis on the large 3D data sets typically produced by episcopic imaging is a non-trivial task that requires specialised approaches depending on the data type and the aims of the analysis.

1.3. Methods of quantitative imaging

There are a few key methods for quantification that are relevant to episcopic imaging data analysis; as the pixel intensities of episcopic images do not correlate with biological properties (unlike techniques such as Bioluminescence Microscopy), analysis is largely dependent on image annotation. Annotation can take the form of object detection or segmentation. Object detection usually takes the form of a bounding box or arrow identifying the feature of interest, while segmentation labels every pixel within the feature as belonging to that feature (where this is done within the feature of interest only, it is referred to as instance segmentation; semantic segmentation requires every pixel in the image to be assigned to a class, e.g. vessel or background). Object detection is generally easier to achieve, but segmentation is more information-dense. Once an image is annotated, further analyses can be performed, for example object counting, distribution and morphometry of objects.

Annotation can be performed manually, by tracing or highlighting objects, or by computational means. Widely used methods include pixel intensity-thresholding, filtering (for example, Frangi filtering for blood vessel enhancement [12]) and, more recently, machine learning. The processing time, labour and level of expertise required varies depending on the method. Fully manual segmentations performed by a highly trained expert are generally considered the gold standard, but are time consuming and can vary between operators. Semi-automated methods may require expertise to fine-tune, whilst fully automated methods do not require human intervention, making them most suited to batch analysis. The ideal image processing pipeline is one which is fully automated to facilitate high throughput, objective results that can be replicated by all users with the same software implementation.

Figure 1. Here a cluster of Paracoccidioides brasiliensis cells are annotated in a photomicrograph (source: CDC Public Health Image Library) by three different methods: object detection, instance segmentation and semantic segmentation. Object detection results in a region of interest (red border, right panel) while instance segmentation produces a mask of the area of each cell (colored regions, center panel) and semantic segmentation produces a binary image with each pixel labelled as ‘background’ (black) or ‘cell’ (white) (left panel).

Figure 1. Here a cluster of Paracoccidioides brasiliensis cells are annotated in a photomicrograph (source: CDC Public Health Image Library) by three different methods: object detection, instance segmentation and semantic segmentation. Object detection results in a region of interest (red border, right panel) while instance segmentation produces a mask of the area of each cell (colored regions, center panel) and semantic segmentation produces a binary image with each pixel labelled as ‘background’ (black) or ‘cell’ (white) (left panel).

2. Methods

Literature searches were performed in Google Scholar using the term "high resolution episcopic microscopy" OR "episcopic fluorescence image capture" (case ignored) with a date range of January 2011 to December 2022. 463 searches were initially returned. Based on the title, abstract and methods for these results, review articles were excluded alongside articles which did not use HREM, EFIC or a related episcopic technique. This left 167 relevant articles.

Subsequent exclusion criteria removed articles which did not use biological samples or were protocol descriptions only, without new experimental data or new image processing methods. Conference abstracts without enough detail to extract meta-analysis data were also excluded. From the remaining 123 articles, ten features were extracted:

• Year
• Contrast type
• Structure under investigation
• Model organism
• Was annotation was performed?
  • If yes: was this segmentation, object detection, or both?
• Was quantitative data presented?
• Method of annotation /quantification – organized into categories determined during analysis
• Software used
• Was the raw image data deposited in an online repository?
• The type of publication (paper, conference paper, thesis)

3. Results

3.1. Trends in prevalence of quantification

Since 2011, episcopic microscopy data has featured in 123 publications, including 115 journal papers, three conference papers, two correspondences, two doctoral theses and one case report. By far the most widely used contrast method has been eosin labelling (featured in 90 papers), sometimes combined with acridine orange or X-gal labelling (Figure 2a,b). EFIC was the second most used contrast method, being featured in 24 papers. Four papers used fluorescence labelling, each of which included segmentation of tissues and quantitative results. Three papers employed cryosectioning, of which two featured quantitative analysis. Data was made available online for 27 papers (22%), 17 of which were a part of the Deciphering Mechanisms of Developmental Disorders (DMDD) project [13].

The most commonly imaged species was mouse (85 papers), followed by human tissue (20 papers). There has been a strong focus on visualizing mouse embryos, in particular for assessing cardiac morphology (represented in 55 papers).

Over these past ten years, the number of papers published has averaged ten per year, with an increase in the last two years to 17 in 2021 and 16 in 2022. Of these papers, the proportion which provide quantitative image data has shifted: beginning at 75% of publications between 2011 and 2012, it dropped to below 50% from 2013 and 2015, then increased again in 2016 to 65% and remained at approximately that level until 2021. The proportion dropped again in 2022 to just 38% (Figure 2c). In total, just over half of the papers published since 2011 (56%) provided quantitative results, but where annotation was performed, that number increased to 70%, signifying the important role annotation plays in quantification. A quarter of the papers published in this time used episcopic imaging for visualization without annotation; often this data was presented alongside other experimental techniques such as in-vivo imaging, immunohistochemistry and genetic sequencing [14,15,16]. Segmentation of episcopic data has been increasingly featured in the literature over the last few years, apparently being favored over object detection since 2016.

3.2. Trends in methods of quantification

Through the analysis of the last decade of HREM papers, we have identified seven broad categories of quantitative analysis being performed on episcopic image data. We define these categories as:

• Scoring /counting of abnormalities (as defined in an atlas)
• Object counting (no agreed ontology needed)
• Registration based
• Hand 2D measurements of anatomic features after reslicing data
• 3D morphological analysis, for example: volumes, networks
• Fractal analysis of heart (standardized method)
• Other (for example, orientation of fibers)

3D morphological measurements were the most commonly applied method of quantification, occurring in 25 papers and becoming more prevalent over time (Figure 3a,b). This morphological analysis was often performed in conjunction with other methods such as scoring or counting of abnormalities. A potential limiting factor for use of this method is the need for segmentation, which was completed by hand in over half of the papers reviewed here. Simple segmentations were employed in many of these papers to remove background and enable measurements of volumes, surfaces areas and distances, but more advanced morphological analysis requires segmentation of complex tissue structures. For example, Sweeney et. al. [17] uses a fully segmented vessel network to create a graphical representation of the vascular connectivity. Producing such a segmentation, when done manually, is extremely labor intensive.

Scoring or counting of abnormalities was the second most widely employed method, appearing in 19 papers, and notable increased in use from 2016 onwards. This is potentially associated with the publication of the open source DMDD database and papers describing the detailed methodology for performing abnormality scoring based on the mammalian phenotyping dictionary. This trend therefore highlights the utility of open-source data platforms and widely available software for establishing a reproducible image quantification protocol. Scoring/ counting is also a method that does not require segmentation and thus can be more quickly achieved by an expert observer. The vast majority of papers employing a scoring of counting method did so manually. The drawbacks of these methods are that they may require a high degree of expertise to recognize abnormalities, and where there is a binary output for a given sample (e.g. normal versus abnormal) a large number of samples must be assessed to allow meaningful quantification.

Seven papers included fractal analysis of the heart. This is a standardized method that produces a quantitative measure of complexity or surface roughness (the fractal dimension, FD) wieldy used in cardiology [18]. Other quantification methods began appearing in papers from 2017 onwards. These include structure tensor analysis for calculating myocyte and neuron orientation [6,19,20], skeletonization algorithms for producing a graphical representation of vasculature [6,21], and measures of asymmetry [22].

3.3. Methods of image annotation

Manual annotation was the most common method for both object detection and segmentation. Overall, 76% of papers presenting quantitative image analysis relied, at least in part, on manual annotation (70% for papers employing segmentation, 89% for papers using object detection) (Figure 4b). Automated and semi-automated methods were less widely used: intensity thresholding was employed in six papers, while vasculature was segmented via Frangi filtering in two papers [23,24]. Some papers employed as specialized methods such as the MOST tracing algorithm in Vaa3D for neuron tracing [6,25] and the Vascular Modelling Toolkit (VMTK) for vessel segmentation [26].

A wide range of software was used to facilitate episcopic image analysis. Of the 20 software packages listed across all the reviewed papers, the most widely used were Amira (Thermofisher) (48 papers), OsiriX (Pixmeo) (24 papers) and Fiji/ ImageJ (18 papers) (Figure 4a). Amira was particularly popular for segmentation, while OsiriX was used more often for object detection. Commercial software continues to be preferred over open source alternatives: there were 35 instances of open source software being used for image analysis, compared to 99 instances of commercial software use. 22 papers did not list the software used for their analysis.

In three papers, deconvolution algorithms were used to overcome the image blurring caused by out-of-plane light from below the sample surface [6,24,25]. The asymmetric nature of this blurring necessitated a custom deconvolution algorithm in each case. Walsh et. al. [25] presents a detailed pipeline for asymmetric deconvolution and demonstrates how this enables more accuracy automated segmentation.

Asymmetric blurring is one of the reasons that automated methods might struggle to correctly annotate episcopic images. Additionally, where fluorescent labels are used, the fact that samples are whole-mount stained means they are likely to suffer from heterogenous levels of staining (due to poor penetration of the staining molecule into the center of the sample). The different forms of episcopic imaging also require different tools for annotation. This could explain the high proportion of papers relying on manual annotation, which represents a significant bottleneck for episcopic image analysis. This result highlights the need for new automated methods of annotation designed specifically for episcopic image data. One area of research that could address this need is future is machine learning-based image annotation.

4. Discussion

Publication of episcopic imaging data has increased over the last decade, demonstrating that episcopic microscopy continues to be a valuable tool for analyzing biological samples. Quantitative analysis of this data has also become more prevalent, with an increasing range of methodologies being used to extract quantitative data from episcopic images. However, the proportion of papers providing quantitative image analysis is still fairly low, averaging 54%, possibly due in part to the difficulties associated with image annotation: episcopic images are mostly annotated by hand, which can be extremely labor intensive. While some areas of analysis follow standardized methods (for example, fractal analysis of the heart and scoring of abnormalities according to an atlas) much of the quantification being performed is not standardized across research groups. 20 different software packages are used for analysis, most of which are commercially licensed and potentially expensive, while 22 papers did not list the software used. This makes reproducing and comparing results challenging, especially as different software packages provide different tools for visualization (for example, arbitrary reslicing) which can influence the manual segmentation produced.

6. Future Directions

For quantitative episcopic imaging to continue to grow as a technique, future work must focus on automated image annotation and standardized pipelines for quantitative analysis. Given recent trends, it is likely we will see continue to see more use of complex or multi-stage methodologies, for example segmentations that feed in to computer simulations. In these cases, a reproducible analysis pipeline is even more crucial to provide confidence in the analysis results. Making data easily available, using open source software, and clearly defining the analysis pipeline used in a published study are all ways researchers can contribute to more standardized and reproducible quantification methodologies. One exciting avenue for further research is machine learning-based image annotation. Episcopic image data is a prime candidate for machine learning (ML) approaches due to the availability of data in online repositories that could be used for training. The development of new tools, such as ML models, for episcopic image analysis will expand the use of episcopic imaging and create further possibilities for new applications.