Trackerless 3D Freehand Ultrasound Reconstruction: A Review

1. Introduction

Two-dimensional ultrasound (2D US) has established itself as an invaluable imaging modality in clinical settings due to its low cost, portability, and non-invasive nature. Its real-time visualization capability offers significant utility in intraoperative procedures, providing clinicians with essential anatomical and functional information. Despite these advantages, 2D US is hindered by limitations such as limited spatial orientation, lacking valuable 3D context, and its dependence on the operator’s experience and interpretive skills. These constraints underscore the inherent variability and the potential for subjectivity in 2D US [1]. The introduction of three-dimensional ultrasound (3D US) marks a significant advancement, addressing many of these limitations. Specifically, 3D US provides a more comprehensive visualization of anatomical regions of interest (ROIs) and provides the flexibility of examining the acquired data from multiple viewpoints post-acquisition. This allows for more accurate quantitative measurements and additionally reduces the variability due to operators. Moreover, 3D US allows for integration with other 3D imaging modalities, such as MRI or CT. This enables a multifaceted view of the patient’s anatomy, enriching the diagnostic process. With these advancements, 3D US extends the potential of 2D US across a broader spectrum of clinical applications, like surgical planning, monitoring of treatment progression, and a wide range of intraoperative and interventional tasks [2].

The path to realizing 3D US reconstructions has seen various methodologies, including the use of 3D US transducers, external tracking, and image-based reconstruction methods. 3D US transducers employ 2D crystal arrays rather than traditional 1D crystal arrays. This design facilitates the electronic steering and focusing of the US beam in multiple dimensions, thereby enabling the real-time acquisition of 3D volumetric data. However, since the transducer elements are spread across two dimensions, this configuration results in fewer elements in each dimension. Therefore, to maintain a sufficient spatial resolution the imaging system must limit the coverage area, resulting in a smaller field of view (FOV), particularly at greater depths. External tracking devices such as mechanical, optical, or electromagnetic (EM) systems are designed to position the US probe in 3D space. These systems are prone to artifacts caused by magnetic interference or optical occlusion and have constraints in the range of motion. Considering the high costs and cumbersome set-up of 3D US transducers and external tracking systems, their suitability for certain clinical applications like intraoperative and point-of-care US (POCUS) is diminished [3,4].

Image-based freehand 3D US represents a promising alternative, eliminating the need for both external tracking devices and 3D transducers, thereby capitalizing on the potential for integration with cost-effective, portable and sometimes wireless, US devices [5]. The computational challenge of these systems lies in the reconstruction process, which necessitates an accurate estimation of the probe’s trajectory, divisible into in-plane and out-of-plane (elevational) movements. While in-plane movements are more readily quantified, out-of-plane estimations remain complex. Prior research on out-of-plane motion dates back to seminal work by Chen et al. [6] and has been mainly based on speckle noise, the granular gray-scale textures in B-mode US images. Speckle decorrelation methods map the transformation between neighboring US images to the correlation of their speckle patterns, i.e. the higher the speckle correlation, the lower the elevational distance between neighboring frames [7]. Under Rayleigh scattering conditions, where the size of these scatters is much smaller than the wavelength of the sound waves, this speckle pattern is theoretically predictable. However, when applied to dynamic clinical scenarios, these models often fall short, as speckle variability and real tissue movement introduce errors that accumulate, leading to drift and a compromise in accuracy [8,9,10].

Recent advancements in artificial intelligence (AI) and especially deep learning (DL) techniques have expanded the horizons for extracting detailed information from image data [11]. This has enabled significant progress in overcoming the intrinsic difficulties of image-based 3D US reconstruction. While traditional tracking technologies that require external reference are less suited to point-of-care and intraoperative settings, inertial measurement units (IMUs) have emerged as a viable and less obtrusive alternative for position tracking. These devices integrate a tri-directional magnetometer, a gyroscope, and an accelerometer into compact units. This offers a favorable balance between hardware independence and the need for positional information in trajectory reconstruction. With the growing adoption of IMU technology in clinical devices and developments in DL techniques, image-based reconstruction methods without external tracking are becoming increasingly viable. This approach additionally aligns with the ongoing shift towards more accessible and efficient POCUS applications.

Consequently, the aim of this systematic review is to identify current state-of-the-art methods for freehand 3D US reconstruction that forgo the need for external tracking devices. This review will critically assess various experimental setups, data acquisition strategies, reconstruction methodologies, and their outcomes to define the state-of-the-art in this rapidly advancing field.

2. Materials and Methods

2.1. Literature Search

2.1.1. Search Strategy

A systematic review was conducted according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines [12]. A literature search was carried out across the PubMed, Scopus, and IEEE Xplore databases on February 18th, 2024. Only studies published from 2014 onwards were taken into account to ensure the focus remained on advanced reconstruction methods, particularly those leveraging more sophisticated machine learning (ML) and DL algorithms instead of rule-based techniques [2,13]. The search string encompassed the main concepts of “freehand”, “ultrasound”, “three-dimensional”, and “Artificial Intelligence”, supplemented with synonyms and related keywords. The detailed search string employed for all databases can be found in Appendix A. An additional snowballing method was utilized, examining the references of the selected papers. All duplicates were removed during the search process.

2.1.2. Eligibility Criteria

Studies were considered eligible if they met the following inclusion criteria: (I) focus must be on 3D US reconstructions, encompassing trajectory reconstruction of the US probe, volume reconstructions for regions of interest (ROIs), or both, (II) data employed for 3D reconstruction must be derived from 2D US, (III) 2D US must be freehand or handheld acquired, thereby excluding the use external tracking but permitting IMU.

Studies were excluded with the following criteria: (I) lack of methodological descriptions for the 3D reconstruction process or those that do not specify the tools or software utilized, (II) employing additional imaging modalities or external tracking as input data for reconstruction, with the exception of IMU, (III) preliminary publications conducted by the same research team as subsequent included publications presenting only minor enhancements, such as hyperparameter adjustments or minor dataset expansions, (IV) full text is inaccessible, and (V) full text not written in English. If studies presented multiple datasets or methods, only the results of the datasets and methods aligned with the eligibility criteria were considered.

2.1.3. Study Selection Process

Following the literature search, all identified studies were initially screened based on their titles and abstracts. Papers that did not adhere to the eligibility criteria were excluded. The full text was retrieved for the remaining papers and screened for eligibility based on complete texts. Papers identified through snowballing were similarly assessed. Each assessment was meticulously carried out by a single reviewer. The included studies were categorized based on the focus of 3D reconstruction: estimating the complete 3D trajectory of the US probe, reconstructing a volume within an ROI without tracking the entire sweep trajectory, or addressing both objectives.

2.2. Data Extraction

2.2.1. Study Characteristics

Data extraction of study characteristics from the included articles focused on three main aspects: datasets and acquisition, reconstruction methods, and their evaluation by outcome measures. For the datasets employed, details regarding the clinical anatomy, as well as the number of sequences, frames, and subjects, were extracted. Furthermore, more detailed sweep characteristics related to the included sequences in the datasets were extracted, such as the motion trajectory, length of the sweep, type of US probe, and available acquisition parameters. Additionally, the methodology used to establish ground truth data was extracted. For the 3D reconstruction methods, information regarding the utilized data inputs, preprocessing methods, network architecture and components for trajectory reconstruction or volume reconstruction, and loss functions was extracted. Lastly, information regarding the robustness of the method was extracted.

2.2.2. Quantitative Outcome Measures Of 3D Reconstruction Methods

For each study, relevant quantitative outcome measures were extracted for the proposed reconstruction method and important ablation experiments. These ablation experiments involve systematically removing or modifying components of a model to evaluate their impact on performance, thereby elucidating the contribution of individual elements within the model. Evaluation of the reconstruction methods entailed extraction of trajectory reconstruction errors, volume reconstruction errors, and the computation time. If none of the following described outcome measures were reported within a study, the main outcome measure deemed relevant was extracted.

Trajectory reconstruction errors

Evaluation of trajectory errors involved two principal groups of metrics:

1) Error between successive frames: The reconstruction of the trajectory relies on the prediction of transformation parameters between successive frames. Consequently, the most direct metric to gauge model performance is the Mean Absolute Error (MAE) between predicted and ground truth transformation parameters across a sequence of N number of frames, as outlined in Equation 1:

$$\mathrm{MAE}(t_x,t_y,t_z,{\theta }_x,{\theta }_y,{\theta }_z)={\displaystyle \frac{1}{N}}\sum _{k=1}^n\left|\overrightarrow{p}\left(T_k{(i\to j)}^{\mathrm{pred}},T_k{(i\to j)}^{\mathrm{GT}}\right)\right|$$

(1)

Here, $\left|\overrightarrow{p}\left(A\right)\right|$ computes the element-wise absolute difference between each corresponding parameter of the predicted $T_k{(i\to j)}^{\mathrm{pred}}$ and ground truth $T_k{(i\to j)}^{\mathrm{GT}}$ transformation matrices for each pair (k) of successive US frames (i → j). This matrix consists of the translation ($t_x$, $t_y$, $t_z$) and rotation (${\theta }_x$, ${\theta }_y$, ${\theta }_z$) parameters. The MAE was considered parameter-wise for all transformation parameters or in subsets denoting only translation, orientation, or out-of-plane parameters.

In addition to element-wise parameter analysis, error measurement was conducted based on the spatial location of frames denoted as the frame error. This method calculates the Euclidean distance between the predicted and ground truth frame locations, averaged over the four corners of each frame. This approach offers a broader perspective on the positional accuracy of the frame sequence without focusing solely on individual transformation parameters.

2) Drift measures: Another key metric in trajectory reconstruction is the Final Drift (FD), defined by the Euclidean distance between the positions of the center of the last frame of the estimated trajectory and the last frame of the ground truth trajectory. Since the prediction of the trajectory of the sweep is based on a transformation between successive frames, errors accumulate over the length of the sweep. This accumulated error is reflected in the metric FD. Additionally, the Final Drift Rate (FDR) (%) was extracted which is the FD normalized by the ground truth length of the sweep. Important to note is that a minimal FD does not indicate a satisfactory reconstruction for complex scan strategies such as a loop scan, requiring additional evaluation metrics.

Volume reconstruction errors

For volume reconstruction errors, quantitative metrics such as Dice overlap and the absolute difference in volume, in milliliter, were extracted. These measures assess the degree of overlap and volumetric accuracy between the reconstructed volume and the ground truth, providing a robust framework for evaluating the performance of 3D reconstruction algorithms.

2.2.3. Analysis of Ablation Experiments

Analyses were conducted on two optimization strategies for reconstruction methods to evaluate the impact on outcome measures by ablation experiments. This included one subgroup that focused on optimizing models incorporating multiple data types, such as IMU alongside US frames. Another subgroup aimed to incorporate additional contextual or temporal information by analyzing more extensive US sequences rather than limiting them to just two adjacent frames. The relevant outcome measures previously discussed were evaluated before and after incorporation, quantifying the improvement in terms of percentage.

3. Results

3.1. Study Selection

An initial search across PubMed, IEEE Xplore, and Scopus identified 79 potentially relevant publications. After removing 21 duplicates, eligibility of the remaining 58 articles was assessed based on our predefined inclusion and exclusion criteria. Of these, 29 were excluded after title and abstract screening and 19 were excluded after full-text review, leaving 10 articles for inclusion. Additionally, four studies were identified through snowballing after eligibility assessment, culminating in a total of 14 reports that were included for data extraction. The study selection process is shown in Figure 1.

3.2. Study Characteristics

Table 1 presents a comprehensive summary of the characteristics extracted concerning the objectives, datasets, acquisition techniques, and reconstruction methods utilized in the studies. The majority of the studies exclusively focus on the evaluation of the full 3D US trajectory reconstruction; despite the ultimate aim being volume reconstruction of a ROI within the trajectory [14,15,16,17,18,19,20,21,22]. Two studies evaluated 3D trajectory reconstruction and additionally visualized the ROI, specifically arteries, within that reconstruction, albeit lacking quantitative assessments of the volume reconstruction [23,24]. Two others quantitatively assessed both trajectory and ROI volume reconstructions, specifically targeting the thyroid and prostate [25,26]. A singular study focused on directly estimating the volume of the cerebral ventricle system (CVS), bypassing trajectory reconstruction [27].

3.2.1. Data Acquisition And Datasets

From the selected studies, a total of 23 datasets adhered to the eligibility criteria of 2D images and were considered for further evaluation. It is important to note that due to the challenges posed by speckle decorrelation theory in real tissue, some studies included phantom datasets [14,17,23], while the remainder of the studies employed clinical datasets. As presented in Figure 2, the clinical datasets show a variety of different anatomies, paving the way for diverse clinical applications. Across the datasets included, there is a large range in the number of sequences utilized (from 9 to 619, $\mu$=194.08 and $\sigma$=213.29), which is also applicable to the number of frames and subjects. This directly influences the data volume available for model training and testing and the anatomical variety between sequences. Seventeen datasets reported on the trajectory motions in the sweep, with seven describing basic linear motions. Four datasets describe more complex sweeps encompassing multiple motion types (like wave, tilt, or shift) [19,21,22,23]. Additionally, variations in speed and sudden movements were occasionally noted. Another important dataset characteristic concerns the length of the sweeps, which was reported for 13 out of the 23 datasets. These datasets provided either a range or an average of sweep lengths, spanning from 20 mm to 323.96 mm ($\mu$=154.60 and $\sigma$=77.06).

3.2.2. Reconstruction Methods

Preprocessing

Minimal preprocessing of US images was undertaken, primarily involving cropping and resizing to suit the specific requirements of the employed network architectures. Prevost et al. [23] conducted experiments with various resolutions and identified an optimal resolution with a pixel size of 0.3 mm, which best accommodates the scale of the speckle pattern. Larger pixel sizes could obscure relevant speckle patterns, whereas smaller pixel sizes might amplify the impact of electronic noise and network demands. Leblanc et al. [24] found that a pixel size of 0.15 mm was optimal for efficient network training, noting that higher resolutions resulted in larger input frames which constrained batch sizes, while lower resolutions reduced precision. These images were obtained post-echograph processing, suggesting that speckle filtering may have occurred; similarly for Li Qi et al. [22] who applied median speckle reduction. Two studies reported no speckle reduction or scanline conversion was employed [17,23], due to a degradation performance using filtered B-mode images [23]. Augmentation strategies employed in four studies included horizontal mirroring, pairing non-consecutive images to enhance robustness against speed variations, and techniques like subsequence intercepting, interval sampling, and sequence inversion [19,21,23,24].

Network architectures

Various methodologies were employed across the studies for 3D US reconstruction. Earlier studies often used traditional modeling approaches. For instance, Tetrel et al. [14] utilized a graph-based trajectory estimation model based on speckle phantom decorrelation curves. This was optimized by modeling measurement errors from an optical tracked sequence through Gaussian process regression, a ML technique that predicts the relationship between the observed motion measurements and their associated errors. This allows for more accurate weighting of the measurements in the graph, improving the overall trajectory estimation. Martin et al. [27] employed a hybrid technique focusing on segmentation of the CVS, combining landmark-based registration of coronal and sagittal sequences with DL-based segmentation. Another study introduced a ML model, a Gaussian SVM regressor, equipped with a novel texture-based similarity metric to capture the dynamic textures of US images for estimating out-of-plane motion [15]. This metric essentially aims to train on speckle similarities. The remaining majority, eleven studies, have adopted DL approaches reflecting more state-of-the-art techniques. Predominantly, feed-forward convolutional neural networks (CNNs) were employed. These included 2D CNNs, like ResNet and EfficientNet [22], for processing 2D US images, and 3D CNNs, such as ResNext [16,26], for handling volumetric data. Some studies further enhanced these architectures by integrating Long Short-Term Memory (LSTM) networks to capture temporal dependencies within sequences, thus learning from sequential dynamics.

Among the studies using CNNs, there was a diverse array of approaches to achieve correct trajectory estimation. For example, Wein et al. [25] utilized two perpendicular US sweeps by co-registration of separate predicted trajectories by a CNN. Combining information and redundancy in overlapping data was exploited to refine the trajectory for a better volume estimation, since utilization of 1 sweep resulted in a volume error of 45%. Le Blanc et al. [24] utilized the CNN merely for predicting out-of-plane motion, while in-plane registration was performed by aligning segmentation masks of the artery. Guo et al. [18] specifically focused on the adaption of the network from a transrectal to transabdominal transducer. In later works, Guo et al. [16,26] introduced a self-attention module that focuses on speckle-rich areas, by utilizing feature maps from the last residual block to produce an attention map highlighting the most informative regions. Lastly, two groups of studies employed comparable methodologies within the CNNs to enhance baseline models, which warrant further evaluation. The first approach involves integrating multiple different input channels in addition to US images, such as optical flow (OF) vector fields and IMU data. The second approach focuses on incorporating more contextual information by analyzing US sequences on a global level, taking into account information from multiple consecutive frames rather than focusing solely on pairs of adjacent frames at a local level. These strategies were assessed through experimental analysis.

Loss functions

Most studies employed Mean Squared Error (MSE) loss over a sequence of N number of frames, as defined in Equation (2).

$$\mathrm{MSE}(t_x,t_y,t_z,{\theta }_x,{\theta }_y,{\theta }_z)={\displaystyle \frac{1}{N}}\sum _{k=1}^n{\left(\overrightarrow{p}\left(T_k{(i\to j)}^{\mathrm{pred}},T_k{(i\to j)}^{\mathrm{GT}}\right)\right)}^2$$

(2)

In this equation, the matrix ${\left(\overrightarrow{p}\left(A\right)\right)}^2$ computes the element-wise squared difference between each corresponding translation and rotation parameter of the predicted $T_k{(i\to j)}^{\mathrm{pred}}$ and ground truth $T_k{(i\to j)}^{\mathrm{GT}}$ transformation matrices for each pair (k) of successive US frames (i → j). The MSE penalizes larger errors more heavily and is therefore valued for its reduced sensitivity to noise during minor frame-to-frame transformations, compared to the Mean Absolute Error (MAE).

However, relying solely on the MSE [23,25] tends to result in the model memorizing general movement patterns of the clinician’s probe rather than detecting subtle nuances or variations. To address this issue, several studies incorporated a correlation-based loss in addition to MSE, focusing on the correlation among each transformation parameter (6 DOF) rather than the mean trajectory. Specifically, a case-wise-correlation loss (L_cc) as outlined in Equation (3) was adopted [16,23,25,26], which is based on a Pearson correlation loss, sometimes combined with the MAE [19,20,21].

$${\mathrm{L}}_{\mathrm{cc}}(t_x,t_y,t_z,{\theta }_x,{\theta }_y,{\theta }_z)={\displaystyle \frac{1}{N}}\sum _{k=1}^n{\displaystyle \frac{Cov\left(T_k{(i\to j)}^{\mathrm{pred}},T_k{(i\to j)}^{\mathrm{GT}}\right)}{\sigma \left(T_k{(i\to j)}^{\mathrm{pred}}\right)\sigma \left(T_k{(i\to j)}^{\mathrm{GT}}\right)}}$$

(3)

Here, the correlation coefficients are quantified for each DOF through the covariance $\left(Cov\right)$ of the predicted and ground truth transformation parameter normalized by the product of their standard deviation $\left(\sigma \right)$. The case-wise correlation loss stabilizes training and smooths trajectories by emphasizing global correlations among transformation parameters. By preventing overfitting to common scanning styles it enhances generalization, increasing the sensitivity of the model for varying speeds and motions. Further refinements include the introduction of a motion-weighted regularization to the Lcc and MSE, adjusting regularization strength based on the motion speed to address discrepancies more effectively in slower movements [20].

Additional loss functions were based on consistency to further increase robustness; Miura et al. [17] was the first to introduce a consistency loss across two US frames, employing techniques such as flipping and inverse prediction to ensure consistent output regardless of input orientation changes. Luo et al. [21] expanded this concept at a sequence level, creating flipped subsequences and utilized this as a self-consistency strategy (SCS) through online learning and adaptive optimization of the model. Luo et al. [20] applied similar strategies, specifically adopting frame-level contextual consistency (FCC) and path-level similarity constraints (PSC). Li et al. [22] developed a multi-transformation loss based on MSE to ensure consistency between predictions across multiple frame intervals. Lastly, Guo et al. [26] implemented a Margin loss, rooted in contrastive feature learning, which enhances the model’s discriminative power by forcing the feature generator to produce similar features for frames with similar transformation parameters, and distinct representations for samples with discrepancies in their transformation parameters.

3.3. Quantitative Outcomes

A detailed overview of the extracted outcome measures for all assessed methods and datasets is presented in Table 2. Frame-to-frame errors were mainly based on the MAE over all transformation parameters [17,23], a subset defined by the out-of-plane transformation parameters [15,24], or the angular transformations [19,20,21]. Studies focusing on trajectory reconstruction reported either FD or FDR. Given that FD accumulates over the length of the sweep, FDR is a more objective and comparable metric. This rate was reported in six studies, visually represented by different colors in Figure 3 for all ablation methods per dataset. The best performing methods, highlighted in bold, ranged from 5.2% [23] to 15.67% [19], with an average of 10.95%. Outcomes regarding volume reconstruction were reported in three studies. The direct estimation of the CVS volume resulted in a Dice similarity coefficient of 0.82 ± 0.04 [27]. Following trajectory reconstruction, the volume reconstruction of the prostate yielded a Dice of 0.89 ± 0.06 and a volume error of 3.21 ± 1.93 milliliter [26]. For the thyroid, a volume error of 1.15 ± 0.12 milliliter was obtained [25]. Computation time, discussed in four studies, varied; 5 seconds for CVS volume segmentation [27], 0.85 seconds for stretched trajectory reconstruction of the femoral artery [24], 2.58 seconds for trajectory reconstruction of a 100-frame US sweep [16,26], and 3 minutes for combined trajectory reconstruction and volume reconstruction of the thyroid [25].

3.3.1. Analysis Of Ablation Experiments

Incorporation of multiple data inputs

Analysis of the reconstruction methods indicated a comparable trend among CNNs for optimizing their baseline model through the use of multiple input channels alongside US frames. Optical flow (OF) vector fields were employed by several studies to capture dynamic motion between consecutive frames, thereby providing crucial temporal information about in-plane movements [17,20,23,25]. Luo et al. [20] also integrated Canny edge maps to accentuate anatomical structures, enhancing the model’s capability to identify anatomical features and in-plane motions.

Three studies explored the integration of IMU data, which provides orientation and acceleration information. However, after calibration with ground-truth position data, the acceleration typically exhibited high noise levels and was therefore less frequently used. Pioneer in this field was Prevost et al. [23] who integrated the IMU orientation data by experimenting with different integration techniques such as concatenating the Euler rotation angles to the network’s penultimate layer, thereby including the IMU orientation data in the training and prediction of the network (Table 2, method D) or directly applying calibrated IMU orientations on top of the CNN trained without IMU (Table 2, method E). Combining these two strategies, i.e. using the CNN with all available input to predict translation and resorting to the IMU orientations for the trajectory estimation, led to a further improvement (Table 2, method F). The pairwise difference between all experiments integrating OF and IMU (methods B-F), was statistically significant. Parameter-wise improvement in MAE yielded by the addition of IMU data is shown in Table 3 with a reduction of final drift by 61.89%.

In addition to the IMU orientation data, Luo et al. [19] first utilized IMU acceleration data despite its high noise. After calibration of IMU with ground truth tracking data, the acceleration showed high noise in absolute values, but a correct trend over a wider range. A temporal and multi-branch structure was employed that effectively utilized the low signal-to-noise ratio (SNR) of IMU acceleration data. Additionally, they introduced a multi-modal online self-supervised strategy leveraging IMU data as weak labels to constrain the predicted parameters. In a subsequent study, Luo et al. [21] further tackled the challenges of acceleration noise of an individual sensor by employing four IMUs. This approach integrated a modal-level self-supervised strategy that fused multiple IMU data sources, mitigating the noise effects of one IMU and reducing output discrepancies. The improvement of MAE and reported drift measures of the described approaches is shown in Table 3.

Contextual and temporal information

The enhancement of baseline models through the integration of contextual and temporal information ranged from minor adjustments to substantial architectural changes. For instance, Guo et al. [16,26] transitioned from 2D to 3D CNNs, employing 3D residual blocks and convolutional kernels. This shift allowed for more effective extraction of feature mappings along the axis of the channel, corresponding to the temporal direction of the sweep in this context. This approach showed that a configuration with five frames yielded significant enhancements over two frames [16].

Further integration of contextual and temporal data was achieved through the implementation of various consistency losses within network architectures, previously outlined in section B2. Luo et al. [21] employed a sequence-level self-consistency strategy, while Luo et al. [20] utilized frame-level contextual consistency (FCC) and path-level similarity constraints (PSC). These strategies led to improvements in MAE and FDR, as detailed in Table 4.

More extensive modifications in network architecture were explored by Li et al. [22], who evaluated the use of recurrent neural networks (RNNs) with LSTM modules and feedforward(ff)-CNNs. Those models are known to effectively handle sequential data over long periods. Since no significant difference between these models was found, it was suggested that ff-CNNs are equally competent in modeling US sequences compared to more specialized RNNs. Long-term dependencies were embedded in the network architecture by analyzing sequences of a length M of past (i) and (j) future frames, ranging from 2-100. This methodology allowed the network to capitalize on the consistency of predicting multiple transformations over various frame intervals, significantly enhancing the precision and stability of the US image reconstructions, as evidenced in Table 4.

3.3.2. Generalization And Robustness Of Methods

Generalization capabilities were assessed by three studies. Prevost et al. [23] used US sweep datasets from forearms and carotids. Networks were trained independently on each dataset, combined, and with fine-tuning of the last two layers or the entire model specifically for the carotid dataset. The findings demonstrated that training exclusively on forearm data produced the least favorable results. Fine-tuning the last two layers for carotid data only enhanced performance, comparably to retraining the network from scratch on solely carotid data. Moreover, extensive fine-tuning of the entire network further improved outcomes, suggesting that initial layers, which handle low-level features, are to some extent anatomy-specific. Training on both datasets involved the greatest amount of data and time but did not significantly surpass the results of comprehensive fine-tuning.

Li et al. [22] evaluated reconstruction performance on the complete original test set with various variance-reduced training sets regarding scanning protocol and anatomical variance. It showed, as depicted in Table 2 Method E-F, that training only on straight sweeps increased the final drift and frame errors extensively. This performance reduction also applies, to a lesser extent, to train sets with a reduction in the number of subjects and thus anatomical variance. Meanwhile, training merely on more complex motions, like c- and s-shape, yielded much less substantial performance losses.

Guo et al. [18] specifically aimed to enhance model adaptability across different US domains, transitioning from a pre-trained network on a large transrectal dataset to a smaller transabdominal dataset. To tailor the model for specific domain characteristics, a paired-sampling strategy was implemented that processed samples from both domains simultaneously during training. This enabled the network to identify domain-invariant features while adapting to nuances specific to the target domain. The fine-tuning phase focused on aligning the learned features more closely with the characteristics of transabdominal scans. The effectiveness of this approach was demonstrated through the improved feature distribution alignment between the domains, indicating robust cross-domain adaptability. Notably, their Targeted Adaptive Ultrasound Volume Reconstruction (TAUVR) approach achieved comparable results to the network trained exclusively on the target domain, as shown in Table 2 Method A and B.

4. Discussion

This systematic review provides an extensive evaluation of advancements in freehand 3D US reconstruction without external tracking, with a focus on DL methodologies that have progressively dominated medical imaging applications. The majority of the studies in this review have implemented DL techniques, predominantly CNNs. These methods have greatly enhanced the precision and usability of 3D US reconstructions, especially in settings involving real tissue. The effectiveness of CNNs can be related back to the speckle decorrelation theory, which formed the basis of previous methodologies. This was substantiated by preprocessing experiments obtaining optimal performance with a resolution correlating to the speckle noise and mitigation of the speckle reduction filter. Additionally, a self-attention module highlighted a focus on speckle-rich areas, indicating their critical role in the model’s transformation predictions [16].

Studies using CNNs showed a clear trend toward optimizing baseline models to manage complex motion patterns and anatomical variations. Enhancements were achieved by incorporating more contextual information by adapting network components of architectures and integrating IMU data or channels with optical flow vector fields. These strategies improved the performance of baseline models. Especially the integration of IMUs, providing additional position information, has emerged as a significant advancement. This showed substantial improvements in model accuracy and a reduction in drift errors, exemplified by Prevost’s achievement of a state-of-the-art FDR of 5.2% [23]. These enhancements underscore the potential of IMUs to replace traditional, cumbersome tracking systems, although further progress in noise reduction and data fidelity is necessary.

However, this review also identifies several limitations within the included studies impacting outcome measures. There are significant disparities in dataset characteristics, encompassing variations in quality and a variety of clinical applications. These different clinical applications necessitate different relevant outcome measures. For instance, in applications requiring 3D reconstructions for measuring the vessel length, length error is more critical than rotational errors, influencing the choice of metrics used to evaluate reconstruction methods [23]. Additionally, the reported lengths of US sweeps are crucial as they affect metrics such as final drift, which accumulates throughout the sweep. However, lengths are frequently unreported, with only 3 out of 8 papers discussing final drift providing this data, rendering it a subjective measure. Furthermore, the type of motion performed during the sweep also affects model training and generalization capabilities. Simpler motions are easier to learn but might be less representative of clinical scenarios. Complex motions, utilized in only four studies, are more challenging to learn, leading to varied performance across datasets with different anatomy and motion characteristics. This was also observed by Prevost et al. [23], where dataset 4 including more complex sweep motions showed diminished performance compared to dataset 1-3, including basic motion trajectories. These findings are corroborated by generalization experiments where evaluating the methods on different datasets led to decreased performance, indicating dependency on specific protocols, anatomical features, and acquisition parameters. However, the ability to fine-tune or adapt networks to different acquisition parameters or anatomical features using smaller datasets, as demonstrated by Prevost et al. [23] and Guo et al. [18], suggests that retraining from scratch for each new application is generally unnecessary.

Subsequently, due to the diverse range of outcome measures and its dependence on specific dataset characteristics, quantitative outcomes are hard to compare. This makes it challenging to conclusively determine the most effective network architectures. Notably, only four studies have reported computation times, thus providing limited information on the suitability of these methods for real-time applications. Consequently, this review serves more as an overview of existing methods rather than a comparative analysis of quantitative measures for reconstruction methods across studies.

Given the focus of this review on freehand-acquired 2D US images, to facilitate point-of-care use, studies utilizing datasets acquired with 3D US transducers were not considered. These studies sampled the 2D image planes from 3D volumes, to obtain 2D US image data. Nonetheless, knowledge of network fine-tuning suggests that some of these excluded reconstruction methods might have been interesting. Thus, the exclusion of studies using 3D transducers or non-freehand 2D images utilized in reconstruction may have narrowed the range of methodologies evaluated, potentially omitting advanced techniques. As such, the scope and conclusions of this review are confined to directly applicable network architectures for 2D US images used in 3D reconstruction methods.

As previously described, the applicability of the networks is currently constrained to representative datasets utilized in training, influenced by anatomical features, scanning protocols, and acquisition parameters. This is not necessarily considered a significant limitation, as US systems typically include application-specific presets that allow for network fine-tuning or adaptation. However, for further advancements, it is crucial that networks possess the capability to handle more representative sweeps in clinical settings. For example, intraoperative conditions may depend on specific movements where a simple sweep is not always possible. To meet these requirements, it is essential to consider factors such as managing back-and-forth sweeps where models may need to process redundant information. This consideration is vital when the organ of interest does not fit within a single US frame and requires multiple passes. Additionally, many models are direction-dependent, which may result in mirrored images.

Furthermore, assessing the accuracy of trajectory estimation compared to ground truth tracking (either EM or optical) is an initial step toward accurate 3D reconstructions. However, future evaluations of performances should also incorporate more clinical parameters. The focus of the included studies has been predominantly on trajectory estimation and its accuracy, primarily assessed by final drift. While this serves as a preliminary gauge of accuracy, it may not directly correlate to clinical relevance. Therefore, considering the current advancements in trajectory estimation accuracy, future research should aim for more clinical validation tailored to its specific applications. For instance, if the goal is measuring the volume as an important parameter for diagnosis, the Dice coefficient is paramount, even if the final drift error remains suboptimal. Alternatively, evaluating the tracking error of the ROI, rather than the frames within the entire trajectory may offer more clinically relevant insights. These clinically relevant outcomes tailored to the specific application should become more central in the evaluation of the 3D reconstruction methods.

Achievement of high accuracy in the clinical evaluation of 3D US reconstruction methods could have a transformative impact on various medical applications. A model that accurately reconstructs 3D images without relying on external tracking devices or complex setups has the potential to revolutionize fields such as surgical planning and diagnostic imaging. For instance, real-time, detailed 3D visualizations could enhance the precision of surgical procedures, reduce operation times, and improve patient outcomes. In diagnostic imaging, the ability to reconstruct 3D structures from 2D US data allows for quantitative volume measurements, which could lead to earlier detection and better treatment monitoring, e.g. in measurements of the prostate volume, critical for diagnosing and monitoring prostate cancer. This could be achieved in a non-invasive, low-cost, point-of-care setting, making advanced diagnostics more accessible, particularly in primary care environments. Additionally, this technology could democratize access to advanced imaging techniques. As these models continue to evolve, they could seamlessly integrate with other imaging modalities, providing a comprehensive view of patient anatomy and enabling more personalized and effective treatment plans. Therefore, while this review highlights significant advancements in 3D reconstruction methods, the future applications of this technology hold even greater promise for enhancing patient care across diverse clinical contexts.

5. Conclusions

This systematic review has comprehensively evaluated the progress on freehand 3D US reconstruction methodologies that circumvent the need for external tracking devices. This is a dynamically evolving field, particularly influenced by developments in DL. The application of CNNs and the integration of IMUs have notably advanced the capabilities of image-based reconstruction methods compared to conventional speckle decorrelation methods. However, the diversity in study designs and the heterogeneity in datasets present challenges in quantitatively comparing reconstruction methods and deriving definitive conclusions. While there have been significant improvements in trajectory accuracy, clinical validation of these advancements remains important. Ongoing research must address the adaptability of these systems to varied clinical environments and further refine the models to handle more complex US sweeps and anatomical variations. By eliminating the reliance on external tracking and expensive 3D transducers, these advancements provide significant reductions in both cost and complexity while expanding its usability across various medical settings.