Evaluation of 3D Facial Scanners: Trueness & Precision

1. Introduction

Today, we are witnessing an increasing adoption of digital technologies in the dental field [1,2]. This can be attributed to improved patient comfort, as these technologies are less invasive and reduce the time required for analysis and evaluation of results [3]. Digital technologies are being applied in various areas of dentistry, such as prosthodontics, implantology, orthodontics, and restorative dentistry [4,5]. The latest innovation is the Facial Scanner (FS), which creates a virtual craniofacial model through a fully digitalized three-dimensional image [3,6]. As part of a fully digital workflow, facial scan images and intraoral scans can be integrated with CBCT data, enabling the creation of a 3D virtual patient that supports treatment planning and improves results [6,7,8]. The main advantage of using FSs lies in their ability to scan perioral soft tissues, which cannot be reproduced through CBCT or intraoral scanning, allowing for the creation of a fully digitalized 3D model of the patient's face [3,9,10]. FSs, like Intraoral Scanners (IOS) [11], must achieve a high level of trueness to create reliable 3D craniofacial models [1,10,12]. Facial scanners are used for diagnostic records, virtual patient creation, smile design, virtual articulation, and the diagnosis and treatment of Obstructive Sleep Apnea (OSA) [13]. They deliver precise 3D modeling for personalized treatment planning and improved clinical outcomes. These capabilities are further enhanced by incorporating jaw motion tracking (JMT) devices, which offer innovative and valuable tools for achieving fully facially driven treatment plans. With JMT, a fourth dimension (dynamic patient-specific jaw motion) can be integrated, enabling even greater refinement and personalization in treatment [14,15]. Accuracy refers to how closely a test result matches the accepted reference value. Two related terms are also used: precision, which measures the consistency of results under specific conditions, and trueness, which describes how close the average of multiple measurements is to the true or reference value, regardless of result variability [16,17]. Therefore, while trueness is essential, it alone is not sufficient for evaluating a FS. The FS must also demonstrate precision by delivering reproducible results across multiple measurements of the same object. In addition to professional facial scanning devices, which provide high-performance levels at a significant investment, dental apps are now available for download on mobile devices, offering low-cost options for capturing images of faces and objects [18]. The aim of this study was to evaluate the precision and trueness of facial scans obtained from ten different FSs, including both professional devices and mobile applications: Metascan (App Store), TRNIO (App Store), Scaniverse (App Store), MagiScan (App Store), Polycam (App Store), Face Hunter (Zirkonzahn), Space Spider (Artec3D), Ray Face (Micerium), Easy Face Scan (Revopoint), and MetiSmile (Shining 3D). The files generated by these various scans were compared to a reference file created using the industrial scanner Edge 14,000 (FARO: Certified ISO 9001 - calibration ISO/IEC 17025:2017 accreditations). If scans are not accurate, treatment failures can occur, such as misfit dental restorations [19]. Therefore, this research aimed to assess the accuracy of the virtual patient. The null hypothesis is that facial scans from different scanners show significant differences in precision and trueness compared to the reference scanner, with no difference between mobile apps and professional facial scanners.

2. Materials and Methods

To standardize the measurements, a three-dimensional facial model made of polyvinyl chloride was used, with dimensions, shape, and proportions comparable to those of the human face (Figure 1).

The local markers were created using Cobalt-Chrome laser fusion technology and subsequently coated with pink enamel, similar to the color of the face. The geometry and positioning of the local markers were carefully considered, opting for a convex spherical shape that is more accurate to scan [13]. To ensure repeatability and ease of assessment, six linear measurement markers were selected: tragus, ala, and chin on both the right and left sides.

The number and arrangement of the points intended for matching always have a spherical geometry, which is larger in size compared to the reference points used for linear measurements, in order to distinguish them from the others. After selecting their distribution, the local markers were permanently affixed using a liquid cyanoacrylate adhesive to ensure the repeatability of the different scans (Figure 2). The scanning phase began with the acquisition of the reference file, which is the representation of the three-dimensional facial model. This file is necessary to compare the scans produced by the various FSs used in the study, in order to determine their local and global trueness. An industrial laser scanner, specifically the Edge 14000 model by FARO®, was used to acquire the reference file. The .stl format was chosen for export.

After an in-depth product review prior to the study, five professional FSs intended for dental use and available on the Italian market were selected. Similarly, five scanning apps were chosen for download, including only those with the highest number of reviews and good ratings. The professional FSs included in the study were: Face Hunter (Zirkonzahn), Space Spider (Artec3D), Ray Face (Micerium), Easy Face Scan (Revopoint), and MetiSmile (Shining 3D); the apps tested were: Metascan (App Store), TRNIO (App Store), Scaniverse (App Store), MagiScan (App Store), and Polycam (App Store). All FSs and applications are capable of exporting files in .stl and .obj formats. The applications differ from the professional scanners in terms of the technology they use, as while all scanners employ structured light technology, the applications primarily rely on photogrammetry, which reconstructs 3D objects by analyzing multiple 2D images, relying on feature detection and triangulation.

Six scans of the model were performed for each scanner, and, like the reference file, the .stl format was chosen so that the resulting files could be imported into the CAD software used for the subsequent measurements. The Exocad DentalCAD software was then used for the measurement phase; Exocad also offers a wide range of implant libraries integrated within the software to support precise and customized implant design. In collaboration with New Ancorvis S.r.l. (Via dell'Industria, 15, 40012 Bargellino (BO) - Italy), and using the CAD method to create implant libraries, a volumetric conversion library was created. This allowed a conical geometry to be associated with each spherical local marker, facilitating the subsequent measurement phase. Distances were measured from the apex of each cone, corresponding to the reference point. The measurement process was carried out using a virtual ruler integrated into the software. To verify the correct cone-sphere matching, there is a specific function that allows the software to calculate the best possible placement based on the acquired data and actual alignment (Figure 3).

To evaluate the trueness of the various scanners along the x, y, and z axes, and thus perform a 3D assessment based on the previously established reference points, the following distances were chosen: Right ala - Right tragus, Left ala - Left tragus, Right tragus - Right chin, Left tragus - Left chin, Right ala - Left ala, Right chin - Left chin, Right ala - Right chin, Left ala - Left chin (Figure 4).

The same protocol was applied to each FS in the study. First, the .stl file from the scan was uploaded to the Exocad platform, and the custom volumetric conversion library was opened to align the reference points (Figure 5 and Figure 6). Then, the predefined linear distances in millimeters were measured using the specific measurement tool.

After taking the measurements, the results from the six scans of each facial scanner were analyzed for precision.

These measurements were then compared with the reference values to assess overall trueness. Trueness analysis was conducted globally by comparing linear distances between the files produced by the different facial scanners and the reference file. Local trueness was evaluated at the reference point surface by measuring the axial distance between the scanned reference surface and the hemisphere surface on the reference model, corresponding to the apex of the associated cone (Figure 7).

Statistical analyses were conducted using R software (version 4.2.2, R Foundation for Statistical Computing), an open-source platform for data manipulation, computation, analysis, and graphical visualization. Numerical data were reported as means and standard deviations. Differences in linear measurements obtained from the scanners relative to the reference file, along with discrepancies in markers between the study group files and the reference file across different scanners, were evaluated using linear mixed regression models. In these models, scanner type and measurement were treated as fixed effects, while a random intercept was assigned to each scan. This methodology enables the analysis of complex datasets by distinguishing between systematic and random variability, enhancing the trueness of estimates in scenarios involving repeated measurements or data collected from multiple groups. Correlation classifications were defined as follows: poor for ICC ≤ 0.2, fair for 0.2 < ICC ≤ 0.4, moderate for 0.4 < ICC ≤ 0.6, substantial for 0.6 < ICC ≤ 0.8, almost perfect for 0.8 < ICC < 1, and perfect for ICC = 1. The same scale was applied to the concordance correlation coefficient (CCC). The ICC quantifies the similarity or consistency of measurements across different instruments, while the CCC measures the closeness of the measurements to each other and to the reference value. Bland-Altman analysis was performed to estimate the bias (the mean difference between measurements) and the limits of agreement (LoA), which represent the range within which differences between the two methods are considered acceptable. These estimates were accompanied by confidence intervals for each scanner, indicating the probability that the true value lies within a specified range. Confidence intervals for bias were calculated at 95%, while those for LoA were calculated at 90%, as the latter were determined using two one-sided tests (TOST). The significance level (α) was set at 0.05.

3. Results

The results from the statistical analysis were divided into two categories of scanners examined: applications downloadable on devices and professional scanners. Regarding the applications, although the scan preview appears to closely resemble reality, substantial differences emerge in terms of image resolution when analyzing the files exported in .stl format. Additionally, it was observed that the files generated from scanning the model were exported either over- or under-scaled relative to the reference. This discrepancy makes it impossible to align the hemispherical local markers scanned by the applications with the conical geometry of the volumetric conversion library specifically created in Exocad. As a result, due to these significant dimensional differences, it was not possible to proceed with the intended measurements. The preliminary check with 3Shape Viewer confirmed that the five professional FSs produced scan files accurately scaled to the reference, unlike the apps. This ensured the feasibility of proceeding with the measurement phase.

For each FS, including the reference model (Table 1), the data generated by the various scans were sorted based on their distances in millimeters.

The mean values and standard deviations of the linear distances measured across the six scans were also calculated, along with the discrepancies at each local marker relative to the reference for each scanner (Table 2).

Globally, by comparing the distances obtained from the various FSs to those from the reference, the performance of the scanners was evaluated using the intraclass correlation coefficient (ICC). In this case, all scanners demonstrated a consistency of 1 relative to the reference, which is considered perfect.

The concordance correlation coefficient (CCC), similar to the ICC, was calculated to assess the agreement between the measurements produced by the various scanners and the reference values. The CCC accounts for both the correlation between measurements and the systematic bias relative to the reference, both of which were also equal to 1. A comparison was made between the average linear distances measured across the six scans for each scanner and the corresponding values from the reference. Most scanners did not show statistically significant differences compared to the reference (p-value > 0.05). The only scanner that produced statistically significant differences was Easy Face Scan (p-value = 0.003).

Analyzing the mean difference (MD) and its confidence interval, a slight underestimation was observed, with an average discrepancy of only 0.304 mm. This indicates that the scanner generates a 3D facial model slightly smaller than the actual size. However, the average discrepancy was minimal, with a 95% confidence interval ranging from -0.497 mm to -0.110 mm.

In the global context, all scanners demonstrated high reliability in terms of precision, as evidenced by the absence of statistically significant differences in the linear distances measured across the six scans.

The data presented in Table 3 represent the local trueness values of the five scanners, measured at the markers relative to the reference file. As shown, the presence of discrepancies with a mean greater than 0 compared to the reference was tested under the null hypothesis. Consequently, all scanners showed statistically significant differences (p-value < 0.001). However, for almost all scanners, these differences were very close to 0.

In particular, the scanner that showed the highest local trueness values was the Space Spider by Artec 3D, followed by MetiSmile by Shining 3D, Easy Face Scan, and Face Hunter by Zirkonzahn. The only scanner that produced more noticeable differences compared to the reference was RayFace by Micerium.

Regarding local precision, which refers to the ability of scanners to obtain similar measurements during different scans of the same object, without considering the reference value, all the scanners examined showed high levels of precision. This was confirmed by the absence of statistically significant differences between the measurements, irrespective of the reference value (p-values > 0.05) (Table 4).

4. Discussion

This study aimed to evaluate the performance of facial scanning technologies in generating accurate and precise facial representations for clinical use. While professional facial scanners demonstrated high trueness and precision, scanning applications were found unsuitable due to significant limitations in their resolution and dimensions (Figure 8). Thus, the null hypothesis is rejected.

The study highlights several novel contributions to the field. Firstly, it underscores the superior performance of professional facial scanners in both global and local measurements. At the global level, the intraclass correlation coefficient (ICC = 1.000) confirmed perfect agreement for all tested professional scanners, indicating excellent repeatability and reliability. This level of reliability has significant implications for clinical workflows, as consistent and repeatable scans are crucial for ensuring precise treatment planning and execution.

Among the professional scanners evaluated, the Space Spider by Artec 3D emerged as the top performer, as seen in previous studies that highlighted its superior trueness and precision in capturing facial geometries [21,22], excelling in both global precision and local trueness.

In contrast, RayFace by Micerium showed notable limitations in local trueness, indicating potential constraints in its application for treatments requiring high spatial resolution, such as orthodontics or prosthodontics.

Scanning applications, such as Scaniverse, Polycam, MagiScan, and TRNIO Plus, demonstrated significant deviations from the reference models, with undersized or oversized 3D representations. These findings are consistent with earlier research [23] highlighting the limitations of low-cost, non-specialized scanning solutions in capturing complex anatomical details when compared with professional FSs. However, recent studies have demonstrated positive outcomes for these applications [24], and further research is needed to validate their clinical reliability.

Professional-grade facial scanners offer a reliable solution for capturing patient-specific facial geometries, supporting facially driven treatment planning. This approach is particularly beneficial in complex treatment scenarios, such as occlusal plane orientation and the integration of facial aesthetics into prosthetic and orthodontic workflows. Moreover, the study emphasizes the role of high-precision scanning in improving patient involvement and understanding, thereby enhancing overall treatment outcomes. Future research should aim to establish standardized guidelines for the clinical use of facial scanners, addressing factors such as trueness thresholds, compatibility standards, and best practices for integrating scanning technologies into routine workflows. Additionally, using a rigid model removes patient-related variability but fails to reflect real clinical conditions. Future studies should address in vivo variables to enhance applicability.

5. Conclusion

The apps downloadable on devices, due to the limitations of this in-vitro study, showed significant dimensional discrepancies and should continue to be tested. In contrast, professional FSs demonstrated high global trueness and reproducibility, making them reliable tools for the three-dimensional reconstruction of facial morphology.