Exploring AI-Driven Machine Learning Approaches for Optimal Classification of Peri-Implantitis Based on Oral Microbiome Data: A Feasibility Study

1. Introduction

Dental implants are an increasingly utilised alternative for replacing missing teeth, with an annual growth rate of 14-23% per year [1]. Peri-implantitis (PI) is a common inflammatory and biofilm-related disease that affects the surrounding tissues of the dental implant [2]. This occurs when bacteria biofilms grow around the implant, causing soft tissue inflammation and bone resorption, leading to the eventual removal of the implant [3]. Current prevention strategies are based on the analysis of risk factors such as hygiene and smoking habits to assess patient eligibility to undergo a dental implant procedure [4,5]. Unfortunately, identifying risk factors alone is insufficient for accurately predicting peri-implant diseases, which exhibit a weighted mean prevalence rate of 43% across Europe [6]. As a result, there is a pressing need for effective predictive tools to assist clinicians in preventing and managing peri-implantitis to prevent dental implant failures [7,8].

The oral microbiome is composed of a diverse and complex community of hundreds of different species of microorganisms that inhabit the human mouth [9,10]. These include a wide range of bacteria, fungi, viruses, archaea, and protozoa, varying in diversity and abundance, depending on the status of oral health [11]. Several studies support the concept of characteristic microbiome profiling for PI, highlighting its potential as a diagnostic tool [12,13,14,15,16,17]. These propose a set of bacterial pathogens characteristic of PI subgingival biofilm, supporting the evidence of their association with the aetiology of PI. However, the application of such bacteria as microbial biomarkers for predictive purposes before dental implant placement is inadequate, as they are found in increased abundance or frequency when PI is installed but not in the precedent phase. Saliva is a possible viable solution for predicting PI as it may have key species or combinations of species that play a role in PI development. Additionally, saliva holds potential as a non-invasive diagnostic tool for predicting PI, as it may contain inflammatory markers, metalloproteinases, and key species or combinations of species involved in PI development [18].

Supervised machine learning (ML) is a computational technique that enables a computer algorithm to learn complex interactions from datasets with numerous variables and, consequently, the generation of a predictive model [19,20]. Depending on the algorithm, these models can perform binary phenotype classifications (yes/no), which in turn, can be applied to disease diagnosis, prognosis and effective treatment assessment [19,21]. Support Vector Machines (SVM), Random Forests (RF) and Logistic Regression (LR) have been recently applied for the classification of PI based on patient clinical data, rendering model’s performances with sensitivities up to 66% and specificities up to 72% [22]. Although these performances are reasonably acceptable, they fall short of the ideal standards required for practical application in dental practices (> 90%). Oral microbiome-based classifiers for PI using RF algorithm on plaque microbiome data have been attempted, rendering high predictive power models [23]⁠. So far, saliva microbiome-based classifiers have not been explored as a solution for PI prediction.

Developing bespoke machine learning (ML) algorithms and optimizing models is a complex and labour-intensive process, requiring the testing and refinement of hundreds of algorithms while considering combinations of methods, architectures, and customisations [20,21]. These can be further optimised for specific cases by tuning different hyper-parameters and feature engineering techniques [20]. AutoML approaches are suitable solutions for effective finding bespoke optimal ML models [20,24,25]. These are often based on genetic and/or evolutionary algorithms that explore and test the dataset accounting for various algorithms and hyperparameter combinations [20,24]. In this work, we apply auto-ML approaches to explore a wide range of ML algorithms for finding bespoke predictive models for PI using oral metagenomic data. Here, we further analyse the resulting models from saliva and subgingival biofilm to evaluate their potential as predictive tools for use in dental practices.

2. Materials and Methods

2.1. Oral Metagenomic Datasets Acquisition and Information

A total of 100 metagenomes from the NCBI SRA database under BioProject number PRJNA1163384 (https://www.ncbi.nlm.nih.gov/sra/?term=PRJNA1163384) were used in this study (see Table S1 in the Supplementary Materials). This data was obtained from 40 anonymised patients, with one or more dental implants. Of these, 20 had healthy implants (HI group), while the remaining 20 had co-occurrence of healthy and peri-implantitis-affected implants (PI group). Each individual provided a saliva sample and one or two subgingival biofilm samples. The data pertains to male and female patients over 18 years of age (excluding pregnant or breastfeeding women) seeking dental care at the Egas Moniz Dental Clinic, Almada, Portugal, between January 2023 and September 2023.

Metagenomic data processing involved quality filtering of sequenced reads using Trimmomatic version 0.39 [26] with the following parameters: (1) removal of sequencing adapters, (2) trimming of bases with an average quality score below Q25 within a 5-base sliding window, and (3) discarding reads shorter than 100 bases. High-quality reads were then aligned to the reference human genome sequence assembly GRCh38/hg38 using Bowtie version 2.5 [27] to exclude host-derived sequences. The remaining high-quality sequences from each sample were assembled de novo using metaSPAdes version 3.15.5 [28] with default settings. The relative abundances based on mapped read counts of each identified bacteria and archaea species were obtained using the MetaPhlAn version 4.0.6 [29] computational tool and using the MetaPhlAn clade-specific marker genes mpa_vOct22_CHOCOPhlAnSGB_202212 database. Viral and fungi species read counts were obtained using the Kraken2 version 2.1.1 software tool and using the 2019 viral genome and fungi genome Kraken databases [30]. Abundance estimation at the species level was computed with Bracken version 2.9 using a Bayesian re-estimation method with Kraken [31]. Abundances were further normalized to the sample total read counts and expressed in ppm. Comprising all collected microorganisms’ relative abundances (ppm), we constructed two datasets in csv files, one containing all subgingival biofilm samples (SF1) and another containing the data of all saliva samples (SF2). These datasets include all bacteria, fungi and viruses identified in the study samples, including those associated with healthy implants and with peri-implantitis (Table 1). Metadata files containing the association of each sample with taxa were constructed for the biofilm and saliva datasets.

2.2. Statistical Analysis

To assess the statistical significance of microorganism abundances between healthy and PI groups, we employed the non-parametric Mann–Whitney U test, which is suitable for comparing two independent groups when the data may not follow a normal distribution. The two-tail Mann–Whitney U test was systematically applied to compare the abundance of microorganisms. We implemented the statistical analysis in Python 3.8 using the SciPy library version 1.10. Specifically, we used the mannwhitneyu() function from the SciPy.stats module to perform the Mann–Whitney U tests. For each test, we computed the U statistic and the associated p-value, which indicates the likelihood that the observed differences between groups occurred by chance. A p-value threshold of 0.05 was used to determine statistical significance. Only microorganisms with p-values less than this threshold were considered significant, representing a less than 5% probability that the observed differences between groups were due to random variation.

2.3. Single Microorganism Classifiers

Single microorganism binary PI classifiers (yes/no) were defined based on an abundance cut-off values for microorganisms identified with statistical significance between HI and PI groups. For cases where the average microorganism abundance in the PI group was higher than in the HI group, the classifiers were set to predict PI if the abundance value exceeded the cut-off value; otherwise, the model predicted a healthy implant. Conversely, microorganism abundance in the PI group was lower than in the HI group, the classifiers predicted PI if the abundance value fell below the cut-off value; otherwise, a healthy implant was predicted. Abundance cut-offs were computed based on finding the value within the maximum and minimum range, resulting in the optimal sensitivity of the classifier tested with the datasets described in section 2.1. Classifiers and all required parameter estimations were implemented in Python 3.8.

2.4. ML-Based Microbiome Classifiers

The Tree-Based Pipeline Optimization Tool (TPOT) version 0.12.0 was used for the generation of binary classifiers (yes/no) of PI based on the entire microbiome data. TPOT is an open-source software Python package for the automated generation of ML-derived predictive tools [24,32]. TPOT is an auto-ML approach that relies on genetic programming to generate predictive models with optimal performance, testing a wide range of ML algorithms and hyperparameter tuning [24]. The Python TPOT Classifier method was used for model training, testing and optimisation. The method was implemented in a Python script for the systematic generation of models as previously implemented for other systems [33]. We set TPOT Classifier to perform 100 optimization attempts (generations), which randomly selected 50 distinct ML algorithms (population size) with a 5-fold cross-validation setting for training and testing of generated models. Here, the Optimal Receiver Operating Curve Area Under the Curve (ROC-AUC) was set as the optimisation criterion. Model generation scripts were run on a 16-core Virtual Private Server with a 2.6 GHz processor.

2.5. ML-Driven Microorganism Combination Scoring Classifiers

The O2Pmgen tool version 1.1 was used for the generation of binary classifiers (yes/no) of PI based on scoring combinations of microorganism abundances. O2Pmgen tool is an AI-driven auto-ML approach that relies on an evolution-inspired algorithm that finds the best combination of biomarkers under a multi-objective fitness function for selecting models with optimal sensitivity and specificity [25]. This tool is available online on the digital phenomics platform (https://digitalphenomics.com). Using this approach, the AI algorithm conducts supervised ML on the uploaded data where the scoring of n biomarker combinations evolved by the AI using a proportion of the data for training always inferior to 50% of the dataset, leaving the remaining data for testing purposes [34,35]. In this approach, the AI was programmed to learn to combine alterations in data that reflected up-regulations, down-regulations, and microorganism binary presences in the training set (healthy vs. PI implants). Classifiers were generated using the scoring function (S), where: Pi is the absolute distance between the median level of microorganism i on the PI group and the sample microorganism abundance value; Ni is the absolute distance between the median level of microorganism i on the HI group and the sample value; Wi is the enrichment of microorganism i on the PI group; and n is the total number of microorganisms combined in the model.

$$S={\sum }_i^n{\displaystyle \frac{100Wi\left(-Pi+Ni\right)}{Pi+Ni}}$$

(1)

2.6. Classifier Performance Analysis

Generated classifiers (predictive models) were evaluated by calculating performance metrics such as ROC-AUC, sensitivity, and specificity [36]. The ROC-AUC metric was calculated to assess the ability of the model to distinguish between positive and negative classes. The ROC curve data points were generated and the AUC was computed based on the area under this curve, which we calculated using a 5% interval window for higher precision [36]. Sensitivity (True Positive Rate) and specificity (True Negative Rate) were calculated to evaluate the model’s ability to correctly classify positive and negative instances, respectively [36]. These metrics were computed using standard formulas:

$$Sensitivity={\displaystyle \frac{TruePositives}{TruePositives+FalseNegatives}}$$

(2)

$$Specificity={\displaystyle \frac{TrueNegatives}{TrueNegatives+FalsePositives}}$$

(3)

Data visualizations were generated to provide insights into the model’s performance and the distribution of features. Boxplots, bar plots and pie charts were conducted using the matplotlib and seaborn libraries. For data manipulation, model performance filtering and ranking we use the Pandas and numpy Python libraries (e.g., array manipulations, numerical calculations). All analysis and visualisations were implemented within a Jupyter Notebook using Python 3.8. Custom Python functions were written to automate the calculation of key performance metrics, including ROC-AUC, sensitivity, and specificity. Model evaluation code was designed to run iteratively, allowing for easy comparisons between different models or model configurations.

3. Results

Model Performance

Statistical analysis of saliva and biofilm datasets identified 60 and 47 microorganism species, showing abundances significantly different between HI and PI groups, respectively (see Table S2 in the Supplementary Materials). These results indicate putative biomarker candidates for PI classification based on their relative abundance in a peri-implant biofilm or saliva. Among the putative biomarker microorganisms identified, only Stomatobaculum longum, Streptococcus sanguinis and Treponema denticola were statistically significant in both biofilm and saliva microbiome. We generated PI classifiers based on a single microorganism with statistical significance from biofilm and saliva origins. We calculated the performance of each of these single biomarker PI classifiers and ranked them according to their predictive potential. The top 10 predictive biomarkers for biofilm data demonstrated sensitivities of up to 75% (e.g. Selenomonas infelix), but specificities remained low, not exceeding 43%, with AUC values reaching a maximum of 56% (Figure 1a). Regarding the saliva data, we obtained top-ranked biomarkers with sensitivities of up to 95% (Fusarium fujikuroi) with very low specificities (up to 42%) and AUC values of up to 55% (Figure 1b). These results suggest that individual microorganisms from biofilm and saliva exhibit poor predictive performance, despite demonstrating high sensitivity. The results further reveal that only Streptococcus sanguinis appears among the top-performing single biomarkers, suggesting distinct PI biomarkers for biofilm and saliva samples.

AI-driven ML was applied to biofilm and saliva microbiome datasets to identify optimal combinations of microorganism species (biomarkers) that result in bespoke predictive tools for PI classification. We generated 100 models for each dataset based on the biomarker combinations scoring approach, rendering optimal predictive classifiers containing 2 to 4 combinations of different microorganisms. Using this approach, we obtained 35 distinct combinations of microorganisms for biofilm data PI classifiers and 23 distinct combinations for saliva data PI classifiers. These combinations are composed of 29 microorganisms for biofilm data (Figure 2a) and 23 for saliva data (Figure 2b). Interestingly, only around 30% of the identified biomarkers belonged to the microorganisms with statistical significance, suggesting a complex non-linear behaviour. Our results further showed an over-representation of 3-5 biomarkers in the generated models, indicating consistent ML behaviour that is incompatible with random selection. Specifically, Neurospora crassa, Prevotella nigrescens and Alloprevotella tannerae were prominent in the biofilm data PI classifiers, while Prevotella salivae, Streptococcus sanguinis and Fusarium fujikuroi were key biomarkers in the saliva data PI classifiers. Furthermore, our results showed very few common biomarkers between saliva and biofilm (Prevotella nigrescens, Peptostreptococcus stomatis and Bacteroidetes oral taxon 27).

To explore the diversity of available ML algorithms that can be applied for predicting PI from microbiome data, we generate 100 models using an auto ML framework. Using this approach, we tested 300 distinct AI-generated pipelines (model algorithms for binary classification) in each optimisation attempt, resulting in 30,000 distinct model classifiers for each dataset. The obtained classifiers generated using biofilm data showed a variety of types of ML algorithms as optimal choices to be applied for this data (Figure 3 a). Decision trees-based classifiers such as Random Forest and ExtraTrees classifier were the most frequent optimal solutions obtained in 39% of the AI-generated models. ML algorithms based on Bayes theorem such as Multinomial and Gaussian Naive Bayes classifiers were also present in 20% of optimal models generated. Less frequent (< 10%) ML algorithms such as the artificial neural network MLP classifier were also obtained as optimal solutions. In contrast, classifiers generated using the saliva dataset showed only 2 types of optimal ML algorithms, the Bernoulli Naive Bayes (BernoulliNB) algorithm and the Linear Support Vector Classifier (LinearSVC) algorithm (Figure 3 b). BernoulliNB was overrepresented in the generated models, accounting for 99% of the models, whereas the LinearSVC appeared in only 1% of the generated models. These results highlight significant differences in the optimal ML algorithms for application to biofilm and saliva metagenomic data.

The ROC of the PI classifiers generated from biofilm and saliva microbiome data showed substantially higher AUC values for the ML-driven models when compared with the single biomarker approach (Figure 4). Our results demonstrated that ML-driven models based on scoring combinations of biomarkers achieved AUC values ranging from over 70% to as high as 98%, indicating good to excellent predictive power. The results further showed that about half of the ML-driven models generated from saliva data exhibited higher AUC values compared to those derived from biofilm data. This positions ML-driven models based on scoring saliva biomarkers as having higher predictive power compared to those using biofilm data. Regarding the models based on ML algorithms only 25% of achieved AUC values above the acceptable threshold for reasonable predictive power (>60%), with a maximum of 90% for biofilm microbiome data and 65% for saliva microbiome data.

The sensitivities and specificities of the generated PI classifiers revealed distinct performance categories across models and their respective explored approaches (Figure 5). Poor sensitivities and specificities for single biomarker classifiers were obtained. For biofilm-derived PI classifiers, we identified 10 models with good performance, where sensitivities and specificities ranged from 80 to 90% (Figure 5a, see Table 2 for model details). Among these models, the 5 top-ranking performances were based on scoring combinations of 3-4 microorganisms with a predominance of Alloprevotella tannerae, Prevotella nigrescens, and Neurospora crassa (Table 2). The artificial neural network MLPClassifier applied to the entire biofilm microbiome data rendered a model on the 6^th best ranking, with a sensitivity of 90% and a specificity of 85%. Similarly, Gaussian Naive Bayes and the Support Vector Machines SGDClassifier were also among the best-performing models. For the remaining ML-driven models, the majority of the models based on scoring biomarker combinations had reasonable performances (70-80%) in sensitivity and specificity. In contrast, the majority of the models that correspond to models based on ML algorithms applied to entire microbiome biofilm data often resulted in poor performances (sensitivity and specificity < 70%), with only 19% of the models demonstrating reasonable performances.

For saliva-derived models, we identified 45 models with excellent performance (90-100%) in their sensitivity and specificity (Figure 5b). We also obtained 52 models with good performances (80-90%) in sensitivity and specificity. The majority of the models with good and excellent performances were ML-driven models based on scoring biomarker combinations. Only one model based on a ML algorithm had a good performance for saliva data. The top 10 best-performing PI classifiers for saliva data showed only ML-driven models based on the scoring of 2-4 microorganism species combinations (Table 3). Among these microorganism species, Prevotella salivae and Streptococcus sanguinis appeared in most models (8 and 7 respectively), including the top 3 best-performing models, which achieved 100% sensitivity and 95% specificity. Other microorganism species such as Pochonia chlamydosporia and Fusarium fujikuroi were also components of the top best-performing models indicating alternative predictive combinations.

4. Discussion

Modern AI and ML frameworks are being progressively applied in dental and oral health research, including basic, translational and clinical research [37]. ML classifiers based on applying LR, SVM, and RF algorithms on demographic data and parameters known to be risk factors have been proposed as solutions for the prediction of PI [22]. Although the resulting ML classifiers showed acceptable predictive power (AUC 70-87%), these models have shown limited performance with reported sensitivities and specificities up to 66% and 72%, respectively [22,38].

In our study, we analysed a wide range of ML algorithms to benchmark oral microbiome data as an alternative data-rich solution capable of resulting in predictive tools (classifiers) with performances suitable for its application to dental practices. From our results, the AI-driven ML classifiers based on scoring the abundances of 2-4 saliva microorganisms have shown the highest performances and predictive power among all generated classifiers, including the ones derived from subgingival biofilm microbiomes and the single microorganism approach. These model performances have achieved remarkably high sensitivities (95-100%) and specificities (90-95%), suggesting that ML-driven scoring of microorganism abundances in saliva can be a suitable approach for the generation of alternative predictive tools that outcompete current approaches [22,37,38].

Although our study can already propose ML models for predicting PI with good performances above 80% of sensitivity and specificity, these models were generated with a small dataset, which may have overestimated the performance metrics. Therefore, we can only withdraw qualitative benchmarking conclusions regarding optimal approaches and systems design for future studies that aim to develop novel PI predictive tools. Despite this data limitation, our best-performing ML models from biofilm data rendered similar performances in comparison with the work of Ghensi et al. [23] that applied a RF algorithm to subgingival biofilm microbiome for the diagnostics of PI. This supports the idea that our models are consistent with work performed in a larger cohort of patients. Moreover, our work showed that ML algorithms such as the MLPClassifier, the Gaussian Naive Bayes and the SVM SGDClassifier may be alternative solutions for best-performing models using biofilm data.

The results from our study further support the idea that applying AI-driven solutions for finding and scoring bespoke microorganism combinations (biomarkers) for predicting PI is more effective than using a classical ML approach that uses algorithms such as LR, RF, SVM on the “entire” microbiome. This suggests a paradigm change that challenges the current reported models by using AI-driven approaches that perform the identification of optimal biomarker combination scoring models. This is a significant finding as models derived from a few combinations of biomarkers offer advantages for clinical applications. Those microbial biomarkers can potentially be identified using more affordable and straightforward tests, such as qPCR. In contrast, the sequencing costs for the entire microbiome determination and quantification using metagenomics techniques may constrain the application in dental practices of ML models that rely on the whole microbiome. Although these ML approaches account for more information, they are usually “black boxes” and prone to artefacts, bias and overfitting, which is difficult to identify [20,36,39]. On the other hand, models based on a few combinations of microorganisms are much less informative but offer the possibility of being assessed from the biological point of view.

Interestingly, our work showed few similarities between biofilm and saliva-predictive biomarker species, identified in both single and combined biomarker classifiers. Further, their predictive potential in comparison with best-performing models was reduced. This suggests that saliva and biofilm microbiomes are distinct ecosystems, however, saliva biomarkers are good proxies for PI prediction and biofilm biomarkers with high predictive power.

Using AI-driven ML applied to biofilm and saliva metagenomic datasets, we identified optimal combinations of microbial species that serve as bespoke predictive tools for PI classification. Notably, Neurospora crassa, Prevotella nigrescens, and Alloprevotella tannerae emerged as prominent biomarkers in the biofilm-based PI classifiers, while Prevotella salivae, Streptococcus sanguinis, and Fusarium fujikuroi outstood in the saliva-based PI classifiers.

Neurospora crassa and Fusarium fujikuroi are two fungal species. However, it is important to recognize that all microbial species within the oral cavity engage in inter- and intra-species interactions, which may play a crucial role in determining health or disease states [40]. Neurospora is generally considered non-pathogenic to humans, whereas Fusarium is recognized as an opportunistic pathogen; however, there is evidence that Neurospora can occasionally be implicated in infections, including those of the oral cavity and other body sites, highlighting its potential, albeit rare, role in human disease [41]. To the best of our knowledge, these fungi have not been identified as part of the PI microbiome.

Prevotella nigrescens and Alloprevotella tannerae are two common oral bacterial species, involved in oral inflammatory processes being the former frequently found in dysbiotic biofilms of periodontal diseases and the latter in carious lesions [42]. These two species among others have also been found to be more abundant in the PI subgingival biofilm sites than in healthy implant sites [43]. Moreover, Verdugo et al. [44] reported that PI lesions were more likely to harbor Prevotella nigrescens than healthy implants and saliva from patients with PI showed an even higher prevalence of P. nigrescens. Consistently, our findings identified P. nigrescens as one of the three biomarkers shared between saliva and biofilm datasets, alongside Peptostreptococcus stomatis and Bacteroidetes oral taxon 27. Peptostreptococcus stomatis has been reported to be more abundant in peri-implantitis than in periodontitis [13,45].

Studies with much larger datasets are still needed before such ML classifiers can be implemented into dental practices. In particular, datasets comprising several hundred saliva samples will enable validation using independent datasets that were not involved in model construction. Models that combine saliva microorganisms with demographic data and risk factors may also be an idea worth exploring in future work to develop best-performing PI models, applicable to dental practices. Finally, the development of affordable experimental testing platforms encompassing these ML models is still needed.

The implementation of advanced AI and ML strategies in dental practices presents significant challenges, particularly due to the complexity of interpreting such data by clinicians. Although contemporary AI frameworks have demonstrated high predictive accuracy, such as scoring microorganism abundances for PI prediction, these models often rely on intricate data structures that are not readily comprehensible to practitioners. Clinicians, who are primarily trained in diagnostics and treatment rather than computational analysis, may find difficulties integrating these data-intensive systems into routine practice. Moreover, we should avoid AI-driven approaches to operate as "black boxes," rendering it challenging to translate their outcomes into actionable clinical decisions.

Researchers should simplify these technologies by converting complex ML outputs into intuitive classifiers. For instance, predictive tools could use smaller biomarker sets, such as specific microbial species combinations identified through affordable tests like qPCR, rather than relying on entire microbiome metagenomic data. These tools would enhance accessibility while preserving high sensitivity and specificity, as recent studies have shown. These technologies can improve patient outcomes in dental settings by bridging the gap between complex AI research and user-friendly applications.

Acknowledging these challenges, we designed this feasibility study to bridge the gap between complex AI-driven research and practical dental applications. By benchmarking ML algorithms and exploring saliva and biofilm microbiomes as predictive tools, this study lays the groundwork for developing intuitive, cost-effective, and clinically relevant diagnostic platforms.