Topic Modeling for Faster Literature Screening Using Transformer-Based Embeddings

1. Introduction

The advancement of technology and internet has ushered in a new era of information sharing, greatly affecting the way research is conducted at all stages [1]. With unprecedented ease, scholars and researchers have now the ability to publish a plethora of data, reviews, and opinions on various platforms that can be accessed worldwide, using new publishing models that make sharing information easy and fast. However publishing does not only serve to disseminate knowledge for furthering research but also involves complex dynamics related to career progression and securing grant funding [2,3]. As a result, an overwhelming number of publications covering diverse topics is published every day, making it challenging to efficiently identify relevant papers amidst a sea of sometimes only tangentially related literature [4]. The challenge becomes particularly critical when conducting systematic reviews.

Systematic reviews are rigorous and thorough examinations of the existing literature to answer specific research questions [5], which rely on experimental and sometimes observational studies, with the purpose of gathering all the existing evidence on a given medical condition and, usually, its therapy. To conduct a successful systematic review, researchers must consider a wide range of sources and databases to ensure comprehensive coverage of the relevant literature [5,6], and make sure to identify all the pertinent data, minimizing evidence loss. For this purpose, researchers usually rely on established library repositories to search for scientific articles [7]. These databases, e.g. Medline, are commonly searched using specific keywords that may appear in the article, e.g. in its title or abstract. Though this approach is fast and robust, it may fail to capture relevant articles if they use different wordings or synonyms [8]. Moreover, as the lexicon may be ambiguous, this approach usually yields large numbers of publications that do not match the exact focus of the query – or are even off topic -, and forces investigators to sift through the query results to handpick the papers they need [9].

Automation has significant potential to expedite and reduce the cost of systematic reviews [10]. Recent advancements in natural language processing (NLP) and machine learning have demonstrated the possibility to automate or assist several tasks within the systematic review process. Innovations in this area have led to the development of softwares like Abstractr, ASReviews, EPPI-reviewer, and RobotSearch, which utilizes convolutional neural network architectures to identify RCTs [11,12]. Large language models are promising tools to automate some aspects of systematic reviews by enhancing literature retrieval through semantic understanding and contextual analysis of search terms [13,14]. An essential component for achieving semantic understanding is embeddings - numerical representations that encode word or even sentence meaning - which are key in NLP for capturing complex relationships between words and sentences using special architectures known as transformers [15]. Embeddings based on transformer architectures have proved very effective in several NLP tasks, and many efforts have been devoted to developing better embeddings for specific tasks, also in the biomedical field, including topic modeling [16].

Topic modeling involves identifying the main theme of unlabeled documents. This approach can be valuable in understanding complex scientific literature corpora by automatically organizing and categorizing large sets of publications based on their topics [17]. One of the latest examples among the topic-modeling algorithms is BERTopic - an advanced algorithm developed by Grootendorst in 2022 - which harnesses the power of the embeddings obtained from BERT, a well-known transformer architecture [18]. BERTopic can segment a dataset of text documents – an operation commonly known as clustering – by their semantic content and extract a series of representative keywords, which BERTopic then concatenates to create a topic label for each cluster.

The main aim of the present paper is to explore how BERTopic can be applied effectively on datasets of the scientific literature and identify relevant papers for systematic reviews in the dental field. This approach proves helpful in filtering out articles that are not pertinent for further assessment, thereby enhancing the speed and efficiency of the search process.

2. Materials and Methods

2.1. Datasets

Our analysis focused on two separate datasets, which had been previously used to identify articles of interest (henceforth “target articles”) for two published systematic reviews in the dental field, on peri-implantitis and bone regeneration, respectively [19,20].

These two datasets were manually screened, searching for Off-topic articles (henceforth labelled OffTA). We adopted a broad definition of OffTAs, as those articles that did not focus on dentistry, e.g. penile prostheses, or breast implants. Pre-clinical studies were not considered OffTAs unless they were investigating areas that were clearly related to fields other than dentistry. So, for instance, a report on cellular behavior in an in vitro setting would not necessarily be considered an OffTA, but a pre-clinical investigation on a fracture model in rodent would. All the papers that investigated areas of dentistry (or applicable to dentistry) were considered On-topic articles (OnTAs), for both datasets. It may be argued that, when it comes to peri-implantitis, orthopedic implants may be closer to dental implants than e.g. orthognatic surgery (which is related to dentistry), or that, orthopedic research articles may be more relevant to bone regeneration of the alveolar ridges than many dental-related research areas, but, as the goal of our investigation was to filter out papers from the dataset, to make systematic reviews faster, we worked under the assumption that it would be safer to discard articles that focused on different clinical areas than dentistry, and that would not increase the risk of losing important pieces of evidence for both datasets.

The first dataset included 6137 articles on the treatment of peri-implantitis and was generated through a state of the art and well-detailed keyword-based search across several databases, including Medline and Embase [20]. In that systematic review, the investigators eventually identified 24 target articles that answered the following Focused Questions (FQ):

FQ1 In patients with peri-implantitis, what is the efficacy of different bone reconstructive therapies compared to access flap surgery (AFS) in terms of pocket reduction and change in bleeding and suppuration on probing (BOP and SOP), at a minimum of 12 months of follow-up?

FQ2 In patients with peri-implantitis, what is the long-term (≥12 months) performance of reconstructive therapies in terms of pocket reduction and change in BOP/SOP?

Upon inspection, this dataset resulted composed of 3810 OnTAs (i.e. dentistry-related), and 2327 OffTAs (38%).

The second database had been used as a basis for another published systematic review by Calciolari et al. on bone augmentation techniques [19]. The dataset used for this work comprised 5309 articles obtained through a systematic literature search across several databases to address the following FQs:

FQ1: In patients receiving GBR simultaneous to implant placement, what is the impact of biomaterials (membranes, grafts, bioactive factors) on the stability of peri-implant bone levels as assessed through 2D or 3D radiographs in RCTs/CCTs with ≥ 12 months of follow-up?

FQ2: In patients receiving GBR simultaneous to implant placement, what is the impact of biomaterials (membranes, grafts, bioactive factors) on bone defect dimension (width and/or height) changes as evaluated at re-assessment procedures performed at ≥4 months post GBR in RCTs/CCTs?

Upon inspection, this dataset appeared to include only 814 OffTAs, or 15% of the total.

2.2. Purpose of the Study

The purpose of the study was to investigate whether running BERTopic, a topic-modeling algorithm, on these two datasets to segment them into topic clusters could make subsequent screening faster, by identifying groups of articles constituted only (or prevalently) of OffTAs – henceforth designated as Off-topic Groups or OffTGs – that could be safely discarded to narrow down the corpus.

2.3. Data Analysis

The data were analyzed using Google Colab Pro notebook powered by Python 3.10.12 [21] and running on T4 GPUs [22].

The analysis of the publications was conducted on their titles, based on the assumption that titles are a summary of the content of a paper, and are thus representative of their topic [23,24]. The datasets did not need to undergo any preprocessing, besides removing entries when titles were missing. Unlike previous publications [25], we did not deem necessary to lowercase the titles, nor to remove stopwords, to rely on BERTopic’s capability to produce contextual embeddings [26].

Embeddings in NLP are dense vectors that represent the semantics of words in a multidimensional space [27]. Unlike older algorithms [28], BERT (Bidirectional Encoder Representations from Transformers) understands the context and creates unique word embeddings based on their usage in different contexts [15,29]. BERTopic operates through several stages, including transformer embedding models, dimensionality reduction, clustering, and cluster tagging using cTF-IDF [18]. For our study, we chose the Huggingface’s ‘all-mpnet-base-v2’ model. Embeddings from this model are too large for efficient clustering and must be thus reduced. As several dimensionality reduction algorithms are available, we used UMAP (Uniform Manifold Approximation and Projection), which has been shown to be very effective in preserving the topological structure of data [30]. We empirically decided to reduce all-mpnet-base-v2’s 768-dimension embeddings to 5-dimension embeddings. Reduced embeddings were then clustered with HDBSCAN (Hierarchical Density-Based Spatial Clustering of Applications with Noise) [31] and cTf-Idf was applied to extract topic keywords in each cluster. Unlike Tf-Idf [32], cTf-Idf adjusts the weight based on the term frequency within a cluster of documents rather than within an individual document [33].

More specifically, we decided to use the following set-up:

- UMAP metric: cosine distance;

- size of the neighborhood: 15;

- number of components: 5;

- HDBSCAN clustering metric: Euclidean;

- minimum cluster size: 50.

To increase the system’s speed, we used the cuML GPU-based implementation of UMAP and HDBSCAN [34]. Besides BERTopic’s default representation model, we adopted KeyBERT, a more recent algorithm to improve keyword extraction [35].

We also used the OpenHermes-2.5-Mistral Large Language Model [36] to generate better labels for the topics, instead of just a sequence of concatenated keywords, which is BERTopic’s standard output. A large language model (LLM) is an A.I. algorithm that can generate human-like responses [37]. LLMs are typically large and require vast computer resources, thus reduced versions have been elaborated [38], which are commonly referred to as quantized LLMs [39]. We opted for the OpenHermes-2.5-Mistral-7B-GGUF/openhermes-2.5-mistral-7b.Q4_K_M.gguf quantization, available for download on Huggingface.com.

LLMs need a prompt from the users [40] to generate a response, and we set the following prompt:

prompt = „„„ Q:

I have a topic that contains the following documents:

[DOCUMENTS]

The topic is described by the following keywords: ‘[KEYWORDS]’.

Based on the above information, can you give a short label of the topic of at most 5 words?

A:

„„„

We used BERTopic’s inbuilt functions, and the matplotlib [41] and seaborn libraries [42] for data visualization. The Datamapplot library [43] was used for effective cluster visualization. For specificity, sensitivity, and F1 assessment, we considered OffTAs as positive, and OnTAs as negative. OffTAs clustered within a OffTG were considered true positive, while OnTAs clustered in an OffTG were considered false positives.

3. Results and Discussion

3.1. Peri-Implantitis Dataset Analysis

Running BERTopic on a Colab notebook, after loading the necessary libraries and packages, required few seconds for such a small dataset. Although BERTopic proved to be capable of easily identifying several topics based on the titles in the dataset, the actual number of topics depended on the parameters of the algorithm. The four main steps that operators can control in BERTopic are:

1) Creating the embeddings

2) Reducing the embeddings’ dimensions

3) Clustering the embeddings

4) Labelling the embeddings.

To create sentence embeddings from the titles we decided to use the all-mpnet-base-v2 model, which has been pre-trained on a large dataset and has proved to be effective also on academic titles [44]. It is relatively slower when compared to smaller models, but this did not significantly affect the computation time with little more than 6000 titles.

To improve effective clustering, we reduced the dimensions of our initial embeddings, using UMAP. This algorithm allows for a thorough customization of its parameters, including the granularity of the topological structure that it aims to preserve during dimensionality reduction, through the “number of neighbors” parameter. The clustering algorithm we chose, HDBSCAN, offers the great advantage of not requiring operators to pre-determine the number of clusters (e.g. unlike K-means algorithms), although, as a result, it tends to cluster unclear documents in a null group, which is identified by the -1 label. HDBSCAN can be customized through several parameters too, including the minimum acceptable size for a cluster. The last step consists in finding a label for each cluster, which corresponds to its topic. To do that, we have chosen two representation models. A large language model, to create human-like labels, and KeyBERT, to generate keywords that are characteristic to the topic. Figure 1 shows that, by reducing the minimum size of the clusters, BERTopic was able to find more topics in our dataset, which is intuitive, because there might be niche topics, constituted by only a handful of articles, which are lost if the size threshold is too high. At the same time, changing the sensitivity of UMAP through its number of neighbors parameter altered the slope of the curve. The choice of the best settings is an arbitrary decision, which depends on the purpose of the investigators; in our case, we wanted to have enough granularity to isolate unrelated topics, while keeping the number of topics small enough to be easily manageable and several orders of magnitude smaller than the dataset itself, to make it a convenient step in the workflow. For that reason, we empirically determined the optimal number of neighbors to be 15, and the minimum cluster size to be 50. These settings generated 14 topics (Table 1). Table 1 lists the topics identified by the algorithm, from the biggest one to the smallest one. The topic list includes the ‘-1’ null topic, where all the unclassified papers are supposed be allocated.

Quite remarkably, however, both KeyBERT and the LLM convened that these non-classified (-1 cluster) articles were anyway mostly centered on implants (and implant infections) and assigned them the title “Treating Implant Infections”. This null topic cluster was quite small (n=357), and BERTopic managed to identify several further topics that were, unsurprisingly, related to peri-implantitis (e.g. topic #0 which included the bulk of the papers (n=3733) or topic #6), but also more broadly related to implant dentistry (e.g. topics #5, #7, and #12).

However, this dataset also included at least 8 completely unrelated OffTGs, some of them quite conspicuous in size, such as the following:

#1 Valves and Stents in Coronary Arteries (n=587), e.g. “Carotid-subclavian bypass grafting with polytetrafluoroethylene grafts for symptomatic subclavian artery stenosis or occlusion: a 20-year experience” [45];

#2 Intraocular Lens Inflammation (n=232), e.g. “Double-masked, placebo-controlled evaluation of loteprednol etabonate 0.5 for postoperative inflammation” [46];

#5 Parkinson’s Disease and Deep Brain Stimulation (n= 174), e.g. “Three-dimensional space fluid-attenuated inversion recovery at t to improve subthalamic nucleus lead placement for deep brain stimulation in Parkinson’s disease: from preclinical to clinical studies” [47];

#16 Cochlear Implantation (n=96), e.g. “Online support group users’ perceptions and experiences of bone-anchored hearing aids (bahas): a qualitative study” [48].

A careful observation of the topic list suggests that more topics could be considered unrelated to the query, albeit dental-related. Some of them are small niche topics, such as #12 Zirconia Implants and Abutments (n=61), some are larger, such as topic #5 Sinus Floor Elevation (n=144); although it is thematically related to implants, nothing in its keyword descriptors mentions peri-implant disease:

[‘sinus floor’, ‘sinus elevation’, ‘osteotome sinus’, ‘sinus augmentation’, ‘sinus surgery’, ‘maxillary sinus’, ‘sinus implants’, ‘sinus lift’, ‘transcrestal sinus’, ‘eluting sinus’]

Clearly, and similarly to identifying OffTAs, choosing OffTGs to discard based on their descriptors is arbitrary and there is a degree of risk in discarding dentistry- or implant-related topics. We assumed that a cautious attitude would be to retain them, and we thus decided to discard only those topics that were unrelated to dentistry as a whole, to avoid risking losing relevant articles.

To get a better grasp of how the titles of the dataset were semantically distributed, we reduced the embedding dimensionality that we used for topic modeling from 5 down to 2 dimensions, so that each title could be represented as a data point in a scatter plot (Figure 2). Closer points correspond to articles whose titles have closer meaning, and therefore topic, while farther points represent articles that belong to less closely related topics, including frank OffTAs. As it can be easily noticed, the conceptual space for this dataset of scientific publications is not homogeneous, but there are several areas of higher density, which constitute the individual topics, and some topics appear isolated. Unsurprisingly, OffTAs tend to be distributed peripherally, as a satellite constellation of articles with looser association to the core of the dataset, which appears in the middle of the plot, and includes most of the dental-related articles (Figure 2).

Even if we just consider the topics highlighted in Table 1 and that are grossly unrelated to the purpose of the systematic review (i.e. unrelated to dentistry in general), they alone account for about 30% of the papers in the dataset (or 1856 papers out of the 6137 titles that had to be manually screened at the time the systematic review was performed).

As this dataset had already been published, we also already knew which articles had been selected for the review. As Table 2 shows, the target articles that corresponded to the query were not contained in any of the OffTGs. OffTGs (and the OffTAs contained in them) removal during screening would thus not have negatively affected the review.

If we consider where the target articles were allocated (Figure 3), it is apparent that most of them ended up in the #0 group (n=23), followed by group #6 Photodynamic Therapy for Peri-implantitis Treatment (n=1).

This supports the idea that BERTopic working on all-mpnet-base-v2 embeddings is robust enough to discriminate not only OffTGs from dental-related topics, but, within dental topics, what the peri-implantitis topics are.

We then further analyzed the dataset, and found that 6 OnTAs had been misclassified into OffTGs, and more precisely in topic #8 Porous Polyethylene Orbital Reconstruction; the first misclassified paper was about orthognatic surgery, and so not directly related to implants:

• “Long-term evaluation of the use of coralline hydroxyapatite in orthognathic surgery” [73]

The remaining five papers focused on craniofacial surgery, and thus understandably closer to orbital surgery than to dental -related topics:

• “Timing of cranial reconstruction after cranioplasty infections: are we ready for a re-thinking? A comparative analysis of delayed versus immediate cranioplasty after debridement in a series of 48 patients” [74]
• “HTR® polymer facial implants: A five-year clinical experience” [75]
• “Gore-Tex chin implants: a review of 324 cases” [76]
• “The application of alloplastic materials for augmentation in cosmetic facial surgery” [77]
• “Japanese National Questionnaire Survey in 2018 on Complications Related to Cranial Implants in Neurosurgery” [78]

In this case, therefore, it can be argued that BERTopic did not really misclassify these papers and discarding them would not have jeopardized the systematic review. Upon inspection, we also found 469 OffTAs (i.e. unrelated to dentistry) that had not been clustered in the OffTGs but had been allocated to dental topics. This classification thus yields a Specificity=0.99, a Sensitivity=0.79, and F1 score= 0.88.

3.2. Bone Augmentation Dataset Analysis

We again applied BERTopic on the titles of the articles of the second dataset and tuned the clustering parameters as previously described to get an approachable number of topics. As Figure 4 shows, generally speaking, decreasing the minimum size of the acceptable clusters in the HDBSCAN algorithm again increased the number of topics identified by BERTopic.

At the same time, changing the parameters of the UMAP algorithm allows BERTopic to better recognize local features in the distribution of the data points. We maintained the same parameters as for the first dataset, which yielded 22 topics. Table 3 lists the topics identified in this dataset.

The topics were visually inspected and, based on their LLM description and their keyword descriptors, we assessed that most topics were, as expected, related to bone regeneration (e.g. topic #3 Alveolar Ridge Augmentation Techniques), ridge preservation (e.g. topic #5 Ridge Preservation Bone Allograft) or otherwise implant-related (e.g. topic #9 Implants in Fresh Extraction Sockets).

However, a careful inspection of the dataset also revealed 5 OffTGs that appeared completely unrelated to the dental field and that altogether totaled 610 OffTAs (Table 3, bold). These topics included:

#2 Hip Arthroplasty (n=296), e.g. “Timing of tibial tubercle osteotomy in two-stage revision of infected total knee arthroplasty does not affect union and reinfection rate. A systematic review” [79]

#12 Cervical and Lumbar Fusion Studies (n=93), e.g. “A Long-Term Follow-up, Multicenter, Comparative Study of the Radiologic, and Clinical Results between a CaO-SiO2-P2O5-B2O3Bioactive Glass Ceramics (BGS-7) Intervertebral Spacer and Titanium Cage in 1-Level Posterior Lumbar Interbody Fusion” [80]

#10 Ventricular Assist Devices (n=117), e.g. “Heart transplantation of patients with ventricular assist devices: impact of normothermic ex-vivo preservation using organ care system compared with cold storage” [81]

The dataset also contained 1 topic that, albeit related to dentistry, was not centered on implant dentistry or bone regeneration, i.e. topic #15 Periodontal defect treatment, and that could thus be potentially discarded, but was retained following a conservative attitude, as explained above.

When we plotted the dimensionally reduced embeddings for these 5 selected OffTGs, these were again mostly located at the periphery of the scatter plot (Figure 5), at some semantic distance from the bulk of the data points. Taken together, the bone augmentation dataset comprised a smaller proportion of OffTAs than the peri-implantitis dataset, because our analysis identified only 11% of papers that could be safely excluded from further consideration (632 OffTAs out of 5309 articles).

We then assessed the allocation of the 36 target articles that were identified by the manual screening in the systematic review. No target article had been allocated to any of the OffTGs, so that their removal from the dataset prior to the manual screening would not have negatively affected the outcome of the review (Table 4).

Interestingly, most target papers belonged to either group #16 Collagen Membranes for Guided Bone Regeneration (16 papers out of 36 target articles), group #6 Bone Grafting for Dental Implants (8 target articles) or group -1 (9 target articles) (Figure 6). This should actually give readers some pause, as, out of 5309 articles, 45% of the target articles were found in topic #16, which comprises only 71 papers, and all the 36 articles were contained in 6 topic groups, which contained 2695 articles (noticeably, topic -1 alone contained 1515 papers).

We then manually screened the corpus again and found that no (dental) OnTA had been clustered in the OffTGs, yielding a classification Specificity=1. Upon inspection, we also found 204 OffTAs (i.e. unrelated to dentistry) that had not been clustered in the OffTGs. The Recall (or Sensitivity) for this task is thus 0.74, and the F1 score for this classification task is 0.85, in line with the results for the first dataset. This indicates that a few more papers could have been filtered out from the dataset, but it also confirms that no real dental paper (i.e. a potential target article for a systematic review) was inadvertently lost due to misclassification.

Overall, high performing topic-modeling algorithms, such as BERTopic, open up the possibility to segment whole datasets into clusters labelled by their topic. Some of these topics are clearly unrelated to the matter at hand and can be potentially, and based on our data, safely removed and excluded from further screening, saving a variable amount of time to investigators. The first dataset, on peri-implantitis, was more heterogeneous and 8 OffTGs were identified, which contained about 30% of the total number of papers in the dataset. It can be assumed that their exclusion would have significantly impacted the total manual screening time. The second dataset, on bone regeneration, was cleaner, and the 5 frank OffTGs that BERTopic identified were smaller. It can be therefore assumed that cleaner, more focused datasets will benefit from this semantic filtering less than broader and more heterogeneous datasets. At the same time, it can also be hypothesized that the availability of these protocols of semantic filtering could affect the way keyword-based searches are conducted, relaxing the need for more stringent queries, and allowing for broader and more heterogeneous datasets, which can be filtered using semantic-based algorithms before manual screening.

Our brief analysis also suggests that some topics might even be positively selected to conduct a more restricted and focused screening, with some caveats. In our situation, it was easy to retrospectively identify the topics where the target articles were contained but doing that with a new dataset would not be as straightforward, because many topics would be about closely related areas, and excluding them would be risky. As an example of this, considering an hypothetical adoption of BERTopic in the pre-screening phase of thed bone regeneration dataset for a systematic review, topic #16 Collagen Membranes for Guided Bone Regeneration could have been an easy pick for further assessment, but at this stage there would have been no rationale to safely exclude e.g. the quite large (n=692) #1 Sinus Floor Elevation group by just looking at its LLM descriptor or its keywords.

4. Conclusions

Our analysis focused on two separate datasets, and relied on BERTopic, a popular topic-modeling algorithm, to identify sets of articles to discard from datasets of the biomedical literature prior to manual screening, to make evidence identification for systematic reviews faster. Taken together, our data show that encoding article titles with the all-mpnet-base-v2 model and then running BERTopic on them is an inexpensive, user-friendly, and quick way to cluster articles into topics and have an effective overview of the dataset’s semantic structure. This procedure has a degree of granularity sufficient to identify groups of articles that can be safely removed from the dataset before it is processed by the investigators during the reliable - but slow and labor intensive - step of manual inspection. The number of topics in the dataset that are unrelated to the query varies according to the query itself, the keywords used to conduct it, and the database used to create the dataset, but, in one of the two datasets we used, we were able to filter out more than 1800 articles, or 30% of the dataset. Furthermore, we also observed that the target articles that had been actually identified for the systematic reviews tended to be found in few clusters, opening the possibility that semantic search can not only help investigators identify unrelated articles to discard, but also select a subset of the initial dataset to focus manual inspection on, with faster processing.