Enhancing Adverse Event Monitoring and Management in Phase IV Chronic Disease Drug Trials: Applications of Machine Learning

1. Introduction

Phase IV clinical trials, the final phase of drug testing, serve a key role in post-marketing surveillance. While classical early-phase clinical trials are designed to assess a drug’s initial safety, efficacy in small populations, and pharmacokinetics, the main objective of Phase IV is to assess a drug’s long-term performance in a real-world setting, including safety, efficacy, and economic benefits. The phase doesn’t conduct research in a controlled clinical setting but instead studies how the drug performs or behaves in a variety of real-world use cases. In order to enable informed clinical decisions and public health policies, Phase IV studies determine how drugs performed in the post-marketing market, thus providing patients, physicians, and regulatory agencies with pertinent data [1]. Hence, Phase IV studies are vital in assessing the real clinical value of a drug, which will impact market sustainability, health insurance coverage policies, and the formulation of public health strategies.

Unlike conventional RCTs, Phase IV studies often enroll a population of patients that is more representative of patients in the “real-world,” including patients of all ages, genders, and ethnic backgrounds with various comorbidities. Due to strict enrollment conditions, RCT samples are unlikely to reflect real world effectiveness; in contrast, Phase IV subjects reflect true drug performance across different populations and various clinical settings. Patients’ disease state can often be more complex and multiple diseases might coexist within patients at different treatment stages. This enables the results of Phase IV studies to not just be representative of the general efficacy of the drug but also give a measure of treatment efficacy and safety for specific patient populations [2]. These studies typically have high external validity (i.e., generalizability to the real-world clinical situation) and can therefore provide more substantive evidence for common use of the drug.

The real-world data and real world evidence based analysis is the most crucial part of the phase IV studies. These data are gathered from the treatment work of health care providers and patients, and are primarily collected through electronic health records, health insurance claims data, patient registration databases, and medical cost data. This includes information about drug effectiveness, safety, patient compliance, and the socio-economic advantages of various treatment choices, as it covers long-term use of drugs in clinical settings [3]. According to RWD and RWE, Phase IV studies can now go beyond tightly controlled conditions in clinical trials to obtain more representative clinical information in the real world. Analyzing this data, researchers can not only assess the effectiveness of drugs in certain groups of people, but also examine the economics of drugs in different treatment settings, offering additional evidence for public health policies, health cover policies, and drug pricing. As the concept of personalized medicine is gradually promoted, the results of the Phase IV study will help promote the adjustment of medical policies in different regions of the world and support the adoption of personalized and precise treatment plans [4].

Nonetheless, due to the prevalence of drugs in clinical applications and the expanding population of patients, monitoring and managing adverse drug events (AEs) has become increasingly complicated. In particular, the drugs for chronic diseases often take long-term or even life-longing use, and so the adverse events of drugs often have the cumulative effect and may also show different characteristics in different time points of drug use. For instance, the side effects of certain drugs for chronic diseases may at first be subtle, but the frequency of adverse events may increase exponentially with extended utilization [5]. Moreover, patients with chronic diseases frequently have several comorbidities, meaning that the effects of drugs could be influenced by multiple factors, thus complicating monitoring. Because of the individuality of patients and the heterogeneity in drug administration, real-time monitoring and effective management of adverse events during long-term use should be addressed in Phase IV studies.

Conventional methods of safety monitoring depend on clinician and patient self-reporting and manual screening, all of which have major limitations. First, patients and physicians may not use this information in a timely manner due to information asymmetry, time pressure, or neglecting symptoms. Second, the manual screening for large amounts of data is time-consuming and labor-intensive and vulnerable to influence by factors like patient-reported bias and clinical judgment bias that might lead to underestimation of adverse drug events [6]. As the number of drug users and the amount of data increase, efficiently mining valuable information from the huge clinical data and identifying the potential drug risk factors is becoming more and more important. As the patient population expands and the volume of adverse event data grows, the monitoring work cannot rely on traditional manual methods, but must use modern technical means, especially machine learning and artificial intelligence technology, to improve the efficiency and accuracy of analysis.

2. Related Work

Medeiros-Ribeiro et al. [7] assessed the immunogenicity and safety of the inactivated CoronaVac vaccine manufactured by Sinovac in patients with autoimmune rheumatism. The results of a recent study showed that the CoronaVac vaccine induced an effective immune response among patients with rheumatoid arthritis and that the vast majority of patients experienced no serious adverse events. The study also found that a small number of people reported mild side effects like pain and fever at the injection site. This multicenter, open-label, Phase 4 study has provided helpful clinical data regarding COVID-19 vaccination in patients with autoimmune diseases.

Humby et al. [8] investigated the effectiveness of biopsy-guided precision treatment strategies in patients with rheumatoid arthritis (RA). This study applied a multicenter, randomized-controlled study in the study of the response of different subtypes of patients with RA after 16 weeks of treatment. Moreover, the results showed that individualized treatment strategies can significantly improve the symptoms of patients with specific subtypes, which has put forward a new evidence for the treatment of rheumatoid arthritis. The study indicates that patient stratification in clinical practice through individualized treatment not only improves efficacy, but could also serve as effective guidance in clinical practice. Lagoumtzi et al. [9] the use of senolytics and senomorphics in aging and chronic diseases. Their study notes that the huge potential of aging treatments does require further validation of their efficacy and safety through clinical trials. The study also mentioned potential adverse effects of treatments targeting aging processes, particularly in the context of chronic diseases like cardiovascular, metabolic, and neurodegenerative diseases.

The study proposed by Crosby et al. [10] discussed effects of ketogenic diet for chronic disease management By conducting a review of the available literature they assessed the impacts of the ketogenic diet on chronic diseases, including diabetes, obesity, and cardiovascular disease. In some patients, the ketogenic diet has a much deeper impact on the chronic disease management, however, it may come with side effects like kidney problems and electrolyte imbalance. Dravet syndrome cannot be treated, and patients with it only have medications available to control seizures. Nevertheless, the ketogenic diet is a potential treatment that is still in the clinic and it should be used carefully and monitored closely. Lazarus et al. [11] performed a Phase 4 UK feasibility trial (ComFluCOV) to assess interaction between COVID-19 vaccination (ChAdOx1 or BNT162b2), given with seasonal influenza vaccination/*. “This study found that the COVID-19 vaccine with the flu vaccine resulted in a robust immune response in most adults without significant side effects,” said study co-author, Dr. Richard McLarnon, from the University of Wisconsin, whose expertise in infectious disease and vaccine development helped design the study. Such empirical data will assist in informing public health policies for vaccine combinations and will support vaccination strategies worldwide.

Clemens et al. [12] discussed a Phase 4 non-inferiority study to assess immune response in adults who received two doses of the CoronaVac vaccine after a booster (third booster) dose. Study results demonstrated notable increases in antibody levels following booster vaccination with no serious adverse events. The study have important clinical evidence for booster vaccination that showed the third dose has a more stimulating effect on the immune system.

3. Methodologies

3.1. Spatiotemporal Graph Convolutional Network and Self-Attention Mechanism

In the analysis of the long-term application effect of chronic disease drugs, it is crucial to capture the changes in time series data and the spatial dependence between patients. First, we introduce the dynamic adaptive timing adjustment function $\gamma \left(t\right)$, which is used to weighting the effects of different time points in the self-attention mechanism. By modifying the calculation formulas of queries, keys and values, the following self-attention formulas are obtained, such as Equation 1:

$\begin{array}{c}A\left(Q,K,V,t\right)=Softmax\left({\displaystyle \frac{Q{K}^T}{\sqrt{d_k}}}+\gamma \left(t\right)\right)·V,\left(1\right)\end{array}$

where $Q$ is the query vector, $K$ is the key vector, $V$ is the value vector, and $d_k$ is the dimension of the key vector. To enhance the flexibility of the model to the change in drug efficacy, we designed a time-weighted function $\gamma \left(t\right)$, as shown in Equation 2:

$\begin{array}{c}\gamma \left(t\right)={\lambda }_1·\left(1+\exp \left(-{\lambda }_{2}t\right)\right).\left(2\right)\end{array}$

The function of this function is to adjust the influence of each time step to ensure that the model is less responsive to the drug effect in a short period of time, and in the long term, the model pays more attention to the long-term effect as the data accumulates.

At the same time, the model obtains the time-series weighted output ${\mathcal{T}}_t$ through the self-attention output of the weighted time step, which is calculated by Equation 3:

$\begin{array}{c}{\mathcal{T}}_t={\displaystyle \sum _{i=1}^N}\phi \left(t\right)·\mathcal{A}\left(Q_i,K_i,V_i,t\right),\left(3\right)\end{array}$

where $\phi \left(t\right)$ is the time gain function, defined as Equation 4:

$\begin{array}{c}\phi \left(t\right)={\lambda }_3·\left(1+\exp \left(-{\lambda }_{4}t\right)\right).\left(4\right)\end{array}$

This gain function ensures that over time, the model will place more and more emphasis on learning about long-term drug effects. Hyperparameter tuning is a critical step when building a machine learning model, and hyperparameters like lambda (regularization strength) control the complexity of the model and help avoid overfitting. In order to process multimodal data such as patient history, laboratory data, and imaging data, we introduce a weighted cross-modal fusion mechanism. The features of each modality are weighted and fused by the following Equation 5:

$\begin{array}{c}F\left(X_1,X_2,\dots ,X_M\right)={\displaystyle \sum _{m=1}^M}\left({\omega }_m·ML{P}_m\left(X_m\right)\right)+{\displaystyle \sum _{m=1}^M}{\eta }_m·Attention\left(X_m\right),\left(5\right)\end{array}$

where ${\omega }_m$ and ${\eta }_m$ are the weighting coefficients corresponding to the mode $m$, respectively, which are obtained by backpropagation learning. In this way, the model can weighted and fused information from different data sources to improve the predictive ability of drug efficacy and adverse events.

At the same time, considering the spatial relationship between patients, especially the similarity of drug responses of the same patient population, we introduced the Spatiotemporal Graph Convolutional Network (ST-GCN). The network can capture the spatial dependencies between patients through the graph convolutional layer, and infer the overall effect of the patient population while learning the individual effects of patients. The formula for calculating the convolution of a spatiotemporal graph is Equation 6:

$\begin{array}{c}{H}^{\left(l+1\right)}=\sigma \left(\widehat{A}{H}^{\left(l\right)}{W}^{\left(l\right)}\right),\left(6\right)\end{array}$

where $\widehat{A}$ is the normalized adjacency matrix between patients, $H^{\left(l\right)}$ is the node feature matrix of the $l$-th layer, $W^{\left(l\right)}$ is the weight matrix of the graph convolutional layer, and $\sigma$ is the activation function. Through graph convolution operations, the model can share information about drug efficacy among different patient populations to improve the overall prediction ability.

3.2. Self-Supervised Learning with Graph Embedding

Data scarcity is often an issue when dealing with adverse event predictions, especially in long-term studies of chronic disease drugs. To this end, we introduce a self-supervised learning strategy to improve the robustness of the model by designing a self-supervised loss function. The self-supervised loss function combines drug efficacy prediction and negative sampling strategies, as in Equation 7:

$\begin{array}{c}{\mathcal{L}}_{SSL}={\displaystyle \sum _{i=1}^n}{\left({\widehat{X}}_i-X_i\right)}^2+{\lambda }_5·\left({\displaystyle \sum _{j=1}^k}Loss\left(X_j,Y_j\right)\right)+{\lambda }_6·R\left(\theta \right),\left(7\right)\end{array}$

where ${\widehat{X}}_i$ is the predicted value, $X_i$ is the actual observed value, $Loss\left(X_j,Y_j\right)$ is the prediction error of each pair of samples, and $\mathcal{R}\left(\theta \right)$ is the regularization term of the model parameters to prevent overfitting. Through self-supervised learning, the model can extract the relationship between potential drug effects and adverse events from scarce data, thereby improving the prediction accuracy.

In terms of drug effect prediction, graph embedding technology provides an effective way to combine the individual characteristics of patients with drug effects. Figure embedding derives the effects of drugs in different patient populations by learning the relationship between drugs and patients. The graph embedding process is calculated by Equation 8:

$\begin{array}{c}z=GraphEmbed\left(G,\theta \right)=Softmax\left({\displaystyle \frac{AZ+B}{C}}\right),\left(8\right)\end{array}$

where $A$ and $B$ are the matrices of the relationship between the patient and the drug in the graph, $C$ is the normalized coefficient, and $Z$ is the embedded representation of the drug effect. AZ + B means that Z (the embedded representation of the drug) is first linearly transformed with A (the relationship between the patient and the drug), and then the B bias term is added to obtain a weighted result, which can be used as a predictor of the drug’s efficacy in different patients.

In order to further improve the accuracy and stability of the model, especially in the monitoring of adverse drug events, we optimized the model by combining the negative sampling strategy and the regularization mechanism. When dealing with adverse events, learning from negative samples becomes particularly critical. Prediction errors and regularization terms are minimized by optimizing the following loss functions, as in Equation 9:

$\begin{array}{c}{\mathcal{L}}_{final}={\mathcal{L}}_{SSL}+{\lambda }_7·{\mathcal{L}}_{reg}+{\lambda }_8·{\mathcal{L}}_{neg},\left(9\right)\end{array}$

where ${\mathcal{L}}_{reg}$ is the regularization term, ${\mathcal{L}}_{neg}$ is the learning loss of negative samples, and ${\lambda }_7$ and ${\lambda }_8$ are hyperparameters.

By using the self-supervised learning and negative sampling strategies, the model can effectively reduce overfitting and improve the prediction accuracy of adverse events. Through the design of these innovative formulas, our model can effectively improve the accuracy of adverse event monitoring and drug efficacy evaluation in the study of long-term use of drugs for chronic diseases, and maintain high robustness in the face of scarce data.

The threshold selection method for cost is to choose a threshold to minimize the overall misclassification cost or maximize a custom utility function to optimize the model’s decision-making effect when the cost of false positive and false negative misclassification is different.

4. Experiments

4.1. Experimental Setup

Above all, the national rheumatic disease patient registry (NRDPR) was utilized. This database integrates long-term clinical data from patients with rheumatic diseases in several countries/regions, particularly patients with rheumatoid arthritis (RA) and other autoimmune disorders. The dataset is multi-dimensional including the personal data, medical history, medication, lab tests, disease activity index, the complications and responses to treatment, etc. It has the characteristics of high-dimensional multi-type data (time series data, categorical data, continuous numerical data, etc.), and a large number of missing and inconsistent respectively, bringing difficulties to the training of machine learning models. The data includes information on patients of different ethnicities, gender, and ages, with a wide range of severity and types of complication, and over an extended time frame, with some patients followed for greater than 5 years. The analysis of safety and efficacy during the long-term use of drugs for patients with chronic diseases, based on this real-world data, provides insight into the clinical responses that cannot be obtained through clinical trials, including an in-depth analysis of clinical responses, such as adverse events that may occur during treatment and their influence on drug efficacy.

4.2. Experimental Analysis

To validate the effectiveness of the proposed model, four common comparison methods were used, random forest (RF), support vector machine (SVM), gradient boosting tree (GBDT) and long short-term memory network (LSTM). As an ensemble learning method, random forest is well suited for handling high-dimensional data and missing values, having been commonly used for predicting diseases and drug efficacy. A support vector machine (SVM) processes nonlinear data by constructing an optimal hyperplane, which is suitable for small-sample, high-dimensional tasks. Gradient boosting trees are able to reduce errors by developing many decision trees, which is suitable for capturing complex nonlinear relationships and derive high precise prediction results. Unlike convolutional neural networks, which have high computational costs associated with their quadratic time complexity, long short-term memory networks are recurrent neural networks that work well with time-series clinical data, such as drug long-term effects or cumulative adverse events..

Using specificity as the primary evaluation metric, we assess the performance of various algorithms across diverse thresholds. Specificity is the true negative rate, a measure of a classification model’s ability to identify negative class samples. As we can see from our results for each algorithm including the mentioned RF, SVM, GBDT, LSTM, and our Ours method. We plotted the specificity as a function of threshold which visually compared performance under different thresholds.

The results of Figure 1 indicated that the specificity of all methods increased to varying degrees as the threshold increased. More specifically, this is because with a higher threshold, the model is more stringent in predicting positive samples and thus, the false positives decrease while specificity increases. The Ours method outperforms all other methods consistently and has the highest specificity. This shows that the Ours method is superior to identify negative samples under multiple thresholds through provide more accurate judgment on some occasions (such as drug safety monitoring). In summary, the Ours method is particularly advantageous for fast specificity increase as well as final stability—especially under high threshold conditions, outperforming other methods. By means of the visual analysis, we further confirm the feasibility and practical relevance of our model.

The following the Matthews correlation coefficient (MCC) was taken as the evaluation index, and MCC is an important metric to assess between the performance of dichotomous model and has high value especially in the event of data imbalance. The range of MCC is -1 to 1, where 1 is perfect classification effect, -1 is completely wrong classification rules, and 0 is random classification.

From above Figure 2, we can observe that the Ours method outperforms all other works in terms of MCC in the comparison of multiple methods, and the MCC values of Ours method increase with the increase of threshold setting, which indicates that it can maintain good classification performance for different settings of thresholds. Compared with other methods ( such as RF, SVM, GBDT, and LSTM ), the MCC performance of the Ours method is better than that of other methods, especially at higher thresholds, which can balance positive and false positive better and get more accurate classification results. Such that the Ours method has the capability to increase stability and predictive ability on the complex data background, especially in the long-term efficacy analysis of chronic disease drugs and better ability of exploration base for therapeutic effect and potential risks.

5. Conclusion

In general, in the analysis of drug effects for chronic diseases and especially for real-world data, this study presents the benefits the proposed method can achieve for traditional machine learning method measured by Matthews correlation coefficient (MCC). Under different threshold settings, the proposed method achieves a better classification performance, which confirms that the proposed method is more robust on the monitoring for drug efficacy and adverse events. This is especially critical for Phase IV clinical trials, in which long-term efficacy and safety of the drug is analyzed to inform deliberation and public health policy. In future research, others may incorporate more diverse data inputs to enhance the model’s predictive capabilities. Simultaneously, advanced technologies, such as semi-supervised learning or active learning, could address the data imbalance and missing data issue, and further enhance the business performance of the model.