Continued Process Verification Monitoring: Optimizing Control Chart Design by Reducing the False Alarm Rate and Nuisance Signals

1. Introduction

The Food and Drug Administration (FDA)’s 2011 Process Validation Guidance recommends continued process verification (CPV) during stage-3 process monitoring for drug manufacturers. In CPV, Shewhart control charts are employed as a statistical tool for monitoring process parameters and attributes (critical process parameters (CPPs), Key process parameter (KPPs), In-process controls (IPCs), and release specifications). In control charts, the calculated process average is utilized as the center line, and control limits are placed at distance k = ±3 short-term standard deviations (σ_R) away from the process average. The control chart, along with Nelson rules, serves as a tool for detecting out-of-control (OOC) data, defined as data that fall beyond the set control limits, and out-of-trend (OOT) data, defined as a group of data that forms a non-random pattern (shift in mean or increasing/decreasing pattern) [1,2,3]. Although control charts are commonly applied across the industry to assess the statistical stability of a process, this approach introduces two types of risk: 1) false alarms, i.e., signals that are incorrectly identified as OOC or OOT when the data are actually belong to normal variability of the process itself, and 2) nuisance alarms, i.e., signals that are too small to have any impact on the process or lack sufficient statistical reliability [4,5]. In practice, small process variations and shifts are anticipated in many processes; therefore, control charts should detect moderate to large shifts that have more practical application rather than small or practically insignificant shift signals. Both high false alarm rates (FARs) and frequent nuisance alarms will reduce the reliability of control chart signals. Therefore, in this article, we provide a review on the current state of understanding on FARs associated with different Nelson rules, properties of data distributions (skewness and kurtosis) and their impact on control chart FAR, and methods for reducing FARs by optimizing the usage of Nelson rules and adjusting the value of k (the number of standard deviations from the mean) to accommodate variability and asymmetricity in data distribution.

2. Functional Modules of CPV

CPV program is driven by four functional models as identified in Figure 1. CPV standard operating procedure (SOP): Drug manufactures define standard operating procedure for CPV as a guidance document that minimum should discuss 1) criteria for selection parameters to be monitored and trended 2) Frequency of cross functional team to perform data review meeting to monitor and analyze the trends of parameters and attributes, 3) Participants in the cross functional CPV data review meeting, 4) Discuss different statistical methods to assess the process capability and stability based on type of data distribution, 5) Method of controlled way of documenting the meeting minutes of CPV data review meeting, 6) Guidance on frequency of CPV report generation based on batch run rate, and 7) Guidance on reaction and response to signals from CPV data monitoring system.

CPV Plan: As per the recommendations in CPV SOP, CPV plan is generation for each drug substance intermediate stage, that list out all the parameters to be monitored and trended, type of data distribution and statistical method may be applied to assess the stability and capability.

CPV data review meeting: As recommended in the CPV SOP the parameters listed in CPV plans are charted and analyzed for signals of unexpected variation from historical performance as a cross functional team that includes a minimum participation from manufacturing, quality assurance (QA), manufacturing science and technology (MSAT) and quality control (QC). During the data review the identified statistical process control signals for OOT or OOC are disused on; 1. Magnitude of signal, 2. Statistical reliability of signal 3. Risk and severity on product quality or/and process performance, and decision on how to respond to the signal is warranted. If the signal is identified as nascence signal or not practically significant enough or not statistically significant enough, then no response or action is required. If the signal is identified to be real and has both practical and statistical strength but still not significant enough to impact product quality, then a technical evaluation may be warranted to understand the cause of variation, details of evaluation and results should be followed up in subsequent CPV meetings and findings should be documented in CPV meeting minutes. If the signal is identified to be real, has both practical and statistical significance, and the magnitude is significant enough to potentially impact the product quality, then an investigation under quality management system may be required.

CPV reports: CPV reports are generated either annually or semiannually or quarterly basis based on batch run rate. Here all parameters and attributes listed in CPV pans are monitored and trended along with summary stability and capability discussions along with all the CPV data review meeting minutes. Recommendations from the CPV reports on changes in the frequency of monitoring or retiring parameters from CPV monitoring may be implemented to CPV plans.

3. False Alarm Rate in Control Charts

In 1924, Walter Andrew Shewhart introduced the concept of data analysis using control charts, which was later adopted by the pharmaceutical and biopharmaceutical industries when regulatory agencies requested that drug manufacturers to perform CPV [6]. In control charts or XBar charts, the center line is placed at the average, and control limits are typically placed at k standard deviations from the average. Typically, k = 3 is utilized, for which there is a 0.135% probability that any data point belonging to a normally distributed population will fall outside ±3 standard deviations (σ). This approach gives an interval of expected future observations if the process does not change. If observations are within these limits, it can be assumed that the validated state is maintained; if observations are outside these limits, the mean and/or standard deviation has probably changed, potentially requiring attention. σ is often calculated as a short-term σ_R, which causes the limits to be more sensitive to drift, as shown in Equation (1).

$$CL=\mu \pm 3\left({\sigma }_R\right)$$

(1)

The control chart is divided into six zones based on distance from the center line as shown in Figure 2. Each zone is one standard deviation (σ_R) wide. The 3-σ_R limits on both sides of the center line show the upper control limit and lower control limit. The 2-σ_R limits indicate the upper and lower warning limits, and the 1-σ_R limits on each side of the center line are the upper and lower one-sigma limits. Control charts, along with Nelson rules 1–8 (summarized in Table 1), are used to monitor process control and process stability based on the location of data in different zones, the frequency of occurrence of any patterns, and the distance from the mean [6]. To correctly apply the Nelson test rule signals in a control chart, it is imperative to understand the normal acceptable probability for a data point to fall within or outside of the control chart zones. For instance, for a hypothetical mean µ=0 and standard deviation σ=1, the probability that normally distributed data will fall between -3 SD and +3 SD is given by Equation (2).

$$p=1-\left[\Phi \left(x=3\right)-\Phi \left(x=3\right)\right]$$

(2)

Optimizing the Number of Nelson Test Rules Used in Control Chart

In statistical process control, the FAR or rate of type I errors, often identified by the symbol α, indicates the likelihood that data will be falsely identified as an abnormal signal when the data actually correspond to the normal variability of the process. Table 1 summarizes the different Nelson rules and their purposes, along with probability of a false alarm for each Nelson rule [7].

The FARs from all eight Nelson rules identified in Table 1 sum up to 2.65%. Although it might be intriguing to use all eight Nelson test rules, Nelson (1984) suggested keeping the FAR below 1% [6]. Nelson also stated that tests 5 and 6 should be included only when there is an economical desire to have an early warning, as these rules increase the FAR. Tests 7 and 8 are used to diagnose stratifications and to identify subgroups taken from two different means. Therefore, test rules 7 and 8 have application only when subgroups are used in the control chart. However, in the biopharmaceutical industry, testing is typically performed for a single sample taken from a batch; therefore, unless multiple samples are tested and averaged for the single value reported in the control chart, rules 7 and 8 do not add value for data monitoring and trending. Adhibhatta et al. (2017) showed that Nelson tests 1, 2, and 3 are adequate for monitoring OOC and OOT data, with a combined false alarm probability of 0.94 [8]. Thus, in an effort to limit the FAR to < 1.0% and considering the practical application of test rules for biopharmaceuticals, it is recommended that only Nelson test rules 1–3 be utilized in a control chart for biopharmaceutical application.

4. FAR in Out-of-Control Signal from Limited Sample Size

OOC data are detected via Nelson rule 1 when a single data point falls outside 3 SD from the mean, with the assumptions that the data are normally distributed, the underlying true population mean (µ) and population standard deviation (σ) are known, and there is no error due to sampling variability. However, for new products, based on the batch run rate, there is a high likelihood of having a very limited sample size during the first couple of years. For small sample sizes, the replacement of the population means with the sample mean (x̄) and of the population standard deviation with the sample standard deviation (SD) and the establishment of control limits will lead to an increased FAR [9,14]. The risk of an increased FAR due to a limited sample size in control limits has been calculated by Bischak et al. (2007) as a function of the number of standard deviations from the mean (k), the number of observations (n), and the size of the subgroup in each observation (m). As discussed above, biopharmaceutical companies typically perform testing on a single sample taken from a batch; therefore, two subsequent lots are used to calculate the short-term standard deviation, and hence, m is assumed to be 2. The FAR for a limited number of observations or sample size (n) is expressed in Equation (3). Here, (C₄) is an unbiased estimator of standard deviation. The uncertainty that arises from using the sample standard deviation (SD) instead of the underlying true population standard deviation is compensated for by using an unbiased standard deviation estimator (C₄) calculated as a function of subgroup size, as expressed in Equation (4) [10,13]. Here, Φ is the cumulative normal probability density function.

$${FAR}_{\left(n\right)}=2\times \Phi \left(\frac{-k}{\sqrt{1+\frac{1}{n}\left(1+\frac{k^2\left(1-C_4^2\right)}{C_4^2}\right)}}\right)$$

(3)

$$C_4=\left[\frac{\mathsf{\Gamma }\left(\frac{m}{2}\right)}{\mathsf{\Gamma }\left(\frac{m-1}{2}\right)}\right]\times \sqrt{\frac{2}{m-1} }$$

(4)

The probability of a false alarm caused by utilizing a limited sample size when setting the control limit at k = 3 (i.e., 3 standard deviations from the mean) is calculated by applying Equations (3) and (4), as shown in Figure 3. Figure 3 shows that the reduction in FAR follows an exponential decay curve; we observe a sharp drop below 5% for a sample size of n = 10, followed by a plateau after approximately n = 30 and a further decrease below 1% at n = 100. Therefore, we recommend using a run chart when the sample size is ≤10, using the calculated control limits as tentative control limits for monitoring and updating control limits for sample sizes of 30–100, and using fixed long-term control limits when the sample size is ≥100. Another alternative solution for mitigating the uncertainty in the calculation of control limits for a small sample size (n < 30) is the use of prediction limits. Prediction limits account for the estimation uncertainty that arises from using the sample standard deviation (SD) by replacing the normal quantile with the t quantile at (n–1) degrees of freedom and also accommodates the uncertainty due to the sample mean by including the standard error of the mean, as expressed in Equation (5).

$$CL=\overline{X}\pm t_{(1-\frac{\alpha }{2}, n-1)}.SD.\sqrt{1+1/n}$$

(5)

5. FAR in Out-of-Control Signal from Skewed and Heavy Tailed Distribution

The application of Nelson test rules in a control chart is based on a normality assumption. However, if the data are severely skewed or if the variance in the data is excessively broad with a heavy-tailed distribution, this can lead to an increase in the FAR. The asymmetry and tail length in data with reference to a normal distribution are measured as skewness (K₃) and kurtosis (K₄), respectively [14].

5.1. Modelling FAR from Asymmetric Data Distribution

Control charts are based on the assumption that a data distribution is symmetric on either side of the center average line. However, in practice, the data distribution may be off-centered or asymmetric. Skewness (K₃) is a measure of the degree of asymmetry observed in a data distribution. A distribution can have either right-skewed or left-skewed data or no skew at all. A right-skewed distribution is longer on the right side of its peak, and a left-skewed distribution is longer on the left side of its peak. Skewness in a distribution can be calculated from Equation (6) [11]. A highly skewed distribution can lead to a high FAR in control chart signals. The probability distribution function of a skewed data spread can be modeled by a log-normal distribution [12], which is delineated by two parameters, namely, the shape (Sh) or mean (µ) and the scale parameter (Sc) or standard deviation (σ). The FAR caused by skewness in a data distribution with control limits placed at ± 3 (σ_R) from the mean can be determined by knowing the relationship between the skewness value and shape parameter (Sh). A correlation expression between skewness and shape parameter (Sh) was developed from the data reported by Derya et al. (2012), as shown in Figure 4A [12]. Assuming a scale parameter of Sc = 0 with the shape parameter (Sh) calculated via the correlation shown in Figure 4A, the FAR for various degrees of skewness can be calculated via Equation (7), as shown in Figure 4B. Here, k is the number of standard deviations from the mean; for control limits placed at 3 SDs from the mean, k = 3. Figure 4B shows that a skewness below 1.5 keeps the control limit signal at an acceptable FAR of <1.0%.

$$K_3=\frac{{\sum }_1^n{\left(X_i-\overline{X}\right)}^3}{\left(n\right)\times {\sigma }^3}$$

(6)

$${FAR}_{K3}=1-\Phi \left(\frac{ln\left(k\right)-Sc}{Sh}\right)$$

(7)

5.2. Modelling FAR from Heavy Tailed Distribution

Kurtosis is a measure of whether data are heavy-tailed or light-tailed relative to a normal distribution. Data distribution with high kurtosis tends to have heavy tails or outliers. In contrast, data distribution with low kurtosis tends to have light tails or a lack of outliers. A standard normal distribution has a kurtosis of zero. The kurtosis (K₄) can be calculated from Equation (8) [11,14]. A positive kurtosis indicates a "heavy-tailed" distribution. A t-distribution has fatter tails than a normal distribution. Therefore, a t-distribution can be used as a model to represent kurtosis, which will allow for a more realistic calculation of the FAR from excess kurtosis [11]. The FAR caused by kurtosis can be modeled via a t-distribution, as expressed in (9) and as shown in Figure 5, by knowing the relationship between kurtosis (K₄) and degrees of freedom (df) for the t-distribution. Here, k is the number of standard deviations from the mean; for control chart limits placed at 3 SDs from the mean, k = 3. df is the number of degrees of freedom for the t-distribution, and df_t.dist can be calculated from the kurtosis (K₄) in the data distribution, as expressed in Equation (10) [15].

$$K_4=\frac{{\sum }_1^n{\left(X_i-\overline{X}\right)}^4}{\left(n\right)\times {\sigma }^4}-3$$

(8)

$${FAR}_{K4}=2\times \left[1-t.dist\left(\left(k\right),\left({df}_{t.dist}\right)\right)\right]$$

(9)

$${df}_{t.dist}=4+\left(\frac{6}{K_4-1}\right)$$

(10)

Adjustment to “k” to accommodate the skewness and heavy tail distribution

Braden et al. (2022) showed that the FAR for a severely skewed or heavy-tailed distribution can be reduced by replacing the control limits at k = 3 (σR) from the mean with limits based on adjusted k values (k’), as expressed in Equations (11) and (12), to accommodate skewness or heavy tails in the data distribution. Branden also stated that Equations (11) and (12) are valid only when the skewness or kurtosis of the distribution meets the criteria expressed in Equation (13) [14]. The factor adjustment for skewness and kurtosis is shown in Figure 6.

$$k\text{'}=3+\frac{4}{3}\left(\frac{K_3}{\sqrt{n}}\right)$$

(11)

$$k\text{'}=3+\left(\frac{K_4}{n+0.33{ K}_4}\right)$$

(12)

$$-2.489\le \frac{K_3}{\sqrt{n}}\le 2.489 and 0\le \frac{K_4}{n}\le 11.549$$

(13)

6. Nuisance Signal from Nelson Rule 2 and Its Reliability

Nelson rule 2 stating nine subsequent data points on side of the center line used to detect shift in the mean is attributed to nuisance signal as the magnitude of shift in the mean can be potentially insignificant. Therefore, when a signal is identified for Nelson rule 2, it is imperative that the strength of the signal be assessed against the magnitude of the shift. Wheeler (2017) classified Nelson test rule 2 signals as small shift when magnitude of signal is less than 1.5 standard deviation from mean (Δ/σ). An intermediate shift when signal is 1.5 ≥ (Δ/σ) ≥ 2.5, and as a large shift when the magnitude of signal (Δ/σ) > 2.5 [16]. The reliability of the signal for the desired shift effect can be calculated from the type II error. In control charts, the (β) value or type II error indicates the probability of a true signal going undetected. The widely acceptable minimum probability for β is 0.20[17], which indicates a reliability (1–β) of 100% – 20% = 80%. The relationship between sample size, type I error (α), type II error (β), and (Δ/σ) for the mean is expressed as a two-sided limit and one-sided limit, as shown in Equations (14) and (15) [21].

$$n_T=\frac{{\left(t_{1-\alpha /2,df}+t_{1-\beta ,df}\right)}^2 }{{\left(\frac{∆}{\sigma }\right)}^2}$$

(12)

$$n_T=\frac{{\left(t_{1-\alpha ,df}+t_{1-\beta ,df}\right)}^2 }{{\left(\frac{∆}{\sigma }\right)}^2}$$

(13)

Here, n_T is the total sample size in the analysis, where n_T = n_his + n_sis. n_his is the number of historic samples, and n_Sis is the number of data points on one side of the center line showing a positive signal for Nelson test rule 2; therefore, n_Sis = 9. df = n_sig + n_his – 2, which represents the degrees of freedom, and t(1-α, df) represents the inverse of the cumulative t-distribution at 1 – α and df.

Here, an acceptable value of the significance (α) value or type I error is often set at 0.05 [18], indicating a 5% probability of falsely identifying a signal as positive when the data still corresponds to normal variability within the process. In this case, the statistical confidence (1 – α) is 100% – 5% = 95%.

The detectability/shift from the mean (Δ/σ) is the acceptable distance from the mean, expressed as a number of standard deviations. The FDA guidance document on “statistical review and evaluation” states that a (Δ/σ) value of 1.5 at 90% confidence (1 – α) is adequate to conclude equivalence between two groups [19,20].

Equations (14) and (15) for the sample size are rearranged to solve for the power of estimation, where t.dist is the cumulative distribution function for a one-sided or two-sided t-distribution, respectively. According to Cohen (1988), the generally accepted minimum power (1 – β) is 80% for a confidence (1 – α) of 95% [17]. Figure 6 shows the reliability of a signal for Nelson rule 2 for different combinations of the shift from the mean (Δ/σ) and historic sample size. Assuming a historic data sample size of n_his = 30 (The number 30 is selected because t distribution approaches normal distribution at sample size of 30) and a sample size of n_sis = 9 on one side of the center line for the Nelson test rule-2 signal, the reliability is calculated via Equation (14) for different values of the shift from the mean (Δ/σ), as shown in Figure 6. Figure 7 demonstrates that if the observed shift in the mean is ≤1 SD, then the signal can be considered as a nuisance signal without sufficient reliability.

$$Power \left(1-\beta \right)=t.dist\left(\left(\frac{\left(\frac{∆}{\sigma }\right)}{\sqrt{\frac{1}{n_{sig}}+\frac{1}{n_{his}}}}-t_{(1-\raisebox{1ex}{$\alpha $}\!\left/ \!\raisebox{-1ex}{$2$}\right., df)}\right),\left(df\right)\right)$$

(14)

$$Power \left(1-\beta \right)=t.dist\left(\left(\frac{\left(\frac{∆}{\sigma }\right)}{\sqrt{\frac{1}{n_{sig}}+\frac{1}{n_{his}}}}-t_{(1-\alpha , df)}\right),\left(df\right)\right)$$

(15)

7. Conclusions

Challenges in CPV review sessions include assessing a control chart signal for reliability and deciding whether to initiate an investigation followed by action for a corrective or preventive action. However, control charts are susceptible to false alarms and nuisance alarms. Trusting incorrect signals from the control chart and initiating quality control actions can lead to a waste of resources and time. Therefore, it is imperative to understand false alarm probabilities in control charts in association with different Nelson test rules. In this article, we reviewed the current state of understanding on FARs inherent to control charts and the impact on FARs of data distributions that deviate from normal distribution assumptions. We also discussed methods for optimizing the use of control charts by selecting the Nelson test rules to be applied and correcting the number of sigma factors (k) used for calculating control limits to accommodate skewness and kurtosis in a data distribution to reduce the FAR and nuisance signals and to improve the reliability of data trending and monitoring.