Assessing the Accuracy of Air Quality Personal Exposure Monitors

1. Introduction

Air pollution comprises a mix of gases and solid particles of varying composition. Particulate matter (PM), carbon monoxide (CO), ozone (O₃), and nitrogen dioxide (NO₂) are major pollutants of concern and their impact on health has been extensively studied [1,2,3]. Studies of this type are, however, often limited by a lack of understanding of personal exposure to poor quality air [4].

Personal exposure to air pollutants can be assessed using a variety of methods. Most studies have been restricted to indirect approaches, such as stationary monitors, dispersion modeling, and land use regression models [5,6], while others favor combining two or more methods in an attempt to provide a more accurate estimate of air pollution exposure [7,8]. Each of these methods has its own set of limitations, the most important of which is the inability to accurately quantify an individual’s actual exposure to multiple pollutants, and the temporal and/or spatial variability of the exposure [6,9].

In recent years, low-cost personal exposure monitors (PEMs), with technologies like compact pollution sensors enabled with GPS, have been employed as an approach for assessing personal exposure to air pollution and are now widely available commercially [10,11]. They are designed to be attached and carried by the person of interest, thus providing a measure of the personal exposure to air pollution [12,13]. PEM characteristics potentially provide important opportunities for improving of our understanding of the impact of air pollutants on health, [14,15] but also may support education and public awareness [16]. They could be used in combination with routine ambient air monitoring networks [17] to assess air pollution exposures at population levels, e.g. across cities [18]. However, data quality and lack of guidance on how to use these devices limit their use [19,20,21]. Their accuracy and reliability must be robustly tested before they can be adopted specially in health research studies.

Field and laboratory evaluations are vital for validating the performance of personal air quality monitors in order to ensure that the quality of data obtained is of high quality [19,20,21]. There is an ever-growing number of studies that asses the accuracy of a suite of static air pollution sensors designed to be used in networks, but studies showing the performance of personal air quality sensors are much more limited despite their use, at scale, in measuring personal exposure. Plume Lab Flow has been developed as a PEM and recently evaluated in a laboratory setting [22]. Plume Lab Flow 2’s performance has been reported by the South Coast AQMD, where they only compared three PLFs outdoors alongside a high reference-grade instruments [23]. All which were limited to certain environmental conditions. The assessment of PLF is still limited and needs to be evaluated in different field conditions that can represent the individual exposure level. In this study we evaluated the performance of 21 PLF against reference-grade instrumentation in a range of field environments over sufficient periods of time, to determine whether they can be effectively utilized.

2. Materials and Methods

Twenty-one PLFs were deployed in three field tests: two outdoor (roadside, urban background) for short duration (3 to 4 hours) and one indoor location for long duration (3 weeks). The PM_2.5, PM₁₀, NO₂ measurements results from 21 PLFs were compared to those of side-by-side measurements using three reference instruments (depending on the field test): a) Portable Optical Particle Spectrometer (POPS) [24], b) ARISense Sensor [25], and c) monitors from the Manchester NERC Air Quality Supersite (http://www.cas.manchester.ac.uk/restools/firs/).

2.1. Instrumentation

2.1.1. Personal Exposure Monitor

The PLF (https://plumelabs.com/en/flow/) is a wearable low-cost (~150 USD/sensor) air quality sensor that connects to a mobile application via Bluetooth connection. It can be carried or worn by a person during their regular daily routine. It weighs 70 g, and the unit charge lasts approximately 24 hours. It is intended to track indoor and outdoor air quality. In this study, the PLF was chosen as it is one of the market leaders (at the time of the study) and measures (every minute) particulate matter (PM_2.5 and PM₁₀; measured in µg/m³), NO₂ (measured in ppb), volatile organic compounds (VOCs measured in ppb). The PLFs were operated according to the manufacturer’s instructions, where the data were synchronized every two to three days to ensure uploading of the data onto the manufacturer’s server. As per the manufacturer's instructions, each flow device was operated for a week before utilizing them in the fields. This data is not considered usable for the analysis.

2.1.2. Reference Instrumentation

The Portable Optical Particle Spectrometer (POPS) and ARISense (version 1.0 system) are portable air quality instruments suitable for indoor and outdoor use [24]. POPS is a lightweight particle counter that measures particle diameters between 0.13 to 3.0 µm by using a single-particle light scattering algorithm [26]. This study used POPS to measure PM_2.5 (µg/m³), at outside roadside and indoor. The ARISense sensor is equipped with a variety of electrochemical sensors designed to measure the ambient levels of multiple pollutants in real-time, but were used here for reference data for two pollutants: NO₂ (ppb) and PM₁₀ (µg/m³) [24], which used to measure indoor air quality levels. In addition, two reference-grade instruments from the Manchester Natural Environment Research Council (NERC) Air Quality Supersite (the NERC supersite monitors) were employed for outside background measurements: FIDAS200 (Palas GmbH, Germany) instrument to measure the PM₁₀ (µg/m³) and PM_2.5 (µg/m³) concentrations, and Teledyne API T500U Cavity Attenuated Phase Shift (CAPS) Analyser to monitor NO₂ (ppb) concentrations, instruments type are described in detailed previously [27]. As per the manufacturer's instructions, each flow device was operated for a week before utilizing them in the fields. This data is not considered usable for the analysis.

It is important to note that the ARISense is a low-cost sensor; measurement errors in this instrument are possible. However, its high-performance quality has been demonstrated, making it a viable option for use as a reference in this study [24].

2.2. Sites and Measurement Periods

2.2.1. Indoor site measurements periods

In the indoor evaluation, PM_2.5, PM₁₀, and NO₂ levels measured from the PLFs were compared with the PM_2.5 levels measured by POPS, and PM₁₀ and NO₂ measured by ARISense. The measurement period lasted three weeks, from October 20 to November 16, 2020, inside the Centre for Atmospheric Science at the University of Manchester. During the sampling period, the PLFs operated continuously inside the building, except for the days when they were taken away to be used at other sites, as detailed in the following sections. In the building, the PLFs’ batteries were constantly charged alongside the reference device. The PLFs were placed within one meter of each other and next to a window. In order to simulate a natural use of the building, windows were opened and closed at various intervals during the indoor trial. In the UK, we spend up to 90% of our time indoors, it was therefore decided to prioritize the indoor evaluation aspect of this study [28].

2.2.2. Roadside site measurement period

The PLFs and POPS devices were used to measure PM_2.5 levels on Upper Brook Street, Ardwick, Manchester (UK), a major arterial road route into the city centre. The measurement period for the road site was on November 13, 2020, from 11:26 AM to 3:41 PM, excluding the period between 1:00 PM and 2:00 PM. The PLFs were set side-by-side on top of the POPS instrument, and both were placed next to the road at the ground level.

2.2.3. NERC Supersite and measurement period

The NERC Supersite provides continuous reference-grade measurements, and therefore provides the ideal site to understand the performance of the sensors and allowed the characterization of the PLFs for PM_2.5, PM₁₀, and NO₂. The station is located on the University of Manchester’s Fallowfield Campus, Wilmslow Rd, Manchester (UK); the location are described in detailed previously [27]. The measurement duration was 12:10 PM to 3:49 PM on November 19, 2020.

Despite the short period of co-location specially at roadside and background site, 3 to 4 hours, measurements at these reference sites are seen as best practice for understanding the performance of the PLF in an outdoor environment and is therefore included in this study.

2.3. Statistical Analysis

All analyses were performed with R statistical software (version 4.0.1 – © 2004–2016 The R Foundation for Statistical Computing; (http://www.R-project.org), and “ggplot2”, “dplyr”, and “tidyquant” packages were used for all data processing. All data from the portable monitoring device and the reference monitoring instruments were recorded at one-minute intervals.

To compare the co-located PLF data with the reference measurements a variety of statistical tools were used, including descriptive statistical analysis and time series plot charts. Four PLFs were excluded due to the devices’ short battery life or the absence of data at the time the measurements were taken. Consequently, only 17 of the 21 PLF devices were included in the analysis. To determine how well the PLF exposure values agreed with those from the references, a Pearson correlation analysis (r) was conducted, to illustrate the relationships of their agreement. Descriptive and correlation coefficient analyses were performed for each site locations.

A linear least squares regression model was performed also for each PLFs and for PM_2.5, PM₁₀, and NO₂ pollutants from the indoor location. The results were summarized using the most common error value metrics: the coefficient of determination of the linear fit (R²) and the root mean squared error (RMSE). This was completed assuming that the reference measurements were free from error and the PLFs would be subjected to measurement error.

The three-week indoor measurement period included a period of exceptionally high ambient pollution, during Guy Fawkes Night. For that reason, the regression was completed over two time periods: a) indoor monitoring, from November 3 to November 7 (five days only, including a Guy Fawkes Night event with high pollutant concentration levels), and b) from October 20 to November 16, 2020 (complete three-week period including the period of high pollution), where the regression was conducted with reference instruments’ (POPS and ARISense) values as the independent variable (x-axis), and PLF values as the dependent variable (y-axis). The equation for the linear regression is (Y = bX + a), “b” is the slope of the regression line, and “a” is the intercept. We evaluated the tool's performance for two different time intervals (1-minute and 1-hour). RMSE provides a good measure of measurement error; an RMSE of zero indicates that all predictions lie on the regression line, suggesting no errors (i.e., good performance). In addition, the higher R² values (close to 1), which range from 0 to 1, indicate better performance.

3. Results and Discussion

For all measured pollutants, the overall correlation coefficient between the PLFs and the reference instruments was often weak (r<0.4). Moderate correlation coefficient with the reference instruments was observed with one of PLFs at indoor location (r = 0.58) and two of PLF at roadside location (0.4 < r < 0.6) when measuring PM_2.5, but not for PM₁₀ and NO₂ concentrations. When analyzing only a subset of the data when the high pollution periods were observed during the Guy Fawkes Night (POPS measurement showed a maximum of 118.7 μg/m³), 11 PLF units showed stronger regression results (R² values > 0.5) with a one-hour compared to a one-minute (n=9) for PM_2.5 . For the full indoor measurement period (3 weeks), 4 PLF units showed stronger regression results (R² values > 0.5) for PM_2.5. PM₁₀ and NO₂ showed consistently poor regression results (R² values < 0.5) in both the raw (1 min), 1 hour averaged, for both the high pollution period and for the full measurement interval.

3.1. Indoor Monitoring

Figure 1 shows measured time series of PM_2.5, PM₁₀, and NO₂ by the reference instrumentation, POPS and ARISense (first column) and 3 PLF devices (PLF2, PLF11, and PLF12) out of 17, illustrating their highly variable performance. The series for the remaining 14 PLF devices are shown in Figure S1.

The reference indoor air quality instruments show strong variability in concentrations over the full measurement period. Levels of air pollution exceeded the WHO threshold during periods from October 20 to November 16, 2020 with a high exceedance detected by the POPS device at 9:24 PM for PM_2.5 pollutant with a level of 118.77 μg/m³ (see Figure 1A). The PM₁₀ and NO₂ reference monitor readings also revealed a significant peak that exceeded the WHO threshold of 242.5 μg/m³ and 59.13 ppb, respectively. This occurred during Guy Fawkes Night from November 5, 2020 at 5:00 PM to November 6, 2020 at 3:00 AM (highlighted with a blue line), and the event also had a clear effect on the indoor air quality at this time. Outdoor air quality has a significant impact on indoor air quality, especially evident when indoor sources of pollution are absent [24]. Natural ventilation, open windows and doors, are the most prevalent ways for outside air to enter and influence indoor environments. Figure 1A, the high and low PM_2.5 readings of the PLFs (PLF11 and PLF12) generally coincided with the PM_2.5 concentrations measured by POPS. However, they did not follow the reference ARISense for PM₁₀ and NO₂ concentrations, as shown in Figure 1B,C. PLF2, for instance, showed inconsistent tracking patterns (high-frequency readings that exceed the WHO threshold) in comparison to the references for PM_2.5, PM₁₀, and NO₂ levels. Most of the other PLFs, as shown in Figure S1, exhibit the same unreliable multiple high peak patterns during this time.

The summary statistics of the indoor monitoring pollutants measured by the PLFs and the POPS and ARISense instruments are shown in Table S1. The mean of indoor PM_2.5 concentration reading taken by POPS over the duration of indoor sampling (three weeks) was 6.21 μg/m³, while that of PLFs ranged between 3.39 and 8.06 μg/m³. On the other hand, PM₁₀ mean concentration from ARISense was 16.7 μg/m³, and that of PLFs ranged between 8.39 and 34.85 μg/m³. The mean of indoor NO₂ reading concentration taken by ARISense was 6.91 ppb, while that of the PLFs ranged between 13.1 and 21.08 ppb.

Overall, the correlation coefficient average between the PLFs and the references was very weak (r values less than 0.4), indicating poor agreement (Supplemental digital content: Tables S4–S6). There was a moderate correlation between PLF and POPS reference for only one PLF unit (PLF19) (r = 0.58), the remining 16 of the 17 PLF revealed a weak degree of agreement for the PM_2.5 (see Table S4). PM₁₀ and NO₂ concentration data measured by the PLFs revealed no to very weak agreement (0.005 < r < 0.29) and (0.02 < r < 0.22), respectively.

Table 1,Figure 2, Figure 3 and Figure 4 show the results of the regression analyses for each PLF. The range of the PLFs’ R² values for PM_2.5 is 0.0–0.63, PM₁₀ is 0.0–0.05, and for NO₂, it is 0.00–0.05. The RMSE value range is 3.2–8.8 for PM_2.5, 20.2–47.5 for PM₁₀, and 13.1–28.7 for NO₂. For the PM_2.5 measurements, 13 out of 17 PLFs have lower R² values (below 0.5; close to 0) and higher RMSE values (far from zero), this suggests that the regression model has a relatively weaker goodness of fit and less accurate prediction, indicating a poor measurement performance of the PLF sensors. Exceptional have been seen in 4 PLF units (PLF11,12,13, and 19), for which R² values ranged between 0.45 and 0.63 (R² close to 1), and their RMSE values range between 3.4 and 4.2 (RMSE close to 0), indicating that all of the variance in the four PLF units is explained moderate to substantial portion of the variance in the references and the model’s predictions have relatively small deviations from the actual observed values, which suggesting a better measurement performance for PM_2.5. The performance of all PLFs, not limited to those four units, showed poor measurements for PM₁₀ and NO₂.

Specifically looking at the performance of PLF11 and PLF12, in Figure 1, where PLF11 and PLF12 seem to show reasonable performance with the PM_2.5 variations as recorded by POPS, the error value metrics (R² and RMSE) also improved. Their R² and RMSE values are 0.58 and 3.4 and 0.58 and 3.6, respectively. The PM₁₀ and NO₂ variation from those two personal units still performed poorly (see Figure 1B,C). The poor linear responses were observed for all PLFs regarding PM₁₀ and NO₂, including those two units (PLF11 and PLF12, see Figure 3 and Figure 4). Their R² and RMSE values are (0.05 and 20.2), and (0.03 and 26.1), respectively, for PM₁₀, and (0.00 and 22.7), and (0.05 and 14.0), respectively, for NO₂.

Figure 4. Regression plots for PLFs and ARISense reference device for the 1-min NO₂ concentrations carried out over a month at the indoor monitoring location. X-axis represents ARISense reference instrument; Y-axis represent PLFs tools; ----: Blue line represents the linear regression fit line between the measured NO₂ concentrations from PLFs and ARISense; ----: Black line represents the 1:1 line. The equation for the linear regression is (Y = bX + a); R² = coefficient of determination.

Figure 4. Regression plots for PLFs and ARISense reference device for the 1-min NO₂ concentrations carried out over a month at the indoor monitoring location. X-axis represents ARISense reference instrument; Y-axis represent PLFs tools; ----: Blue line represents the linear regression fit line between the measured NO₂ concentrations from PLFs and ARISense; ----: Black line represents the 1:1 line. The equation for the linear regression is (Y = bX + a); R² = coefficient of determination.

Table 2 show regression results comparing 1-min and 1-hour over the full indoor measurement period (3 weeks). During this period only 4 out of 17 PLFs showed better performance results (R² values > 0.5, highlighted with gray in Table 2) when measuring PM_2.5 levels with a 1-minute, and 1 out of 17 PLFs performed better (highlighted with blue in Table 2) with a 1-hour. R² values for PM₁₀ and NO₂ showed poor performance (R² values < 0.5) in both 1 min and 1 hour.

Similar have been seen for the short-time period (5 days) that included high levels of pollution (the Guy Fawkes Night event), see Table 3. During this period of high pollution 9 out of 17 PLFs showed better performance results (R² values > 0.5, highlighted with gray in Table 3) when measuring PM_2.5 levels with a 1-minute, and 11 out of 17 PLFs performed better (highlighted with blue in Table 3) with a 1-hour. R² values for PM₁₀ and NO₂ showed poor performance (R² values < 0.5) in both 1 min and 1 hour.

It is clear that, 1-hour intervals and during the selected high pollution period, 11 (out of 17) of the PLFs demonstrate some measurement proficiency. For example, the PM_2.5 R² values for PLF12 are 0.58 (1-min measurements for three weeks period in Table 2), 0.68 (1-min measurements for five days in Table 3), 0.75 (1-hr for five days). An earlier study that tested the same type of tools showed similar results[22]. In their study, they found that only three and four devices (out of 32) gave high precision (80 – 100%) for PM_2.5 and PM₁₀, respectively.

3.2. Roasdside and supersite intercomparison

PM_2.5 pollution levels were measured only at the roadside with the POPS instrument, and PM_2.5, PM₁₀ and NO₂ levels were measured at the Supersite. Due to the short time period of the measurements, only descriptive and correlation coefficient analyses for each site were conducted. At the time of measurements, the air quality measurements at both sites (road and Supersite locations) showed concentrations lower than the WHO 24-hour mean limit value (see Figures S2 and S3). An elevated concentration of PM_2.5 was observed for periods between 3:30 PM and 3:41 PM (November 13, 2020), as detected by POPS at the roadside location for PM_2.5, with the maximum level reaching 18.60 μg/m³ (see Figure 4A).

Figure 4. Time series of POPS and Supersite monitors (reference) and three PLF tools for PM_2.5, 1-min concentrations carried out for over three hours at the (a) roadside and (b) Manchester Air Quality Supersite. ---- : Red dash line represents the WHO limited value (15 µg/m³ for 24-hr mean PM_2.5).

Figure 4. Time series of POPS and Supersite monitors (reference) and three PLF tools for PM_2.5, 1-min concentrations carried out for over three hours at the (a) roadside and (b) Manchester Air Quality Supersite. ---- : Red dash line represents the WHO limited value (15 µg/m³ for 24-hr mean PM_2.5).

Tables S2 and S3 provide a summary of the roadside and Supersite pollutants measured by the PLFs, and references POPS and Supersite instruments. The roadside mean concentration of PM_2.5 recorded by PLFs ranged between 2.4 and 18.54 μg/m³ (maximum of 12.54–75.18 μg/m³), while the mean concentration of 7.12 μg/m³ (maximum of 18.6 μg/m³) with measurements taken by POPS. Findings from the Supersite location, the mean concentration of PM_2.5, PM₁₀, and NO₂ recorded by PLFs ranged between 2.12 and 7.72 μg/m³ (maximum of 11.25–78.91 μg/m³), 8.14 and 27.59 μg/m³ (maximum of 115.41–572.93 μg/m³), 1.8 and 23.89 ppb (maximum of 15.97–183.3 ppb), respectively, while the mean concentration was 3.05 μg/m³, 6.48 μg/m³, and 9.92 ppb (maximum of 4.13 μg/m³, 8.8 μg/m³, and 18.38 ppb) for PM_2.5, PM₁₀, and NO₂, respectively with measurements taken by the Supersite monitors.

Despite the short measurement periods in the outdoor environments, the measured pollutant levels obtained from the PLFs (n= 15 units) do not show a good correlation (r < 0.4) with the concentrations measured by the POPS and supersite instruments at both locations. Exceptional results were seen with two PLFs (PLF2 and PLF17) that showed moderate correlation (0.4 < r < 0.6) when measuring PM_2.5 at roadside location. The correlation tables for the road and Supersite locations are given in the supplementary material (see Tables S8 and S9).

Our field evaluation results show, at best, a weak correlation between the majority of the PLF devices (n=16) and the reference instruments for all measured pollutants. Only one PLFs showed a moderate correlation with the reference data. Our results are in agreement with another field evaluation reported by the South Coast AQMD [23]. Their R² for PM_2.5 measurement between 0.02 and 0.15 for three PLF units and a Federal Equivalent Monitor GRIMM (FEM GRIMM) over an hour period, and was stronger over 24 h of observation (0.02 < R² < 0.72) compere to 5 min (0.01 < R² < 0.09) [22].

4. Conclusions

There is increasing adoption of commercial low-cost air quality monitors by health researchers and public authorities. The body of literature assessing the accuracy of these devices is continuously expanding, but little is reported on commercially available personal exposure monitors. In this study, we evaluated the performance of a market-leading real-time air quality exposure monitor, the Plume Lab Flow2, by comparing its air pollutant measurements (PM_2.5, PM₁₀, and NO₂) against readings from several high-quality air pollution monitor (ARISense, POPS, and Manchester Air Quality Supersite monitoring sites) under field conditions at indoor, outdoor, and roadside locations. Our field evaluation study demonstrated that the PLF cannot offer reliable measurements for use in exposure studies. The results suggest that the PLF is not sensitive enough to measure the very low concentration levels detected by the reference instruments. A limitation of the study is the relatively short duration of roadside measurements. Another limitation arises from the use of multiple different reference instruments, which may introduce a potential bias that is challenging to quantify. Further studies should examine whether PLFs maintain their performance over extended durations to accurately represent their potential real-world applications. Accordingly, a comparison of personal monitoring devices with corresponding reference methods should be a routine quality assurance to continue developing more efficient sensors, to guarantee the reliability of the data.