Evaluation of an Automatic Meteorological Drone Based on a 6-Months Measurement Campaign

1. Introduction

Measurements of temperature, humidity and wind in the boundary layer are sparse, although they are important for the initialization of numerical weather prediction models [2]. In particular, the prediction of fog and convection could profit from additional measurements in the boundary layer [3].

Uncrewed Aircraft Systems (UAS) or drones are often presented as an opportunity to fill the observational gap in the boundary layer [4,5,6]. The World Meteorological Organization recognized the potential of drones and will organize a demonstration campaign in 2024 [7]. They are evaluated for case studies or research application[8,9,10], however, to the best of our knowledge, the operational capabilities of drones, i.e. their capability to operate unattended and continuously, are not yet sufficiently evaluated.

MeteoSwiss operationally assimilates atmospheric profiles from the following measuring instruments into its numerical weather prediction model COSMO (Consortium for Small-scale Modelling): radiosounding, radar and lidar wind profilers, microwave radiometers and aircraft derived observations (AMDAR and Mode-S). Drones represent an innovative technology that can be used to record profiles in the boundary layer and above. To better understand the maturity and the data quality of meteorological drones, MeteoSwiss, together with Meteomatics, conducted a proof-of-concept campaign, which included the long-term operation of a fully automated meteorological drone in Payerne. The main goal of this campaign was to evaluate the system and data availability, as well as the data quality.

In this study, we first focus on the data availability, then we analyze two case studies. In the last section of the document, we investigate the quality of the Meteodrone measurements compared to the sounding and remote-sensing instruments.

2. Materials and Methods

Meteodrone

The Meteodrone MM-670 is a hexacopter designed and manufactured by Meteomatics (https://www.meteomatics.com/en/meteodrones-weather-drones/). The Meteodrone is designed to operate up to a maximum altitude of 6000m, for wind speeds up to 25m.s^-1 (90 km/h) and for temperatures down to −50°C. It is equipped with a parachute in case of emergency. The propellers are heated to be able to fly in icing conditions. The drone has a total weight of 5kg and measures 70cm in diameter.

Figure 1. Meteodrone and Meteobase installed at Payerne.

Figure 1. Meteodrone and Meteobase installed at Payerne.

The Meteobase is an automated shelter used to control the drone remotely and to recharge and protect the drone when it is not in use. A heating system keeps this base above ambient temperature. The drone can be programmed to perform and repeat customized flight patterns fully automatically. For safety reasons, it was mandatory that the drone operations were always supervised by a remote pilot. A “Specific Operations Risk Assessment” was required to obtain clearance from the Federal Office of Civil Aviation (FOCA). One of the safety rules was to call every night the air traffic controller to ensure no air traffic was planned around the drone. The base is equipped with a FLARM system on the ground to avoid potential collisions. The operations were also declared on the Daily Airspace Bulletin in Switzerland (DABS) to warn pilots and drone operators. From April to May, another restriction was added by FOCA: Flight operations were prohibited for wind directions between 315 and 45° to minimize the risk for the drone to come down on the Payerne camping in case of an incident.

For this campaign, the strategy was to fly at least 8 times per 24 hours during working days. Due to aeronautical restrictions, it was not possible to fly during daytime, so the flights were scheduled every hour from 20:00 to 04:00 UTC. To avoid interferences with the radiosonde launched at 23:00 UTC, the corresponding Meteodrone flight was scheduled for 23:15 UTC. The flights were planned to go up to 2000m a.s.l. (around 1500m a.g.l.). The Meteodrone MM-670 is equipped with two temperature sensors, one humidity sensor and one pressure sensor. The wind speed and direction are derived from the power of the 6 engines necessary to maintain its horizontal position.

In this study we assess the manufacturer's final data product which is available in real-time from the drone system. The data were averaged on the way up and down but the details of the data processing (time lag correction, averaging between the sensors…) were not made available by Meteomatics referring to the manufacturing intellectual property.

Sounding

Radiosoundings are operated by MeteoSwiss at Payerne since 1942 [11]. Soundings are launched twice a day at 11:00 and 23:00 UTC. Since 2018, the sondes are of type RS-41 manufactured by Vaisala [12]. The soundings are launched manually during office hours (typically weekdays during daytime) and automatically the rest of the time. The typical maximum altitude is above 30km [13].

Remote-sensing

Payerne is a WMO “Measurement Lead center” equipped with several remote-sensing instruments (https://community.wmo.int/activity-areas/imop/cimo-testbeds-and-lead-centres/Testbed_Switzerland ). To put in perspective the performance of the Meteodrones, we aimed to compare Meteodrone measurements to state-of-the-art remote-sensing instruments.

At Payerne, wind is measured at high temporal and spatial resolution with a Doppler lidar WLS-200 manufactured by Leosphere (Vaisala Group). It is a scanning lidar measuring around 1500nm with a resolution of 50m. The wind is calculated using the Doppler Beam Switching (DBS) technique with an independent wind profile every 15 seconds and an update rate of 3 seconds. In its operational configuration the measurement range reaches from 100m above ground to the top of the boundary layer. A radar wind profiler PCL1300 manufactured by Degreane is also measuring wind profiles [14,15]. It is a UHF radar also using the DBS technique with 5 beams at an elevation angle of 75°. The radar can provide an independent profile every 20 minutes with an update rate of 10 minutes and a vertical resolution of 144m starting at 350m above ground level.

In this study, we use the temperature from a microwave radiometer HATPRO-G5 manufactured by RPG [16,17,18]. The radiometer is measuring temperature profiles every 5 minutes by performing a scan with 11 elevation angles. A neural network is used to calculate the profile from the brightness temperatures measured by seven channels between 51 to 58 GHz. The profile is calculated at 55 altitudes between the ground and 2.5km above the ground. Humidity measurements of the HATPRO are not used in this study since the vertical resolution of these humidity profiles is not sufficient for our purpose [19].

The humidity is instead measured by the Raman Lidar for Meteorological Observations, RALMO [20,21]. RALMO is emitting at 355nm and is using the nitrogen and water vapor rotational-vibrational Raman signals to estimate the specific humidity. RALMO provides an independent profile every 30 minutes with a vertical resolution of 30m starting at 90m above the ground. The temperature retrieval of RALMO [22] is not used in this study.

The cloud base height is derived by a CL31 ceilometer manufactured by Vaisala [23]. The instrument is a low power lidar emitting in the infrared at a wavelength of 910nm and reports cloud base height from the ground to 7.7km. The instrument is reporting an independent cloud detection every 30 seconds with a resolution of 10m.

WMO requirements and quality analysis

WMO defines requirements for the observation of physical variables for a number of application areas including high-resolution numerical weather prediction [1].

WMO defines the "threshold" as the minimum requirement to be met to ensure that data are “useful” (table 2). The "goal" is the threshold above which further improvements are of no added value for the given application. The "breakthrough" is an intermediate level between "threshold" and "goal" which, if achieved, would result in a significant improvement for the targeted application. The “breakthrough” level may be considered as an optimum, from a cost-benefit point of view, when planning or designing observing systems.

According to WMO, the "uncertainty" characterizes the estimated range of observation errors on the given variable, with a 68% confidence interval (1σ). To estimate this uncertainty, we make the hypothesis that the sounding has a negligible error compared to the other sensors and that the uncertainty should be understood as the root mean square (RMS) of the difference between the sounding and the drone or the remote-sensing instrument, respectively. To compare the measurements, all profiles were resampled on a vertical reference grid with 20m vertical spacing taking the average of all points on the original grid within +/- 10m of the levels of the reference grid. When a remote-sensing instrument was not reporting data in this vertical interval the level was ignored in the comparison. Both the radiosounding and the drone measurements were considered as instantaneous. The profiles were evaluated when the drone measured within a 45-minute interval around the sounding start time. Remote-sensing instruments were averaged from the drone average time plus or minus 10 minutes.

WMO uncertainty requirements for high-resolution NWP for the planetary boundary layer (PBL) or the free troposphere (FT) are identical for the temperature and the specific humidity. For the wind, the “threshold” requirement is 5m/s in the PBL and 8m/s in the FT. In this study we selected 5m/s as it is also the threshold used to monitor wind measurements in the E-PROFILE network [24].

These WMO requirements were recently used by Gaffard et al. to evaluate novel measurements like a DIAL Lidar [25]. In this study, we aim to evaluate if Meteodrones fulfil these requirements.

3. Results and discussion

The evaluation is divided into 3 sections: first, a discussion on availability, second a case study demonstrating the potential of the Meteodrone. Finally, the quality is evaluated over the whole period.

Availability

The campaign took place from December 2021 to May 2022.

The Meteobase was installed at Payerne on 02 November 2021. In November 2021, 45 flights were performed. Because of technical problems during landing including an emergency landing due to a problem in the communication between the drone and the base, the official start of the campaign has been set to December and the flights performed in November are not considered in this evaluation.

864 flights took place during the main campaign between December 2021 and May 2022. With 128 working days during this period and 8 flights per 24h planned, 1024 flights were scheduled to be performed during the campaign leading to an availability of 75.7%. For 87 nights, flights were performed according to schedule with at least 8 flights per night (68%).

For the missing flights, 50.3% were not performed due technical problems, 22.8% due to weather conditions and 26.9% due to air space restrictions. Around 10 flights encountered an important technical problem that required a manual intervention to put the drone back on its base. In January 2022, a motor failure led to a second emergency landing on which the drone was damaged. In total, five different drones were used to continue measuring after technical difficulties.

During the same period, the availability was 95% for the Doppler Lidar 94.3% for the radar wind profiler, 99.3% for the microwave radiometer and 70.2% for the Raman lidar.

Table 1. Number of flights performed by the Meteodrone.

	# flights	Complete nights
Planned	1024	128
Effective	864	87
Effective in %	75.7%	68 %

Table 1. Number of flights performed by the Meteodrone.

	# flights	Complete nights
Planned	1024	128
Effective	864	87
Effective in %	75.7%	68 %

709 flights (82.1% of the effective flights) reached the nominal altitude of 2000m above sea level (Figure 2). The remote Meteodrone operator aborted the remaining 155 flights (17.9% of the effective flights), mostly due to delicate atmospheric conditions like high wind speeds. The maximum wind speed recorded by the wind lidar between the surface and 2000m was 31.3m.s^-1 suggesting that the drone flew above the theoretical limit of 25m.s^-1. The minimum temperature recorded by the microwave radiometer between the ground and 2000m was -11.2°C.The minimum temperature recorded by the microwave radiometer between the ground and 2000m was -11.2°C.

91.3% of the flights were performed within 10 min around the nominal time. Data were transferred in real time to the MeteoSwiss server and 92.6% of the files were available on the MCH server within 10 min.

Case study 1: 24/05/2022

During the night from 24 to 25 May 2022, 9 flights were performed. The overview of the measurements is shown in appendix A1. Figure 4 shows the profiles recorded by the radio sounding at 23.00 UTC and at 23:19 UTC by the drone. In blue, the Meteodrone measurements are presented, in black the radio soundings and in red the remote-sensing measurements. Figure 4 shows temperatures around 13°C at the ground, decreasing to 4°C at 2000m with a small temperature inversion at around 750m asl. All instruments show the temperature inversion, but the sounding reported a more detailed structure compared to the other instruments. A humid layer was measured around 1500m with a maximum relative humidity around 90%. That night, the Raman Lidar RALMO was not operational, therefore there is no humidity profile measured by a remote-sensing instrument displayed in Figure 4. Light precipitation was recorded at the beginning of the night (not shown). The wind speed was low with values below 5m/s. All instruments provide comparable results, even if the drone overestimated the wind speed by 1.2m/s (67%) on average for this profile. The wind direction was shifting from easterly winds at the ground to northerly winds at 1500m asl.

This case study highlights the good data quality a Meteodrone can achieve. It provides profiles of temperature, humidity and wind in the boundary layer with a higher temporal frequency compared to a radio sounding. It is also providing these measurements with one instrument instead of having different remote sensing instruments.

Nevertheless, like for all instruments, there are some drawbacks: case study 2 illustrates challenges that might be encountered with a Meteodrone.

Case study 2: 15/12/2021

On the night between the 15 and 16 December 2022, 9 flights were performed. The overview of the measurements is shown in appendix A2. Figure 5 shows the profiles recorded by the radio sounding at 23.00 UTC and at 23:20 UTC by the drone. In blue, the Meteodrone measurements are presented, in black the radio soundings and in red the remote-sensing measurements. Figure 5 shows temperature around 0°C at the ground decreasing to -3°C at 1000m with an important temperature inversion above, with a maximum temperature of 7.7°C at 1916m. All instruments show the temperature inversion, but the sounding again reported a more detailed structure. The minimum and the maximum of the remote sensing measurements between 23:00 UTC to 23:30 UTC are represented by the grey areas, showing that the conditions were steady over time during the case study Thus, the differences between the radio sounding and the Meteodrone cannot be attributed to atmospheric variability.

A thick stratus was present this night with cloud base height reported by the ceilometer at 530m. It is confirmed by the relative humidity of 100% from 660m to 1060m in the radiosonde relative humidity measurements. That night, the Raman Lidar RALMO was not measuring because of the low clouds, therefore there is no humidity profile measured by a remote-sensing instrument on Figure 5.

The Meteodrone was not measuring relative humidity above 97.4% even if the sensor seems saturated between 700m to 900m. Similar underestimations of high relative humidity were observed on several occasions at the beginning of the campaign: the Meteodrone never measured 100% in December, even in clouds.

The dry layer around 1800m was detected by the Meteodrone but with a relative humidity of 25% instead of 4.6%. This is likely due to the overestimation of the temperature at this altitude or a contamination of the sensor. According to Meteomatics, the positioning of the sensor was not ideal and might have led to a contamination of measurements. The location of the sensor was changed later in January. In terms of specific humidity, the Meteodrone reported a minimum of 1.67g/m³ instead of 0.3g/m³ measured by the radio sounding. Due to this major difference, this case was excluded from the humidity analysis in this document.

The wind speed at the ground was low, progressively increasing to 10m/s at 2500m. All instruments provide comparable results even if the drone overestimated the wind speed by 1.18m/s (20%) on average. The wind lidar was measuring in the lower part of the stratus and the wind radar was measuring above. The wind direction was relatively constant with north-easterly winds.

Quality evaluation

In order to provide a quantitative evaluation, the measurements from Meteodrones and remote-sensing instruments were compared to the radiosoundings. 97 flights were performed around 23:15UTC, close to the radiosonde launch. These profiles are visible in Appendix A5 and A6. For these profiles the difference and root mean square (RMS) of the difference was calculated.

For this analysis, 8 humidity profiles were excluded:

• 3 cases with a dry layer above fog that was not correctly measured (15, 16 and 20 12/2021). As discussed in case study 2, a contamination of the humidity sensor might be the origin of these differences with an RMS for specific humidity reached values up to 285% the 20/12/2021.
• 4 cases where the humidity of the sensor was not recorded correctly (24, 25 and 26 01/2022 and 16/03/2022).
• 1 case with a very dry layer (RH<1.6% measured by the radiosonde and RH<6.95% measured by the Meteodrone on 28/02/2022) leading to significant difference in terms of specific humidity (72%). This result is not representative of the performance of the Meteodrone this day as the Relative Humidity measured by the Meteodrone was close to the one measured by the RS-41 (see Appendix A6).

The profile of the temperature, relative humidity and wind speed differences are visible on Figure 6 and the RMS are presented in table 2, together with the WMO requirements for these parameters. On Figure 6 the agreement between the Meteodrone and the radiosonde is visible with median differences lower than 0.5°C.

However, some biases for temperature and relative humidity are visible at the bottom and the top of the profile. It is likely that the overestimation close to 0.5°C at low altitude is due to the effect of the Meteobase which keeps the drone at controlled temperature affecting the lower part of the measurements. The explanation for increasing temperature bias from -0.27°C at 650m to 0.56°C at 1990m is not known by the authors and more detailed data would be needed to investigate this behavior (for example the variability of the sensors, the difference between up and down measurements or the inclination of the drone).

On average, the relative humidity is underestimated by the Meteodrone by -1.6%. Close to the ground, the relative humidity is underestimated by -4%, it is reaching 0% at 600m and decreasing with altitude reaching -3.3% at 1990m. This effect seems linked to the temperature overestimation and to the underestimation of RH in clouds, as mentioned in case study 2.

Figure 6 shows that the wind speed is overestimated by the Meteodrone. The bias is nearly constant around 1.5m.s^-1 up to 1900m but the value is increasing at the top of the profiles with a median overestimation of 3.25m.s^-1. For the wind lidar the median wind difference is -0.1m.s^-1, suggesting a better performance of the lidar. For the wind direction the mean difference is 10.6° with a standard deviation of 45.3°. It can be compared with a mean wind direction difference of -1.8° for the wind lidar and a standard deviation of 30°.

As explained in section 3, there is a 15 min delay between the radiosonde and the other measurements. We assume that the impact of this delay is negligible, as the bias shown by the Meteodrone is not visible on the remote sensing instrument even if they are evaluated at the same time as the Meteodrones.

As shown in Table 2, Meteodrones and HATPRO-G5 radiometers meet the “breakthrough” target for temperature with an RMS smaller than 1K.

For the humidity evaluation, after filtering the inaccurate profiles, the RMS for specific humidity is 8.3% and hence meeting the threshold requirement of 10%. The Raman Lidar presented comparable results with a slightly higher uncertainty estimated at 9.2%. Using a similar technique, Gaffard et al. [25] evaluated a Dial lidar and reported an uncertainty between 5 % and 10%, suggesting similar performances.

For the wind, the root-mean-square-vector-difference (RMSVD) calculated for the Meteodrone was 3.1m.s^-1. The value was significantly better for the wind lidar with an RMSVD of 1.8m.s^-1, meeting the “breakthrough” requirement. Meteomatics mentioned that the algorithm was not updated after a drone modification and the results should be better in an updated version of the algorithm (personal communication 2022). As the wind speed is based on the propellers speed adjustments, a minor modification on the drone can lead to significant differences in the wind estimation. Meteomatics is working on a bias correction to improve the wind estimation.

Table 2. WMO requirements for high resolution NWP compared to drone and remote-sensing performances for temperature, humidity and wind profiles.

	Goal	Break-through	Threshold	Drone	Remote-sensing
Atmospheric temperature	0.5 K	1 K	3 K	0.68K	0.72K	Microwave radiometer
Specific humidity	2%	5%	10%	8.3%*	9.2%*	RALMO
Wind (horizontal)	1m/s	2m/s	5m/s	3.1 m.s-1	1.8 m.s-1	Wind Lidar

Table 2. WMO requirements for high resolution NWP compared to drone and remote-sensing performances for temperature, humidity and wind profiles.

	Goal	Break-through	Threshold	Drone	Remote-sensing
Atmospheric temperature	0.5 K	1 K	3 K	0.68K	0.72K	Microwave radiometer
Specific humidity	2%	5%	10%	8.3%*	9.2%*	RALMO
Wind (horizontal)	1m/s	2m/s	5m/s	3.1 m.s-1	1.8 m.s-1	Wind Lidar

Time evolution

Figure 7 represents the monthly average of the differences between radiosoundings and Meteodrones or remote-sensing instruments, respectively. The RMS of the temperature difference is evolving from 0.75K in December to 0.31K in May. This improvement can be partly explained by the meteorological conditions, but it is likely that the quality of the Meteodrone measurements increased after replacing the sensors on 31/01/2022. For the radiometer, the evolution of the RMS of the temperature difference is smaller: from 0.73K to 0.58K. Again, this suggests that improvement seen in the RMS of the Meteodrone is likely due to technical improvements made during the campaign. For the humidity, the evolution is even bigger with an RMS of the specific humidity difference of 30.0% difference in December and 4.7% in May. The significant differences observed in December and January could be explained by a chemical contamination on the humidity sensor leading to a bias like observed by Wang et al. [26] on radiosondes. Again, the replacement of the sensors on 31/01/2022 clearly improved the performances of the Meteodrone. In comparison, the performance of the lidar is stable with an RMS of 8.55% in December and 9.1% in May.

For the wind, the evolutions of the RMS of the wind vector difference from 2.7m.s^-1 in December to 3.0m.s^-1 in May for the Meteodrone and 1.71m.s^-1 in January to 1.32m.s^-1 in May for the Doppler lidar do not demonstrate a clear trend for the two instruments.

4. Conclusions

For the first time, an extensive drone campaign was conducted over a 6-months period and the results were compared to co-located radiosoundings. A total of 864 flights were performed during the campaign, resulting in an availability rate of 75.7%. The missing flights were due to technical problems (50.3%), air space restrictions (26.9%) and adverse weather conditions (38%). Among the flights, 82.1% reached the nominal altitude of 1500m above ground, while the remaining flights were aborted mainly due to higher wind speeds at altitude. Additionally, 91.3% of the flights were conducted within a 10-minute window around the nominal time. To ensure optimal availability, five different Meteodrones were used throughout the campaign. Furthermore, sensor replacements were made after January 31, 2022, to enhance the quality of humidity data.

In this study, we selected two cases: one demonstrating the performance of the Meteodrone, another one illustrating the challenges encountered during the campaign like measuring within and above clouds or wind speed over-estimation.

The quality was analysed for 97 flights performed at the same time as the radiosoundings with Vaisala RS-41. The results were put in perspective with the same methodology applied on remote-sensing measurements. Based on these findings, we find that the Meteodrone meets WMO’s “breakthrough” requirements for high resolution NWP in terms of temperature. The Meteodrone also meets the WMO requirements for wind for high resolution NWP. For humidity, the requirement was met after excluding 8 cases where the humidity sensor did not perform optimally. The classifications for temperature and humidity derived from this campaign measurements are the same for remote-sensing instruments and the Meteodrone. The wind lidar, however, performed better than the Meteodrone wind measurement during this experiment, reaching the “breakthrough” requirement. Meteomatics is currently working on a bias correction for the wind and we expect that the “breakthrough” requirement will be reached. At the end of the campaign, Meteodrones reached the “goal” value for temperature and the “breakthrough” for humidity, thus becoming a unique tool to measure temperature, humidity and wind profiles with a high temporal and vertical resolution under a wide range of meteorological conditions.

Further analysis is needed to validate the drone measurements during daytime as the flights were allowed only during night.

The Meteodrone is now operated at Payerne by Meteomatics in the framework of the DETAF 2.0 project. On the 01/06/06, 1373 flights were performed by Meteomatics at Payerne. In this project, not financed by MeteoSwiss, up to 15 stations should be operational in Switzerland at the end of 2024. One of the aims is to assimilate these data into MeteoSwiss numerical weather prediction model to provide better forecasts.

This proof of concept is opening the door to operational measurements with drones for providing accurate meteorological observations in the boundary layer and is a first step towards the worldwide demonstration campaign that will be organized by WMO in 2024.