Precision and Characteristics of Satellite Spatial Quality Estimators Measurement using Edge Target imaged with KOMPSAT-3A

1. Introduction

Remote sensing satellites’ spatial quality values play two important roles in the satellite development process. The first involves modulation transfer function (MTF) requirement value, i.e., MTF value at Nyquist frequency (hereinafter MTF50). Using this value, satellite developers can verify that MTF50 requirement is met during satellite design, development, and pre-launch testing. The second role is in measuring the satellite’s MTF50 value in orbit after launch and verifying whether MTF50 requirement is satisfied, which constitutes the calibration and validation phase. This work describes the details of the post-launch on-orbit spatial quality measurement method.

The basic measurement method used to determine the spatial quality (RER, FWHM, MTF50, MTFA) (Figure 1) of an optical remote sensing satellite in orbit after launch has already been proposed and described by several organizations, including ISO12233 [1,2,3,4,5,6,7,8,9,10]. Many other satellite organization and companies have developed and used their own ground reference targets and measurement codes, Maxar (USA), Astrium (France), TELESPAZIO (Italy), I2R Corp. (USA), ESA ESRIN (Italy), CNES (France), ONERA (France), CSIR (South Africa), USGS (USA), NASA (USA), AOE (China), and KARI (Korea) [11,12,13,14]. However, there are many uncertain factors within the procedure for the measurement of spatial quality values [16,17,18]; as a result, even if the same Edge target is used, the resulting values will differ depending on the spatial quality code used [11]. Therefore, the uncertain factors that affect the accuracy and precision of the measured values within the spatial quality measurement procedure have been analyzed with simulated Edge target images [19,20,21,22,23]. However, due to the limitations of simulated Edge target images, it is difficult to identify various problems that exist in real Edge target images.

MTF50 is the most commonly used estimator for satellites’ spatial quality, but the measurement precision of MTF50 is known to be much lower than that of RER, FWHM, and MTFA [25,29]. Therefore, internal calibration and validation (hereinafter Cal/Val) engineers use more precise and intuitive spatial quality estimators, such as RER, FWHM, and MTFA, instead of MTF50 [24,25]. Among these, MTFA is less commonly used. It was first proposed by KARI along with MTF50 as a spatial quality estimator with high precision on the MTF curve [17,24,25]. The precision of each spatial quality estimator in practice has not been quantitatively compared. This study is the first to do so using the Coefficient of Variance (hereinafter CV), which is used to denote the precision of a measurement in statistics [15].

The precision of spatial quality measurement values varies depending on the conditions at the time of imaging, the state of the Edge targets, and the measurement code used, etc. This work compares and summarizes the results of the spatial quality measurement of real Edge targets using KOMPSAT-3A (Ground Sample Distance (GSD), 0.55m; MTF50 > 8%) imagery covering the period of January 29, 2017 to March 16, 2024. Finally, the most reliable estimator in terms of precision is proposed. In addition, as only an MTF50 value of 8% or more is provided as a requirement for KOMPSAT-3A, and RER, FWHM, and MTFA values are not specified, it is difficult to compare the accuracy between these spatial quality estimators. Thus, this work focuses on the analysis and comparison of the precision.

In practical terms, the spatial quality measurement with the Edge target seeks to reflect the state of the satellite and the Edge target as accurately possible at the moment of imaging, while minimizing the influence of factors that could reduce the accuracy and precision of the measurement. Additionally, under the assumption that the satellite is not perfectly manufactured, it is necessary to explain the measurement results using asymmetric Edge Spread Function (ESF) and Line Spread Function (LSF) plots [26].

The Committee on Earth Observation Satellite Working Group on Calibration and Validation, Infrared and Visible Optical Sensors (CEOS WGCV IVOS) [27] has organized the Geo-Spatial Quality Working Group within IVOS since 2012. It seeks to resolve the problems associated with the satellite spatial quality measurement procedure mentioned above, and its findings were summarized and published in 2019 [11,28,29]. The authors of the present work also participated in this working group, which led to the spatial quality measurement code provided by the KARI Cal/Val team and using KOMPSAT-3 (GSD, 0.7m) satellite imagery. They specified the main purpose of this working group as follows: “Get the reasonable quantity of Spatial quality for the remote sensing satellite in the Real conditions” [17]. This work focuses on the findings presented at that time.

Section 2 describes the spatial quality estimators and Edge targets, and Section 3 summarizes the main factors that affect the errors of the measured values within ESMP. Section 4 compares and analyzes the values of the spatial quality estimators measured from the Edge target images taken by KOMPSAT-3A. Section 5 concludes the paper and briefly introduces future directions and topics for future papers.

2. Spatial Quality Measurement Estimators and Edge Targets

2.1. Spatial Quality Measurement Estimators

Figure 1 is a plot showing the procedure of measuring the spatial quality estimators (RER, FWHM, MTF50, MTFA) using the Edge target. We show ESMP output plot, using an edge Region Of Interest (ROI) image from Level 0 (raw image data) in the Across (detector; static) direction, imaged by KOMPSAT-3A on February 26, 2019, at a Roll tilt of 29.31deg, for the Edge target located in Zuunmod, Mongolia. Since KOMPSAT-3A uses a Time Delayed Integration (TDI) CCD, the Along (flight; dynamic) direction has a predicted value known as the line rate and is inaccurate, so only the Across direction’s spatial quality measurement is used.

RER is the slope value at the center inflection point of ESF, and FWHM is LSF width value at a ‘0.5’ normalized Y-axis value in LSF. MTF50 is MTF value at Nyquist frequency of the MTF curve, and MTFA is the area value in the region from ‘0’ to ‘0.5’ (Nyquist frequency). In order to calculate RER in an asymmetric ESF and LSF [24,26], it was suggested by the authors at the Geo-Spatial Working Group to calculate RER as a tangent from the RER center, rather than using the conventional method. This is still under discussion [17] (see Section 3).

2.2. Edge Targets

Figure 2 is a list of Edge targets deployed and in operation around the world that is currently available on the CEOS WGCV IVOS portal webpage [30] and USGS EROS webpage [31]. The Edge target located in Zuunmod, Mongolia was constructed and operated by KARI [32].

3. Factors Affecting Measurement Values in the Spatial Quality Calculation Procedure

This section describes the factors listed in Table 1 that affect the errors of the measured values within ESMP. All factors in Table 1 are implemented in ESMP according to the calculation procedure. The initial values in Column 3 of Table 1 for each factor are obtained according to the statistical InterQuartile Range (IQR) outlier exclusion method (Q1 - IQR * 1.5 to Q3 + IQR * 1.5) using KOMPSAT-3A, which involves a set of 966 Across edge images extracted from Edge targets (Baotou, India, Salon, Zuunmod) imagery acquired from January 2017 to March 2024, and the initial values given in this work are entered in ESMP. The initial values of all factors are not finalized yet.

3.1. ‘1. Asymmetric ESF and LSF’

An asymmetric ESF and LSF are some of the most important factors determining the accuracy and precision of spatial quality measurements [26]. Almost all satellite cameras have asymmetric ESF and LSF characteristics, but to varying degrees. Thus, when fitting ESF, non-parametric fitting should be used. As a result, as shown in Figure 3a, the location of the RER center changes, and the proportional relationship between RER value and FWHM value becomes weaker (Section 4.2).

3.2. ‘2. Straighness of Edge’

‘2. Straightness of Edge’ is a quantitative value used to determine the straightness of the line of edge, which is the most important requirement for the Edge target (Figure 3b). ESMP uses a ‘2. Straightness of Edge’ constraint that excludes edges with a statistical ‘Fitting Error (FitErr)’ greater than ‘0.1’ pixels according to the IQR outlier method using 966 Across edge images acquired by KOMPSAT-3A. ‘FitErr’ is obtained as the standard deviation (StDev) of the edge points over the line of the edge. If ‘4. Straightness of Edge’ value exceeds the constraint value; it directly affects the ESF fitting and reduces the accuracy and precision of the spatial quality measurement.

3.3. ‘3. Noise of Bright and Dark Area on ESF’

‘3. Noise of Bright and Dark area on ESF’ was intended to provide a minimum SNR (Signal-to-Noise Ratio) value to measure the spatial quality, but it is difficult to intuitively provide a minimum SNR value due to the complexity of the factors related to SNR. In the general SNR calculation method [37], the value is proportional to the signal (average DN; Digital Number), so SNR values in bright and dark area are different. Thus, the proposed SNR calculation formula [4], in which it is calculated as the brightness difference between bright and dark area (ΔDN = average bright DN – average dark DN), is incorrect [38]. A lower SNR results in larger noise on ESF, which leads to relatively lower fitting accuracy and precision. ESF fitting accuracy and precision directly depend on noise rather than the signal. Moreover, ‘4. DN difference between Bright and Dark area (ΔDN)’ is somewhat related to the ESF fitting accuracy and precision. The smaller ΔDN, the more sensitive it is to noise. But there is no proportional relationship between SNR and spatial quality results (Figure 6 right).

Increased noise on the ESF can result in poor fitting accuracy and precision in bright and dark area and unreliable MTF curves, as shown in Figure 4. ESMP statistically calculates the standard deviation of each bright and dark area as a noise value and uses it as a constraint. If the ‘noiseBr (noise in bright area)’ is greater than ‘0.05’ and the ‘noiseDa (noise in dark area)’ is greater than ‘0.045’, the result is excluded as a constraint according to the IQR outlier method [36].

3.4. ‘4. DN Difference between Bright and Dark Area (ΔDN)’

‘4. DN difference between Bright and Dark area (ΔDN)’ is the average DN value of the bright area minus the average DN value of the dark area, which indirectly affects spatial quality calculation. Low noise and a high SNR mean that the spatial quality calculation is more accurate and precise. Therefore, it is difficult to use ΔDN alone as a constraint because it must be considered together with ‘3. Noise of Bright and Dark area on ESF’. When ΔDN is small, the spatial quality measurement precision is low, but there is no proportional relationship between ΔDN and the spatial quality result (Figure 6 left).

ΔDN value varies depending on the DN range (radiometric resolution) and SNR of the satellite. KOMPSAT-3A has 14bit radiometric resolution (0~16383 DN) and SNR > 100 at 3000 DN, and constraint values of ΔDN > 1000.

3.5. ‘5. Edge Angle between Line of Edge and Across Direction (EdgeAng)’

‘5. Edge angle between Line of Edge and Across direction (EdgeAng)’ theoretically has no effect on spatial quality calculation. The Edge angle value is reflected in the ESF calculation step as the COS(EdgeAng), and it is found that the spatial quality measurement results of KOMPSAT-3 may not be related to the Edge angle [25]. However, small Edge angle will cause pixel data to clump together on ESF, preventing the use of the Edge angle to ensure that pixel data is uniformly and densely distributed on ESF, resulting in lower ESF fitting accuracy and precision. Thus, a constraint can be used that excludes the Edge angle values less than ‘2.2 deg’ according to the IQR outlier method. It also excludes the Edge angle values greater than ’30 deg’ [36]. In theory, there should be no correlation between the Edge angle and spatial quality values since COS(EdgeAng) is applied, but the spatial quality measurement results of KOMPSAT-3A show that there is a somewhat correlation. (Figure 7).

3.6. ‘6. RER Center’

In general, the ‘0.5’ point of the ESF Y-axis is used as the RER center, but, in practice, the RER center’s location should be determined differently depending on the degree of asymmetry ESF and LSF. ESMP calculates the tangent slope at the RER center, which is the inflection point of the edge region on ESF and the location of the peak on LSF (green line in Figure 3a). There is controversy regarding the definition of the RER center in an asymmetric LSF [17,24]. RER value of the ESMP’s RER center is relatively larger and has higher precision than RER value of the normalized DN = 0.5 center.

3.7. ‘7. Total Trim Width of ESF’

As satellite cameras are not perfectly manufactured, there are two problems: (1) ESF and LSF are asymmetric [26], and (2) the bright and the dark area of ESF are not flat. Therefore, in spatial quality calculation, the edge width of ESF depends on the characteristics of the satellite camera [24]. Due to (2), a large trim width results in a lower spatial quality value, but the result is relatively more reliable. However, it is very difficult to obtain a large edge area with uniform bright and dark area on a ground Edge target, so the total trim width ‘18 pixels’ of KOMPSAT-3A is used in ESMP as a compromise [24].

3.8. ‘8. Fitting Method for ESF’

The fitting method used to measure ESF from the edge pixel data is the factor that has the greatest impact on the results. There are two main types of fitting methods available here: parametric and non-parametric fitting methods. If the satellite camera is symmetrical, a parametric fitting method such as the Fermi–Dirac equation can be used. But most satellite cameras are asymmetrical [26], so a non-parametric fitting method must be used (Figure 3a). As a result, the fitting method used is a critical factor regarding the results of spatial quality measurements. For asymmetric cases, the Savitzky–Golay method is popular, but ESMP uses the Cubic Smoothing Approximation Spline fitting (CSAPS) by default, which has the smallest fitting error though its own testing. CSAPS is sensitive to the number of edge row lines (EdgeLine) and its weight value, so it has the disadvantage that these two must be determined as initial values. However, CSAPS tends to produce better spatial quality values than other fitting methods.

3.9. ‘9. Noise Removal Method for ESF’

The Edge target on the ground is rarely kept perfectly clean, so the final spatial quality value depends on the method applied to remove noise contained in the edge pixel data on ESF. The suitable method depends on the state of the Edge target and how it is coded within ESMP, so the results are different. ESMP uses CSAPS twice to remove noise on the ESF (Figure 4). After the first CSAPS fitting with weight = 0.95 to remove the ESF pixel data as outlier noise over 2σ of noise in the bright and dark area, a second CSAPS fitting with weight = 0.95 is performed. Normalization is performed via the fitted ESF.

3.10. ‘10. Number of Edge Row Lines (EdgeLine)’

‘10. Number of Edge row lines (EdgeLine)’ is directly related to the CSAPS fitting used by ESMP. Due to the nature of CSAPS, ESF is not perfectly fitted, and the fitted ESF is normalized, so there is a proportional relationship between the EdgeLine value and the spatial quality result [24,33]. However, it is assumed that most fitting methods also have the same problem. Statistically, the spatial quality values converge to a certain value when EdgeLine is above 40pixel, but, realistically, it is very difficult to obtain an EdgeLine value above 40pixel on the Edge target. Thus, at present, ESMP has fixed EdgeLine at ‘21’ and continues to conduct further research [33].

4. Results and Discussion

This section presents a comparative analysis of the spatial quality estimators via the spatial quality results obtained using ESMP for 483 Edge target images acquired by KOMPSAT-3A between January 29, 2017, and March 16, 2024. Two Across edges and two Along edges are counted for each Edge target. The Across MTF can be measured in the laboratory before launch, and it is commonly used to represent the spatial quality. Thus, this works describes 840 results obtained with ESMP’s constraint, out of a total of 966 Across edge spatial quality results. Table 2 compares the 966 Across edges and 840 constrained results. After applying the constraint, the average value is slightly improved, and the comparison the Coefficient of Variation (CV; StDev/Average) value, which represents the precision, indicates that the precision is clearly improved.

This section is divided into two main parts: first, it describes the characteristics and biases of the spatial quality measurement results obtained by KOMPSAT-3A; second, it compares and analyzes the representative spatial quality estimators of RER, FWHM, MTF50, and MTFA to highlight the precision of each estimator.

4.1. Characteristics and Biases of Spatial Quality Measurement Results

The characteristics and biases of the spatial quality measurements from 840 Across edge images are plotted and summarized for the following factors:

• Julian date (since 20120517 (May 17, 2012), when KOMPSAT-3 was launched);
• Roll tilt angle;
• ΔDN (bright average DN – dark average DN);
• SNR (DN / StDev at DN 3000);
• Edge angle;
• Edge target (Baotou, India, Salon, Zuunmod).

Figure 5 (left) shows the variation in KOMPSAT-3A’s spatial quality over time with the Julian date. The red number at the top right is CV value. The Y-axis scales for each of RER, FWHM, MTF50, and MTFA are given according to CV value. In Figure 5 (left), we can note the following:

• KOMPSAT-3A’s spatial quality is very stable over time and does not show any adverse effect due to aging.

Figure 5 (right) shows the relationship between KOMPSAT-3A’s spatial quality and the roll tilt angle at the moment of imaging. In Figure 5 (right), we can note the following:

• There is no proportional relationship between the roll tilt angle and spatial quality.

Figure 6 (left) shows the relation between KOMPSAT-3A’s spatial quality and ΔDN. In Figure 6 (left), we can observe the following:

• There is no proportional relationship between ΔDN and the spatial quality.

Figure 6. (left) ΔDN vs. RER, FWHM, MTF50, and MTFA. (right) SNR vs. RER, FWHM, MTF50, and MTFA (upper right in red: CV). Scale of Y-axis was changed to CV value.

Figure 6. (left) ΔDN vs. RER, FWHM, MTF50, and MTFA. (right) SNR vs. RER, FWHM, MTF50, and MTFA (upper right in red: CV). Scale of Y-axis was changed to CV value.

Figure 6 (right) shows the relation between KOMPSAT-3A’s spatial quality and SNR. In Figure 6 (right), we can observe the following:

• There is no proportional relationship between SNR and spatial quality.

Figure 7 shows the relationship between the KOMPSAT-3A’s spatial quality and the Edge angle. In Figure 7, we can observe the following:

• In Figure 7 (left), there appears to be a correlation between the Edge angle and the spatial quality, and, in Figure 7 (right), this may be due to the difference in state with the Edge target. However, since the Edge angle is different for each Edge target and Baotou's Edge angle is the largest, it is presumed that it is due to the Edge angle.

Figure 7. (left) Edge angle vs. RER, FWHM, MTF50, and MTFA, separated by the Edge target. (right) X-axis for Edge targets in the order of Baotou (blue), India (red), Salon (green), and Zuunmod (purple), separated by imaging order vs. RER, FWHM, MTF50, and MTFA (upper left: average and CV). Scale of Y-axis was changed to CV value.

Figure 7. (left) Edge angle vs. RER, FWHM, MTF50, and MTFA, separated by the Edge target. (right) X-axis for Edge targets in the order of Baotou (blue), India (red), Salon (green), and Zuunmod (purple), separated by imaging order vs. RER, FWHM, MTF50, and MTFA (upper left: average and CV). Scale of Y-axis was changed to CV value.

Figure 7 (right) shows the relationship between KOMPSAT-3A’s spatial quality and Edge targets (Baotou, India, Salon, Zuunmod), and Figure 7 (left) shows the Edge angle separated according to the Edge target. The black numbers at the top left of each plot are the overall average and CV values, while the colored numbers are the average and CV values for Baotou, India, Salon, and Zuunmod, respectively. Table 3 shows the average and CV values of the spatial quality according to the Edge target. India has only 29 results, so it is statistically excluded from the comparison. In Figure 7 and Table 3, we observe the following:

• The spatial quality values are different for each Edge target. Those of Baotou is the best, while those of Salon and Zuunmod are comparable.
• Each Edge target has different CV values, representing the precision. That of Baotou is relatively better than that of Salon and Zunnmod. Baotou is clean and well maintained.

The following two explanations can be given for the correlation between the spatial quality value and the Edge angle:

• There is a correlation between the Edge angle and the spatial quality values.
• There may be an element of the spatial quality measurement procedure that is missed in the application of the Edge angles.

Looking only at Figure 7 (left), it appears that spatial quality is related to the state of the Edge target. But the spatial quality measurement procedure multiplies COS(EdgeAng) with all pixel data and then the ESF fitting is performed [2,3,6,7], so the possible effects of other factors related to the Edge angle may also need to be considered. If the Edge angle value is large, the Across and the Along boundaries may become unclear, and other factors may be relevant. According to this result, the Edge angle affects both accuracy and precision of the spatial quality measurements of ESMP. Thus, it is necessary to modify and improve ESMP by studying and analyzing the factors that affect the spatial quality measurement via Edge angle. This can be used to draw conclusions through additional research and analysis.

4.2. Comparative Analysis of Spatial Quality Estimators’ Results

We plot and compare the relationships between the spatial quality estimators (RER, FWHM, MTF50, MTFA). In Figure 9, the scale of the plot is changed to a z-distribution (standard normal distribution) because RER, FHWM, MTF50, and MTFA satisfy the conditions of a normal distribution (-0.5 < skew < 0.5). The following formula was used. The red numbers in the top right in Figure 8 is the skew values

z-distribution = (Pixel data – Average) / StDev

Figure 9 and Table 4 demonstrate the relationships among RER vs. FWHM vs. MTF50 vs. MTFA. The first red value at the top right of Figure 9 is Pearson’s correlation coefficient, and the second red value is the Pearson’s value for 766 spatial quality values after excluding outliers using the IQR method. The blue number is CV value in the full case and the IQR. In Figure 9, we observe the following.

• RER vs. FWHM in spatial domain and MTF50 vs. MTFA in frequency domain are the best correlations according to Pearson’s correlation coefficient. After excluding outliers with the IQR, the Pearson’s correlation coefficient value for RER vs. FWHM reflects the best correlation, while FWHM vs. MTFA has the worst correlation, and the rest are similar.
• It is difficult to identify a strong proportional relationship between the spatial quality estimators other than RER vs. FWHM.

It is necessary to determine the reason that the Pearson’s correlation coefficient values are no greater than ‘0.73’. In particular, the reason for the low value of FWHM vs. MTFA (0.243) should be determined. This is because, if there are any independent characteristics between the spatial quality estimators, it is difficult to represent the spatial quality of a satellite using only one spatial quality estimator. Moreover, it can be assumed that each spatial quality estimator has different spatial quality-related factors involved. It is difficult to say which spatial quality estimator is better based on the results of IQR. It should be noted that only FWHM shows slightly different results from RER and MTFA. This is a subject for further research and analysis.

4.3. Summary

The comparison of CV values confirms that RER is the best spatial quality estimator in terms of precision. RER is more than four times better than MTF50 in terms of precision. In cases where it is difficult to measure RER, FWHM and MTFA are also useful estimators, with more than three times better precision than MTF50.

• KOMPSAT-3A’s spatial quality is very stable over time and does not show any adverse effect due to aging.
• There is no proportional relationship between the spatial quality and the Roll tilt angle, ΔDN, and SNR.
• There appears to be a correlation between the Edge angle and the spatial quality; this may be due to the difference in the Edge angle.
• RER vs. FWHM and MTF50 vs. MTFA are the good correlations, and it is difficult to identify any other strong proportional relationship between the spatial quality estimators.

5. Conclusions

By comparing and analyzing the spatial quality measurement results of the 840 edge images of KOMPSAT-3A, it is confirmed that the accuracy and precision of ESMP are sufficient. Of course, the methods used and the initial values of the constraints lead to some statistical values in ESMP, and there is a minor difference in the spatial quality values of the Edge angle and different Edge targets. Thus, further verification of ESMP’s internal algorithms is needed. Nonetheless, in conclusion, all of the presented findings are the result of objective application within the scope of explanation.

As a spatial quality estimator, it is confirmed that RER has the highest precision, and MTF50 has the lowest relative to its reputation. Meanwhile, FWHM and MTFA also exhibit sufficient precision. In addition, it is confirmed that the spatial quality estimators have no correlation with the Roll tilt angle, ΔDN, and SNR, but it is possible that the Edge angle may affect the spatial quality measurement values.

In this work, a comparative analysis was performed using the Edge target images acquired by KOMPSAT-3A with a system MTF50 of ‘> 8%’. This work described the internal structure of ESMP using Edge targets, the initial input conditions, and the factors that affect the results. The following additional research work will do:

• Spatial quality measurement with edges of Terrain Feature via statistics [39];
• Cause analysis of the differences in the spatial quality measurement with the Edge angle;
• Cause analysis results with a Pearson’s correlation coefficient value no greater than ‘0.73’ and a low FWHM and MTFA;
• The results and a comparison of the spatial quality measurements of other KARI satellite images, including those of KOMPSAT-3 (GSD, 0.7m; MTF50 > 8%).