A Multispectral Camera Suite for the Observation of Earth’s Outgoing Radiative Energy

1. Shortwave camera suite

In the [11] reference design, the ShortWave (SW) camera - also called solar camera - consists of an RGB CMOS detector array with a wide field of view lens allowing to view the Earth from limb to limb. The field of view is ${140}^{\circ }$. The Wide Field Of View (WFOV) lens design is illustrated in Figure 4. The nadir resolution is 2.2 km. The f-number is 3. The used RGB CMOS spectral responses are illustrated in Figure 5. On a stand alone basis the RSR can be estimated from an RGB CMOS imager through spectral regression with a noise level of 3 %.

The only on-board calibration source for the SW camera suite will be a shutter, allowing to verify the dark current, and also allowing to shield the camera from direct solar illumination during solar pointing. The camera spectral gains and dark current will be determined pre-flight during ground calibration. The stability of the RGB spectral responses and the gains will be monitored, and if needed adjusted in-flight through vicarious calibration monitoring stable Earth targets similar to [13,14,15].

The spectral regression noise level by which the RSR broadbance radiance can be estimated, can be improved by increasing the number of narrowband spectral channels used as input for the regression. In [16] a commercially available CMOS sensor with 4x4 filter bands in the VIS (470-620nm) is described. Also a commercially available CMOS sensor with 5x5 filter bands in the VIS-NIR (650-975nm) is described.

The newly proposed ECO Shortwave Camera Suite (SCS) consists of 3 separate ShortWave (SW) cameras. Camera 2 and 3 can be based on the same CMOSIS CMV2000 sensor array. Since the size of the detector is comparable to the detector used in [11], the reference wide field of view lens from Figure 4 can be re-used. Camera 1 can be realized using the higher resolution CMOSIS CMV12000 sensor array. Since the size of the CMV12000 sensor is about 3 times larger than the detector size used in [11], a new lense design is appropriate, and will be presented in Section 2.The 3 shortwave cameras, have different spectral responses, by using different filters directly integrated on the CMOS chip, with spatial patterns illustrated in Figure 6.

The 3 ECO SW cameras are the following:

• ECO SW Camera 1 will be the RGB high-resolution SW camera, formed by a CMOS sensor, with spectral response between 400 and 1000 nm, with standard ’RGGB’ Bayer pattern. The active area of the CMOS sensor will be 3000*3000 pixels, yielding a nadir resolution for a single pixel of 0.57 km, and a nadir resolution for the 2x2 RGGB pixels of 1.1 km. The pixel pattern of SW camera 1 is illustrated in the left part of Figure 6.
• ECO SW Camera 2 will be the VIS SW camera from [16]. The nadir resolution of the 4x4 VIS filter pattern - illustrated in the middle part of Figure 6 - is 6.8 km.
• ECO SW Camera 3 will be the VIS-NIR SW camera from [16], using the same underlying CMOS sensor as Camera 2. The nadir resolution of the 5x5 VIS-NIR filter pattern - illustrated in the right part of Figure 6 - is 8.5 km.

The detectors from [16], considered for camera 2 and 3, are also considered for the CubeMAP [17] and Hyperscout-H [18] space missions.

2. Optical design high-resolution SW camera

The SW camera uses the CMOSIS CMV12000 4K CMOS detector with pixel size 5.5 $\mu m$. For the envisioned FOV of ${140}^{\circ }$, we use a circular area of 16.5 mm diameter, corresponding to 3000 pixels. For a WFOV lens with field angle ${140}^{\circ }$ observing the Earth from an altitude of 700 km, the size of the nadir pixel is 0.57 km. Considering that the sensor is used with a Bayer color filter, the RMS spot size we require should be smaller than 2 pixels - corresponding to 11 $\mu m$. The barrel distortion is considered to be corrected in post processing.

2.1. The Design parameters

The lens train - illustrated in Figure 7 - consist of 6 lenses, two of which form an achromatic doublet. The doublet along with the third lens works as a chromatic aberration correction system along with focusing the image on the image plane. The main lens properties are summarised in Table 1. The Lens materials were selected from the SCHOTT ® catalog. The LAK14 is a crown glass with relatively high refractive index and SF6 is a flint glass with high refractive index as well.

There are 3 aspherical surfaces on the system. the first surface, the last surface and the final surface of the achromatic doublet. This is done to ensure increased light collection and to reduce spherical aberrations. The initial reference design for this lens train was the one shown in Figure 4 [11].

The system was designed and optimised using the Zemax OpticStudio® software. Only a few parameters were constrained, namely, the ’Effective focal length’, at 15.6 mm, the lens thickness minimum being 2 mm. The initial merit function for spot size was done through the ’Quick focus’ function provided by Zemax. Once a substantial percentage of criteria is fulfilled during optimization and satisfactory RMS values along with a good relative illumination were achieved, further optimization was done using the ’Hammer optimization’ function provided by Zemax.

2.2. Optical performance

2.2.1. RMS error

The RMS error - see Figure 8 - shows the root mean square deviation of a bunch of rays of light from its intended spot of incidence. This value is supposed to be below $11\mu m$ approximately. This is widely true for the said lens train and only reaches $12\mu m$ for the 400 nm wavelength at the extreme ${70}^{\circ }$ incidence angle. This can be considered sufficient for our purposes.

2.2.2. Seidel aberrations

The Seidel aberrations - see Figure 9 - apart from distortion have been optimized and kept low. The distortion/ barrel distortion is to be corrected on ground as post-processing.

2.2.3. Point Spread Function (PSF)

The optical PSF is the characterization of how a point of light is transformed when passing through an optical system. The optical PSF is related to the RMS spot size as shown in Figure 8, which mostly lies below the desired 11 $\mu$m. The average optical PSF of all wavelengths is given in Figure 10.

To evaluate the joint effect of the optical PSF, and the pixel shape, we calculate the convolution op the optical PSF with the detector pixel shape. We use a layer of fixed size containing several pixels of the same type, either Red/Blue active pixels or Green-Green active pixels. The two types of pixels exist due to the presence of the Bayer filter. The individual pixels have sizes of $5.5\mu m$ but to emulate real pixels, we use an active area of 80% of the nominal pixel size as shown in Figure 11. We assume the variation of the optical PSF is negligeable within the layer of pixels.

The convolution of these pixel layers with the optical PSF for the $0^{\circ }$ incidence angle, is shown in Figure 12. It can be seen that, after the convolution, individual pixels are still recognisable.

From the RMS Figure 8, we see that the 400 nm wavelength at ${70}^{\circ }$ incidence angle and 500 nm at ${38}^{\circ }$ incidence angles have high RMS spot radius. We therefore examine in detail the PSFs and convolutions at these particular wavelengths and angles.

The optical PSF for light with a wavelength of 400 nm incident at ${70}^{\circ }$, is shown in the left of Figure 10. The convolution of this optical PSF with the blue-type active pixel layer is shown in the right of Figure 10.

Figure 13. Left : The PSF of 400 nm wavelength for ${70}^{\circ }$ incidence angle. Image size 53.236 by 53.236 $\mu m$. Right: The convolution of active pixel layer of blue-type with the PSF of ${70}^{\circ }$ incident light of 400 nm wavelength, where individual pixels are overlaid over the convolution.

Figure 13. Left : The PSF of 400 nm wavelength for ${70}^{\circ }$ incidence angle. Image size 53.236 by 53.236 $\mu m$. Right: The convolution of active pixel layer of blue-type with the PSF of ${70}^{\circ }$ incident light of 400 nm wavelength, where individual pixels are overlaid over the convolution.

The optical PSF for light with a wavelength of 500 nm incident at ${38}^{\circ }$, is shown in the left of Figure 14. The convolution of this optical PSF with the green-type active pixel layer is given in the right of Figure 10.

We see that the light spreads towards the top and bottom of the pixels in the right part of Figure 13 and Figure 14. However, these spreads are acceptable for the function of the detector.

2.2.4. Relative intensity

The relative intensity of light as a function of angle of incidence is shown in Figure 15.

2.2.5. Chromatic aberration

The chromatic aberration of the system is that of a compensated system as shown in Figure 16.

2.3. Simulated Performance for Earth observation

2.3.1. SNR High-resolution 4K-RGB camera

For camera 1, with a nadir resolution of 570 m, and a nominal satellite altitude of 700 km, Lambertian Earth leaving flux is attenuated towards the entrance of the camera by a factor ${(0.57/700)}^2/\pi =2.11\times {10}^{-7}$ . Within the optical system, with a $0^{\circ }$ radiometric aperture diameter of 5.38 mm, and a detector pixel size of 5.5 micron, the entrance flux is amplified towards the detector by a factor $\pi /4\times {(5.38/5.5\times {10}^{-3})}^2=7.5\times {10}^5$. Assuming no light loss, the ratio between the detector flux and the Earth flux is then $2.11\times {10}^{-7}\times 36.7\times {10}^4=0.16$. Assuming a 50 % light loss, the ratio becomes $0.16\times 0.5=0.08$.

The detector for camera 1, is the CMOSIS CMV12000 CMOS detector array, with $4096\times 3072$ pixels, and a full well capacity of 1485 counts, corresponding to approximately 10.5 bit per pixel. For a $3000\times 3000$ pixel subarray, it can be read out at a maximum frame rate of 400 fps, corresponding to a minimum readout time for a single frame of 2.5 ms. We calculate the maximum integration time before saturation occurs for the theoretical case of 100 % diffuse reflection for an incident solar irradiance of 1500 W/m². The wavelength dependent saturation time for Red, Green and Blue wavelengths with peaks at 634 nm, 547 nm, 453 nm respectively, are 16.7 $\mu$s, 18.9 $\mu$s , 26 $\mu$s respectively. An average Quantum efficiency of 0.45 is assumed.

Before motion blurring occurs, a further time averaging is allowed within a time period of 0.57 km / (7 km/s)= 81.4 ms, so over $81.4ms/2.5ms\approx 32$ frames. The SNR as a function of wavelength for 32 integrations of 15 $\mu$s each for red, green and blue wavelengths are $47\times {10}^3$, $41\times {10}^3$ and $30\times {10}^3$ respectively. If we follow the procedure of [11] to estimate the RSR from the RGB camera measurements, the quantisation errors corresponding to these SNR values cause an error on the RSR of 0.04 W/m². This 0.04 W/m² quantisation error, is small compared to the [11] spectral regression error of 3 %, which for the reference scene of 1500 W/m² corresponds to 3 % × 1500 W/m² = 45 W/m².

2.3.2. SNR multispectral cameras

For cameras 2 and 3, with a nadir resolution of 1.71 km, and a nominal satellite altitude of 700 km, Lambertian Earth leaving flux is attenuated towards the entrance of the camera by a factor ${(1.71/700)}^2/\pi =1.9\times {10}^{-6}$. Within the optical system, with a $0^{\circ }$ radiometric aperture diameter of 1.14 mm, and a detector pixel size of 5.5 micron, the entrance flux is amplified towards the detector by a factor $\pi /4\times {(1.14/5.5\times {10}^{-3})}^2=3.37\times {10}^4$. The ratio between the detector flux and the Earth flux is then $1.9\times {10}^{-6}\times 3.37\times {10}^4=0.064$ without light loss, and $0.064\times 0.5=0.032$ with 50 % light loss.

The detector for camera 2 and 3 is the CMOSIS CMV2000 CMOS detector array, with 1024×2048 pixels, and a full well capacity of 1012.5 counts, corresponding to $\approx 10$ bit per pixel. For a 1000×1000 pixel subarray, it can be read out at a maximum frame rate of 680 fps, corresponding to a minimum readout time for a single frame of 1.47 ms. Before motion blurring occurs, a further time averaging is allowed within a time period of 1.71 km / 7 km/s = 243 ms, so over $243ms/1.47ms\approx$ 166 frames. The Signal to Noise Ratio (SNR) as a function of wavelength for 166 readouts of 400 $\mu$s each is illustrated in Figure 17. For the least sensitive wavelength at 950 nm, the SNR for a single readout is 771, the SNR for 166 readouts is $128\times {10}^3$.

3. Longwave camera suite

3.1. Multispectral thermal cameras

A reference design of the thermal imager is described in [12]. The thermal imager consists of a microbolometer detector array with a WFOV lens allowing to view the Earth from limb to limb. The WFOV lens design is illustrated in Figure 18. The nadir sampling distance is 2.2 km, and the nadir effective resolution is 4.4 km. Using a single wavelength band of 8-14 micron, on a stand alone basis the OLR can be estimated from the thermal imager with an accuracy of 5 %.

The LongWave (LW) cameras will be equipped with a shutter with known temperature and known high emissivity. This shutter will act as a flat blackbody. Combined with the deep space view a two-point in-flight calibration for the longwave cameras will be implemented. The shutter view will be used for the characterization of the thermal offset as a function of the lens temperature. The deep space view will be used for an in-flight verification of the gain, that will also be measured on the ground before flight.

For improving the spectral regression OLR noise level, the number of ECO LW channels can be increased. This will be done by using multiple cameras, with filters inserted in front of the microbolometer detector array. In [19], the TIRI/HERA space instrument is described using the same detector as in [12], and a filter wheel with 6 filters each having a bandwidth of approximately 1 $\mu$m. For ECO we will adopt similar filters, listed in Table 2.

The 6 ECO filter spectral responses are also similar to filter responses used for the SEVIRI/MSG [20] and FCI/MTG [21] instruments.

The ECO LW Camera Suite (LCS) will be formed by 6 separate LW cameras, each consisting of the same microbolometer array and WFOV lens, and each equiped with a different filter realising the spectral responses from Table 2. For the microbolometer array, the Lynred 1024x768 array with 17 $\mu$m pixel pitch [22] can be used.

The LW spectral responses are illustrated in Figure 19.

3.2. Noise Equivalent Differential Temperature (NEDT)

For the thermal cameras, with a nadir sampling distance of 2 km, and a nominal satellite altitude of 700 km, Lambertian Earth leaving flux is attenuated towards the entrance of the camera by a factor (2/700)²=8.2$\times {10}^{-6}$. Within the optical system, with a $0^{\circ }$ radiometric aperture diameter of 6.61 mm, and a detector pixel size of 17 micron, the entrance flux is amplified towards the detector by a factor $\pi$/4*(6.61/17$\times {10}^{-3}$)² = 119$\times {10}^3$. Asssuming no light loss, the ratio between the detector flux and the Earth flux is then 8.2$\times {10}^{-6}$ * 119$\times {10}^3$ = 0.97. Assuming 50 % light loss, the ratio becomes 0.97 * 0.5 = 0.49.

The Lynred 1024x768 microbolometer detector array has an NEDT of 50 mK for a f/1 optics at 300 K and at 30 Hz. For the above calculated optical magnification of 0.49, the NEDT will be increased to 50 mK / $\sqrt{0.49}$ = 72 mK. Before motion blurring occurs, the microbolometer signal can be integrated during 4.4 km / 7 km/s = 0.63 s. After 0.63 s averaging, the NEDT will be reduced to 72 mK / $\sqrt{0.63s*30Hz}$ = 17 mK for a hypothetical microbolometer without spectral filter. From the ECO spectral filters - illustrated in Figure 19 and quantised by equations 3 - the fractions measured for an ideal black body at 300 K are calculated, and the respective resulting NEDTs are calculated and listed in Table 3.

3.3. Longwave spectral regression

In [12], the OLR was estimated from a single narrowband irradiance $I_{nb}$ measurement, by the following steps:

• conversion of the narrowband irradiance $I_{nb}$ to a narrowband brightness temperature $T_{nb}$
• conversion of the narrowband brightness temperature $T_{nb}$ to a broadband brightness temperature $T_{bb}$
• conversion of the broadband brightness temperature $T_{bb}$ to the OLR

Here we will generalise this approach using 6 narrowband irradiances $I_{nb,i}$, for i=1,...,6. The 6 narrowband irradiances are obtained from the 6 thermal cameras with spectral response illustrated in Figure 19. Using the sigmoid function $s\left(x\right)$, with

$$s\left(x\right)=\frac{1}{1+e^{-x}}$$

(1)

we define the synthetic microbolometer spectral response $f_{mb}\left(\lambda \right)$

$$f_{mb}\left(\lambda \right)=s((\lambda -7.5\mu m)*4)*s(\lambda -13\mu m)$$

(2)

with $\lambda$ the wavelength, and we define the synthetic narrowband spectral responses $f_{nb,i}\left(\lambda \right)$, for i=1,...,6 as

$$\begin{array}{c}\hfill c{f}_{nb,1}\left(\lambda \right)=f_{mb}\left(\lambda \right)*s((\lambda -8\mu m)*10)*s(-(\lambda -9\mu m)*10)\\ \hfill f_{nb,2}\left(\lambda \right)=f_{mb}\left(\lambda \right)*s((\lambda -9\mu m)*10)*s(-(\lambda -10\mu m)*10)\\ \hfill f_{nb,3}\left(\lambda \right)=f_{mb}\left(\lambda \right)*s((\lambda -10\mu m)*10)*s(-(\lambda -11\mu m)*10)\\ \hfill f_{nb,4}\left(\lambda \right)=f_{mb}\left(\lambda \right)*s((\lambda -11\mu m)*10)*s(-(\lambda -12\mu m)*10)\\ \hfill f_{nb,5}\left(\lambda \right)=f_{mb}\left(\lambda \right)*s((\lambda -12\mu m)*10)*s(-(\lambda -13\mu m)*10)\\ \hfill f_{nb,6}\left(\lambda \right)=f_{mb}\left(\lambda \right)*s((\lambda -13\mu m)*10)*s(-(\lambda -14\mu m)*10)\end{array}$$

(3)

For a given general spectral irradiance distribution $I\left(\lambda \right)$, the narrowband irradiance $I_{nb,i}$ is given by

$$I_{nb,i}=\int I\left(\lambda \right)f_{nb,i}\left(\lambda \right)d\lambda$$

(4)

and the broadband irradiance, also known as the OLR, is given by

$$OLR=\int I\left(\lambda \right)d\lambda$$

(5)

For the specific case of blackbody radiation, the irradiance distribution is given by the Planck curve and depends only on the temperature T of the blackbody, $I\left(\lambda \right)=I_{BB}(T,\lambda )$. We define the brightness temperature $T_{nb,i}$ from an exponential fit to this blackbody radiation:

$$\int I_{BB}(T,\lambda )f_{nb,1}\left(\lambda \right)d\lambda \approx {(a_i*T_{nb,i})}^{b_i}$$

(6)

and we define the broadband brightness temperature $Tbb$ from Planck’s law:

$$OLR=\sigma *T_{bb}^4$$

(7)

For the general narrowband radiance $I_{nb,i}$ - given by equation 4, the narrowband brightness temperature $T_{nb,i}$ is obtained by inverting the exponential fit from equation 6:

$$T_{nb,i}=\frac{{\left(I_{nb,i}\right)}^{\frac{1}{b_i}}}{a_i}$$

(8)

and the broadband brightness temperature $T_b$ is obtained by inverting Planck’s law:

$$T_{bb}={\left(\frac{OLR}{\sigma }\right)}^{\frac{1}{4}}$$

(9)

Similar to [12], we use the libradtran radiative transfer simulation tool, to simulate the narrowband irradiances and OLR for 15 reference scenes, listed in column 1 of Table 4.

For the ensemble of these 15 reference scenes, we regress $T_{bb}$ as a linear function of $T_{nb,i}$:

$$T_{bb}\approx c_0+\sum _{i=1}^6{c}_i*T_{nb,i}$$

(10)

next we evaluate the OLR from $T_{bb}$ using Planck’s law - using equation 9. The resulting OLR regression error is given as column 3 of Table 4. It lies within the range +/- 0.61 %.

The general recipee for estimating the OLR from measured narrowband irradiances $I_{nb,i}$, for i=1,...,6 is:

• convert $I_{nb,i}$ to $T_{nb,i}$ using equation 8
• estimate $T_{bb}$ from $I_{nb,i}$, using equation 10
• convert $T_{bb}$ to the OLR, using equation 7

4. Discussion

The monitoring of the Earth Energy imbalance is one of the most critical aspects of mastering our planet’s climate change [4,5], and is the main mission objective of the Earth Climate Observatory (ECO) space mission concept. The ECO payload concept consists of four WFOV radiometers as main instruments, targeting an accurate differential measurement of the EEI as the difference of the ISR and the TOR, at a low spatial resolution of approximately 6000 km for a single measurement of the TOR. As auxiliary ECO instruments, cameras are proposed, allowing to spectrally separate the TOR into RSR and OLR, and to resolve the RSR and the OLR at a spatial resolution of at least 5 km at nadir, allowing to distinguish clear-sky from cloudy scenes. In [11] a 1500x1500 RGB CMOS camera was proposed, providing a nadir resolution of approximately 1 km for a single pixel, and allowing to estimate the RSR with a spectral regression error of 3 % at the 2x2 pixel level. In [12] a 750x750 microbolometer camera was proposed, providing a nadir resolution of approximately 2 km, and allowing to estimate the OLR with a spectral regression error of 5 % at the pixel level. In this paper we are introducing some improvements in the ECO camera concept, building further on the [11,12] baseline designs.

For the SW camera, in [11] a single RGB camera - with pixel size of 3.2 $\mu$m, and image diameter of 1536 pixels, corresponding to 1536*3.2 $\mu$m = 4.9 mm - was used. The accuracy by which the high resolution RSR radiance can be estimated from this camera, is limited to 3 % by the information contained in the three spectral channels, the Red, Green and Blue channels with typical spectral responses shown in Figure 5.

Here we have proposed a dramatic improvement in the number of spectral channels by adopting the IMEC multispectral cameras described in [16], which we adopt as our SW cameras 2 and 3. Camera 2 and 3 provide a total of 31 spectral channels, in the form of a 4x4 VIS and 5x5 VIS-NIR MultiSpectral Filter Array (MSFA) respectively, as illustrated in Figure 6. The IMEC multispectral detectors are based on the CMOSIS CMV2000 detector, with a pixel size of 5.5 $\mu$m, and image diameter of 1000 pixels, corresponding to 1000*5.5 $\mu$m = 5.5 mm. For the SW camera lense we can use the optical design of [11], with a scaling of 5.5/4.9 = 1.1.

The use MSFA based cameras requires a demosaicing algorithm, for which Deep Learning (DL) based methods are currently an active area of research [23,24]. The quality of the demosaicing will improve if we can provide additional high resolution input information. For this purpose we propose to include SW camera 1, a high resolution RGB camera, based on the CMOSIS CMV12000 detector, with a pixel size of 5.5 $\mu$m, and image diameter of 3000 pixels, corresponding to 3000*5.5 $\mu$m = 16.5 mm. For this camera, we need a new optical design, which is presented in Section 2 and Figure 7.

For the thermal or LongWave (LW) camera, in [12] a single ’broadband’ microbolometer camera with spectral response between 8 and 14 $\mu$m - see the purple curve in Figure 19 - is proposed. The corresponding OLR spectral regression error is 5 %. Here we propose to use 6 different copies of this basic camera - using the same microbolometer detector array and WFOV LW lense, see Figure 18 - with 6 different filters in front of it. Inspired by the TIRI instrument [19] to be launched as part of the Hera mission [18], we choose 6 adjacent filters with a width of 1 $\mu$m, with spectral responses illustrated in Figure 19.

Extending the methodology of [12], with the newly proposed 6 channel LW camera concept, in Section 3.3 we evaluate the OLR radiance spectral regression error to be 0.6 %. Thus, comparing our newly proposed LW camera design to the one of [12], by going from 1 to 6 channels, we reduce the OLR spectral regression error from 5 % to 0.6 %.

A detailed analysis of the optical performance of the newly designed high-resolution SW camera, as well as an evaluation of the SNR of all SW cameras and the NEDT of all LW cameras is given in Section 2.2. The optical performance is well above what is required for the ECO mission scientific objectives.

5. Conclusions

We propose in this work, an update to the work previously done for the ECO space mission to measure what can be considered as the most essential of all climate variables: the Earth energy imbalance. The proposed payload consists of a combination of highly accurate, but low resolution radiometers measuring the incoming solar radiation and the total outgoing terrestrial radiation in a differential way, enhanced with lightweight uncooled camera systems aiming to separate the total outgoing terrestrial radiation spectrally in reflected solar - or shortwave, and emitted thermal - or longwave radiation, and to increase the spatial resolution to 5 km at nadir or better. Compared to our earlier proposal for the camera system, which consisted of a single SW camera and a single LW camera, we now propose a suite of 3 SW cameras and 6 LW cameras, with strongly improved spectral information content, and with an optical performance that is largely sufficient for the intended application.