Enrichment of the Information Extracted From Hyperspectral Reflectance Images for Noninvasive Phenotyping

Hyperspectral reflectance imaging is an emerging method for rapid non-invasive quantitative screening of plant traits. This method is essential for high-throughput phenotyping and hence for accelerated breeding of crop plants as well as for precision agriculture practices. However, extraction of sensible information from reflectance images is hindered by the complexity of plant optical properties, especially when they are measured in the field. We propose using reflectance indices (Plant Senescence Reflectance Index, PSRI; Anthocyanin Reflectance Index, ARI; and spectral deconvolution) previously developed for remote sensing of vegetation and point-based reflectometers to infer the spatially resolved information on plant development and biochemical composition using ripening apple fruit as the model. Specifically, the proposed approach enables capturing data on distribution of chlorophylls and primary carotenoids as well as secondary carotenoids (both linked with fruit ripening and leaf senescence during plant development) as well as the information on spatial distribution of anthocyanins (known as stress pigments) over the plant surface. We argue that the proposed approach would enrich the phenotype assessments made on the base of reflectance image analysis with valuable information on plant physiological condition, stress acclimation state, and the progression of the plant development.


Introduction
An important general task of plant performance monitoring is recording the progression of plant development (phenological phases); crucial indicators of the produce quality include leaf senescence and fruit ripeness. Monitoring these parameters is normally accomplished by visual observations and scoring [1,2], although a trend for automation via image analysis with machine learning algorithms becomes increasingly evident [3][4][5][6][7]. At the same time, developmental changes in pigment composition as well as those induced by environmental stresses and phytopathogen attacks manifest themselves as specific changes in plant reflection properties [8][9][10] linked with the changes in pigment content and composition. A valuable complement to visual assessments is provided by biochemical markers such as content and composition of the photosynthetic and photoprotective pigments displaying profound, directional changes during plant development, stress acclimation and damages [8,[11][12][13][14]. That makes spatially resolved information an important asset and proximal sensing of the information on spatial variation-a powerful tool in the array of methods of plant phenotyping and crop breeding [15][16][17][18][19][20][21]. The approaches based on automated proximal sensing are also getting increasing usage in precise agriculture [6,7,[15][16][17].
Indeed, wet biochemical analyses are laborious, expensive and in some cases even not feasible at all (e.g., when continuous monitoring of the same set of objects is required). Therefore, non-invasive approaches to retrieval of plant pigment composition on the scale of individual organs (leaves and fruits) as well as canopies based on the analysis of reflected light spectra have been suggested [18,19]. Unfortunately, despite spectral sensing of vegetation having a strong and established foundation, most of the earlier techniques were developed for "point-based" measurements [20]. In such measurements, the reflected signal is captured (and hence inevitably averaged) over a sampling area of plant surface encircled by the field of view (FOV) of a detector. This is a critical disadvantage as the heterogeneity of traits being monitored is inherent to many, if not all plant objects, and especially pronounced in fruit-bearing crops [21,22]. This limitation can be circumvented by applying imaging hyperspectrometers (IH) capable of capturing hyperspectral reflectance images (HRIs) of the whole plant or plant organ(s) with a sufficient spatial resolution [19,23,24].
Extracting information from HRIs that is not just pertinent to the plant phenotyping but also readily interpretable is of a special concern. Currently, machine learning-based methods of advanced image analysis as well as other mathematical tools are becoming widespread [25][26][27][28][29][30]. Although quite efficient in many cases, these methods normally do not take physiologically relevant information as an input. On the other hand, a significant effort has been invested into the development of vegetation indexes (VIs). VIs allow to retrieve quantitative information on vegetation conditions, including pigment (chlorophylls, Chl, carotenoids, Car, and anthocyanins, AnC) concentration from the point-based reflectance measurements as well as from HRIs recorded from satellite and UAV-based airborne platforms. That found extensive use in the remote sensing of vegetation [8,31]. The most widespread VIs are the VIs for Chl assessment such as NDVI, further complemented by the VIs for the assessment of plant development (leaf senescence and fruit ripening) [10,32]. However, while overall development of combining spectral and spatial features for HRI processing could be described as blooming, these VIs have not yet been tested, to the best of our knowledge, for the processing of HRIs of singular plant objects and organs.
In this work, we tested the VIs such as CI700, PSRI, and ARI initially developed for the retrieval of the foliar and fruit Chl content, Car/Chl ratio, and AnC [31,33] to extract quantitative information from HRIs about the pigment content. In addition, we have tested an earlier developed approach based on the reflectance spectra deconvolution [12] to extract physiologically relevant information on fruit ripeness from HRI. Special attention has been paid to the demonstration of the heterogeneity of plant objects regarding their pigment composition and its relevance as an additional source of information on plant physiological condition.

Plant material
2.1.1. Lettuce leaves Lettuce (Lactuca sativa L.) variety "Revolution" plants were grown from commercially available seeds (Bayer, Nunhems) first in a nursery on a turf substrate (for 15 d) and then in 1.2-L vessels in the same substrate (for 20 d) in vertical greenhouses (Panasonic, Japan). The fresh-cut plants were kept in a climatic chamber (Liebherr, Germany) at a constant temperature of 15 °C, relative humidity of 45%, and illuminated by white fluorescent tubes with PAR photon flux density of 50 µmol quanta m -2 s -1 as measured by LI-850 quantum meter (LiCOR, USA) at the leaf surface level. The measurements on the plants were conducted at 0, 1, 4, 7, and 11 days of the experiment. Three experiments, each made in triplicate, have been conducted with this object (n = 9).

Apple fruits and chemical treatments
Fruit of apple (Malus × domestica Borkh.) (n = 30) variety "Golden Delicious" (yellowgreen-colored) or "Ligol" were grown either at an experimental orchard of Michurin Federal Scientific Center (Michurinsk, Tambov region, Russia) or in the experimental orchard of the Botanical Garden of Lomonosov Moscow State University. Depending on the measurement goal, the fruit were imaged while they attached to the tree or after three months of storage at +4 °C in conventional atmosphere.
Apple fruit ripeness was assessed and expressed in form of Streif Index (SI) representing a quotient of the apple fruit firmness divided by the product of starch index times Total Soluble Solids concentration (TSS): = ( ℎ ) * Fruit firmness was determined with a digital penetrometer FHT-05 (FHT, China). TSS were determined with a manual refractometer BRIX COK-21 (Aqua-Lab, Russia).
To perform the starch-iodine test, apples were cut in half across equator, dipped into iodine solution (1%) and incubated for the development of color for 3-5 min. The coloration intensity was evaluated as starch index score [34].

Spectral reflectance measurements
2.2.1. Reflectance spectra measurement with a conventional spectrophotometer Diffuse reflectance spectra of apple fruits were recorded at a 400-800 nm range with an Agilent Cary Bio 300 (Agilent, USA) spectrophotometer equipped with an integrating sphere attachment (internal diameter 100 mm) against Spectralon plate as a 100% reflectivity standard.

Hyperspectral reflectance imaging
The hyperspectral reflectance data-containing images of the lettuce plants and apple fruits were captured with a frame-based imaging hyperspectrometer IQ (SPECIM, Finland). For each pixel of the hyperspectral image, a reflectance spectrum (spectral range 400-1000 nm; spectral resolution 1 nm; 512 × 512 pixels/frame) was recorded against a reflectivity standard made of Spectralon. Objects were illuminated by two 150 W cold daylight fluorescent lamps mounted in softboxes, supplemented with one 100 W incandescent lamp.

Reflectance indices calculation
The reflectance indices for hyperspectral datasets were calculated essentially as described previously [36,37]. Briefly, for each pixel of the hyperspectral images, two indices were computed. The first index, CI700, is a sensitive indicator of Chl content, [Chl]; it was defined as follows [11]: where R800 is the reflectance in a band in the near infra-red (NIR) region unaffected by the pigment absorption of light and R700 is the reflectance in the Red Edge region of the red Chl absorption maximum [31]. The second spectral index calculated was PSRI, an index tightly related with [Car]/[Chl] ratio in the samples and, ultimately, indicative of the rate and stage of senescence of plant objects [11,32,33]: where R800 is the reflectance in a band in the near infra-red (NIR) region unaffected by pigment absorption of light, R480 is the reflectance in a band affected by both [Car] and [Chl], and R678 is the reflectance in the band of the red Chl absorption maximum. For the processing of HRI, the point-based PSRI index was adapted as described in [27] and transformed to the Hyperspectral image PSRI (HPSRI) as follows:

Deconvolution of reflectance spectra
An alternative approach to the extraction of sensible information from the hyperspectral reflectance images was based on a previously developed technique for spectral reconstruction of the spectral reflectivity of plant objects such as apple fruits [12]. This technique involves calculation of the best (in the sense of L2 norm) approximation to the reflection spectrum as a linear combination of the known spectra of the following pigment pools: thylakoid-bound chlorophylls and carotenoids, RC(λ), cuticular/vacuolar phenolics, RP(λ), extrathylakoid carotenoids, RX(λ), and vacuolar anthocyanins, RA(λ), as well as a featureless scattering background. These known spectra are denoted below as fiducial spectra [12]. This model allows to simulate reflection spectra of plant objects widely differing in pigment content and composition: where R´(λ) is the reconstructed reflectance spectrum; RC(λ), RP(λ), RX(λ), and RA(λ)the fiducial spectra, and a-d are on-negative fitting coefficients. The fitting coefficients were adjusted by the least squares approach (implemented in Microsoft Excel spreadsheet using the Solver tool) to minimize , the sum of squares of the residuals in the spectral range 400-750 nm (if not stated otherwise): where is the sum of squared residuals; ( ) and ´( ) are the measured and reconstructed reflectance coefficients at the wavelength λi.
Relative contributions by the different pigment pools into the total reflection as quantified by the fitting coefficients are shown to be sensitive indicators of apple fruit ripening and accumulation of secondary carotenoids under stress conditions [12].

Results and discussion
We tested the feasibility of HRI processing with the approaches previously developed for processing of "point-based" reflectance spectra of plants using the models of vegetable and fruit crops comprised by fresh-cut lettuce leaves and apple fruits, both stored and attached to a tree. Below, we present the results of our attempts to extract quantitative information on different aspects of plant development and functioning from HRI data.

Leaf senescence
Higher plants contain in their chloroplasts two types of chlorophylls and several carotenoids as principal pigments involved in light harvesting and photochemical reactions. Under stress conditions and/or in a course of senescence an induction of the carotenoid and phenolic biosynthesis frequently takes place. The absorption bands of leaf pigments span over the visible and UV parts of the solar spectrum and there is a considerable overlap between the absorption bands of different pigments. Chlorophylls are the primary photosynthetic pigments capturing and transforming light energy during photosynthesis; carotenoids (Car) are important both for light harvesting and photoprotection [38].
In the experiments with lettuce, we followed changes in the reflective properties accompanying the transformation of the pigments, Chl and Car, during 11-day storage of fresh-cut leaves (Figures 1 and S1; for more details, see Methods and [37]). Towards this end, we have measured the distribution of foliar Chl over the leaf surface using the Chlorophyll Index (CI) based on the reflectance in the Red Edge band (700 nm), CI700, featuring a wide dynamic range and high linearity in a broad range of Chl contents (see e.g. [31]).
The CI700 index images manifested high heterogeneity of the studied leaves regarding their Chl content, which was evident throughout the observation period. This heterogeneity was apparent as a broad peak in the histogram reflecting the distribution of Chl contents over the leaf surface as captured by the HRI (Figure 1a). This heterogeneity was apparent on the spectral index images as well (see the false-colored images in Figure  1a): the regions adjacent to the leaf veins tended to have lower CI700 and the regions between the veins displayed higher CI700 values and hence higher Chl content.
The incubation of the fresh-cut plants for 11 days resulted in a gradual change of the shape of distribution: specifically, additional modes appeared on the histogram and the overall modal value shifted towards lower CI700 values (see the histograms in Figure 1a). Overall, the observed changes are consistent with degradation of Chl normally occurring during the senescence of leaves.
In addition to the monitoring of plant condition based solely on changes in Chl content, we have further investigated the approach to gauging plant senescence and fruit ripening based on the Plant Senescence Reflectance Index (Figure 1b; [32]). The applicability of PSRI for HRI processing, particularly to the HRIs of salad, has been established recently [37]. Here, we have analyzed the connection between heterogeneity and changes in distribution of PSRI values over the leaf surface (Figure 1b). In the beginning of the experiment (day 0), PSRI values calculated for the most of the leaf surface were uniformly low. Later, the distribution of the PSRI values changed to a bimodal one suggesting (i) co-existence of physiologically disparate leaf parts at different stages of senescence and (ii) gradual expansion of more senescent regions.
The analysis of the HRIs of fresh-cut lettuce leaves supports the possibility of assessing the stage of senescence in plants by registering changes in their optical properties, i.e. non-destructively. This approach also allows for the visualization of heterogeneity hidden within an individual plant, which hardly can be detected visually.

Fruit ripening
Due to its relatively low pigment content and high reflectance, apple fruit represents a simple natural system in which general plant developmental and/or stress-induced pigment dynamics could be followed non-destructively, quantitatively and in detail [11,13,[39][40][41][42][43][44]. Similarly to leaves, fruit optics is determined by the overall content of pigments, their local concentration, interactions and distribution within cell structures, as well as by the role played by scattering in defining internal optical properties. Numerous studies have shown remarkable and specific changes of the whole-fruit spectral reflection during ripening, adaptation to excessive solar radiation, photo-oxidative damage and development of physiological disorders [11,13,[39][40][41][42][43][44].
Ripening apple fruit detached from a tree (see Materials and methods) served as an another model of pigment transformation manifesting plant developmental changes [45]. Previously we have found that changes in the content of both chlorophylls and carotenoids rather than tracking either of these pigments singularly should be used to follow both on-and off-tree ripening in apple fruit. Then, in the course of off-tree ripening, chlorophyll degradation was found to be strongly tied with the accumulation of carotenoids. Acordingly, carotenoid-to-chlorophyll ratio being biologically significant regardless of the harvest date and pigment content of fruit [11,13,42]. For this experiment, a single batch of fruits with different maturity (as assessed by the destructive tests) have been selected. Apart from the conventional color (RGB) images, hyperspectral images of the same fruit differing in their maturity were taken. Then, vegetation index images have been calculated from the spectra contained in these HRIs.
The RGB images of the apples of different maturity were visually similar, so it was very difficult to determine the maturity stage from them with any degree of confidence (Figures 2a-c). Further analysis has been conducted on the HRIs taken from the same fruit. The first part of the analysis involved the calculation of PSRI index images (Figures 2df). The second part included pixel-wise deconvolution of the reflectance spectra comprising the HRIs of the apple fruits according to the previously developed algorithm (see the Methods and [12]). As a result, spectral contributions of the pigments typical of unripe fruit, e.g. chlorophyll and primary (photosynthetic) carotenoids, were estimated for each pixel constituting the HRIs of corresponding fruits (Figures 2g-i). The comparison of the resulting index images has shown that the fruits at more advanced stages of ripening were characterized by larger areas of the fruit surface with increased PSRI values (Figures 2d-f) and by dwindling areas characterized by a large content of the photosynthetic pigments.
To obtain a quantitative understanding of the changes observed on the calculated index images, we have analyzed histograms of the corresponding spectral parameters (Figures 3 and 4).  Figure 3a).
The values of the PSRI index version adapted to HRI analysis, HPSRI [22], computed on the basis of the corresponding HRI [10], correlated with the corresponding values of the Streif index; the destructive assay of apple fruit maturity served as the "ground truth" for this analysis (see Materials and Methods). The close relationship between parameters derived from the HRI with fruit maturity evaluated destructively supports the possibility of the non-invasive ripening assessment using the indices correlated with the Car/Chl ratio. Another important outcome of this analysis is finding a plausible approach to the visualization of the inherent heterogeneity of fruit tissues regarding its ripeness. This finding is compatible with our previous finding of ripening kinetics of sunlit and shaded sides of a single apple fruit being different [13]. An alternative approach to the analysis of the reflectance spectra extracted from the hyperspectral images involved their deconvolution using a set of fiducial spectra (see Materials and Methods and [12,46]). This approach is based on an approximation to the whole apple fruit reflection spectrum, which could be achieved with known spectral properties of basic pigment pools and a simple assumption on the shape of the featureless non-selective scattering spectrum. The shapes of the fiducial spectra are also assumed to remain essentially the same at different stages of fruit development. It allows to estimate relative spectral contribution by individual pigment pools such as Chl and associated with them Car.
As a result, sets of coefficients characterizing the contribution of each pigment group represented by the corresponding fiducial spectrum into the measured spectrum from each pixel of the hyperspectral image. The set corresponding to the spectral contribution of photosynthetic (primary) carotenoids declining in the progress of fruit ripening ( Figure  4a; see also [13,14,47]), has been selected for further analysis. The analysis of distribution of the coefficient's values revealed the characteristic shift of the maximum (modal value) of the distribution to lower values of the corresponding coefficient (Figure 4b). These changes correspond to the increase in the relative proportion of the fruit with significant accumulation of the esterified forms of secondary carotenoids typical of apple ripening at the expense of primary (photosynthetic) Car and Chl [13,46,47]. Some lines of evidence suggest that these changes proceed along with the induction of carotenogenesis in apples that involves the buildup of xanthophylls (mainly violaxanthin and its fatty acid esters) occurring in chloroplast/chromoplast plastoglobuli [47]. Collectively, these results as well as the experiments with long-term fruit storage suggest that relative contribution of Chl and Car into whole fruit spectral reflection could serve as a sensitive indicator of apple ripening.
(a) (b) Figure 4. Non-invasive assessment of apple fruit ripening with hyperspectral reflectance deconvolution (for further details, see Methods and [12]). (a) The changes in the distribution of the spectral contribution of chlorophylls and associated with them primary carotenoids to overall reflectance of the representative fruits from the studied batches. Note the shift of the modal value towards lower Chl and primary Car contents; (b) relationship of the contribution the photosynthetic pigments (inversely related with fruit ripeness) estimated through hyperspectral reflectance deconvolution with the destructive assessment of ripeness (the Streif index). The numbers of apple fruit batches featuring different actual ripeness are indicated near the data points.

Anthocyanic coloration
An important phenotypic manifestation in plants is red coloration of their leaves and/or fruits determined by the presence of anthocyanin pigments [48][49][50][51]. It frequently manifests acclimatory responses to diverse stresses and hence increased resilience of plants, including crop plants, to these stresses [49,52]. The formation of anthocyanins in apples (mainly cyaninidin-3-galactoside) is genetically determined but may be influenced by various factors, including high irradiance. The red coloration of fruit is also of practical significance since red coloration increases fruit acceptance by customers [53].
The approach to quantitative monitoring of anthocyanin content in plant tissues suggested in this work is based on the previously developed Anthocyanin Reflectance Index. ARI [56]. We tested it using red apple fruit as the model and the results of a field experiment aimed at the modulation of apple red coloration with the PHJ treatment ( Figure 5; see also Methods).
The HRI with the images of fruit have been processed by calculating ARI index for each pixel comprising images of the apple fruits. The resulting datasets were integrated and recalculated using a model previously developed for non-invasive estimation of anthocyanin content in apple fruit peel [11]. As a result, we obtained false-colored images highlighting the distribution of AnC over fruit surface (Figure 5b) as well as integral estimations of the average AnC content expressed per unit fruit surface (Figure 5c). These results allow one to visually estimate the abundance and heterogeneity of fruits regarding their AnC content. More systematic, quantitative estimations that represent AnC content in the whole plant organs or individual plants can be obtained this way.

Conclusions
This study is an extension of our previous effort aimed at deepening of current understanding of in vivo optical properties of principal pigment pools responsible for light absorption. Towards this end, we have leveraged the previously developed vegetation indices for inferring quantitative information on pigment content and distribution over plant surface from their hyperspectral reflectance images [10,22]. The analysis of the cases presented above supports the applicability of the approaches developed for quantitative interpretation of conventional "point-based" reflectance measurements for extracting of spatially resolved information on pigment composition of plants. An added benefit of the proposed approach to the refining of HRI with the VIs developed for remote sensing applications is comprised by sensitivity and selectivity ensuring the enrichment of the resulting images with information on plant trait in question.
Processing of the HRI of the model plant objects employed in this work with reflectance indexes CI700, PSRI, hPSRI, and ARI allowed to capture the complex patterns of pigment transformation during leaf senescence and fruit ripening. Images rendered from the HRI with the reflectance indexes mentioned above revealed the inherent heterogeneity of physiological and biochemical traits such as pigment content and composition. This kind of heterogeneity normally cannot be revealed with conventional point-based spectral measurements and "wet" analytical methods.
However, the application of the VIs to HRI analysis should be done with caution. Although many VIs employ an internal normalization to cope with diverse interferences, a routine quality-control of the reflectance spectra should be used to exclude the spectra distorted from under-or overexposure of plant object and/or the reflectivity standard. Another important issue is comprised by the choice of suitable post-processing of the HRI processing results. Arguably, more advanced analysis of the VI values calculated on the basis of HRI is necessary beyond simple averaging. Thus, confident interpretation of plant optical properties, also in the context of HRI analysis requires knowledge of spectral features of pigment absorption in vivo. Here, we demonstrated that the basic models developed with such knowledge in mind can be applied to HRI of leaves and fruits with a wide variation in pigment content and composition, at different stages of their development.
To conclude, the HRI processing with the VI developed for remote sensing of vegetation can potentially yield a plethora of information useful for automated noninvasive phenotyping of crop plants. This information would be a welcome complement to currently widespread methods of morphological analysis of plant images with machine learning algorithms enriching their results with quantitative information on plant physiological condition and biochemical composition. This approach would find a broad application also in precision agriculture e.g. in gauging of fruit quality and ripening.
Supplementary Materials: The following are available online at www.mdpi.com/xxx/s1, Figure S1

Data Availability Statement:
The data presented in this study are available from the corresponding author on reasonable request.