Time-Course Morphological Analysis Using Normal and Local Curvature Information from 3D Modelling Data for Leaf Lettuce

1. Introduction

Physical contact that occurs during the manual measurement of plants can suppress plant growth [1]. Therefore, it is desirable to use noncontact and nondestructive methods to accurately measure plant growth. Noncontact and nondestructive image-based analyses of plant morphology are continually advancing. Data obtained from image analysis technology such as quantitative characteristic loci (QTL) analysis and genome-wide association studies (GWAS) has been used in molecular genetics [2,3] and is beginning to be used for growth monitoring and growth prediction in crop cultivation [4,5,6,7,8]. Furthermore, image-based 3D measurement methods are also being developed and researched [9,10,11,12]. Notably, 3D measurement is high throughput and can capture a large amount of information with a single measurement [13,14]. Traditionally, plant characteristics have been measured and evaluated manually by ruler or by visual assessment. The former is a labour-intensive and time-consuming process, especially when multiple measurements and plants are involved. The latter method also presents issues such as inter-observer variability and a requirement for skilled evaluators. Our research aims to address these challenges by noncontact 3D analysis of a wide variety of plants to reduce the labour required for measurement and to quantify subjective human evaluation. Therefore, we conducted a review of noncontact 3D measurement methods and developed a highly accurate all-around 3D modelling system to record plant growth [11]. This device uses images to perform 3D modelling and can be applied to various sizes of plants depending on the camera configuration. In this study, we developed a compact device that can be used even in a plant growth chamber that can measure plants ranging in size from a few millimetres to approximately 40 cm. To verify the possibility of measuring plants through 3D modelling, we first chose lettuce, which is easy to obtain and relatively easy to cultivate. Previous studies on the 3D analysis of lettuce have included 3D reconstruction using Kinect [15], analysis of light use efficiency using ray tracing with a handheld laser [16], growth prediction models using 2D projections obtained from 3D models [17], and estimation of harvest yield using volume estimation with 3D models [18]. However, as of now, a 3D measurement method for morphological features such as blistering or undulation (wavy morphology) of leaves has not been established.

There are several types of lettuce, including leaf and head types. Leaf lettuce was selected for analysis in this study. Leaf lettuce is one of the representative vegetables cultivated in plant nurseries. There are many varieties of leaf lettuce, and they have various characteristics, such as large or small wavy leaves or flat leaves, depending on the variety. For lettuce leaves, genes involved in serration have been reported, and these are also associated with susceptibility to tip burn [19]. Furthermore, photomorphogenesis, that is, the phenomenon of plant morphology being controlled by light conditions, has been well studied [20]. In leaf lettuce, it has been reported that light quality and quantity affect leaf shape and colouration [21]. It is important to accurately identify changes in lettuce growth to control for optimal growing conditions.

In Japan, there are 62 evaluation characteristics for lettuce included in the Lettuce Variety Inspection Standards (LVIS; September 2022, Ministry of Agriculture, Forestry and Fisheries) [22] and 55 characteristics in the test guidelines for the International Union for the Protection of New Varieties of Plants (UPOV) [LACTU_SAT Lactuca sativa L TG/13/11 Rev. 2 : 2017-04-05] [23]. The Japanese National Test Guidelines include some characteristics that overlap with those of the UPOV. Of the 62 LVIS characteristics, 42 are related to appearance, excluding characteristics related to disease resistance and pest resistance. In this study, we mainly examined characteristics adopted in both the Japanese National Test Guidelines and the UPOV Test Guidelines. notably, we focused on morphological characteristic evaluation using 3D analysis and excluded seven characteristics related to colour and five characteristics related to stems. In addition, this study focused on the characteristics that appeared from seeding to harvest in leaf lettuce. As a result, we focused on the 21 characteristics shown in Table 1 that could be analysed with the 3D model.

The characteristics described as “Measure” in the “Survey method” column of Table 1 can be measured with the reconstructed 3D model. Specifically, these are four characteristics encompass the length and volume of each part (Lettuce Variety Inspection Standards: 2, 4, 19, 20; UPOV characteristic 2). The measurement accuracy of our system was evaluated for characteristics related to these items, such as plant size, vertical projected area, and volume measurement.

The characteristics described as "Visual assessment" in the "Survey method" column of Table 1 are qualitative and are evaluated based on subjective human senses. However, there is a possibility that such characteristics can be quantified. We estimated that the characteristics denoted with ** marks in the table could be quantified with 3D models. The characteristics denoted with * marks were by combining AI and recognition technologies.

In addition, seven characteristics related to visual assessment (Lettuce Variety Inspection Standards: 22-28; UPOV: 18-24) were quantified with our newly proposed local curvature (hereinafter referred to as LC) method using normal vectors. Additionally, we examined two characteristics related to blisters on leaves. Then, we confirmed the accuracy of the 3D models by comparing the measured data with the visual assessment results.

In this paper, we describe these results and propose the use of a 3D measurement method for further assessment. Specifically, we develop a new parameter to quantify the blistering of leaves using normal vectors and LC data. Additionally, the growth process of leaf lettuce is quantified using length, projected area from above, volume, and compactness as parameters.

2. Materials and Methods

2.1. Overview of the 3D modelling system

Figure 1a shows the 3D modelling system that we previously developed and that was used in this study [24]. The system consists of four cameras, LED lighting, a turntable, measuring posts, a blue background, and a control PC. The system was designed to be used in a compact plant growth chamber (external dimensions of 70×70×75 cm). The cameras used were Lucid Triton TRI089S-CC (camera body 29×29×45 mm, 4096×2160 CMOS sensor resolution). A wide-angle 8 mm lens was used to capture the turntable and measurement post within the field of view. The shooting time per sample was approximately 7 minutes. The background colour of the device was painted blue to eliminate the background. The images taken with this device were processed using a closed-loop coarse-to-fine method (CLCFM) that we developed by improving the structure-from-motion/multiview stereo (SfM/MVS) method [25]. Camera positions were estimated by CLCFM, and background and noise were removed by the multimask matching method to obtain high-density 3D point cloud data (Figure 1b).

2.2. Basic System Performance

2.2.1. Shape Measurement Accuracy

We have previously reported that this system can measure the shape of a plant body up to approximately 1 mm accuracy for objects up to approximately 1 m in size [24] . Additionally, we have also reported that compared to the accuracy of a handheld laser scanner, the accuracy of this system in surface measurement of fruits is within 1 mm [26]. At this stage, we conducted an evaluation of the accuracy of the angle estimation for the detected surface by using the normal vectors (angle accuracy of the obtained point cloud) that could be obtained from the 3D point cloud generated in the present study.

2.2.2. Angle measurement accuracy

To confirm the accuracy of angle measurement for the normal vectors obtained by this system, we tilted a flat plate at angles of 0°, 15°, 30°, 45°, and 60° and reconstructed the point cloud under the same conditions as those for the lettuce. The angle of the plate was measured with a digital angle metre to serve as the actual value. We divided the point cloud into 1 mm squares, 5 mm squares, and 10 mm squares on the x-y plane and estimated the angle for each region. The mean values and standard deviations of the angles are shown in Table 2 (units are in degrees). The standard deviation of the local angle (orientation) of the plate was 1.28-3.72° for a 1×1 mm area, 0.99-1.78° for a 5×5 mm area, and 0.63-1.2° for a 10×10 mm area. These results indicate that the accuracy of angle estimation improves as the observation area is expanded. When estimating the orientation of an entire leaf or a local part of a leaf with this system, the accuracy is approximately 1° in the 5 × 5 to 10 × 10 mm area.

2.3. Plant Materials and Cultivation Methods

We tested four varieties of leaf lettuce, namely, Red Cos, Fancy Green, Frillice, and Red Leaf, that are commercially available on the market for the measurements using our device. Seeds were sown in polyurethane foam blocks (2.5 cm × 2.5 cm × 2.5 cm) soaked in tap water. Then, the plants were placed in a plant growth chamber under a white LED light source with a light intensity of 150 μmol m-2 s-1 and a 16-hour photoperiod at 22°C. The seedlings were grown hydroponically in ½ Otsuka A solution (electrical conductivity of 1.35-1.6 dS m-1, pH 5.5-6.5, OAT Agrio, Tokyo, Japan) in plastic trays. One week after sowing, the seedlings were transplanted to plug trays (3 cm × 3 cm × 4 cm) and cultivated using a hydroponic system.

2.4. Measurement and Analysis Methods for Lettuce Morphology

2.4.1. Shape Measurements

2.4.1.1. Length and Volume

We measured the changes in length and volume representing six characteristics related to dimension using this device, which provided the most fundamental information about the growth status of the lettuce. We captured images of three varieties of leaf lettuce, namely, Fancy Green, Frillice, and Red Leaf, every other day from 16 to 54 days after sowing. Plant height and long and short diameters were measured using a ruler to obtain actual measurements. Plant height was measured from the ground to the top of the lettuce, while the major and minor axis lengths were measured as the longest and shortest distances, respectively, when viewed from above. We measured three plants for each variety and then calculated the average value of the measurements. We also measured the volume and weight of 17 individuals of Red Cos lettuce of different sizes. We harvested the lettuce plants and measured their volume by submerging them in a graduated cylinder filled with water and recording the water level change as the actual value. For 3D measurements, we used the 3D model reconstructed from images captured by this device to measure the plant height, major axis length, and minor axis length of the lettuce. We estimated the volume by approximating the 3D model using four methods: bounding box, convex hull, columnar solid, and voxel. The bounding box was the smallest rectangle in which the 3D model of a lettuce plant could be enclosed (Figure 2a). The convex hull was the smallest convex polyhedron containing the 3D model of a lettuce plant (Figure 2b). The columnar solid was the product of the height of the bounding box and the area of the vertical projection as viewed from above (Figure 2c). The voxel volume was the volume calculated by converting the 3D point cloud data into a collection of small cubes (voxels) (Figure 2d). The voxel size was 0.5 mm per side.

2.4.1.2. Structural Estimation (Compactness)

There is an index called compactness that is calculated by using the two aforementioned values of voxel volume and convex hull volume. By using this, it may be possible to estimate plant structure, harvesting time, and so on. The compactness of plants has been evaluated in various ways in the past. In 2D, compactness is defined as the total leaf area divided by the convex hull area [27,28,29]. In 3D, McCormick et al. (2016)[30] measured shoot compactness by measuring the surface area of the smallest convex polyhedron that contained the entire shoot. In another study that was on grapes, a method was proposed to evaluate the structure of grape clusters using the compactness of 3D models [31]. In the case of lettuce, compactness corresponds to the size of the space between lettuce leaves and may be a potential method for evaluating the density of the leaves. In this paper, compactness is defined by the following equation.

compactness = voxel volume/ volume of convex hull

(1)

From this equation, it can be seen that the closer the value is to 1, the more space is filled without gaps.

2.4.2. Normal Vector-based Analysis

Leaf lettuce has a wide variety of cultivars available on the market. Lettuce leaves have differences in leaf morphology, such as wrinkles and flatness, and these morphologies differentiate each cultivar. Wrinkles and flatness are referred to as "blistering" and "size of blisters" in the Lettuce Variety Inspection Standards (Table 1). Therefore, we investigated a quantitative method for evaluating leaf morphology using normal vector information obtained from 3D models.

2.4.2.1. Normal Vectors

Normal vectors are vectors perpendicular to a plane and can be obtained from the point cloud reconstructed using this system. By utilizing the direction information of these normal vectors, it is possible to estimate the orientation (angle) of plant leaves. The normal vectors are calculated using a set of points, including the target point and its neighbouring points in the point cloud. That is, the normal vector is calculated by fitting a local surface to the selected neighbouring points, with different sizes such as 3×3, 5×5, and 7×7, depending on the local surface size of the points (Figure 3).

The normal vector was obtained by solving the relevant plane equation:

z = ax + by + c,

(2)

y = ax + bz + c,

(3)

x = ay + az + c

(4)

from the selected point on the 3-dimensional coordinate, and each coefficient was obtained using the least-squares method.

2.4.2.2. Local Curvature (LC)

Normal vector data have also been used in plant visualization by Pound et al. (2014) [32]. They used LC calculated via the normal vector data for 3D reconstruction. We surmised that LC could also be used for plant shape evaluation. It is possible that wrinkles in lettuce leaves could be detected as edges. We applied this method to measure the lettuce plants and calculated LC using the following equation.

$$\mathrm{L}\mathrm{o}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{c}\mathrm{u}\mathrm{r}\mathrm{v}\mathrm{a}\mathrm{t}\mathrm{u}\mathrm{r}\mathrm{e}=\sqrt{{\mathit{s}}_{\mathit{n}\mathit{x}}^2+{\mathit{s}}_{\mathit{n}\mathit{y}}^2+{\mathit{s}}_{\mathit{n}\mathit{z}}^2}$$

(5)

where $\left(nx,ny,nz\right)$: Normal vector; $s_{nx}$,$s_{ny}$,$s_{nz}$: standard deviation of$nx,ny,nz.$By using the LC value, it is possible to evaluate "blistering" and "size of blisters" (Table 1). As an example, an arbitrarily created 3D model is shown in Figure 4 a-c. Figure 4 d-f shows images in which differences in the direction of the plane are visualized by colour coding based on the normal vectors. Figure 4 g-i shows a heat-map visualization of the difference in LC calculated using the normal vector data. The areas where the 3D model has peaks and valleys have large LC values and are shown in yellow to red. The edge part indicated by arrows attached to Figure 4 e and f is detected at the positions indicated by arrows in the heatmap of h and i. In the case of a highly convex and concave shape, such as that shown in Figure 4f, many regions with high LC values are detected. In such cases, when the distribution of LC values is shown in a histogram, the distribution is skewed towards larger values of LC on the horizontal axis (Figure 4l). On the other hand, when there are few or small concavities and convexities, such as in Figure 4d and e, the distribution of LC values is skewed towards smaller values (Figure 4j, k). Figure 4 k shows high values in the series 0-0.05, which is a flat region with little slope variation, shown in black in the heatmap of Figure 4 h.

2.4.2.3. Quantification of LC: Centroid of Histogram

As mentioned above, the distribution of LC values tends to skew towards higher values as the surface becomes more uneven. Therefore, it is possible to evaluate the changes in the blistering of lettuce leaves during growth as a change in the skewness of the distribution of LC values. In this study, we attempted to quantify this change by calculating the centroid of the histogram using the following formula.

$${\mathrm{Centre}\mathrm{of}\mathrm{Gravity}\text{：}\mathrm{L}\mathrm{C}}_{\mathrm{g}}＝\frac{\sum _{\mathrm{L}\mathrm{C}=0}^1\left(\mathrm{L}\mathrm{C}\times \mathrm{f}\mathrm{r}\mathrm{e}\mathrm{q}\mathrm{u}\mathrm{e}\mathrm{n}\mathrm{c}\mathrm{y}\right)}{\sum _{\mathrm{L}\mathrm{C}=0}^1\mathrm{f}\mathrm{r}\mathrm{e}\mathrm{q}\mathrm{u}\mathrm{e}\mathrm{n}\mathrm{c}\mathrm{y}}$$

(6)

LC: Local curvature

3. Results

3.1 Measurement Accuracy

3.1.1. Accuracy of Lettuce Length Measurement

The 62 lettuce characteristics listed in the Japanese National Test Guidelines (April 2022, Ministry of Agriculture, Forestry and Fisheries)[22] include 4 characteristics related to size: Characteristic 2, cotyledon size; Characteristic 4, plant diameter: Characteristic 19, leaf length and Characteristic 20, leaf width. In this study, the following measurements were made and evaluated for plant size, which can be measured nondestructively. Using the 3D modelling system developed in this study, we continuously captured images of three leaf lettuce varieties, namely, Fancy Green, Frillice, and Red Leaf, every other day from 16 to 54 days after sowing and obtained temporal 3D data of the growth of the leaf lettuce. For each of the three varieties, various parameters were measured using the constructed 3D model data and were compared with manual measurements. We investigated the correlation between the values measured on the 3D model and those measured by hand using a ruler for the plant height, major diameter, and minor diameter of the leaf lettuce for the three varieties. As a result, the correlation coefficients of the measured values for plant height and the major and minor diameters were all 0.99 or higher for all three varieties (Figure 5). The differences between the 3D measurement values and hand measurement values were 0.22 ± 1.82 mm to 0.81 ± 2.15 mm (Table 3). The standard deviation of the measurement error was approximately ±1 mm for plant height, while it ranged from ±1 mm to 3 mm for diameter. One of the reasons for the errors in these measurements may be that the errors occurred when the ruler was pressed against the lettuce leaves and that the measurement positions of the manual and computer measurements did not perfectly match. However, since the correlation between the two measurements was high, it should not be considered to be a problem for practical use.

3.1.2. Accuracy of Lettuce Volume Measurement

We investigated four methods for measuring the shape and volume of lettuce using 3D models. These methods are shown in Figure 2: (a) bounding box, (b) convex hull, (c) columnar solid, and (d) voxel volume. Seventeen Red Cos lettuce plants at different growth stages and of different sizes were used as plant material. The correlation coefficients between the volumes measured by the above four methods and the actual volumes measured by submerging the plants in water in a graduated cylinder were 0.919, 0.920, 0.937, and 0.962, respectively (Figure 6 a-d). The accuracy of volume measurement was proportional to the computational cost, and the voxel volume method showed the highest correlation. The other methods also provided high correlation for evaluating the growth and characteristics of lettuce.

Voxel-based volume measurements showed the highest correlation but also had the highest computational cost. In this study, we attempted to calculate at voxel sizes of 1 mm or less, but even with a high-performance PC (CPU: Intel Core i9-10900K), it was impossible to perform the calculations. The measurement accuracy of convex hull volume and columnar solid volume was lower than that for voxel volume, but their correlation coefficients with actual measurements were above 0.9, which is considered sufficient for evaluating crop growth and characteristics. Furthermore, it is possible to automatically calculate characteristics such as length and width using these methods.

3.2. Normal Vector-based Analysis (consideration of quantification of visual assessment)

3.2.1. Visualization

Visualizing the normal vector information that represents the orientation of the point cloud on the 3D model can enable more intuitive visualization and quantification of the blisters. Therefore, a series of 3D data that recorded the morphological changes in lettuce growth from Section 3.1 were used to visualize the normal vector calculated by the formula shown in Section 2.4.2.1.

3.2.1.1. Visualization of Normal Vectors

Figure 7 shows a three-colour visualization of the normal vectors in the x, y, and z directions for a 3D model of Fancy Green at 30 to 54 days after sowing. Blue indicates when the leaf surface is facing straight up (z axis direction), while green or red is displayed when it is facing sideways (x, y axis direction). Visualizing the normal data makes it easier to visually read the direction the leaf surface is facing and the complexity of the leaf shape.

3.2.1.2. Visualization of LC

Figure 8 shows 3D models of a single lettuce leaf cut out and visualized with normal vectors and a heatmap of LC. LC values are displayed in a gradient from red to yellow, green, and blue, with the highest values shown in red. Regions with strong blistering are shown in red, while smooth regions are shown in blue.

Next, we visualized the whole lettuce plant in the same way. Figure 9-1, 9-2, and 9-3 show the results of the normal vector analysis for three lettuce varieties (Fancy Green, Red Leaf, and Frillice). (a) 3D model viewed from the side, (b) 3D model viewed from the top, (c) heatmap visualization of the LC of the 3D model viewed from the top, and (d) image of the normal vector viewed from the top with three colours for the x, y, and z axis directions. In the heatmap of (c) in Figure 9-1 to 9-3, the more uneven areas are shown in red to yellow, and the smoother areas are shown in green to blue. Comparing the three varieties, Fancy Green had many flat areas in blue and green from 26 to 44 days after sowing (Figure 9-1c), while Frillice and Red Leaf had more wrinkled areas in red from 30 days after sowing, and the wrinkles spread to the entire leaf thereafter (Figure 9-2c). The wrinkles spread to the entire leaf (Figure 9-2c and 9-3c).

3.2.2 Quantification of Convexity and Concavity

As mentioned above, differences in the distribution of LC values among the varieties can be quantitatively compared by determining the centroid position of the LC histogram, as shown in Section 2.4.2.3. The LC data for the three lettuce varieties shown in Figure 9 were represented in histograms, and their centroids were calculated (Figure 10). When the LC values are skewed towards smaller values, there are many flat areas with small changes in the normal vectors of neighbouring points. In this case, the centroid value is small. On the other hand, when the LC values are skewed towards larger values, there are many areas with large changes in normal vectors, indicating many areas with high convexity and concavity. In this case, the centroid value is large.

As shown in the heatmap of Figure 9-1c, Fancy Green had many flat areas that were represented in blue and green colours from 26 to 44 days after sowing. In the histogram of LC shown in Figure 10a, the centre of gravity remained at approximately 0.46 to 0.47 from 26 to 44 days after sowing (Figure 10a). Approximately 54 days after sowing, the area with wrinkles, as shown in Figure 9-1c, expanded, and the centroid of the LC histogram also increased to 0.56. In contrast, Frillice and Red Leaf showed a tendency of an increase in areas with high convexity and concavity, represented in red in Figure 9-2c and Figure 9-3c, from approximately 30 days after sowing. At this time, the centroid of the LC histogram increased along with growth, ranging from 0.52 to 0.64 for Frillice (Figure 10b) and from 0.51 to 0.62 for Red Leaf (Figure 10c). These results indicate that changes in the distribution of visualized LCs are consistent with changes in the value of the centre of gravity of the histogram of LCs. This suggests that quantitative evaluation of convexity and concavity by LC is possible.

3.3. Time-series Measurement Evaluation

We examined the quantification of changes in leaf morphology. We then checked whether it is possible to track these changes. Morphological changes during growth were quantified using time series data from Section 3.1, in which lettuce growth was continuously recorded every other day. Using these 3D models, plant height, vertical projected area viewed from above, voxel volume, compactness, and local curvature were measured and compared among the three cultivars (Figure 11a-e).

Red Leaf showed a lower trend in plant height change in the early growth stage at approximately 20 days after sowing (Figure 11a). This is because Red Leaf leaves grow horizontally in the early growth stage; thus, the height of the plant increases slowly. As the plants grew, new leaves were stacked upon the old leaves, and the plant height increased; almost no difference was observed among the three varieties at approximately 35 d after sowing (Figure 11a).

The most common method of evaluating lettuce yield is weighing. However, it is desirable to measure lettuce during growth using a nondestructive method. Therefore, the vertical projected area (ratio of leaf area) and volume were measured from the 3D model and evaluated. On the last day of measurement, the lettuce plants were harvested, and the fresh weights were measured. Red Leaf was heavier than Fancy Green, weighing 31.7 ± 0.5 g and 28.3 ± 1.3 g, respectively. As shown in Figure 11c, Red Leaf > Fancy Green was also observed when the volume was evaluated using voxels. However, as shown in Figure 11b, the vertical projected area was Fancy Green > Red Leaf. These results suggest that voxel volume, a three-dimensional evaluation, reflects characteristics more accurately than the vertical projected area, a two-dimensional evaluation. The reason for the opposite relationship between the vertical projected area or voxel volume of Red Leaf and Fancy Green may be related to the morphology of the lettuce leaves. Comparing the leaf morphology of Red Leaf and Fancy Green, the entire leaf of the former was severely crumpled, while that of the latter was less crumpled and had more areas with smooth surfaces. Leaves that are severely crumpled occupy a larger space. This cannot be quantified using vertically projected areas.

Therefore, we evaluated the ratio of leaves occupying the space using compactness (Figure 11d). When compactness was compared, the relationship between the cultivars was Red Leaf > Fancy Green (Figure 11d). This indicates that the ratio of leaves occupying space is larger in Red Leaf than in Fancy Green leaves. Furthermore, to compare the difference in the crumpling of leaves between the two cultivars, the distribution of local curvature values was expressed in terms of the centroid of the LC histogram as described above, indicating that Red Leaf had a larger value than Fancy Green (Figure 11e). This indicates that Red Leaf leaves were more severely crumpled than Fancy Green leaves.

These results suggest that leaves of Red Leaf occupy a higher ratio of space than those of Fancy Green due to their severe crumpling, resulting in a larger volume in relation to the vertical projected area.

Among the three cultivars, Frillice appeared smaller in size, which was reflected in the vertical projected area and voxel volume values viewed from directly above (Figure 11b, c). Frillice showed similar changes to Fancy Green, with lower compactness values in the early stages of growth. Subsequently, the value of compactness increased from 35 d after sowing and showed a pattern similar to that of red leaf from 40 d after sowing (Figure 11d). These changes in values may quantitatively characterize the morphological change of Frillice, in which the leaves that develop in the early stages of growth are not crumpled, but severely crumpled leaves begin to appear in the later stages of growth.

The local curvature of Fancy Green tended to be smaller than that of Red Leaf until the plants matured (Figure 11e). This is consistent with the subjective impression that Fancy Green has fewer crumpled areas and flatter areas on its leaves than Red Leaf.

These results showed that the 3D measurement method provides a great deal of information that cannot be obtained from 2D image measurement and enables analysis by quantification of growth characteristics based on time-series information.

4. Discussion

As mentioned above, the LC-based characteristic evaluation was shown to be consistent with subjective evaluation. As shown in the Lettuce Variety Inspection Standards (Table 1), characteristics related to leaf concavity and convexity are evaluated by visual assessment. We believe that the LC-based characteristic evaluation described in this paper may be applicable to this. The following preliminary data have been obtained.

4.1. Leaf Lettuce Characteristic Evaluation

We evaluated two lettuce characteristics, Characteristic 22 "Leaf blistering" and Characteristic 23 "Size of blisters", that are used in the examination for registration of lettuce varieties. We compared the evaluations by visual assessment, which is currently conducted in the Center for Seeds and Seedlings, NARO, with those by 3D measurements. The visual assessment is to be performed in five or three stages according to the Test Guidelines for Lettuce (April 2022; Ministry of Agriculture, Forestry and Fisheries)[22], as shown in Table 4. We checked whether this visual assessment could be quantified by LC.

First, we performed 3D measurements for a single leaf of 7 lettuce types, Lettuce A to G, sold on the market and created 3D models (Figure 12). We extracted the area excluding the marginal part and part of the midrib (the part where the vascular bundles running through the centre of the leaf are concentrated). We performed LC analysis on this extracted area by the dotted line in the 3D model in the upper part of Figure 12. The area was manually cut out. The heatmap in the middle row of Figure 12 shows that the region marked by orange to red corresponds to the area evaluated by human observation as wrinkles. Samples judged as 7 (strong) for blistering by visual assessment showed many areas marked in orange or red, samples judged as 3 (weak) showed few of those areas, and samples judged as 5 (medium) showed a tendency that was intermediate between the two. When we calculated the centroid of the LC histogram, samples judged as 3 (weak) had values of 0.35 and 0.44, samples judged as 5 (medium) had values of 0.46, 0.41, and 0.39, and samples judged as 7 (strong) had values of 0.48 for both samples (Figure 12). For six samples, excluding lettuce B, there was a suggested correspondence between the value of the centroid of the LC histogram and the strength of blistering evaluated by visual assessment. For the exception of lettuce B, the region extraction was considered incomplete. If the periphery and midrib are not excluded, wrinkles other than blistering may be detected incorrectly. In the future, we will accumulate more data and consider the best method for selecting the area.

Using the same LC analysis data, we also attempted to analyse Characteristic 23, size of blisters. When viewed by the human eye, the larger the flat part of the blister, the larger the blister is judged to be. In flat areas, the value of LC is considered small. Therefore, we analysed the percentage of leaves occupied by areas exhibiting each LC value and examined the correlation with the visual assessment (Figure 13). As shown in the heatmap in the middle of Figure 12, the regions shown in blue to green corresponded to areas that were evaluated as smooth by the human eye. Therefore, assuming that regions with LC < 0.5 shown in blue to green are smooth regions, we calculated the percentage of those regions that occupy the leaf. As a result, in samples where the size of blistering was judged as 3, considered small by visual assessment, the regions with LC < 0.5 occupied 74.9% and 70.4%, while in samples where it was judged as 5, considered medium, they occupied 96.4%, 91.3%, 90.8%, 89.5%, and 76.3%, respectively.

From these results, we suggest that there is a correlation between the LC-based evaluation and the visual assessment for Characteristic 22, leaf blistering and Characteristic 23, size of blisters.

In the surveyed samples, there was some correlation between the evaluation based on LC and the observed evaluation, but the evaluation of the size of the convex and concave parts cannot be fully assessed by the evaluation of the flat parts. However, the evaluation criterion only evaluates the convexity (blister), while the LC data calculate the amount of convexity and concavity but not the orientation. This problem can likely be addressed by combining the normal vector data indicating the orientation of the curved surface to distinguish between convex and concave parts.

From these results, the new LC-based characteristic evaluation approach proposed in this paper displays excellent practical utility. However, further validation is needed due to the lack of data in this case. The correspondence between characteristic evaluation using LC data and characteristic evaluation based on human senses must be further investigated.

Some evaluation characteristics are related to the shape of leaf edges, such as Characteristics 24-28, and these characteristics can be applied in modified LC-based analysis.

For example, Characteristic 24, the degree of undulation of margins, can be evaluated in 3D. Characteristics 25, 26, 27 and 28, which require cutting at the tip of the leaf blade, can be evaluated by projecting the leaves onto a flat surface. We believe that these characteristics can also be quantitatively evaluated from the 3D information acquired by our system. In addition, for the evaluation of these characteristics, it is necessary to extract the margin on the apical part, which in our data (Figure 8c) was detected as an area with a large LC value (red area). It is likely that the evaluation of the undulation and incisions of margins can be analysed by performing region extraction as 3D data, and this will be a future challenge.

The concavity and convexity of leaves can also be listed in test guidelines for vegetables such as cabbage, broccoli, and cauliflower, so our technique could be applied to these vegetables as well.

5. Conclusions

In this paper, we proposed an algorithm and its principles for measuring the dimensions and evaluating the visual observation of lettuce using the 3D measurement device that we developed. Ultimately, we demonstrated that various characteristics that cannot be assessed through 2D image analysis can be quantified and evaluated using 3D data.

The system developed in this study has a measurement accuracy of 1 mm and an angular accuracy of approximately 1°, which is considered sufficiently high to evaluate the morphology of crops such as lettuce. The 3D model created using a previous 3D imaging system contains nonnegligible noise and occlusion [24] and is considered insufficiently accurate, especially for analysis using normal data. However, the accuracy of the 3D model reconstructed using the system developed in this study was sufficiently high; thus, we attempted to quantitatively evaluate the morphology of the plant leaf using normal data. Therefore, we attempted to evaluate the angle accuracy of our developed system and quantitatively evaluate the morphology of plant leaves using 3D measurements and normal data. As a result, nondestructive 3D measurements of leaf lettuce with high accuracy were possible. The parameters that can be measured are the length, vertical projected area, volume, compactness, and the visualization and quantification of leaf curvature using a newly introduced local curvature based on normal vectors. Furthermore, we visualized and quantified the curvature of leaves using LC based on the newly proposed normal vectors in this paper.

These results show that it is possible to evaluate characteristics based on visual assessment using the proposed device and evaluation method. These parameters can be used for growth monitoring to predict the harvest time of crops and for the early detection of growth problems and diseases. In addition, in cultivation in artificial environments such as plant nurseries, crop conditions can be precisely monitored, and the accuracy of cultivation environment control can be improved.

However, these methods need to be validated with more data. Such data and subsequent experiments are in preparation.

Recently, omics analyses such as those related to the transcriptome, proteome, and metabolome have been accelerated, and the data produced have accumulated, but the technology to collect phenotype information efficiently is still under development, and it has been difficult to quantify the complexity of morphology. However, the morphological analysis of crops using local curvature and compactness by normal vectors attempted in this study succeeded in visualizing and quantifying the complex morphological data of concavity and convexity. Phenotype data as objective variables for explanatory variables, such as omics data, will be more enriched than ever before, which will also contribute to improving the accuracy of genetic analysis.