Weed Segmentation in Sugarcane Fields from UAV Orthomosaics Using Convolutional Neural Networks and Promptable Mask Annotation

1. Introduction

Weeds compete with sugarcane for water, nutrients, light, and space, and the resulting interference translates directly into lost yield. Field experiments in Brazilian sugarcane have quantified this: infestations dominated by Brachiaria decumbens reduced stalk yield by up to 82%, and mixed infestations of signalgrass and guineagrass caused reductions near 40% [1]. The same body of work established that interference is concentrated in a critical window early in the cycle [2], which is precisely the period in which a grower benefits most from knowing where the infestation is rather than simply that it exists.

Uniform herbicide application ignores that spatial structure. It treats clean and infested areas alike, raising input costs and selection pressure for resistance without a corresponding yield benefit. Site-specific management is the alternative, but it presupposes a map, and producing that map by scouting is slow and does not scale to the field sizes typical of commercial sugarcane.

Automated identification from imagery is difficult for reasons specific to this crop. Sugarcane and many of its associated weeds are grasses; they share colour and coarse texture, so the discrimination is a green-on-green problem in which spectral thresholds and hand-designed features perform poorly. Morphology varies with growth stage, illumination changes across a flight, and weed patches merge visually into the canopy that surrounds them. Convolutional neural networks (CNNs) address this by learning discriminative representations directly from data, and they have become the dominant approach in crop–weed recognition [3]. dos Santos Ferreira et al. [4] demonstrated the principle in Brazilian soybean using UAV imagery, and Verçosa [5] applied CNNs specifically to weed detection in sugarcane.

A practical obstacle remains upstream of the model: supervised segmentation needs pixel-level labels, and producing them by hand over orthomosaics covering hundreds of hectares is prohibitive. Promptable segmentation models change this economics. The Segment Anything Model [6] and its successor SAM 2 [7] generate high-quality masks from cheap prompts such as points or boxes, which allows an operator to supply a bounding box in seconds and obtain a mask that would otherwise take minutes to trace.

This paper reports a pipeline built around that idea and evaluated on commercial sugarcane fields. Our contributions are:

1.

a semi-automatic annotation workflow in which operator bounding boxes prompt SAM 2, and the resulting masks are georeferenced and exported as GeoJSON so that predictions remain tied to field coordinates;

2.

a dataset of 5000 tiles cropped from UAV orthomosaics of commercial sugarcane, annotated at pixel level across agronomically defined weed categories;

3.

a per-class evaluation over approximately 6.9 million pixels showing that the limiting error in this task is the weed–canopy boundary rather than discrimination among weed types—a finding that contradicts the intuition that morphologically similar weed groups are the hard case.

Section 2 reviews related work. Section 3 describes the data, the annotation pipeline, and the training setup. Section 4 presents the results, Section 5 interprets them and states the limitations, and Section 6 concludes.

2. Related Work

• Weed recognition with deep learning.

Adhinata et al. [3] survey machine learning and deep learning for weed and crop classification and identify the recurring difficulties: visual similarity between weed and crop, occlusion, illumination variability, and the need for early-stage detection when plants are small and least distinctive. Among applications in Brazilian agriculture, dos Santos Ferreira et al. [4] combined SLIC superpixel segmentation with CNN classification on UAV soybean imagery, separating soil, crop, grass, and broadleaf weeds. Work specific to sugarcane is comparatively sparse; Verçosa [5] developed a CNN-based detector for weeds in sugarcane fields and reported that image quality and class diversity dominate performance.

• Semantic segmentation.

Fully convolutional networks [8] established dense prediction by replacing fully connected layers with convolutions and upsampling. U-Net [9] added skip connections between a contracting encoder and an expanding decoder, recovering the spatial detail lost to pooling; it remains a strong default when annotated data are limited, which is the usual situation in agriculture. Garcia-Garcia et al. [10] review the field more broadly.

• Backbone architectures.

VGG16 [11] stacks small $3\times 3$ convolutions into a deep, uniform network; its roughly 138 million parameters make it expensive, but its features transfer well. MobileNetV2 [12] uses depthwise separable convolutions with inverted residuals and linear bottlenecks to reach about 3.4 million parameters, targeting devices with limited compute. EfficientNet [13] later formalised the trade-off through compound scaling. In agricultural deployment the choice matters because inference may run on an embedded platform in a sprayer rather than on a workstation.

• Promptable segmentation for annotation.

SAM [6] introduced promptable segmentation trained on a billion-mask corpus, achieving strong zero-shot transfer. SAM 2 [7] extends the architecture with streaming memory and improves both accuracy and speed on images. Using such a model as an annotation accelerator, rather than as the deployed predictor, is the role it plays here.

• Loss functions for imbalanced segmentation.

Cross-entropy weights every pixel equally and therefore under-serves small structures. Region-overlap losses respond to this: Dice loss [14] optimises overlap directly, Tversky loss [15] adds asymmetric control over false positives and negatives, focal Tversky loss [16] concentrates gradient on hard examples, and focal loss [17] addresses the analogous imbalance in dense detection.

3. Materials and Methods

3.1. Image Acquisition and Dataset Construction

The imagery consists of orthomosaics assembled from UAV flights over commercial sugarcane fields. The aircraft carried a high-resolution RGB sensor; the resulting mosaics capture both healthy stands and infested areas, and each retains its geographic referencing so that any derived product can be mapped back to field coordinates. From these orthomosaics we cropped 5000 tiles.

Five categories were annotated, following the groups agronomists use when prescribing control (Figure 1):

• Sugarcane canopy (background)—healthy stand with no relevant infestation;
• Narrow-leaf weeds—grasses (Poaceae) morphologically close to the crop;
• Broadleaf weeds—dicotyledonous species with wide leaf blades;
• Castor bean (Ricinus communis)—treated separately because of its distinctive palmate leaf and its agronomic importance in the region;
• Other creeping weeds—prostrate species not covered by the categories above.

Separating narrow-leaf from broadleaf weeds is not a taxonomic formality: the two groups respond to different herbicide classes, so the distinction is what makes a prediction actionable.

The models reported in Section 4 were trained and evaluated on the first four of these categories. The other creeping weeds class was heterogeneous by construction—it was defined residually, as whatever the other four did not cover—and excluding it removed a source of label noise without changing the agronomic decision the system supports, since creeping species are treated with the same broad-spectrum products as the broadleaf group. All results below are therefore four-class.

3.2. Semi-Automatic Annotation with SAM 2

Pixel-level annotation was produced through the four-step procedure summarised in Figure 2.

1.

Prompting. An operator draws bounding boxes around the elements of interest in a tile. Boxes are colour-coded, and the operator does not trace outlines—a coarse box is sufficient.

2.

Mask generation. The boxes are passed to SAM 2 [7] as prompts. The model returns a binary mask per prompted region.

3.

Polarity convention. For weed classes, foreground (white) denotes the weed. For background tiles, which contain only healthy sugarcane, the operator marks the regions that are not of interest and the mask is inverted, so that white consistently denotes the class named by the tile label.

4.

Georeferencing and export. Mask contours are vectorised and written to GeoJSON, preserving the geographic coordinates inherited from the orthomosaic. Masks, source tiles, and GeoJSON files are stored together.

Every mask was inspected. Where SAM 2 produced an unsatisfactory boundary, the prompt was redrawn and the mask regenerated; ambiguous cases were referred to an agronomist. No mask entered the training set without approval.

3.3. Network Architectures

Both models follow an encoder–decoder design in the style of U-Net [9], differing only in the encoder (Table 1). Encoders were initialised from ImageNet [18] and fine-tuned rather than frozen. VGG16 [11] was chosen for its representational capacity and reliable transfer; MobileNetV2 [12] for its suitability to embedded inference.

3.4. Training Configuration

Table 2 lists the hyperparameters. Learning rates in the range $1\times {10}^{-5}$ to $5\times {10}^{-5}$ are conservative, which is appropriate when fine-tuning pretrained weights on a comparatively small dataset; Smith [19] discusses the sensitivity of training to this choice.

Three mechanisms guard against overfitting. Early stopping halts training when validation loss stops improving, which Prechelt [20] analyses as an implicit capacity constraint. Checkpointing retains the best validation state rather than the final one. $L_2$ regularisation penalises large weights [21]; the theoretical background for all three is treated in Goodfellow et al. [22].

The colour-space difference in the last row of Table 2 is not a controlled variable but a property of the two configurations as they were run. It is a confounder, and Section 5.4 states what follows from that.

3.5. Loss Functions

Three losses were evaluated (Table 3). Dice loss performed best and was used for the reported models, which is consistent with the class-imbalance argument: weed pixels are a minority of every tile, and a loss that optimises region overlap is less easily dominated by the majority class than pixel-wise cross-entropy.

3.6. Evaluation Protocol and Metrics

The dataset was partitioned into training, validation, and test subsets. Validation guided early stopping and checkpointing; the test subset was used once, for the figures reported in Section 4.

Evaluation is at pixel level. Writing ${\mathrm{TP}}_c$, ${\mathrm{FP}}_c$, and ${\mathrm{FN}}_c$ for the true positive, false positive, and false negative pixel counts of class c, we report

$$P_c={\displaystyle \frac{{\mathrm{TP}}_c}{{\mathrm{TP}}_c+{\mathrm{FP}}_c}},R_c={\displaystyle \frac{{\mathrm{TP}}_c}{{\mathrm{TP}}_c+{\mathrm{FN}}_c}},F{1}_c={\displaystyle \frac{2P_c{R}_c}{P_c+R_c}},$$

(1)

$${\mathrm{IoU}}_c={\displaystyle \frac{{\mathrm{TP}}_c}{{\mathrm{TP}}_c+{\mathrm{FP}}_c+{\mathrm{FN}}_c}},\mathrm{mIoU}={\displaystyle \frac{1}{C}}{\displaystyle \sum _{c=1}^C}{\mathrm{IoU}}_c,$$

(2)

together with overall pixel accuracy and the frequency-weighted IoU, which weights ${\mathrm{IoU}}_c$ by the share of ground-truth pixels in class c. Overall accuracy is reported for continuity with earlier work but should not be read alone: with background accounting for 62.4% of all pixels, a model that simply predicted background everywhere would already score above 0.6.

4. Results

Both models were evaluated over approximately $6.9\times {10}^6$ labelled pixels.1

4.1. VGG16

The VGG16 model reached 83.06% overall pixel accuracy and 0.692 mIoU (Table 4). Per-class results are given in Table 5 and the confusion structure in Figure 3a.

Narrow-leaf weeds were the strongest class ($F1=0.908$, $\mathrm{IoU}=0.832$) and broadleaf weeds the weakest ($F1=0.700$, $\mathrm{IoU}=0.539$). The reason is visible in Figure 3a: 25.0% of broadleaf pixels were assigned to background, and 7.5% of background pixels were assigned to broadleaf. Castor bean shows the same asymmetry in the opposite direction—its recall is high (84.8%) but its precision is only 0.712, because 8.0% of the much larger background class was predicted as castor bean.

Confusion between weed types is minor. Only 2.6% of narrow-leaf pixels were labelled broadleaf and 1.6% of broadleaf pixels labelled narrow-leaf. The limiting factor is not telling weeds apart; it is telling weeds from the canopy they sit in.

The loss curves (Figure 4a) fall steeply over the first epochs and flatten around the sixth; early stopping triggered at epoch 10, with training loss ending near 0.47 and validation loss near 0.67.

4.2. MobileNetV2

The MobileNetV2 model reached 73.87% accuracy and 0.549 mIoU. Its error pattern differs from VGG16’s in an important way. Narrow-leaf weeds, the strongest class for VGG16, are its weakest ($F1=0.649$, $\mathrm{IoU}=0.480$), driven by low precision (0.559): 7.8% of background pixels were predicted as narrow-leaf, and because background is roughly six times larger than the narrow-leaf class, that small percentage overwhelms the true positives.

Background recall fell to 71.5%, against 84.1% for VGG16. In operational terms this is the more consequential figure: 28.5% of clean canopy was assigned to some weed class, which in a variable-rate prescription would mean spraying more than a quarter of the uninfested area.

Training ran 19 epochs before early stopping, with training loss ending near 0.28 and validation loss near 0.45 (Figure 4b).

4.3. Comparison

VGG16 leads on every aggregate metric: 9.19 percentage points of pixel accuracy, 0.143 of mIoU, and 0.106 of mean $F1$. It also leads on all four classes individually, by margins ranging from 0.058 $F1$ for castor bean to 0.259 for narrow-leaf. MobileNetV2 uses roughly 2.5% of the parameters.

Three clarifications matter for reading these numbers.

First, the final loss values are not comparable across the two runs. MobileNetV2 ends at a lower validation loss (0.45) than VGG16 (0.67) while segmenting substantially worse. Loss magnitude depends on the configuration that produced it, and comparing absolute values across runs that differ in input representation says nothing about segmentation quality. The IoU and $F1$ figures, which are computed on the same label space, are the comparable quantities.

Second, epochs are not compute. VGG16 requires substantially more arithmetic per forward pass than MobileNetV2, so reaching its stopping criterion in fewer epochs does not mean it trained faster in wall-clock or energy terms, and it does not make it cheaper to deploy.

Third, reducing input channels does not make VGG16 efficient. Moving from three channels to one affects only the first convolutional layer; it does not meaningfully reduce the cost of a 138-million-parameter network. Any claim that grayscale input made VGG16 computationally competitive with MobileNetV2 would be unsupported, and we do not make it.

5. Discussion

5.1. The Limiting Error Is the Weed–Canopy Boundary

The per-class analysis overturns the intuition that guided the class design. Narrow-leaf and broadleaf weeds were separated as distinct categories on the assumption that telling them apart would be the hard problem, since both are green vegetation seen from above and the two groups are morphologically closer to each other than to bare soil. In the VGG16 model, that confusion accounts for 2.6% and 1.6% of the respective classes—close to negligible.

What dominates instead is confusion with background. Of the 1 168 899 misclassified pixels in the VGG16 model, 1 104 104, or 94.5%, involve background on one side or the other. This is what a green-on-green problem looks like at pixel level: the network can tell a broad leaf from a grass blade, but it cannot reliably decide where a weed patch ends and the sugarcane canopy begins. Boundary pixels are intrinsically ambiguous—leaves overlap, shadows fall across both—and they are also where annotation itself is least certain.

This reframes what would improve the system. Higher-capacity weed classifiers address a problem that is largely solved here. Better boundary delineation—higher ground sample distance, boundary-aware loss terms, or multispectral bands that separate species spectrally where RGB cannot—addresses the problem that actually binds.

5.2. Operational Reading

Which error is costly depends on the decision the map feeds. For a variable-rate sprayer, background recall is close to a direct measure of wasted input: every clean pixel called a weed is area sprayed unnecessarily. VGG16’s 84.1% against MobileNetV2’s 71.5% is therefore the difference between spraying about 16% and about 28% of clean canopy, which bears directly on whether the system delivers the input saving that motivates it.

Conversely, weed recall bounds the infestation that escapes treatment. Both models recover roughly 72–73% of broadleaf pixels, so approximately one broadleaf pixel in four would go untreated under either. Given that interference is concentrated early in the cycle [2], that residue is not negligible, and it is the figure we would want to improve before field deployment.

5.3. The Annotation Pipeline

The component we expect to transfer most readily is the annotation workflow. Prompting SAM 2 with a coarse box replaces manual contour tracing, and because the exported GeoJSON retains geographic coordinates, a prediction remains a location in a field rather than a label on an image. That property is what connects the model to a prescription map. The approach is not specific to sugarcane and should apply to any crop where pixel-level annotation cost is the bottleneck.

5.4. Limitations

The architecture comparison is confounded. VGG16 was trained on grayscale input and MobileNetV2 on RGB. Architecture and colour space vary together, so the differences in Table 4 cannot be attributed to architecture alone. Colour is plausibly informative for distinguishing vegetation types, so the direction of the confound is not obvious a priori—if anything, the RGB model had the representational advantage and still performed worse, but that observation is not a substitute for the experiment. Resolving this requires training both architectures on both representations.

Single split, no repetitions. Results come from one train/validation/test partition and one training run per configuration. Without cross-validation or repeated seeds we cannot separate a genuine difference from run-to-run variance, and none of the reported gaps carries a confidence interval. The partitions were also not pixel-identical between the two models.

Class exclusion. The other creeping weeds category was annotated but excluded from the reported experiments. Performance on the full five-class problem is therefore unknown, and the reported figures are optimistic relative to it.

Narrow geographic and temporal scope. All imagery comes from one production region, one crop cycle, and one sensor configuration. Generalisation to other regions, varieties, growth stages, and illumination conditions is untested.

Labels inherit SAM 2’s behaviour. Although every mask was reviewed, the boundaries originate from a model. Any systematic bias in how SAM 2 delineates vegetation edges would be inherited by the labels and therefore by the trained networks—and since the weed–canopy boundary is precisely where the models fail, this limitation and the main finding are not independent.

5.5. Future Work

The immediate priorities follow from the limitations: a factorial experiment crossing architecture with colour space, and k-fold cross-validation with repeated seeds on identical partitions. Beyond that, data augmentation [23] would address the limited sample size; encoders with better accuracy-per-FLOP such as EfficientNet [13] are worth evaluating for embedded inference; and given that the binding constraint is the weed–canopy boundary, boundary-aware losses and multispectral or near-infrared bands are the most promising directions. Validating predicted maps against ground-truth scouting, and quantifying the herbicide actually saved, is what would establish agronomic value rather than only segmentation performance.

6. Conclusions

We presented a pipeline for weed segmentation in sugarcane from UAV orthomosaics that couples SAM 2-assisted annotation with encoder–decoder CNNs over four agronomically defined classes. Evaluated on approximately 6.9 million labelled pixels, a VGG16-based model reached 83.06% pixel accuracy and 0.692 mIoU against 73.87% and 0.549 for MobileNetV2.

The per-class analysis carries the more useful result. Discriminating among weed types—the difficulty the class design anticipated—is close to solved, accounting for 1.9% of the pixels in those two classes in the better model. Over 94% of all errors involve the boundary between weed patches and the surrounding canopy. That is where the remaining accuracy lies, and it points at sensing and boundary modelling rather than at larger classifiers.

The annotation workflow is the component we consider most portable: promptable segmentation reduces the cost of pixel-level labelling enough to make supervised segmentation practical at field scale, and georeferenced GeoJSON output keeps predictions tied to coordinates a sprayer can act on. The architecture comparison, by contrast, is an operational observation rather than a controlled result, for the reasons given in Section 5.4; treating it otherwise would overstate what these experiments establish.