Enhanced Deep Learning for Food Semantic Segmentation Using Skip Connections Fine-Tuning and Explainability

1. Introduction

General Context

Global population growth, evolving consumption patterns, and increasing food losses are major challenges to the sustainability of food systems. According to the U.S. Environmental Protection Agency (EPA), food waste management must follow a hierarchy of value that prioritizes strategies with the greatest environmental and economic benefits. The EPA’s *Food Recovery Hierarchy* classifies the different ways of managing food surpluses, from preventing waste, considered the most sustainable approach, to landfill disposal, which is the least favorable solution due to its environmental impacts (EPA, 2023). Between these two extremes, redistribution for human consumption, food processing, animal feed, composting, and energy recovery are different strategies for preserving the value of food resources and promoting the principles of the circular economy (Papargyropoulou et al., 2014) [1].

However, the effective application of this hierarchy requires precise characterization of the food products involved. Indeed, before determining the best way to utilize surplus food, it is necessary to automatically identify its composition, condition, and the various components present. In real-world environments, a single food scene can simultaneously contain several food categories, such as rice, meat, vegetables, or various side dishes, making their automatic analysis particularly complex. Thus, beyond simple overall meal recognition, a nuanced understanding of the visual structure of food becomes essential for developing intelligent systems capable of assisting in the sorting, utilization, and sustainable management of food resources (Gao et al., 2026) [2].

From this perspective, computer vision and artificial intelligence technologies offer promising solutions for the automatic analysis of food scenes. Unlike traditional methods based on human inspection, approaches based on deep learning enable rapid, non-destructive analysis that can be adapted to real-world conditions. Convolutional neural networks (CNNs) have enabled major advances in automatic food recognition thanks to their ability to hierarchically extract visual features (LeCun et al., 2015) [3]. These advances have led to the development of numerous applications in the food sector, including dish recognition, nutritional analysis, and automatic monitoring of eating habits (Zhou et al., 2019) [4]. However, these approaches remain primarily focused on the overall classification of images and do not always allow for the precise identification of the different food components present in a complex scene.

Early approaches to food recognition relied primarily on image classification methods, aiming to assign an overall category to a given image. Among the most widely used databases, Food-101, proposed by Bossard et al. (2014) [5], is a major reference, providing over 100,000 images categorized into 101 food categories. However, this dataset is mainly used for classification purposes, while Food-Seg 103, introduced by Wu et al. in 2021 [6], is a dataset used for semantic segmentation of food. Published work using these datasets has led to significant advances in automatic food recognition. However, classification methods remain limited when an image contains multiple foods simultaneously, as they cannot accurately identify the location and shape of the different food components.

With the emergence of deep neural networks, convolutional neural networks (CNNs) have significantly improved the performance of food image analysis systems. Zhou et al. [4], presented a comprehensive study on the use of deep learning for automatic food understanding, including the recognition, classification, and analysis of visual features. However, a nuanced understanding of culinary scenes requires detailed spatial information, which has led to the development of semantic segmentation approaches.

Evolution of Food Recognition Towards Semantic Segmentation

Early research in the field of food vision focused primarily on image classification, the goal of which is to assign an overall category to a complete image. Among the most widely used databases, Food-101, introduced by Bossard et al. (2014) [5], is a major reference in the field of food recognition, with 101 annotated food categories. This database has enabled the evaluation of the performance of numerous deep learning models for automatic food recognition and has contributed to the rise of approaches based on convolutional neural networks. However, these global classification methods remain limited when an image simultaneously contains several food components, as they provide no information on the spatial location or proportion of each ingredient present in the scene. With recent advances in deep learning, convolutional neural networks (CNNs) have significantly improved the performance of visual food understanding systems. LeCun et al. (2015) [3], demonstrated the effectiveness of deep learning architectures for the automatic extraction of complex visual features, while Zhou et al. (2019) [4], presented a detailed analysis of deep learning applications in the food domain, including the automatic recognition, classification, and interpretation of food images. Despite these advances, classification remains insufficient for applications requiring a detailed understanding of a meal’s composition, especially when multiple foods appear in the same image.

Semantic segmentation thus represents a natural evolution that overcomes the limitations of classification by associating a category with each pixel of an image. Unlike classical approaches that only provide a global prediction, pixel-by-pixel segmentation allows for the precise identification of the position, shape, and extent of each food component. This capability is particularly important for complex food scenes, in which multiple ingredients may be mixed or partially overlapping.

In this context, several deep segmentation architectures have been proposed to improve object accuracy and localization. Ronneberger et al. (2015) [7], introduced U-Net, an encoder-decoder architecture widely used in biomedical segmentation and subsequently adapted to several computer vision domains thanks to its hop connections that preserve fine spatial information. Chen et al. (2018) [8], proposed DeepLab, which leverages atrous convolutions to improve multiscale context capture and the segmentation of objects of varying sizes. More recently, Xie et al. (2021) [9], introduced SegFormer, a visual transformer-based architecture that efficiently models global relationships in complex scenes while maintaining a lightweight architecture.

Food Segmentation and Specialized Databases

In the food industry, semantic segmentation is a crucial step for automatically analyzing meal composition and identifying the different ingredients present in a scene. Unlike traditional food recognition applications, food segmentation systems must handle significant visual variability related to differences in food presentation, lighting, texture, and shape. This complexity necessitates databases containing detailed, pixel-level annotations.

To address this need, Wu et al. (2021) [6], proposed FoodSeg103, a specialized database for semantic food segmentation comprising 103 food categories with pixel-by-pixel annotations. Unlike traditional classification databases such as Food-101, FoodSeg103 allows for more nuanced analysis of scenes containing multiple ingredients simultaneously. This characteristic makes it a suitable benchmark for evaluating food segmentation models under conditions closer to real-world applications.

Several recent studies have explored the use of artificial intelligence for the intelligent recognition, classification, and management of food waste. Mazloumian et al. [10] investigated deep learning-based approaches for automatic food waste classification using images acquired from smart waste bins, demonstrating the potential of computer vision techniques for food waste monitoring and characterization.

Rahman et al. [11] explored image-based waste analysis through the development of a dedicated waste image dataset and deep learning-based classification approaches, highlighting the importance of automated visual recognition for efficient waste segregation.

More recent studies, notably by Rahman et al. [11] and other data-driven approaches, have further demonstrated the increasing interest in artificial intelligence techniques for automated food waste segmentation, classification, sorting, and sustainable management processes.

Current Limitations and Motivation for the Study

Despite significant progress in food semantic segmentation, several challenges remain. First, the limited availability of pixel-level annotated images represents a major constraint for training deep learning models. Second, the high intra-class variability of foods, differences in presentation, and real-world acquisition conditions make model generalization difficult. Finally, modern deep learning architectures are often still considered difficult to interpret, limiting their adoption in applications requiring an understanding of the network’s decisions.

In this context, the explainability of deep learning models becomes essential. Interpretation methods such as Grad-CAM, proposed by Selvaraju et al. (2017), [12], allow visualization of the image regions that contributed to the model’s decision. This analysis provides a better understanding of network behavior and allows us to assess whether the extracted features truly correspond to relevant food regions.

Finally, to improve the transparency of the developed models, an interpretability analysis based on Grad-CAM is applied to the best-performing model. This analysis allows visualization of the food regions activated by the network and verification that the model’s decisions are based on relevant visual features. Thus, this work provides a comprehensive evaluation combining segmentation performance, architectural analysis, and explainability of deep models for automatic understanding of food images.

2. Materials and Methods

This research offers a comparative evaluation of two deep learning architectures for the semantic segmentation of food images. The experimental framework compares a standard U-Net model with modified versions incorporating skip connections and fine-tuning, as well as a ResNet34-based segmentation model analyzed with and without fine-tuning. The primary objective is to conduct an ablation study measuring the impact of architectural modifications and transfer learning on segmentation performance, while keeping the training and evaluation pipeline consistent across all models. Model evaluation relies on metrics such as pixel accuracy, Dice/F1 score, mean intersection over union (mIoU), precision, recall, and loss. Following quantitative analysis, the best-performing model is selected for qualitative evaluation using Grad-CAM, which provides visual insights into the regions influencing the model’s predictions, as illustrated in Figure 1.

2.1. Data Preparation

The FoodSeg103 dataset [6], consists of a training set containing 4,983 RGB food images and a validation set including 2,135 images. Each sample includes the original image, its corresponding pixel-level semantic segmentation mask, the food categories present in the image, and a unique identifier.

The segmentation masks provide pixel-wise annotations for 103 food categories, enabling supervised training and evaluation of semantic segmentation models. Owing to the large number of classes, complex meal compositions, intra-class variability, and frequent occlusions between food items, FoodSeg103 constitutes a challenging benchmark for semantic food segmentation. Before training, the FoodSeg103 images and their corresponding annotations were preprocessed according to the requirements of the segmentation and multilabel classification tasks.

2.1.1. U-Net-Based Segmentation Models

The basic U-Net, U-Net optimized with skip connections (U-Net Skip), and the fine-tuned U-Net (U-Net FT) used the same preprocessing procedure. All RGB images were resized to a fixed resolution of $256\times 256$ pixels and normalized to the interval $[0,1]$ according to:

$$I_{norm}={\displaystyle \frac{I_{in}}{255}}$$

(1)

where $I_{in}$ denotes the original RGB image and $I_{norm}$ the normalized image provided as input to the network. The corresponding segmentation masks were resized using nearest neighbor interpolation to preserve the original class indices and avoid any modification of the pixel-level annotations. Thus, the same image and mask preprocessing procedure was consistently applied to the three U-Net-based models.

2.1.2. ResNet34-Based Semantic Segmentation

ResNet34 was employed as an encoder-based semantic segmentation model for pixel-level food image segmentation. The ResNet34-based segmentation model predicts a semantic class for each pixel of the input image, allowing the food regions to be explicitly delineated. The RGB input images were resized to $512\times 512$ pixels and normalized using the ImageNet channel-wise mean $\mu$ and standard deviation $\sigma$:

$$I_{\mathrm{norm}}={\displaystyle \frac{I_{\mathrm{in}}-\mu }{\sigma }}.$$

(2)

The corresponding segmentation masks were resized to the same $512\times 512$ spatial resolution using nearest-neighbor interpolation in order to preserve the original class indices and avoid the creation of invalid intermediate labels. Each pixel of the ground-truth mask therefore retains its original semantic class. Let $M\in {\{0,\dots ,C-1\}}^{H\times W}$ denote the ground-truth segmentation mask, where C represents the number of semantic classes. The ResNet34-based segmentation model produces a pixel-wise prediction

$$\widehat{M}=f_{\theta }\left(I\right),$$

(3)

where I is the input image and $f_{\theta }$ denotes the ResNet34-based encoder–decoder network. For each pixel $(i,j)$, the predicted semantic class is obtained from the maximum class probability:

$${\widehat{M}}_{i,j}=arg\underset{c}{max}P{(c\mid I)}_{i,j}.$$

(4)

The ResNet34 encoder extracts hierarchical visual features from the input image, while the segmentation decoder progressively restores the spatial resolution to generate a dense pixel-level prediction. The model is trained using the corresponding ground-truth segmentation masks. Two configurations were evaluated in the comparative study: a ResNet34-based segmentation model without fine-tuning and the same architecture with fine-tuning (ResNet34 + FT). This comparison was designed to assess the contribution of fine-tuning to semantic segmentation performance while keeping the training and evaluation pipeline consistent across both configurations. The resulting segmentation predictions were evaluated using the underluying performance metrics. The best-performing model was subsequently selected for qualitative analysis and Grad-CAM-based visual interpretation of the learned feature responses.

2.2. Deep Learning Architectures

2.2.1. Construction of the UNet Model

SimpleUNet Baseline

The proposed SimpleUNet model is a lightweight convolutional architecture designed for pixel-wise semantic segmentation of FoodSeg103 images. The network takes an RGB image tensor of dimension $8\times 3\times 256\times 256$, corresponding to a batch of eight images, and preserves the spatial resolution throughout the network. It produces a dense segmentation logit tensor of dimension $8\times 104\times 256\times 256$. Here, 104 corresponds to the 103 food categories and the background class. The architecture consists of three convolutional layers with ReLU activation functions. The first convolutional layer transforms the three RGB channels into 64 feature maps using $3\times 3$ kernels, with 1,792 trainable parameters. The second convolutional layer increases the feature depth from 64 to 128 channels using $3\times 3$ kernels and contains 73,856 trainable parameters. Finally, a $1\times 1$ convolutional layer performs pixel-wise semantic classification by projecting the 128 feature maps at each spatial location onto the 104 semantic classes. This final layer contains 13,416 trainable parameters.

The resulting output is not an image-level classification vector but a spatially dense prediction map. For each pixel location $(i,j)$, the network produces 104 class logits, corresponding to the 103 food categories and the background class. The predicted semantic label for each pixel is obtained as

$${\widehat{y}}_{i,j}=arg\underset{c\in \{0,\dots ,103\}}{max}{S}_{i,j,c},$$

(5)

where $S_{i,j,c}$ denotes the logit associated with class c at pixel location $(i,j)$.

The SimpleUNet model, Table 1, contains 89,064 trainable parameters and no non-trainable parameters. It was used as a lightweight baseline for semantic segmentation to investigate the contribution of subsequent architectural improvements, particularly the introduction of skip connections and fine-tuning.

Forward Propagation

the successive convolutional layers of the SimpleUNet model. The RGB image, with a spatial resolution of $256\times 256$ pixels and three input channels, is processed by a sequence of convolutional layers that extract increasingly rich feature representations while preserving the spatial resolution.

The first convolutional layer transforms the three input channels into 64 feature maps, followed by a ReLU activation. The second convolution increases the feature representation to 128 channels while maintaining the same spatial resolution. Finally, a $1\times 1$ convolution projects the 128-dimensional feature representation at each pixel onto the 104 semantic classes. Consequently, the network produces a dense pixel-wise segmentation output rather than an image-level classification vector.

The forward propagation process of SimpleUNet is summarized in Table 2, showing the evolution of tensor dimensions from the input image to the final segmentation output.

2.2.2. Optimisation of U-Net with Skip Connections

A U-Net architecture with skip connections was implemented for semantic segmentation on the FoodSeg103 dataset. The model was designed with an encoder–decoder structure in which feature maps extracted by the encoder are transferred to the corresponding decoder layers through skip connections, [7]. The encoder consists of convolutional blocks with progressively increasing numbers of filters, from 64 to 128 and 256 filters, followed by a bottleneck with 512 filters. The decoder progressively reconstructs the spatial resolution using transposed convolution layers with 512, 256, 128, and 64 filters. The final layer is a $1\times 1$ convolution producing 104 output channels corresponding to the segmentation classes. Figure 2 shows the configurations of the two segmentation architectures. The SimpleUNet configuration is presented on the left, while the U-Net architecture with skip connections is shown on the right. The figure highlights the main differences in the layer organization and the feature connections between the two architectures.

Mathematical Formulation of U-Net and Skip Connections

The U-Net architecture proposed by Ronneberger et al. relies on a contracting path for extracting contextual features and an expansive path for reconstructing spatial resolution, enabling precise localization of segmented regions [7]. In an encoder block, a convolution followed by a ReLU activation is expressed as:

$${\mathbf{F}}^{\left(l\right)}=ReLU\left({\mathbf{W}}^{\left(l\right)}*{\mathbf{F}}^{(l-1)}+{\mathbf{b}}^{\left(l\right)}\right),$$

(6)

where ${\mathbf{F}}^{(l-1)}$ is the feature map from the previous layer, ${\mathbf{W}}^{\left(l\right)}$ the convolution filters, ${\mathbf{b}}^{\left(l\right)}$ the bias, and $ReLU\left(x\right)=max(0,x)$ [29]. Each encoder block uses two successive convolutions before downsampling [7], performed via max-pooling:

$${\mathbf{P}}^{\left(l\right)}(i,j,c)=\underset{(m,n)\in \mathcal{N}}{max}{\mathbf{F}}^{\left(l\right)}(2i+m,2j+n,c),$$

(7)

where $\mathcal{N}$ is the pooling window (typically $2\times 2$) and c the channel.

The skip connections, central to U-Net, directly transmit high-resolution encoder features to the corresponding decoder levels, preserving fine spatial details lost during downsampling. With ${\mathbf{E}}^{\left(l\right)}$ the encoder features and ${\mathbf{U}}^{(l+1)}$ the upsampled decoder features, their fusion is given by:

$${\mathbf{C}}^{\left(l\right)}=Concat\left({\mathbf{E}}^{\left(l\right)},{\mathbf{U}}^{(l+1)}\right),$$

(8)

combining high-level semantic information with fine spatial detail [7]. In the decoder, resolution is increased via transposed convolution:

$${\mathbf{U}}^{\left(l\right)}=ConvTranspose\left({\mathbf{D}}^{(l+1)};{\mathbf{W}}_u^{\left(l\right)}\right),$$

(9)

after which features are refined by convolution:

$${\mathbf{D}}^{\left(l\right)}=ReLU\left({\mathbf{W}}_d^{\left(l\right)}*{\mathbf{C}}^{\left(l\right)}+{\mathbf{b}}_d^{\left(l\right)}\right).$$

(10)

The final layer projects features onto K semantic classes via a $1\times 1$ convolution followed by Softmax:

$${\mathbf{p}}_{i,k}={\displaystyle \frac{exp\left(z_{i,k}\right)}{{\sum }_{j=1}^{K}exp\left(z_{i,j}\right)}},$$

(11)

where $z_{i,k}$ is the logit for pixel i and class k. The predicted class is then:

$${\widehat{y}}_i=arg\underset{k\in \{1,\dots ,K\}}{max}{\mathbf{p}}_{i,k}.$$

(12)

Training relies on backpropagation, adjusting network parameters according to the error between predicted and reference segmentation. The pixel-wise cross-entropy loss is used:

$${\mathcal{L}}_{CE}=-{\displaystyle \frac{1}{N}}{\displaystyle \sum _{i=1}^N}{\displaystyle \sum _{k=1}^K}{y}_{i,k}log\left(p_{i,k}\right),$$

(13)

where N is the total number of pixels, K the number of classes, and $y_{i,k}$ the ground-truth value for pixel i and class k. Gradients are propagated from the output layer to preceding layers via the chain rule, and parameters are updated by gradient descent:

$${\mathbf{W}}_{t+1}^{\left(l\right)}={\mathbf{W}}_t^{\left(l\right)}-\eta {\displaystyle \frac{\partial {\mathcal{L}}_{CE}}{\partial {\mathbf{W}}_t^{\left(l\right)}}},$$

(14)

where $\eta$ is the learning rate and t the iteration index. This process is repeated across batches to progressively reduce the segmentation error. Among the mechanisms formalized above, skip connections play a particularly important role, as they preserve and reintroduce fine spatial information into the decoder that would otherwise be lost during successive downsampling operations [7].

2.2.3. Optimization of the U-Net Model with Residual Connections by Fine-Tuning

The U-Net model used in this study is based on an encoder-decoder architecture incorporating jump connections to preserve the spatial information necessary for accurate segmentation [7]. To improve the network’s learning capacity, residual connections were also integrated, following the principle of residual blocks introduced in ResNet, to facilitate gradient propagation and the extraction of more discriminating features [16]. During fine-tuning, the encoder layers were initially frozen, while the decoder and task-specific layers were optimized on the target dataset. The deeper encoder layers were subsequently unfrozen to allow further adaptation, Figure 3.

Mathematical Foundations of Fine-Tuning for U-Net with Skip Connections

In a second step, a fine-tuning is applied to adapt the parameters learned by the model to the specific characteristics of the FoodSeg103 dataset. Unlike training performed from randomly initialized weights, fine-tuning starts from parameters already learned and continues their optimization on the target segmentation task. The loss function used to guide this adaptation can be expressed as the pixel-to-pixel cross-entropy loss [13], as:

$${\mathcal{L}}_{FT}=-{\displaystyle \frac{1}{N}}{\displaystyle \sum _{i=1}^N}{\displaystyle \sum _{c=1}^C}{y}_{i,c}log\left({\widehat{y}}_{i,c}\right),$$

(15)

where N represents the number of pixels considered, C the number of classes, $y_{i,c}$ the actual value of pixel i for class c, and ${\widehat{y}}_{i,c}$ the corresponding predicted probability. The network parameters are then adjusted from the pre-trained weights according to:

$${\mathbf{W}}_{t+1}^{\left(l\right)}={\mathbf{W}}_t^{\left(l\right)}-{\eta }_{FT}{\displaystyle \frac{\partial {\mathcal{L}}_{FT}}{\partial {\mathbf{W}}_t^{\left(l\right)}}},$$

(16)

where ${\eta }_{FT}$ denotes the learning rate used during fine-tuning, generally chosen to be lower than that used during initial training, in order to preserve relevant representations already acquired while allowing their adaptation to the target domain [14]. In the case of U-Net with skip connections, this adaptation jointly concerns the encoder, the decoder, and the parameters associated with the skip connections. The features extracted by the encoder can thus be specialized for food images, while the decoder progressively refines the reconstruction of segmented regions. Fine-tuning therefore makes it possible to exploit previously learned representations while improving their fit to the specific distribution of the segmentation data considered.

2.2.4. Construction of the ResNet34 Model Before Fine-Tuning

The ResNet34 model used for multi-label classification in this study is initially based on an architecture pre-trained on ImageNet. It comprises a succession of four groups of residual blocks, composed respectively of 3, 4, 6, and 3 BasicBlocks, with an increasing number of filters from 64 to 512. After the residual blocks, an Adaptive Average Pooling operation produces a representation of dimension 512, which is then flattened and passed to a fully connected layer transforming the 512 features into 103 outputs corresponding to the FoodSeg103 categories. A sigmoid function is used to obtain the probabilities associated with the different food categories. These same encoder weights, learned during multi-label classification training, are subsequently reused as the encoder of the segmentation model during the testing phase and fine-tuned on the target segmentation task during the training phase Figure 4.

Table 3. Architecture of the ResNet34 model trained on the FoodSeg103 dataset for multi-label classification.

Block	Output Size	Description
Input Image	$3\times 512\times 512$	RGB image from the FoodSeg103 dataset
Initial Convolution	$64\times 256\times 256$	$7\times 7$ convolution, stride 2
Max Pooling	$64\times 128\times 128$	Spatial downsampling
Layer 1	$64\times 128\times 128$	3 residual blocks (BasicBlocks)
Layer 2	$128\times 64\times 64$	4 residual blocks (BasicBlocks)
Layer 3	$256\times 32\times 32$	6 residual blocks (BasicBlocks)
Layer 4	$512\times 16\times 16$	3 residual blocks (BasicBlocks)
Adaptive Average Pooling	$512\times 1\times 1$	Global feature extraction
Flatten	512	Feature vector
Fully Connected Layer	103	Multi-label classification
Sigmoid Activation	103 probabilities	Presence probability for each food category

Table 3. Architecture of the ResNet34 model trained on the FoodSeg103 dataset for multi-label classification.

Block	Output Size	Description
Input Image	$3\times 512\times 512$	RGB image from the FoodSeg103 dataset
Initial Convolution	$64\times 256\times 256$	$7\times 7$ convolution, stride 2
Max Pooling	$64\times 128\times 128$	Spatial downsampling
Layer 1	$64\times 128\times 128$	3 residual blocks (BasicBlocks)
Layer 2	$128\times 64\times 64$	4 residual blocks (BasicBlocks)
Layer 3	$256\times 32\times 32$	6 residual blocks (BasicBlocks)
Layer 4	$512\times 16\times 16$	3 residual blocks (BasicBlocks)
Adaptive Average Pooling	$512\times 1\times 1$	Global feature extraction
Flatten	512	Feature vector
Fully Connected Layer	103	Multi-label classification
Sigmoid Activation	103 probabilities	Presence probability for each food category

Mathematical Foundations for ResNet34 as an Encoder and Backpropagation

ResNet34 is used as the encoder to extract hierarchical representations from food images. ResNet introduces residual learning, where blocks learn a residual function relative to their input, easing the optimization of deep networks and improving gradient propagation [16].

For an input $\mathbf{x}$, a residual block outputs:

$$\mathbf{y}=\mathcal{F}(\mathbf{x},\mathbf{W})+\mathbf{x},$$

(17)

where $\mathcal{F}(\mathbf{x},\mathbf{W})$ is the learned residual transformation and $\mathbf{x}$ is passed directly via the shortcut connection [16]. When dimensions differ, a linear projection ensures compatibility:

$$\mathbf{y}=\mathcal{F}(\mathbf{x},\mathbf{W})+{\mathbf{W}}_s\mathbf{x},$$

(18)

with ${\mathbf{W}}_s$ the projection parameters [16].

Parameters are optimized via the segmentation loss $\mathcal{L}$, updated at iteration t as:

$${\mathbf{W}}_{t+1}^{\left(l\right)}={\mathbf{W}}_t^{\left(l\right)}-\eta {\displaystyle \frac{\partial \mathcal{L}}{\partial {\mathbf{W}}_t^{\left(l\right)}}},$$

(19)

where $\eta$ is the learning rate, following the backpropagation principle of Rumelhart et al. [15].

2.2.5. Construction of the ResNet34 Model After Fine-Tuning

After fine-tuning, the overall architecture of the ResNet34 model remains unchanged, with the same residual blocks, the same representation of 512 features, and the same classification layer $512\to 103$. The main modification concerns the configuration of trainable parameters: the convolutional backbone is frozen to preserve the learned visual representations, while only the classification layer is made trainable. The model thus retains its total of 21.3 million parameters, of which 52,839 parameters are trainable after the backbone is frozen.

Mathematical Formulation for Fine-Tuning of ResNet34

After initial training, fine-tuning is applied to adapt the pre-trained ResNet34 representations to the specific characteristics of the FoodSeg103 dataset, continuing optimization from previously learned parameters rather than a random initialization. Selected network parameters are again adjusted by backpropagation according to:

$${\mathbf{W}}_{t+1}^{\left(l\right)}={\mathbf{W}}_t^{\left(l\right)}-{\eta }_{FT}{\displaystyle \frac{\partial {\mathcal{L}}_{FT}}{\partial {\mathbf{W}}_t^{\left(l\right)}}},$$

(20)

where ${\eta }_{FT}$ is the fine-tuning learning rate and ${\mathcal{L}}_{FT}$ the loss computed on the target data. A lower learning rate than that of initial training is typically used to allow gradual adaptation while preserving previously learned visual characteristics.

For ResNet34 as an encoder, fine-tuning enables the gradual adaptation of representations across residual blocks, from low-level features such as contours and textures to higher-level semantic structures, which is particularly relevant when the target data distribution differs from the pre-training domain. The experimental analysis thus distinguishes ResNet34 before and after fine-tuning, allowing a direct evaluation of its contribution to segmentation performance.

2.3. Ablation Study

An ablation study was conducted to separately analyze the contribution of the main architectural components and the adaptation strategies used in the studied models. [20] For the U-Net architecture, three configurations were considered. The first corresponds to the reference U-Net, without hop connections and without fine-tuning. The second corresponds to the U-Net incorporating hop connections, but without fine-tuning. The third corresponds to the U-Net with hop connections and after fine-tuning.[7]. This organization allows us to separately evaluate the contribution of hop connections by comparing the first two configurations, and then the effect of fine-tuning by comparing the last two.

For ResNet34, two configurations were considered, corresponding respectively to the initial model and the model after fine-tuning. Comparing these two configurations allows us to evaluate the effect of adapting the model to the task of classifying food categories. In the fine-tuned configuration, the convolutional backbone is frozen while the classification layer is updated. All configurations are evaluated with the same experimental protocol and metrics to ensure a consistent comparison between the different variants.

2.4. Grad-CAM for Model Interpretability

To analyze the interpretability of the ResNet34 model’s predictions,

the Grad-CAM (Gradient-weighted Class Activation Mapping) method was used. Grad-CAM produces activation maps that locate image regions that contribute positively to a given prediction, by leveraging the gradients of the target class relative to the feature maps of a convolutional layer, [12]. This approach allows us to visualize the image areas primarily considered by the model during classification.

In this study, Grad-CAM is applied to the ResNet34 model to generate heat maps superimposed on the input images. These visualizations are used to provide a qualitative interpretation of the model’s decisions and to examine whether the highlighted regions correspond to the relevant visual characteristics of food categories.

2.4.1. Software and Computational Environment

The models were implemented in Python using the PyTorch framework. This framework was used for building the architectures, managing tensors, training, fine-tuning, and evaluating the models. The various image preprocessing and transformation operations were performed using libraries from the Python and PyTorch ecosystem.

The experiments were run on GPU-accelerated computing resources. Two NVIDIA Tesla T4 GPUs were used specifically for the different experimental phases. Additional GPU resources were also used when computational needs required it. The GPU environment was leveraged via CUDA to accelerate the computational operations necessary for deep learning.

For the analysis of the interpretability of the ResNet34 model, Grad-CAM (Gradient-weighted Class Activation Mapping) was integrated into the experimental environment to generate activation maps from the model’s gradients. These maps are used in the qualitative analysis of the predictions presented in the interpretability section.

2.5. Evaluation Metrics

2.5.1. Training Metrics

During the training phase, the loss function (Loss) and accuracy (Accuracy) were used to track the model’s learning progress. The loss function allows the evaluation of the gap between the model’s predictions and the reference annotations and to track the convergence of the optimization. Accuracy allows the tracking of the evolution of prediction performance over the different training epochs.

2.5.2. Evaluation Metrics on the Test Set

After training, the final models were evaluated on the independent test set. For segmentation models, performance was evaluated using pixel accuracy, mean intersection over union (mIoU), and Dice score. These metrics assess the proportion of correctly classified pixels, the overlap between predicted regions and reference annotations, and the similarity between the two segmentation maps. The loss function (Loss) and the accuracy were also calculated on the test data in order to evaluate the final performance of the models on data not used during parameter optimization.

2.6. Data Pipeline Orchestration for ResNet34 Fine-Tuning

For better understanding of the proposed fine-tuning procedure, the overall pipeline based on ResNet34 is illustrated in Figure 5. Three-stage experimental pipeline for ResNet34-based semantic segmentation, comprising (1) image–mask preprocessing and batch generation, (2) model training, fine-tuning, and best-model selection based on validation performance, and (3) evaluation of the selected model on the unseen test set through pixel-wise segmentation and quantitative performance metrics. To avoid replication, the same pipeline stages are applied to the U-Net with skip connections architecture, including its fine-tuning process, with the corresponding model-specific components adapted accordingly.

3. Results

This section presents the experimental results obtained with the improved U-Net model applied to the semantic segmentation of food categories in the FoodSeg103 dataset. The model’s performance is analyzed through learning evolution and quantitative evaluation across the test set.

3.1. Ablation Study: UNet Improvements

An ablation study was conducted to evaluate the contribution of each modification introduced into the baseline U-Net architecture, based on the comparison of training loss and accuracy over 50 epochs across configurations. Three configurations were compared: the standard U-Net, the Skip U-Net incorporating enhanced skip connections, and the fine-tuned Skip U-Net, the training results for the loss and the accuracy metrics through 50 epochs are reported in Figure 6.

The final performance of the optimized model was evaluated on the test set using commonly used semantic segmentation metrics, including accuracy, precision, mean intersection over union (mIoU), Dice score, and F1 score, Table 4. These metrics quantify the degree of overlap between the predicted masks and the reference annotations, [17].

The baseline U-Net achieved a pixel accuracy of 62.47%, with a mean IoU of 0.4116 and a Dice score of 0.4894. These results show that the conventional encoder-decoder architecture can learn semantic representations; however, the progressive downsampling operations may lead to the loss of fine spatial details, which affects the accuracy of object boundaries.

After introducing skip connections, the Skip U-Net significantly improved the segmentation performance, reaching 85.90% pixel accuracy, 0.7150 mean IoU, and 0.7899 Dice score. Compared with the baseline model, this corresponds to an increase of 23.43 percentage points in pixel accuracy, while the mean IoU and Dice score improved by 0.3034 and 0.3005, respectively. This improvement confirms the effectiveness of skip connections in recovering high-resolution features and preserving spatial details during the decoding process [7].

Following the fine-tuning stage, the optimized Skip U-Net achieved the best performance, with 94.45% pixel accuracy, 0.8626 mean IoU, and 0.9135 Dice score. Compared with the model before fine-tuning, the optimized model improved pixel accuracy by 8.55 percentage points, mean IoU by 0.1476, and Dice score by 0.1236. This improvement demonstrates that fine-tuning enables the adaptation of previously learned parameters to the target domain, [14], improving generalization and reducing segmentation errors.

The final results demonstrate that both architectural modification through skip connections and optimization through fine-tuning contribute to improving the segmentation quality. The evaluation using IoU and Dice metrics follows common practices in semantic segmentation evaluation, where overlap-based metrics are used to measure the agreement between predicted masks and ground-truth annotations [17]. The different scores are reported in Table 4.

3.1.1. Ablation Study for Adapting the ResNet34 Model to the FoodSeg103 Domain

To evaluate the impact of fine-tuning on the performance of the ResNet34 model, an ablation study was conducted on the FoodSeg103 dataset, comparing the base model with a fine-tuned version. The reported results in Figure 7, show how fine-tuning affects the training dynamics of the architecture. Without fine-tuning, the model exhibit a progressive improvement as the number of epochs increases toward 50, neither the tendency of the training loss nor that of the accuracy shows a noticeable slope. During this training phase, it can be observed that the fine-tuned Skip U-Net achieves training performance relatively close to that of ResNet34, despite the substantial difference in their initial training trajectories, suggesting that fine-tuning allows U-Net to progressively narrow the gap with the already high-performing ResNet34 model, Figure 8.

In the initial configuration, ResNet34 achieves an overall accuracy of 97.27%, with an F1 score of 62.53%, a mean IoU of 0.4549, and a Dice Score of 0.6253. After adjusting the classifier parameters on the FoodSeg103 data, the fine-tuned model showed a slight improvement in performance, with an Accuracy of 97.28%, a Recall increasing from 51.40% to 52.27%, an F1 score reaching 63.05%, and an increase in the Mean IoU and Dice Score to 0.4604 and 0.6305, respectively. These results demonstrate that fine-tuning allows for better adaptation of the model to the specific characteristics of the food domain, notably an increased ability to detect existing classes, while maintaining the robustness of the representations extracted by the ResNet34 backbone. The different scores are reported in Table 5.

3.2. Qualitative Visualization of ResNet34 Predictions

To complement the quantitative evaluation, qualitative visualizations were generated to examine the behavior of the ResNet34-based segmentation model before and after fine-tuning. Figure 9 presents ten representative food images together with their corresponding segmentation masks, obtained before fine-tuning, providing a visual representation of the regions identified by the model. Overall, the different food ingredients are well distinguished by their respective masks. However, some fine details are occasionally missed, such as the whipped cream inside a roll, which does not appear in the predicted mask. Conversely, in other cases, a partially filled glass is entirely colored by the model, as if it corresponded to a solid piece of meat. Such inconsistencies motivate the use of Grad-CAM to better understand the underlying reasoning behind these predictions.

After fine-tuning, Grad-CAM was additionally applied to visualize the spatial regions contributing most strongly to the model predictions. The resulting heatmaps highlight the areas receiving higher activation from the network, while their superposition onto the original images facilitates the visual interpretation of these activations in the context of the food images, Figure 10. The comparison shows how the selected model distributes its discriminative responses across the image and provides a complementary qualitative view of the model behavior beyond the quantitative segmentation metrics. The Grad-CAM analysis shows a color gradation from yellow to red over the food content of the plate, while blue tones appear outside the plate, indicating that the model correctly attends to food-relevant areas in the vast majority of cases.

4. Discussion

4.1. Impact of Architectural Improvements: Ablation Study

4.2. UNet-Model Architecture Analysis vs UNet with Skip Connections

To evaluate the impact of the various architectural improvements, an ablation study was conducted by comparing the reference convolutional model with the improved U-Net architectures.

Although initially named SimpleUNet, the first model used in this study does not implement the original U-Net encoder-decoder topology proposed by Ronneberger et al. (2015), [7]. Indeed, it contains neither spatial reduction operations by pooling, nor oversampling layers, nor jump connections between the encoding and decoding levels. Rather, it corresponds to a lightweight fully convolutional segmentation network used as a baseline model. In this study, it is therefore considered a Simple Fully Convolutional Network (Simple-FCN). Comparative analysis with U-Net architectures highlights the importance of skip connections in semantic segmentation tasks. Unlike a simple convolutional architecture, where detailed spatial information can be progressively lost during feature extraction, U-Net allows the high-resolution representations from the first layers of the encoder to be transferred directly to the corresponding layers of the decoder.

These connections facilitate the merging of the deep contextual information extracted by the encoder with the fine spatial details necessary for accurate contour reconstruction. This property is particularly important for segmenting food images, where objects may exhibit irregular shapes, similar textures, and low-contrast boundaries.

The results obtained after 50 training epochs confirm that the introduction of an encoder-decoder architecture with hop connections progressively improves segmentation performance compared to the Simple-FCN model. The observed improvement in segmentation metrics (Accuracy, Precision, Recall, Dice Score, and IoU) demonstrates the effectiveness of multi-scale feature fusion in preserving spatial information during the reconstruction process. This ablation study thus confirms that the performance gains do not stem solely from increased network depth, but primarily from the ability of U-Net architectures to efficiently combine global semantic features and the local details necessary for accurate pixel-by-pixel segmentation.

4.3. Optimization of the U-Net Model with Residual Connections by Fine-Tuning

For this U-Net model used, residual connections were also integrated, following the principle of residual blocks introduced in ResNet, to facilitate gradient propagation and the extraction of more discriminating features [16].

After an initial training phase, a fine-tuning step was applied using the already learned weights. This strategy allows the network parameters to be progressively adjusted to the specific characteristics of the food domain, without restarting the training from a random initialization. During this phase, the model continues to optimize the already extracted representations to improve the quality of the segmented masks. The gradual decrease in the loss function and the improvement in segmentation performance demonstrate that fine-tuning allows for a more precise adaptation of the network to the visual variations present in FoodSeg103.

4.4. Discussion Results Through Metrics

The improvements observed primarily concern better localization of regions belonging to different food categories, confirmed by the increase in the mIoU score, as well as a reduction in pixel-by-pixel classification errors, confirmed by the increase in the accuracy score, calculated from the pixel-by-pixel annotations available in FoodSeg103.

Since Dice and mIoU (mean intersection over union) scores are, according to recent literature, relatively insensitive to errors at the boundaries of segmented masks, particularly for large objects, [18], the improvement in the accuracy of segmented object contours cannot be validly inferred from the overall increase in these scores. A rigorous quantification of this improvement would require a dedicated boundary metric, such as Boundary IoU, [18], which was not calculated in this study.

Finally, the improvement attributed to better use of spatial information thanks to residual connections, [16], is based on the results of the ablation study: the configuration integrating these connections obtains higher scores than the equivalent configuration without residual connections, which allows us to attribute this improvement to the architectural mechanism considered, the latter constituting the only difference between the two configurations compared.

4.5. Discussion Resnet vs Unet

The results obtained show that segmentation performance depends not only on the depth or complexity of the network, but also on its ability to preserve and utilize fine spatial information. In our experiment on FoodSeg103, the U-Net model with skip connections and fine-tuning achieves a higher mIoU than that obtained with the ResNet34 encoder. This difference can be explained by the essential role of skip connections, which allow features extracted at shallow encoder levels to be transferred directly to the decoder. This information includes details about the contours, shapes, and precise location of food regions, which are particularly important for pixel-by-pixel segmentation. Conversely, although ResNet34 provides deeper and more discriminating semantic representations thanks to its residual structure, successive convolution and downsampling operations can lead to a loss of fine spatial information. Thus, a deeper architecture does not necessarily guarantee better segmentation. Fine-tuning also plays an important role, as it allows the learned representations to be adapted to the specific characteristics of FoodSeg103. The improvement in mIoU observed after fine-tuning therefore indicates that adapting the parameters to the domain data is crucial for improving segmentation accuracy. These results suggest that, for this type of data, preserving spatial detail and adapting the model to the domain may be more decisive than simply increasing the backbone depth. However, these results should be considered in light of the experimental protocol, particularly the fine-tuning parameters, the learning rate, the number of epochs, the pre-training weights, and the decoder architecture, to ensure a fair comparison between the different architectures.

Table 6. Overview of representative food image analysis studies, including their tasks, methodologies, and contributions.

Table 6. Overview of representative food image analysis studies, including their tasks, methodologies, and contributions.

4.6. Discussion on the Trade-Off Between Performance, Complexity, and Interpretability

4.6.1. Analysis of the Computational Complexity of the Simple-FCN Model

The architectural simplicity of the SimpleUNet model, considered in this study as a lightweight, fully convolutional network (Simple-FCN), significantly reduces the number of parameters and the hardware resource requirements. The estimated computational cost is 46.70 GMAC, with an approximate memory consumption of 1.25 GB during training and inference.

This low complexity represents a significant advantage for applications requiring fast execution or deployment on resource-constrained platforms. However, this reduced complexity also limits the model’s ability to learn complex representations, particularly in food scenes containing multiple categories with similar textures and significant spatial interactions.

Thus, the Simple-FCN model provides a relevant baseline for evaluating the contribution of more advanced architectures incorporating multi-scale fusion mechanisms, learning transfer, or pre-trained encoders.

4.6.2. Impact of Architectural Improvements on the Quality of Segmentation: Ablation Study

U-Net architectures rely on an efficient encoder-decoder structure with jump connections that combine deep semantic features with fine spatial information. This strategy, initially proposed by Ronneberger et al. [7], Pixel-level segmentation has become a fundamental task in computer vision, enabling precise localization and delineation of objects within images [27].

In this study, several U-Net-based variants were evaluated, including classic U-Net, U-Net with optimized jump connections, and U-Net with fine tuning. Jump connections allow the preservation of spatial details lost during encoding by directly transferring high-resolution features to the decoding layers.

This property is particularly important for food images, where edge accuracy and separation between visually similar regions directly influence segmentation quality.

The ablation study results thus allow us to identify the actual contribution of each architectural improvement and confirm the value of hierarchical feature blending strategies.

4.6.3. Comparison Between CNN Classification and Segmentation Architectures

The experimental results highlight the complementary roles of convolutional neural network architectures for food image analysis. ResNet34 and U-Net are both based on convolutional operations; however, they address different learning objectives.

ResNet34 is primarily designed for image classification tasks. Through its deep residual blocks, it learns hierarchical visual representations by progressively extracting low- and high-level features from input images. The residual learning strategy introduced by He et al. enables efficient training of deep networks by improving gradient propagation and feature reuse [16].

In contrast, U-Net is specifically designed for pixel-level semantic segmentation. Its encoder-decoder structure combined with skip connections allows the network to preserve spatial information while recovering detailed object boundaries during the decoding process [7].

Compared with classification-oriented CNN models, U-Net provides a more precise localization capability because it generates pixel-wise predictions rather than a single class label. This characteristic is particularly important for food segmentation scenarios, where different ingredients may exhibit similar visual patterns but require accurate separation at the pixel level.

Recent food segmentation methods have explored alternative strategies to address this challenge. For instance, MVEANet [34] relies on a Transformer-based backbone (STViT) combined with a SAM-based decoder to achieve state-of-the-art segmentation accuracy, particularly in delineating fine-grained food boundaries. However, this comes at the cost of increased model complexity and inference time. In contrast, Muñoz et al. [33] proposed a lightweight CNN-based alternative by replacing the DeepLabv3+ backbone with EfficientNet-B1, achieving competitive segmentation performance while significantly reducing computational cost. These contrasting strategies illustrate the broader trade-off between Transformer-based and CNN-based architectures in food segmentation: the former favors global attention-driven accuracy, while the latter favors efficiency and deployability.

Building on this CNN-based direction, ResNet34 is commonly adopted as a classification backbone due to its residual connections, which mitigate the vanishing gradient problem and enable effective extraction of discriminative, high-level image features. Therefore, ResNet34 is effective for learning discriminative image-level features and category recognition, whereas U-Net-based architectures are more suitable for detailed food region delineation. The combination of deep feature extraction and spatial localization represents an effective strategy for intelligent food image analysis.

4.6.4. Interpretability Analysis by Grad-CAM

To improve the interpretation of the model’s decisions, a complementary analysis based on ResNet34 and Grad-CAM was performed.

ResNet34, proposed by He et al. [16], introduces residual connections, enabling efficient learning of deep representations. The Grad-CAM maps proposed by Selvaraju et al. [12], allow visualization of image regions that strongly contribute to the network’s predictions.

This analysis provides a better understanding of the behavior of deep learning models. Interpretability has become essential for promoting the adoption of artificial intelligence systems, particularly in applications where decisions must be analyzed by human experts [26].

Discussion of Grad-CAM Visualizations

The Grad-CAM visualizations provide additional qualitative evidence regarding the effect of fine-tuning on the ResNet34-based segmentation model. The superimposed activation maps indicate that, after fine-tuning, the model focuses more strongly on image regions associated with the food content. This behavior suggests that the adaptation of the pretrained representations enables the encoder to capture features that are more relevant to the target food-segmentation task. In particular, the concentration of high-activation regions within or around the segmented food areas indicates a better alignment between the discriminative features learned by the network and the spatial regions of interest. This qualitative behavior is consistent with the improvement observed in the quantitative segmentation metrics after fine-tuning, suggesting that the adaptation of ResNet34 contributes not only to higher segmentation performance but also to more task-relevant visual representations. Nevertheless, Grad-CAM should be considered as a complementary interpretability analysis rather than direct evidence of segmentation accuracy, since the activation maps indicate regions contributing to the model prediction and do not constitute segmentation masks themselves. Overall, the combined quantitative and qualitative analyses support the benefit of fine-tuning ResNet34 for adapting the pretrained encoder to the FoodSeg103 domain.

4.6.5. Synthesis of the Trade-Off Between Performance, Complexity, and Interpretability

The overall study shows that no architecture offers an absolute advantage for all the criteria considered. Convolutional models based on U-Net offer an effective trade-off between segmentation accuracy, architectural complexity, and deployment capacity. Thanks to their encoder-decoder structure and their jump connections, these architectures make it possible to preserve fine spatial information while improving the reconstruction of segmented regions [7].

The ablation study showed that integrating jump connections significantly improves segmentation performance by facilitating the transmission of spatial features between the encoder and the decoder. Furthermore, the fine-tuning phase allows for the gradual adaptation of the learned parameters to the specific characteristics of the food domain, thus improving the generalizability of the model [14].

In parallel, ResNet34 provides an efficient representation of visual features thanks to its residual architecture, enabling robust extraction of discriminating information for food category classification. The use of residual connections facilitates the learning of deep learning networks by improving gradient propagation.

Grad-CAM analysis complements this evaluation by adding a dimension of interpretability. This method allows visualization of the image regions that contributed to the network’s decisions, thus improving the understanding of the behavior of deep learning models.

Thus, the results obtained show that the combination of a suitable segmentation architecture (U-Net), fine-tuning optimization, and explanatory analysis provides a relevant compromise between performance, computational cost, and interpretability for intelligent systems of automatic food recognition and segmentation.

5. Conclusion

In this work, we proposed and evaluated a comparative approach to two Deep Learning architectures for the semantic segmentation of food images, with particular emphasis on the contribution of skip connections, fine-tuning and interpretability by Grad-CAM. Overall, the experimental results demonstrate that the proposed architectural improvements provide measurable benefits for semantic food segmentation. The ablation study confirms the contribution of skip connections to the preservation of spatial information, while fine-tuning further improves the adaptation of the models to the FoodSeg103 domain. More specifically, the results reveal a differentiated effect of this fine-tuning process between the two models: fine-tuning substantially improves the U-Net with skip connections, whereas ResNet34, which converges more rapidly during training, benefits only marginally from fine-tuning. This contrast suggests that the effectiveness of fine-tuning is closely tied to each architecture’s initial learning dynamics, with U-Net’s slower initial convergence leaving considerably more room for improvement compared with ResNet34’s rapid early convergence.

The complementary Grad-CAM analysis further provides visual evidence of the regions contributing to the model predictions, supporting the interpretability of the learned representations. These findings highlight the added value of combining architectural design, fine-tuning, quantitative evaluation, and explainability rather than relying exclusively on model complexity or final segmentation scores.