A Survey of Image Segmentation for Industrial Applications with a Focus on Quality Control

1. Introduction

In modern industrial manufacturing, ensuring consistently high product quality is a key success factor. Quality control (QC) [1] and quality assurance (QA) [1] are necessary to detect production errors at an early stage, avoid rejects and meet customer requirements and legal specifications. Traditional manual inspection methods, such as visual inspections or measurements, reach their limits, especially in highly automated production environments. These methods are considered time-consuming, subjective and difficult to scale.

Automated processes are therefore becoming increasingly important. With the advance of digitalisation, computer vision is establishing itself as a key technology in industrial quality assurance. Camera systems and image processing algorithms enable the automatic inspection of products during or after production and reduce the need for manual inspection.

In particular, image segmentation methods, which analyse and classify image areas with pixel precision, offer great potential for the automated detection of manufacturing defects [2]. These include surface defects, dimensional deviations, cracks or breaks. Contemporary deep learning methodologies, especially convolutional neural networks (CNNs), have proven to be formidable instruments in this domain, since they adeptly extract picture information, exhibit resilience to interference, and can be readily tailored to various industrial contexts. Different types of segmentation enable precise detection of defects and reduce the amount of manual annotation required, in some cases through the use of synthetic training data.

Despite the increasing prevalence of deep learning-based segmentation methods, challenges remain in the areas of real-time capability, adaptability to different product variants, and minimisation of false alarms, especially in safety-critical scenarios. However, modern methods show that highly accurate and flexible segmentation is increasingly feasible even under demanding production conditions.

The aim of this thesis is to summarise current state-of-the-art (SOTA) methods of image segmentation for industrial quality assurance, evaluate their advantages and disadvantages, and compare the most relevant approaches based on defined criteria and typical application scenarios. Particular attention is paid to segmentation types, training strategies, and practical applications in industrial production processes.

2. Fundamentals of Image Segmentation

Image segmentation is a fundamental technique in computer vision, facilitating comprehensive analysis of visual data. Unlike pure object recognition, which solely identifies the existence and location of objects, segmentation offers pixel-level accuracy in defining structures within an image. This makes it an indispensable tool for industrial quality control systems, where even the smallest defects or material deviations must be reliably detected and localised. Depending on the objective, a distinction is made between semantic segmentation [3], instance segmentation [4] and panoptic segmentation [5]. The effectiveness of modern segmentation methods today relies heavily on deep learning architectures supported by large data sets and powerful frameworks. The following section therefore begins by presenting the most important software ecosystems that enable the efficient development and application of segmentation methods, before going on to describe key data sets that serve as the basis for training and evaluation.

2.1. Frameworks

Many validated open-source frameworks and specialised tools facilitate the deployment and effective use of image segmentation methods. They provide significant support in the creation, training and inference of segmentation models, thereby contributing to a considerable simplification of the development process. Well-maintained and versatile software ecosystems are particularly important in industry, where robustness, efficiency and adaptability to specific requirements are crucial.

PyTorch [6] and TensorFlow [7] are among the most important deep learning frameworks. Both offer extensive libraries for the development and training of neural networks that support both classic CNN architectures and modern transformer-based models [8]. Thanks to its dynamic computation graphs, PyTorch is particularly popular for research and experimental developments, as it offers very intuitive and flexible modelling options. TensorFlow, on the other hand, has advantages for industrial real-time applications, as it offers a stable production environment and enables the efficient execution of models on various hardware platforms (GPU, TPU, edge devices).

The Detectron2 framework [9], created by Facebook AI Research, is regarded as a premier tool in instance and semantic segmentation. It provides a modular and powerful platform on which different segmentation methods can be implemented. It impresses with its high flexibility, extensive documentation and numerous pre-trained models that can be used for domain-specific adaptations. It is used in particular for tasks that place high demands on segmentation accuracy or involve complex image content.

Another established tool is the Segment-Anything-Model (SAM) architecture from Meta AI [10]. This framework aims to provide a universal segmentation model that has been trained on a large amount of diverse image data. It enables both precise segmentation of individual objects and rapid adaptation to new application areas using prompting mechanisms.

In addition, there are specialised frameworks and tools for specific application areas of image segmentation, such as DeepMask and SharpMask [11,12]. These were developed by Facebook Research as Torch implementations for object detection and segmentation. MultiPath Network is also a Torch implementation of a network for object detection and segmentation based on the work A MultiPath Network for Object Detection [13].

OpenCV [14] is an important preprocessing tool for image data and an interface between camera systems and deep learning models. It offers basic and advanced image processing operations such as scaling and filtering. In addition, OpenCV can be easily integrated into existing industrial image processing systems, enabling efficient processing of image data for downstream segmentation models.

2.2. Datasets

In an industrial context, where specific requirements and error types must be taken into account, the accessibility and quality of datasets are crucial for the development and evaluation of image segmentation models. Industrial image datasets often contain a variety of defect types, different materials and varying image conditions. This results in special requirements for the robustness and generalisability of models.

In addition to such industry-specific datasets, generally available segmentation datasets are also used, especially in the pre-training phases of transfer learning [15]. Here, robust feature extractors are first trained on large, broadly annotated datasets before fine-tuning with domain-specific data. This significantly reduces the training effort and improves recognition performance, even when only a small amount of industrial data is available.

One of the most widely used general datasets is COCO (Common Objects in Context) [16], which, in addition to object recognition and keypoint detection, also provides extensive instance segmentation with more than 330,000 images and 80 categories. Another classic in computer vision is Pascal VOC [17], which is used for classification, object recognition and, in particular, semantic segmentation and comprises around 20,000 images with 20 classes. For scenes in urban areas, the Cityscapes dataset [18] plays a central role, as it contains both semantic and instance-based segmentations of street scenes, with 5,000 finely annotated images and 19 classes. LVIS (Large Vocabulary Instance Segmentation) [19] is an extension of COCO that contains significantly more classes and is particularly suitable for fine-grained instance segmentation.

2.3. Quality Control and Assurance in Industry

Industrial quality assurance [1] encompasses all organised and methodical measures necessary to ensure that products and manufacturing processes meet specified quality standards. These include both testing and preventive measures that can take place during production (inline testing) [20,21] or after completion of the manufacturing process (end-of-line testing) [22]. The aim is to detect defective products at an early stage, reduce waste and minimise the costs of rework or complaints. These measures ensure product quality, improve processes and increase customer satisfaction.

Automated inspection systems increasingly use technologies such as AI-based image processing or anomaly detection. Such systems can detect surface defects, dimensional deviations, missing components and other quality deviations with high precision and speed. However, conventional rule-based image processing systems often reach their limits when dealing with complex or variable visual patterns.

To overcome these limitations, companies are increasingly turning to segmentation-based models based on deep learning. These models offer greater flexibility and accuracy, especially when detecting subtle defects or deviations that are difficult to define using fixed rules. The following sections provide an overview of such models, their types and specific objectives in the context of industrial inspections.

The aforementioned principles, encompassing frameworks and pertinent data sets, constitute the foundation for comprehending the practical implementation of image segmentation models. Various segmentation-based deep learning models have been created to efficiently exploit these ideas in industrial applications. These differ in architecture, processing speed and detection accuracy and are crucial for overcoming specific challenges in automated quality assurance. The following section presents and compares the model types used in modern industrial applications.

2.4. Traditional Methods

Conventional image segmentation methods rely on traditional image processing techniques and statistical analysis. They do not usually require large amounts of data or extensive training processes like modern deep learning approaches, but instead use direct properties of image data such as intensity, colour or spatial relationships. One of the most important classic segmentation methods is presented below.

• Thresholding

Thresholding [23] is a fundamental and widely employed technique for image segmentation. It entails establishing one or more threshold values T to categorise picture pixels $I(x,y)$ into distinct classes, such as foreground and background. Simple global thresholding can be described mathematically as:

$$\begin{array}{c}\hfill S(x,y)=\left\{\begin{array}{cc}1\hfill & \mathrm{if}I(x,y)\ge T,\hfill \\ 0\hfill & \mathrm{if}I(x,y)<T.\hfill \end{array}\right.\end{array}$$

(1)

Local or adaptive thresholding methods take regional image differences into account and define the threshold value $T(x,y)$ depending on the pixel’s environment, e.g. by means of averaging or Gaussian weighting within a window:

$$\begin{array}{c}\hfill S(x,y)=\left\{\begin{array}{cc}1\hfill & \mathrm{if}I(x,y)\ge T(x,y),\hfill \\ 0\hfill & \mathrm{if}I(x,y)<T(x,y).\hfill \end{array}\right.\end{array}$$

(2)

• Region Growing

Region Growing [24] is an iterative segmentation-based approach. Starting from a seed point $s_i$, neighbouring pixels p are added to region $R_i$ if a similarity criterion $f(p,R_i)$ is satisfied, e.g. intensity difference:

$$\begin{array}{c}\hfill f(p,R_i)=|I\left(p\right)-{\mu }_{R_i}|<ϵ\end{array}$$

(3)

where ${\mu }_{R_i}$ is the mean intensity of region $R_i$ and $ϵ$ is a threshold value. The region grows until no further pixels satisfy the condition.

• Clustering

Clustering methods segment the image by grouping pixels into clusters $C_k$ based on a feature vector ${\mathbf{x}}_j$ (e.g., intensity, colour, or texture). In K-means [25], the sum of the squared distances is minimised:

$$\begin{array}{c}\hfill \underset{C_1,\dots ,C_K}{min}\sum _{k=1}^K\sum _{{\mathbf{x}}_j\in C_k}{\parallel {\mathbf{x}}_j-{\mu }_k\parallel }^2\end{array}$$

(4)

where ${\mu }_k$ is the mean value of the features in cluster $C_k$.

Fuzzy C-Means [26] generalises this by assigning each pixel a membership function $u_{jk}\in [0,1]$, and the cost function is:

$$\begin{array}{c}\hfill J_m=\sum _{j=1}^N\sum _{k=1}^K{u}_{jk}^m{\parallel {\mathbf{x}}_j-{\mu }_k\parallel }^2,\end{array}$$

(5)

with the fuzzification parameter $m>1$.

• Watershed Algorithm

The watershed algorithm [27] interprets the image intensities as topographical relief $I(x,y)$. Catchment areas $W_i$ are defined by letting ‘water’ flow into the landscape from the local minimum. The pixel assignment can be formally described by minimising the distance to the nearest minima:

$$\begin{array}{c}\hfill W_i=\{(x,y)\mid \mathrm{flowed}\mathrm{from}(x,y)\mathrm{to}\mathrm{minimum}{m}_i\}.\end{array}$$

(6)

The segmentation boundaries arise at the pixels where catchment areas meet. Marker-based variants additionally use manually or automatically defined starting points $M_i$ to control the region expansion.

3. Types of Image Segmentation

Image segmentation is a crucial procedure in the domains of computer vision and machine learning. The objective of segmentation is to partition an image $I:\Omega \subset {\mathbb{R}}^2\to {\mathbb{R}}^c$ into significant parts ${\left\{R_k\right\}}_{k=1}^K$ such that each region conveys pertinent information and facilitates a streamlined representation of the picture.

Formally, this can be described by partitioning the image domain $\Omega$:

$$\begin{array}{c}\hfill \Omega =\bigcup _{k=1}^K{R}_k,R_i\cap R_j=\varnothing \mathrm{for}i\ne j.\end{array}$$

(7)

3.1. Semantic Segmentation

Semantic segmentation [3] assigns a semantic class $c\in \{1,\dots ,C\}$ to each pixel $p\in \Omega$. This can be formally described as a classification function $f:\Omega \to \{1,\dots ,C\}$:

$$\begin{array}{c}\hfill f\left(p\right)=c,\forall p\in \Omega .\end{array}$$

(8)

For an image with multiple objects of the same class, all pixels $p_i$ are assigned to the same class c, regardless of which instance they belong to:

$$\begin{array}{c}\hfill f\left(p_i\right)=f\left(p_j\right)=c\mathrm{if}{p}_i,p_j\mathrm{belong}\mathrm{to}\mathrm{objects}\mathrm{of}\mathrm{the}\mathrm{same}\mathrm{class}.\end{array}$$

(9)

In neural networks, the function f is approximated by a parametrised model $f_{\theta }$ that predicts the class membership of each pixel based on its features $x_p$:

$$\begin{array}{c}\hfill {\widehat{c}}_p=f_{\theta }\left(x_p\right),x_p\in {\mathbb{R}}^d\end{array}$$

(10)

where ${\widehat{c}}_p$ is the predicted class for pixel p. Training is typically performed by minimising a loss function L, e.g. cross-entropy:

$$\begin{array}{c}\hfill \mathcal{L}\left(\theta \right)=-\sum _{p\in \Omega }\sum _{c=1}^C{y}_{p,c}log{\widehat{y}}_{p,c},\end{array}$$

(11)

where $y_{p,c}$ is the ground truth assignment and ${\widehat{y}}_{p,c}=\mathbb{P}({\widehat{c}}_p=c)$ is the predicted probability.

A summary of notable semantic segmentation methods is presented in Table 1.

• Fully Convolutional Networks (FCNs)

Fully Convolutional Networks (FCNs) [28] were the inaugural deep learning architectures composed exclusively of convolutional layers and tailored for semantic segmentation. Rather of employing fully connected layers at the conclusion of a traditional CNN, FCNs utilise transposed convolutions [28] to restore spatial resolutions. The model integrates semantic and geographical information by amalgamating characteristics from various depth levels.

• U-Net

The U-Net [29] was initially designed for biomedical image segmentation and is especially effective in situations with constrained training data. The architecture comprises a contractive pathway for context acquisition and a symmetric expanding pathway that facilitates accurate localisation. A defining aspect is the skip connections between contractive and expanding layers, which retain precise information.

• Efficient Neural Network (ENet)

In 2016, Paszke et al. presented ENet [30], a notably resource-efficient neural network designed for semantic segmentation. Unlike many previously developed architectures, ENet focused on real-time capability and efficiency, so that the model can also be used on devices with limited computing power, such as mobile platforms or embedded systems. The architecture employs an encoder-decoder framework while utilising considerably fewer parameters than similar approaches. This is achieved, among other things, through the use of asymmetric and factorised convolutions, early downsampling steps and bottleneck modules. Despite its low complexity, ENet delivers robust results in semantic segmentation, making it particularly attractive for time-critical applications such as autonomous driving.

• V-Net

In 2016, Milletari et al. [31] presented V-Net, an architectural framework founded on three-dimensional convolutional neural networks, designed explicitly for medical segmentation.In contrast to two-dimensional approaches, V-Net processes volumetric data directly in 3D, which allows spatial context information to be captured more effectively. The architecture is structured as an encoder-decoder network, with a symmetrical design with skip links that mitigate information loss during depth reduction. Another significant innovation of V-Net was the introduction of the Dice Loss function, which targets the segmentation result directly and is particularly suitable for datasets with highly unbalanced class distributions. This concept contributed significantly to improving the performance of neural networks in medical segmentation and served as the basis for many subsequent architectures.

• Efficient Residual Factorized Network (ERFNet)

In 2017, Romera et al. [32] presented ERFNet (Efficient Residual Factorized Network), an architecture for semantic segmentation that is specifically designed for efficient computation and real-time applications.ERFNet employs residual and factorised convolutional blocks to diminish model complexity while preserving the capacity to extract profound features. This combination enables the network to attain an optimal equilibrium between precision and processing resources, rendering it especially appealing for applications like autonomous driving or embedded devices. The architecture shows that targeted simplification and optimisation of convolution blocks can achieve high segmentation performance even with limited hardware resources.

• SegNet

SegNet [33] is another encoder-decoder network designed specifically for semantic segmentation tasks. Its distinctive feature is the use of max pooling indices from the encoder to efficiently reconstruct the feature maps in the decoder. This not only saves memory but also improves segmentation accuracy, as important structural information is retained during upsampling.

• Pyramid Scene Parsing Network (PSPNet)

In 2017, Zhao et al. [34] presented the Pyramid Scene Parsing Network (PSPNet), an architecture that adeptly leverages global scene context for semantic segmentation. The Pyramid Pooling module is a fundamental element of the model, aggregating characteristics at several scales to consider both local details and global contextual information. This multi-level pooling structure enables PSPNet to better understand complex scenes and reliably segment objects of different sizes and distances. The architecture is based on a deep ResNet backbone for feature extraction, complemented by the Pyramid Pooling strategy, and delivers state-of-the-art results on many semantic segmentation benchmark datasets.

• DeepLab

In 2017, Chen et al. [35] introduced DeepLab v1, an architecture for semantic image segmentation that attracted attention primarily through the use of dilated convolutions. This technique allows for the expansion of the filters’ receptive field without compromising the spatial precision of the feature maps, facilitating precise segmentation in high-resolution images. DeepLab employs an encoder-decoder methodology, wherein the encoder derives profound semantic characteristics and the decoder reconstructs the segmentation to the original image resolution. Employing this strategy, DeepLab attained notable enhancements in the precision of segmentation outcomes, particularly for objects of varying dimensions. The release of v1 laid the foundation for further versions, which introduced additional improvements such as Atrous Spatial Pyramid Pooling (ASPP) [35] and more effective decoder modules.

• Image Cascade Network (ICNet)

In 2018, Zhao et al. [36] introduced the Image Cascade Network (ICNet), an architecture for semantic segmentation optimised for real-time, high-resolution applications. The model follows a multi-stage cascade approach in which the input images are processed at different resolutions. Through the astute integration of these features, ICNet attains elevated segmentation precision while preserving a minimal computing burden. The architecture is especially appropriate for applications like autonomous driving or video processing, where real-time performance is essential for handling big image sizes.

• Bilateral Segmentation Network (BiSeNet)

In 2018, Yu et al. [37] introduced BiSeNet (Bilateral Segmentation Network), an architecture for semantic segmentation that enables high accuracy in real-time applications. The model combines two separate paths. One is the spatial path, which maintains spatial details, and the other is a context path, which extracts deep semantic information. This bilateral structure allows both fine edges and global context to be used efficiently. BiSeNet is characterised by low latency and efficient computation, making it particularly suitable for applications such as autonomous driving or mobile systems that require fast and accurate segmentation.

• Attention U-Net

In 2022, Zhu et al. [38] introduced Attention U-Net, a further development of the classic U-Net that is specially optimised for medical image segmentation. The central innovation consists of attention modules that are integrated into the skip connections of the encoder-decoder network. Those components allow the network to concentrate on relevant features and diminish extraneous information, thus enhancing segmentation precision, particularly for intricate or noisy images. By combining U-Net’s proven symmetric architecture with attention-based mechanisms, Attention U-Net can achieve more accurate segmentations without significantly increasing network complexity, making it particularly suitable for demanding medical applications.

• SEgmentation TRansformer (SETR)

SETR [39] is a pioneer among pure Transformer-based segmentation models. It substitutes the conventional CNN encoders with a purely Vision Transformer architecture that treats image patches as tokens, akin to natural language processing. The decoder architecture subsequently reconstructs the segmentation masks at high resolution. This architecture allows SETR to attain excellent performance, particularly for large-scale structures.

• Segmenter

In 2021, Strudel et al. [40] introduced Segmenter, a model that applies the concepts of Vision Transformers (ViT) [8] to semantic segmentation. Instead of classic convolutional neural networks, Segmenter uses patch-based Transformer encoders that effectively model the global dependencies in the image. A special decoder reconstructs the complete segmentation map from the patch embeddings. Thanks to its Transformer-based architecture, Segmenter can utilise long-range contextual information, which leads to more accurate segmentations, especially in complex scenes. The model shows that combining transformer mechanisms with segmentation tasks can achieve SOTA results, especially in scenarios where global image information is crucial.

• SegFormer

SegFormer [41] is a modern model for semantic image segmentation that combines the advantages of transformer architectures with efficient feature extraction mechanisms. SegFormer eliminates traditional positional encodings and employs a hierarchical transformer encoder architecture that captures information across various scales. This encompasses both global contextual information and intricate characteristics of the objects, resulting in enhanced segmentation accuracy, particularly in complicated scenarios. The model is distinguished by its low computing complexity and considerable adaptability, rendering it appropriate for diverse applications, including autonomous driving and medical picture processing.

Table 1. Overview: Semantic Segmentation Models.

Model	Year	Description
FCN (32s,16s,8s) [28]	2015	First fully convolutional network for pixel-wise segmentation
U-Net [29]	2015	Encoder-decoder with skip connections, biomedical focus
ENet [30]	2016	Lightweight real-time network
V-Net [31]	2016	3D extension of U-Net for volumetric data
ERFNet [32]	2017	Efficient residual factorized convs
SegNet [33]	2017	Encoder-decoder with pooling indices for efficient upsampling
PSPNet [34]	2017	Pyramid scene parsing, global context pooling
DeepLab v1 [35]	2017	Atrous convolutions, multi-scale context
ICNet [36]	2018	Cascade for real-time semantic segmentation
BiSeNet [37]	2018	Bilateral path for speed + accuracy balance
DeepLab v2–v4	2018–2020	Improved ASPP + encoder-decoder refinement
SETR [39]	2021	First pure Vision Transformer for segmentation
Segmenter [40]	2021	Transformer encoder + lightweight decoder
SegFormer [41]	2021	CNN-Transformer hybrid, efficient
Attention U-Net [38]	2022	U-Net with attention gates for better localization

Table 1. Overview: Semantic Segmentation Models.

Model	Year	Description
FCN (32s,16s,8s) [28]	2015	First fully convolutional network for pixel-wise segmentation
U-Net [29]	2015	Encoder-decoder with skip connections, biomedical focus
ENet [30]	2016	Lightweight real-time network
V-Net [31]	2016	3D extension of U-Net for volumetric data
ERFNet [32]	2017	Efficient residual factorized convs
SegNet [33]	2017	Encoder-decoder with pooling indices for efficient upsampling
PSPNet [34]	2017	Pyramid scene parsing, global context pooling
DeepLab v1 [35]	2017	Atrous convolutions, multi-scale context
ICNet [36]	2018	Cascade for real-time semantic segmentation
BiSeNet [37]	2018	Bilateral path for speed + accuracy balance
DeepLab v2–v4	2018–2020	Improved ASPP + encoder-decoder refinement
SETR [39]	2021	First pure Vision Transformer for segmentation
Segmenter [40]	2021	Transformer encoder + lightweight decoder
SegFormer [41]	2021	CNN-Transformer hybrid, efficient
Attention U-Net [38]	2022	U-Net with attention gates for better localization

3.2. Instance Segmentation

Unlike semantic segmentation, instance segmentation differentiates not only between semantic categories but also recognises specific object instances within each category [4]. Formally, let $I:\Omega \subset {\mathbb{R}}^2\to {\mathbb{R}}^c$ be an image, and let $\mathcal{C}=\{1,\dots ,C\}$ be the set of classes. In instance segmentation, each pixel $p\in \Omega$ is assigned a pair $(c,i)$:

$$\begin{array}{c}\hfill f\left(p\right)=(c_p,i_p),c_p\in \mathcal{C},i_p\in \mathbb{N},\end{array}$$

(12)

where $c_p$ represents the semantic class and $i_p$ represents the instance number within that class. This means that objects of the same class but different instances are assigned different masks:

$$\begin{array}{c}\hfill {\left\{R_{c,i}\right\}}_{i=1}^{N_c},R_{c,i}=\{p\in \Omega \mid f\left(p\right)=(c,i)\},\end{array}$$

(13)

where $N_c$ is the number of instances of class c.

Modern neural network architectures, such as Mask R-CNN, approximate the function $f_{\theta }$ parameterised by $\theta$ and combine object localisation (bounding boxes) with precise pixel mapping:

$$\begin{array}{c}\hfill {\widehat{f}}_{\theta }\left(p\right)=({\widehat{c}}_p,{\widehat{i}}_p),\forall p\in \Omega .\end{array}$$

(14)

The training loss function takes into account both the classification of the objects ($L_{\mathrm{cls}}$) and the segmentation accuracy of the mask ($L_{\mathrm{mask}}$):

$$\begin{array}{c}\hfill \mathcal{L}\left(\theta \right)=L_{\mathrm{cls}}+L_{\mathrm{box}}+L_{\mathrm{mask}}.\end{array}$$

(15)

Instance segmentation facilitates the accurate analysis and monitoring of distinct items, even when they are part of the same category. A summary of notable instance segmentation methodologies is presented in Table 2.

• R-CNN

R-CNN was developed by Girshick et al. [42] and represents one of the first approaches that proposes regions in the image and then classifies them with a CNN. The method first extracts region proposals using an external algorithm such as Selective Search and then processes each region individually through a CNN to assign classes and localise objects. R-CNN laid the foundation for modern object detection and segmentation models, but was very computationally intensive due to the sequential processing of regions.

• Fast R-CNN

Fast R-CNN [43] improved the original R-CNN architecture by applying feature extraction to the entire image and then using regions-of-interest (RoI) pooling to extract features for each region. This drastically reduces the computational effort while enabling end-to-end training. Fast R-CNN thus significantly accelerated detection without sacrificing accuracy.

• Sequential Grouping Networks (SGN)

Liu et al. [44] introduced Sequential Grouping Networks (SGN), an architecture that detects objects by sequentially grouping segmentation proposals. SGN systematically processes the proposed segments and consolidates pertinent regions to generate the final instance segmentation. This method enhances accuracy for significantly overlapped or fragmented entities.

• Mask - R-CNN

Mask R-CNN extends Faster R-CNN with an additional mask prediction branch [45], which creates a precise segmentation mask for each region. This architecture thus enables instance segmentation, i.e. the separation of individual objects within the same class. Mask R-CNN continues to use RoIAlign to increase the accuracy of the masks and combines detection and segmentation in an end-to-end trainable framework.

• PANet

The Path Aggregation Network (PANet) [46] was originally developed as an extension to Mask R-CNN to increase the accuracy of instance segmentation. PANet adds a bottom-up path to the Feature Pyramid Network (FPN) structure, which efficiently aggregates finer details from deeper network layers. In addition, an adaptive feature pooling method is introduced, which enables better mask prediction across different object sizes.

• MaskLab

Chen et al. [47] developed MaskLab, a framework for instance segmentation that combines classification, semantics and mask formation. MaskLab uses both region-based features and contextual information to precisely align masks with object boundaries. MaskLab demonstrates high accuracy, especially in complex scenes with overlaps.

• Cascade Mask R-CNN

Cascade Mask R-CNN [48] is an advancement of Mask R-CNN that addresses the problem of accuracy degradation when processing difficult objects. It connects several stages of detectors and mask predictions in series, with each stage building on the outputs of the previous stage. This cascading structure improves precision in both object detection and instance segmentation, as errors from earlier stages can be corrected in later stages.

• Hybrid Task Cascade (HTC)

Chen et al. [49] introduced Hybrid Task Cascade (HTC) in 2019, which extends the advantages of Cascade Mask R-CNN by more closely integrating detection and segmentation tasks. HTC combines multiple cascaded stages and uses cross-task feature fusion, allowing semantic information from mask segmentation to flow back into the detection pipeline. This further increases both object detection and segmentation accuracy. HTC is regarded as one of the most robust systems for instance segmentation on intricate datasets.

• You Only Look At CoefficienTs (YOLACT)

YOLOACT (You Only Look At CoefficienTs) [50] is a real-time instance segmentation model that offers significantly higher speed than previous models such as Mask R-CNN, while maintaining comparable accuracy. The model separates the generation of proto masks and the prediction of mask coefficients, enabling parallel processing. The final masks are then generated by a weighted combination of both components.

• Tensormask

In 2019, Chen et al. [51] introduced TensorMask, a model for instance segmentation that treats segmentation as a dense prediction of masks over a grid of tensors. Unlike classical Mask R-CNN-based approaches, which generate masks per region, TensorMask generates masks for the entire image in a single pass, directly mapping the spatial structure of the objects into the tensors. This enables more accurate and consistent mask generation, especially for closely spaced or overlapping objects. TensorMask thus combines the advantages of dense predictions with modern deep learning architectures for instance segmentation.

• Segmenting Objects by Locations (SOLO)

SOLO (Segmenting Objects by Locations) [52] is an instance segmentation approach based on an anchor-free concept. The model divides an image into a regular grid, with each grid point tasked with generating a mask for an object in its respective region. Unlike classical methods, SOLO does not require separate region proposals or RoI operations, making the architecture simpler and more efficient while achieving precise segmentation of individual objects.

• Segmenting Objects by Locations (SOLOv2)

SOLOv2 (Segmenting Objects by Locations) is a further development of SOLO and pursues an anchor-free approach to instance segmentation [53]. The model divides the image into a regular grid and assigns each grid point the task of generating a mask for an object within its region. SOLOv2 improves both accuracy and efficiency over its predecessor, including through more dynamic feature assignments and improved mask prediction.

• Conditional Instance Segmentation (CondInst)

In 2020, Tian et al. [54] introduced CondInst (Conditional Instance Segmentation), a framework for instance segmentation that generates masks directly through conditional convolution filters that are generated individually for each object. This eliminates the need for classic RoI-based processing, enabling more efficient and flexible mask predictions. CondInst can generate masks for any number of objects simultaneously and demonstrates high accuracy while reducing computation time.

• DEtection TRansformer (DETR)

Carion et al. [55] developed DETR (DEtection TRansformer), which formulates object detection as an end-to-end transformer problem. DETR replaces classical region proposal methods with attention-based global image representations, enabling precise detection of objects in complex scenes. This model paved the way for transformer-based segmentation approaches.

• Deformable DETR

Zhu et al. [56] introduced Deformable DETR, an evolution of DETR that addresses the long convergence time and computational costs of the original model. Deformable DETR replaces the standard self-attention mechanisms with deformable attention, which focuses on a limited set of relevant key positions. This allows the model to train faster while delivering accurate object detection and segmentation in high-resolution images.

• Conditional DETR

Chen et al. [57] and Meng et al. [58] extended DETR with Conditional DETR, which improves convergence speed and enables more stable training processes. By conditioning on object prior information, the model can learn faster and more accurately, especially in complex scenes and with a large number of objects. The v2 version incorporates more optimisations and attains superior performance in several object detection and segmentation benchmarks.

3.3. Panoptic Segmentation

Panoptic segmentation [5] integrates the principles of semantic and instance segmentation. Formally, let $I:\Omega \subset {\mathbb{R}}^2\to {\mathbb{R}}^c$ be an image and $\mathcal{C}=\{1,\dots ,C\}$ the set of classes. For each pixel $p\in \Omega$, a pair $(c_p,i_p)$ is assigned:

$$\begin{array}{c}\hfill f\left(p\right)=(c_p,i_p),\end{array}$$

(16)

where

$$\begin{array}{c}\hfill i_p=\left\{\begin{array}{cc}0,\hfill & \mathrm{if}{c}_p\in \mathrm{stuff}\mathrm{classes},\hfill \\ 1,2,\dots ,N_{c_p},\hfill & \mathrm{if}{c}_p\in \mathrm{things}\mathrm{classes}\left(\mathrm{instances}\right).\hfill \end{array}\right.\end{array}$$

(17)

This simultaneously segments and classifies things (objects with instances) and stuff (flat regions without instances). The regions for instances and semantic classes can be written as:

$$\begin{array}{c}\hfill R_{c,i}=\{p\in \Omega \mid f\left(p\right)=(c,i)\},c\in \mathcal{C},i\in {\mathbb{N}}_0.\end{array}$$

(18)

Neural networks for panoptic segmentation, such as EfficientPS, approximate the function $f_{\theta }$ parameterised by $\theta$:

$$\begin{array}{c}\hfill {\widehat{f}}_{\theta }\left(p\right)=({\widehat{c}}_p,{\widehat{i}}_p),\forall p\in \Omega .\end{array}$$

(19)

The training loss function integrates both semantic and instantiation aspects:

$$\begin{array}{c}\hfill \mathcal{L}\left(\theta \right)=L_{\mathrm{sem}}+L_{\mathrm{inst}}+L_{\mathrm{consistency}},\end{array}$$

(20)

where $L_{\mathrm{sem}}$ evaluates the semantic classification, $L_{\mathrm{inst}}$ evaluates the instance masks, and $L_{\mathrm{consistency}}$ evaluates the consistency between the two components.

Panoptic segmentation thus enables a holistic and consistent interpretation of image content in complex scenes. An overview of representative panoptic approaches is provided in Table 3.

• Unified Panoptic Segmentation Network (UPSNet)

Xiong et al. [60] presented UPSNet (Unified Panoptic Segmentation Network), a methodology for panoptic segmentation that integrates semantic and instance information into a cohesive framework. UPSNet incorporates distinct branches for semantic and instance segmentation, with a unified loss function to consolidate the outcomes. This facilitates uniform panoptic segmentation while preserving elevated efficiency.

• Adaptive Instance Selection Network (AdaptIS)

Sofiiuk et al. [61] created AdaptIS, a framework for instance segmentation employing adaptive instance-aware convolutions. AdaptIS generates masks for individual objects without relying on RoI pooling, making the model more flexible and efficient with objects of different sizes.

• Efficient Panoptic Segmentation Network (EPSNet)

Chang et al. [62] presented EPSNet (Efficient Panoptic Segmentation Network), which executes panoptic segmentation in a resource-efficient and effective manner. EPSNet uses a unified backbone architecture for semantic and instantiation branches and performs optimised feature fusion. This allows EPSNet to achieve high accuracy with lower computational effort, making it particularly interesting for mobile and embedded systems.

• Fast Panoptic Segmentation Network (FPSNet)

De et al. [63] developed FPSNet (Fast Panoptic Segmentation Network), a framework for panoptic segmentation that is optimised for efficient processing and fast inference. FPSNet integrates semantic and instance data into a streamlined network, yielding robust panoptic segmentation outcomes while substantially decreasing calculation times, thereby rendering it appropriate for real-time applications.

• Panoptic-Deeplab

Cheng et al. [64] introduced Panoptic-DeepLab, which combines the advantages of DeepLab-based semantic segmentation with instance segmentation. The model employs atrous Spatial Pyramid Pooling (ASPP) [64] and decoder-based feature fusion to effectively handle both semantic and instance data. Panoptic-DeepLab produces strong outcomes in panoptic segmentation tasks.

• Efficient Panoptic Segmentation (Efficientps)

Mohan et al. [65] developed EfficientPS, an efficient framework for panoptic segmentation. It combines a lean backbone with separate branches for semantics, instances and panoptic fusion, achieving high accuracy with low computational load, which makes it particularly attractive for real-time applications.

• Mask2Former

Cheng et al. [66] presented Mask2Former, a cohesive transformer-based model that facilitates semantic, instance, and panoptic segmentation. MaskFormer conceptualises segmentation as a mask classification challenge, wherein each pixel is designated to a particular mask class. This architecture combines transformer-based global context modelling with flexible mask prediction and delivers SOTA performance on multiple segmentation benchmarks.

• Mask DINO

Li et al. [67] introduced Mask DINO, an advanced framework for instance and panoptic segmentation based on transformer architectures. Mask DINO extends the DINO detection pipeline with mask prediction, where each pixel is precisely assigned to an instance mask. By combining self-attentive mechanisms and dynamic mask classification, Mask DINO achieves high accuracy and robustness, especially in complex scenes with many overlapping objects. The model is considered a state-of-the-art solution for modern segmentation tasks.

• Video Panoptic Segmentation Network (VPSNet)

Wen et al. [68] introduced VPSNet (Video Panoptic Segmentation Network), a model explicitly engineered for panoptic segmentation in video content. VPSNet enhances traditional panoptic designs by incorporating temporal consistency and object tracking across frames, facilitating stable and coherent panoptic segmentation in video sequences. VPSNet is very advantageous for autonomous systems and video analysis.

3.4. Hybrid and Combined Models

These models cannot be strictly assigned to a single category, as they combine different approaches. The following section therefore presents methods that are neither exclusively semantic, instance-based nor panoptic. Particular attention is paid to models that combine different paradigms, for example by linking object recognition and segmentation, CNNs and transformers, prompt-based methods or generative approaches. An overview of representative hybrid approaches is provided in Table 4.

• Diffusion Network (Difnet)

DifNet is a diffusion-based model for image segmentation that aims to refine the boundaries of objects through iterative information propagation [69]. Initial segmentation estimates are diffused and adjusted over several steps, which improves segmentation accuracy, especially at object edges. DifNet is characterised by its ability to efficiently integrate both global context information and local details.

• SEG-YOLO

SEG-YOLO [70] is an extension of the well-known YOLO architecture [71], which combines real-time object detection with segmentation capability. The classic bounding box-based YOLO model is supplemented by additional segmentation heads, allowing masks to be generated directly for detected objects. As with object detection, YOLO-Seg aims for high speed with acceptable accuracy, making it particularly interesting for real-time applications.

• DeepLabCut

[72] Nath et al. [72] developed DeepLabCut, a tool for markerless animal pose estimation. It is based on deep learning segmentation and keypoint tracking techniques that enable the precise identification of body points in animals in videos. DeepLabCut is particularly useful in behavioural research and neuroscience, as it allows highly accurate analysis of movements without invasive markers.

• SegDiff

SegDiff [73] is a diffusion-based model for image segmentation that applies the principles of generative diffusion processes to the task of segmentation. Instead of directly predicting masks, SegDiff generates segmentation iteratively by reconstructing and refining noisy masks over several steps. This approach allows for particularly flexible and precise modelling of complex structures and fine details.

• Swin-Unet

Swin-Unet [74] integrates the U-Net architecture [29] with Swin transformers functioning as encoders. Swin transformers [75] utilise a hierarchical representation and sliding windows, facilitating the effective processing of both local and global image information. The recursive structure of U-Net is retained, allowing the architecture to benefit from both the advantages of skip connections and the powerful global context modelling provided by transformers.

• Segment Anything Model (SAM)

The Segment Anything Model (SAM) [10] was introduced by Meta AI and aims to create a universal model for segmenting arbitrary objects in arbitrary images, regardless of the object domain or class. It is based on a Transformer-based encoder-decoder architecture and uses prompts such as points, boxes, or masks as input. SAM is characterized by its ability to generalize to new, unseen data, making it particularly attractive for zero-shot and few-shot applications.

• FastSAM

FastSAM [76] is an optimized variant of the Segment Anything Model (SAM) that aims to achieve significantly faster inference times while maintaining segmentation accuracy. Through more efficient network architectures, reduction of redundant computations, and accelerated feature extraction, FastSAM can segment large images or video streams in real time. The model retains SAM’s flexibility to segment multiple objects using points, bounding boxes, or text instructions, making it particularly suitable for applications with high performance requirements such as robotics or interactive image processing.

• Diffumask

DiffuMask is another diffusion-based segmentation model based on the idea of iteratively reconstructing image masks from noisy estimates [77]. By gradually refining the masks, DiffuMask can reliably capture complex object shapes and fine structures. The model combines the advantages of probabilistic diffusion processes with modern deep learning architecture to achieve accurate and consistent segmentation results.

• Grounded-SAM

Grounded-SAM [78] extends the Segment Anything Model (SAM) with the ability to link segmented objects to semantic descriptions. While SAM is primarily specialized in generating object masks based on input points or bounding boxes, Grounded-SAM additionally allows segmentation to be specifically controlled by text or context information. This allows users to select specific objects in complex scenes without the need for separate training data for each class. This model combines the flexibility of SAM with targeted controllability through semantic inputs.

• PS-YOLO-seg

PS-YOLO-Seg [79] is an extension of the well-known YOLO architecture [71] for instance and object detection, which additionally integrates a segmentation component. The model combines YOLO’s fast and efficient object localization with precise mask predictions, so that objects are not only detected but also segmented with pixel-level accuracy. PS-YOLO-Seg is particularly suitable for applications that require both real-time performance and accurate segmentation, such as robotics or autonomous driving systems.

• GS-YOLO-Seg

[80] GS-YOLO-Seg builds on the basic principles of YOLO [71] and extends them with a segmentation-capable architecture similar to PS-YOLO-Seg [79], but with an additional focus on improving accuracy for highly overlapping objects. Through optimized feature aggregation and special mask modules, GS-YOLO-Seg can accurately segment instances even in complex scenes without compromising high processing speed. This model is especially appropriate for applications requiring the reliable differentiation of objects in dense settings.

4. Method Comparison: Advantages and Limitations

Image segmentation can be categorised into four primary groups: [semantic]semantic segmentation, [instance]instance segmentation, [panoptic]panoptic segmentation, and [hybrid]hybrid approaches. Each of these methods has specific advantages and disadvantages that depend on the respective use case, the available computing resources, and the requirements for speed and accuracy.

5. Application Examples in the Industry

Image segmentation and deep learning-based quality assurance are used in a wide range of industrial applications. In automotive production, surface inspections are performed on car body parts to automatically detect paint defects, scratches, or dents. In electronics manufacturing, segmentation-based algorithms enable the detection of solder joint defects, missing components, or short circuits on printed circuit boards. Computer vision is also increasingly being used in the food industry to check product shape, size, or surface quality, for example in fruits, baked goods, or packaged foods. Other examples include industrial textile production, where weaving errors such as broken threads or irregularities are automatically detected, and metal processing, where cracks, scratches, or dimensional deviations are reliably identified. These applications show that automated segmentation systems not only make quality assurance more efficient and consistent, but can also complement and, in many cases, replace human inspection tasks.

5.1. Common Requirements and Issues in Industrial Environments

The use of image segmentation systems in industrial quality assurance processes places special demands on the technologies used. In production lines, image processing must keep pace with the production speed or cycle time in order to ensure that the manufacturing process runs without delays. This often requires real-time processing, with the decisive criterion being compliance with the time specifications of the production system.

The system must also work reliably under difficult conditions, such as changing lighting conditions, contamination, reflections, or varying positions and orientations of the objects to be inspected. Image segmentation solutions should be flexibly adaptable to different inspection characteristics, product variants, and production conditions. Seamless integration into the existing production infrastructure is also necessary in order to communicate with the respective control systems, databases, and machines.

In addition, factors such as cost, maintenance effort, and the processing of large amounts of data play an important role. The selection of suitable image segmentation methods in an industrial environment therefore requires careful consideration of technical performance and practical feasibility.

6. Meta Analysis of Image Segmentation Methods

In order to evaluate the practical application of current image segmentation methods for industrial quality assurance tasks, a comparison based on relevant criteria is necessary. Depending on the application, different factors such as segmentation accuracy, processing speed, and hardware requirements are of crucial importance.

6.1. Search Strategy and Selection Criteria

A systematic literature search was performed in the scientific databases IEEE Xplore, ScienceDirect, SpringerLink, and the preprint platform arXiv to locate pertinent publications. This research aimed to locate contemporary studies on image segmentation techniques within the industrial domains of quality control and quality assurance, and to conduct a comparative analysis of their performance metrics.

Relevant search terms and keywords were specifically defined for the literature search in order to precisely narrow down the thematic focus. Among others, the keywords “image segmentation,” “quality control,” “industrial inspection,” “image segmentation quality control,” and combinations such as “semantic segmentation quality control industry,” “instance segmentation quality control manufacturing,” and “panoptic segmentation quality control production.” The search phrases were amalgamated utilising Boolean operators to attain an exact and thorough search outcome.

Explicit inclusion and exclusion criteria were delineated to guarantee the pertinence and integrity of the studies incorporated. Only peer-reviewed articles and high-quality preprints (e.g., from arXiv) from the publication period of 2018 to 2025 were considered in the evaluation. Peer-reviewed works were identified based on their publication in established, indexed journals or conferences. High-quality preprints were selected based on the reputation of the authors, citation frequency (where available), and recognition by the professional community.

Only papers with a clear reference to industrial quality control and assurance were considered. Studies without experimental performance data and papers dealing exclusively with non-industrial applications such as medical imaging or autonomous vehicles were excluded from the analysis. Review papers and publications without their own empirical investigations were also not considered. Table 5 provides an overview.

6.2. Qualitative Evaluation

The overview of selected works shows a wide variety of approaches to image segmentation in industrial quality control. Semantic segmentation dominates many applications, especially in the detection of surface defects or material deviations, as in [81,82,83,84,85,86]. The advantages of these methods lie primarily in the reduction of manual annotations, robustness with limited data sets, and the ability to obtain precise, pixel-accurate information about defects. Panoptic segmentation is used when both the class membership and the identity of individual objects are relevant, as in [87,88]. It enables the combination of semantic and instance information and reduces the effort required for manual annotations by using synthetic training data.

Instance-based segmentation is primarily used in high-precision applications such as microscopic analysis [89], 3D object recognition [90], and laser cutting control [92]. The advantage here is the precise localization of individual objects, even in cases of overlap or moving objects, which enables automated quality checks.

Hybrid approaches that combine multiple methods, such as Mask R-CNN with data augmentation [91] or CNN-based transfer learning [93], offer a good balance between accuracy and efficiency, especially in scenarios with small or heterogeneous datasets.

In general, qualitative analysis shows that the choice of segmentation method depends heavily on the use case. Semantic segmentation is suitable for general surface inspections, while instance and panoptic methods are suitable for applications that require precise object boundaries and individual objects, and hybrid approaches offer a flexible solution for complex industrial scenarios.

7. Discussion

The methods developed to date for industrial defect detection and quality assurance are largely based on convolutional neural networks (CNNs). These models have proven themselves in numerous applications because they extract image features locally, can be implemented efficiently, and offer high performance in the segmentation of surface defects, cracks, or corrosion on different materials. In particular, they enable real-time detection and classification of defects, which is of great importance for industrial production lines.

However, CNN-based methods reach their limits, especially when it comes to capturing global relationships in the image and processing complex, heterogeneous data. The rigid local recording of CNN filters makes it difficult to detect defects that are characterized by subtle changes or varying contextual information. Certain types of defects are also difficult to detect based on local features alone, which limits the accuracy and robustness of the models.

In this context, transformer-based architectures represent a promising advance. The self-attention mechanism makes it possible to effectively model global dependencies in image data and capture more complex relationships. This opens up new possibilities for more precise and versatile segmentation of errors. Studies show that transformer models are more robust to noise and variations in the data. Advances in the real-time capability of transformer-based methods also show that these powerful architectures are increasingly becoming practical for resource-constrained and time-critical applications.

An application-specific approach is recommended for practical industrial use. Mask R-CNN or DeepLabv3+ are suitable for high-precision offline or safety-critical inspections, while lightweight U-Net variants or YOLACT models are more suitable for real-time and inline inspections. Flexible or changing product scenarios benefit from the combination of object detection and segmenting models or, experimentally, from the use of universal models such as Segment Anything. Hybrid image processing systems that combine classic image analysis and deep learning to further increase robustness and adaptability are also a useful addition.

8. Conclusion and Outlook

The review of contemporary techniques for image segmentation in industrial quality control indicates that convolutional neural networks (CNNs) remain crucial. They provide exceptional precision in identifying and segmenting surface imperfections, fissures, and material discrepancies, and have been integrated into numerous applications. Weakly supervised and semi-supervised approaches reduce the annotation effort and enable the efficient use of small data sets. Semantic, instance, or panoptic/hybrid segmentation is utilised based on the application. Semantic segmentation is effective for flat faults, instance segmentation enables the recognition of distinct objects and overlapping components, and hybrid methodologies integrate both benefits to tackle more intricate situations. The use of synthetic data, transfer learning, and data augmentation further contributes to increased accuracy and robustness.

The rapid development in the field of computer vision and object segmentation opens up numerous possibilities for industrial quality assurance in the future. Resource-efficient models allow direct use on production lines without a cloud connection, while Explainable AI (XAI) [94] increasingly delivers comprehensible and interpretable segmentation results that are relevant for industrial quality standards. In addition, the expansion to 3D data and multi-sensor setups will improve the capture of complex structures, and no-code platforms [95] simplify the integration and adaptation of segmentation models even without in-depth AI knowledge. Zero- and few-shot learning approaches [96] reduce the training data requirements and enable the flexible use of universal models such as Segment Anything.

In summary, it can be said that the combination of these developments will further increase automation and process reliability in industrial quality assurance. Hybrid methods, optimized real-time applications, and targeted integration into industrial workflows will make industrial image segmentation more efficient, accurate, and practical in the future, with the focus remaining on balancing high performance and practical application.