CoSOV1Net: Cone- and Spatial-Opponency Primary Visual Cortex-Inspired Neural Network for Lightweight Salient Object Detection

1. Introduction

The human visual system (HVS) has the ability to select, among the large amount of information received, which is relevant and to process in detail only the relevant one. This relevant information in an image is called salient objects [1]. The salient object detection models in computer vision try to mimic this phenomenon by detecting and segmenting salient objects in images.

The salient object detection is an important task given its many applications in computer vision such as object tracking, recognition and detection [2] , advertisements optimization [3], image/video compression [4], image correction [5], analysis of iconographic illustrations [6], images retrieval [7], aesthetic evaluation [8], image quality evaluation [9], image retargeting [10], image editing [11], image collage [12], to name a few. Thus, it has been the subject of intensive research in recent years and is still being investigated [13].

Salient object detection models generally fall into two categories, namely conventional and deep learning-based models, which differ by the feature extraction process. The first use hand-crafted features, while the latter use features learned from a neural network. Thanks to the powerful representation learning methods, deep learning-based salient object detection models have recently shown superior performance over conventional models [13,14].

The high performance of deep learning-based salient object detection models is undeniable, however, they are also generally heavy if we consider their number of parameters and memory occupied in addition to their high computational cost and slow detection speed. This makes these models less practical for resource-limited vision sensors or mobile devices that have many constraints on their memory, computational capabilities and for real-time applications [15,16]. Hence, the need for lightweight salient object detection models, whose performance is comparable to state-of-the-art models, with the advantage of being deployed on resource-limited vision sensors or mobile devices and having a detection speed allowing them to be used in real time applications.

The authors of these existing models of lightweight salient object detection authors have used different methodologies such as imitating primate hierarchical visual perception [17], human attention mechanism [16,17], etc.

In this work, we propose an original approach for a new lightweight neural network model of salient object detection that can be therefore adapted to mobile environments and limited or resources-constrained devices with the following additional interesting properties of being able to be trained from scratch, without having to use backbones developed from image classification tasks and with few parameters but while having comparable performance with state-of-the-art models.

Given that detecting salient objects is a capability of the human visual system, and that a normal human visual system does this quickly and correctly, we used images or scenes encoding mechanism research advances in neuroscience especially for the human visual system early stage [18,19,20]. Our strategy in this model is therefore inspired by two neuroscience discoveries in human color perception, namely:

• the color-opponency encoding in the HVS (Human Visual System) early stage [21,22,23,24];
• the fact that the color and pattern are linked inextricably in human color perception [18,25].

Inspired by these neuroscience discoveries, we propose a Cone- and Spatial-Opponency Primary Visual cortex (CoSOV1) module that extracts the features at the spatial level and between the color channels at the same time to integrate color in the patterns. This process is applied first on opposing color pairs channels two by two and then to grouped feature maps through our deep neural network. Thus, based on the CoSOV1 module, we build a novel lightweight encoder-decoder deep neural network for salient object detection: CoSOV1Net. Our proposed neural network CoSOV1Net has only $1.14M$ parameters while having comparable performance with state-of-the-art salient object detection models. CoSOV1Net predicts salient maps on a speed of $4.4fps$ on an Intel CPU, i7-11700F, and $211.2fps$ on a NVIDIA Geforce RTX 3090 TI GPU for 384× 384 images; and has a low $FLOPS=1.4G$. Therefore, CoSOV1net is a lightweight salient object detection that can be adapted for mobile environments and limited resources devices.

Our contribution is threefold:

• we propose a novel approach to extract features from opposing color pairs in a neural network to exploit the strength of color opponent principle from human color perception. This approach permits to accelerate neural network learning;
• we propose a strategy to integrate color in patterns in a neural network by extracting features locally and between color channels at the same time in successively grouped feature maps that results in reducing the number of parameters and the depth of that neural network, while keeping good performance;
• we propose a lightweight salient object detection neural network architecture based on the proposed approach for learning opposing color pairs along with the strategy of integrating color in patterns. This lightweight salient object detection neural network has few parameters while having performance comparable to the state-of-the-art methods

The rest of this work is organized as follows: Section 2 presents some lightweight models related to this approach ; Section 3 presents our proposed lightweight salient object detection model; Section 4 describes the datasets used, evaluation metrics, our experimental results and the comparison of our model with state-of-the-art models; Section 5 discusses our results ; Section 6 concludes this work.

2. Related work

Many salient object detection models have been proposed and most of the influential advances in image-based salient object detection have been reviewed by Gupta et al. [13]. Herein, we present some conventional models and lightweight neural network models related to this approach.

2.1. Lightweight Salient Object Detection

In recent years, lightweight salient object detection models have been proposed with different strategies and architectures. Qin et al. [26] designed $U^{2}net$ a lightweight salient object detection with a two-level nested Unet[27] neural network able to capture more contextual information from different scales thanks to the mixture of receptive fields of different sizes. Other models are based on streamlined architecture to build lightweight deep neural networks. MobileNets [28,29] and ShuffleNets [30,31] with their variants are among the latter models. MobileNets [28] uses architecture based on depth-wise separable convolutions. ShuffleNets [30] used architecture based on point-wise group convolution and channel shuffle and depth-wise convolution to greatly reduce computation cost while maintaining accuracy. Other authors have been inspired by primate or human visual system process. Thus, Liu et al. [17] designed the HVPNet, a lightweight salient object detection network, based on a hierarchical visual perception (HVP) module which mimic the primate visual cortex for hierarchical perception learning. Liu et al. [16] were inspired by human perception attention mechanism in designing SAMNet a lightweight salient object based on a stereoscopically attentive multiscale (SAM) module that adopts a stereoscopic attention mechanism for effective and efficient multi-scale learning.

2.2. Opponent Color models

The color opponency, which is a human color perception propriety, has inspired many authors who have defined channels or feature maps to tackle their image processing tasks. Frintrop et al. [32] used three opponent channels $RG$, $BY$ and I to extract features for their salient object detection model.

To extract features for salient object detection, Ndayikengurukiye et Mignotte [1], used nine (9) opponent channels for RGB color space (RR: Red-Red, RG: Red-Green, RB: Red-Blue, GR: Green-Red, GG: Green-Green, GB: Green-Blue, BR: Blue-Red, BG: Blue-Green and BB: Blue-Blue) with a non linear combination thanks to the OCLTP (opponent color local ternary pattern) texture descriptor which is an extension of the OCLBP (opponent color local binary pattern)[33,34] and Fastmap [35] which is a fast version of MDS (Multi-dimensional Scaling).

Most authors apply the opponent color mechanism to the input image color space channels and not on the resulting feature maps. However, Jain et Healey [36] used opponent features computed from Gabor filter outputs. These authors compute opponent features by combining information across different spectral bands at different scales obtained by Gabor filters for color texture recognition [36]. Yang et al. [37] proposed a framework based on the color-opponent mechanisms of color-sensitive double-opponent (DO) cells in the human visual system primary visual cortex (V1) in order to combine brightness and color to maximize the boundary detection reliability in natural scenes.

In this work, we propose a model inspired by the human visual system but different from other models because our model uses the primary visual cortex (V1) cone- and spatial-opponency principle to extract features at channels spatial level and between color channels at the same time to integrate color into patterns in a manner allowing lightweight deep neural network design.

3. Materials and Methods

3.1. Introduction

Our model for tackling the lightweight salient object detection challenge is inspired by the human visual system (HVS) early visual color process especially its cone-opponency and spatial-opponency in the primary visual cortex (V1). Indeed, the human eye retina (the retina is located in the inner surface of the human eye) has two types of photoreceptors namely rods and cones. Rods are responsible for monochromatic vision under low levels of illumination while cones are responsible for color vision at normal levels of illumination. There are three classes of cones: the L, M and S. When light is absorbed by cone photoreceptors, the L cones absorbs long wavelength visible light, the M cones the middle ones and the S cones the short wavelength [22,23,25].

The cone signals are then processed by single-opponent retina ganglion cells. The single-opponent operates an antagonistic comparison of the cone signals [21,23,24,38]:

• L - M opponent for Red - Green;
• S - (L + M) opponent for Blue - Yellow.

The Red - Green as well as the Blue - Yellow signals are carried by specific cells (cells for Red - Green and different cells for Blue - Yellow) through the lateral geniculate nucleus (LGN) to primary visual cortex (V1).

Shapley [25] and Shapley et Hawken [18] showed that the primary visual cortex (V1) plays an important role in color perception through the combined activity of two kinds of color-sensitive cortical neurons namely single-opponent and double-opponent cells. Single-opponent cells in V1 operate in the same manner as those of retina ganglion cells and could provide neuronal signals that could be used for estimating the color of the illumination [25]. Double-opponent cells in V1 compare cone signals across space as well as between cones [19,20,22,25]. Double-opponent thus have two oppenencies: spatial-opponency and cone-opponency. These properties permit them to be sensitive to color edges and color spatial patterns. They are thus able to link color and pattern inextricably in human color perception [18,25].

As the primary visual cortex (V1) is known to play a major role in visual color perception as highlighted above, we propose in this work a deep neural network based on the primary visual cortex (V1) to tackle lightweight salient object detection challenge. We especially use two neuroscience discoveries in human color perception, namely:

• the color-opponency encoding in the HVS early stage;
• the fact that the color and pattern are linked inextricably in human color perception.

These two discoveries in neuroscience inspired us to design a neural network architecture for lightweight salient object detection, which hinges on two main ideas. First, at the beginning of the neural network, our model opposes color channels two by two by grouping them (R-R, R-G, R-B, G-G, G-B, B-B) and extract from each channels pair the features at the channels spatial level and between the color channels at the same time to integrate color in patterns. So, instead of doing a subtractive comparison or an OCLTP (opponent color linear ternary pattern) like Ndayikengurukiye et Mignotte [1], we let the neural network learn the features that represent the comparison of the two color pairs. Second, this idea of grouping and then extracting the features at the channels spatial level and between the color channels at the same time is applied on feature maps at each neural network level until the saliency maps are obtained. This process allows the proposed model to mimic the human visual system capability of linking inextricably color and pattern in color perception [18,25].

It is this idea that differentiates our model from other models that use depth-wise convolution first and after point-wise convolution [28,29] to extract features at each individual color channel level (or feature map) first not through a group of color channels (or feature maps) at same time as our model does. This idea differentiates also our model from models that combine a group of color channels (or feature maps) pixel by pixel first and apply depth-wise convolution after[30,31]. The idea of grouping color channels in pairs (or feature maps groups) differentiates our model from models that, while extracting features at color channels spatial level and between color channels at the same time, consider all color channels (or feature maps) as a single group.

Our model takes into account non-linearities in the image at the beginning as well as through our neural network. For this purpose, we use an encoder-decoder neural network type whose core is a module that we called CoSOV1 (Cone- and Spatial-Opponency Primary Visual Cortex) module.

3.2. CoSOV1 : Cone- and Spatial-Opponency Primary Visual Cortex module

The CoSOV1 (Cone- and Spatial-Opponency Primary Visual Cortex) module is composed of two parts (see Figure 1).

In the first part, input color channels (or input feature maps) are split into groups of equal depth. Convolution ($3\times 3$) operations are then applied on each group of channels (or feature maps) in order to extract features from each group as opposing color channel (or opposing feature maps). This is done thanks to a set of filters that convolve the group of color channels (or feature maps). Each filter is applied to the color channels (or input feature maps) through a convolution operation which detects local features at all locations on the input. Let ${\mathcal{I}}^g\in {\mathbb{R}}^{\mathcal{W}\times \mathcal{H}\times S}$ be a input group of feature maps where $\mathcal{W}$ and $\mathcal{H}$ are respectively the width and the height of each group’s feature map and $W\in {\mathbb{R}}^{3\times 3\times S}$, a filter with learned weights, S being the depth of each group or the number of the channels in each group g. The output feature map ${\mathcal{O}}^g\in {\mathbb{R}}^{\mathcal{W}\times \mathcal{H}}$ for this group $g\in \{1,\dots ,\mathcal{G}\}$ has pixel value in $(k,l)$ position defined as follow:

$${\mathcal{O}}_{k,l}^g=\sum _{s=1}^S\sum _{i=0}^2\sum _{j=0}^2{W}_{i,j,s}{\mathcal{I}}_{k+i-1,l+j-1,s}^g$$

(1)

The weights matrix $W\in {\mathbb{R}}^{3\times 3\times S}$ is the same across the whole group of channels or feature maps. So, each resulting output feature map represents a particular feature at all locations in the input color channels (or input feature maps) [39]. We call the $3\times 3$ convolution on grouped channels (or grouped feature maps) a group-wise convolution. The zero padding is applied during the convolution process to keep the input channels size for the output feature maps. After the group-wise convolution, we apply the batch normalization transform which is known to enable faster and more stable training of deep neural networks [40,41]. Let $\mathfrak{B}=\{X_1,\dots ,X_m\}$ be a mini-batch that contains m examples from the dataset $\{X_1,\dots ,X_m\}$, the mini-batch mean is

$${\mu }_{\mathfrak{B}}=\frac{1}{m}\sum _{i=1}^m{X}_i$$

and the mini-batch variance is

$${\sigma }_{\mathfrak{B}}^2=\frac{1}{m}\sum _{i=1}^m{(X_i-{\mu }_{\mathfrak{B}})}^2$$

The batch normalization transform $B{N}_{\gamma ,\beta }$ : $\{X_1,\dots ,X_m\}⟶\{Y_1,\dots ,Y_m\}$ ($\gamma$ and $\beta$ being parameters to be learned):

$$Y_i=\gamma \widehat{X_i}+\beta$$

(2)

where $i\in \{1,\dots ,m\}$ and

$$\widehat{X_i}=\frac{X_i-{\mu }_{\mathfrak{B}}}{\sqrt{{\sigma }_{\mathfrak{B}}^2+ϵ}}$$

and $ϵ$ is a very small constant to avoid division by zero.

In order to take into account the non-linearities present in the color channels input (or feature maps input), given that group-wise is a linear transformation, the batch normalization is followed by a non-linear function, Exponential Linear Unit (ELU) defined as follows:

$$ELU\left(x\right)=\left\{\begin{array}{cc}x\hfill & \mathrm{if}x\ge 0,\hfill \\ \alpha \times (exp(x)-1.)\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

$\alpha =1$ by default.

The non-linear function which is the activation function is placed after batch normalization as recommended by Chollet [42].

The second part of the module searches the best representation of the obtained feature maps. It is similar to the first part of the module except for group-wise convolution which is replaced by the point-wise convolution but the input feature maps for the point-wise convolution in this model are not grouped. The point-wise convolution allows us to learn the filters weights and thus obtain feature maps that best represent the input channels (or input feature maps) for the salient object detection task while having few parameters.

Let $\mathcal{O}\in {\mathbb{R}}^{\mathcal{W}\times \mathcal{H}\times M}$ be the output of the first part of the module, M being the number of feature maps in this output, $\mathcal{W}$, $\mathcal{H}$ the width and the height respectively. Let a filter of the learned weights $V\in {\mathbb{R}}^M$ and $\mathcal{FM}\in {\mathbb{R}}^{\mathcal{W}\times \mathcal{H}}$, its output feature map by a point-wise convolution. Its pixel’s value ${\mathcal{FM}}_{k,l}$ in $(k,l)$ position is:

$${\mathcal{FM}}_{k,l}=\sum _{m=1}^M{V}_m{\mathcal{O}}_{k,l,m}$$

(3)

Thus, $V\in {\mathbb{R}}^M$ is a vector of learned weights which, to the input feature maps $\mathcal{O}\in {\mathbb{R}}^{\mathcal{W}\times \mathcal{H}\times M}$ associates a feature map $\mathcal{FM}\in {\mathbb{R}}^{\mathcal{W}\times \mathcal{H}}$ which is the best representation of the input feature maps $\mathcal{O}\in {\mathbb{R}}^{\mathcal{W}\times \mathcal{H}\times M}$. The point-wise convolution in this module uses many filters and thus it outputs many feature maps that are the best representation of the input feature maps $\mathcal{O}$. As point-wise convolution is a linear combination, we apply again a batch normalization followed by a exponential linear unit function (ELU) on the feature map $\mathcal{FM}$, to get the best representation of the input feature maps for the learned weights $V\in {\mathbb{R}}^M$ that takes into account non-linearities in the feature maps $\mathcal{O}\in {\mathbb{R}}^{\mathcal{W}\times \mathcal{H}\times M}$.

Our scheme is different from depth-wise separable convolution in that depth-wise convolution does not use the convolution of a group of channels but each channel individually [28,43]

In addition, after the non-linear function, noise is injected in the resulting feature maps during the neural network learning stage thanks to dropout process (but not in the prediction stage) to facilitate the learning process. We used in this model the dropblock [44] if the width of the feature map is greater than 5 and the common dropout [45] otherwise.

The CoSOV1 module allows to have few parameters but also good performance for our neural network.

3.3. CoSOV1Net neural network model architecture

Our proposed model is built on the CoSOV1 module (see Figure 1). It is a neural network of the U-net encoder-decoder type [27] and is illustrated in Figure 2. Thus, our model consists of three main blocks:

• The input RGB color channel pairing;
• The encoder;
• The decoder.

3.3.1. Input RGB color channel pairing

At this stage, the input RGB image is paired in 6 opposing color channels pairs R-R, R-G, R-B, G-G, G-B, B-B [1,33,46]. These pairs are then concatenated which give 12 channels R,R,R,G,R,B,G ,G,G,B,B,B as illustrated in Figure 3. This is the step for choosing the color channels to oppose. The set of color channels concatenated are then fed to the encoder.

3.3.2. Encoder

The Encoder, in our proposed neural network model, is a convolutional neural network (CNN) [47] where an Encoder Unit (see Figure 2) is repeated eight times. Each Encoder Unit is followed by a max pooling ($2\times 2$) with strides=2 except for the 8^th neural network level where the max pooling is $3\times 3$ with strides=3 (the max pooling is a downsampling operation, like a filtering with a maximum filter). While the size of each feature map is reduced by half, the depth of the feature maps is doubled except for the first level where it is kept at 12 and the last two levels where it is kept at 128 to have few parameters.

The Encoder Unit (see Figure 4 (a)) is composed of a residual block (Figure 4 (b)) repeated three (3) times.

We used residual block because this kind of block is known to improve training deeper neural networks [48]. The residual block consists of two CoSOV1 modules with a residual link. The reason of all these repetitions is to encode more information and thus allow to increase our network performance.

In the encoder, schematically, as explained above (Section 3.2), the CoSOV1 module (Figure 4 (c)) splits the input channels in groups and apply a group-wise convolution ($3\times 3$ convolution). After a point-wise convolution is applied to the outputs of the concatenated groups (see Figure 5 for the first level input illustration). Each of these convolutions is followed by a batch normalization and a non-linear function (ELU: Exponential Linear Unit activation). After these layers, during the model training, the regularization is done in the CoSOV1 module using the dropout [45] method for small feature maps (dimension smaller than $5\times 5$) and the Dropblock [44] for feature maps with dimension greater than $5\times 5$ which is a variant of dropout that zeroes a block instead of pixels individually as Dropout does.

At its end, the encoder is followed by the middle unit (see Figure 6 (a) ) which is the CoSOV1 module (see Figure 6 (b)) where we remove the group-wise convolution, since at this stage the feature maps are $1\times 1\times 128$ in size, and add a residual link.

3.3.3. Decoder

The decoder transforms the features from the encoder to obtain the estimate of the salient object(s) present in the input image. This transformation is achieved through a repeating block, namely the Decoder Unit (see Figure 7 (a)). The Decoder Unit consists of two parts: the Decoder Residual Block (see Figure 7 (b)) and the Decoder Deconvolution block (see Figure 7 (c)). The Decoder Residual Block is a modified CoSOV1 module that allows the model to take into account the output of the corresponding level in the encoder. The output of the Decoder Residual Block took two directions. On the one hand, it is passed to the next level of the Decoder and on the other to the second part of the Decoder Unit which is the Decoder Deconvolution block. The latter deconvolves this output obtaining two feature maps having the size of the input image ($384\times 384\times 2$ in our case). At the last level of the decoder, all the outputs from the Deconvolution blocks are concatenated and fed to a convolution layer followed by a softmax activation layer which gives the estimation of the salient object detection map.

4. Experimental Results

4.1. Implementation Details

For our proposed model implementation, we used the deep learning platform TensorFlow with Keras deep learning application programming interface (API) [49]. All input images are resized to $384\times 384$ and pixels values are normalized (each pixel channel value $\in [0.0,\dots ,1.0]$ and ground truth pixels $\in \{0,1\}$). Experiments were conducted on a single GPU, nvidia Geforce RTX 3090 TI (24 GB) and an Intel CPU, i7-11700F.

4.2. Datasets

Our proposed model’s experiments were conducted on public datasets which are the most widely used in the salient object detection field [50]. Thus, we used Extended Complex Scene Saliency dataset (ECSSD) [51], DUT-OMRON (Dalian University of Technology—OMRON Corporation) [52], DUTS [53], HKU-IS [54], THUR15K [55] datasets.

The ECSSD [51] contains 1000 natural images and their ground truth. Many of its images are semantically meaningful, but structurally complex for saliency detection [51].

DUT-OMRON [52] contains 5168 images and their binary mask with diverse variations and complex background.

DUTS dataset [53] is divided into DUTS-TR (10553 training images) and DUTS-TE (5019 test images). We train and validate our proposed model on the DUTS-TR and DUTS-TE was used for tests.

HKU-IS [54] is composed by 4447 complex images, which contains many disconnected objects with different spatial distributions. Furthermore, it is very challenging for the similar foreground/background appearance [56].

THUR15K is a dataset of images taken from the “Flickr” web site divided into five categories (butterfly, coffee mug, dog jump, giraffe, plane), which contains 3000 images. The images of this dataset represent real world scenes and are considered complex for obtaining salient objects [55] (6232 images with ground truths).

4.3. Model Training Settings

For the reproducibility of the experiments, we set the seed=123. We train our proposed model on DUTS-TR (10553 training images). We split DUTS-TR dataset in a train set (9472 images) and a validation set (1056 images). That is approximately $90\%$ of the dataset for the training set and $10\%$ for the validation set. We didn’t use the 25 images because we wanted the training set and the validation set to be divisible by batch size which is 32.

Our proposed model is trained on scratch without pre-trained backbones from images classification (i.e. VGG [57], etc.) or lightweight backbones (i.e. MobileNets [28,29] or ShuffleNets [30,31]). As DUTS-TR is not a big dataset, we used data augmentation during training and many epochs in order to overcome this problem. Indeed, more there are epochs more the data augmentation process transforms data. Thus, our proposed model training has two successive stages:

• The first stage is with data augmentation. The data augmentation is applied on each batch with random transformation ($40\%$ zoom in or horizontal flip or vertical flip). This stage has 480 epochs: 240 epochs with learning rate = $0.001$ and the following 240 epochs with learning rate=$0.0001$;
• The second stage is without data augmentation. It has 620 epochs: 240 epochs with learning rate = $0.001$, followed by 140 epochs with learning rate = $0.0001$ and 240 epochs with learning rate = $0.00005$.

We also use a same initializer for all layers in the neural network: HeUniform keras initializer [58] which draws samples from a uniform distribution within [-limit, limit], where limit = $\sqrt{\frac{6}{fan\_in}}$ ($fan\_in$ is the number of input units in the weight tensor). The dropout rate is set to $0.2$. We used the RMSprop [59] keras optimizer with default values except for the learning rate, the centered which is set to true and the clipnorm=1. The loss function used is the “sparse_categorical_crossentropy” keras function; the keras metrics is “SparseCategoricalAccuracy; the keras check point monitor is “val_sparse_categorical_accuracy”

4.4. Evaluation Metrics

4.4.1. Accuracy

The metrics used to evaluate our proposed model accuracy are: $F_{\beta }$ measure, $MAE$ (mean absolute error), weighted $F_{\beta }^w$ measure [60]. We also used Precision-Recall and $F_{\beta }$ measures curves.

The $F_{\beta }$-measure ($F_{\beta }$ ) is the weighted harmonic mean of Precision and Recall:

$$F_{\beta }=\frac{(1+{\beta }^2)\times \mathrm{Precision}\times \mathrm{Recall}}{{\beta }^2\times \mathrm{Precision}+\mathrm{Recall}}$$

(4)

During evaluation ${\beta }^2=0.3$ as it is often suggested [16,56].

Let $\overline{S}$ be the saliency map estimation with pixels values normalized in order to be in $[0.0,\dots ,1.0]$ and $\overline{G}$, its ground truth also normalized in $\{0;1\}$. The $MAE$ (mean absolute error) is:

$$MAE=\frac{1}{W\times H}\sum _{x=1}^W\sum _{y=1}^H|\overline{S}(x,y)-\overline{G}(x,y)|$$

(5)

where W and H are the width and the height respectively of the above maps ($\overline{S}$ and $\overline{G}$).

The $F_{\beta }^w$ measure [60] fixes the interpolation flaw, dependence flaw, and equal importance flaw in traditional evaluation metrics and its value is:

$$F_{\beta }^w=(1+{\beta }^2)\frac{{\mathrm{Precision}}^w\times {\mathrm{Recall}}^w}{{\beta }^2\times {\mathrm{Precision}}^w+{\mathrm{Recall}}^w}$$

(6)

${\mathrm{Precision}}^w$ and ${\mathrm{Recall}}^w$ are the weighted $Precision$ and the weighted $Recall$ respectively.

4.4.2. Lightweight measures

Since, in this work we proposed a lightweight salient object detection model, thus, we also evaluate the model with lightweight measures: the number of parameters, the saliency maps estimation speed (FPS: frames per second) and the computational cost by measuring the FLOPS (the number of floating-point operations). The FLOPS is related to the devices energy consumption (more FLOPS more energy consumption).

4.5. Comparison with state-of-the-art

We compare our proposed model with 20 salient object detection state-of-the-art and 10 lightweight salient object detection state-of-the-art models. We divided these methods because the lightweight methods outperform others with respect to lightweight measures. However, the lightweight methods accuracy is lower than the accuracy of those with huge parameters. We mainly use the salient object detection results provided by Liu et al. [16] except for $F_{\beta }$ measure and Precision-Recall curves where we use saliency maps provided by these authors. We also used saliency maps provided by the HVPNet authors [17] to compute HVPNet $F_{\beta }^{\omega }$ measures.

In this section we describe the comparison with the 20 salient object detection namely DRFI[61], DCL [62], DHSNet [63], RFCN [64], NLDF [65], DSS [66], Amulet [67], UCF [68], SRM [69], PiCANet [70], BRN [71], C2S [72], RAS [73], DNA [74], CPD [75], BASNet [76], AFNet [77], PoolNet [78], EGNet [79] and BANet [80].

Table 1 shows that our proposed model CoSOV1Net outperforms all the 20 salient object detection state-of-the-art for lightweight measures (#parameters, FLOPS and FPS) with a large margin ( i.e. the best among them for the FLOPS is DHSNet [63] with $FLOPS=15.8G$ and $F_{\beta }=0.903$ for ECSSD; the worst is EGNet [79] with $FLOPS=270.8G$ and $F_{\beta }=0.938$ for ECSSD; while our proposed model CoSOV1Net has $FLOPS=1.4G$ and its $F_{\beta }=0.931$ for ECSSD) (see Table 1).

Table 1 also shows that CoSOV1Net is among the top 6 models for ECSSD, among the top 7 for DUT-OMRON around the top 10 for the other three datasets for F-measure. Table 2 and Table 3 compare our model with the state-of-the-art models for respectively MAE and $F_{\beta }^{\omega }$ measures. From this comparison, we see that our model is ranked around the top 10 for all the 4 datasets and the 15^th rank for the HKU-IS dataset. This demonstrates that our model is also competitive with respect to the performance of state-of-the-art models.

We also compare our proposed model CoSOV1Net with the lightweight salient object detection state-of-the-art: MobileNet [28], MobileNetV2 [29], ShuffleNet [30], ShuffleNetV2 [31], ICNet [81], BiSeNet R18 [82], BiSeNet X39 [82], DFANet [83], HVPNet [17], SAMNet [16].

For the comparison with lightweight state-of-the-art, Table 4 shows that our proposed model outperforms these lightweight state-of-the-art for the models parameters number and $F_{\beta }$ measure for ECSSD dataset and is competitive for other measure and dataset. Table 5 shows that our model outperforms these lightweight state-of-the-art for MAE measure for ECSSD and DUTS-TE datasets, is ranked 1^st ex aequo with HVPNet for DUT-OMRON, ranked 1^st ex aequo with HVPNet and SAMNet for HKU-IS dataset and 2^nd for THUR15K dataset. Our model also outperforms these lightweight state-of-the-art for $F_{\beta }^{\omega }$ measure for ECSSD and DUTS-TE and is competitive for the 3 other datasets (see Table 6).

4.6. Comparison with SAMNet and HVPNet state-of-the-art

We chose to compare our CoSOV1Net model specifically with SAMNet [16] and HVPNet [17] because they are among the best state-of-the-art models. Figure 8 and Figure 9 show that the proposed model is competitive with these two lightweight salient object detection state-of-the-art with respect to precision-recall and $F_{\beta }$ measure curves.

For qualitative comparison, Figure 10 shows some images highlighting that our proposed model (CoSOV1Net) is competitive with regard to the state-of-the-art SAMNet [16] and HVPNet [17] models which are among the best ones.

Images from columns 1 and 2 show a big salient object on a cloudy background, and a big object on a complex background respectively: CoSOV1Net (ours) performs better than HVPNet on these saliency maps. Columns 3 shows salient objects with same colors and column 4 shows salient objects with multiple colors: SAMNet and CoSOV1Net saliency maps are slightly identical and HVPNet saliency map is slightly better. Column 5 shows image with 3 salient objects with different sizes and colors: 2 are big and 1 is very small; CoSOV1Net saliency map is better than the SAMNet’s and the HVPNet’s. Column 6 shows red salient objects on black and yellow background; SAMNet’s saliency map is worst while CoSOV1Net and HVPNet perform well on that image. Column 7 shows a complex background and multiple salient objects with different colors: CoSOV1Net performs better than SAMNet and HVPNet. Column 8 shows tiny salient objects: the 3 models perform well. On column 9, SAMNet has worst performance while CoSOV1Net is the best. Column 10 shows colored glasses as salient objects: the CoSOV1Net performance is better than the SAMNet’s and HVPNet’s. On column 11, SAMNet has worst performance. On column 12 and 13, CoSOV1Net has the best performance. Column 18 shows a sub-marine image: CoSOV1Net is better than SAMNet.

5. Discussion

The results shows the performance of our model CoSOV1Net for accuracy measures and lightweight measures. The CoSOV1Net’s rank, when compared to state-of-the-art models, shows that CoSOV1Net behaves as a lightweight salient object detection by dominating the lightweight measures and having good performance for accuracy measures (see Table 7).

The results also showed that when the proposed model CoSOV1Net is compared to the lightweight salient object detection state-of-the-art, its measures results are ranked generally among the best for datasets and measures used. Thus, we can conclude that CoSOV1Net behaves as a competitive lightweight salient object detection model.

As we didn’t use backbones from images classification (i.e. VGG [57],…) or lightweight backbones (i.e. MobileNets [28,29] or ShuffleNets [30,31]), we conclude that our proposed model CoSOV1Net performance is intrinsic to this model itself.

Finally, putting together the measures for the salient object detection models and the lightweight salient object detection models in a graphic, we notice that the CoSOV1Net model is located for $F_{\beta }$ measures with respect to FLOPS and for the number of parameters up left while for the FPS measure it is located up right thus showing its performance as lightweight salient object detection model (see Figure 11). This shows that our proposed model CoSOV1Net is competitive with the best state-of-the-art models used.

The quantitative and the qualitative comparisons with SAMNet [16] and HVPNet [17] showed that our proposed model has good performance, given these state-of-the-art models are among the best ones.

6. Conclusion

In this work, we presented a lightweight salient object detection deep neural network, CoSOV1Net with very low parameters number (1.14M), low floating-point operations number (FLOPS=1.4G) thus low computational cost and respectable speed ($FPS=211.2$ on GPU: nvidia Geforce RTX 3090 TI) yet with comparable performance with state-of-the-art salient object detection models that use significantly more parameters and other lightweight salient object detection such as SAMNet [16] and HVPNet [17].

The novelty of our proposed model (CoSOV1Net) is that it uses the principle of integrating color in pattern in a salient object detection deep neural network, since according to Shapley [25] and Shapley et Hawken [18] color and pattern are inextricably linked in color human perception. This is implemented by taking inspiration from the primary visual cortex (V1) cells especially cone- and spatial-opponent cells. Thus, our method extracts features at the color channels spatial level and between the color channels at the same time on a pair of opposing color channels. The idea of grouping color pushed us to group feature maps through the neural network and extract features at the spatial level and between feature maps as done for color channels.

Our results showed that this strategy generates a model which is very promising, competitive with most salient object detection state-of-the-art and lightweight salient object detection state-of-the-art, and practical for mobile environments and limited resources devices.

As future work, our proposed CoSOV1Net model, based on integrating color into patterns, can be improved by coupling it with human visual system attention mechanism, which is the basis of many lightweight models, to produce a more efficient lightweight salient object detection model.