A Convolutional Neural Network as a Potential Tool for Camouflage Assessment

1. Introduction

Camouflage refers to the use of any combination of materials, coloration, or illumination to reduce the visual signature of an object in its local background. Both in nature and in military operations, camouflage is applied to maximize survivability by minimizing the chances of being seen. In the military context, camouflage measures such as camouflage nets, low-emissive paints, camouflage patterns, etc. are typically applied to visually blend high-value assets into their environment. It is of great importance to assess how effectively such camouflage measures reduce an object’s visual signature and to know why and when they work. This assessment requires a metric of detectability.

Various metrics of visual object detectability have been developed over the years. These metrics either involve human observer experiments or objective computational (image-based) measures [1,2]. Human observer studies designed to assess camouflage efficiency typically involve visual search and detection experiments. However, such behavioral studies are time-consuming and expensive since they involve many observers and repetitions. Also, natural scenes are typically highly heterogeneous, making it difficult to assess camouflage effectiveness for moving objects since their local contrast varies with their momentary position. Hence, there is a need for models that compute camouflage effectiveness from digital imagery.

In the rest of this section we first discuss the pros and cons of observer and computational object visibility metrics. Then we suggest to adopt the YOLO predicted certainty (confidence level) of an object’s identity as a measure of camouflage efficiency.

1.1. Observer camouflage metrics

Given the fact that the goal of camouflage is to decrease the probability of detection, human visual search performance measured in field trials using precisely controlled scenarios and closely monitored conditions is often considered to provide the golden standard for camouflage assessment. However, field tests are costly, time-consuming, labor-intensive, and logistically difficult. Moreover, the number of conditions in which camouflage efficiency can be tested is limited and the variation in performance across conditions is inherently large (as different participants show different scan paths). Hence, field search trials are often replaced by laboratory experiments, using photo- and video-simulation techniques and imagery that was recorded in field trials. Laboratory experiments allow complete control over all influential parameters. However, human performance in artificial laboratory conditions may differ considerably from performance in real-world scenarios. This may be due to differences between the simulated environment and the real world, such as limitations in resolution [3], luminance, color [4] reproduction, dynamic range, etc., while the relative importance of each of these factors is still largely unknown.

A viable and efficient alternative for extensive search and detection experiments is visual object conspicuity. Visual conspicuity quantifies the contrast between the visual signature of an object and its local background. This contrast can arise from differences in size, shape, luminance, color, texture, binocular disparity, motion, and clutter. An object is easier to detect when it stands out from its background, and the more it stands out the quicker it will be found [5,6,7]. It is therefore likely that visual conspicuity is the main determinant of human visual search performance. Visual conspicuity can be operationalized as the maximum angular gaze deviation at which objects can still be distinguished from their immediate surroundings. We previously developed and validated an efficient psychophysical measurement procedure to quantify this concept [8,9,10,11]. This method is quick and efficient, can be used in situ, with full prior knowledge of the object and its location in the scene, and requires only a few observers for statistical significance. The resulting conspicuity measures reliably predict human visual search performance in realistic and complex scenarios [8,10] for both static objects [10] and dynamic objects [12], in the lab [10,13,14] and in the field [10,15]. Visual conspicuity can also quantify the saliency of camouflaged objects in visual, near-infrared, and thermal imagery [15,16,17]. Depending on the criterion used, one can assess the object’s detection or identification conspicuity. Detection conspicuity corresponds to the subjective observer criterion that the visual object signature differs noticeably from the local background, while identification conspicuity corresponds to the criterion that the image details at the location of the object can actually be identified as the target. Since identification conspicuity also contains a top-down (cognitive, recognition) component, it predicts mean search time more closely than detection conspicuity [8].

1.2. Computational camouflage metrics

Image based camouflage effectiveness assessment algorithms compute object saliency using digital object-background contrast metrics [1,18]. These metrics have frequently little or no relation to human visual perception. To compute overall object saliency, an object’s position or area should be known in advance. In addition, there is no unique definition of object saliency (e.g., one can arbitrarily adopt the average or maximum saliency over the object support area [1]).

Another approach to image based assessment of camouflage effectiveness are visual saliency models [1]. These models typically use human vision models to compute local contrast maps (local saliency) from multiple multiresolution feature maps. Similar to simple contrast metrics, the overall object saliency metric is ill-defined while the object position and outlines are required for its computation [1].

1.3. YOLO camouflage metric

State-of-the-art neural networks can detect and classify objects in digital imagery and provide a degree of certainty, or confidence, of their classification. Since this confidence level reflects an object’s visual distinctness [19], we propose to adopt it as a measure of camouflage effectiveness. The benefit of this approach is that it requires no a priori knowledge of the nature, size and position of the object of interest.

Today, some of the best object detection results are achieved by models based on convolutional neural networks (CNN). YOLO [20] (You Only Look Once) is currently one of the fastest and most accurate object detection algorithms available. YOLO uses features learned by a deep convolutional neural network to detect specific objects in videos, live feeds, or images. It looks at the whole image at test time, so its predictions are informed by the global context in the image. The YOLO algorithm first divides the image into a grid. For each grid cell, it predicts a number of object bounding boxes (see the green squares in Figure 1a) with their associated class probabilities or confidence scores using a single feed forward convolution neural network. The confidence scores represent the degree of similarity of the detected object to each of the predefined object classes (such as person, train, cat, etc.) and reflect the accurateness of the model prediction. In this study, we validate the YOLO model for the assessment of effectiveness of camouflaged persons by relating its output to human observer data (identification conspicuity and search performance) obtained in the lab.

In the next sections, we first investigate whether the level of confidence predicted by YOLO can serve as a measure of camouflage effectiveness. To investigate whether the predicted certainty (confidence level) of an object’s identity correlates with its degree of camouflage (i.e., whether it decreases with decreasing visual object distinctness), we apply YOLO to images of a single person situated in different environments and wearing clothing that provides different levels of camouflage. Then, we investigate whether the level of confidence predicted by YOLO also predicts human ratings of object identification conspicuity and to human visual search performance. Since identification conspicuity and the level of confidence predicted by YOLO both increase with the visible amount of characteristic object details we expect that both object distinctness measures will be correlated. A part of this work was previously published as a conference paper [21].

2. Experiment 1: YOLO detection performance for camouflaged persons

Using a photosimulation approach, we examined whether YOLO is able to detect a person wearing camouflage clothing in two different (bush/desert and urban) environments, and whether the associated confidence levels of the detections can serve as a measure of camouflage effectiveness. Thereto, an image of a person in camouflage clothing was digitally inserted at multiple locations in photographs of both environments, and YOLO determined the probability that a person was present.

2.1. Methods

The background images used in this study were color photographs of a bush/desert background (Soesterduinen, the Netherlands, see Figure 1a, left panel) and an urban background (Amsterdam, the Netherlands, see Figure 1a, right panel). The size of the images was 1000 × 666 pixels (width × height). These images were also used in a previous study [22].

Test scenes were generated by inserting the image of a person wearing a grey camouflage suit at various locations in both the bush/desert and urban background scenes. Figure 1a shows the image of the person that was used, superimposed on different locations in the two backgrounds. Each test scene contained a single person at a location (in x,y coordinates) that was both naturalistic (the person was never presented in the air, in a tree, or in a river, etc.) and systematically manipulated within a certain range. For the bush/desert environment, the x (horizontal) position varied between 66 and 900 pixels (at intervals of half the person width), and the y (vertical; position) varied between 275 and 417 pixels (intervals of half the person height), such that the person was positioned at most locations in the background image. For the urban environment, x varied between 66 and 900 pixels (at intervals of half the person width), and y varied between 150 and 417 pixels (at intervals of half the person height). The size (i.e., height and width) of the person scaled with distance in the scene. As a result, only a few persons were presented nearby while most were presented at larger distances.

We used the YOLOv4-tiny pre-trained neural network to detect persons [23]. The Python code is available online: https://data-flair.training/blogs/pedestrian-detection-python-opencv/. The detection was set to classify persons only, and the input images used were small (1000 × 666 pixels) to speed up the process. The threshold for overlapping predictions was low (0.03), since we are only interested in a single class (‘person’). Furthermore, the minimum confidence was set to 0.0001 (to increase the probability that the person was found). The minimum confidence reflects the threshold for the confidence score returned by the model above which a classification (in our case a ‘person’) is considered correct.

For each person detected by YOLO (i.e. a hit), we determined the detection probability (i.e., the associated confidence score generated by YOLO, indicating the probability that the detected object belongs to the class ‘person’). The probability was set to zero if the person was not detected by YOLO (i.e., a miss). Note that there were no correct rejections since the person was always present. Neither were there any false alarms, indicating that no aspects of the environment were classified as persons. Subsequently, for each background environment separately, we created a detection probability heatmap by summing the YOLO confidence scores for the persons over the image area covered by their body. Then finally, we determined the average detection probability for each pixel in the image.

2.2. Results

The green squares (bounding boxes) in Figure 1a represent persons found by YOLO. It is clear from the few examples shown in Figure 1a that YOLO succeeds in finding most persons in both scenes. Only a few were not found (as illustrated by red arrows in Figure 1a). In general, YOLO is able to detect the persons, at least when they are presented nearby and that the detection probability (i.e., the confidence score) decreases with increasing distance (since the person scales with distance, resulting in a decreasing amount of detail).

The detection probability heatmaps are illustrated in Figure 1b for the bush/desert and the urban environment (left and right panel, respectively). Figure 1c illustrates the mean detection probability (collapsed over the horizontal plane) as a function of the person height (i.e., distance of the person, since the person size scaled with distance). It is clear from this illustration that the detection probability decreases with increasing distance (i.e., with decreasing object resolution or distinctness). YOLO did an excellent job in detecting the persons even though they were rather tiny (see the green squares in Figure 1a). More specifically, a probability of 80% corresponds to an object size of about ~60 and ~40 pixels in the urban and the bush/desert environment, respectively (see Figure 1c). Therefore, we conclude that the person was slightly better camouflaged in the urban environment than in the bush environment. This is probably due to the higher homogeneity of this background.

3. Experiment 2: YOLO vs human camouflaged person detection on photosimulations

It is known from the literature that a visual object captures attention when a specific feature, such as its color, shape, or orientation deviates from its immediate background [24]. Similar results were reported in a military context, indicating that camouflage efficiency is best if a camouflage suit shares properties (such as colors and orientations) with its local background [22,25].

In Experiment 2a, we examine whether YOLO is able to distinguish between good and bad performing camouflage patterns based on a comparison of their spatio-chromatic features with those of their local surroundings. Therefore, we manipulated both the pattern and the colors of the camouflage suit and presented the person in either an urban or bush/desert environment (like Experiment 1). The pattern was either an average color, or consisted of a 1/f fractal pattern (see Figure 2). The colors used for the camouflage suits were either drawn from the bush/desert environment or from the urban environment. Therefore, we specified a region of interest (ROI) from which the colors were drawn for each environment separately (see Figure 2a). Throughout the experiment, we kept the person’s pose fixed to avoid it to influence YOLO’s performance. Moreover, we included another condition by adding a shading to the person’s camouflage suit, since it is known that shading can affect YOLO’s performance (for example for the detection of fruit, see [26,27]). We manipulated the shade presence to examine whether shades influence YOLO’s performance, as they make it feasible to distinguish human body parts. Figure 4 illustrates the eight camouflage suits used in Experiment 2 (2 ROI’s × 2 camouflage patterns × shade presence). We expect that YOLO has more difficulty in detecting the person when the camouflage suit consists of colors selected from the same background (e.g., a bush/desert camouflage suit in a bush/desert environment) than when it consists of colors selected from a different background (e.g., a bush/desert camouflage suit in an urban environment). Moreover, we expect that YOLO finds it more difficult to detect a person wearing a 1/f fractal pattern than one wearing an average colored uniform, given that natural images also consist of similar 1/f structures [28,29,30,31,32,33]. Furthermore, we expect that it is easier for YOLO to detect a person with shading than without shading as the former is more realistic (and YOLO is trained to identify realistic persons). Finally, we expect the detection rate to decrease with distance, and that the effect of shading ceases to exist with distance (as the articulation of details decreases with distance).

Figure 2. (a) Region of interest (ROI) in the bush/desert (left panel) and the urban environment (right panel). (b) Camouflage patterns used in Experiment 2 without shading. (c) Camouflage patterns with shading.

Figure 2. (a) Region of interest (ROI) in the bush/desert (left panel) and the urban environment (right panel). (b) Camouflage patterns used in Experiment 2 without shading. (c) Camouflage patterns with shading.

Figure 3. Detection probability heatmaps in the bush/desert background. (a) Mean detection probability (i.e., the confidence score generated by YOLO) when the person was wearing a uniform in an average color derived from a bush ROI. The panels on the left indicate the detection probability when no shading was added, whereas a shading was added for the right panels. (b) Mean detection probability when the person was wearing a 1/f fractal camouflage pattern derived from a bush ROI. (c) and (d) Same as panels (a) and (b), but for an urban ROI.

Figure 3. Detection probability heatmaps in the bush/desert background. (a) Mean detection probability (i.e., the confidence score generated by YOLO) when the person was wearing a uniform in an average color derived from a bush ROI. The panels on the left indicate the detection probability when no shading was added, whereas a shading was added for the right panels. (b) Mean detection probability when the person was wearing a 1/f fractal camouflage pattern derived from a bush ROI. (c) and (d) Same as panels (a) and (b), but for an urban ROI.

Figure 4. Detection probability heatmaps in the urban background. (a) Mean detection probability (i.e., the confidence score generated by YOLO) when the person was wearing a uniform in an average color derived from a bush ROI. The panels on the left indicate the detection probability when no shading was added, whereas a shading was added for the right panels. (b) Mean detection probability when the person was wearing a 1/f fractal camouflage pattern using a bush ROI. (c) and (d) Same as panels (a) and (b), but for an urban ROI.

Figure 4. Detection probability heatmaps in the urban background. (a) Mean detection probability (i.e., the confidence score generated by YOLO) when the person was wearing a uniform in an average color derived from a bush ROI. The panels on the left indicate the detection probability when no shading was added, whereas a shading was added for the right panels. (b) Mean detection probability when the person was wearing a 1/f fractal camouflage pattern using a bush ROI. (c) and (d) Same as panels (a) and (b), but for an urban ROI.

In Experiment 2b, we examine whether the detection probability provided by YOLO in Experiment 2a generalizes to human observers, using a visual search and an object identification conspicuity paradigm. Thereto, we randomly selected a set of 64 images that were also used in Experiment 2a. Each image was shown four times (randomly mirrored) and participants performed two tasks. First, an image appeared on the screen and participants were instructed to press the spacebar as soon as they found the person in the scene (i.e., search task). Subsequently, a mask replaced the image and participants were asked to indicate the location of the person in the scene using their mouse to verify whether the participant correctly detected the person. Second, we measured the object identification conspicuity [8,10,12]. Thereto, the image reappeared on the screen, and participants were asked to fixate a location on either the left or right in the scene, far away from the person’s location so that they were no longer able to resolve (identify) the person. Next, participants slowly fixated locations in the scene that were progressively closer to the person, until they were able to identify the object in their peripheral vision as a ‘person’. Finally, participants used the mouse to indicate this location. The identification conspicuity reflects the distance between the indicated location and the object’s location (i.e., the eccentricity). If the detection probability derived by YOLO is a realistic measure for detecting persons, then we expect that the detection probability increases with increasing identification conspicuity (i.e., a positive correlation). Moreover, we expect that the detection probability increases with decreasing search times (i.e., a negative correlation), and that the search time increases with decreasing object conspicuity (i.e., a negative correlation).

3.1. Methods Experiment 2a

The method was identical to the previous experiment, except for the following changes. In the present experiment we used the images of the same person, but manipulated the pattern on his camouflage suit. Therefore, we first defined a ROI (the area in which persons could reasonably be expected to appear in the scene) for each background environment (see Figure 2a). Based on the ROI of the background (bush/desert or urban environment), a fractal (1/f) camouflage pattern was created with colors (statistically) matching the background, given that fractal structures are known to match the statistics of natural images [32]. The pattern was created as follows [14]. First, a 1/f pattern (pink-noise) pattern in RGB was created. Then, a texture is created for each of the layers of a multiband image (R, G, B) using a random sample of this pink noise pattern. Finally, the color distribution of the background sample (ROI) was applied to the texture using 3D histogram equalization [34]. A second camouflage pattern that was used was the average color over the ROI. Furthermore, we added shading to the camouflage suit to make the persons more realistic. Thereto, half of the camouflage suits included shading (See Figure 2c), whereas the other half of the camouflage suits were without shading (Figure 2b).

3.2. Methods Experiment 2b

3.2.1. Participants

Six participants (2 females, 4 males, mean age: 31 years, ranging from 23 to 48 years) participated in both Experiment 2b and Experiment 3b. Two participants were the authors (PP, EVDB), while the others participated voluntary and were naïve as to the purpose of the experiment. The experimental protocol was reviewed and approved by the TNO Internal Review Board (TNO, the Netherlands: reference 2019-024) and was in agreement with the Helsinki Declaration of 1975, as revised in 2013 [35].

3.2.2. Stimuli and apparatus

The experiment was programmed and run in Python using the Psychopy module [36,37]. For the validation of Experiment 2a, we selected 64 different images from Experiment 2a (only persons with shading added). More specifically, for each background condition (bush/desert and urban), we randomly selected eight different object locations. For each object location, we manipulated the ROI (bush/desert and urban) and pattern (average and 1/f) of the camouflage suit (see Section 3.1). It is important to note that we fixed the object location within each background condition, since this makes it feasible to compare the behavioral performance across the different camouflage suits (i.e., ROI and pattern combination). The images (1000 × 562 pixels; visual angle: 54.7˚ × 30.7˚) were shown four times, either presented normally or mirrored horizontally. Participants were seated approximately 100 cm from a 4k Samsung Monitor (UE65MU6100; size: 140 × 80 cm; resolution: 1280 × 720 pixels; Refresh: 60 Hz), and used a standard mouse and QWERTY keyboard to make responses.

3.2.3. Design and procedure

A trial started with the presentation of a white fixation cross on a grey background for a duration of 800 ms. Subsequently, one of the 64 images was shown and participants were instructed to press the spacebar as quickly as possible when they found the target object (the person). The time between the image onset and the response reflects the search time. A white noise mask replaced the image as soon as the participants pressed the space bar and participants were instructed to indicate the person’s location using the mouse. Then, the identification conspicuity for the person was measured. Participants used the mouse to indicate the maximum distance from the detected object to identify it as being indeed a person (on either the left or right side from the person). The next trial was initiated when participants made the unspeeded mouse response. Participants performed 256 experimental trials (64 images × 4 repetitions) divided over four blocks. Participants were allowed to take a break between each block, and received verbal instructions prior to the experiment. Participants were already familiar with the task, since they first performed Experiment 3b, followed by Experiment 2b.

3.3. Results

Figure 3 and Figure 4 illustrate the results of Experiment 2a for the bush/desert and urban background environment, respectively. Here, the detection probability heatmap is shown for each of the eight camouflage suits used.

Figure 5a,b illustrate the detection probability as a function of object height (i.e., distance) for each camouflage suit used in the bush/desert and urban environment (collapsed over the horizontal plane), respectively. Even though for each panel the camouflage suit was always the same, the error bars represent a variance of detection probability over the horizontal plane. This variance is most likely a result of the heterogeneity of the environments. Indeed, the object-background similarity varied over the locations, making the same person highly visible at certain locations and invisible at other locations. This is consistent with the human visual search literature, showing that it is more difficult to find an object that is similar to its background than one that is dissimilar to its background [24]. Furthermore, it is clear from these figures that, in general, the proportion of detected objects decreased with distance. This is in line with the results of our previous behavioral study using the same background images [22]. However, the 1/f camouflage suit with bush/desert colors revealed a somewhat different pattern of results for both backgrounds used. It is clear here that the detection probability did not decrease with distance, but instead increased with distance for the urban background and was rather low for the bush/desert environment in general. A feasible explanation for these deviating results is that the 1/f camouflage suit was generated as a whole, without taking important properties of a human being (like having arms, legs, etc.) into account. As a result, YOLO did not classify a large person as a ‘person’. It is interesting to see that for more distant persons, the performance of the 1/f pattern was rather similar to that of the average uniform. Furthermore, it is also interesting that the detection probability increased when shading was added to the camouflage suits (compare the red lines with the black lines in each panel). This makes perfect sense as adding shading results in more realistic ‘persons’ by emphasizing the shape characteristics of a human being. That the effect of shading disappeared over distance also makes perfect sense as environmental details (such as a trunk of a tree or a person’s hand) that are clearly visible when observed nearby, can no longer be resolved at larger distances.

The persons were located at naturalistic locations for both background environments (so that they were not presented in the air, tree, etc.). Due to this restriction, the persons were smaller for the bush/desert background (ranging between 14.82 – 201.28 pixels) than for the urban background (ranging between 24.37 – 214.20 pixels). Figure 5c,d illustrate the overall mean detection probability for each camouflage suit for both backgrounds. However, unlike the data presented in Figure 5a,b, we collapse the results over all locations with the constraint that the height of the person was between 24.4 and 201.2 pixels for both background environments, thus enabling a comparison of the camouflage suits across the two different background conditions.

In general, the camouflage efficiency was better (i.e., lower detection probability) for the 1/f pattern (0.55) compared to the average colored suit (0.85). Interestingly, when focusing on the average colored suit in the bush/desert background, the camouflage efficiency was better when the colors were drawn from the bush/desert environment (0.46) than from the urban environment (0.88). The opposite effect was observed for the urban background, but only for the average colored suit. That is, the camouflage efficiency was worse when the colors were drawn from the bush/desert environment (0.93) than from the urban environment (0.84). This opposing effect was not observed for the 1/f fractal pattern, which is most likely due to the deviating results for the 1/f pattern with colors drawn from the bush/desert environment (which is expected to be due to the high contrast pattern breaking the human form rather than to the colors). As expected, the detection probability was lower when the camouflage suits’ color matched the environment (0.63; e.g., urban ROI in an urban environment) than when it did not match the environment (0.77; e.g., urban ROI in a bush/desert environment). This was the case for all the four camouflage suits used. This is consistent with the literature that it is easier to find an object when the color is dissimilar to the background than when it is similar to the background [24].

Figure 6 illustrates the results from the behavioral tasks in Experiment 2b. Here, Figure 6a depicts the number of responses as a function of the localization error. Trials were excluded from further analyses if the participants were not able to localize the person accurately (i.e., the localization error was > 150 pixels) or when the search time was larger than 10 seconds. As a result, 4 trials were excluded as participants were not able to localize the person correctly (0.3%), and 1 trial was excluded because of the long search time (0.07%).

Figure 6b illustrates the search performance as a function of detection probability for each image used in Experiment 3b. There was a significant negative correlation between the search time (human performance) and detection probability (YOLO performance), Pearson r = -0.46, p < 0.001, indicating that the search time increased with decreasing detection probability. Figure 6c illustrates the identification conspicuity as a function of the detection probability for each frame tested. As is clear from this figure, identification conspicuity correlated positively with the detection probability, Pearson r = 0.71, p < 0.001, indicating that YOLO predicted the conspicuity performance quit well. The search performance did also correlate with the identification conspicuity performance, Pearson r = -0.69, p < 0.001 (see Figure 6d), indicating that search times increased with decreasing object identification conspicuity. What’s interesting is that these three correlations were significant for each participant (see Table 1).

Figure 7 illustrates the mean search time and mean object identification conspicuity as a function of ROI (bush/desert versus urban) and pattern (average versus 1/f) for each background condition. Although our overall aim was not to examine which camouflage suit performed better according to the behavioral measure, for each background condition separately, we conducted a post-hoc repeated measures ANOVA on the mean search time and mean object identification conspicuity with ROI (bush/desert versus urban) and pattern (average versus 1/f) as within subject variables.

For the bush/desert background environment (Figure 7, left panels), the ANOVA on the identification conspicuity yielded a significant ROI effect, F(1, 5) = 7.877, p = 0.038, as the mean conspicuity was smaller when the camouflage suit contained colors from the bush/desert background (5.3 dva) than from the urban background (6.4 dva). The mean conspicuity was smaller when the camouflage pattern was 1/f (5.4 dva) than when it was the average color (6.3 dva), as revealed by a significant pattern effect, F(1, 5) = 71.376, p < 0.001. The two-way interaction failed to reach significance, F(1, 5) = 5.014, p = 0.075. The ANOVA on the mean search time yielded a significant ROI effect, F(1, 5) = 50.369, p < 0.001, as the search times were significantly larger when the suit’s color matched the background color (ROI: bush/desert, 883 ms) than when it did not match the background color (ROI: urban, 671 ms). The main effect of pattern and the two-way interaction failed to reach significance, F-values ≤ 3.490, p-values ≥ 0.12

For the urban background environment (Figure 7, right panels), the ANOVA on the identification conspicuity yielded a significant ROI effect, F(1, 5) = 11.375, p = 0.020, as the mean conspicuity was larger when the camouflage suit contained colors taken from the bush/desert background (7.6 dva) than from the urban background (6.5 dva). There was a trend towards a significant pattern effect, F(1, 5) = 6.192, p < 0.055, as the mean conspicuity was smaller when the camouflage pattern was 1/f (6.8 dva) than when the uniform has an average color (7.3 dva). The two-way interaction failed to reach significance, F(1, 5) = 0.609, p = 0.471. The ANOVA on the mean search time yielded a significant ROI effect, F(1, 5) = 10.030, p < 0.025, as the search times were significantly larger when the suit’s color matched the background color (ROI: urban, 935 ms) than when it did not match the background color (ROI: bush/desert, 860 ms). The main effect of pattern and the two-way interaction failed to reach significance, F-values ≤ 0.117, p-values ≥ 0.746.

The results are clear. As expected form the literature, camouflage efficiency was better when the color of the camouflage suit matched the colors of the background than when it did not [14,22]. This was revealed by both the mean search times as well as the mean object identification conspicuity. Furthermore, the identification conspicuity was smaller for the 1/f pattern than for the average color pattern. However, this effect was only significant for the object identification conspicuity measure, but failed to reach significance for the search time, probably due to the low number of observations in this study (given the large inherent variation that is typical for search times, reliable estimates of mean search time require many observers and a large number of repetitions [38]).

Summarizing, the results of Experiment 2 show that YOLO predicted the human detection of a camouflaged person quite well: detection probability correlated negatively with mean search time and positively with mean identification conspicuity.

4. Experiment 3: YOLO vs human camouflaged person detection on naturalistic images

In the photosimulation experiments described in the previous sections, we showed that YOLO is able to detect persons wearing camouflage clothing, and that its detection performance correlates with human detection performance. Although the results are convincing, it is not clear to what extent this result holds for settings with naturalistic lightning (shadows) and contrast. YOLO has been trained using urban environments, which are typically more homogenous and consist of regular structures, making it potentially easier to detect a person. Thus, it may be less efficient for detecting persons wearing a camouflage suit and a helmet in a natural environment. To investigate this issue we performed two more experiments.

First, in Experiment 3a, we examine whether YOLO is able to detect persons wearing camouflage clothing in naturalistic conditions. Therefore, we recorded a movie of a person walking through the forest while wearing a standard Netherlands woodland camouflage suit and a helmet. We expect that YOLO can also detect persons wearing camouflage clothing in naturalistic conditions with a given probability.

Next, in Experiment 3b, we examine whether the detection probability provided by YOLO for persons wearing camouflage clothing in naturalistic conditions correlates with human performance. Thereto, we selected 14 different samples from the movie shown in Experiment 3a and replicated the tasks from Experiment 2b (i.e., a visual search task followed by an object conspicuity task). If YOLO detects the person in the naturalistic environment, then we expect to replicate the results from Experiment 2b.

4.1. Methods Experiment 3a

The movie was recorded in Soesterberg, The Netherlands (August 9th 2022). The person was wearing a Netherlands woodland camouflage suit and a green helmet. The short movie (950 samples; 59 seconds) was resized (1000 × 562 pixels) and YOLO predicted the camouflage efficiency for each sample (as determined by the detection probability).

4.2. Methods Experiment 3b

The experiment was identical to experiment 2b, except for the following changes. We selected 14 different samples from the movie shown in Experiment 3a. The samples were randomly selected such that the detection probability varied from trial to trial (detection probability for the 14 different samples ranged from .26 till .97). Participants performed 54 experimental trials each (14 movie sample × 4 repetitions). Participants received verbal instructions prior to the experiment and practiced 6 trials to get familiar with both tasks. Figure 8 illustrates a couple of samples from the movie for illustrative purposes.

4.3. Results

The results of the Experiment 3a are illustrated in Figure 8. Here, five random samples from the movie are shown (the person is always present). If YOLO detected the person, the green square signifies its location with a given detection probability as shown in the graph. The graph illustrates the detection probability for each movie frame. The movie can be downloaded online from https://osf.io/vpcer/files/osfstorage/62fd376b7b16410c0c12f3b1.

Figure 8 illustrates that YOLO did a good job in finding the person. Even though motion detection is not incorporated (i.e., our version of YOLO considers each image as a static individual image), the person was detected on the vast majority of samples. Interestingly, poses such as kneeling down, and (partly) hiding behind a tree affected performance significantly.

Figure 9 illustrates the results from Experiment 3b. Figure 9a illustrates the number of responses as a function of the localization error. Trials were excluded from further analyses if the participants were not able to localize the person accurately (i.e., the localization error was > 150 pixels), and responded too slow (RTs > 10 seconds). As a result, we excluded 0.6% of the trials (only localization errors).

Figure 9b illustrates the search performance as a function of detection probability for each frame used in Experiment 3b. There was a trend towards a significant negative correlation between the search time (human performance) and detection probability (YOLO performance), Pearson r = -0.52, p = 0.057. Figure 9c illustrates the identification conspicuity as a function of the detection probability for each frame tested. As it is clear from this figure, like in Experiment 2b, the identification conspicuity performance correlated positively with the detection probability, Pearson r = 0.69, p < 0.01, indicating that YOLO predicted the conspicuity performance quite well. Table 2 illustrates the correlations for each participant separately. Interestingly, the identification conspicuity correlated positively with the detection probability for 5 out of 6 participants. The search performance did not correlate with the identification conspicuity performance, Pearson r = -0.2, p = 0.502 (see Figure 9d). This is probably due to the low accuracy of the estimated mean search time since the number of search trials in this study was rather low [38].

Summarizing, the results of Experiment 3 are generally in line with those from Experiment 2: detection probability correlated positively with mean identification conspicuity and mostly negatively with mean search time.

5. Discussion

In general, YOLO was well able to detect the persons both in photosimulations (Experiments 1 and 2) and in naturalistic stimuli (Experiment 3). Camouflage efficiency (as determined by YOLOs’ confidence rating: detection probability) correlated with human performance. More specifically, camouflage efficiency correlated positively with the object identification conspicuity, and negatively with mean search time. However, the latter correlation was only significant for the photosimulation experiments (Experiment 2b), but failed to reach significance for the naturalistic setting (Experiment 3b). The absence of a significant correlation in the naturalistic setting may be attributed to the combination of a low number of trials (i.e., four) per image and the person being always highly salient. Indeed, the person was always easy to find, with mean search times between 500 – 750 ms, indicating that the person did pop-out from its environment [5,6,7]. In contrast, in the photosimulation experiment the mean search time across the different images varied between 500 – 2000 ms, indicating that search was more difficult for at least some of the images shown. As a result, less trials are required to find significant differences across the different images used for the photosimulation than in the naturalistic setting. However, in the visual search literature, a frequently used rule of thumb is that at least 20 trials are required to estimate search performance for a particular condition in a lab setting [7,39,40], and at least 60 trials in a more applied setting [38]. Nevertheless, the strong positive correlation between the object conspicuity measure and YOLO’s performance is a good indicator that the YOLO is suitable for estimating camouflage efficiency.

It is interesting that the detection of persons by YOLO was influenced by whether we superimposed shading to the person or not. It was easier to detect a person with shading than without. This suggest that not only the outline or silhouette of the person is an important factor, but also the articulation of other body parts, such as arms, which became visible when adding shading (see Figure 2). Moreover, as expected, the detection probability decreased with distance, and the effect of shading disappeared with distance as well. That the effect of shading disappeared with distance makes perfect sense, as for the human eye, details such as the trunk of a tree, the arms of a person, or the letters on billboards, typically can no longer be resolved at larger distances. Although the distance effect was rather consistent and present for the vast majority of conditions, it was interesting to observe that YOLO had more difficulties in finding a person wearing a 1/f fractal camouflage suit derived from the bush ROI when the person was nearby than when he was further away. A feasible explanation for this counterintuitive finding is that the structures (pattern elements) of the camouflage suit gave the percept that some diagnostic body parts, such as the arms, were missing when the person was nearby but not when he was far away. This occurred even in the condition where we superimposed a shading to the person. A feasible explanation is that the shading had little effect when using a 1/f fractal pattern with colors derived from the bush/desert ROI, and not from the urban ROI. In the former condition, the shading effect was minimized due to the dark textures in the camouflage suit (see the shading for both 1/f fractal patterns in Figure 2c). Apparently, the high contrast pattern is capable of breaking the human form as perceived by YOLO, while this is not the case (to the same extent) for the human observer. In the future, it would be good to realistically model shading and the different parts of the camouflage suit, such that the arms (and other body parts) become more articulated (and thus easier to detect for YOLO).

The present study also replicates earlier findings. For instance, the camouflage efficiency significantly improved when the colors of the camouflage suit (e.g., bush/desert) consisted of colors taken from its immediate background (bush/desert environment) than when it consisted of colors taken from another background urban environment [14]. This color effect was observed for both human behavior and for YOLOs’ behavior. Furthermore, again human performance and YOLO’s performance both indicated that the 1/f fractal pattern is a more efficient camouflage pattern compared to a uniform average color. Interestingly, as is clear from the detection probability heatmaps, even though the camouflage suit and the person’s pose was kept constant for each condition, the camouflage efficiency not only depended on the distance, but also on where the person was presented on the horizontal plane. This variance not only illustrates that the camouflage efficiency for a particular camouflage suit depends on the local contrast with its immediate environment, but it also illustrates the value of using YOLO as a tool to measure of camouflage efficiency. Indeed, estimating camouflage efficiency for each location using behavioral experiments is simply too time consuming, if not practically impossible.

Summarizing, the finding that YOLO’s confidence score associated with the detection of a camouflaged object correlates with human detection performance suggests that it may be developed into a valid measure of camouflage effectiveness.

5.1. Limitations

In this study we used the pretrained YOLOv4-tiny model. Since this model was introduced several newer versions have appeared. However, recent versions of the YOLO model typically prioritize balancing the tradeoff between speed and accuracy rather than focusing on accuracy [41,42]. This study primarily investigates the inherent capability of YOLO to predict human detection performance. Given its relatively high accuracy [42] YOLOv4 still appears a valid tool for this purpose [43].

In its present form YOLO’s classification performance appears sensitive to high contrast, human shape breaking camouflage (e.g. the 1/f fractal bush/desert camouflage pattern at short distances). Future studies should investigate this effect in more detail.

In this exploratory study we used only two background scenes and the image of a person as target object. Future studies should investigate whether YOLO can be applied to assess camouflage performance across a wider range of different backgrounds, target objects, camouflage patterns and combinations thereof.

In this study we applied YOLO-V4-tiny to detect persons in the individual frames from a video sequence. It would be interesting to replicate the present study using a version of YOLO that incorporates motion detection [44]. Motion is a special case in the visual search literature, as a minimal motion change is known to capture attention immediately [7,45,46,47]. In the military context, it is also known that motion breaks camouflage [48], even if a person wears an optimal camouflage suit [22].