Resolving Contrast and Detail Trade-Offs in Image Processing with Multi-Objective Optimization

1. Introduction

Image enhancement is the process of applying specific techniques to boost an image’s visual quality. These techniques can imply diverse criteria, such as increasing the image contrast, noise reduction, highlighting relevant details, or adjusting brightness and color saturation. The main goal of image enhancements is to make the information within the image more easily interpretable and perceptible to human viewers, as well as to boost the automatic process of applications such as pattern recognition, medical analysis, and computer vision, among others [1,2,3].

Traditional image enhancement methods are divided into two well-defined categories [4,5]. The first one is the spatial domain enhancement that directly modifies pixel values to adjust aspects such as contrast and image detail, using techniques like smoothing and highlighting filters. The second one is the frequency domain enhancement, which transforms the image into a mathematical domain to adjust the frequency components, allowing fine detail enhancement and reduction of undesirable patterns through techniques such as high-pass or low-pass filtering. Both categories have specific applications and are chosen according to the type of improvement desired.

It is remarkable that not all images require the same enhancement process since the enhancement strategy relies upon the image’s specific characteristics. For example, a low-contrast image can significantly benefit from spatial domain enhancement techniques, like the contrast adjustment or the histogram equalization approach, which enhance the image’s sharpness [6,7]. On the other hand, in medical images such as magnetic resonances, the fine details highlighting can be critical; therefore, frequency domain enhancement techniques are more suitable for adjusting the high frequencies and spotlighting the image’s internal structures [8]. Hence, there is no unique ideal operator for all images, and there is no unique quantitative metric that automatically evaluates the image quality. Automatic image enhancement is a process that produces enhanced images without human intervention and is an extremely complicated task in image processing [9].

Image enhancement methods are commonly parametric, implying that their efficacy significantly relies upon fine-tuning various parameters. Within this context, evolutionary algorithms (EAs) stand out as highly effective tools. Pioneer work in this area is that of Bhandarkar, et al. [10], where a genetic algorithm (GA) is employed for an image segmentation problem. The outcomes showed that the GA outperforms traditional image segmentation methods regarding accuracy and robustness against noise. In the following years, numerous studies have proposed different approaches to address image enhancement problems through EAs. In [11], an optimization-based process solved through the particle swarm optimization (PSO) algorithm is employed to enhance the contrast and details of images by adjusting the parameters of the transformation function. This adjustment takes into account the relationship between local and global information. Subsequently, in [12], an accelerated variant of PSO for the aforementioned transformation function is proposed, achieving a more efficient algorithm in terms of convergence.

The use of EAs to solve image enhancement problems remains a common trend in recent years. This is observed in works such as [13], where the differential evolution (DE) algorithm is employed to maximize image contrast through a modified sigmoid transformation function. This function adjusts parameters that control the contrast magnitude and the balance between bright and dark areas, with optimal values determined through the evolutionary process. Similarly, Bhandari and Maurya [14] develop a novel optimized histogram equalization method, preserving average brightness while improving the contrast by means of a cuckoo search algorithm. The proposed method uses plateau boundaries to modify the image histogram, avoiding the extreme brightness changes often caused by traditional histogram equalization. In [15], GA is applied to optimize the histogram equalization through optimal subdivision considering different delimited light exposition regions. In [16] the directed searching optimization is applied in the medical image enhancement process, improving the contrast while preserving the texture in specified zones through a threshold parameter. The popularity of EAs has led to more sophisticated approaches, as in [17], where hybridization between whales optimization and chameleon swarm algorithms is proposed specifically to find the optimal parameters of the incomplete beta function and gamma dual correction. Several other EAs have been applied to image enhancement problems, such as monarch butterfly optimization [18], chimp optimization algorithm [19], sunflower optimization [20], slime mold algorithm [21], among others.

In the last decade, multi-objective optimization in image processing has also been the subject of several investigations. The relevance of the multi-objective approach lies in the need to balance multiple quality criteria simultaneously. In many cases, improving one image characteristic may worsen another. This inherent conflict between particular objectives requires an approach that obtains a set of optimal solutions, known as the Pareto front. For example, in [22], a PSO variant is proposed to address a multi-objective problem aimed to simultaneously maximize the available information quantity (by means of the entropy) and minimize the resulting image distortion (measured by the structural similarity index). In [23], GA is employed to maximize the Poisson log-likelihood function (used to measure the quantitative accuracy) and the generalized scan-statistic model (measures the detection performance). Similarly, in [24], a multi-objective cuckoo search algorithm is employed to tune the parameter of adaptive histogram equalization. The objectives were to enhance contrast by maximizing entropy and to minimize fast noise variance estimation. In [25], the Non-dominated Sorting Genetic Algorithm based on Reference Points (NSGA-III) is used to obtain optimal parameters for anisotropic diffusion, aiming to produce effective filtering results. The proposed methodology seeks to balance two competing objectives: image noise content and contrast.

The Pareto front approach implemented in image enhancement tasks offers a variety of solutions. A decision-maker can select the most suitable solution based on the specific needs of the application. In the context of multi-objective optimization, the incorporation of preferences in the decision-maker is known as preference articulation [26,27,28]. Even though preference articulation is a fundamental aspect of multi-objective optimization, its explicit use in works regarding image enhancement with a multi-objective optimization approach seems scarce.

Therefore, the current work is focused on image enhancement, considering two essential properties of image processing: contrast and details. Improving these properties through transformation functions is generally compromised, meaning that increasing contrast can lead to a significant loss of details in the image. Hence, a multi-objective optimization problem with two objective functions is established: one related to the image contrast and the other to the image details. These functions are measured by means of the entropy and the standard deviation for the image contrast, while the pixels’ quantity and intensity in high-frequency regions measure the image details. The contrast enhancement is performed by the sigmoid transform function, and the detail enhancement is performed by the unsharp masking and highboost filtering. To address the problem, the NSGA-II [29] is used with a posterior preference articulation, meaning that specific solutions are selected once the Pareto front is finally computed. The current work offers the following contributions:

• The trade-off between the enhancement of image contrast and details is set as a multi-objective problem. Unlike the traditional mono-objective approach, which only provides an optimal solution with a predefined priority, the current proposal offers the best solutions regarding the compromises between both criteria along the Pareto front.
• A posterior preference operator is articulated, providing three key images from the Pareto front: the image with maximum contrast, the image with maximum detail, and the image at the knee of the front, which represents the image closest to the utopia point. This operator allows the user to select the most suitable solutions to their particular needs.
• An experiment is conducted with images of two categories: medical and natural scene images. Both categories represent research fields where image processing is an essential endeavor. The results of this experiment demonstrate that the NSGA-II achieves images of superior quality compared to the original instances. Furthermore, a thorough analysis is conducted regarding the suitability of the obtained images according to the established preferences. For medical images, the evaluation focuses on how the selected solutions enhance the clarity and detail of relevant structures, which is crucial for diagnostics and analysis. For natural scene images, the analysis shows how the solutions improve contrast and detail, making the images more visually appealing and impactful.

The remainder of this paper is organized as follows: Section 2 outlines the sigmoid correction and unsharp masking with highboost filtering methods used in this study. In Section 3, we present the proposed multi-objective optimization model and posterior preference articulation method. Section 4 provides the benchmark results, including experimental design, graphical analysis, and quantitative evaluations. Finally, Section 5 offers conclusions and suggestions for future research directions.

2. Materials and Methods

2.1. Sigmoid Correction

Sigmoid correction is a technique used in image processing to adjust the contrast of an image in a nonlinear manner. This method is particularly useful for enhancing the contrasts in both dark and bright regions of an image. The correction is achieved using a sigmoid function, which maps the input intensity values to a new range according to a sigmoid curve [13,30,31]. The sigmoid function used for image correction is defined as:

$$g\left(x\right)={\displaystyle \frac{1}{1+e^{-\alpha (x-\Delta )}}}$$

(1)

where $g\left(x\right)$ represents the transformed pixel values; x is the original pixel intensity, scaled in the range of 0 to 1; $\alpha$ is the steepness of the sigmoid curve, which controls the degree of contrast adjustment; $\Delta$ is a value that determines the midpoint of the sigmoid curve related to the normalized pixel intensity, allowing control over the balance between the bright and dark regions of the image.

The parameter $\alpha$ affects how rapidly the transition occurs between dark and light regions. A higher value of $\alpha$ results in a steeper curve, increasing contrast by making transitions more abrupt, while lower values of $\alpha$ produce a gentler curve with smoother transitions, as shown in Figure 1a. The $\Delta$ parameter enables fine-tuning the balance between bright and dark regions. By adjusting this value, you can control where the midpoint of the contrast adjustment occurs, thus affecting how the dark and light areas of the image are processed. A higher $\Delta$ value results in a larger range of intensities being considered dark, leading to a darker overall image, whereas a lower value results in a larger range of intensities being considered bright, making the image brighter, as depicted in Figure 1b.

2.2. Unsharp Masking and Highboost Filtering

Unsharp masking and highboost filtering (UMH) are techniques for enhancing image sharpness and detail by manipulating high-frequency components. The process begins with creating a blurred version of the original image using a smoothing filter, such as an average filter. For a filter size of S, the average filter ${\mathbf{A}}_f\in {\mathbb{R}}^{S\times S}$ is defined as an $S\times S$ squared matrix, where each $a_{ij}\forall i,j\in \{1,\cdots ,S\}$ element within ${\mathbf{A}}_f$ is equal to ${\displaystyle \frac{1}{S^2}}$ . With $S=5$, the average filter is represented explicitly as follows:

$${\mathbf{A}}_f={\displaystyle \frac{1}{5^2}}\left[\begin{array}{ccccc}1& 1& 1& 1& 1\\ 1& 1& 1& 1& 1\\ 1& 1& 1& 1& 1\\ 1& 1& 1& 1& 1\\ 1& 1& 1& 1& 1\end{array}\right]$$

(2)

Applying this low-pass filter to the input image yields a blurred version of the original image, denoted as $f_b\left(x\right)$. In the unsharp masking process, a mask $m\left(x\right)$ is created by subtracting the blurred image from the original image:

$$m\left(x\right)=x-f_b\left(x\right)$$

(3)

This mask highlights the high-frequency details that are suppressed by the blur. However, not all high-frequency components contribute meaningfully to the image’s detail. To exclude insignificant details, a threshold $d_{th}$ is applied, where any values in the mask below this threshold are set to zero:

$$m\left(x\right)=\left\{\begin{array}{cc}0,\hfill & \mathrm{if}\left|m\left(x\right)\right|\le d_{th}\hfill \\ m\left(x\right),\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

(4)

This step ensures that only significant details are enhanced. To enhance the image, this mask is then added back to the original image with a scaling factor k to control the extent of the enhancement. The resulting enhanced image $g\left(x\right)$ is calculated as:

$$g\left(x\right)=x+k·m\left(x\right)$$

(5)

3. The Proposed Algorithm

In this section, we present the novel multi-objective optimization model and a posterior preference articulation method developed specifically for this study. This method is designed to find optimal trade-offs between the objective functions, while also providing preferential solutions capable of capturing different characteristics of the images.

3.1. Multi-Objective Optimization Problem

This work focuses on the enhancement of images considering two fundamental properties of image processing: contrast and details. The enhancement of these properties through transformation functions is often compromised, meaning that an increase in contrast can result in a significant loss of details in the image. Therefore, a multi-objective problem is defined with two objective functions:

• Contrast Function: Defined as the product of entropy $H\left(I\right)$ and the normalized standard deviation of the pixel intensities ${\sigma }_{norm}\left(I\right)$ of the I image. The contrast function is expressed as:

  $$f_1(\alpha ,\Delta )=H\left(I\right)\times {\sigma }_{norm}\left(I\right).$$

  (6)
• Details Function: Defined as the product of the number of pixels in high-frequency regions $N_{HF}\left(I\right)$ and the intensity of these high-frequency pixels $I{N}_{HF}\left(I\right)$ of the I image. The details function is denoted as:

  $$f_2(\alpha ,\Delta )=N_{HF}\left(I\right)\times {log}_{10}\left({log}_{10}\left(I{N}_{HF}\left(I\right)\right)\right).$$

  (7)

Formally, the multi-objective optimization problem can be expressed as in (8), where $\mathcal{F}=-\{f_1,f_2\}$ represents the set of objective functions to be minimized. The decision vector $\varphi ={[\alpha ,\Delta ]}^T$ controls the behavior of the sigmoid correction function, with ${\varphi }_{min}$ as the lower and ${\varphi }_{max}$ as the upper boundary.

$$\begin{array}{cc}\hfill & \underset{\varphi \in {\mathbb{R}}^2}{min}\mathcal{F}\left(\varphi \right)\hfill \\ \hfill & \mathrm{subject}\mathrm{to}\text{:}{\varphi }_{min}\le \varphi \le {\varphi }_{max}\hfill \end{array}$$

(8)

Before evaluating these objective functions, a series of image transformations are performed. Contrast enhancement is achieved using a sigmoid transformation function, where $\alpha$ and $\Delta$ are the decision variables that control the degree of contrast adjustment. Additionally, UMH is applied for detail enhancement. These transformations impact the objective functions by modifying the entropy, standard deviation, number of high-frequency pixels, and their intensities. Figure 2 illustrates the overall procedure for solving the multi-objective optimization problem using NSGA-II.

3.2. A Posterior Preference Articulation

A posterior preference articulation involves selecting specific solutions from the Pareto front based on particular criteria. In this work, the focus is on choosing the extreme solutions of the Pareto front and its knee point. The knee point is identified as the solution on the Pareto front that is closest to the utopia point, which typically represents the ideal but unattainable objective values. Within the optimal trade-offs of the Pareto front, these three solutions represent opposing balances of contrast and detail, helping to capture essential image characteristics that may be relevant according to the specific requirements of each application.

To determine the knee point, the Pareto front is first normalized, ensuring that the values of each objective function fall within the range [0, 1]. The utopia point in this normalized space is a vector of zeros, representing the best possible values for each objective. The Euclidean distance of each solution on the normalized Pareto front to the utopia point is calculated as follows:

$$d_i=\sqrt{\sum _{j=1}^{N_f}{(f_{ij}-p_j)}^2}$$

(9)

where: $N_f$ is the number of objective functions; $f_{ij}$ is the normalized value of the j-th objective for the i-th solution; $p_j$ is the value of the j-th objective at the utopia point (typically 0 after normalization). The knee point is identified as the solution with the minimum distance to the utopia point.

4. Benchmark Results and Discussion

In this session, the fundamental aspects of the experiment are described. Subsequently, the results are presented and discussed in two parts: first, the visual analysis of the results is conducted, followed by a discussion of the numerical results from the images in relation to the well-established indicators.

4.1. Experimental Design

A set of twenty images is selected to assess the effectiveness of the method developed in this work. This set is divided into two groups: the first includes 10 natural scene images extracted from the Kodak dataset [32], specifically from kodim01 to kodim10 (hereinafter referred to as Natural1 to Natural10, in that order). The second group consists of 10 medical images (referred to as Medical1 to Medical10, respectively) selected from various libraries, including brain images [33,34], blood composition images (white blood cells of the basophil and eosinophil types) [35,36], X-rays [37], ocular nodules [38], dental infections [39], microphotographs of pulmonary blood vessels [40], and traumatic forearm positioning [41].

The NSGA-II algorithm is executed for each image with a maximum of 30,000 function evaluations, aiming to produce a Pareto front containing the best solutions. From this front, solutions are extracted according to the defined articulation preference operator. The objective is to evaluate the quality of these solutions in terms of contrast and details, complemented by a visual analysis to determine the suitability of each image for specific purposes. Finally, the similarity of the enhanced images to the original ones is assessed using the structural similarity index (SSIM), which allows for quantifying the degree of similarity between the processed and original images.

The parameter values used in this experiment are as follows: population size ($N_p=50$), number of variables ($N_{var}=2$), number of objective functions ($N_f=2$), number of evaluations ($N_{eval}=30,000$), mutation probability ($P_m=0.5$), crossover probability ($P_c=0.7$), simulated binary crossover parameter ($N_c=5$), polynomial mutation parameter ($N_m=5$), details threshold ($d_{th}=0.1$), lower bound (${\varphi }_{min}={[0,0]}^T$), and upper bound (${\varphi }_{max}={[10,10]}^T$).

4.2. Graphical Results

Table 1, Table 2, Table 3 and Table 4 present the results obtained through the multi-objective optimization image enhancement approach. Specifically, Table 1 and Table 2 show the results for natural images, while Table 3 and Table 4 display medical images. The tables are organized as follows: the first and second columns list the image names and their corresponding original, unenhanced versions. The third to fifth columns showcase the selected points from the Pareto front, representing the maximum contrast, knee point, and maximum detail, respectively. The final column illustrates the obtained Pareto front through the optimization process, with red, green, and orange points indicating the images that achieved maximum contrast, knee point, and maximum detail, respectively.

As observed in the results, the images extracted from the Pareto front significantly maximize both contrast and detail compared to the original images. In all study cases, the original image is dominated by the solutions extracted from the fronts, demonstrating the approach’s effectiveness in improving visual quality. However, the differences among the three enhanced images for each problem require a more detailed analysis.

In the natural images, the differences among the three preferred images are more subtle, given that these are high-quality images with inherently low contrast, specifically selected for contrast enhancement exercises. More pronounced differences are observed in the images Natural1, Natural6, and Natural8. For the Natural6 image, there is a general improvement in overall details. However, specific regions, such as the highlighted flower in the yellow box, may lose details compared to the original image, which retains more information. This suggests that for future work, it may be advisable to apply local and/or adaptive image enhancement techniques to preserve details in specific regions while maintaining overall image quality.

For medical images, there are instances where differences are more perceptible. For example, in the Medical3 image, the maximum contrast solution makes it difficult to visualize the internal details of the basophil (white blood cell), which could result in less accurate interpretation. In contrast, the knee solution and maximum detail solution provide a clearer view of the interior of the white blood cell. Similarly, in the Medical5 image, the maximum contrast solution highlights the bone structures of the hand and arm. However, the maximum detail image offers a more precise view of the internal structures within the bones, which is crucial for a more detailed evaluation. Another notable example is the Medical8 image, where the maximum detail solution offers a more detailed view of the internal structure of the eosinophil (another type of white blood cell). However, the maximum contrast image improves the visibility of red blood cells. As shown in the yellow box, this solution reveals a red blood cell that is nearly imperceptible in the other solutions. An interesting case is the Medical6 image, where only a few non-dominated solutions are present on the Pareto front. Despite the similarities among the preferred solutions, the nodules are much more perceptible in the enhanced images than in the original image.

The solutions extracted from the Pareto front represent optimal trade-offs between contrast and detail. For natural images, these three alternatives can be considered useful based on aesthetic criteria or in subsequent automatic processes that require prioritizing one property over another. In the case of medical images, these alternatives allow for a more precise evaluation suited to different diagnostic needs, providing a flexible approach to enhance the visualization of critical details according to the clinical context.

4.3. Quantitative Results

Table 5 and Table 6 present noteworthy information regarding several criteria following the next structure. For each image whose name is presented in the first column, a set of three rows displays the outcomes of the Pareto front’s maximum contrast, knee point, and maximum details solutions. The outcomes per individual regarding their entropy, normalized standard deviation, number of pixels, and pixel intensity are displayed from the third to the sixth column. The seventh and eighth columns display the objective function values of the multi-objective optimization problem. The last column displays the SSIM with respect to the original images, where all those images archived values above 0.7, i.e., SSIM>0.7, are in boldface, implying that these images accomplished the enhancement with an acceptable similarity to the original image.

As can be seen, the maximum contrast solutions generally yield higher entropy and normalized standard deviation, indicating a broader range of pixel intensities and greater variability in the enhanced images. In contrast, maximum detail solutions focus on enhancing the finer details within the images. This often results in lower entropy and normalized standard deviation compared to maximum contrast solutions but may increase the number and the intensity of high-frequency pixels (indicating more detailed textures). These results highlight differences in contrast and detail between the solutions extracted from the Pareto front that may not be perceptible in the previous visual discussion. If we examine some cases, we can find images such as Natural2, Medical1, and Medical4, where the extreme points on the Pareto front do not show a visual difference. However, their associated values for contrast and detail exhibit numerical differences.

Through the reported values of the objective functions, it can be observed that all exposed solutions are non-dominated, indicating that they represent an optimal trade-off between the two evaluated criteria. Regarding the SSIM index, 65% of the solutions exhibit values above 0.7, indicating a generally high level of structural similarity. Furthermore, by only analyzing the medical images, 85% of the solutions reached SSIM outcomes beyond 0.7, indicating that the proposal is a trustworthy tool when dealing with this kind of information. Nonetheless, if only the natural scene images are evaluated, the number of solutions that archived this SSIM outcome decreases to 50%. This may be an influence of the artificially imposed low contrast in this set of images. Consequently, future work should consider incorporating SSIM as an additional objective function could be considered, especially for images where image fidelity can be crucial.

5. Conclusions

The conflict between contrast and detail in image processing is presented as a multi-objective problem. Through this approach, a set of optimal solutions is obtained, forming a Pareto front in all cases, highlighting the trade-off that can exist between these two properties. Therefore, it is demonstrated that a single-objective approach to this problem will only lead to a particular solution among all the optimal solutions obtained through the multi-objective approach.

A proposed model integrates the sigmoid transformation function and UMH into the NSGA-II. Additionally, a posterior preference articulation is added, which selects three key solutions from the Pareto front: the maximum contrast solution, the maximum detail solution, and the knee point solution. These three solutions showed significant superiority in terms of contrast and detail compared to the original images. Furthermore, the outcomes visually and numerically demonstrated how these three image solutions, though all optimal solutions differ in terms of entropy, standard deviation, number of detail pixels, and detail intensity. This influenced the perception of fundamental characteristics within the images, highlighting the relevance of the proposed preferences in different contexts.

Despite an improvement in the overall details of the images, specific regions may lose details compared to the original image. This suggests that for future work, it may be advisable to apply local and/or adaptive image enhancement techniques to preserve details in specific regions while maintaining overall image quality. It is also advisable to incorporate SSIM or a similar performance index as an objective function to improve image fidelity, especially in medical applications where the image’s information trustworthiness is crucial.