LLMSeamCarver: LLM-Enhanced Content-Aware Image Resizing

1. Introduction

LLMSeamCarver is an image resizing method designed to maintain important image details during resizing. By integrating LLMs, LLMSeamCarver goes beyond traditional methods, enabling a smarter, context-aware resizing process. LLMs enhance tasks such as region prioritization, interpolation, edge detection, and energy calculation by analyzing image context or textual inputs. This allows LLMSeamCarver to preserve crucial image features like faces and text while optimizing resizing for different scenarios.

This paper explores how LLMs improve the accuracy and efficiency of image resizing.

2. Background

2.1. Image Resizing and LLMs

LLMSeamCarver enhances the traditional seam carving method by incorporating LLMs. The integration of LLMs allows for dynamic resizing of images based on real-time context and user input, rather than relying on fixed parameters. The LLM dynamically optimizes the energy functions that determine how seams are selected and removed, making the resizing process adaptive and context-sensitive. This approach allows for superior preservation of image details and better quality when resizing for specific tasks, such as creating thumbnails or preparing images for various screen sizes.

LLMs contribute by adjusting the parameters based on the content of the image and the desired effect, improving the quality of resized images while providing flexibility. The dynamic optimization of the resizing process using LLMs represents a major leap in the flexibility and efficiency of image resizing techniques.

2.2. LLM-Related Work

Recent research in LLMs has demonstrated their potential in a variety of domains, including text, image, and video generation.

Recently, LLMs has shown its power and potential to enhance traditional image processing workflows through advanced model architectures and optimization techniques. For instance, researchers have developed methods that optimize convolutional neural network (CNN) layers for feature extraction in image processing tasks, thereby improving the performance of deep learning models in image classification and segmentation [2,22]. Zhang et al. (2024) explored the use of LLMs in multi-modal fusion networks, enabling the integration of both visual and textual information, which enhances image analysis tasks [10,11,24,26].

Additionally, efficient algorithm design [1,6,24,27,30] and efficient LLMs [17,18] have shown promising prospective in efficient model design with LLMs. Through dynamic optimization, LLMs allow for more context-aware resizing by adjusting energy functions during the process. This flexibility ensures that fine-grained details are preserved, which is especially crucial for tasks like content-aware resizing. Studies on text-to-image models have demonstrated how LLMs can modify images based on contextual prompts [7,13,15,16,28], providing further advancements in content-aware image resizing.

2.3. Image and Vision Generation Work

The application of deep learning techniques in image and vision generation has seen remarkable progress in recent years [3,4,9,21,23,25,26]. Deep convolutional networks have been effectively utilized for texture classification, achieving high accuracy in distinguishing fine-grained patterns and structures (Bu et al., 2019) [1]. These approaches significantly enhance detail preservation during image resizing by maintaining textures and edges, ensuring that visual fidelity is retained across transformations.

Moreover, advances in multi-modal fusion networks and techniques for image-driven predictions—as demonstrated by Dan et al. (2024)—illustrate how artificial intelligence can process and modify images in real-time [3,14,26]. These methods integrate data from diverse sources, facilitating applications such as enhanced video editing and real-time object tracking.

Additionally, model compression is increasingly favored from both model optimization and system design perspectives, enabling efficient resource usage without sacrificing performance [8,19,20]. These innovations support dynamic, user-controlled visual generation, opening new possibilities for customizable and interactive media content creation.

2.4. Image Resizing and Seam Carving Research

The traditional seam carving method was proposed by Avidan and Shamir (2007), other studies have contributed to enhancing seam carving methods. Kiess (2014) introduced improved edge preservation methods within seam carving [12], which is crucial for ensuring that resized images do not suffer from visible distortions along object boundaries. Zhang (2015) compared classic image resizing methods and found that seam carving provided superior results when compared to simpler resizing techniques, particularly in terms of detail preservation [29].

Frankovich (2011) further advanced seam carving by integrating energy gradient functionals to enhance the carving process, providing even more control over the resizing operation [5].

3. Functionality

LLMSeamCarver leverages LLM-Augmented methods to ensure adaptive, high-quality image resizing while preserving both structural and semantic integrity. The key functionalities are:

• LLM-Augmented Region Prioritization: LLMs analyze semantics or textual inputs to prioritize key regions, ensuring critical areas (e.g., faces, text) are preserved.
• LLM-Augmented Bicubic Interpolation: LLMs optimize bicubic interpolation for high-quality enlargements, adjusting parameters based on context or user input.
• LLM-Augmented LC Algorithm: LLMs adapt the LC algorithm by adjusting weights, ensuring the preservation of important image features during resizing.
• LLM-Augmented Canny Edge Detection: LLMs guide Canny edge detection to refine boundaries, enhancing clarity and accuracy based on contextual analysis.
• LLM-Augmented Hough Transformation: LLMs strengthen the Hough transformation, detecting structural lines and ensuring the preservation of geometric features.
• LLM-Augmented Absolute Energy Function: LLMs dynamically adjust energy maps to improve seam selection for more precise resizing.
• LLM-Augmented Dual Energy Model: LLMs refine energy functions, enhancing flexibility and ensuring effective seam carving across various use cases.

4. LLM-Guided Region Prioritization

To enhance seam carving, we propose a formalized method where LLMs are used to compute semantic importance $S(x,y)$ for each pixel, directly modifying the energy map $E(x,y)$. This ensures content-aware resizing with minimal disruption to critical features.

4.1. Semantic Importance Scoring by LLMs

The semantic importance $S(x,y)$ is computed as:

$$S(x,y)=g\left(h\right(I),q(D\left)\right),$$

(1)

where:

• $h\left(I\right)$: Image embedding derived from input image I using a vision feature extractor.
• $q\left(D\right)$: Text embedding derived from optional user description D using a language transformer.
• g: A cross-modal scoring function combining $h\left(I\right)$ and $q\left(D\right)$, implemented via attention mechanisms.

If no description D is provided, $q\left(D\right)$ defaults to a generic embedding, allowing g to focus solely on image features.

4.2. Energy Map Adjustment

The refined energy map $E^{\prime }(x,y)$ integrates semantic importance into the standard energy formulation:

$$\begin{array}{cc}\hfill E^{\prime }(x,y)& =E(x,y)+\alpha ·S(x,y),\hfill \\ \hfill E(x,y)& =\sqrt{{(\nabla I_x(x,y))}^2+{(\nabla I_y(x,y))}^2},\hfill \end{array}$$

(2)

where:

• $\nabla I_x(x,y),\nabla I_y(x,y)$: Gradients in x- and y-directions.
• $S(x,y)$: Semantic score indicating the importance of pixel $(x,y)$.
• $\alpha$: Weighting factor balancing pixel-based energy and semantic importance.

The function $S(x,y)$ ensures that higher semantic scores reduce the likelihood of important regions being removed during seam carving.

4.3. Semantic Score Calculation with LLMs

To compute $S(x,y)$, the LLM processes image embeddings $h\left(I\right)$ and textual embeddings $q\left(D\right)$:

$$h\left(I\right)=\mathrm{VisionEncoder}\left(I\right),$$

(3)

$$q\left(D\right)=\mathrm{TextEncoder}\left(D\right),$$

(4)

$$S(x,y)=\mathrm{Attention}\left(h\right(I),q(D\left)\right),$$

(5)

where:

• VisionEncoder: Extracts regional features from I (e.g., object locations, edges).
• TextEncoder: Encodes user-provided descriptions into contextual embeddings.
• Attention: Combines $h\left(I\right)$ and $q\left(D\right)$ to assign $S(x,y)$ based on pixel relevance.

The score $S(x,y)$ is normalized to a range $[0,1]$ for compatibility with $E(x,y)$.

4.4. Cumulative Energy Map Update

The cumulative energy map $E^{\prime }(x,y)$ propagates semantic adjustments through the seam carving process. For a given pixel $(i,j)$, the update is:

$$\begin{array}{cc}\hfill E^{\prime }(i,j)& =E(i,j)+\alpha ·S(i,j)\hfill \\ \hfill & +min\{C_L(i,j)+E^{\prime }(i-1,j-1),\hfill \\ \hfill & C_R(i,j)+E^{\prime }(i-1,j+1),\hfill \\ \hfill & C_U(i,j)+E^{\prime }(i-1,j)\},\hfill \end{array}$$

(6)

where:

• $C_L,C_R,C_U$: Left, right, and upward cost terms adjusted with $S(x,y)$.
• min: Ensures the optimal seam path minimizes distortion of high-priority regions.

5. LLM-Augmented Bicubic Interpolation

The LLM-Augmented Bicubic Interpolation method enhances traditional bicubic interpolation by incorporating semantic importance scores $S(x,y)$ derived from LLMs. This approach ensures content-aware resizing, prioritizing regions of high semantic value such as faces, text, and objects.

5.1. Traditional Bicubic Interpolation

In standard bicubic interpolation, the value of a pixel at position $(x,y)$ is computed using a 4x4 grid of neighboring pixels. The interpolation weights $w\left(x\right)$ are determined by the relative distances between the target pixel and its neighbors:

$$w\left(x\right)=\left\{\begin{array}{cc}{(a+2)\left|x\right|}^3-(a+3){\left|x\right|}^2+1,\hfill & \mathrm{if}\left|x\right|\le 1,\hfill \\ {a\left|x\right|}^3-{5a\left|x\right|}^2+8a\left|x\right|-4a,\hfill & \mathrm{if}1<\left|x\right|<2,\hfill \\ 0,\hfill & \mathrm{otherwise},\hfill \end{array}\right.$$

(7)

where a is a constant (commonly $a=-0.5$) controlling the interpolation smoothness.

The pixel value at position $(X,Y)$ is computed as:

$$B(X,Y)=\sum _{i=0}^3\sum _{j=0}^3{a}_{ij}·w(x-X_i)·w(y-Y_j),$$

(8)

where $a_{ij}$ are the pixel intensities in the 4x4 grid.

5.2. LLM-Augmented Interpolation

In the augmented method, the interpolation weights are modified to incorporate semantic importance:

$$w^{\prime }\left(x\right)=w\left(x\right)·(1+\beta ·S(x,y)),$$

(9)

where:

• $S(x,y)$: Semantic importance score for pixel $(x,y)$, computed by the LLM.
• $\beta$: Scalar factor controlling the influence of $S(x,y)$ on the interpolation process.

The updated pixel value is then calculated as:

$$B^{\prime }(X,Y)=\sum _{i=0}^3\sum _{j=0}^3{a}_{ij}·w^{\prime }(x-X_i)·w^{\prime }(y-Y_j),$$

(10)

ensuring higher weights for regions with greater semantic importance.

5.3. Semantic Importance Calculation

The semantic score $S(x,y)$ is derived using the LLM:

$$S(x,y)=f_{\mathrm{LLM}}(I,D),$$

(11)

where:

• I: Input image.
• D: Optional user-provided description specifying priorities (e.g., "preserve faces").
• $f_{\mathrm{LLM}}$: A function combining image embeddings $h\left(I\right)$ and text embeddings $q\left(D\right)$ through an attention mechanism:

  $$S(x,y)=\mathrm{Attention}\left(h\right(I),q(D\left)\right).$$

  (12)

5.4. Cumulative Interpolation Update

The augmented interpolation integrates the adjusted weights:

$$f^{\prime }(x,y)=\sum _{i=0}^3\sum _{j=0}^{3}f(x_i,y_j)·w^{\prime }(x-x_i)·w^{\prime }(y-y_j),$$

(13)

where $w^{\prime }\left(x\right)$ and $w^{\prime }\left(y\right)$ ensure greater emphasis on semantically significant regions.

6. LLM-Augmented LC (Loyalty-Clarity) Policy

The LLM-Augmented LC Policy improves contrast-based resizing by integrating semantic importance parameters from LLMs. These parameters ensure that important regions, such as faces and text, receive higher priority during resizing.

6.1. Global Contrast with Semantic Guidance

We calculate the global contrast of a pixel $I_k$ as:

$$C^{\prime }\left(I_k\right)=\sum _{\forall I_i\in I}\parallel I_k-I_i\parallel +\lambda ·\sum _{r\in R}{w}_r·\parallel I_k-I_r\parallel ,$$

(14)

where:

• $\lambda$ scales the influence of semantic importance, computed by the LLM.
• $w_r$ represents the semantic weight of region r, also derived from the LLM.
• R is the set of semantically significant regions.

This formulation integrates semantic relevance into the contrast computation, preserving critical regions.

6.2. Frequency-Based Contrast Refinement

The traditional frequency-based contrast uses intensity distributions:

$$C\left(I_k\right)=\sum _{n=0}^{255}{f}_n·\parallel a_m-a_n\parallel ,$$

(15)

where $f_n$ is the frequency of intensity $a_n$. We extend this by including semantic importance:

$$C^{\prime }\left(I_k\right)=C\left(I_k\right)+\lambda ·\sum _{r\in R}{w}_r·f_r·\parallel a_m-a_r\parallel ,$$

(16)

where:

• $f_r$ adjusts the frequency of region r based on the LLM’s analysis.
• $w_r$ weights the region according to its semantic importance.

6.3. LLM Parameter Computation

The LLM computes $w_r,f_r,\lambda$ through embeddings and attention mechanisms:

• Image Embeddings $h\left(I\right)$: A vision encoder extracts pixel-level and global features:

  $$h\left(I\right)=\mathrm{VisionEncoder}\left(I\right).$$

  (17)
• Text Embeddings $q\left(D\right)$: A text encoder processes user descriptions:

  $$q\left(D\right)=\mathrm{TextEncoder}\left(D\right).$$

  (18)
• Spatial Embeddings ${\xi }_r$: Positional embeddings represent region-specific attributes:

  $${\xi }_r=\mathrm{PositionalEmbedding}\left(r\right).$$

  (19)
• Semantic Weights $w_r$: Attention mechanisms combine embeddings:

  $$w_r=\mathrm{Attention}(h\left(I\right),q\left(D\right),{\xi }_r).$$

  (20)
• Frequency Adjustment $f_r$: The LLM refines the frequency distribution:

  $$f_r={\displaystyle \frac{{\sum }_{p\in r}f\left(p\right)}{\mathrm{Area}\left(r\right)}}.$$

  (21)
• Scaling Factor $\lambda$: A sigmoid function ensures $\lambda \in [0,1]$:

  $$\lambda =\sigma \left(g\right(h\left(I\right),q\left(D\right)\left)\right).$$

  (22)

6.4. Contrast-Based Resizing Decision

The adjusted contrast $C^{\prime }\left(I_k\right)$ guides the resizing process. The seam path is chosen to minimize distortion in semantically important regions:

$$min\left\{\sum _{k\in P}{C}^{\prime }\left(I_k\right)\right\},$$

(23)

where P represents the seam path.

7. LLM-Augmented Canny Line Detection

The LLM-Augmented Canny Line Detection integrates semantic importance parameters $S_{\mathrm{LLM}},{\sigma }_{\mathrm{LLM}},T_{\mathrm{LLM}}$ derived from a Large Language Model (LLM) to enhance edge detection. This approach ensures that edges in critical regions, such as faces or text, are prioritized during resizing.

Figure 1. Outlier detected by the LLM-LC algorithm.

Figure 1. Outlier detected by the LLM-LC algorithm.

7.1. Gaussian Filtering

The image is smoothed using a Gaussian filter:

$$G(x,y)={\displaystyle \frac{1}{2\pi {\sigma }^2}}{e}^{-{\displaystyle \frac{x^2+y^2}{2{\sigma }^2}}}.$$

(24)

The LLM adjusts the smoothing factor $\sigma$ for semantically important regions:

$${\sigma }_{\mathrm{LLM},r}=\sigma ·(1+\alpha ·S_{\mathrm{LLM},r}),$$

(25)

where:

• $S_{\mathrm{LLM},r}$: Semantic importance score for region r.
• $\alpha$: Scaling factor for the semantic influence.

7.2. Gradient Calculation

Gradients are computed using the Sobel operator:

$$G(i,j)=\sqrt{G_x{(i,j)}^2+G_y{(i,j)}^2},$$

(26)

$$\theta (i,j)=arctan\left({\displaystyle \frac{G_y(i,j)}{G_x(i,j)}}\right),$$

(27)

where $G_x(i,j)$ and $G_y(i,j)$ are horizontal and vertical derivatives.

The LLM adjusts the gradient magnitude for each pixel:

$$G_{\mathrm{LLM}}(i,j)=G(i,j)·S_{\mathrm{LLM}}(i,j).$$

(28)

7.3. LLM Semantic Parameter Calculation

The semantic scores $S_{\mathrm{LLM}}(i,j)$, smoothing factor ${\sigma }_{\mathrm{LLM},r}$, and thresholds $T_{\mathrm{LLM},\mathrm{high}},T_{\mathrm{LLM},\mathrm{low}}$ are computed as follows:

$$\begin{array}{cc}\hfill h\left(I\right)& =\mathrm{VisionEncoder}\left(I\right),\hfill \end{array}$$

(29)

$$\begin{array}{cc}\hfill q\left(D\right)& =\mathrm{TextEncoder}\left(D\right),\hfill \end{array}$$

(30)

$$\begin{array}{cc}\hfill {\xi }_{i,j}& =\mathrm{PositionalEmbedding}(i,j),\hfill \end{array}$$

(31)

$$\begin{array}{cc}\hfill S_{\mathrm{LLM}}(i,j)& =\mathrm{Attention}(h\left(I\right),q\left(D\right),{\xi }_{i,j}),\hfill \end{array}$$

(32)

$$\begin{array}{cc}\hfill {\sigma }_{\mathrm{LLM},r}& =\sigma ·(1+\alpha ·S_{\mathrm{LLM},r}),\hfill \end{array}$$

(33)

$$\begin{array}{cc}\hfill T_{\mathrm{LLM},\mathrm{high}}& =T_{\mathrm{high}}·(1+\beta ·S_{\mathrm{LLM},r}),\hfill \end{array}$$

(34)

$$\begin{array}{cc}\hfill T_{\mathrm{LLM},\mathrm{low}}& =T_{\mathrm{low}}·(1+\beta ·S_{\mathrm{LLM},r}).\hfill \end{array}$$

(35)

7.4. Edge Refinement

Edges are refined using non-maximum suppression and hysteresis thresholding. The final edge map is defined as:

$$G_{\mathrm{LLM}}(i,j)=\left\{\begin{array}{cc}0,\hfill & \mathrm{if}{G}_{\mathrm{LLM}}(i,j)<T_{\mathrm{LLM},\mathrm{high}},\hfill \\ G_{\mathrm{LLM}}(i,j),\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

(36)

Figure 2. Original Image.

Figure 2. Original Image.

Figure 3. Edges Detected by the Canny Detector (with LLM Augmentation).

Figure 3. Edges Detected by the Canny Detector (with LLM Augmentation).

8. LLM-Augmented Hough Transformation

The LLM-Augmented Hough Transformation integrates semantic importance $S_{\mathrm{LLM}}(x,y)$ derived from LLMs to enhance line detection. This ensures that key structural features, such as text and faces, are preserved during image resizing.

8.1. Mathematical Formulation

The Hough Transformation maps edge pixels $(x,y)$ to Hough space. For each pixel, the radial distance r is computed as:

$$r=xcos\left(\theta \right)+ysin\left(\theta \right),$$

(37)

where $\theta$ is the angle of the line in Hough space.

In the LLM-Augmented Hough Transformation, the accumulator $A(r,\theta )$ is updated as:

$$A(r,\theta )\leftarrow A(r,\theta )+S_{\mathrm{LLM}}(x,y),$$

(38)

where:

• $A(r,\theta )$: Accumulator value for the line parameterized by $(r,\theta )$.
• $S_{\mathrm{LLM}}(x,y)$: Semantic importance score computed for pixel $(x,y)$.

8.2. Semantic Score Calculation

The semantic score $S_{\mathrm{LLM}}(x,y)$ is computed as:

$$\begin{array}{cc}\hfill h\left(I\right)& =\mathrm{VisionEncoder}\left(I\right),\hfill \end{array}$$

(39)

$$\begin{array}{cc}\hfill q\left(D\right)& =\mathrm{TextEncoder}\left(D\right),\hfill \end{array}$$

(40)

$$\begin{array}{cc}\hfill {\xi }_{x,y}& =\mathrm{PositionalEmbedding}(x,y),\hfill \end{array}$$

(41)

$$\begin{array}{cc}\hfill S_{\mathrm{LLM}}(x,y)& =\mathrm{Attention}(h\left(I\right),q\left(D\right),{\xi }_{x,y}).\hfill \end{array}$$

(42)

8.3. LLM-Augmented Hough Transformation Algorithm

Algorithm 1:LLM-Augmented Hough Transformation

Require: I, D: Input image I, optional description D. Ensure:  $\mathcal{L}$: Detected lines with semantic weighting.  1: $\mathcal{E}\leftarrow \mathrm{EdgeDetector}\left(I\right)$  2: $A(r,\theta )\leftarrow 0$  3: for$(x,y)\in \mathcal{E}$do  4:     $S_{\mathrm{LLM}}(x,y)\leftarrow \mathrm{Atten}(h\left(I\right),q\left(D\right),{\xi }_{x,y})$  5:     for $\theta \in [0,\pi ]$ do  6:         $r\leftarrow xcos\left(\theta \right)+ysin\left(\theta \right)$  7:         $A(r,\theta )\leftarrow A(r,\theta )+S_{\mathrm{LLM}}(x,y)$  8:     end for  9: end for 10: $\mathcal{P}\leftarrow \mathrm{PeakDetector}\left(A\right)$ 11: $\mathcal{L}\leftarrow \mathrm{MapToImageSpace}\left(\mathcal{P}\right)$

8.4. Threshold Adaptation

To refine line detection, thresholds for peak detection are dynamically adjusted using the average semantic score $S_{\mathrm{avg}}(r,\theta )$:

$$T_{\mathrm{LLM}}(r,\theta )=T_{\mathrm{base}}·(1+\beta ·S_{\mathrm{avg}}(r,\theta )),$$

(43)

where:

• $T_{\mathrm{LLM}}(r,\theta )$: Adaptive threshold for $(r,\theta )$.
• $S_{\mathrm{avg}}(r,\theta )$: Average semantic score for lines contributing to $(r,\theta )$.
• $\beta$: Scaling factor controlling semantic influence.

Figure 4. Original Image.

Figure 4. Original Image.

Figure 5. Lines Detected by LLM-Augmented Hough Transformation.

Figure 5. Lines Detected by LLM-Augmented Hough Transformation.

Figure 6. Prominent Lines Detected by LLM-Augmented Hough Transformation.

Figure 6. Prominent Lines Detected by LLM-Augmented Hough Transformation.

9. LLM-Augmented Absolute Energy Equation

The LLM-Augmented Absolute Energy Equation refines energy calculations by integrating semantic weights $S(x,y)$ derived from LLMs. This ensures seam carving preserves critical features such as text and faces.

9.1. Semantic Weighting

The LLM assigns a semantic score $S(x,y)$ to each pixel:

$$S(x,y)=\mathrm{LLM}(I,(x,y),\mathrm{context}),$$

(44)

where I is the input image and context includes features like object and text importance. $S(x,y)\in [0,1]$ scales pixel importance, with higher values for semantically significant regions.

9.2. Gradient Refinement

The original energy gradient:

$$e\left(I\right)=\left|{\displaystyle \frac{\partial I}{\partial x}}\right|+\left|{\displaystyle \frac{\partial I}{\partial y}}\right|$$

is modified as:

$$e_{\mathrm{LLM}}\left(I\right)=\left(\left|{\displaystyle \frac{\partial I}{\partial x}}\right|·S_x(x,y)\right)+\left(\left|{\displaystyle \frac{\partial I}{\partial y}}\right|·S_y(x,y)\right),$$

(45)

where $S_x(x,y)$ and $S_y(x,y)$ are direction-specific weights computed by the LLM.

9.3. Cumulative Energy Update

The cumulative energy function integrates $S(x,y)$ into the seam carving process:

$$\begin{array}{cc}\hfill e_{\mathrm{LLM}}(i,j)& =e(i,j)+|e(i,j+1)-e(i,j)|·S_x(i,j)\hfill \\ \hfill & +|e(i+1,j)-e(i,j)|·S_y(i,j)\hfill \\ \hfill & +min\{C_L(i,j)+e(i-1,j-1),\hfill \\ \hfill & C_R(i,j)+e(i-1,j+1),\hfill \\ \hfill & C_U(i,j)+e(i-1,j)\}.\hfill \end{array}$$

(46)

Here, $C_L,C_R,C_U$ are adjusted by $S(x,y)$ to prioritize semantically significant pixels.

10. LLM-Augmented Dual Gradient Energy Equation

The proposed LLM-Augmented Dual Gradient Energy Equation to refine edge detection by dynamically adjusting numerical differentiation and gradient computation. LLMs provide context-aware corrections for each computational step.

10.1. Numerical Differentiation with LLM Adjustments

Taylor expansions are dynamically adjusted with LLM corrections. The forward expansion is expressed as:

$$f(x+\Delta x)=f\left(x\right)+\Delta x{f}^{\prime }\left(x\right)+{\displaystyle \frac{\Delta x^2}{2}}{f}^{\prime \prime }\left(x\right)+{\Delta }_{LLM},$$

(47)

where ${\Delta }_{LLM}$ includes context-aware corrections predicted by the LLM. Similarly, for the backward expansion:

$$f(x-\Delta x)=f\left(x\right)-\Delta x{f}^{\prime }\left(x\right)+{\displaystyle \frac{\Delta x^2}{2}}{f}^{\prime \prime }\left(x\right)+{\Delta }_{LLM}.$$

(48)

LLMs adapt $\Delta x$ dynamically based on local image gradients and refine higher-order terms to reduce numerical error.

10.2. Gradient Approximation with Adaptive Refinements

Gradient approximations in the x- and y-directions incorporate corrections from LLMs:

$$f_x(x,y)\approx {\displaystyle \frac{f(x+\delta ,y)-f(x,y)}{\delta }}+\Delta f_x,$$

(49)

$$f_y(x,y)\approx {\displaystyle \frac{f(x,y+\delta )-f(x,y)}{\delta }}+\Delta f_y.$$

(50)

Here, $\Delta f_x$ and $\Delta f_y$ are LLM-predicted corrections based on local edge strength and texture complexity. The LLM also adapts $\delta$ to handle regions with high-gradient variations.

10.3. Energy Calculation with LLM Refinements

The energy of a pixel is computed using the LLM-enhanced gradients for each RGB channel. For the x-direction:

$${\Delta }_x^2(x,y)=\sum _{C\in \{R,G,B\}}{\left(C_x(x,y)+\Delta C_x(x,y)\right)}^2,$$

(51)

and similarly for the y-direction:

$${\Delta }_y^2(x,y)=\sum _{C\in \{R,G,B\}}{\left(C_y(x,y)+\Delta C_y(x,y)\right)}^2.$$

(52)

The total energy is:

$$E(x,y)=\sqrt{{\Delta }_x^2(x,y)+{\Delta }_y^2(x,y)}.$$

(53)

LLM contributions include predicting $\Delta C_x(x,y)$ and $\Delta C_y(x,y)$ to improve accuracy and dynamically adjusting channel weights for better feature preservation.

11. Result Evaluation

The evaluation of the proposed LLM-augmented methods focuses on three primary dimensions: semantic preservation, visual quality, and computational efficiency. Each method’s contribution to image resizing is assessed quantitatively and qualitatively, providing a comprehensive understanding of its effectiveness.

11.1. Evaluation Metrics

The following metrics are used to evaluate all methods:

• Semantic Preservation ($\mathcal{S}$): Measures the alignment of detected features or preserved regions with semantically significant areas, as defined by the LLM:

  $$\mathcal{S}={\displaystyle \frac{{\sum }_{i\in {\mathcal{F}}_{\mathrm{detected}}}{S}_{\mathrm{LLM}}\left(i\right)}{{\sum }_{i\in {\mathcal{F}}_{\mathrm{ground}}\mathrm{truth}}{S}_{\mathrm{LLM}}\left(i\right)}},$$

  (54)

  where $S_{\mathrm{LLM}}\left(i\right)$ is the semantic importance score for feature i.
• Visual Quality ($\mathcal{V}$): Evaluates the perceptual quality of resized images using metrics such as PSNR (Peak Signal-to-Noise Ratio) and SSIM (Structural Similarity Index):

  $$\begin{array}{cc}\hfill {\mathcal{V}}_{\mathrm{PSNR}}& =10{log}_{10}\left({\displaystyle \frac{{\mathrm{MAX}}^2}{\mathrm{MSE}}}\right),\hfill \end{array}$$

  (55)

  $$\begin{array}{cc}\hfill {\mathcal{V}}_{\mathrm{SSIM}}& ={\displaystyle \frac{(2{\mu }_x{\mu }_y+C_1)(2{\sigma }_{xy}+C_2)}{({\mu }_x^2+{\mu }_y^2+C_1)({\sigma }_x^2+{\sigma }_y^2+C_2)}}.\hfill \end{array}$$

  (56)
• Computational Efficiency ($\mathcal{C}$): Measures the average runtime per image:

  $$\mathcal{C}={\displaystyle \frac{\mathrm{Total}\mathrm{Runtime}}{\mathrm{Number}\mathrm{of}\mathrm{Images}}}.$$

  (57)

11.2. Results and Discussion

The results, including accuracy metrics and error rates for each sub-experiment, are provided below. The experiment revealed that LLM-augmented resizing methods led to superior performance in image classification, particularly in cases where maintaining fine image details was critical.

Figure 7. Error and accuracy of a sub-experiment showing the improvements from LLM-augmented methods.

Figure 7. Error and accuracy of a sub-experiment showing the improvements from LLM-augmented methods.

Illustrative examples of images processed by different methods, including LLM-augmented techniques, are shown below, highlighting the visual differences in the resized images and how LLM-augmented methods contribute to better feature preservation.

Figure 8. Image processed by the LLM-Bicubic Method.

Figure 8. Image processed by the LLM-Bicubic Method.

Figure 9. Image processed by the LLM-Absolute Energy Method.

Figure 9. Image processed by the LLM-Absolute Energy Method.

Figure 10. Image processed by the LLM-Canny Method.

Figure 10. Image processed by the LLM-Canny Method.

Figure 11. Image processed by the LLM-Dual Gradient Energy Method.

Figure 11. Image processed by the LLM-Dual Gradient Energy Method.

Figure 12. Image processed by the LLM-Hough Transformation Method.

Figure 12. Image processed by the LLM-Hough Transformation Method.

Figure 13. Image processed by the LLM-LC Method.

Figure 13. Image processed by the LLM-LC Method.

11.3. Conclusion

This experiment shows the significant improvements by LLM-augmented methods in image resizing. LLMSeamCarver can preserve finer image details, resulting in improved performance for semantic preservation, visual quality and computational efficiency.