Enhanced Image Retrieval Using Multiscale Deep Feature Fusion in Supervised Hashing

1. Introduction

The surge in high-dimensional multimedia data, driven by advancements in computer networks and social media platforms, underscores the need for efficient storage and retrieval solutions [1,2,3]. Approximate Nearest Neighbor (ANN) search [4] has emerged as a pivotal area of study in computer vision and information retrieval, a technique essential for reducing storage requirements and improving search efficiency in high-dimensional spaces. Hashing [5] has garnered considerable attention as a potent strategy within ANN search, which transforms high-dimensional data into compact binary codes while preserving spatial relationships between data points. Deep hashing techniques [6,7,8,9] have been devised to concurrently learn visual features and binary hash codes, enriching encoded information with semantic context. Deep hashing approaches have emerged as powerful tools for simultaneous feature learning and binary code generation. Unlike conventional hashing methods that rely on independently trained hash functions and quantization algorithms, deep hashing techniques adopt an end-to-end framework to extract semantic representations and construct binary hash codes. These end-to-end frameworks enable the cohesive construction of semantic representations and binary hash codes, surpassing traditional hashing techniques that rely on separate hash functions and quantization steps. While recent advancements in deep hashing have shown promise in information retrieval [10,11,12,13,14], there remains a need for further improvements, particularly in retaining local structural details within images. Most deep hashing approaches emphasize high-level features from fully connected (FC) layers [15,16,17], leading to a dearth of local feature information due to the global nature of these representations. Integrating multi-level features that capture both local and global details has shown promise for enhancing retrieval accuracy [18,19,20]. Despite attempts to fuse multi-feature representations, existing methods often fall short of achieving true end-to-end compatibility between feature representation and binary hash coding.

To address these limitations and to effectively harness the complementary nature of deep multi-scale features, this paper introduces Multiscale Deep Feature Fusion for High-Precision Image Retrieval through Supervised Hashing (MDFF-SH). Leveraging ResNet50 [21], convolutional multi-scale features are aggregated from images of varying sizes and fused within corresponding convolutional layers to yield robust representations. Inspired by the feature pyramid network [22], this fusion process incorporates top-down pathways and lateral connections, enabling exploration of both top-layer semantic and bottom-layer spatial features (Figure 1). Moreover, the MDFF-SH adapts hashing results based on different scale features, enhancing retrieval recall without sacrificing precision. In summary, the key contributions of this paper are as follows:

• Dual-Scale Approach: We propose a dual-scale approach that considers both feature and image sizes to preserve semantic and spatial details. Moreover that compensates for the loss of high-level features and ensures the generated hash codes are more discriminative and informative.
• Multi-Scale Feature Fusion: MDFF-SH learns hash codes across multiple feature scales and fuses them to generate final binary codes, enhancing retrieval performance.
• End-to-End Learning: Our MDFF-SH model integrates joint optimization for feature representation and binary code learning within a unified deep framework.
• Superior Performance: Extensive experiments on three well-known datasets demonstrate that MDFF-SH surpasses state-of-the-art approaches in retrieval performance.

2. Related works

Hashing techniques have gained significant popularity in image retrieval due to their minimal storage requirements and fast processing capabilities [23,24]. The primary purpose of hashing is to map high-dimensional data into low-dimensional hash codes, ensuring that similar data points have minimal Hamming distances while dissimilar points have maximized distances.

Hashing methods are categorized into supervised [25,26] and unsupervised [27,28,29,30,31] approaches, based on the use of labeled data. Unsupervised hashing methods [32,33,34,35] focus on learning hash functions using unlabeled training samples to transform input images into binary codes. Locality-Sensitive Hashing (LSH) [36] is among the most well-known unsupervised methods, with other significant approaches like Spectral Hashing (SH) [33] and Iterative Quantization (ITQ) [34] being prominent in the field.

In contrast, supervised hashing techniques leverage labeled data to learn hash functions, often yielding higher accuracy compared to unsupervised methods. Supervised Hashing with Kernels (KSH) [37] is notable for employing kernel methods to create nonlinear hash functions. Minimal Loss Hashing (MLH) [38] uses structured SVMs to define an objective for learning hash functions. Supervised Discrete Hashing (SDH) [28] refines the objective function to produce high-quality hash codes without relaxation.

The emergence of deep neural networks has propelled the development of deep hashing algorithms [12,16,39,40,41,42,43,44], which outperform traditional methods by using rich feature representations. Pairwise and triplet-based similarity preservation are common strategies in these methods to utilize label information. CNN-based Hashing (CNNH) [16] extracts features using CNNs but separates feature learning from hash function training, limiting feedback integration. Deep Pairwise-supervised Hashing (DPSH) [12] uses a Bayesian approach to model relationships between pairwise labels and hash codes, optimizing this relationship for better learning outcomes. HashGAN [39] employs a Wasserstein GAN to generate hash codes while utilizing pairwise similarities within a Bayesian framework. Zhuang et al. [40] developed a binary CNN classifier leveraging triplet loss to maintain semantic relationships. Deep Triplet Quantization (DTQ) [41] incorporates triplet-based quantization in a supervised learning framework for joint optimization of quantization and feature learning. In Supervised Semantics-Preserving Hashing (SSDH) [42], hash functions are embedded as a fully connected layer, with training focused on minimizing classification error. Wang et al. [43] offered a comprehensive framework for distance-preserving linear hashing extended to deep learning, where the fully connected layer’s features support hashing. Shen et al.’s Similarity-Adaptive Deep Hashing (SADH) [44] uses outputs from fully connected layers to refine a similarity graph matrix for enhanced hash code learning.

Traditional approaches often rely on high-level features, typically from the final fully connected layers. However, capturing diverse features for a more comprehensive representation is crucial. Multi-level image retrieval methods address this need. Lin et al. introduced Discriminative Deep Hashing (DDH) [14], integrating end-to-end learning and multi-scale feature extraction from convolution-pooling layers. Yang et al. [45] developed Feature Pyramid Hashing (FPH), a dual-pyramid framework for learning detailed and semantic features for fine-grained retrieval. Redaoui et al. proposed Deep Feature Pyramid Hashing (DFPH) [46] for leveraging multi-level visual and semantic data, and Deep Supervised Hashing with Multiscale Feature Fusion (DSHFMDF) [47], which extracts and combines multiscale features from various convolutional layers for robust image retrieval. Ng et al. [48] introduced Multi-Level Supervised Hashing (MLSH), which separately trains tables at different feature levels to enhance both structural and semantic representation.

2.1. Problem Definition

Let $X={\left\{x_i\right\}}_{i=1}^N\in {\mathbb{R}}^{d\times N}$ represent a training dataset with N images, where $Y={\left\{y_i\right\}}_{i=1}^N\in {\mathbb{R}}^{K\times N}$ are the associated ground truth labels for the $x_i$ samples, and K denotes the number of classes. To express the semantic similarities between images, we use a pairwise label matrix $S=\left\{s_{ij}\right\}$, where $s_{ij}\in \left\{0,1\right\}$ indicates whether images $x_i$ and $x_j$ are semantically related $s_{ij}=1$ or not $s_{ij}=0$. The objective of deep hashing is to learn a function $f:x\mapsto B\in {\left\{-1,1\right\}}^L$, which maps each input image $x_i$ to a binary code $b_i\in {\left\{-1,1\right\}}^L$, where L represents the length of the binary code.

2.2. Model Architecture

The architecture of the proposed MDFF-SH model, depicted in Figure 1, is structured to achieve high-efficiency and high-precision image retrieval through five main components: (1) feature extraction, (2) feature reduction, (3) feature fusion, (4) hash coding, and (5) classification. This modular approach ensures a cohesive understanding of each component and how they contribute to the model’s overall functionality.

Figure 1. Enhanced Image Retrieval Using Multiscale Deep Feature Fusion in Supervised Hashing (MDFF-SH).

Figure 1. Enhanced Image Retrieval Using Multiscale Deep Feature Fusion in Supervised Hashing (MDFF-SH).

• Feature Extraction: The initial feature extraction stage is crucial for gathering informative details from the input image. In MDFF-SH, the ResNet50 network serves as the backbone due to its capability to capture complex and distinguishing image features. Each layer in ResNet50 is designed to capture image details at increasing levels of abstraction, making it an ideal foundation for extracting both structural and semantic features.
  MDFF-SH systematically collects features from distinct levels of ResNet50. This includes low-level features, capturing fine details such as edges and textures, and high-level features, encapsulating semantic attributes. This multi-level approach ensures that the image representation integrates both granular details and overall semantic meaning.
• Multiscale Feature Focus: The model’s multiscale feature extraction focuses on layers from several convolutional blocks—specifically, the final layers of ‘conv3’, ‘conv4’, and ‘conv5’ blocks, along with the fully connected layer fc1. Lower-level layers like ‘conv1’ and ‘conv2’ are excluded to optimize memory usage, as their semantic contribution is limited. The selected layers effectively capture a balanced mix of structural and semantic information, providing a comprehensive representation of the image that includes both low- and high-level characteristics.
• Feature Reduction: After extraction, the dimensionality of the multiscale features is reduced to retain discriminative power without excessive computational overhead. Using a 1x1 convolutional kernel, the model combines features across levels in a linear manner, creating a streamlined yet rich representation. This step enhances the depth and robustness of the features while minimizing redundancy.
• Feature Fusion: In the fusion stage, the reduced features from different levels are combined to produce a unified representation. By merging both low- and high-level information, the fusion layer enables the model to construct an image representation that captures local structures alongside global context. This fusion provides a robust basis for generating binary codes that reflect a detailed and semantically rich image profile.
• Hash Coding: To generate the final hash codes, the fused feature representation undergoes nonlinear transformations through hash layers, each of which outputs binary codes of the desired length L. This transformation ensures that the binary codes retain the core characteristics of the images in a compact and retrieval-optimized format.
• Classification: The classification layer, which corresponds to the number of classes in the dataset, assigns the generated hash codes to specific image categories. This final component allows MDFF-SH to distinguish among classes based on learned binary representations, reinforcing the network’s retrieval effectiveness.
  Through this structured architecture presented in Table 1, MDFF-SH captures both local and global image information, resulting in a powerful and compact feature representation that is tailored to high-precision image retrieval.

After extracting features from multiple scales, we employ a 1x1 convolutional kernel to reduce dimensionality while preserving discriminative information. This process enhances feature depth and robustness and eliminates redundancy.

Subsequently, a fusion layer composed of 1024 nodes integrates these multi-scale features, combining low-level structural details with high-level semantic information. This fusion step creates a comprehensive image representation that balances fine-grained local structures with broader contextual understanding.

To generate compact binary hash codes, we apply a nonlinear mapping through multiple hash layers, each with L nodes. This nonlinear transformation effectively encapsulates key image characteristics into binary codes. The concatenated hash code representation is further refined in the final hashing layer to ensure semantic integrity and discriminative power.

Finally, a classification layer with neurons corresponding to the number of classes is employed to categorize images based on their learned representations. The discriminative nature of the hash codes enables accurate image classification.

By integrating multi-scale features and a well-structured architecture, our model generates diverse and informative hash codes. These hash codes effectively capture both local details and global context, leading to improved retrieval performance and accurate image classification.

2.3. Loss Functions and Learning Rule

To ensure that the generated hash codes effectively preserve semantic similarity, our MDFF-SH method combines three distinct loss functions: pairwise similarity loss, quantization loss, and classification loss. These losses are harmonized to support efficient and effective training.

2.3.1. Pairwise Similarity Loss

The MDFF-SH method is designed to maintain similarity between pairs of input samples within Hamming space. Pairwise similarity is evaluated by calculating the inner product between hash codes $b_i$ and $b_j$ , defined as dist $dis{t}_H\left(b_i,b_j\right)={\displaystyle \frac{1}{2}}{b}_i^T{b}_j$ . Given a set of binary codes $B={\left\{b_i\right\}}_{i=1}^N$ and pairwise labels $S=\left\{s_{ij}\right\}$, the probability of the pairwise labels is represented as:

$$p\left(s_{ij}\right|B)=\left\{\begin{array}{cc}\sigma \left(w_{ij}\right)\hfill & \mathrm{if}{s}_{ij}=1\hfill \\ 1-\sigma \left(w_{ij}\right)\hfill & \mathrm{if}{s}_{ij}=0\hfill \end{array}\right.$$

(1)

where $\sigma \left(w_{ij}\right)={\displaystyle \frac{1}{1+e^{-w_{ij}}}}$ and $w_{ij}={\displaystyle \frac{1}{2}}{b}_i^T{b}_j$

This formulation implies that a larger inner product $\langle b_i,b_j\rangle$ corresponds to a smaller $dis{t}_H\left(b_i,b_j\right)$ and a higher value of $p\left(1|b_i,b_j\right)$. When $s_{ij}=1$, the binary codes $b_i$ and $b_j$ are considered similar.

The optimization problem then becomes minimizing the negative log-likelihood over labels in S, resulting in

$$J_1=-logp\left(S|B\right)=-\sum _{s_{ij}\in S}(s_{ij}{w}_{ij}-log(1+e^{w_{ij}}))$$

(2)

This objective aims to minimize the Hamming distance between similar samples while maximizing the distance between dissimilar samples, aligning with the principles of similarity-based hashing.

2.3.2. Quantization Loss

In practical applications, binary hash codes are commonly used for measuring similarity. However, optimizing discrete hash codes directly within a neural network can be challenging. To address this, we employ a continuous approximation for hash coding. Let $u_i$ denote the output of the hash layer, with $b_i$ defined as $b_i=sgn\left(u_i\right)$. To minimize the gap between continuous and discrete representations, we introduce quantization loss as a secondary objective:

$$J_2=\sum _{i=1}^Q{|b_i-u_i|}_2^2$$

(3)

where Q represents the mini-batch size.

2.3.3. Classification Loss

To support robust learning of multiscale features across the network, we employ cross-entropy loss for classification, which helps the model correctly categorize input samples. The classification loss is given by:

$$J_3=-\sum _{i=1}^Q\sum _{k=1}^K{y}_{i,k}log\left(p_{i,k}\right)$$

(4)

where $y_{i,k}$ denotes the true label and $p_{i,k}$ represents the softmax output of the i-th training sample for the k-th class.

In conclusion, the total loss function combines the pairwise similarity, quantization, and classification losses, as follows:

$$J=J_1+\beta J_2+\gamma J_3$$

(5)

where $\beta$ and $\gamma$ are balancing parameters that control the contributions of quantization and classification losses, respectively.

3. Experiments

This section evaluates the performance of MDFF-SH and its variations on three extensive public datasets: CIFAR-10, NUS-WIDE, and MS-COCO. Our objective is to demonstrate the effectiveness of the proposed method compared to several leading hashing approaches. We begin with an overview of these datasets and follow with the experimental setup. Section 3.3 details the evaluation metrics and baseline methods. Finally, we present the results, including a comparative analysis with state-of-the-art hashing techniques.

3.1. Datasets

CIFAR-10,krizhevsky2009learning: This dataset consists of 60,000 color images across ten object classes, with each class containing 6,000 images sized 32×32 pixels. Following the protocol in [50], we randomly select 1,000 images (100 per class) as the query set, with the remaining images serving as the database. From the database, we sample 5,000 images (500 per class) as the training set.

NUS-WIDE,chua2009nus: Comprising 123,287 color images (40,504 validation images and 82,783 training images), this dataset includes images labeled with one or more of 80 categories. For our experiments, we randomly select 5,000 images as the query set, 10,000 as the training set, and use the remaining images as the database.

MS-COCO,lin2014microsoft: is a dataset consists 123287 colour images (40504 validation and 82783 training). Each image is labelled by one or more of 80 categories. We randomly selected 5,000 images as queries points and 10000 images as the training datasets. While the rest of the images as the database.

3.2. Experimental Settings

The MDFF-SH method is implemented using the PyTorch deep learning framework, and we initialize the network parameters with the ResNet50 convolutional model pretrained on the ImageNet dataset [53]. All experiments are conducted using the RMSProp optimizer [54] with a learning rate of 1 × 10⁻⁵ and a batch size of 32. Hyperparameters are set as follows: $\alpha =0.01$ and $\beta =0.1$.

3.3. Evaluation Metrics and Baselines

To assess the performance of our image retrieval method and facilitate comparisons with alternative approaches, we use the following metrics:

• Mean Average Precision (MAP) results,
• Precision-Recall (PR) curves,
• Precision at top retrieval levels (P@N), and
• Precision within a Hamming radius of 2 (P@H ≤ 2).

The MDFF-SH method is compared with a selection of traditional and state-of-the-art methods, including five unsupervised methods: LSH [4], SH [33], SGH [55], ITQ [34], PCAH [56], as well as two supervised hashing methods: SDH [28] and KSH [37]. Additionally, we include nine deep supervised hashing methods: CNNH [16], DNNH [8], DCH [57], DHN [7], HashNet [58], DHDW [59], DPH [60], LRH [50], and MFLH [61]. For multi-label datasets such as MS-COCO and NUS-WIDE, two samples are considered similar if they share one or more semantic labels.

3.4. Results

Table 2 presents a comparison of MAP results for our method and competing hashing methods on CIFAR-10 and NUS-WIDE with hash code lengths of 12, 24, 32, and 48 bits. Our MDFF-SH method consistently outperforms all other methods. Specifically, compared to the best traditional hashing method, SDH [28], MDFF-SH achieves an average MAP improvement of 52.7% and 23.55% on CIFAR-10 and NUS-WIDE, respectively. Deep hashing methods generally perform better than classical methods due to their ability to generate more robust feature representations. For CIFAR-10 and NUS-WIDE, MDFF-SH delivers an average MAP increase of 9.58% and 2.95%, respectively, over the second-best method, MFLH [61], across all hash code lengths. For example, the MAP values of MDFF-SH at different lengths are enhanced by 52.6%, 52.5%, 53.3%, and 52.4%, respectively, compared to the SDH method. The MAP of the MDFF-SH method is also significantly improved compared to the deep hash method. These results indicate the capability of MDFF-SH to produce high-quality hash codes for efficient image retrieval.

Table 2 shows the performance of MDFF-SH on the MS-COCO dataset. MDFF-SH achieves superior retrieval performance at all code lengths compared to all baseline methods. As a multi-label dataset, MS-COCO presents a more complex semantic structure than single-label datasets, which poses a greater challenge for maintaining semantic integrity in hash codes. For example, the MDFF-SH method improves the MAP values over different lengths of hash codes by 7.4%, 11.7%, 14.1%, and 15.2%, respectively, compared with the DCH method. Nonetheless, MDFF-SH achieves the best results, underscoring the effectiveness and robustness of the proposed approach for high-precision image retrieval in complex datasets.

Figure 2a and Figure 3a present the precision curves for P@H=2, demonstrating that our MDFF-SH method consistently outperforms other techniques by achieving the highest precision within this Hamming radius. Although a slight decline in P@H=2 performance is observed as the code length increases, MDFF-SH maintains strong retrieval accuracy, indicating its ability to focus on relevant points within a Hamming radius of 2, even with longer hash codes.

Additionally, Figure 2b,c and Figure 3b,c compare the Precision-Recall and precision-at-top-results performance of MDFF-SH with other methods. In particular, Figure 2c and Figure 3c show that MDFF-SH achieves the highest precision with 48-bit codes across varying numbers of returned samples, especially in the range of 100 to 1,000. Furthermore, Figure 2b and Figure 3b illustrate that MDFF-SH achieves notably high precision at low recall levels—a crucial feature for precision-first retrieval systems widely used in practical applications. Overall, these results underscore the superior performance of MDFF-SH compared to other methods evaluated.

Figure 2a and Figure 3a display the precision curves for P@H=2, clearly showing that our MDFF-SH method outperforms other approaches by achieving the highest precision within this Hamming radius. While a slight decrease in P@H=2 performance occurs as code length increases, this result highlights MDFF-SH’s effectiveness in concentrating on relevant points within a Hamming radius of 2, even with longer hash codes.

In summary, our MDFF-SH method consistently outperforms the compared methods across various evaluation metrics, underscoring its superiority in image retrieval tasks. To visually illustrate its effectiveness in eliminating irrelevant images, we present Figure 4, showcasing the retrieval accuracy of different image categories in the CIFAR-10 dataset using MDFF-SH with 48-bit binary codes. The figure features query images in the first column, while the subsequent columns display images retrieved using MDFF-SH. This example reinforces our approach’s capability to precisely retrieve pertinent images, further substantiating its practical utility.

In summary, our MDFF-SH method consistently surpasses the compared techniques across multiple evaluation metrics, affirming its superior performance in image retrieval tasks. To visually demonstrate its effectiveness in filtering out irrelevant images, Figure 4 presents the retrieval accuracy for various image categories within the CIFAR-10 dataset using MDFF-SH with 48-bit binary codes. In this figure, query images are shown in the first column, followed by images retrieved through MDFF-SH in the subsequent columns. This example highlights our method’s ability to accurately retrieve relevant images, underscoring its practical value in real-world applications.

3.5. Ablation Studies

(1)

Ablation Studies on Multi-Level Image Representations for Enhanced Hash Learning: To investigate the impact of multi-level image representations on hash learning, we conducted ablation studies. Unlike many existing methods that primarily focus on semantic information extracted from the final fully connected layers, we explored the contribution of structural information from various network layers.

Table 4 presents the retrieval performance on the CIFAR-10 dataset using different feature maps. We observed that features from the fc1 layer yielded the highest mAP of 75.8%, emphasizing the importance of high-level semantic information. However, using features from Conv 3-5 resulted in an average mAP of 62.5%, highlighting the significance of low-level structural details. Our proposed MDFF-SH approach outperformed all other configurations, achieving an average mAP of 85.5%, demonstrating the effectiveness of combining multi-scale features for enhanced retrieval performance.

(2)

Ablation Studies on the Objective Function: To assess the impact of different loss components in our objective function, we conducted ablation studies on the CIFAR-10 dataset using the MDFF-SH model. We evaluated the performance of the model when either the Pairwise Quantization Loss ($\beta =0$, MDFF-SH-J3) or the Classification Loss ($\gamma =0$, MDFF-SH-J2) was excluded. As shown in Table 5, the inclusion of both J2 and J3 resulted in an 8.55% performance improvement. This finding highlights the importance of both quantization loss, which minimizes quantization error, and classification loss, which preserves semantic information, for generating high-quality hash codes.

4. Conclusions and Future Work

This paper introduces a novel end-to-end framework, Multiscale Deep Feature Fusion for High-Precision Image Retrieval through Supervised Hashing (MDFF-SH), designed to generate robust binary codes. Our approach optimizes three key components: similarity loss, quantization loss, and semantic loss, to effectively integrate structural information into hash representations. By leveraging multiscale features, MDFF-SH achieves a balance between structural detail and retrieval accuracy, leading to improved recall and precision.

Extensive experiments on standard image retrieval datasets demonstrate the superior performance of MDFF-SH compared to state-of-the-art methods. In future work, we aim to extend this approach to medical imaging, where the presence of multi-scale objects could benefit significantly from our method’s ability to capture both fine-grained and coarse-grained details.

The scalability of our model makes it adaptable to various computer vision tasks, providing robust feature representations that have the potential to advance a wide range of applications.