Self-Supervised Foundation Model for Template Matching

1. Introduction

In computer vision, visual template detection is a technique used to identify and locate a pattern in a larger image. This usually involves comparing the image of the template on an input image at different positions to find its best match. This technique can be used for detection of familiar objects, image registration and object tracking tasks.

Despite the significant progress in the development of this technique, the solutions developed so far do not emphasize a sufficiently wide range of essential properties important for practice, such as generalization capability, real-time execution and easy retraining that does not require annotated data, etc.

Self-supervised models [1,2,3] are artificial neural networks designed to extract knowledge from visual data without relying on annotations. The lack of necessity for a large amount of annotated data is the reason for its increased popularity in recent years. As self-supervised models evolve, they continuously improve their generalization capabilities while being more data efficient.

This work presents a novel approach for self-supervised learning of a CNN designed for visual template detection. Our Self-TM models provide a high degree of generalization capabilities across various tasks, which classifies them as so-called “foundation” models dealing with high accuracy on data of a different modality, i.e., highly different from those used for training. If accuracy increase is needed, these foundation models can be easily and quickly fine-tuned on a small set of real data.

The applied approach in Self-TM can be attributed to “Siamese network-based trackers” [4,5,6,7,8,9,10,11,12,13] where typically the same model (encoder) is used to extract both image features of the search object and image features in which it is searched. Then, the features from the last layers are correlated using learnable cross-correlation [12,13], which result is then processed by one or multiple decoders (convolutional neural networks or transformers) [14,15,16,17,18,19]. Most often, the result obtained by the decoders is one or several regression and classification maps [12,13].

Based on this framework, the present work proposes highly efficient architecture, in addition to a simple yet accurate correlation approach. Combined with the intuitive training method, Self-TM can be easily fine-tuned on any type of images.

The novelty in the proposed Self-TM may be summarized by the following:

• The high degree of generalization eliminates the need to retrain the model with real data;
• If further training (fine-tuning) is still necessary, very few images of the real data are needed until the desired accuracy is reached;
• The specially designed encoder is trained on the task of locating a searched object, what differs from the standard approaches of using a network trained on some other tasks, most often classification or a more general one;
• The integrated correlation operator we use provides real information about the location of the searched object, rather than being a separate layer of the network [12,13] which has to be further trained and the result to be decoded by one or more neural networks. This approach has not been found in the literature so far;
• Hierarchical propagation of activations from the last to the first layer leads to precise localization while precluding the need to use an additional decoder that needs to be trained. The result is an extremely simplified and lightweight architecture, while providing high accuracy. This approach has not been found in the literature so far;
• Two step self-supervised training, involving two types of data augmentations: color augmentation, and color and geometric augmentations;
• Self-TM is rotationally invariant. In this work, the Self-TM models’ family is trained over the entire interval from -90 to +90 degrees.

2. Related Work

2.1. Hand-Crafted Methods

Template matching has a long history in the field of computer vision with many hand-crafted methods, mostly involving correlation-based techniques in which the searched template is slid over the target image, computing various similarity scores.

Traditional methods applied directly on pixel intensities, such as SSD (sum of squared differences) [20] and SAD (sum of absolute differences) [21] are very sensitive to noise, deformations and visual transformations of the object or region of interest, making them inapplicable to real tasks. Papageorgiou and Poggio (in 1998) introduced a novel method [22] that uses normalized cross-correlation (NCC), allowing a certain degree of robustness against illumination and contrast variations. Their approach has spawned later advances, for example with zero-mean NCC (ZNCC) [23] in pattern matching, emphasizing the importance of normalization.

Another type of methods, based on keypoints’ visual features such as SIFT [24], SURF [25] and ORB [26], which have emerged as an alternative to traditional template matching techniques, have wide popularity, because they can efficiently handle variations in scale, rotation, and perspective. Despite their accuracy fluctuations, they are used in various systems since they have high performance, good generalization capabilities on new data, and do not need high computational resources.

2.2. Learnable Methods

Driven by the idea of increasing the accuracy of template matching, in recent years the focus of development has mainly shifted to the development of two types of learnable methods: sparse and dense.

Sparse methods have gained popularity for their efficiency due to the use of keypoints features, resulting in reduced computational cost and improved robustness. For example, SuperPoint [27] is a keypoints extraction algorithm that introduces a trained CNN to detect keypoints and generate their features. SuperGlue [28] is focused on feature matching detection. It evaluates and aligns each pair of keypoints by constructing an association matrix based on their features and applies Sinkhorn [29] to determine the optimal match between them. LightGlue [30] shows significant progress with its efficient matching determination. It uses a small architecture, combining the ability to early stop inference when matches are already found, and exclude for later processing points that cannot be compared. OmniGlue [31] presents a framework using SuperPoint [27] to keypoints detection that attempts to improve the generalization properties by exploiting features from foundation model [32].

Dense methods [33,34,35,36] have demonstrated remarkable improvements in accuracy for similarity search between images, especially with recent advances introduced by the transformers [15,17,18,19]. Unlike sparse methods, they aim to establish correspondences for each pixel of the processed images, resulting in dense information flow. This approach is particularly useful in applications where understanding the entire image is critical. However, recent methods mainly emphasize accuracy, thereby increasing computational requirements to unacceptable levels, challenging even systems with significant GPU resources.

Self-TM offers a learnable dense approach for localizing a search template, taking advantage of the dense method’s high accuracy, while being characterized by a very efficient architecture that does not require high computational resources. Self-TM can also be integrated into a sparse-based framework such as OmniGlue [31] (described in section 4.3), increasing its accuracy and preserving the characteristic properties of this type of methods.

Although learnable methods have proven to perform well on their training data, they are characterized by a limitation in the similarity of the tasks in which they can be applied due to their limited generalization that is characteristic for the foundation models.

2.3. Foundation Models

Foundation models are pre-trained, data-rich, highly generalizable models designed to perform more than one task with sufficiently high accuracy. These models serve as a foundation for a variety of applications, allowing for fine-tuning or adaptation to specific tasks with less data and computational resources.

Key features of these models are: pre-training with large datasets covering a wide range of visual concepts, the ability to fine-tune on a small but specific corpus of data focused on a particular task, the ability to adapt to different tasks or subtasks, high robustness and generalization ensuring robust performance in different domains and tasks.

For example, DINO [32] and DINOv2 [37] are transformers trained by a self-supervised learning approach that allow semantic segmentation without relying on annotated data, making them invaluable for training on large unannotated image sets. DINOv2, as a successor, extracts higher quality features, providing improvements that allow it to solve more complex tasks with greater accuracy. Both methods use a self-distillation approach, where a student model learns from a teacher model of the same architecture. The teacher model is updated based on the student parameters using the exponential moving average technique. DINO and DINOv2 use “contrastive” learning [38], which encourages the model to produce similar embeddings under variations of the same image, by applying different color and geometric augmentations, while discriminating between different images.

SAM [39] and SAM2 [40] with their accuracy and generalization, revolutionized semantic image and video segmentation by requiring minimal annotations compared to traditional methods. In addition to images, the input for these frameworks also includes prompts in the form of point positions, a binary mask, rectangle coordinates, and even a textual description, allowing them to focus on a specific region to extract desired features and output.

SegGPT [41] is a segmentation model built on results from [42] that allows the model to adapt to different tasks using a minimum of examples. The model is useful for all segmentation tasks such as instance, object, and semantic segmentation. During training, it performs contextual coloring using a random coloring scheme (rather than specific colors) to identify segments by learning their contextual information, resulting in improved generalization.

Self-TM is a foundation model oriented towards learning high-quality hierarchical features incorporating localization properties. Similar to DINO [32] and DINOv2 [37], the Self-TM models’ family is trained using a self-supervised learning approach, not relying on annotated data, making them invaluable for training on large unannotated image sets. The localization properties provide high accuracy in tasks such as object detection, image registration, object tracking, sparse and/or dense image matching.

3. Theoretical Formulation of the Proposed Method

In this section, the setup of our proposed Self-Supervised Foundation Model for Template Matching (Self-TM) method, illustrated in Figure 1, is discussed.

Technical details such as the choice of architecture and layers, data, and training steps involved are explained in detail, as well as essential properties of the hierarchical features that are the overall goal of the present work, including properties for image templates localization without any annotations.

The next section (Section 4) presents the experimental results of Self-TM on data with different modalities (HPatches [43], MegaDepth [44] and ScanNet [45]) and a variety of tasks including template localization, patch verification, patch retrieval and image matching.

Section 5 presents details related to the development of the approach: the iterative learning approach; a comparison of Self-TM against the architecture on which it is based.

3.1. Model

A convolutional neural network is chosen since the proposed method involves hierarchical activations from the final to the first layer of the network where the deeper the layer in which the neuron is located, the larger receptive field it has, i.e. a larger region of the input pixels is covered.

The proposed method is independent of the CNN architecture and can be applied to models with different properties. Our chosen architecture is based on the recently introduced ConvNeXt architecture [14], which is similar to ResNet [16] but designed to compete in performance with the state-of-the-art vision transformers [18,19]. In addition, this architecture is one of the few effectively used in self-supervised learning.

Our model Self-TM uses the same blocks of ConvNeXt [14], but with modified filter size and stride, see Table 1. Down sampling blocks now use 3 × 3 filters and stride 3, which in terms of receptive field gives a neuron from layer N field of view with size 3 by 3 of layer N-1.

The layers used, following ConvNeXt [14], are a standard two-dimensional convolution (Conv2D), a linear layer (Linear), a normalization layer (Norm) [46] and a non-linear GELU activation function [47], see also Section 5.2.

The size of Self-TM Small is significantly compressed, containing only 13 million learnable parameters, which is significantly smaller than standard architectures (Table 2) used for encoders and decoders in template matching or tracking methods. This is given by the network design entirely focused on this type of tasks.

3.2. Data

ImageNet-1K Train [53] is used for training the Self-TM models because it has established as a standard for training image processing models, as well as because of the high diversity of classes (1000 classes) covering enough different visual concepts for achieving high generalization properties.

3.3. Training

The often-preferred Invariance-based [54] approach has been used to the training, where the idea is to learn similar features for compatible images and discriminative features for incompatible images. These compatible images are produced by applying random image augmentations modifying the input image such as: changing the contrast, blurring, etc. Using this learning approach, it is possible to obtain the so-called representation collapse, where the network’s output is constant regardless of the various input data applied.

The training is performed in two successive steps, which differ in the type of augmentations applied to the input images: color augmentations and color and geometric augmentations. This is necessary because geometric augmentations add increased variation to the data that the model cannot initially overcome. To overcome this, Self-TM is initially trained only on data with color augmentations applied. Once this training step is complete, the model continues its training, now adding geometric augmentations to the color augmentations, which are: perspective transform with factor up to 0.5, rotation from -90 to +90 degrees, and rescale with tolerance from -0.3 to +0.3.

Training starts with 15 epochs with color augmentations and continues with 30 epochs with color and geometric augmentations. 4 NVIDIA A5000 24GB graphics cards are used for the training, and an epoch of Self-TM Large takes about 6.5 hours.

The training steps are as follows:

• An input image I, is taken from an unannotated image database, to which a random crop $R$ is applied, and then resized by $S$, to 189 × 189 pixels yielding a “query” image, $S\left(R\left(I\right)\right)=Q$, see Figure 1;
• Another random crop $R$, is performed on $Q$, and then random image augmentations (color and/or geometric) $A$, modifying the input image to obtain a “template” image, $A\left(R\left(Q\right)\right)=T$, see Figure 1. This step can produce one or many different templates. In the Self-TM training, only two “template” numbers are used;
• The position $gt\_p$ (ground truth position), of the resulting “template” on the “query” image is stored (the coordinates of the red rectangle’s center on “query”, see Figure 1);
• “Query” and the two templates are fed as input to the Self-TM network, $f_{\theta }:\left(Q,T\right)\to (y,y^{\mathrm{\text{'}}})$, and the resulting feature maps from all layers are stored, $y=f_{\theta }\left(Q\right),{ y}^{\mathrm{\text{'}}}=f_{\theta }\left(T\right)$, ${(y,y^{\mathrm{\text{'}}})=\left\{\right(y}_n,y_n^{\mathrm{\text{'}}}\left)\right|n=1,\dots ,N\}$, where $N$ is the number of the network’s layers. In the current architecture the number of layers is 3, denoted above by: first, mid and last;
• A correlation operator $CORR\left(y,y^{\mathrm{\text{'}}}\right)$, is applied to every two corresponding feature maps sequentially starting from the deepest layer. Then the position of the maximum value (predicted position), ${pred\_p}_n=softmax\left(CORR\left(y_n,y_n^{\mathrm{\text{'}}}\right)\right)$, is found on its output (see “hierarchical activations propagation” in Figure 1);
• A mean squared error is calculated on every two corresponding feature maps, ${MSE}_n=MSE\left(y_n,y_n^{\mathrm{\text{'}}}\right)={\displaystyle \frac{1}{\left|D\right|}}\sum _D{{\left(R\right(y)}_n-y_n^{\mathrm{\text{'}}})}^2,D=def\left(y_n\right)\cap def\left(y_n^{\mathrm{\text{'}}}\right)$, where $D$ is the region of summation, i.e. the intersection of the two definition domains of the “template” feature map $y_n^{\mathrm{\text{'}}}$ and the “query” feature map ${R\left(y\right)}_n$, ${(y,y^{\mathrm{\text{'}}})=\left\{\right(y}_n,y_n^{\mathrm{\text{'}}}\left)\right|n=1,\dots ,N\}$. Multiplication is scalar, i.e. elementwise in $D$;
• Parameter optimization (gradient descent): by minimizing the errors obtained from ${MSE}_N,\dots ,{MSE}_1$, and the offset of the positions of $pred\_p_N,\dots ,{pred\_p}_1$, relative to the real positions of the template $gt\_p_N,\dots ,{gt\_p}_1$, which is possible due to compatibility of positions (in pixels).

3.4. Augmentation

Because geometric augmentations add increased variation to the data that the model cannot initially overcome, training is performed in two steps: training begins on data with applied color augmentations, and then it continues on data with both color and geometric augmentations.

3.4.1. Color augmentations

The applied color augmentations are based on the approach in [55], where:

• To obtain the “query” image are applied sequentially:

  ○

  Random crop: scale: from 0.1 to 0.9

  ○

  Rescale: 189 × 189 pixels

  ○

  Normalization: mean = [0.485, 0.456, 0.406], std = [0.228, 0.224, 0.225], computed on ImageNet
• To obtain the two “templates” on the “query” are applied sequentially:

  ○

  Random color jitter with independent probability* of 80%: brightness = 0.4, contrast = 0.4, saturation = 0.2, hue = 0.1

  ○

  Random grayscale with independent probability* of 20%

  ○

  For „template“ 1:

  ▪

  Random Gaussian blur: radius from 0.1 to 0.2

  ○

  For „template“ 2:

  ▪

  Random Gaussian blur with independent probability* of 10%: radius from 0.1 to 2.0

  ▪

  Random invert of all pixel values above a given threshold (Solarization) with independent probability* of 20%: threshold = 128
• Normalization: same as in “query”
• Random crop: scale from 0.14 to 0.85, ratio from 0.2 to 5.0

* Applying random operation with independent probability following the law of uniform distribution

3.4.2. Geometric Augmentations

Geometric augmentations (aligned by the center of the resulting cropping square) are applied on the “template” images only:

• Random square crop: scale from 0.14 to 0.45
• A randomly selected geometric augmentation is applied with a 50% probability:

  ○

  Random perspective transformation: distortion scale = 0.5

  ○

  Random rotation: degrees from -90 to +90

  ○

  Random rescale: factor from -0.7 to 1.3

Table 4. Visualization of random color and random geometric augmentations on random images from ImageNet-1K. For better visualization, the images are cropped in a 1:1 height-to-width ratio.

Image crop	Color and geometric augmentation

Table 4. Visualization of random color and random geometric augmentations on random images from ImageNet-1K. For better visualization, the images are cropped in a 1:1 height-to-width ratio.

Image crop	Color and geometric augmentation

3.5. Hierarchical Activations Propagation

One of the main novelty in the proposed framework is the hierarchical propagation of activations from the last to the first layer $pred\_p_N,\dots ,{pred\_p}_1$ (currently $pred\_p_{last},pred\_p_{mid},{pred\_p}_{first}$). This results in precise localization although not training an additional decoder is required (“hierarchical activations propagation” in Figure 1).

The localization of the “template” $T$, in a selected image $Q$ is performed by finding the maximum value (the highest value activation position visualized by a red square in Figure 1 and Figure 2) in the correlation operator’s output produced between any two corresponding layers starting from the deepest to the first, i.e. ${pred\_p}_{N,\dots ,1}=$ $softmax\left(CORR\left(y_{N,\dots ,1}, y_{N,\dots 1}^{\text{'}}\right)\right)$.

We start by finding the position of the maximum value (${pred\_p}_N$) in the last layer $N$, which is propagated to the previous layer $N$-1 as the position ${pred\_p}_{N-1}$. This new position (${pred\_p}_{N-1}$) is again maximum value but here selected from a limited region denoted as $RF$ (receptive field), of the maximum value of $N,$ ${pred\_p}_{N-1}=$ $softmax\left({RF}_{{pred\_p}_N}\left(CORR\left(y_{N-1}, y_{N-1}^{\text{'}}\right)\right)\right)$, see Figure 2. In this way, each subsequent layer refines the assumed position of $T$.

To detect the position of the maximum value, a softmax function [56] is used. It is a differentiable argmax function allowing the gradient to flow freely from the end to the beginning of the network during training:

$${softmax\left(z\right)}_i={\displaystyle \frac{e^{-\beta z_i}}{{\sum }_{j=1}^K{e}^{-\beta z_i}}} \mathrm{f}\mathrm{o}\mathrm{r}\mathrm{}i=1,\dots ,K, \mathrm{a}\mathrm{n}\mathrm{d} z=\left(z_1,\dots ,z_K\right)\in {\mathbb{R}}^K.$$

(1)

3.6. Correlation

Another novelty in the proposed framework is the integration of a static correlation operator in the network, the result of which provides information about the location of the searched object, instead of being a separate layer [12,13] that needs to be further trained and then its result decoded by one or more neural networks.

The correlation operator used here is normalized cross-correlation representing a dense measure of a pixel-wise similarity metric:

$$CORR\left(x,y\right)={\displaystyle \frac{\sum _{x^{\text{'}}{y}^{\text{'}}}(T\left(x^{\text{'}},y^{\text{'}}\right)\bullet I\left(x+x^{\text{'}}, y+y^{\text{'}}\right))}{\sqrt{\sum _{x^{\text{'}}{y}^{\text{'}}}{T\left(x^{\text{'}},y^{\text{'}}\right)}^2 }\sqrt{ \sum _{x^{\text{'}}{y}^{\text{'}}}{I\left(x+x^{\text{'}}, y+y^{\text{'}}\right)}^2}}},$$

(2)

where $(x^{\mathrm{\text{'}}},y^{\mathrm{\text{'}}})\in def\left(Q\right)\cap def\left(T\right)$, i.e. the summation is (only) over the intersection of the domains, of the input $I$ and the template $T$ positioned at the point $(x,y)$ of $I$.

The result of the correlation operator is a grayscale image, where the position of the template is the pixel with the highest intensity. In this case, the operator is not applied to pixel values, but to the activation values of the layers (feature maps).

3.7. Loss

Two terms $MSE$ and $CORR$ are used for computing the loss, $\mathcal{L}$ while training the network, $f_{\theta }$:

$${\mathcal{L}}_{MSE}={\displaystyle \frac{1}{N}}{\sum }_{n=1}^N\sqrt{MSE(y_n, y_n^{\text{'}})}, y_n\mathbb{R}, y_n^{\text{'}}\mathbb{R}$$

(3)

$${\mathcal{L}}_{CORR}={\displaystyle \frac{1}{N}}{\sum }_{n=1}^N\sqrt{{({pred\_p}_n-{gt\_p}_n)}^2}, {pred\_p}_n{Z}^2, {gt\_p}_n{\mathbb{R}}^2$$

(4)

$$\mathcal{L}={\displaystyle \frac{{\mathcal{L}}_{MSE}+{\mathcal{L}}_{CORR}}{2}},$$

(5)

where ${\mathcal{L}}_{MSE}$ is the average of the mean squared error ($MSE$) results for each layer, and ${\mathcal{L}}_{CORR}$is the average of the Euclidean distances between the detected maximum values position $pred\_p$, and the real template positions, $gt\_p$, for each layer.

3.8. Hyperparameters/Optimization

An AdamW optimizer [57] with a $learning\_rate=0.001$ and a regularization technique $weight\_decay={10}^{-6}$ is used in the loss calculation:

$$\mathcal{L}=\mathcal{L}+\underset{\_}{weight\_decay}*\underset{\_}{L2 norm of the weights}.$$

(6)

3.9. Prediction

A correlation operator $CORR\left(y,y^{\mathrm{\text{'}}}\right)$ is applied on every two corresponding feature maps obtained from all layers of the network for an input image $y=f_{\theta }\left(I_Q\right)$and a selected template$y^{\text{'}}=f_{\theta }\left(I_T\right)$. Starting with $CORR$ result of the deepest layer by using “hierarchical activations propagation” the maximum activation value of the first layer is reached. The position of this activation is the coordinates of the template $I_T$ in the input image $I_Q$. But, since the receptive field in the down sampling blocks for layer N is 3 by 3, this resulting position is applicable to the input image $I_Q$ downsized 3 times.

4. Experiments

This section presents experimental results and a demonstration of our proposed Self-TM method on data having the same and different modality as the one used in the training (ImageNet-1K Test [53], HPatches [43], MegaDepth [44] and ScanNet [45]) as well as a variety of tasks:

• ImageNet-1K Test [53], showing the accuracy of template localization on data having the same modality as the training data;
• HPatches [43], evaluating the properties of feature maps as local descriptors for finding matches between different image patches in corresponding images;
• MegaDepth [44], evaluating image matching accuracy on outdoor scenes having different modality from the training data;
• ScanNet [45], evaluating image matching accuracy on indoor scenes having different modality from the training data.

4.1. ImageNet-1K Test

ImageNet-1K test set [53] was used to compare the trained Self-TM model family (Small, Base and Large) based on template localization accuracy.

Since the training is carried out in two successive steps, which differ in the type of applied augmentations of the input images: color augmentations; color and geometric augmentations, this test is performed separately for each of them.

The result is presented in Table 5, where for all Self-TM models (small, base and large) the displacement in pixels of the three layers $({D\_L}_{last}, {D\_L}_{mid},{D\_L}_{first})$ of $pred\_p_N, \dots , {pred\_p}_1$ is calculated, relative to the real positions of the template $gt\_p_N, \dots , {gt\_p}_1$.

The test steps are similar to training steps and are as follows:

• An input image is taken from an unannotated dataset (ImageNet-1K Test), from which a "template" is obtained. On each two corresponding feature maps positions of the maximum values of the correlation operator’s result are then calculated (see training steps 1 to 5 in Section 3.3).
• For each two corresponding feature maps (i.e., for each layer), the position displacement measured in pixels (${D\_L}_{last}, {D\_L}_{mid},{D\_L}_{first}$) of $pred\_p_N, \dots , {pred\_p}_1$ is computed, relative to the actual template positions $gt\_p_N, \dots , {gt\_p}_1$, using Euclidean distance:

  $${D\_L}_n=\mathrm{}|gt\_p_n-pred\_p_n|={\displaystyle \frac{1}{M}}{\sum }_{i=1}^M\sqrt{{(gt\_p_n(i)-pred\_p_n(i\left)\right)}^2},$$

  (7)

  where $M$ is the number of images in the dataset, $gt\_p_n\left(i\right)$ and $pred\_p_n\left(i\right)$ are the two-dimensional vectors of each image $i=1,\dots ,M$, and $n$ is the currently selected layer of the network ${\{D\_L}_n|n=1,\dots ,N\}$. In this case, $N=3$ and for clarity instead of $n=1,2,3$ we are using $first,mid,last$ indexation. ${D\_L}_N$ or ${D\_L}_{last}$ is the average Euclidean distance for the deepest layer.

Table 5 clearly shows that the models trained and tested using only color augmentations despite their size difference manage to achieve the same localization accuracy, respectively an average displacement of $0.15$ pixels for the deepest layer, followed by an average displacement of $0.17$ pixels for the middle layer and an average displacement of $0.57$ pixels for the first layer. This shows that even the smallest Self-TM model (13M) has sufficient parameters capable of reaching successful training, competing with the largest Self-TM (130M). However, this is not the case when geometric augmentations are added to the color augmentations, where we observe almost twice the displacement (${D\_L}_{first}=2.214,{D\_L}_{mid}=0.767,{D\_L}_{last}=0.409$), compared to Self-TM Large (130M) (${D\_L}_{first}=1.331,{D\_L}_{mid}=0.452,{D\_L}_{last}=0.273$). The reason is the increased variation in the training data by the applied geometric augmentations, which require bigger models and more training time.

Table 6 shows the performance of the Self-TM Large model on random images from the ImageNet-1K Test set. A randomly selected region on which random color and geometric augmentations were applied to obtain a template. From the examples, it can be clearly seen that the selected model successfully manages to cope with the applied complex color and geometry augmentations.

4.2. HPatches

HPatches [43] is a dataset for finding matches between different patches, which is used to evaluate the generalization capabilities of Self-TM models.

This dataset has an out-of-domain modality than the one used for training, containing scenes (116 groups of scenes with 6 images in each) with different illumination and geometric augmentations. Patch matches are detected by feature points located by HoG, Hessian and Harris detectors between the first and the other images. The standard metric mAP (mean average precision) was used for evaluation on three different tasks: patch verification, image matching and patch retrieval.

Patch Verification evaluates how well positive pairs are separated from negative pairs. There are two subtasks: SameSeq containing pairs of patches from the same set of scenes and DiffSeq containing pairs of patches from different sets of scenes. The SameSeq is considered more challenging because the textures in different parts of the image are often similar.

Image Matching evaluates how accurately images from the same scene are detected. Here, the overall mAP results are much lower than in Patch Verification, since the ratio of positive to negative examples is significantly lower here and all the negatives come from the same group of scenes. Another interesting observation is that results obtained from images with photometric changes (illum) are significantly lower than those with viewpoint changes (viewpt). This is because the “illum” images include extreme changes in illumination.

Patch Retrieval evaluates how accurately patches correspondences are detected in a large set of patches. Here, a single patch is compared to a large collection containing mainly distractors obtained randomly from the remaining image sets.

In the Self-TM experiments on HPatches, the feature maps from the last layer of the network are used to obtain the patch descriptors. The input data is resized to 32x32 yielding descriptors of 1x512 (Self-TM Small), 1x1024 (Self-TM Base) and 1x2048 (Self-TM Large).

Benchmark of Self-TM with state-of-the-art methods on all three HPatches tasks [43] is shown in Figure 3.

In all three tasks the color of the markers indicates the amount of “geometric noise” $Easy$, $Hard$, $Tough$ contained in the images used. The accuracy percentages represent the average values of the selected subtask.

The experiment used all the Self-TM models: Self-TM Small, Self-TM Base, Self-TM Large, trained on ImageNet1K Train [53] using color and geometric augmentations, where (+) denotes models that are finetuned on the HPatches dataset.

These results demonstrate robust generalization of all the Self-TM models, where Self-TM Large exhibits impressive performance on all three tasks - competitive with previous results of models trained with annotated data (supervised training), approaching the best.

The performance of the Self-TM Large+ finetuned on HPatches even outperforms the previous results on the Patch Verification task, and on Image Matching it gives the first place with a difference of only $0.2\mathrm{\%}$ mAP.

4.3. Image Matching

Image matching by sparse methods is a suitable task for verifying the high degree of generalization capabilities of Self-TM models, also being very close to template matching. Here, matching is performed by finding sparse keypoints matches in a set of images (Figure 4).

To demonstrate the effectiveness of Self-TM, we use two well-known databases, namely:

• MegaDepth [44] is a large-scale outdoor image dataset. Ground truth matches between images are computed using Structure-From-Motion [59] algorithms. Test data used contains 1500 predefined image pairs, following previous work [34].
• ScanNet [45] which contains many indoor images composed of challenging scenes with low texture and large perspective changes. Here again, test data used contains 1500 predefined image pairs, following [34].

Both datasets contain only out-of-domain images different from the Self-TM training data.

We perform estimation of the matching accuracy between each pair of images by finding the camera pose estimation and computing the essential matrix with the RANSAC [60] algorithm, then decomposing it into rotation and translation. Finally, the angular error between the calculated and real rotation between all pairs of images is reported, represented by AUC (area under the ROC Curve) at thresholds of 5°, 10° and 20°, see [44,45]. AUC summarizes the performance of classification models with different classification thresholds using the ratio of True Positive Rate (TPR) to False Positive Rate (FPR).

Since Self-TM itself is a convolutional neural network model, it cannot be applied alone to solve the image matching tasks. Therefore, it is integrated into the OmniGlue [31] framework, designed and focused on a high degree of generalization using DINOv2 foundation model [37].

More generally, OmniGlue [31] uses frozen weights of DINOv2 [37] and SuperPoint [27] to extract sparse keypoints and their features, which are used to construct an internal graph between each image keypoints and an external graph between all the potential correspondences. Both graphs are fed to a learning module containing multiple blocks with two attention layers ending with a layer giving the potential corresponding keypoints.

Self-TM is included in the OmniGlue framework [31] replacing DINOv2 [37] without any further training. Thus, a direct comparison between Self-TM and DINOv2 [37] features in terms of quality and generalization is performed.

Table 7 presents a comparison between the novel OmniGlue + Self-TM framework against the original OmniGlue [31] and various competing methods (requiring training) following the experiments performed in [31]. These experiments also include traditional approaches like SIFT [24] and SuperPoint [27] still used in tasks where training is not possible. In these hand-crafted rules a Mutual Nearest Neighbor is used to find a match between keypoints.

Table 7 shows a significant advantage of OmniGlue + Self-TM Large/Base over OmniGlue [31] on both MegaDepth [44] and ScanNet [45] datasets. OmniGlue + Self-TM Small improves accuracy at AUC @5° but lags behind at AUC @20°.

In addition to the accuracy evaluation the comparison of DINOv2 [37] and Self-TM should also consider their architectures size, which directly affects training time and inference speed. The DINOv2 [37] architecture used is ViT-14-base [18] having 87M parameters compared to Self-TM Base having 40M and Self-TM Large having 130M parameters.

Considering the results of Table 7 and the size of the architectures used, Self-TM gives a higher degree of generalization than DINOv2 [37] (ViT-14-base) and at the same time provides a more optimized architecture - in number of trainable parameters and speed (Table 8).

5. Implementation Details

5.1. Two Step Training

Table 9 aims to present our experiments based on which the iterative training approach is developed (two successive steps with different image augmentations: color, color and geometric). HPathces dataset [43] is used, which effectively evaluates feature maps properties as local descriptors for finding matches between different image patches in corresponding images.

Our experiment uses Self-TM Base model trained from scratch on ImageNet or HPatches dataset using color or color and geometric augmentations, and the same one finetuned on the same data but using weights of already trained models on: HPatches (color); ImageNet (color); ImageNet (color and geometric)

The iterative training approach (Table 9 row 3), where the model is first trained on HPatches with color augmentations and then finetuned with color and geometric augmentations increases the accuracy in: Patch Verification +7.73%; Image Matching +11.96%; Patch Retrieval +5.41% over the model trained directly on HPatches with color and geometric augmentations (Table 9 row 2). The reason for this lies in the fact that geometric augmentations introduce additional variability in the data, which the model cannot initially overcome.

HPatches contain scenes (116 groups of scenes with 6 images in each) with different illumination and geometric augmentations, where models trained with color and geometric augmentations, respectively, have increased accuracy than those using only color augmentations:

• ImageNet training (Table 8 row 5 vs. row 4) results on increased model accuracy such as: Patch Verification +31.77%; Image Matching +81.76%; Patch Retrieval +66.63%;
• HPatches training (Table 9 row 3 vs. row 1) results on increased model accuracy such as: Patch Verification +9.23%; Image Matching +33.89%; Patch Retrieval +18.18%.

High degree of generalization can be explained by the high number of images and the high diversity of classes (1000 classes) covering a sufficient number of visual concepts in ImageNet. Comparing ImageNet and HPatches training (Table 9 row 5 vs. row 3), a significant advantage is observed for the model trained on ImageNet: Patch Verification +22.59%; Image Matching +248.03%; Patch Retrieval +102.73%, despite out-of-domain training data compared to the HPatches data used for test evaluation.

When accuracy increase is required, Self-TM allows for easy finetuning with a small number of images to bring in precise information about the objects and visual concepts to be detected. Comparing ImageNet training result and the same model finetuning with HPatches (Table 9 row 8 vs. row 5) shows an increase in accuracy: Patch Verification +1.15%; Image Matching +4.08%; Patch Retrieval +3.79%.

5.2. Model Architecture

Computer vision field has seen increased model refinement based on improved architectures. They all focus on creating general-purpose visual features that perform well on images of out-of-domain modalities and various tasks. Such models are designed to cover different types of tasks, which usually results in increased size and complexity due to incorporation of additional layers for each type of task (classification, segmentation, tracking, etc.).

The current work shows that designing an architecture targeting a domain of tasks (object detection, image registration, object tracking, sparse and dense image matching) significantly reduces the network size, simplifies the training and inference, increases accuracy and retains its generalization capability, performing well on out-of-domain images.

Our proposed architecture (described in Section 3) uses ConvNeXt [14] blocks with a modified filter size and stride. Down sampling blocks now use 3 × 3 filters and stride 3, which in terms of a receptive field gives a layer $N$ neuron field of view of size 3 by 3 on layer $N-1.$ This results in smoother layer-to-layer hierarchical propagation of activations and precise centering due to the size oddness.

The layers used, following ConvNeXt [14], are:

• Standard two-dimensional convolution (Conv2D) using a certain number of filters performing scalar multiplication on their input data;
• Linear layer (Linear), representing a fully connected layer where each neuron is connected to each neuron from the previous layer;
• Normalization layer (Norm) [46], a faster alternative to Batch Normalization [61], used to normalize network weights, in order to reduce the training time;
• Non-linear activation function GELU [47], used in all modern transformers;

A template positioning accuracy comparison of the proposed Self-TM architecture versus ConvNeXt, is shown in Table 10 via the pixel displacements ${D\_L}_{First}$ and ${D\_L}_{Last}$ between the calculated positions ${pred\_p}_1$ и ${pred\_p}_N$, relative to the ground truth positions $gt\_p_1$ and ${gt\_p}_N$ of the template. Here ${D\_L}_{Fisrt}$ represents the average displacement using the first layer feature maps and ${D\_L}_{Last}-$using the last (the deepest) layer feature maps.

Here Self-TM Base (40 million parameters) and ConvNeXt-S (50 million parameters) are used as they are closest in number of parameters. The result shows that Self-TM Base having 20% less parameters manages to match the templates more accurately by 48.44% using only color augmentations and by 30.39% with geometric augmentations also applied, respectively.

Additional comparison between Self-TM Base and ConvNeXt-S is shown in Table 11, aiming to evaluate the properties of feature maps as local descriptors for detecting matches between different image patches in their corresponding images, using the HPatches dataset [43].

Results of multiple trainings steps on each model (sequentially with different augmentations) are presented starting with ImageNet training with color augmentation and finetuning it to a model trained on HPatches with color and geometric transformations.

Despite the smaller number of parameters in Self-TM Base, increased accuracy is observed in each of the three tasks: patch verification, image matching, and patch retrieval. The best result using a model trained on HPatches with color and geometric augmentations shows an increase in accuracy by 2.96% (from 84.39 to 86.89) in Patch Verification, by 15.68% (from 34.88 to 40.35) in Image Matching and by 5.61% (from 60.61 to 64.01) on Patch Retrieval.

6. Conclusions

The present work introduces a novel end-to-end approach using a family of foundation models Self-Supervised Foundation Model for Template Matching (Self-TM) for precise template matching, image retrieval, dense template matching and sparse image matching using hierarchical propagation of activations from the last to the first layer in an efficient-sized convolutional neural network.

Trained on ImageNet-1K, without any annotations, Self-TM provides robust generalization capabilities out-of-domain data involving challenging geometric augmentations. On the sparse image matching task, Self-TM Base significantly outperforms DINOv2 [37] by +19.6% / +6.7% / +0.3% (AUC@5° / 10° / 20°) on MegaDepth-1500 [44], and on ScanNet-1500 [45] by +57.1% / +20.5% / +6.2% (AUC@5° / 10° / 20°). On HPatches [43], Self-TM has competitive performance compared to methods using a supervised learning approach. When an increase in accuracy is required, the model is easily fine-tuned, outperforming the previous supervised results in the patch verification task on HPatches.

Code and Data: We shared source code using the GitHub repository for this project: https://github.com/anhristov/self-tm.