A Dual-Stream CLIP–ViT Framework for Open-Set Animal Re-Identification: Multi-Seed Ablation, Background-Bias Bracketing, and Query-Time Robustness Analysis

1. Introduction

Animal re-identification (Re-ID) asks whether two images depict the same individual of a given species rather than the same species. It is a fine-grained visual recognition task in which intra-class variability (pose, illumination, expression, viewpoint) is large and inter-class variability (different individuals of the same breed) is small. Among its applications, the recovery of lost companion animals is one of the most socially consequential: a landmark survey [1] found that roughly one companion dog or cat in seven is lost over a five-year period in the United States, and that about $10\%$ of lost dogs and $25\%$ of lost cats are never reunited with their families. Search remains largely manual — canvassing the neighbourhood, paper flyers, shelter visits [2] — and microchips help only once an animal is physically scanned at a facility [3], which is not the case for the sightings shared informally on social media [4]. The dominant signal is therefore a photograph, yet no automated system performs individual-level matching from user-submitted “in-the-wild” images.

Three theoretical gaps. Deploying Re-ID into this setting requires bridging three gaps that controlled benchmarks tend to hide. (i) The Clutter Gap: state-of-the-art Re-ID models are trained and evaluated on pre-cropped subjects but deployment scenarios involve, in many cases, cluttered phone photographs — with complex backgrounds, occluded individuals, poor lighting conditions, etc. Yu et al. [5] show that background overfitting alone can cost $10.2$ percentage points of mmAP when the background is removed from both training and testing on ATRW. (ii) The Pose Alignment Gap: animals are non-rigid bodies whose appearance changes substantially across postures, and global average pooling conflates body-part contributions. (iii) The Data Scarcity Gap: typical lost-pet scenarios provide 1–5 images per individual, below the regime where deep metric learning is comfortable.

Contributions. This work proposes a hierarchical two-stage framework that targets the three gaps and then stress-tests the resulting embedding under conditions rarely reported jointly in the animal Re-ID literature. Our contributions are:

1)

A dual-stream Stage 2 network that fuses a frozen CLIP ViT-B/16 [6] encoder, adapted by a lightweight trainable projection (Stream A), with a fine-tuned ViT-Base/16 [7] carrying L2-norm part attention (Stream B), trained jointly under ArcFace [8] through a single BNNeck [9] embedding, preceded by a YOLOv8 [10] soft-crop Stage 1.

2)

A multi-seed ablation on a combined cat+dog open-set benchmark (173 test identities) that reports sample mean and standard deviation over three seeds for every ViT-based row, complemented by an open-set verification protocol (TAR at FAR $=0.01$ and $0.001$) on both the body benchmark and the $14,716$-identity PetFace dog test split [11].

3)

A background-bias bracket that bounds the residual background reliance of the embedding between a coarse bounding-box mask (a reliable lower bound) and a pixel-level silhouette mask obtained by prompting SAM ViT-B [12] with a specialised dog/cat YOLOv8 detector; a manual audit of the silhouette operator then motivates retaining the reliable soft crop — which never discards the subject — as the Stage 1 front-end for a recall-critical task.

4)

A query-time robustness audit covering nine controlled corruptions (four positional occluders, motion blur, additive Gaussian noise, brightness reduction, JPEG-quality 20, $4\times$ down/up-sampling) applied to the query split with the gallery held fixed, providing a deployment-relevant complement to throughput-oriented audits of prior work [13].

5)

A hardened negative result for a Descriptor Vector Exchange (DVE) [5,14] extension: under full backbone backpropagation it is Pareto-dominated by the base model at seed 42 and at the three-seed mean, and a hyperparameter sweep over descriptor-block depth, loss weight and seed finds no Pareto-superior point. We attribute the outcome to the coarse feature-map resolution of the ViT backbone and to architectural redundancy, not to the data.

2. Related Work

Animal Re-Identification. Early wildlife Re-ID exploited patterned species: tigers (ATRW [15]), Grevy’s zebras [16], and seals or whale sharks (ALFRE-ID [17]). Schneider et al. [18] showed that Siamese networks can match or exceed human-level identification on several species, and the recent WildlifeReID-10k benchmark [19] consolidates the field around shared evaluation protocols. For domestic animals, PetFace [11] provides over a million AnyFace face crops of $257,484$ individuals across 13 families, with ArcFace dominating its loss comparison ($51.23\%$ average top-1 accuracy versus $41.88\%$ for Softmax and $9.81\%$ for Center loss). LostPaw [20] targets the lost-pet retrieval task with a single-stream contrastive ViT and reports a held-out $F_1$ of $91.08\%$ on $31,860$ dogs. OpenAnimals [21] provides a per-axis study of which person Re-ID techniques transfer to animals, finding that square inputs and disabling Random Erasing help whereas many augmentation and attention add-ons inherited from person Re-ID do not. Yu et al. [5] identify background overfitting as the principal failure mode on ATRW and propose a DVE-based unsupervised part-alignment module on top of a SAM/ISNet foreground-removal pipeline.

CLIP and prompt learning. CLIP [6] contrastively aligns an image encoder and a text encoder over 400M web image–text pairs, producing a transferable semantic prior. Naive fine-tuning on small Re-ID datasets erodes that prior; CoOp [22] instead trains a small bank of static context vectors on the text side, and CoCoOp [23] makes those prompts image-conditioned. For animal Re-ID, IndivAID [24] generates image-specific prompts via a Meta-Net and reports consistent gains over CLIP-ReID across eight benchmarks (largest on the pose-variable LionData, $+13.95$ mAP). Our Stream A is deliberately the most conservative member of this family: a frozen CLIP encoder with a single learned projection and no prompt learning, with text-side prompt learning and the full IndivAID Meta-Net left as the highest-priority future-work item.

Deep metric learning. ArcFace [8] introduces an additive angular margin into the softmax objective and trains class-prototype embeddings on the unit hypersphere; it scales to the $\sim 85,000$ identities of MS1MV2 in the face-recognition literature at $m=0.5$. Triplet loss [25] requires hard-pair mining; Circle Loss [26] unifies class- and pair-based losses with self-paced weighting; Multi-Similarity Loss [27] mines informative pairs by jointly weighting positive and negative similarities. The BoT recipe [9] combines ArcFace with a BNNeck, PK sampling, and cosine annealing into a strong person Re-ID baseline that transfers robustly to animals [21].

Part-based alignment. Descriptor Vector Exchange (DVE) [14] learns dense pixel-level correspondences across geometrically warped views without keypoint supervision. Yu et al. [5] apply it on an SE-ResNet50 backbone and report a Re-ID gain on ATRW from DVE together with multi-identity batch sampling.

3. Method

3.1. System Overview

The system decouples “finding” from “identifying” (Figure 1). Stage 1 applies a YOLOv8 detector [10], initialised from COCO [28] and fine-tuned on the dog/cat classes of OpenImages V7, to localise the subject; the bounding box is expanded by a $15\%$ context margin and cropped (soft-crop). If no detection fires, a square centre crop is substituted, so the pipeline always yields a valid input. The detector is deliberately recall-favoured: a missed detection removes the subject entirely and is catastrophic, whereas a loose box is absorbed by the soft-crop margin. Because the crop is non-differentiable, it is run offline and cached, which keeps Stage 2 trainable within a modest VRAM budget and lets the same soft-crop be baked into training when the integrated pipeline is evaluated (Section 4.6).

The dual-stream rationale is to combine a class-agnostic semantic prior (Stream A, addressing the Data Scarcity Gap) with a part-aware spatial representation (Stream B, addressing the Pose Alignment Gap).

3.2. Stream A: Frozen CLIP Semantic Prior

We use CLIP ViT-B/16 [6] as a frozen image encoder. Its weights are kept frozen throughout training because fine-tuning CLIP on a few hundred identities erodes the very prior that makes it valuable: the gradient signal is too weak to improve the encoder yet strong enough to overwrite it. To adapt the encoder to the identity task without touching the backbone, we attach a single trainable projection — a linear layer followed by layer normalisation — that maps the frozen pooled output to a 512D vector. Only this projection receives gradients. This frozen-backbone, learned-adapter design is strictly less expressive than the prompt-learning family from which it draws motivation (CoOp, CoCoOp, IndivAID); we adopt it deliberately so that the measured contribution of Stream A (Section 4.2) reflects the frozen visual prior alone, and we leave text-side prompt learning as future work.

3.3. Stream B: ViT-Base with L2-Norm Part Attention

The second stream is a ViT-Base/16 pre-trained on ImageNet-21K (loaded through timm). Given the 196 output patch features $\mathbf{F}\in {\mathbb{R}}^{196\times 768}$ of the final block, we compute the L2 norm $r_i={\parallel {\mathbf{f}}_i\parallel }_2$ of each patch as a proxy for its discriminative importance and aggregate the top-$K=6$ patches by a softmax-weighted average:

$${\mathbf{h}}_{\mathrm{part}}=\sum _{i\in \mathcal{S}}{\alpha }_i{\mathbf{f}}_i,{\alpha }_i={\displaystyle \frac{exp\left(r_i\right)}{{\sum }_{j\in \mathcal{S}}exp\left(r_j\right)}},\left|\mathcal{S}\right|=K.$$

(1)

This magnitude-based rule is a deliberately lightweight, parameter-free saliency heuristic; more principled selection criteria, such as PCA- or attention-based part selection, are a natural direction for future work. To retain a global summary alongside this part-level evidence, the part vector is averaged with the classification ([CLS]) token, $\mathbf{h}=\frac{1}{2}({\mathbf{h}}_{\left[\mathrm{CLS}\right]}+{\mathbf{h}}_{\mathrm{part}})$. The mechanism adds no parameters and supplies no training signal of its own; all discriminative pressure comes from the ArcFace objective acting on the fused embedding. The ViT backbone is frozen for the first 10 epochs and subsequently fine-tuned at a learning rate $100\times$ lower than the heads ($3\times {10}^{-6}$ vs. $3\times {10}^{-4}$) to avoid catastrophic forgetting.

3.4. Fusion, BNNeck and Loss

The 512D Stream-A and 768D Stream-B features are concatenated into a 1280D vector and passed through a BNNeck [9] that here is a single bottleneck rather than the canonical pre-/post-BN split:

$$\mathbf{e}=\mathrm{Dropout}\left(\mathrm{ReLU}\left(\mathrm{BN}\left(W\mathbf{z}\right)\right)\right)\in {\mathbb{R}}^{512}.$$

(2)

The L2-normalised embedding $\mathbf{e}$ serves a dual role: at training it is the input to ArcFace, and at inference it is the descriptor compared against the gallery by cosine similarity. A single shared embedding suffices because the objective is purely angular — the original split exists only to serve a Euclidean triplet term simultaneously with an angular one. ArcFace is computed against a class-prototype matrix $\mathbf{W}\in {\mathbb{R}}^{C\times 512}$:

$${\mathcal{L}}_{\mathrm{ArcFace}}=-{\displaystyle \frac{1}{N}}{\displaystyle \sum _{i=1}^N}log{\displaystyle \frac{e^{scos({\theta }_{y_i}+m)}}{e^{scos({\theta }_{y_i}+m)}+{\sum }_{j\ne y_i}{e}^{scos{\theta }_j}}},$$

(3)

with $s=64$, $m=0.5$ on the body benchmark ($C=449$) and $m=0.3$, $\mathrm{lr}=1\times {10}^{-4}$, an 8-epoch warm-up on the larger PetFace pool ($C=3,040$), which eases early pressure on the more crowded angular space and avoids the degenerate fixed point identified in preliminary runs.

3.5. Optional DVE Extension

For the part-alignment ablation we add a spatial descriptor head ($768\to 128$, Linear + LayerNorm) on the ViT block-3 intermediate features and train it jointly with a DVE loss [5,14]. For each patch u of an image $\mathbf{x}$ and its affinely warped view ${\mathbf{x}}^{\prime }=T\left(\mathbf{x}\right)$, descriptor similarities define a soft correspondence whose expected matched position $\widehat{g}\left(u\right)$ is regressed to the known warped position $T\left(u\right)$:

$$\begin{array}{cc}\hfill p(v\mid u)& ={softmax}_v\left(\frac{\langle \Phi {\left(\mathbf{x}\right)}_u,\Phi {\left({\mathbf{x}}^{\prime }\right)}_v\rangle }{\tau }\right),\hfill \\ \hfill {\mathcal{L}}_{\mathrm{DVE}}& =\frac{1}{N}\sum _u{∥\widehat{g}\left(u\right)-T\left(u\right)∥}^2,\hfill \end{array}$$

(4)

with $\widehat{g}\left(u\right)={\sum }_{v}p(v\mid u){\mathbf{c}}_v$, ${\lambda }_{\mathrm{DVE}}=0.2$, $\tau =0.1$, and random affine warps ($\pm {30}^{\circ }$, scale $0.85$–$1.15$, translation $\pm 10\%$); the total loss is ${\mathcal{L}}_{\mathrm{ArcFace}}+{\lambda }_{\mathrm{DVE}}{\mathcal{L}}_{\mathrm{DVE}}$. DVE gradients propagate through the descriptor head and through ViT blocks 0–3, as the formulation requires; memory feasibility is restored by gradient checkpointing [29], at the cost of $\approx 80\%$ additional step time once the backbone is unfrozen. An earlier detached-backbone variant is included as a diagnostic baseline (Section 4.3).

3.6. Training Protocol

We train with AdamW (weight decay $5\times {10}^{-4}$) under a warm-up cosine schedule whose rate rises linearly from $20\%$ to $100\%$ of its target over epochs 0–4, PK sampling ($P=10$, $K=4$, batch 40), BF16 mixed precision, and early stopping (patience 15 evaluations) on a combined cat+dog validation Rank-1. RandomErasing is deliberately disabled, following the OpenAnimals finding [21] that it removes exactly the subtle local markings (stripes, spots, facial patterns) that discriminate individuals; a square input geometry is kept for the same animal-oriented reasons. We depart from the OpenAnimals ARBase recipe on a single axis: a pretrained ViT-B/16 backbone rather than an IBN-ResNet-50.

4. Experiments

4.1. Datasets and Protocols

We use three datasets spanning a deliberate naturalistic-to-curated spectrum (Table 1, Figure 2). CatIndividuals (354 train / 75 val / 77 test identities; $9,192$ training images) is the principal source of in-the-wild training signal: user-submitted phone photographs containing furniture, secondary animals, hands and occluding objects, and the dataset closest in distribution to the queries a lost-pet service would actually receive. MultiposeDog (95 train / 96 test; Market-1501 format) supplies tight, pre-cropped body crops with substantial pose variation, and the dog subset of PetFace [11] supplies AnyFace face crops. The combined cat+dog body benchmark pools the MultiposeDog query/gallery (96 dog identities) with the held-out CatIndividuals test split (77 cat identities), for 173 identities and 181 query / $2,319$ gallery images; validation pools the MultiposeDog query/gallery with the 75-identity CatIndividuals validation split, so checkpoint selection reflects both species. All evaluations are open-set with respect to identity (train and test identity sets disjoint) and closed-gallery at the retrieval level (each query has at least one ground-truth match).

We report Rank-k accuracy (CMC) and mean Average Precision (mAP). Of the two, mAP is the primary metric for overall retrieval quality, while Rank-5 is the application-facing metric for the lost-pet task, where an owner inspecting the top-5 results has a high probability of finding the pet.

4.2. Multi-Seed Ablation on Body Re-ID

We ablate four architectural conditions on the combined benchmark and report each ViT-based row as the multi-seed mean and sample standard deviation over three independent runs with seeds $\{7,42,123\}$, alongside the seed-42 single-run value for context (Table 2). All other hyperparameters are identical across rows.

Three findings emerge.

ResNet→ViT is the largest step. Replacing the convolutional backbone with a ViT-Base lifts the seed-42 Rank-1 by $+8.29$ and mAP by $+15.26$ percentage points; the ResNet baseline also peaks early in validation and then degrades under continued angular-margin pressure, whereas the transformer remains stable.

The CLIP→ViT gain survives variance. The difference of multi-seed means between C and B is $+2.39$ Rank-1 and $+2.07$ mAP, both above the pooled standard deviation; the contribution is not an artefact of a favourable seed. The frozen CLIP prior also halves the Rank-1 seed variance (sample standard deviation $0.0166\to 0.0083$, a $2.0\times$ reduction). The stabilisation is, however, specific to the top-1 decision: the mAP variance is not reduced ($0.0165$ for ViT-only against $0.0185$ for CLIP+ViT). We report the two effects separately rather than folding them into a single “CLIP is better” claim.

The DVE extension is Pareto-dominated. Row D sits below the base CLIP+ViT on both metrics at seed 42 ($0.9669$ / $0.8523$ vs. $0.9834$ / $0.8731$) and at the three-seed mean ($0.9669\pm 0.0055$ / $0.8513\pm 0.0029$ vs. $0.9742\pm 0.0083$ / $0.8597\pm 0.0185$); its only favourable property is a tighter seed distribution. We analyse this negative result next.

4.3. DVE: A Hardened Negative Result

A natural concern is whether the DVE outcome is an artefact of one hyperparameter choice. With seed 42 fixed and all other settings as in Table 2, we sweep the ViT block whose patch tokens feed the descriptor head $\{3,6,9\}$ and the loss weight ${\lambda }_{\mathrm{DVE}}\in \{0.05,0.1,0.2,0.5,1.0\}$. (With a ViT backbone the token grid stays at $14\times 14$ across the depth sweep, so the axis varied is the descriptor’s receptive-field depth, not its spatial resolution as in Yu et al.’s SE-ResNet adaptation.) Across all three depths and all five loss weights — combined with the three seeds at the canonical $(\mathrm{block}3,\lambda =0.2)$ point — no configuration is Pareto-superior to the base: $\lambda =0.2$ and $\lambda =0.5$ tie for the sweep’s highest Rank-1 ($0.9669$), and although $\lambda =0.5$ attains the sweep’s best mAP ($0.8651$) it still trails the base ($0.8731$).

We attribute the result to two architectural factors rather than to the data. First, feature-map resolution: DVE learns dense per-position correspondences and was validated where the SE-ResNet map is downsampled only $4\times$ ($56\times 56$), with a modest reported gain of $+2.0$ to $+2.4$ mmAP on ATRW; our ViT-B/16 exposes a $14\times 14$ token grid — about sixteen times fewer spatial positions — far coarser than the regime in which DVE was shown to help. Second, architectural redundancy: the frozen CLIP prior (Stream A) and the L2-norm part attention (Stream B) already aggregate the discriminative regions a correspondence objective is meant to align. Dataset scale is not the differentiator — our 449-identity pool exceeds the ATRW Re-ID split ($\sim 107$ training entities). A faithful test of DVE on a higher-resolution feature map (a finer token grid or a convolutional backbone) is identified as future work. The detached-backbone variant performs no better and isolates a methodological lesson: detaching the backbone silently disables the very mechanism DVE is designed to apply.

4.4. Open-Set Verification

Retrieval metrics presuppose that the query has a match in the gallery. The deployment scenario is structurally different: most queries received by a lost-pet service have no match, and the operational quantity of interest is the cost-weighted error rate of a yes/no decision at a chosen similarity threshold. We construct class-balanced positive and negative pairs from each query/gallery split (RNG $=42$; impostors restricted to the same species), score them by cosine similarity of the L2-normalised 512D embeddings, and report AUC, Equal Error Rate (EER), and True Acceptance Rate at two False Acceptance Rate operating points, FAR $=0.01$ and $0.001$ (Table 3).

Three observations follow. CLIP leads at the moderate operating points: CLIP+ViT (C) attains the best AUC ($0.9860$), the lowest EER ($5.52\%$) and the best TAR at FAR$=0.01$ ($84.63\%$, $+4.11$ pp over ViT-only). The configurations converge at the strictest FAR: at FAR$=0.001$ — the regime most representative of a deployed matcher — all three body models sit within about a point of one another ($67.6$–$68.7\%$ TAR), DVE marginally highest, and the gaps are smaller than the per-seed standard deviations, so no model holds a robust low-FAR advantage. The CLIP variance reduction does not carry over: the AUC seed standard deviation is in fact largest for CLIP+ViT ($0.0053$ vs. $0.0024$ for ViT-only), so the stabilising effect seen on retrieval Rank-1 is specific to that decision and does not extend to the threshold-based scores. The single-seed face checkpoint reaches $74.38\%$ TAR at FAR$=0.001$ over an $85\times$ larger gallery, indicating that the face-cropped pipeline scales better into the low-FAR regime than the body pipeline.

4.5. Open-Set Generalisation at Scale and the Few-Shot Regime

“At scale” here qualifies evaluation, not training. We train the same CLIP+ViT architecture on the PetFace dog filtered set ($3,040$ IDs / $16,335$ imgs; $m=0.3$, an 8-epoch warm-up) and evaluate on the dog test split: $14,716$ unseen identities, each contributing one query and one gallery image. The model reaches $Rank-1=0.5414$, $Rank-5=0.6933$, $mAP=0.6132$ — a test gallery nearly two orders of magnitude larger than the body benchmark (173 vs. $14,716$ identities) in a strict open-set regime. We deliberately make no SOTA claim against the full $46,755$-identity PetFace training benchmark, since we train on a filtered subset. These open-set retrieval numbers are not directly comparable to the seen-identity top-1 accuracies tabulated by Shinoda and Shiohara [11] (whose ArcFace baseline averages $51.23\%$ across families, with $77.86\%$ for dogs under independent per-family training), which evaluate individuals also present in training.

The filtered baseline excludes precisely the identities that motivate the Data Scarcity Gap. We therefore re-train under a relaxed $\ge 2$ images-per-identity filter with $K=2$ images per identity per batch ($P=20$ so the batch size stays 40), expanding the pool to $11,471$ IDs / $41,073$ imgs (median 3 images per identity, $90.2\%$ in the 2–5 range). The result is $Rank-1=0.5885$, $Rank-5=0.7386$, $mAP=0.6589$, exceeding the filtered baseline by $+4.71$ Rank-1 and $+4.57$ mAP percentage points despite a $3.77\times$ larger and substantially sparser training pool. The defensive filter is therefore unnecessary on this benchmark; the truly singular case ($K=1$, one image per identity, which removes within-batch positive-pair construction) is left as future work.

4.6. Background-Bias and the Stage-1 Choice

Stage 1 retains a $15\%$ context margin rather than a tight cut-out. We ask two coupled questions: how much does the Stage 2 embedding rely on the background the soft crop leaves in frame, and is the suppression operator one we can trust as a front-end on real photographs? Each test image (query and gallery) is re-rendered under three masking conditions — original; bounding-box (everything outside the YOLO box filled with the ImageNet-mean colour; reproducible, but it leaves the rectangular ring of background inside the box); and SAM silhouette (the box prompts SAM to remove the in-box background to the animal outline). We cross this with three training regimes (base, soft-crop, silhouette), giving the $3\times 3$ matrix of Table 4.

The base model fixes the magnitude. Removing only the out-of-box background costs it just $-1.65$ Rank-1 and $-4.80$ mAP, whereas removing the in-box background as well costs $-19.9$ and $-28.8$. The true reliance of the deployed soft crop is therefore bracketed between the two: the $15\%$ margin removes very little of the cue the model uses, but a substantial dependence survives inside the box, where the soft crop does not reach. The direction matches Yu et al. [5], whose comparable train-original / test-masked transition costs $-19.2$ mmAP; our upper-bound drop is of the same order. Reading down the columns, suppressing the background at training time reduces the reliance monotonically (the original→SAM gap shrinks from $-19.9$ to $-18.2$ to $-14.9$ Rank-1), but at a cost in clean accuracy (the silhouette-trained model loses $-1.65$ Rank-1 on unmasked images, against $-0.55$ for soft-crop training).

On the matrix alone the silhouette regime would win. That verdict does not survive an audit of the masking operator. Because what matters for re-identification is the loss of discriminative content (eyes, nose, ears, coat markings) rather than pixel area, and no automated proxy we tried matches human judgement (a whole-frame grey fraction, an interior-hole fraction and an embedding-shift score all reach AUC $\le 0.60$ for predicting human-judged damage), we measure the effect by manual audit. On a uniform random sample of 400 training images, the silhouette removes identity-relevant content in $20.8\%$ of the images ($95\%$ CI $17.1$–$25.0$) and erases the subject outright in $4.2\%$ ($2.7$–$6.7$); see Figure 3. The failure is intrinsic to SAM on close-up pet imagery — the polarity is correct, the cat is removed and the background kept — and cannot be screened at inference without a second segmenter. For a recall-critical task a destroyed query cannot be recovered downstream, so the front-end must be judged against a conservative reliability bar. The soft crop clears it: built with the YOLO soft crop, the same training copies fall back to a safe centre crop for only $0.73\%$ of images and never remove the subject; baking it into the pipeline reaches $0.9669$ Rank-1 (train and test both soft-cropped), only $-1.65$ below the no-Stage-1 figure — the same small drop as the bounding-box bound. Foreground segmentation is therefore not adopted as a deployment stage; the silhouette regime is retained only as a cautionary upper-bound ablation, the more so because its mask artefacts are image-deterministic and a silhouette-trained, silhouette-tested model can latch onto them.

4.7. Query-Time Robustness Audit

The deployment scenario is asymmetric: shelter gallery images are typically curated, but owner queries are not. We embed the unperturbed combined gallery once and re-embed the 181 combined query images (104 dog + 77 cat) under nine controlled corruptions, recomputing the metrics against the same gallery (Table 5; NumPy seed 42 for the stochastic perturbations). Single-run figures should be read against the multi-seed noise floor of ${\sigma }_{R_1}=0.0083$ for this model.

The two most damaging corruptions are both capture-time losses of high-frequency detail: a $4\times$ down/up-sampling pass ($-16.0$ Rank-1) and a 15-pixel horizontal motion blur ($-13.3$ Rank-1, and the worst single mAP drop at $-14.0$). Both remove the fine markings a ViT embedding leans on without changing global appearance, and both cost more than any occluder — though both are milder than the SAM background-removal upper bound of Section 4.6, which deletes background and part of the subject. Occlusion is strongly position-dependent but not head-biased on this combined benchmark: the left and centre quarters (which cover the subject’s core for the portrait-framed cats) cost most ($-6.63$ Rank-1, and $-9.18$ mAP for the centre), while the bottom quarter costs least. The embedding is, by contrast, robust to photometric and storage artefacts (brightness halving $-2.76$, JPEG-q20 $-4.42$). The cheapest mitigation — a client-side pre-flight check that rejects low-resolution queries and warns on detected motion blur — therefore recovers most of the lost Rank-1 budget at near-zero cost, more cheaply than a wider augmentation regime that would conflict with the rationale for disabling RandomErasing. This is complementary to throughput-oriented edge audits such as RAPID [13]: rather than optimising frame rate, we map which corruptions a deep embedding is most sensitive to.

4.8. Loss-Function Comparison

To validate the choice of ArcFace, we re-train the proposed architecture under three alternative metric losses with all other components fixed, each at the hyperparameters recommended in its originating paper (Table 6).

No paper-default alternative Pareto-dominates ArcFace. Batch-hard Triplet is the closest, only $-1.10$ Rank-1 and $-0.53$ mAP behind and in fact winning Rank-5 ($0.9945$); it is competitive but not Pareto-superior. Circle Loss is next ($-2.21$ / $-2.34$); its trajectory is healthy, so the gap measures absolute headroom at the animal-Re-ID configuration we adopt from Yu et al.($\gamma =64$, $m=0.25$) rather than at any of the three task-specific regimes in the Circle Loss paper ($\gamma =256$, $m=0.25$ for face; $\gamma =128$, $m=0.25$ for person Re-ID; $\gamma =80$, $m=0.4$ for fine-grained retrieval). Multi-Similarity, at its CUB/Cars defaults ($\alpha =2$, $\beta =50$), is misaligned with our pretrained-ViT, PK setup — its steep negative-pair gradient pulls apart already-discriminative features — and is the clear outlier. Closing these gaps would require per-benchmark loss-hyperparameter retuning; we retain ArcFace as the canonical loss.

4.9. Comparison with the Literature

We situate the framework in prose rather than a single accuracy table, because the nearest systems are each evaluated on a different benchmark — and, in one case, a different task — so a shared Rank-1/mAP column would invite a comparison the protocols do not support. OpenAnimals [21] reports on its four wildlife benchmarks, IndivAID [24] on ATRW and seven further animal datasets, and LostPaw [20] on its own AdoptAPet dog corpus. LostPaw, the only published system targeting the same lost-pet retrieval task, is moreover not measured in the same currency: it reports a pairwise-verification $F_1$ at a fixed threshold rather than a closed-set Rank-1/mAP, so even its headline cannot be placed directly against ours; the cross-evaluation that would make the two commensurable is set out as future work. What can be stated internally is that the 173-identity body benchmark is a relatively small open-set evaluation on which the proposed model’s Rank-5 is high but short of saturation, and that the $14,716$-identity PetFace evaluation supplies the larger, harder gallery that exposes the remaining headroom. We deliberately do not benchmark on ATRW [15]: its tiger visual statistics differ substantially from the dog/cat domain, and a cross-species generalisation experiment was scoped out at the planning stage. Readers seeking an ATRW benchmark for a comparable CLIP-based method can find one in IndivAID [24].

4.10. Implementation and Reproducibility

All models are implemented in PyTorch 2.x with timm 1.0.24 and HuggingFace transformers, trained in BF16 mixed precision on a single NVIDIA RTX 5070 Laptop GPU (Blackwell, CUDA 12.8, 8 GB). Combined body Re-ID training (449 IDs, 100 epochs with early stopping) takes $\approx 28$ min; PetFace training ($3,040$ IDs, 48 epochs) takes $\approx 1$ h 15 min; the DVE variant adds $\approx 80\%$ step time after the backbone unfreezes. At inference, the YOLO soft crop is applied only to unconstrained query photographs; gallery embeddings are computed once and cached, so each query costs one forward pass plus a dot-product search, and an optional fast $O\left(N{k}_1\right)$ variant of k-reciprocal re-ranking [30] ($k_1=20$) refines the ranking without the full $O\left(N^2\right)$ encoding.

5. Discussion

Where the framework succeeds. The CLIP+ViT model’s $+2.07$ mAP and $+2.39$ Rank-1 gains over a ViT-only ablation, replicated across three seeds, together with the $2.0\times$ Rank-1 variance reduction, support the design rationale of Stream A as a class-agnostic regulariser rather than a magnifier of raw discriminative power. The few-shot retraining probe shows the architecture remains viable in the genuinely sparse two-images-per-identity regime that motivates the Data Scarcity Gap, and the large-scale PetFace evaluation shows the embedding space transfers to $14,716$ unseen identities.

Where it fails, and why. The hardened DVE negative result is a property of the method-and-architecture combination, not of the method alone: the dense-correspondence objective needs a high-resolution feature map and non-redundant structure to act on, and the $14\times 14$ ViT token grid combined with the CLIP and part-attention priors provides neither. We deliberately do not attribute the result to a lack of pose variation in the data — CatIndividuals is cluttered and in-the-wild and MultiposeDog carries substantial pose variation — nor to dataset scale, which exceeds the ATRW split on which DVE was shown to help.

Recall governs the deployment choices. Two of our measurements are decided by the asymmetric cost structure of lost-pet retrieval, where a missed reunion is far costlier than a spurious candidate. The background-bias study shows that aggressive foreground segmentation buys the best background robustness but, by a manual audit, defaces or erases the subject in a non-trivial fraction of cats; for a recall-critical front-end we therefore keep the reliable soft crop, which never discards the subject, and treat silhouette suppression as a cautionary ablation. The robustness audit similarly turns into a concrete, cheap pre-flight check (minimum resolution, motion-blur warning) rather than a heavier augmentation regime.

Methodological defaults. Both the multi-seed protocol and the open-set verification were inexpensive on a single modest GPU, and both changed the reading of the results: a single-row best would have over-stated the DVE extension and the AUC summary alone would have hidden the low-FAR regime where deployment actually operates. We suggest reporting both as defaults in animal Re-ID.

Limitations. (i) Stream A uses no prompt learning; integrating the IndivAID Meta-Net (which reports gains up to $+13.95$ mAP over CLIP-ReID, and whose own ablation shows the symmetric text-to-image loss hurts Stage-2, $16.25\to 16.07$ mAP on NyalaData) is the highest-priority architectural extension. (ii) The 173-identity body benchmark is approaching saturation; an in-the-wild dataset of hundreds to thousands of open-set identities would provide a more discriminative ranking signal. (iii) The single-image-per-identity regime is not evaluated. (iv) A robust foreground segmenter — one guarded against the close-up failure mode documented above, rather than segmentation per se — and an active-learning loop such as Ambiguity-Aware Sampling [31] (which matches supervised accuracy while labelling $0.033\%$ of pairs) are the natural paths to a low-annotation deployable system. (v) A direct cross-evaluation with LostPaw [20], whose code and checkpoint are public, is the most informative external validation.

Societal context. The framework targets a problem of real social scale: roughly one companion dog or cat in seven is lost over a five-year period in the United States [1], the dominant search signal is a photograph, and the missing piece is automated individual-level matching from owner-submitted images. Treating this as the actual deployment scenario — rather than an abstract benchmark — motivates every measurement reported here: the verification protocol because most queries have no gallery match, the background-bias bracket because the deployment-time distribution shift removes the most learnable confound, and the robustness audit because phone photographs are asymmetrically degraded relative to curated gallery images.

6. Conclusion

We presented a hierarchical dual-stream framework for open-set animal Re-ID that fuses a frozen CLIP ViT-B/16, adapted by a learned projection, with a fine-tuned ViT-Base carrying L2-norm part attention, preceded by a YOLOv8 soft-crop localisation stage. On a combined cat+dog open-set benchmark of 173 identities it reaches $Rank-1=0.9834$ / $mAP=0.8731$ at seed 42 and $0.9742\pm 0.0083$ over three seeds, the CLIP contribution surviving multi-seed analysis at $+2.39$ Rank-1 and $+2.07$ mAP; it scales to $14,716$ unseen PetFace identities and remains viable in a genuine few-shot regime. A background-bias bracket and a manual audit of the silhouette operator motivate keeping the reliable soft crop as the recall-critical front-end; a nine-corruption audit identifies down-sampling and motion blur as the dominant query-side vulnerabilities with cheap mitigations; and a DVE extension is a hardened negative result, Pareto-dominated across seeds and a hyperparameter sweep and traced to feature-map resolution and architectural redundancy. The framework takes a concrete step towards a deployable lost-pet retrieval system at the scale at which the problem occurs in practice.