Preprint
Article

This version is not peer-reviewed.

Decision Models as L2 Rerankers for Content-Based Recommendation: A Pre-Registered Evaluation

A peer-reviewed version of this preprint was published in:
Information 2026, 17(11), 1141. https://doi.org/10.3390/info17111141

Submitted:

13 October 2026

Posted:

14 October 2026

You are already at the latest version

Abstract
In content-based recommendation, a second-stage (L2) reranker decides which of roughly a hundred retrieved items fill the first screen of results. The standard L2 model is a cross-encoder trained to score relevance. Decision models, general models that answer a typed question about a text with a probability, could offer an alternative through a single plain-language question. Whether they can is unknown: after retrieval most candidates are already on-topic, so the reranker must separate exact matches from close substitutes, a distinction decision models were not trained for, and sparse relevance labels make the comparison hard to measure. We cast a decision model as a pointwise reranker that asks, for each candidate, whether it exactly matches the request, and uses the answer probability as its score. We evaluate this under a pre-registered protocol: every model reranks the same frozen candidate lists, and a gain counts only if it holds both on the realistic retrieved lists and on fully labelled lists. On Amazon ESCI (1,000 requests) and Wayfair WANDS (299), the commercial Jev decision model, used zero-shot, improves P@10 over embedding order by 0.046 to 0.054 and significantly outperforms the Qwen3-Reranker cross-encoder on ESCI. The open decision model CLM-v0.1-8B ranks no better than random. Jev costs about four times more per request than the cross-encoder run on a rented cloud GPU, so the result is best read as evidence that some decision models can serve as high-precision L2 rerankers when accuracy at the top of the list matters more than serving cost.
Keywords: 
;  ;  ;  ;  ;  

1. Introduction

Content-based recommendation returns the items whose content best matches an expressed need: a search request in an online store, a description of what the user wants, or an item they liked [1]. At catalog scale it runs in two stages [2]. An embedding model retrieves a hundred or so candidates (L1), and a slower, more precise model reorders them (L2). The L2 stage decides what the user sees first, so its quality is measured by precision at 5 and 10.
Reordering a retrieved list is difficult because the list is already relevant in the broad sense. On the labelled lists of both catalogs we study, 84% of the embedding top 10 are exact or close matches; what remains is to place exact matches above near misses, for example a graphic tee above a plain one for the request “t-shirt design”. Cross-encoders learn this distinction from labelled relevance data [3,4]. Prompted large language models can rank without task training [5], but generating a ranking over many candidates is costly per request. Decision models offer a third route: a general model answers a typed question about a text with a probability, in one short call. They were built for agent decisions such as choosing a tool or verifying an answer [6], however, and we know of no published evaluation of whether their probabilities separate exact matches from close ones.
Measuring this is itself difficult. Public catalogs label only a sample of products per request: on ESCI, three of every four products in the embedding top 10 carry no label, so a reranker that surfaces a correct but unlabelled item is scored as wrong. A comparison on retrieved lists alone can therefore reward a model for agreeing with the system that chose which items to label.
We treat a decision model as a pointwise L2 reranker. For each candidate it reads the request and the item text and answers one question, whether the item exactly matches the request; the probability of yes is the item’s score (Figure 1). The question targets the one distinction the L2 stage exists to make, and it can be put to any model that returns a yes/no probability, so a cross-encoder and decision models are compared on identical inputs. To control for sparse labels, every model reranks the same frozen candidate lists, ESCI is scored both on the retrieved lists and on fully labelled lists with a gain required in both, and WANDS, where 87% of the embedding top 10 is labelled, serves as a replication.
We compare two decision models, the commercial Jev API model [7] and the open CLM-v0.1-8B model artifact [6], with the Qwen3-Reranker-0.6B cross-encoder [4] on 1,000 ESCI and 299 WANDS requests. Jev, used zero-shot, gives the highest precision on both catalogs and beats the cross-encoder significantly on ESCI. CLM ranks at random level: it recognises a request’s topic but not which on-topic item matches it. Jev’s advantage costs about four times more per request than running the cross-encoder on a rented cloud GPU. Our contributions are:
1.
A formulation of decision models as pointwise L2 rerankers through a single exactness question, applicable to any model that returns a yes/no probability.
2.
An evaluation protocol for reranking under sparse labels: frozen, hashed candidate lists; co-primary retrieved and fully labelled settings; replication on a nearly fully labelled catalog; and pre-registered decision rules.
3.
Evidence on two catalogs that one zero-shot decision model can outperform a trained cross-encoder at the top of the list (significantly on ESCI, in the same direction on WANDS) while another decision model fails, with cost and latency per 1,000 requests.
4.
Code, frozen candidate lists and every run’s manifest (Appendix A).

3. Method: Decision Models as L2 Rerankers

Given a request and its retrieved candidates, our method scores each candidate with one question and sorts the list by the answers. Everything except the L2 model is held fixed: the candidate lists (Section 3.1), the text each model reads and the question it is asked (Section 3.2), and the reference orderings it is measured against (Section 3.3). Retrieval runs once per catalog, and its lists are frozen before any reranker sees them, so every model is scored on identical candidates (Figure 1).

3.1. Candidate Generation and Freezing

We represent each item by its title, then its brand or class, attributes and description, cut to 512 tokens, and embed it with Qwen3-Embedding-0.6B [4]; requests are embedded with a one-line search instruction. For each request we keep the 100 items with the highest cosine similarity: approximate HNSW search on ESCI (1.2 million items, 97.4% overlap with exact search at 100) and exact search on WANDS (43,000 items). The lists, and the set of requests scored on them, are written once, hashed, and verified on every load.
Freezing removes the retriever as a source of variance. Any difference between rerankers then comes from reranking alone, and a run can be repeated later on byte-identical inputs.

3.2. Question-as-Ranker Scoring

Given a request q and a candidate item d, the model answers one question, Does this product exactly match what the shopper’s search query asks for?, and its probability of yes, p ( yes ∣ q , d ) , becomes the score of d; candidates are sorted by score. Item text is cut to 256 tokens and requests to 64 with one shared tokenizer, so every model reads the same words.
Each model receives the question in its own interface (Table 1). Qwen3-Reranker takes it as the task instruction and scores each pair as the probability of its “yes” token. Jev takes one API call per pair, with the request and item as a state headed “Search query” and “Candidate product” and the question as a yes/no item. CLM takes one call per request, with the request as its state, the question as its instructions and all candidates as options, and returns a probability for each candidate normalised over the list.
The question names the criterion the L2 stage exists for, exact match rather than relevance in general, because retrieval has already made most candidates relevant in the broad sense. The formulation needs no training data and no change to the model, applies to any model that returns a yes/no probability, and scores candidates independently, so cost grows linearly with list length and calls can run in parallel.

3.3. Reference Orderings

Three orderings of the same list bound the comparison. Embedding order (L1 order) is the baseline every reranker must beat. Random order within the list is the floor. An oracle that sorts by the true labels is the ceiling, which is below 1 whenever a list holds fewer exact matches than slots.

4. Evaluation Under Incomplete Labels

Public relevance labels cover only a sample of items per request, so an item a reranker promotes may be correct and still be scored as wrong. The protocol therefore fixes what every model sees, scores each request on lists with and without unlabelled items, and accepts a gain only when it holds on both, under rules written before any reranker was run.
Two settings on ESCI. ESCI labels only the products Amazon’s annotators were shown, and three in four products in the embedding top 10 carry no label (Section 5.5). Each request is therefore scored on two lists, both co-primary:
  • Retrieved: the L1 top 100. Realistic, but an unlabelled product counts as not relevant.
  • Judged: only the request’s labelled products (median 16), in embedding order. Every candidate has a label, so differences reflect ranking alone.
The retrieved setting tests the realistic task; the judged setting removes unlabelled items, so a model cannot win or lose by which items happen to carry labels. WANDS labels 87% of the embedding top 10, so only its retrieved lists are scored, with judged-only P@10 as a robustness check.
Metrics. The primary metric is strict P@10, the share of the top 10 labelled Exact with unlabelled items counted as not relevant, because the L2 stage exists to fill the first screen with exact matches. P@5 is the key secondary. Also reported: P@10 counting Exact and Substitute (or Partial) as relevant; judged-only P@10, where unlabelled items are removed before cutting at 10 [11]; judged@10, the labelled share of the top 10; NDCG@10 with graded gains (ESCI 1/0.1/0.01/0, WANDS 1/0.5/0); latency per request (p50, p95); and cost per 1,000 requests.
Statistics. For each model we take the per-request difference in P@10 from embedding order and run a paired bootstrap over requests (10,000 resamples), with a Holm correction across the models in each setting [12]. The pre-registered rules:
1.
A model beats embedding order if its adjusted p < 0.05 and its mean difference is positive, on ESCI in both settings and, separately, on WANDS.
2.
A difference between two rerankers counts as robust only if it has the same sign in both ESCI settings; on WANDS it must also agree with judged-only P@10.
3.
Head-to-head tests between rerankers (paired bootstrap, Holm across the comparisons in each setting) were added after the results and are reported as exploratory.
Pre-registration. For each catalog, the design, metrics and decision rules were committed before any model was run on it; the WANDS protocol was written after the ESCI results, to test whether they hold when labels are nearly complete. Four dated amendments followed, each recording what had been seen: dropping a second cross-encoder (bge-reranker-v2-m3) before any run (Amendment 1); making the judged setting co-primary after the retrieval dry run, before any reranker output (Amendment 2); adding exploratory head-to-head tests after the results (Amendment 3); and removing one planned API model that its provider’s free-tier rate limits prevented us from running (Amendment 4). No amendment changed a pre-registered result.

5. Experiments

5.1. Setup

Datasets. We use the US English subset of the Shopping Queries Dataset used for its Task 1 (ESCI) [9] and WANDS [10] (Table 2). The two differ in the property our protocol controls for: ESCI labels a median of 16 products per request from a 1.2-million-item catalog, while WANDS labels a median of 221 from 43,000 items. In both, a request is a shopper’s search query; we use the two words interchangeably. On WANDS we keep only the 379 queries with at least one Exact label; for the other 101, strict precision is 0 under any ranking. Validation queries were used only for smoke tests and a diagnostic probe, never for tuning.
Implementation. Qwen3-Reranker-0.6B (revision e61197ed) runs in fp16 on one NVIDIA A40. CLM-v0.1-8B runs on the same GPU, its Qwen3-8B encoder served by vLLM. Jev (jev-1.13.0) is called through TypeSafe’s API with up to 32 concurrent requests. Local models process one request at a time. No model was fine-tuned, and no setting was tuned on validation or test data. Each run writes a manifest with model version, list hash, prompt-template hash, hardware, retries and billed tokens (Appendix A).

5.2. Comparison with Embedding Order

Jev and Qwen3-Reranker beat embedding order on strict P@10 in all three tests, which meets the pre-registered rule; CLM falls below embedding order everywhere and stays within 0.012 of random order (Table 3, Figure 2).
Measured against the ceiling, Jev closes 28% of the gap between embedding order and the oracle on ESCI judged P@10 and 34% on P@5; Qwen3-Reranker closes 7% and 12%. Gains are larger at 5 than at 10 for both, so they come mostly from the very top of the list. The ESCI retrieved panel sits low for every model, the oracle included, because most of its top 10 has no label (Section 5.5); this is a property of the labels, not a weakness of the rerankers.

5.3. Decision Model Against Cross-Encoder

The pre-registered rule compares each model with embedding order, so whether Jev’s lead over Qwen3-Reranker is real needs a direct test. We run the same paired bootstrap on the per-request difference between the two (Table 4, Figure 3), reported as exploratory under Amendment 3.
Jev’s lead over Qwen3-Reranker holds in both ESCI settings, so it is robust under rule 2. On WANDS it points the same way but is about half as large and not significant with 299 requests. Both rerankers beat CLM by wide margins in every setting ( Δ P@10 +0.08 to +0.25, p = 0.001 ).

5.4. Latency and Cost

Jev buys its extra precision at about four times Qwen3-Reranker’s cost per request on 100-candidate lists (Figure 4, Table 5). Its measured latency is 8 to 9 times higher, but that figure is set by our API client rather than by the model (point 1 below).
Local models ran one request at a time on one NVIDIA A40. AWS offers no A40 instance, so we price their GPU time at the on-demand rate of the closest single-GPU instance in the same 48 GB memory class, g6e.xlarge (one NVIDIA L40S, $1.861 per hour in us-east-1, October 2026) [13]: cost per 1,000 requests is the p50 time measured on the A40 times this rate. The L40S is generally faster than the A40, so if anything this overstates local cost. Jev’s cost is billed input tokens at $0.042 per million (about 570 tokens per call on ESCI and 650 on WANDS, whose product texts are longer); the roughly 21 output tokens per call are not included. Three points qualify these numbers:
1.
Jev’s measured latency reflects our client, not the model. Our client opened a new HTTPS connection for every call and, running on a rented cloud machine, sustained about 12 calls per second at any concurrency from 32 to 96, against the 80 requests per second Jev’s documentation allows [7]. A 100-candidate request therefore took about 9 s of queueing in our client. At the documented rate, the same request needs about 1.25 s plus the latency of one call. We did not record per-call latency in the benchmark runs; a probe that separates client from model time is in the repository.
2.
Qwen3-Reranker’s latency is an unoptimised baseline. Batching across requests, a serving engine, or shorter inputs would cut it substantially. Neither model meets the 50 to 200 ms budget typical of online L2 reranking as measured here.
3.
CLM is slower than Qwen3-Reranker, costs nearly as much as Jev on rented GPU time, and is no better than random, so it is dominated on every axis.
On ESCI retrieved, Qwen3-Reranker delivers 56% of Jev’s P@10 gain over embedding order (+0.028 against +0.050) at about a quarter of its cost.

5.5. Analysis

Label completeness decides what a retrieved-list result means.Table 6 compares how much of the embedding top 100 each catalog labels. On ESCI three in four top-10 products carry no label, which caps even the oracle at P@10 0.365. That is why the judged setting is co-primary. On WANDS the retrieved list is itself mostly labelled, and judged-only P@10 tells the same story as strict P@10 (Jev 0.562, Qwen3-Reranker 0.545, embedding order 0.509).
Part of the ESCI retrieved-setting gain is moving labelled products up. Reranking raises judged@10 from 0.248 to 0.292 (Jev) and 0.289 (Qwen3-Reranker), while judged-only P@10 moves less (0.327 to 0.340 and 0.331). ESCI’s labelled products were drawn from Amazon’s own search results, so a reranker that agrees with Amazon’s ranker gets credit for surfacing them. The judged setting and WANDS do not have this bias, and both show the same ordering of models.
The rerankers sort Exact above Substitute rather than finding new relevant items. Embedding order already fills 84% of the top 10 with exact or close matches on the labelled lists (0.839 on ESCI judged, 0.846 on WANDS). On ESCI judged lists, P@10 counting Exact or Substitute as relevant barely moves for Qwen3-Reranker (0.837) and rises only to 0.863 for Jev, while strict P@10 rises clearly. That matches the question both were asked: whether an item exactly matches.
CLM’s failure is not a setup error. On 100 validation requests (judged lists) we compared the configuration used on test with two variants, removing the question and removing the trained head (Table 7). No variant does better, and CLM’s order is nearly uncorrelated with the embedding order it was given. Its top picks are on-topic (for “t-shirt design” all five are t-shirts), but it cannot tell a graphic tee (Exact) from a plain one (Substitute). Every candidate in a retrieved list is already on-topic, so that fine distinction is the whole task. CLM’s training data, general question answering and agent traces, contains little of it; its authors report gains mainly after task-specific fine-tuning.

6. Limitations and Acceptance-Critical Extensions

  • Baseline breadth. The paper compares Jev and CLM with one strong open cross-encoder, embedding order, random order and an oracle. This is enough to establish whether the proposed question-as-ranker formulation is plausible, but not enough to claim dominance over the reranking literature. A conference-strength extension should add at least two additional cross-encoder rerankers, a prompted LLM reranker, and a score-fusion baseline that combines embedding similarity with the L2 score.
  • Prompt and label-form sensitivity. Every model saw one fixed binary question and no task-specific training. A different wording, a four-way Exact/Substitute/Complement/Irrelevant question, calibrated probabilities, or fine-tuning could change the ranking, CLM’s above all. The next version should report a small prompt-ablation table on the validation sets before claiming that the effect is prompt-robust.
  • Closed-model reproducibility. The main positive result is for Jev, a commercial API model. We pin the version, record manifests and report cost, but the model cannot be fully inspected or archived by readers. For this reason, the central claim is intentionally Jev-specific: the study shows that at least one decision model transfers to L2 product reranking; it does not show that all decision models do.
  • Latency is setup-dependent. Local models ran unbatched on one A40, and Jev’s calls went through a client that opened a new connection per call and reached about 12 calls per second, well below the documented limit. Neither number is a model’s best case, and per-call API latency was not recorded.
  • Costs rest on list prices. Local GPU time is priced at AWS’s on-demand g6e.xlarge rate applied to A40 timings; spot, reserved or owned GPUs cost less, which would widen Jev’s cost gap. Jev’s cost counts input tokens only, at the price at the time of the runs.
  • One retriever. All lists come from one dense L1 model. Rerankers might gain more, or less, over a keyword or hybrid first stage.
  • WANDS is small. With 299 test requests it confirms the gains over embedding order but cannot separate Jev from Qwen3-Reranker.
  • ESCI label bias. ESCI’s labels come from Amazon’s own results (Section 5.5). The judged setting and WANDS guard against this, but the retrieved-setting numbers on ESCI should be read with it in mind.
  • Model versions. Results hold for jev-1.13.0, Qwen3-Reranker-0.6B at a pinned revision and CLM-v0.1-8B; API models can change behind a fixed name.

7. Conclusions

This paper asked whether a general decision model can serve as the L2 reranker of a content-based recommender. We cast the test as a pointwise reranking problem: the model asks one question per candidate, whether the item exactly matches the request, and scores the candidate by the probability of yes. The question targets the distinction left after retrieval, exact match against close substitute, and puts a cross-encoder and decision models on identical inputs.
Under a pre-registered protocol that requires a gain on both retrieved and fully labelled lists, Jev, used zero-shot, gave the highest P@5 and P@10 on ESCI and WANDS, improved P@10 over embedding order by 0.046 to 0.054, and beat the Qwen3-Reranker cross-encoder significantly in both ESCI settings. Not every decision model transfers: CLM-v0.1-8B ranked no better than random, because it recognises a request’s topic but not which on-topic item matches it. For a team choosing an L2 model the trade-off is concrete. Qwen3-Reranker delivers a quarter to two-thirds of Jev’s P@10 gain (26% on ESCI judged, 56% on ESCI retrieved, 69% on WANDS), runs locally, and costs $0.63 to $0.82 per 1,000 requests on a rented AWS GPU against Jev’s $2.38 to $2.75.
These findings are bounded by one question wording, one dense retriever and two product catalogs; they are strongest for Jev specifically, and WANDS is too small to separate Jev from Qwen3-Reranker. The current contribution is therefore best positioned as a pre-registered empirical study of whether decision-model probabilities can be used as L2 reranking scores, not as a final claim that decision models should replace cross-encoders. The next version should add a prompt-ablation table, broader reranker baselines, score fusion with embedding order, latency measured at each model’s best serving setup, and replication on a larger fully labelled catalog.

Appendix A. Reproducibility

Code, configurations and both pre-registrations are available in the project repository. Each stage is run by scripts/run_esci.sh; WANDS uses the same script with DATASET=wands. Each run writes a manifest with model version, list hash, template hash, hardware, concurrency, retries, failures and billed tokens.
Frozen test lists. The first 16 characters of each sha256 hash are:
ESCI retrieved aba1d03fff960f02
ESCI judged 78e53c70bff8e07c
WANDS retrieved b87fb97c9fed154b
Full hashes, including the validation lists and query sets, are in the pre-registration files.
Pinned versions.
Qwen3-Embedding-0.6B 97b0c614
Qwen3-Reranker-0.6B e61197ed
Jev jev-1.13.0
CLM-v0.1-8B CLM_v0.1-8B.pt over Qwen3-8B served by vLLM
What the study cost. Jev about $3.70 in API usage across both datasets; one A40 rented for about 17 hours, of which about 4 were GPU work (embedding the 1.2M-product ESCI catalog took about 2) and the rest waiting on API runs. No run had a failed call.

Author Contributions

Conceptualization, P.B. and H.G.; methodology, P.B. and H.G.; software, P.B. and H.G.; validation, P.B. and H.G.; formal analysis, P.B. and H.G.; investigation, P.B. and H.G.; data curation, P.B. and H.G.; writing—original draft preparation, P.B. and H.G.; writing—review and editing, P.B. and H.G.; visualization, P.B. and H.G. All authors have read and agreed to the published version of the manuscript.

Funding

This research received no external funding.

Institutional Review Board Statement

Not applicable.

Data Availability Statement

The code, configurations, pre-registrations, frozen candidate-list hashes, and run manifests are intended to be made available at https://github.com/bangadpurva/esci-l2-bench. The public source datasets used in the study are Amazon ESCI and Wayfair WANDS.

Acknowledgments

The authors have no affiliation with, and received no funding, API credits or other support from, the developers of any model evaluated in this paper. The authors used OpenAI’s ChatGPT to assist with language editing, formatting, and revision of the manuscript. The authors reviewed and take responsibility for all content, analyses, citations, and conclusions.

Conflicts of Interest

The authors declare no conflicts of interest. The authors have no affiliation with, and received no funding, API credits, or other support from, the developers of any model evaluated in this paper.

References

  1. Pasquale Lops, Marco de Gemmis, and Giovanni Semeraro. Content-based recommender systems: State of the art and trends. In Recommender Systems Handbook; Springer, 2011; pp. 73–105.
  2. Paul Covington, Jay Adams, and Emre Sargin. Deep neural networks for YouTube recommendations. In Proc. 10th ACM Conf. Recommender Systems (RecSys); 2016.
  3. Rodrigo Nogueira and Kyunghyun Cho. Passage re-ranking with BERT. arXiv 2019, arXiv:1901.04085.
  4. Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, Fei Huang, and Jingren Zhou. Qwen3 Embedding: Advancing text embedding and reranking through foundation models. arXiv 2025, arXiv:2506.05176.
  5. Weiwei Sun, Lingyong Yan, Xinyu Ma, Shuaiqiang Wang, Pengjie Ren, Zhumin Chen, Dawei Yin, and Zhaochun Ren. Is ChatGPT good at search? Investigating large language models as re-ranking agents. Proc. Conf. Empirical Methods in Natural Language Processing (EMNLP) 14918–14937.
  6. Kwok, J.; Kang, H.; Suresh, T.; Saad-Falcon, J.; Pavone, M.; Ré, C.; Mirhoseini, A. Contrastive language models: A system one model for fast and generalizable decision-making. Blog post. 2026. Available online: https://contrastive-lm.notion.site.
  7. TypeSafe. Jev decision model, version jev-1.13.0. System One API. Available online: https://docs.typesafe.ai (accessed on Oct. 2026 2026).
  8. Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. BEIR: A heterogeneous benchmark for zero-shot evaluation of information retrieval models. In Proc. NeurIPS Datasets and Benchmarks Track; 2021.
  9. Chandan K. Reddy, Lluís Màrquez, Fran Valero, Nikhil Rao, Hugo Zaragoza, Sambaran Bandyopadhyay, Arnab Biswas, Anlu Xing, and Karthik Subbian. Shopping queries dataset: A large-scale ESCI benchmark for improving product search. arXiv 2022, arXiv:2206.06588.
  10. Yan Chen, Shujian Liu, Zheng Liu,Weiyi Sun, Linas Baltrunas, and Benjamin Schroeder. WANDS: Dataset for product search relevance assessment. In Proc. 44th European Conf. Information Retrieval (ECIR); 2022, Available online: https://github.com/wayfair/WANDS.
  11. Tetsuya Sakai. Alternatives to Bpref. In Proc. 30th Annu. Int. ACM SIGIR Conf.; 2007, pp. 71–78.
  12. Sture Holm. A simple sequentially rejective multiple test procedure. Scand. J. Stat. 1979, 6(2), 65–70.
  13. Amazon Web Services. Amazon EC2 on-demand pricing: g6e.xlarge, us-east-1. Available online: https://aws.amazon.com/ec2/pricing/on-demand/ (accessed on Oct. 2026 2026).
Figure 1. Evaluation pipeline, run identically on ESCI and WANDS. Only the L2 model varies; candidate lists are frozen before any reranker is run.
Figure 1. Evaluation pipeline, run identically on ESCI and WANDS. Only the L2 model varies; candidate lists are frozen before any reranker is run.
Preprints 238177 g001
Figure 2. Strict P@10 (top) and P@5 (bottom) on the test sets, 1,000 ESCI and 299 WANDS requests; dashed line = label oracle (ceiling). Jev ranks first in every test and metric; CLM sits at the random-order floor.
Figure 2. Strict P@10 (top) and P@5 (bottom) on the test sets, 1,000 ESCI and 299 WANDS requests; dashed line = label oracle (ceiling). Jev ranks first in every test and metric; CLM sits at the random-order floor.
Preprints 238177 g002
Figure 3. Differences in strict P@10 on the test sets. Intervals against embedding order are at the Holm-adjusted level; Jev minus Qwen3-Reranker at 95%. Hollow marker = not significant; on WANDS the Jev minus Qwen3-Reranker interval reaches zero.
Figure 3. Differences in strict P@10 on the test sets. Intervals against embedding order are at the Holm-adjusted level; Jev minus Qwen3-Reranker at 95%. Hollow marker = not significant; on WANDS the Jev minus Qwen3-Reranker interval reaches zero.
Preprints 238177 g003
Figure 4. P@10 gain over embedding order against cost, test sets at 100 candidates. Local models costed at the AWS g6e.xlarge on-demand rate ($1.861 per hour), Jev at billed input tokens.
Figure 4. P@10 gain over embedding order against cost, test sets at 100 candidates. Local models costed at the AWS g6e.xlarge on-demand rate ($1.861 per hour), Jev at billed input tokens.
Preprints 238177 g004
Table 1. Models compared.
Table 1. Models compared.
Model How it scores a list Runs on
Qwen3-Reranker-0.6B (cross-encoder) P ( yes ) per pair, the question as its instruction A40 GPU, fp16
Jev, jev-1.13.0 (decision model, API) One yes/no question per pair TypeSafe API, 32 concurrent calls
CLM-v0.1-8B (decision model, open) One call per request; candidates scored jointly A40 GPU, Qwen3-8B via vLLM
Table 2. Datasets.
Table 2. Datasets.
Catalog Test Valid. Labels/q Scale
ESCI (US, Task 1) 1,215,854 1,000 500 16 E/S/C/I
WANDS 42,994 299 80 221 E/P/I
ESCI test requests are sampled from its test split, validation from train. Labels/q: median labels per test request. E/S/C/I = Exact/Substitute/Complement/Irrelevant; P = Partial.
Table 3. Strict P@10 and P@5 on test. Parentheses: difference from embedding order; every difference shown is significant (Holm-adjusted p = 0.0008 , the bootstrap floor). Best reranker in bold.
Table 3. Strict P@10 and P@5 on test. Parentheses: difference from embedding order; every difference shown is significant (Holm-adjusted p = 0.0008 , the bootstrap floor). Best reranker in bold.
ESCI judged ESCI retrieved WANDS
(1,000 q) (1,000 q) (299 q)
P@10 ↑
Oracle (ceiling) 0.679 0.365 0.675
Jev 0.560 (+0.046) 0.210 (+0.050) 0.526 (+0.054)
Qwen3-Reranker 0.527 (+0.012) 0.188 (+0.028) 0.509 (+0.037)
Embedding order 0.515 0.159 0.472
CLM 0.445 (−0.070) 0.049 (−0.110) 0.280 (−0.192)
Random 0.433 0.040 0.278
P@5 ↑
Oracle (ceiling) 0.859 0.552 0.763
Jev 0.676 0.284 0.577
Qwen3-Reranker 0.617 0.253 0.559
Embedding order 0.583 0.206 0.520
CLM 0.457 0.054 0.273
Random 0.440 0.043 0.276
Table 4. Head to head, Jev minus Qwen3-Reranker (exploratory; paired bootstrap, 95% CI, Holm across six comparisons per setting).
Table 4. Head to head, Jev minus Qwen3-Reranker (exploratory; paired bootstrap, 95% CI, Holm across six comparisons per setting).
Setting Δ P@10 [95% CI] Δ P@5 [95% CI] Holm p
ESCI judged +0.033 [+0.026, +0.041] +0.059 [+0.046, +0.072] 0.001
ESCI retrieved +0.022 [+0.015, +0.029] +0.032 [+0.020, +0.043] 0.001
WANDS +0.017 [−0.000, +0.034] +0.019 [−0.005, +0.043] 0.11
Table 5. Latency per request (p50 / p95, seconds) and cost per 1,000 requests at 100 candidates. †Jev latency is bound by our client (Section V-D), not by the model.
Table 5. Latency per request (p50 / p95, seconds) and cost per 1,000 requests at 100 candidates. †Jev latency is bound by our client (Section V-D), not by the model.
ESCI judged ESCI retr. WANDS Cost / 1K
(∼20 cand.) (100) (100) ESCI / WANDS
Qwen3-Reranker 0.20 / 0.49 1.21 / 1.23 1.58 / 1.70 $0.63 / $0.82
CLM 0.59 / 1.56 3.82 / 5.27 4.11 / 5.59 $1.97 / $2.12
Jev† 1.30 / 4.98 11.37 / 14.22 12.52 / 14.45 $2.38 / $2.75
Table 6. Label coverage of the L1 top 100 (test).
Table 6. Label coverage of the L1 top 100 (test).
ESCI WANDS
Labelled share of the top 10 (judged@10) 0.248 0.870
Labelled share of the top 100 0.075 0.708
Recall of Exact products at 100 0.498 0.624
Requests with an Exact product in the top 100 0.806 0.940
Table 7. CLM diagnostic probe, 100 validation requests (judged lists). ρ : rank correlation with embedding order.
Table 7. CLM diagnostic probe, 100 validation requests (judged lists). ρ : rank correlation with embedding order.
P@10 P@5 NDCG@10 ρ
Embedding order 0.524 0.602 0.745 1.00
CLM, as run on test 0.463 0.478 0.608 0.10
CLM, without the question 0.470 0.474 0.615 0.16
CLM encoder alone, no trained head 0.471 0.494 0.636 0.03
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.
Copyright: This open access article is published under a Creative Commons CC BY 4.0 license, which permit the free download, distribution, and reuse, provided that the author and preprint are cited in any reuse.