Decision Models as Product Rerankers for E-Business Search and Recommendation: An Independent Evaluation on the Amazon Shopping Queries Dataset

1. Introduction

On an e-commerce site, search and recommendation are where intent meets inventory: both decide which products a shopper sees, and in what order [1]. Shoppers rarely look beyond the first results, so the ranking of products determines what they consider and what they buy. Changes in ranking measurably affect click-through, purchases and platform revenue [2]. Product search is also harder than web search. Queries are short and specific, product texts are semi-structured, and the notion of relevance is graded. A case for the requested phone is not irrelevant to its buyer, and a cheaper model from another brand may be an acceptable substitute [1]. Retailers therefore typically retrieve a candidate set with a fast method and then rerank it with a stronger relevance model. Large-scale recommender systems follow the same retrieve-then-rank design, so the same reranking models can serve both.

The Amazon Shopping Queries dataset (ESCI) formalises graded product relevance [3]. Each query–product pair is labelled Exact (the product satisfies all of the query’s specifications), Substitute (it fails some aspects but could serve as a functional substitute), Complement (it does not fulfil the query but could be used together with an exact item), or Irrelevant. Substitutes and complements are also the two relations behind many product recommendations, such as alternatives to a viewed item and accessories bought with it [4]. The benchmark covers English, Spanish and Japanese queries, and its reranking task asks systems to order a fixed set of judged candidates per query.

Language models have changed how such relevance judgements can be made. Cross-encoders fine-tuned on relevance data [5,6], zero-shot LLM rankers [7,8], LLM-based relevance labelling for product search [9] and dedicated LLM rerankers [10] all compete for the reranking stage. LLMs are also increasingly used in recommender systems [11,12,13]. A further option appeared in September and October 2026: decision models. Given a query and a product, these models answer a typed question (for example, the four ESCI relations) with a probability for every permitted answer, in a single pass and without generating text. TypeSafe AI’s Jev introduced the interface [14]. Cloudflare’s Clef models [15] and Perplexity’s Decider [16] followed, with open weights. Their probabilities can be turned directly into a ranking score. The same judgement underlies substitute and complementary recommendations, and it is also what an AI agent shopping on a consumer’s behalf must make: is this product what the customer asked for, a substitute, an accessory, or none of these?

The only published evidence on decision models for product search is a vendor-adjacent blog post. On 250 US queries, reranking with Jev raised nDCG@10 from 0.935 (Elasticsearch hybrid search) and 0.923 (BM25) to 0.957 [17]. Its absolute values depend on its own query sample and candidate sets and are not directly comparable with ours. That study used one model and a hand-designed scoring policy. The independent evaluations of decision models published so far report no product-search ranking results [18,19,20]. The vendors’ own benchmark suite, the Decision Index, includes ESCI only as a classification task: Clef’s model card reports a macro-F1 score but no ranking quality [21].

This paper provides an independent, pre-specified evaluation at a larger scale. We rerank the full judged candidate sets of 1,000 US-English queries (20,477 pairs), 200 Spanish queries (5,314 pairs) and 200 Japanese queries (5,504 pairs) with four decision models: Jev 1.13, Clef 27B, Clef-flash 9B and Perplexity Decider v1.1 (both self-hosted and through its hosted API). The comparison set comprises a dedicated reranker, two general-purpose LLMs used in two ways each, a cross-encoder fine-tuned on in-domain data, BM25 and random ordering. Five research questions guide the study:

• RQ1: Do decision models improve product ranking over lexical retrieval (BM25)?
• RQ2: How do they compare with a dedicated reranker and with general-purpose LLMs?
• RQ3: Which question design ranks best: a four-way relevance label, an ordinal relevance score, or a yes/no exact-match judgement?
• RQ4: How well do the models recognise Substitute and Complement products, which matter for recommendations?
• RQ5: What does reranking cost per 1,000 queries?

The main contributions are:

• an independent, pre-specified evaluation of decision models as product rerankers that covers three markets and a hard-query subset, with query-level paired statistics and openly released code, prompts and predictions;
• to our knowledge, the first comparison of decision-model question designs for ranking;
• a class-level analysis of the Exact/Substitute boundary and of Complement recognition;
• an analysis of label-order stability and of agreement between hosted and locally run deployments;
• a cost–quality frontier that shows which configurations a retailer should consider.

The paper proceeds as follows. Section 2 places the study within product-search and reranking research. Section 3 details the benchmark, systems and statistics. Section 4 presents the findings by research question. Section 5 draws out what they mean for e-commerce operators and states the limitations, and Section 6 concludes the paper and outlines future work.

2. Background and Related Work

2.1. Product Search and Graded Relevance

E-commerce search differs from document retrieval in its short, attribute-heavy queries, its semi-structured product data and its business objectives [1]. Rank position has economic weight: products shown higher receive disproportionate attention and purchases [2]. Graded relevance is natural in this setting, and the ESCI labels add a commercial dimension. Substitutes and complements are classic relations in recommendation, where they are inferred from co-purchase and co-view behaviour [4]. Ranking quality is commonly measured with normalised discounted cumulative gain (nDCG), which rewards placing highly relevant items near the top [22]. Lexical ranking with BM25 remains the standard first-stage method and baseline [23].

2.2. Neural and LLM Rerankers

Cross-encoders that read a query and a candidate together and output a relevance score set the neural standard for reranking [5], and sequence-to-sequence models extended the approach [6]. Large language models can rerank zero-shot, either by listwise prompting [7,8] or by judging relevance pointwise. Fine-tuned LLMs approach human annotators in judging product relevance [9]. Dedicated rerankers built on LLM backbones, such as the Qwen3-Reranker series, are trained specifically to output a relevance probability for a query–document pair [10]. Generative LLMs that state their own confidence tend to be overconfident [13,24], and multiple-choice prompts are sensitive to option order [25].

2.3. Decision Models

Decision models receive a state and typed questions and return a probability for every permitted answer [14]. Three primitives are relevant here:

• choice: one of several labelled options;
• score: one level of an ordered scale;
• noul: yes or no.

Jev is a hosted model. Clef and Clef-flash are open-weight models with a joint scoring head on Qwen backbones [15]. Perplexity Decider v1.1 adds a decision head to the Qwen3.8-27B base model and is likewise released with open weights [16]. Independent studies found Jev accurate and calibrated on public classification tasks [18], found decision models strong when the evidence is in the input [19], and found fine-tuned classifiers more accurate wherever training labels exist [20]. For product search, the only evidence is the Elasticsearch study [17]. It reported gains over BM25 and hybrid search with a hand-weighted combination of Jev’s probabilities, but evaluated a single model on 250 queries.

3. Materials and Methods

3.1. Data

We used the reduced version (Task 1) of the ESCI test split [3], loaded from a public mirror, and removed rows with duplicate example identifiers. For each locale we drew a seeded random sample of queries and kept all of their judged products (Table 1). Each product was described by its title, brand, colour and first three bullet points, truncated to 1,200 characters. We call a query hard if its candidates include at least one Exact and at least one Irrelevant product. On such queries, a random order cannot score well by chance. The supervised cross-encoder was trained on 50,000 US pairs from the ESCI training split, whose queries are disjoint from the test queries.

3.2. Systems

Decision models.

• Jev 1.13, through the TypeSafe API pinned to jev-1.13.0.
• Clef 27B and Clef-flash 9B, run from their released code.
• Perplexity Decider v1.1 27B, in two deployments: on our own GPU with the vendor’s reference code, and through the vendor’s hosted Decisions API.

General-purpose LLMs. Qwen3.8-27B in its FP8 release [26,27] and Gemma-4-31B-it [28] were served with vLLM [29] in two modes:

• Readout. Each relation was listed under a number, and we computed the model’s exact probability of answering with each number.
• Generative. The model answered in schema-constrained JSON with a label and a 0–100 confidence.

Dedicated reranker. Qwen3-Reranker-4B [10] scored each pair as $P\left(\mathrm{yes}\right)/\left(P\right(\mathrm{yes})+P(\mathrm{no}\left)\right)$ after its standard instruction. Following the model card’s vLLM example, decoding was restricted to the two answer tokens, so both probabilities were available for every pair.

Supervised cross-encoder. ModernBERT-large [30] was fine-tuned as a four-class ESCI classifier on the 50,000 training pairs (two epochs, learning rate $2\times {10}^{-5}$, gradient clipping at 1.0, maximum length 256 tokens). These settings were chosen on a query-disjoint split of the training pairs alone, because the first configuration we tried collapsed to the label prior.

Baselines. BM25 ($k_1=1.5$, $b=0.75$, inverse document frequency computed over each market’s candidate products) [23], and random ordering averaged over 20 seeds. As pre-specified, BM25 split text into runs of word characters, without stemming. Japanese is written without spaces, so this left Japanese text largely unsegmented and made the Japanese BM25 artificially weak. After the run we therefore also computed BM25 with Japanese text split into overlapping character bigrams, the usual treatment of Chinese, Japanese and Korean text in lexical search; all other settings were unchanged. We report the Japanese gains over BM25 against this segmented baseline.

3.3. Question Designs and Ranking Scores

Every model received the same state: the shopping query followed by the product text. The question took one of three designs:

• Four-way (choice): “How does this product relate to the customer’s shopping query?”, with the four ESCI relations as options and their definitions condensed from the ESCI guidelines (Appendix A). The ranking score is the expected ESCI gain ${\sum }_{\ell }p(\ell )g(\ell )$, with gains g = 1 (Exact), 0.1 (Substitute), 0.01 (Complement) and 0 (Irrelevant), as in the benchmark.
• Ordinal (score): “Rate how well this product satisfies the customer’s shopping query”, on the ordered scale Irrelevant < Complement < Substitute < Exact. The ranking score is the expected level (0–3).
• Yes/no (noul): “Is this product an exact match for the customer’s shopping query, satisfying all of the query’s specifications?” The ranking score is $P\left(\mathrm{yes}\right)$.

The decision models and the LLM readouts answered all three designs in English. In Spanish and Japanese, and for the generative LLMs and the cross-encoder, only the four-way design was used. The cross-encoder, trained on US data, was evaluated in English only. The reranker has a single, fixed relevance question.

3.4. Measures

Within each query, candidates were sorted by score (ties broken by pair identifier).

• nDCG@10 uses the ESCI gains above [22].
• MRR is the reciprocal rank of the first Exact product; every sampled query had at least one Exact product, so MRR is defined for all queries.
• Classification measures for the four-way design are accuracy, macro-F1, recall per ESCI class, and the expected calibration error (ECE, 15 equal-mass bins) [31].
• Label-order stability was measured on 50 seeded queries (1,049 pairs): the four-way options were reversed and shuffled, and we report the share of changed labels. To separate reordering effects from ordinary variability, the two hosted APIs were sent the unchanged requests two further times.
• Cost. API calls were costed from the input tokens the providers billed, at the October 2026 list prices of US$0.042 (Jev) and US$0.02 (Perplexity) per million tokens. Models on our own hardware were costed from the GPU time each pair consumed on one NVIDIA H100 NVL (the wall time of a batch divided by its size), at an assumed US$2.50 per GPU-hour. Costs are reported per 1,000 queries with all of their candidates.

3.5. Statistical Inference

All comparisons are paired by query. Differences in nDCG@10 are reported with percentile 95% confidence intervals obtained by resampling queries 10,000 times [32]. Two-sided bootstrap p-values are floored at ${10}^{-4}$ and adjusted with the Holm procedure [33]. Each family of adjusted tests contains all systems compared with the same reference (BM25 or the reranker), in the same market and on the same query subset. Design comparisons form one family per market. Because of the floor, the smallest attainable adjusted p-value is 0.0026 in English (26 systems) and 0.0011 in Spanish and Japanese (11 systems). Differences are computed from unrounded scores, so they can differ by 0.001 from the difference of two rounded values; for example, Jev’s Japanese gain over the segmented BM25 is 0.0995, reported as 0.100.

3.6. Pre-Specification and Data Hygiene

Before the main run, we committed and tagged the analysis plan, the code and the SHA-256 hashes of the test files. Three problems surfaced in pilot runs and were fixed before the freeze:

• Missing brand, colour or bullet fields had been rendered as the literal text “nan” in 6,638 of the 20,477 US product texts. The texts were rebuilt with the same queries and pairs.
• The reranker’s probabilities had been read from a truncated list of next-token candidates, which created artificial ties.
• The first cross-encoder configuration collapsed, as described above.

The analysis plan specified bootstrap p-values with Holm adjustment, but the frozen code reported only the intervals. The p-values and their adjustment were added after the main run; they change no score or interval. The segmented Japanese BM25 (Section 3) and the comparison of mean confidence with accuracy (Section 4.4) are the only other analyses added after the run.

Clef’s model card lists ESCI classification among its own benchmarks (macro-F1 57.5) and publishes no training-data manifest [21], so Clef’s exposure to ESCI can neither be shown nor ruled out. Perplexity Decider’s training manifest includes two Amazon review datasets (review polarity and counterfactual statements) but no ESCI or other product-search data [16].

4. Results

4.1. Ranking Quality in the US Market (RQ1, RQ2)

Table 2 and Figure 1 summarise ranking quality on the 1,000 US queries. BM25 reached an nDCG@10 of 0.707 and random ordering 0.583. Every decision model, under every question design, ranked significantly better than BM25, by 0.084 (Jev, ordinal) to 0.114 (Perplexity Decider). The gains held on the 585 hard queries (0.079 to 0.108). Perplexity Decider was the best system overall, at 0.822 locally and 0.819–0.821 through the hosted API.

The dedicated reranker reached 0.787. Both Perplexity configurations significantly exceeded it under all three designs, by 0.032 to 0.035, on all queries and on the hard subset. Clef exceeded it by 0.022 with the four-way design and by 0.021 with the yes/no design, both significant on both subsets. Clef-flash’s yes/no design was significantly better on all queries (by 0.016) but not on the hard subset. Jev’s four-way design was 0.014 better, but the difference did not survive Holm adjustment ($p_{\mathrm{Holm}}$ = 0.053). Among the general-purpose LLMs, the Qwen3.8 readout also outperformed the reranker with the four-way design (by 0.020). The Gemma-4 readout did not differ significantly from it. Both generative LLMs were significantly worse, by 0.026 (Qwen3.8) and 0.024 (Gemma-4). The fine-tuned cross-encoder (0.780) did not differ significantly from the reranker and ranked below every decision-model configuration. Measured by MRR, the decision models placed the first Exact product at an average reciprocal rank of 0.848–0.888.

4.2. Spanish and Japanese Markets

Table 3 and Figure 2 report the 200-query samples from the Spanish and Japanese stores. BM25 was weaker there (0.677 and 0.643), so the decision models’ gains over it were larger: 0.138 to 0.149 in Spanish and 0.145 to 0.172 in Japanese, all significant. With Japanese text segmented into character bigrams, BM25 reached 0.689 in Japanese, so part of its weakness came from the tokenisation. Against this baseline the decision models gained 0.100 to 0.126 in Japanese, all still significant ($p_{\mathrm{Holm}}=0.0011$); generative Gemma-4 (+0.023, $p_{\mathrm{Holm}}=0.177$) was no longer significantly better than BM25. Against the reranker (0.794 and 0.763):

• Both Perplexity configurations were significantly better in both markets (by 0.027 to 0.052).
• Clef-flash was significantly better in Spanish, and Clef in Japanese.
• Jev, Clef in Spanish and Clef-flash in Japanese had positive intervals before adjustment, but these did not remain significant after Holm adjustment over the smaller samples.
• Among the LLMs, generative Qwen3.8 was significantly worse in Spanish and generative Gemma-4 in Japanese.

4.3. Which Question Design Ranks Best? (RQ3)

Table 4 compares the three question designs within each model on the US queries. The differences were small. For Jev and Clef, the four-way design ranked significantly better than the ordinal design (by 0.009 each, $p_{\mathrm{Holm}}$ = 0.004 and 0.012). No other difference was significant after adjustment, and the four-way and yes/no designs were statistically tied for every model (at most 0.006 apart). The ordinal design was never significantly better than either alternative. For both Perplexity configurations, all three designs were within 0.003 of each other. As a consistency check, the probability of Exact under the four-way design and the probability of “yes” under the yes/no design were strongly rank-correlated within each model (Spearman’s $\rho$ between 0.950 and 0.972 over the 20,477 pairs [34]).

4.4. Substitutes and Complements (RQ4)

Class-level recognition did not mirror ranking quality (Table 5, Figure 3). As four-way classifiers, all systems performed modestly: accuracy ranged from 52.3% to 58.2% and macro-F1 from 0.452 to 0.529. Summed over both directions, confusion between Exact and Substitute was the most frequent error for every system (38–53% of all errors). Recall of Exact itself was only 45.3–63.3%. For every system, the most frequent label given to a missed Exact product was Substitute (46–78% of those errors, depending on the system), a boundary the ESCI annotators also found difficult [3]. For ranking, this confusion matters less than it seems. An Exact product mislabelled as Substitute still outranked the Complement and Irrelevant products of its query in 63–84% of such pairs, depending on the system.

Complement recognition varied widely across the models. Clef (67.6%) and Clef-flash (62.6%) recognised about two thirds of Complement products, and the Gemma-4 configurations about half (51.3–54.1%). Jev (34.9%), the fine-tuned cross-encoder (34.3%), the Perplexity configurations (26.3–26.7%) and the Qwen3.8 configurations (14.8–24.0%) recognised far fewer. Substitute recall ranged from 39.0% (Gemma-4 readout) to 62.5% (Perplexity Decider). Clef-flash was the best-calibrated system (ECE 0.022), followed by Clef (0.047). Every other system was overconfident: its mean confidence exceeded its accuracy by 0.11 to 0.43, most strongly for the Gemma-4 readout and the generative LLMs (ECE 0.406 to 0.434).

4.5. Cost and Latency (RQ5)

Table 6 and Figure 4 relate ranking quality to cost, for 1,000 US queries with their 20.5 candidates on average.

• Cheapest. The fine-tuned cross-encoder (US$0.02) and the dedicated reranker (US$0.04). The cross-encoder’s cost covers inference only, not its training or the 50,000 labels it required.
• Hosted APIs. The hosted Perplexity API cost US$0.10–0.14 depending on the design, and Jev US$0.37–0.51.
• Self-hosted decision models. Clef-flash cost US$0.29–0.37, and Clef and the local Perplexity Decider US$0.76–1.00.
• General-purpose LLMs. The Qwen3.8 readout was the most expensive configuration, at US$3.02–6.60. Like the other readouts, it scored each numbered option separately on our serving stack.

Only four configurations lie on the cost–quality frontier: the cross-encoder, the reranker, the hosted Perplexity API with the yes/no design (0.821 for US$0.10) and the local Perplexity Decider with the four-way design (0.822 for US$0.98). Moving from the reranker to the hosted Perplexity API raised nDCG@10 by 0.034 for an extra US$0.06 per 1,000 queries. Moving on to the local Perplexity Decider added only 0.001 at about ten times the cost of the hosted API.

With the four-way design, the hosted APIs answered each pair in 0.28–0.38 seconds of round-trip time. A query’s candidates can be scored in parallel, so per-query latency is close to the per-pair figure. Locally, the decision models needed 0.026–0.070 GPU-seconds per pair.

4.6. Stability and Deployment

Table 7 reports how often a model’s four-way label changed when the options were reordered.

• The Qwen3.8 readout changed 26.4% (reversed) and 20.0% (shuffled) of its labels, and the Gemma-4 readout 6.8% and 8.8%.
• Jev changed 9.3% and 5.7%, well above the 3.6–3.9% of labels that changed when identical requests were simply resent.
• Perplexity Decider changed 4.9% and 3.5%.
• Clef changed none, because it sorts the options internally. Clef-flash changed one label in 1,049.

The hosted Perplexity API returned identical answers to repeated requests. Its labels agreed with those of the locally run model on 96.7–98.5% of pairs, and their nDCG@10 differed by at most 0.003 in English and by 0.005 and 0.008 in the Spanish and Japanese samples (Table 8).

5. Discussion

5.1. Implications for Retailers and Marketplaces

Decision models are a credible reranking layer. Every decision model lifted ranking quality well above lexical search in all three markets, and the best ones exceeded a dedicated reranker. The largest gains came in the Spanish store, where BM25 was weakest; the Japanese BM25 was weak partly because of its tokenisation. Operators of multilingual storefronts, where simple lexical baselines are weaker, are likely to benefit most.

Pick by measured ranking quality, and check class-level recognition separately. Ranking quality did not predict how well the individual ESCI relations were recognised. Perplexity Decider ranked best and had the highest four-way accuracy (58.2%), yet it recognised only about a quarter of Complement products. Clef recognised two thirds of them while ranking slightly lower, and Clef-flash had the highest macro-F1. A retailer choosing a reranker should evaluate nDCG on its own judged queries. A retailer building recommendation features, such as “frequently bought together”, should evaluate Complement recall separately.

Keep the question simple. The four-way label was never significantly worse than the alternatives and was sometimes better. A yes/no exact-match question gave nearly the same ranking at lower cost per pair, because it requires fewer input tokens. Elaborate scoring policies therefore appear unnecessary for ranking alone.

The hosted API is the efficient choice at moderate volume. The hosted Perplexity API reached the quality of the locally run model for about one seventh of its GPU-equivalent cost, at US$0.10–0.14 per 1,000 queries. For a store serving one million searches a day, that is roughly US$100–140 per day for reranking 20 candidates per search. The cheaper dedicated reranker or a trained cross-encoder remains attractive when latency budgets are tight or when in-house relevance labels exist. However, a cross-encoder trained on 50,000 in-domain pairs did not reach the decision models’ ranking quality here.

Agentic commerce. An AI agent buying on a consumer’s behalf must decide whether a product is exact, an acceptable substitute, or a needed accessory. The Exact/Substitute confusion we observed in every system suggests that agents should confirm substitutions with the user rather than act on a single label. The order sensitivity of several models suggests that agents should present options in a fixed order.

5.2. Limitations

• Fixed candidates. We studied reranking of fixed, fully judged candidate sets, as in ESCI Task 1, not end-to-end retrieval. Gains in production depend on what the first stage retrieves.
• Query-driven, not personalised. Relevance was judged against the query alone, without user history or profile. The results therefore apply to the ranking stage of query-driven search and recommendation, not to personalised recommendation, where relevance also depends on the individual shopper.
• Smaller multilingual samples. The Spanish and Japanese samples have 200 queries each, so their intervals are wide.
• ESCI as a benchmark. ESCI labels are themselves noisy at the Exact/Substitute boundary, and nDCG with ESCI gains rewards Exact far more than Substitute. Other gain choices could change close rankings.
• Possible exposure. Clef’s possible exposure to ESCI cannot be assessed (Section 3). More generally, ESCI has been public since 2022, so exposure during pretraining cannot be ruled out for any LLM-based system.
• Single runs. Each configuration was run once. Jev changed 3.6–3.9% of its labels on identical repeated requests (Table 7), so its scores carry some run-to-run variation; this matters most for its comparison with the reranker ($p_{\mathrm{Holm}}=0.053$).
• Model versions. The hosted models were pinned to the versions they served in October 2026.
• Costs. Costs use list prices and an assumed GPU price.
• General-LLM implementation. The general-purpose LLM readouts were not optimised for throughput, so their costs are upper bounds.
• Product text. We used title, brand, colour and up to three bullet points. Richer product data (images, specifications) might change the results.

6. Conclusions and Future Work

On the Amazon Shopping Queries benchmark, decision models proved strong and economical product rerankers, applicable to the ranking stage of both search and recommender systems. All of them improved substantially on BM25 in English, Spanish and Japanese, and Perplexity Decider significantly outperformed a dedicated LLM reranker in all three markets, and Clef and Clef-flash in two (Clef-flash in English only with the yes/no design). The way the relevance question was asked mattered little, so a simple four-way or yes/no question suffices. As classifiers, the models shared the human difficulty of separating Exact from Substitute products and differed widely in recognising Complements. The hosted Perplexity API offered the best ranking quality per dollar among the strongest systems. Retailers should validate rerankers on their own judged queries, evaluate complement recognition separately from ranking, and fix the order of answer options.

Future work should test decision models in end-to-end retrieval, with richer product data, in personalised recommendation settings that combine the query with the shopper’s history, and in live A/B experiments that measure conversion. Because a decision model’s state is free text, a shopper’s history could be added for personalised recommendation. Uneven Complement recall also calls for evaluating complementary recommendation on co-purchase data [4].