Decision Models for Business Customer-Service Triage: Accuracy, Calibration, Out-of-Scope Detection and Cost of Jev, Clef and Perplexity Decider

1. Introduction

Every customer-facing business must decide, for each incoming message, what the customer wants and who should handle it. Contact centres route requests to queues, trigger self-service flows, escalate urgent cases and recognise requests they cannot serve. These triage decisions are high-volume, repetitive and consequential. A misrouted request costs handling time and customer goodwill, and a request that is wrongly automated can cost more than one that is sent to a human [1,2]. Generative artificial intelligence has already changed front-line service work: in a large field study, access to a generative assistant increased the number of customer issues that support agents resolved per hour by 14% on average [3]. Upstream of the agent, however, the core operational question is a classification decision with a fixed set of options, and the business needs to know how far it can trust that decision.

Three families of tools now compete for this task. The first is the general-purpose large language model (LLM) prompted to pick an intent. It can either generate the answer, optionally with a verbalized confidence, or be read out through the probabilities it assigns to each option [4]. The second is the classical supervised classifier fine-tuned on labelled historical tickets. The third is new. In September and October 2026 several vendors released decision models: models that take a state (for example, a customer message) and a typed question with a fixed set of options and return a probability for every option in a single pass, without generating text. TypeSafe AI’s hosted Jev introduced the format [5]. Cloudflare then released the open-weight Clef and Clef-flash [6,7], and Perplexity released the open-weight Perplexity Decider together with a hosted Decisions API [8]. The vendors market these models for routing, moderation and other business decisions where application code thresholds the returned probabilities.

For a manager, the attraction is clear: a probability per option supports automation thresholds, escalation rules and audit trails without parsing free text. But several practical questions remain open. Are decision models more accurate than simply reading the same probabilities out of a general LLM? Are their probabilities better calibrated, so that a threshold of 0.9 means what it says? Do they recognise requests that fall outside the business’s intent catalogue? Are their decisions stable when the option list is reordered, as happens whenever a catalogue is edited? And what does a decision cost when the model is hosted by the vendor rather than run in-house?

Early independent evaluations have begun to answer some of these questions. They benchmark Jev across many public datasets [9], compare general decision models with LLMs on a bilingual benchmark [10], and compare decision models with trained classifiers and generative LLMs for automated decision gates [11]. None of them, however, isolates what decision post-training adds on top of a given backbone. This question matters for business adoption because two of the new models, Clef and Perplexity Decider, are both post-trained from the same open LLM, Qwen3.8-27B [7,8]. A firm that already runs Qwen3.8 can therefore read decisions out of it directly, without any additional model. Comparing the decision models with this same-backbone readout separates the value of decision post-training from the value of the backbone itself. The vendor-reported numbers on intent benchmarks [6] also come without calibration, out-of-scope, robustness or cost analysis, and so do not answer the operational questions above.

This paper fills that gap with a controlled benchmark on customer-service triage. We evaluate Jev 1.13, Clef 27B, Clef-flash 9B and Perplexity Decider v1.1 27B, both locally and through its hosted API. We compare them with the probability readout and the generative output of Qwen3.8-27B and Gemma-4-31B, with a fine-tuned ModernBERT-large, with TF-IDF logistic regression and with a majority-class baseline. The evaluation uses three public customer-service intent datasets: BANKING77 (77 banking intents), Bitext (27 retail customer-support intents) and CLINC150 (150 intents plus an out-of-scope class). Every model receives identical content: the same message, instruction, options and option descriptions. We address five research questions:

• RQ1: How accurate and how well calibrated are decision models relative to LLM readouts, generative LLMs and trained classifiers?
• RQ2: What does decision post-training add over its own backbone (Clef and Perplexity Decider versus the Qwen3.8-27B readout)?
• RQ3: How well do the approaches recognise out-of-scope (OOS) requests that the business cannot serve?
• RQ4: How stable are decisions when the option order changes?
• RQ5: What does a decision cost, and how fast is it, in hosted and self-hosted deployments?

The study makes four contributions. First, it provides the first same-backbone comparison of commercial decision models with a training-free readout of their base LLM. Second, it evaluates calibration and selective automation, that is, how many requests can be automated at a fixed error rate. These are the quantities that determine business value when decisions are thresholded. Third, it quantifies decision churn under option reordering against a measured run-to-run noise floor, and it documents that hosted and self-hosted copies of the same model do not return identical decisions. Fourth, it translates these results into deployment guidance for customer-service operations. Code, prompts, frozen test-set hashes and all 197,960 raw predictions are released.

The rest of the paper is organised as follows. Section 2 reviews related work. Section 3 describes the data, systems, metrics and statistical procedures. Section 4 reports the results, and Section 5 discusses the managerial implications and limitations. Section 6 concludes.

2. Background and Related Work

2.1. Automated Triage in Customer Service

Artificial intelligence in service moves from mechanical tasks towards analytical and intuitive ones [1], and request triage sits at the mechanical-to-analytical boundary. Intent detection is its technical core. The benchmark datasets used here were created for this purpose: BANKING77 for fine-grained intents in a single domain [12], and CLINC150 for intent classification with out-of-scope prediction [13]. For a business, the relevant output is not only the predicted intent but also a decision rule. Each request is automated, routed to a queue or escalated to a human, and that rule depends on how much confidence can be placed in the prediction.

2.2. Large Language Models as Classifiers

An instruction-tuned LLM can classify text in three ways. It can generate the label, optionally with a confidence statement. Its next-token probabilities over option identifiers can be read as a distribution. Or it can be fine-tuned. Studies of text classification find that fine-tuned smaller models still usually outperform zero-shot or in-context prompting of large generative models when labelled data are available [14,15,16]. Reading probabilities from tokens is not straightforward either. First-token probabilities can disagree with the text the model would generate [17], and LLMs show option-identifier and position biases in multiple-choice settings [18,19,20]. In this paper the readout computes the exact probability of every complete option identifier, including multi-digit numbers, rather than the probability of its first token (Section 3.2).

Calibration means that predicted probabilities match observed frequencies, and modern neural networks are often miscalibrated [21]. Pre-trained transformers are comparatively well calibrated in-domain [22]. Large LMs can be well calibrated on properly formatted multiple-choice questions [23,24], but they are often poorly calibrated in question answering [25]. Verbalized confidence (asking the model to state a probability) is attractive because it needs no access to logits. Elicited verbal confidences tend to be overconfident [26], although for models tuned with human feedback they can be better calibrated than token probabilities on question-answering tasks [27]. Whether this holds for many-option business classification is an empirical question that we test.

2.3. Decision Models

A decision model receives a state and one or more typed questions and returns a probability distribution over each question’s allowed options. The choice question type, for example, carries an instruction and a set of options with short descriptions. No text is generated, so the output cannot be malformed and every option receives an explicit probability. Jev, from TypeSafe AI, is the hosted model that introduced this interface [5]. Clef and Clef-flash use the same interface. They attach a joint schema head to a Qwen backbone and score all options in one forward pass [6,7]. Perplexity Decider v1.1 is likewise post-trained from Qwen3.8-27B, with a decision readout head, non-causal attention in its full-attention layers and a saved calibration temperature. Its training mix draws partly on the tasksource collection [8,28].

Independent evidence on these models is recent. Deußer et al. evaluated Jev on 37 datasets and found it more accurate than Qwen3.8-27B on most of them. They also found its choice probabilities well calibrated, and rotating the options left its accuracy unchanged [9]. Duan et al. introduced a bilingual benchmark on which general decision models did well when answers could be drawn from available evidence. The same models faltered on specialist knowledge and on calibrated probabilities [10]. Rafe and Das compared eight decision-model checkpoints with trained classifiers and generative LLMs. They found that a fine-tuned DeBERTa-v3 classifier led on almost every labelled benchmark, and that Gemma-4-31B exceeded Jev on CLINC150 at higher cost and latency [11]. Our study differs in three respects. We compare decision models with the probability readout of their own backbone, separate calibration from accuracy and measure automation coverage at a fixed risk. We also measure decision churn against a noise floor, and we price hosted and self-hosted deployment of the same weights.

2.4. Selective Automation and Out-of-Scope Detection

Thresholding a classifier’s confidence to decide which cases to automate is the classical reject option [29]. It is evaluated with risk–coverage curves, which plot the error rate among accepted cases against the share of cases accepted [30], and summarised by the area under that curve (AURC) [31]. Out-of-scope requests are a related but distinct failure mode. A request that matches none of the business’s intents should be flagged rather than forced into the closest intent. The maximum predicted probability is the standard baseline score for such detection [32]. CLINC150 showed that classifiers trained on in-scope intents often have poor out-of-scope recall [13], which motivated dedicated open-intent methods [33]. Decision models allow a third approach: offering “out of scope” as an explicit option with its own description. We compare both approaches.

3. Materials and Methods

3.1. Datasets

Table 1 summarises the three datasets. For BANKING77 [12] we used the full official test set. For CLINC150 [13] we used the “plus” configuration and its full test set: 4,500 in-scope requests and 1,000 OOS requests. The Bitext customer-support dataset [34] has no official split. It is hybrid-synthetic: according to its dataset card, it was generated with natural-language-generation tools from natural seed texts and curated by linguists. We removed duplicate texts, split the remainder 80/20 with stratification by intent, and dropped test texts whose normalised form also occurred in the training part. From the rest we drew a stratified test sample of 2,000 requests. Training data were used only by the supervised baselines; for Bitext the training part was capped at a stratified 15,000 requests. The longest test message has 69 words (81 ModernBERT tokens), so no input was truncated by any model. The SHA-256 hashes of the three test files were committed before the full run (Section 3.7).

3.2. Systems

Table 2 lists the twelve systems. Every model was pinned to an exact version: the Hugging Face revision for open weights, or the version identifier returned by each API on every response.

Decision models. Jev and the Perplexity Decisions API were called with the same request: a state and one choice question. Clef, Clef-flash and the local Perplexity Decider were run with the inference code released alongside their weights [7,8], with a batch size of eight. Clef returns one logit per option, and we applied a softmax as its model card prescribes. Perplexity Decider’s predict function applies the checkpoint’s saved calibration temperature. On every batch we checked that our argmax mapping agreed with the library’s own answer function. Clef’s release code sorts the options of a choice question alphabetically before encoding them. Its predictions are therefore independent of option order by construction, a point we return to in Section 4.5.

LLM readout. For Qwen3.8-27B [35,36] and Gemma-4-31B [37], the options were numbered [1]…[K], and the assistant turn was prefilled with Best answer: [. Both tokenizers encode digits and the closing bracket as single tokens; the code verified this before every run. The probability of option k, written with digits $d_1\dots d_m$, is therefore

$$P\left(k\right)\propto {\displaystyle \prod _{j=1}^m}p(d_j\mid \mathrm{prompt},d_{<j})·p\left(\right]\mid \mathrm{prompt},d_{1:m}),$$

(1)

normalised over the K options. This is the exact probability of each complete identifier, not the probability of its first digit. We computed it by walking the digit prefix tree with vLLM [38]. At each node vLLM returned the top 64 next-token log-probabilities, and branches whose path probability fell below ${10}^{-5}$ were pruned. Tokens outside the top 64 were treated as having zero probability. The median probability mass that the readout placed on valid option strings was 0.998, 0.999 and 0.997 for Qwen3.8 on the three tasks, and 1.000 for Gemma-4, so the truncation is negligible. Thinking mode was disabled. We used the official FP8 release of Qwen3.8-27B, the format in which a firm would typically serve it. Clef and Perplexity Decider were post-trained from the original Qwen3.8-27B weights, a difference we return to among the limitations.

Generative LLM. The same two LLMs were asked to answer in JSON, giving an option identifier and an integer confidence from 0 to 100. Decoding was greedy (temperature 0), constrained by a JSON schema whose answer field enumerated the valid options. All 21,160 canonical generations were valid (100%). The stated confidence c was assigned to the chosen option, and the remaining probability $1-c/100$ was spread uniformly over the other options.

Supervised baselines. ModernBERT-large [39] was fine-tuned separately for each task on its training split: three epochs, AdamW, learning rate $5\times {10}^{-5}$, batch size 32, linear schedule with 6% warm-up, maximum length 128 tokens, bfloat16. The TF-IDF baseline concatenated word 1–2-gram and character 2–5-gram features and fitted a logistic regression ($C=10$). Both classifiers were trained with the OOS class for CLINC150, as the dataset provides it. The majority baseline predicts the training-set label frequencies.

3.3. Identical Inputs

Every system received the same content (Appendix A). The state was Customer message: <text>. The instruction was a one-sentence question, for example: Which customer-service intent does this bank customer’s message express? For CLINC150 the instruction added Choose `oos’ (out of scope) if none of the listed intents applies. The option descriptions were derived mechanically from the label names (card_arrival → “Card arrival”), and the OOS option was described as “Out of scope: the request does not match any of the other listed intents.” The descriptions were deliberately not hand-tuned, so that no system benefited from prompt engineering. Decision models received the instruction and descriptions as a choice question. The LLM readout received them as a numbered option list, and the generative LLMs as a bulleted list of option identifiers.

3.4. Experimental Conditions

• Canonical: all test requests, with options in alphabetical order of their identifiers (OOS last for CLINC150).
• Reversed and shuffled: a seeded random subset of 500 requests per task, with the option list reversed or permuted by one seeded random permutation per task.
• No OOS option: all CLINC150 test requests with the OOS option removed. OOS requests are then detected by thresholding $1-{\max }_{k}P\left(k\right)$.
• Repeat: for Jev, the robustness subset was sent twice more without any change, to measure the run-to-run noise floor of the hosted API.

The supervised classifiers have no option order and were evaluated in the canonical condition only. The Perplexity API was likewise evaluated only in the canonical condition, where it serves as the hosted counterpart of the local model. In total the study produced 197,960 predictions. Every request in every condition was answered, with no errors and no duplicates.

3.5. Metrics

Accuracy and macro-F1 measure classification quality, where macro-F1 weights every intent equally.

Calibration is measured with the top-label expected calibration error (ECE) [21,40]. Requests are sorted by confidence (the maximum probability) and split into 15 equal-mass bins, which reduces the bias of equal-width bins [41,42]. ECE is the size-weighted mean absolute gap between accuracy and mean confidence in each bin. Equal-mass binning must order requests with tied confidence values. This matters only for Jev and the two generative LLMs, whose confidences take few distinct values. We report the values produced by the frozen analysis pipeline (NumPy’s default sort order), and Section 5.4 quantifies how much other tie orders change them. We also report the Brier score [43] and the negative log-likelihood (NLL) in Appendix B.

Selective automation. Requests are accepted in decreasing order of confidence, giving a risk–coverage curve [30]. We report AURC [31] and coverage at 2% risk: the largest share of requests that can be automated while the error rate among the automated requests stays at or below 2%. This is the quantity a contact centre would set when it allows at most one misrouted request in fifty.

Out-of-scope detection on CLINC150 is measured with OOS recall, the false-alarm rate (in-scope requests labelled OOS), in-scope accuracy and the area under the ROC curve (AUROC). With the OOS option, the score is $P\left(\mathrm{oos}\right)$; without it, the score is $1-{\max }_{k}P\left(k\right)$ [32].

Option-order robustness is the flip rate: the share of requests whose decision differs between the canonical and the reordered condition. Decisions are compared by option identifier, and the rare exact ties between top probabilities (at most 0.6% of the requests of any system) are broken by canonical order. We also report the mean total-variation distance between the two probability vectors.

Cost and latency. For the APIs, cost is the mean number of billed input tokens per request multiplied by the list price at the time of the study: US$0.042 per million input tokens for Jev and US$0.02 for the Perplexity Decisions API. For self-hosted models, cost is the GPU time per decision multiplied by US$2.50 per GPU-hour, an assumed cloud rental price for one NVIDIA H100. GPU time per decision is the batch wall-clock time divided by the batch size; all costs scale linearly with the GPU price. API latency is the client-side round trip per request at the concurrency we used (12 parallel requests for Jev, 8 for Perplexity). It is therefore not directly comparable with the batch-normalised GPU time of the self-hosted models.

3.6. Statistical Analysis

All comparisons are paired, because the systems answered the same requests. For each task we compared every system with two reference systems: the Qwen3.8-27B readout (the shared backbone, for RQ2) and Jev (the original decision model). For differences in accuracy and ECE we report percentile 95% confidence intervals from 10,000 paired bootstrap resamples of requests [44]. Accuracy differences were tested with the exact McNemar test [45]. The p-values were adjusted with the Holm procedure [46] within each reference family of 33 tests (11 systems × 3 tasks). Where we describe two systems as equivalent, this is a two one-sided tests (TOST) decision [47]: the 90% bootstrap interval of the accuracy difference lies within ±2 percentage points.

3.7. Frozen Analysis Plan, Contamination Check and Compute

The analysis plan, code, prompts and test-set hashes were committed and tagged in version control before the full run. After a 100-request pilot per task, the full run took place on 11–12 October 2026 on four NVIDIA H100 NVL GPUs, each power-capped at 400 W. The software stack was vLLM 0.31.0, PyTorch 2.13.0 and Transformers 5.17.0 (5.10.2 for Clef, as its model card requires). The only change after the freeze was one additional figure.

Benchmark contamination is a concern whenever public datasets are used. Perplexity Decider ships a training-data manifest. It lists other intent datasets (snips_built_in_intents, dnd_style_intents) but none of BANKING77, CLINC150 or Bitext. Cloudflare reports BANKING77 and CLINC150 results for Clef [6], so these two datasets were explicit benchmark targets during its development, and no training-data manifest is published. We therefore treat Bitext as the cleanest test of generalisation in this study.

4. Results

4.1. Accuracy (RQ1)

Table 3 reports accuracy and macro-F1. On BANKING77 and CLINC150 the two Clef models were the most accurate systems of all, including the fine-tuned classifiers. Clef-flash reached 95.5% on BANKING77 and 98.1% on CLINC150, and Clef reached 94.4% and 97.4%. Every other zero-shot system scored between 79.4% and 80.7% on BANKING77 and between 86.5% and 91.3% on CLINC150. On Bitext the ranking changed. The local Perplexity Decider was the most accurate zero-shot system (94.0%), followed by the Perplexity API (93.6%) and the Gemma-4 readout (93.1%). Clef (90.2%) and Clef-flash (89.4%) fell below both Qwen3.8 configurations.

Our Clef macro-F1 scores (94.4 on BANKING77 and 97.6 on CLINC150) reproduce the vendor-reported 94.2 and 97.4 closely [6], as do our Jev scores (79.3 and 89.7 versus the reported 79.7 and 89.3). For Clef-flash, however, we measured far higher macro-F1 (95.5 and 98.4) than the reported 90.9 and 66.8. We confirmed from the run logs that the pinned Clef-flash checkpoint was loaded. The vendor’s evaluation protocol is not published in enough detail to explain the difference.

The trained classifiers were nearly perfect on the synthetic Bitext data: ModernBERT reached 99.8% and TF-IDF 99.9%. ModernBERT was also strong on BANKING77 (93.9%), but on CLINC150 it reached only 90.8%, below both Clef models, mainly because it missed many OOS requests (Section 4.4). Among the general LLMs, readout and generation gave similar accuracy, and Gemma-4 was slightly ahead of Qwen3.8 on Bitext and CLINC150.

4.2. What Decision Post-Training Adds over the Backbone (RQ2)

Table 4 compares each decision model with the Qwen3.8-27B readout on the same requests. Clef raised accuracy over its backbone readout by 14.9 percentage points on BANKING77 (95% CI 13.5 to 16.2) and by 9.3 points on CLINC150 (8.5 to 10.1). On Bitext it was 1.5 points less accurate (CI −2.6 to −0.5). That difference narrowly missed significance after Holm correction ($p_{\mathrm{Holm}}$ = 0.0502). Clef-flash, which uses a different and smaller backbone (Qwen3.5-9B), showed the same pattern: +16.0, −2.3 ($p_{\mathrm{Holm}}$ = 0.002) and +10.0 points. The large gains thus appear on the two datasets that were reported benchmark targets and not on Bitext.

Perplexity Decider improved on its backbone readout by smaller margins, all in the same direction: +1.2 points on BANKING77 (not significant after Holm correction, $p_{\mathrm{Holm}}$ = 0.114), +2.3 on Bitext and +3.0 on CLINC150 (both $p_{\mathrm{Holm}}<0.001$). Jev was statistically indistinguishable from the Qwen3.8 readout on BANKING77 and Bitext. It was equivalent within ±2 points on both by the TOST criterion, and 1.5 points more accurate on CLINC150. The supervised ModernBERT exceeded the readout on all three tasks.

On calibration, decision post-training did not consistently improve on the backbone readout. Compared with the readout, Clef’s ECE was not significantly different on BANKING77 or Bitext and was higher on CLINC150 (+0.025). Perplexity Decider’s ECE was higher on BANKING77 (+0.020), not significantly different on Bitext and slightly lower on CLINC150 (−0.009, upper CI bound −0.0003). Jev’s ECE was higher on BANKING77 (+0.050) and not significantly different on the other two tasks. Section 4.3 explains why: the readout of the backbone is already well calibrated.

4.3. Calibration and Selective Automation (RQ1)

Table 5 and Figure 1 show that the clearest calibration difference is not between decision models and LLMs. It lies between probability readouts and verbalized confidence. The Qwen3.8 readout had ECEs of 0.041, 0.020 and 0.019, within the range of the decision models (0.010–0.091). The two generative LLMs ranged from 0.048 to 0.166. They stated very high confidence almost regardless of correctness: Qwen3.8 used only 11–17 distinct confidence values, and its mean stated confidence on BANKING77 was 96.0% at an accuracy of 79.4%. The Gemma-4 readout was also strongly overconfident (ECE 0.067–0.195), because its probability mass concentrated on a single option. Its mean confidence on BANKING77 was 99.1% at 79.6% accuracy. Calibration quality is therefore a property of the model and the readout jointly, not of the readout method alone.

The reliability diagrams in Figure 2 reveal the direction of the errors. Clef’s mean confidence was lower than its accuracy on all three tasks, by 3.1, 0.6 and 4.5 points. On BANKING77 and CLINC150 this gap equals its ECE, so Clef’s calibration error there is underconfidence. Jev, in contrast, was overconfident on BANKING77, with a mean confidence of 89.2% at 80.1% accuracy. For automation, underconfidence is the safer error: it leads the business to automate fewer requests than it could, not more than it should.

Coverage at 2% risk (Table 5, Figure 3) translates these differences into business terms. On BANKING77, Clef and Clef-flash could automate 90.1% and 93.0% of requests with at most 2% errors. Jev and Perplexity Decider could automate 31.4–31.6%, and the Qwen3.8 readout 13.7%. On CLINC150, Clef-flash met the 2% target even at full coverage (100.0%) and Clef at 98.6%. On Bitext the ordering reversed: the Perplexity API (90.3%) and the local Perplexity Decider (89.2%) led Clef (67.8%) and Clef-flash (45.4%). Despite similar accuracy, the generative LLMs and the Gemma-4 readout could automate little safely, with coverage of 0.0–20.1% on BANKING77 and 2.8–8.2% on CLINC150. Their confidence did not separate correct from incorrect decisions.

4.4. Out-of-Scope Detection (RQ3)

Table 6 and Figure 4 report OOS detection on CLINC150. When OOS was offered as an explicit option, Clef recognised 93.7% of OOS requests and wrongly flagged only 0.2% of in-scope requests. Clef-flash achieved 94.3% and 0.3% (AUROC 0.996 and 0.997). Jev and Perplexity Decider caught 86.7–88.6% of OOS requests at false-alarm rates of 2.1–3.0%. The generative Qwen3.8 caught the most OOS requests (95.0%), but it also flagged 10.3% of legitimate requests as out of scope, which would send one valid customer in ten to the wrong channel. The trained classifiers behaved in the opposite way. ModernBERT and TF-IDF had low false-alarm rates (0.2% and 1.4%) but caught only 64.4% and 56.0% of OOS requests: a classifier trained mostly on in-scope intents tends to force unfamiliar requests into a known intent [13].

Removing the OOS option and thresholding $1-{\max }_{k}P\left(k\right)$ instead lowered the AUROC of every system: Clef from 0.996 to 0.948, Jev from 0.978 to 0.914, Perplexity Decider from 0.985 to 0.938, and the Gemma-4 readout from 0.962 to 0.809. In-scope accuracy rose, by between 0.1 points (Clef and Clef-flash) and 7.8 points (generative Qwen3.8), because no in-scope request could be misclassified as OOS. Offering an explicit OOS option with a plain-language description was therefore the more effective way to detect out-of-scope requests for every model evaluated in both conditions.

4.5. Stability Under Option Reordering (RQ4)

Table 7 and Figure 5 report how many decisions changed when the same 500 requests per task were presented with the options reversed or shuffled. For the Qwen3.8 and Gemma-4 configurations and for Perplexity Decider, reordering changed between 0.8% and 11.2% of decisions. Churn was highest on BANKING77, whose intents are fine-grained and partly overlapping. Jev’s decisions changed in 1.0–7.6% of requests. Its noise floor, the share of decisions that changed when identical requests were simply sent again, was 0.0–2.0% (1.0% on average), so most of Jev’s churn is caused by the reordering itself.

Net accuracy hardly moved: across all systems and both reorderings, accuracy on the subset changed by between −1.8 and +3.2 points. Reordering therefore mostly swaps which requests are decided correctly, rather than making the system better or worse on average. This agrees with the finding that rotating the options left Jev’s accuracy unchanged [9], but accuracy-level invariance hides decision-level churn. For a business, churn means that editing the order of an intent catalogue silently reroutes some customers.

Clef and Clef-flash changed no decision at all. Their probability vectors were identical for at least 496 of the 500 requests in every condition, and the few remaining differences (at most 0.004) are presumably numerical effects of different batch composition. This is a design property, not learned robustness: Clef’s release code sorts the options alphabetically before the model sees them. The same remedy is available for every other system, which can be given options in a fixed canonical order.

4.6. Cost and Latency (RQ5)

Table 8 and Figure 6 report cost and speed. Averaged over the three tasks, the hosted Perplexity Decisions API was the cheapest zero-shot option, at US$18 per million decisions, followed by Jev (US$69). Self-hosted Clef-flash cost US$70, the local Perplexity Decider US$102 and Clef US$207. Cost grows with the length of the option list, which makes CLINC150 (151 options) the most expensive task for every decision model and for both generative LLMs. Running the same Perplexity weights on our own GPU cost 5.7–5.9 times the hosted price under our GPU price assumption. Clef-flash was about three times cheaper than Clef and, on BANKING77 and CLINC150, also more accurate.

The Qwen3.8 readout was by far the most expensive configuration, at US$2,519 per million decisions and 2.7–4.8 seconds of GPU time per request. This is largely an engineering artefact. Qwen3.8 spreads probability over many options, so the digit-tree search needed 55.8 next-token evaluations per request on average (24.9 on Bitext to 70.7 on CLINC150), against 2.9–3.7 for Gemma-4. In addition, the hybrid linear-attention architecture of Qwen3.8 obtained little prefix-cache reuse in our vLLM version, so each evaluation re-processed most of the prompt. The Qwen3.8 readout alone used 17.7 of the 27.9 GPU-hours of measured inference time in the study. An implementation that encodes the prompt once and scores all options from the cached state would be expected to bring the readout close to the cost of a single forward pass. Our cost estimate for the readout should therefore be read as an upper bound.

The trained classifiers cost less than US$1 per million decisions, but they require labelled training data and a training pipeline. This investment bought accuracy that no zero-shot system matched on Bitext and that only the two Clef models exceeded on BANKING77.

4.7. Hosted Versus Self-Hosted Perplexity Decider

The hosted Decisions API and the local checkpoint share the same weights, yet their decisions were not identical (Table 9). They agreed on 98.4%, 99.0% and 98.9% of requests. The mean total-variation distance between their probability vectors was 0.014–0.023, and for single requests individual option probabilities differed by up to 0.376. Accuracy was nearly identical: the largest difference was 0.4 points, on Bitext. The likely cause is numerical: differences in serving hardware, batching and kernels. Even so, a firm that validates a threshold on one deployment and runs it on the other should expect roughly one to two decisions in a hundred to differ.

5. Discussion

5.1. Principal Findings

Four findings stand out. First, decision models are not uniformly more accurate than a well-implemented LLM readout. Clef’s large gains appeared on the two datasets that its developer reports as benchmarks, and on the cleaner Bitext test it was slightly worse than its own backbone readout. Perplexity Decider, whose training manifest excludes all three datasets, gave smaller gains of 1.2–3.0 points, all in the same direction. Second, much of the calibration advantage that decision models offer over prompting an LLM comes from reading probabilities rather than generating text. The Qwen3.8 readout was as well calibrated as the decision models, whereas verbalized confidence from the same model was poorly calibrated. This result contrasts with reports that verbalized confidence can rival token probabilities on question answering [27], and agrees with the overconfidence of elicited confidence documented elsewhere [26]. Third, the most consequential operational difference was not accuracy but coverage at a fixed risk and out-of-scope behaviour, where systems with similar accuracy differed by tens of percentage points. Fourth, every system except the two Clef models changed a material share of its decisions when the option order changed. The Clef models’ invariance comes from their input pipeline rather than their training.

5.2. Managerial Implications

Validate on your own requests, not on vendor tables. Rankings changed with the dataset. Clef led clearly on BANKING77 and CLINC150 but trailed Perplexity Decider and even general LLMs on Bitext. Vendor-reported benchmark scores were reproducible in our runs for Clef and Jev, but they did not predict the ranking on a dataset outside the vendors’ reported benchmarks. In our data, a labelled sample of about 2,000 historical requests gave paired confidence intervals of roughly ±1 percentage point for differences between systems. This is enough to choose between candidates before committing to a vendor.

Choose by coverage at your risk tolerance, not by accuracy. A contact centre that tolerates 2% misrouting among automated requests could, on BANKING77-type traffic, automate about 90% of requests with Clef and about 31% with Jev or Perplexity Decider. With the generative LLMs, despite similar accuracy, it could automate at most a fifth. The business value of a triage model lies mainly in the share of requests it can handle without human review. That share depends on how well confidence separates correct from incorrect decisions, which accuracy alone does not reveal. Managers should ask vendors for risk–coverage curves, or compute them on a pilot sample.

Do not threshold verbalized confidence. When an LLM is used for triage, a confidence that it states in text is a poor basis for automation rules. Generative Qwen3.8 reported 95% or more confidence on most requests regardless of correctness. Where an LLM must be used, option probabilities read out of the model are far better calibrated. Decision models provide such probabilities by design.

Offer “out of scope” explicitly. Every system that we evaluated both ways detected out-of-scope requests better when “out of scope” was a described option than when a threshold was applied to its maximum probability. Trained classifiers missed 36–44% of out-of-scope requests, which a business would then silently route to the wrong team. Generative LLMs erred in the other direction, flagging up to one in ten legitimate requests.

Freeze the option order, and re-validate after catalogue changes. Reordering an intent list changed up to 11.2% of decisions while accuracy moved by at most 3.2 points. Firms should present options in a fixed canonical order, for example alphabetical as Clef does internally, and should re-run validation whenever the catalogue is edited.

Do not assume hosted and self-hosted copies are interchangeable. Hosted and local Perplexity Decider disagreed on roughly 1–2% of requests. Thresholds should be validated on the deployment that will actually serve traffic. At list prices, the hosted APIs were also the cheapest zero-shot option (US$18 and US$69 per million decisions). Self-hosting is justified mainly by data-protection requirements, not by cost.

Inference cost is not the binding constraint. All decision models cost between US$18 and US$207 per million decisions. These amounts are small next to the cost of handling a single misrouted request manually. The selection criterion should therefore be coverage at the target risk and out-of-scope behaviour, with price as a tie-breaker. Trained classifiers remain the cheapest and, where labelled data exist and intents are stable, very accurate. Decision models are most attractive when labelled data are scarce, when intents change often, or when out-of-scope requests are common.

5.3. Theoretical Implications

For research on LLM-based classification, the results separate three components that are often conflated: the backbone, the readout of decisions from it, and decision post-training. On our tasks the readout determined calibration more than post-training did, and post-training gains in accuracy depended strongly on the dataset. Evaluations of decision models should therefore include a same-backbone readout as a baseline and report selective-automation metrics alongside accuracy. Our results also show that order invariance can be engineered in the input pipeline. Robustness comparisons should report whether options are canonicalised before encoding.

5.4. Limitations

Several limitations qualify these conclusions. First, three public datasets cannot represent every customer-service domain. Two of them (BANKING77 and CLINC150) are vendor-reported benchmarks, and the third (Bitext) is hybrid-synthetic and easy for supervised models. Results on a firm’s own tickets may differ, which is precisely why we recommend local validation. Second, contamination cannot be ruled out for Clef or Jev, whose training data are not published. Our contamination check rests on Perplexity’s manifest and on Cloudflare’s reported benchmarks. Third, the readout used the FP8 release of Qwen3.8-27B, while Clef and Perplexity Decider were post-trained from the original weights. The same-backbone comparison is therefore exact in architecture and training lineage but not bit-for-bit in numerics, and Clef-flash uses a different backbone (Qwen3.5-9B) altogether. Fourth, the option descriptions were derived mechanically from label names. Richer, hand-written descriptions could help all zero-shot systems and could change their relative order. Fifth, ECE with equal-mass bins depends on how ties in confidence are ordered. This affects only the systems with few distinct confidence values: under 200 random orderings of ties, the largest variation was 0.006 (0.047–0.053, generative Qwen3.8 on Bitext), which changes no conclusion. Sixth, costs rely on list prices at the time of the study and on an assumed GPU price of US$2.50 per hour. The readout cost reflects an implementation without effective prefix caching, and the API latencies were measured from one client location. Finally, Jev is a hosted service whose version we pinned (jev-1.13.0), but its serving stack may change without notice. Our Jev results describe that version only.

6. Conclusions

Decision models make thresholded business decisions convenient: every option receives a probability, outputs are never malformed and hosted access is inexpensive. On customer-service triage, however, their advantages were narrower and more dataset-dependent than vendor tables suggest. Against a careful probability readout of their own backbone, decision post-training added large accuracy gains on two benchmark datasets that the vendor reports, small gains or none on a third, and no consistent calibration gain. The differences that matter most for operations were automation coverage at a fixed risk, out-of-scope handling and stability under catalogue edits. On these measures the systems differed far more than on accuracy, and the best choice differed by dataset. Firms adopting decision models should validate them on a few thousand of their own labelled requests, set automation thresholds from risk–coverage curves, offer an explicit out-of-scope option, fix the option order, and validate on the deployment that will serve production traffic. Future work should test these models on proprietary ticket streams with real business costs attached to each error, study richer option descriptions, and evaluate efficient single-pass readouts that make LLM option probabilities as cheap as decision-model calls.