Beyond Utility Evaluation for LLM-Based and Generative Recommender Systems: A Focused Survey of Offline Protocols

1. Introduction

Consider two movie recommenders: one ranks a short list containing a relevant film, while the other must find that film in the catalog. Their hit rates answer different questions. A title-generating recommender introduces a further decision: whether its output names exactly one catalog item. These choices affect both accuracy and the diversity or exposure attributed to its recommendations. Surveys of evaluation practice and trustworthy recommendation establish the importance of these distinct outcomes [1,2,3,4].

The limitation of accuracy alone predates language-model recommendation. McNee et al. [5] question the use of accuracy as the dominant design objective, while Ge et al. [6] distinguish coverage and serendipity from predictive accuracy. The present problem is how to preserve these distinctions when a recommender generates the object that the evaluator must identify and score.

LLM-based and generative recommenders also change what receives a score. A model may classify a supplied item, rank a candidate list, generate a title, or emit an item identifier. P5, TALLRec, GenRec, RecRanker, and token-level generative methods illustrate these different uses of language modeling [7,8,9,10,11]. Evaluating them requires a connection from the raw output to the credited recommendation.

That connection can change a conclusion. Jiang et al. [12] observe sensitivity to history, candidate position, and profile generation, and discuss unmatched generated titles under their hallucination label. Di Palma et al. [13] recover public benchmark records from LLMs, raising questions about what familiar-dataset accuracy establishes. These issues compound known effects of baseline tuning, sampled metrics, implementation choices, and incomplete artifacts [14,15,16,17,18]. High utility is interpretable only after the comparison task, credited item, and evaluated population are defined.

Existing LLM4Rec and generative recommendation surveys organize models and tasks [19,20,21]; trustworthy recommendation surveys organize stakeholder objectives. Our focus is the link between these two perspectives: how validity choices change the utility or beyond-utility claim supported by an experiment. Jannach and Chen’s methodological guidance on comparison rigor and reproducibility provides the closest TORS foundation [22]. Table 1 identifies the connections developed here.

We examine 20 detailed protocol comparisons within a 49-paper coded corpus. The synthesis yields two recurring findings:

• Protocol choices change reported orderings. Candidate-pool selection, credit for ambiguous identifiers, and the reference used to measure fairness each change which model or history strategy appears preferable in source-reported comparisons.
• Different measured objects can explain apparent disagreement. Long-tail candidate selection and popular-record recovery describe different operations. Attribute recoverability, user benefit, and item exposure likewise express different objectives. Reading these results together clarifies what each establishes.

We use candidate access, item identity, generalization, and outcome reference to organize the comparisons. Section 2 and Section 4 establish the review method and evaluated components. Section 5 distinguishes beyond-utility outcomes, and Section 6 connects their interpretation to the protocol comparisons. The evaluation controls in Section 7 are practical consequences of this synthesis.

2. Scope and Survey Methodology

We conduct a focused review of offline evaluation in LLM-based preference prediction, candidate ranking, and item generation. The review asks how the evaluated output determines the required evidence, which protocol choices change comparisons between systems, and how those choices affect beyond-utility interpretation. Language-based representation methods provide contrasts with direct generation. The empirical synthesis addresses utility, diversity, consumer fairness, item exposure, and generalization; interaction, explanation, and privacy motivate extensions beyond this core.

• Identification and selection.

The search was completed on 15 September 2026. We followed citation trails from LLM recommendation evaluation, generative recommendation, trustworthy recommendation, and reproducibility studies, consulting proceedings, publisher, DBLP, and arXiv records. Searches combined recommendation with evaluation, protocols, generative models, and LLMs, followed by title-based retrieval. The target publication window is 2019–2026; earlier work is included for established evaluation concepts. Selection is purposive: a study enters the detailed comparison when its reported task, candidate construction, output resolution, evaluated population, or outcome reference supports a concrete protocol contrast. Studies describing adjacent models or general principles provide contextual support. Preprint and published versions are associated with the same work, with the version used for each extracted result recorded separately.

• Extraction and comparison.

The corpus contains 49 records: 40 with full-text coding and nine with bibliographic or source-record coding. The 23-field schema records system role, task, output, data, split, candidates, metrics, validity checks, prompts, baselines, tuning, artifacts, user evaluation, and protocol threats. Source descriptions are distinguished from our interpretations. Table 2 assigns each record one primary contribution for corpus accounting, while the analysis permits a study to inform several questions.

Twenty full-text studies support the detailed protocol comparison. They cover supplied-target prediction, sampled and retrieved candidates, full-catalog scoring, generated names and identifiers, representation and memorization probes, held-out entities and prompts, and four fairness formulations. Figure 1 organizes these contrasts by the question they resolve. We compare the experimental objects and controls before interpreting reported scores; heterogeneous metrics are not pooled. The supplement links each selected study to its inclusion rationale, extracted fields, PDF locations, and the comparison it supports.

• Evidence provenance and scope.

An AI assistant performed the extraction and source rechecks; independent human coding agreement has not been established. The detailed comparison retains unspecified and conflicting source fields, and numerical contrasts were checked against the source tables. Results for CFaiRLLM and the provider-fairness study refer to the archived arXiv versions identified in the supplement, with journal metadata recorded separately. The review compares reported protocols and results rather than rerunning model code.

The initial snowballing search has no complete retrieved-and-excluded ledger. Accordingly, corpus counts describe this collection and do not estimate the prevalence of practices across the field. Privacy attacks and end-to-end agent behavior are outside the detailed empirical comparison. The broader bibliography supplies definitions, methodological foundations, and related work; it is distinct from the coded analytical sample.

Table 2. Evidence clusters used to build the survey taxonomy. The n column reports the number of papers in the conservative coded corpus, not a field-wide prevalence estimate.

Evidence cluster	n	Representative sources	Role in this survey
LLM/generative protocols	19	LLM evaluation, memorization, direct generation, reranking, identifier validity, and cold-start protocols [9,10,11,12,13,25,26]	Connects history and position sensitivity, generated-item identity, contamination, and unseen-entity evaluation to the meaning of reported performance.
Classical evaluation methodology	6	Reproducibility analyses, sampled-metric critiques, evaluation frameworks, and TORS methodological standards [14,15,16,17,18,22]	Provides the baseline for rigorous evidence: tuned comparisons, exact metric definitions, split validity, artifact completeness, and reproducible evaluation pipelines.
Beyond-utility and trustworthy recommendation	8	Evaluation landscape, trustworthy recommendation, popularity bias, and LLM consumer/provider fairness studies [1,2,27,28,29,30]	Supplies mature dimensions beyond top-k utility, including diversity, novelty, coverage, calibration, popularity bias, fairness, privacy, robustness, and stakeholder value.
LLM4Rec and agentic landscapes	4	Broad LLM4Rec, generative recommendation, and agentic recommender surveys [19,20,21,31]	Serves as scope control: these papers organize methods and tasks, while this survey reorganizes the same space by evidence needs and protocol failure modes.
Interaction and judge evaluation	2	Conversational recommender evaluation and LLM-as-judge surveys [32,33]	Extends evaluation from ranked lists to dialogues, subjective judgments, rubric calibration, judge bias, and agreement with human or user-centered evidence.
Multimodal and group boundary cases	9	Multimodal survey, robustness/generalization, fusion, efficiency, consensus, and group recommendation work [34,35,36,37,38,39]	Keeps the paper recommender-specific: these boundary cases show that robustness, modality contribution, efficiency, consensus, and uncertainty are not captured by utility alone.
Additional frontier examples	1	Recent generalist embedding evaluation for zero-shot recommendation and search [40]	Marks an adjacent evaluation question, whether broad pretrained representations should be judged as recommenders, search systems, or transfer models.

Table 2. Evidence clusters used to build the survey taxonomy. The n column reports the number of papers in the conservative coded corpus, not a field-wide prevalence estimate.

Evidence cluster	n	Representative sources	Role in this survey
LLM/generative protocols	19	LLM evaluation, memorization, direct generation, reranking, identifier validity, and cold-start protocols [9,10,11,12,13,25,26]	Connects history and position sensitivity, generated-item identity, contamination, and unseen-entity evaluation to the meaning of reported performance.
Classical evaluation methodology	6	Reproducibility analyses, sampled-metric critiques, evaluation frameworks, and TORS methodological standards [14,15,16,17,18,22]	Provides the baseline for rigorous evidence: tuned comparisons, exact metric definitions, split validity, artifact completeness, and reproducible evaluation pipelines.
Beyond-utility and trustworthy recommendation	8	Evaluation landscape, trustworthy recommendation, popularity bias, and LLM consumer/provider fairness studies [1,2,27,28,29,30]	Supplies mature dimensions beyond top-k utility, including diversity, novelty, coverage, calibration, popularity bias, fairness, privacy, robustness, and stakeholder value.
LLM4Rec and agentic landscapes	4	Broad LLM4Rec, generative recommendation, and agentic recommender surveys [19,20,21,31]	Serves as scope control: these papers organize methods and tasks, while this survey reorganizes the same space by evidence needs and protocol failure modes.
Interaction and judge evaluation	2	Conversational recommender evaluation and LLM-as-judge surveys [32,33]	Extends evaluation from ranked lists to dialogues, subjective judgments, rubric calibration, judge bias, and agreement with human or user-centered evidence.
Multimodal and group boundary cases	9	Multimodal survey, robustness/generalization, fusion, efficiency, consensus, and group recommendation work [34,35,36,37,38,39]	Keeps the paper recommender-specific: these boundary cases show that robustness, modality contribution, efficiency, consensus, and uncertainty are not captured by utility alone.
Additional frontier examples	1	Recent generalist embedding evaluation for zero-shot recommendation and search [40]	Marks an adjacent evaluation question, whether broad pretrained representations should be judged as recommenders, search systems, or transfer models.

3. Background: From Utility to Evaluation Validity

Utility measures the recovery of relevant items or the prediction of held-out preferences. Beyond-utility outcomes concern what else the system delivers, such as diverse recommendations or long-tail exposure. Validity concerns whether the observations and comparison support the intended interpretation. These are different questions: a score may accurately describe a restricted candidate task while providing weak evidence about full-catalog recommendation.

Evaluation frameworks start with the user task and experimental setting, rather than with a universally preferred metric [41,42,43]. Gunawardana and Shani [44] explain why metric choice can select a different algorithm for a different recommendation task. This distinction is essential when a binary preference predictor, a reranker, and an item generator all report improvements under the broad label of recommendation accuracy.

A metric specifies a calculation; a protocol supplies its inputs and population. Generative outputs add an interpretation step between prediction and scoring: titles must resolve to items, identifiers may denote collision groups, and factual explanations require reference evidence.

Figure 2 traces these dependencies. Protocol choices define the task and experimental conditions. Raw outputs are retained, mapped to catalog records where possible, and scored under explicit credit rules. Utility and beyond-utility measurements are derived from a shared set of traceable records, with each metric specifying its statistical unit, evaluated population, and denominator. Request-level utility, catalog coverage, and provider fairness can therefore use different denominators.

• Metric names do not fully specify the comparison.

Cumulated gain measures ranked retrieval with graded relevance [45]; top-N recommendation evaluation concerns which relevant items appear in the returned list [46]. The observed labels and candidate construction still determine what either calculation estimates. Analyses of statistical bias in recommendation metrics [47] and of alternative splits, sampling, and domains [48] make those choices part of the scientific comparison. Tamm et al. [49] additionally show that implementations sharing a metric name can disagree. Thus the audit records the relevance rule, candidate universe, discount, normalization, and aggregation unit alongside the score. This task-specific view is consistent with the broader challenges of offline evaluation identified by Castells and Moffat [50].

Output validity also affects aggregation. Section 7 distinguishes quality over all attempted requests from quality conditional on a valid output, and shows how the two can favor different systems.

4. System Roles in LLM-based and Generative Recommendation

The input and output of an evaluated component determine its role. A preference predictor scores a supplied user–item pair; a reranker orders a supplied set; an enhancer supplies features to another recommender; a generator emits an item name or identifier. A conversational or agentic system composes such components into an interaction. Roles describe operations, so a complete system may contain several. Adaptation by prompting or fine-tuning, sequential or multimodal context, and targets such as utility or exposure are separate facets.

TALLRec and RecRanker illustrate the distinction. Both use tuning, but the former evaluates binary preference prediction with AUC, whereas the latter evaluates candidate reranking [8,10]. Figure 3 illustrates the input–output contracts and the question each exposes.

The same language-model technology supports different contracts. LLMRank orders supplied candidates [51]; TIGER and LETTER generate semantic identifiers [52,53]. UniSRec and Recformer instead use item descriptions to learn representations for scoring [54,55]. OpenP5 supports alternative indexing methods and task configurations [56]. These distinctions determine whether evaluation measures preference discrimination, conditional ordering, or access to a relevant catalog item.

An LLM-enhanced recommender supplies profiles, embeddings, or other features to a backbone. Its contribution is assessed by holding the backbone fixed and varying the added information, encoder, or feature-generation step. A cheaper text encoder and an ablation without generated features help separate useful information from additional capacity or reformatting.

Direct generation exposes item resolution as an evaluation operation. GenRec generates names, while semantic-ID methods generate tokens that may identify multiple items [9,25]. Constrained decoding and post-hoc matching determine which items receive credit. The paper-level comparison in Section 6 examines these choices.

Interaction shifts the output from a list to a trajectory, bringing preference elicitation and task completion into evaluation [32]. Multimodal recommendation adds the contribution of different input sources [34]; group recommendation adds member-level outcomes [38,39]. These extensions require observations beyond a single ranked list.

5. Evaluation Dimensions Beyond Utility

Beyond-utility results depend on both the outcome being valued and the operation being observed. Candidate selection, free generation, and record extraction can produce different popularity patterns. Fairness measures can disagree because they compare different benefits. Table 3 summarizes the selection and extraction findings; Table 4 defines the measurement targets used throughout this section.

• Exposure and memorized knowledge measure different behavior.

Jiang et al. [12] report stronger long-tail exposure for several LLM configurations in Beauty and LastFM, and better reported serendipity despite lower overall accuracy for the best LLM configurations in Beauty (Section 4.3, Figure 2). Their ranking task supplies one positive item and 19 sampled negatives. Di Palma et al. [13], by contrast, recover popular MovieLens items more readily than rarely interacted items in record-extraction probes (Section 3.4, Figure 4). These observations concern different operations: selecting among presented candidates versus recalling a benchmark record. They can coexist. A popularity-skewed knowledge store does not by itself determine the exposure distribution of a candidate-conditioned recommender.

• Input representation changes beyond-utility outcomes.

In Jiang et al.’s Beauty and Sports comparisons, profile-only inputs retain similar accuracy while improving the reported popularity measure; profile-plus-history improves the reported dimensions (Section 4.6). This intervention supplies evidence about recommendation behavior. Explanation faithfulness requires a separate test of whether the profile reflects the recommendation mechanism.

Metric definitions explain what transfers across comparisons. Jiang et al. define long-tail items as the bottom 80% by item frequency and appendix coverage relative to the union of candidate sets. Their serendipity prose invokes usefulness, whereas the displayed Eq. 6 measures non-overlap with MostPop without an explicit relevance indicator. We therefore retain the source-defined metric name without inferring verified useful novelty from that formula. Appendix user-group disparity uses activity groups; those definitions are not demographic-fairness results.

Table 4. Measurement objects and their reference information. Table 3 and Table 5 compare empirical findings; privacy is an extension, and output validity is a condition on scoring.

Target or check	Measured object	Reference information needed
Diversity	Differences or category proportions within returned lists.	Resolved item identities, item features or taxonomy, list length, and aggregation rule.
Novelty and serendipity	Unfamiliarity or unexpectedness relative to prior knowledge or expectations.	User or popularity reference; an explicit relevance component when claiming useful unexpectedness.
Catalog coverage	Distinct catalog items reached across recommendations.	Eligible catalog, evaluated requests, output length, and treatment of unresolved items.
Calibration	Agreement between recommendation and preference distributions.	Reference preferences, attribute categories, distance measure, and user-level aggregation.
Popularity bias	Recommendation exposure relative to item frequency or user preference.	Popularity counts and time window, candidate popularity, and the chosen exposure or preference baseline.
Fairness	Distribution of benefit, sensitivity, or exposure across defined stakeholders.	Beneficiary, groups, benefit reference, allocation rule, statistical unit, and time horizon.
Robustness and generalization	Outcome stability under perturbations or transfer to held-out entities.	Fixed task, perturbed input, held-out unit, available metadata, and repeated observations.
Privacy (extension)	Protected information recoverable under an adversary’s access.	Protected-data definition, attack capability, success measure, or formal privacy guarantee.
Output validity (check)	Catalog resolution and factual support for generated assertions.	Matching rules, ambiguous identities, unresolved requests, and evidence for item or attribute existence.

Table 4. Measurement objects and their reference information. Table 3 and Table 5 compare empirical findings; privacy is an extension, and output validity is a condition on scoring.

Target or check	Measured object	Reference information needed
Diversity	Differences or category proportions within returned lists.	Resolved item identities, item features or taxonomy, list length, and aggregation rule.
Novelty and serendipity	Unfamiliarity or unexpectedness relative to prior knowledge or expectations.	User or popularity reference; an explicit relevance component when claiming useful unexpectedness.
Catalog coverage	Distinct catalog items reached across recommendations.	Eligible catalog, evaluated requests, output length, and treatment of unresolved items.
Calibration	Agreement between recommendation and preference distributions.	Reference preferences, attribute categories, distance measure, and user-level aggregation.
Popularity bias	Recommendation exposure relative to item frequency or user preference.	Popularity counts and time window, candidate popularity, and the chosen exposure or preference baseline.
Fairness	Distribution of benefit, sensitivity, or exposure across defined stakeholders.	Beneficiary, groups, benefit reference, allocation rule, statistical unit, and time horizon.
Robustness and generalization	Outcome stability under perturbations or transfer to held-out entities.	Fixed task, perturbed input, held-out unit, available metadata, and repeated observations.
Privacy (extension)	Protected information recoverable under an adversary’s access.	Protected-data definition, attack capability, success measure, or formal privacy guarantee.
Output validity (check)	Catalog resolution and factual support for generated assertions.	Matching rules, ambiguous identities, unresolved requests, and evidence for item or attribute existence.

5.1. Novelty, Diversity, and Calibration

TIGER provides a decoding-level contrast with prompted ranking [52]. On Beauty, increasing sampling temperature from 1.0 to 2.0 raises category Entropy@10 from 0.76 to 1.38 (Table 3). This measures the category distribution of returned items, whereas Jiang et al.’s long-tail exposure and candidate-union coverage measure item frequency and reach. TIGER’s table does not pair these temperature settings with accuracy values, so it establishes a diversity response to decoding rather than a quantified accuracy–diversity frontier. The shared reporting requirement is to name the output distribution being diversified and retain the decoding configuration that produced it.

Beyond-accuracy objectives differ in their statistical objects. Kaminskas and Bridge [57] distinguish diversity, novelty, serendipity, and coverage; Vargas and Castells [58] make rank and relevance explicit in novelty and diversity measurement. Topic diversification changes the composition of a recommendation list [59], while maximal marginal relevance is an earlier retrieval mechanism for balancing relevance and redundancy [60]. A generated list can therefore become less redundant without becoming more useful, more surprising to its recipient, or broader in catalog coverage. These outcomes require separate operational definitions.

Calibration adds a reference distribution. Steck [61] aligns recommendation proportions with the user’s historical interests, and Kaya and Bridge [62] compare calibrated and intent-aware recommendations. The calibration survey further distinguishes technical objectives from evidence of effectiveness [24]. For an LLM-generated profile, the audit must consequently state whether the reference distribution comes from observed interactions, explicit preferences, or the generated summary itself. Measuring agreement with the summary tests consistency with that representation; it does not independently establish preservation of the user’s preferences.

Popularity mitigation presents a related choice of reference. Controlling popularity in learning-to-rank [63] and evaluating popularity bias from the user’s perspective [64] address different aspects of the problem. Increasing aggregate long-tail exposure does not establish that each user’s preference for popular or less popular items is respected. Results should therefore distinguish an item-frequency statistic, a catalog-wide exposure distribution, and user-level preference mismatch. This separation also explains why a representation-spectrum diagnostic cannot establish exposure diversity.

5.2. Fairness Requires a Beneficiary and an Allocation Rule

Multistakeholder recommendation makes consumer and provider interests explicit [65]. Exposure fairness specifies how positions allocate visibility [66]; equity of attention accumulates that allocation across rankings [67]. Pairwise fairness instead evaluates comparisons between items [68], and FairRec formulates a two-sided allocation problem [69]. These are distinct targets, not alternative names for a single fairness score.

For generated recommendations, this distinction has a concrete consequence. A provider cannot receive catalog-level exposure credit until an output is resolved to its item and provider. Conversely, counting only successfully resolved outputs can conceal unequal service failure across user groups. A fairness report should identify the protected or stakeholder groups, the exposure model, the accumulation period, and the relation between group membership and unresolved requests. It can share the traceable request records used for utility while retaining a different statistical unit and denominator.

• The benefit reference can reverse a fairness comparison.

FaiRLLM [27] compares generated lists under neutral and sensitive-attribute prompts, measuring how their similarity varies across groups. Its constructed music and movie preferences permit controlled prompt interventions, but supply no held-out relevance labels. CFaiRLLM [28] adds a different reference: test interactions used to ground preference alignment. Table 5 places both beside attribute-recoverability and item-exposure evaluations. The choice is substantive: equal sensitivity to demographic wording, equal recommendation benefit, hidden sensitive information, and equal item exposure are different objectives.

Table 5. Four fairness claims and their operational definitions. Source versions and field-level evidence are recorded in the supplement.

Study	Measured object	Protocol	Interpretation
FaiRLLM [27]	Group variation in similarity to neutral-prompt lists.	Constructed preferences; 20 titles; Jaccard, SERP*, PRAG*, SNSR/SNSV (Sections 2–4).	Demographic-prompt sensitivity, without held-out relevance labels.
CFaiRLLM [28]	Group disparities under list or test-preference alignment.	Temporal MovieLens/LastFM profiles; history selection and title matching (Sections 3–4).	Preferred history strategy changes with the reference (Table 4, arXiv v3).
UP5 [29]	Sensitive attributes recoverable by a probe.	MovieLens/Insurance; attribute AUC and sampled Hit@k (Section 5).	Probe-specific recoverability alongside utility, distinct from benefit parity.
Provider-fairness study [30]	Item-count concentration and catalog coverage.	Prompt and ICL variants; title/artist matching; Gini, HHI, entropy, coverage (Sections 3–5).	ICL can concentrate exposure; item equality serves as a provider-side proxy.

Table 5. Four fairness claims and their operational definitions. Source versions and field-level evidence are recorded in the supplement.

Study	Measured object	Protocol	Interpretation
FaiRLLM [27]	Group variation in similarity to neutral-prompt lists.	Constructed preferences; 20 titles; Jaccard, SERP*, PRAG*, SNSR/SNSV (Sections 2–4).	Demographic-prompt sensitivity, without held-out relevance labels.
CFaiRLLM [28]	Group disparities under list or test-preference alignment.	Temporal MovieLens/LastFM profiles; history selection and title matching (Sections 3–4).	Preferred history strategy changes with the reference (Table 4, arXiv v3).
UP5 [29]	Sensitive attributes recoverable by a probe.	MovieLens/Insurance; attribute AUC and sampled Hit@k (Section 5).	Probe-specific recoverability alongside utility, distinct from benefit parity.
Provider-fairness study [30]	Item-count concentration and catalog coverage.	Prompt and ICL variants; title/artist matching; Gini, HHI, entropy, coverage (Sections 3–5).	ICL can concentrate exposure; item equality serves as a provider-side proxy.

The difference changes a reported ordering. In CFaiRLLM’s Table 4 (arXiv v3), the Jaccard-based sex-group SNSR is 0.0010 for random history sampling and 0.0424 for recent sampling under item-list alignment; lower is better. Under preference-grounded alignment, the corresponding values are 0.0210 and 0.0086, favoring recent sampling. Comparing strategies within each reference shows that the preferred history depends on the benefit being measured. The absolute fairness levels remain specific to their respective metric definitions.

• A probe tests recoverability, not the distribution of benefit.

UP5 [29] evaluates sensitive-attribute prediction alongside recommendation accuracy. In its MovieLens gender comparison for matching-based models (Table 3), CFP reports attribute AUC of 54.19%, versus 56.62% for C-PMF and 70.80% for C-SX, while Hit@10 is 65.82%, 65.32%, and 56.02%, respectively. CFP combines lower attribute recoverability with higher recommendation Hit@10 in this comparison. This is evidence about information accessible to the tested probe; CFaiRLLM instead measures differences in recommendation benefit. The two evaluations can therefore complement one another without agreeing on a single fairness ranking.

• More context need not broaden exposure.

Deldjoo’s provider-fairness study [30] measures recommendation-count concentration and catalog coverage after matching generated names to items. In its sequential MovieLens experiment, the no-demographics, frequency-profile setting has coverage 0.02621 with zero-shot prompting and 0.01685 with ICL-2; Gini rises from 0.57739 to 0.65835 (arXiv v3, Table 10). The added demonstrations coincide with narrower, more concentrated exposure in this comparison. Read alongside CFaiRLLM, this result shows why history and demonstration policies belong in a beyond-utility protocol: they change both whose preferences are represented and which items receive exposure. These item-frequency statistics use equality among items as a provider-side proxy; provider ownership and merit require additional definitions.

5.3. Privacy and Explanation Need Different Evidence

Privacy evaluation specifies protected information and adversary access. Re-identification from ratings [70], disclosure through collaborative-filtering outputs [71], membership inference [72], and language-model extraction [73] concern different information and access routes. Differential privacy supplies a formal guarantee for a randomized mechanism [74]. These distinctions identify what a recommendation-specific privacy claim must protect; public-item recall alone is not a disclosure test.

Explanation research distinguishes factual support, user understanding, and the reason for a recommendation [75,76]. Scrutable user models additionally expose preferences for correction [77]. For generated explanations, accurate item descriptions and faithful accounts of the ranking mechanism are separate evaluation targets. Section 8 considers how to connect them to user-observed outcomes.

5.4. Output Validity and Robustness

• Matching failure and hallucination require different checks.

A matching failure occurs when an output cannot be resolved to an eligible catalog item. Causes include malformed output, unresolved aliases, collisions, and real items outside the catalog. We reserve item hallucination for an asserted fabricated item and attribute hallucination for an unsupported factual assertion. An existing but irrelevant item is a relevance error. Jiang et al.’s prose defines hallucination through titles unmatched after case, space, and symbol normalization, including omissions and combined titles. Their displayed Eqs. 10 and 24 instead use an in-catalog membership indicator, reversing this interpretation. We follow the prose definition here; whether the implementation follows that definition or the printed equations remains unresolved. Neither a non-match nor the source’s metric label establishes fabrication.

This distinction also matters to exposure. A failed exact match may denote a real long-tail item; nearest-item repair may instead assign it to a popular item. Novelty and coverage should consequently be computed over resolved item identities with unresolved outputs retained in the audit, rather than treating unfamiliar wording as novel content.

Robustness tests specify both the changed input and the outcome expected to remain stable. Candidate permutations preserve eligibility; missing-modality tests change available information. Multimodal work motivates checking modality contribution and sparse-data behavior [35,36,78], with efficiency a separate practical objective [37]. These extensions follow the same principle of defining the evaluated object before interpreting a score change.

6. How Protocol Choices Change Comparative Findings

The fairness comparisons in Section 5 show that the outcome reference can change which history strategy appears preferable. We now examine how candidates, item credit, and held-out populations determine the recommendation problem being scored. Table 6, Table 7 and Table 8 organize these contrasts by evaluation question.

6.1. Candidate Access Changes the Ranking Problem

Dai et al. use one positive and four negatives; Jiang et al.’s ranking task uses one positive and 19 negatives. RecRanker instead describes retrieval followed by reranking [10,12,79]. The first two designs test ordering when the target is available. The third also depends on whether retrieval finds it. Candidate-order robustness addresses presentation, whereas changing candidate membership changes the task itself.

With one held-out relevant item per request, end-to-end Hit Rate at k factors into the probability of retrieving that item and the probability of ranking it in the top k conditional on retrieval. Inserting the target into every candidate set estimates the conditional term under a constructed set. A pipeline comparison therefore retains retrieval misses and reports conditional reranking separately.

LLMRank separately examines supplied-positive sets, hard negatives, and retrieved candidates [51]; UniSRec and Recformer specify all-item test ranking even though their training objectives use negative examples [54,55]. Thus negative sampling must be located in training or evaluation before a comparison is classified. A model’s language backbone does not determine its candidate protocol.

Complete relevance judgments do not remove candidate-selection effects. Sanner et al. [80] collect judgments from 153 raters over pools combining EASE, text retrieval, and popularity-stratified random selection. In their Table 3, EASE exceeds the language-only three-shot LLM on the full pool (NDCG@10: 0.673 versus 0.640), but falls below it on the random subpool (0.592 versus 0.650). This is a reversal of reported point estimates, not an established paired significant difference. EASE also contributes items to the full pool. The contrast changes the candidate task and shows why label completeness and selection neutrality require separate checks; the random subpool is drawn from specified popularity strata, not uniformly from the entire catalog.

Table 7. Generated outputs and item-level credit. Name resolution, identifier membership, and one-to-one item identity address different sources of scoring ambiguity.

Study	Output and evaluation	Resolution or decoding rule	Consequence for interpretation
GenRec [9], Sections 3–4	Generated names; MovieLens 25M and Amazon Toys; HR/NDCG@5,10.	Last/penultimate interactions held out; matching and failure aggregation not fully specified.	Reproducing a name-level score requires its catalog matching and failure-counting rules.
TIGER [52], Sections 4.3, 4.5	Four-token semantic IDs; generated top-10 lists.	Disambiguating token for known items; approximately 0.1–1.6% invalid IDs across three datasets.	Unique registered IDs do not guarantee that every generated sequence is registered.
LETTER [53], Sections 3.2, 4	Learned item tokenization; tokenizer, regularizer, and backbone comparisons.	A trie restricts generation to valid successor tokens.	Membership in the identifier language does not alone establish one-to-one item credit.
Zhang et al. [25], Sections 3.1, 4	Scientific, Cell, Beauty, Yelp; shared generator across tokenizers; three seeds.	SID Hit/NDCG versus collision-corrected item metrics.	Correcting identifier-level credit changes tokenizer ordering without changing native outputs.

Table 7. Generated outputs and item-level credit. Name resolution, identifier membership, and one-to-one item identity address different sources of scoring ambiguity.

Study	Output and evaluation	Resolution or decoding rule	Consequence for interpretation
GenRec [9], Sections 3–4	Generated names; MovieLens 25M and Amazon Toys; HR/NDCG@5,10.	Last/penultimate interactions held out; matching and failure aggregation not fully specified.	Reproducing a name-level score requires its catalog matching and failure-counting rules.
TIGER [52], Sections 4.3, 4.5	Four-token semantic IDs; generated top-10 lists.	Disambiguating token for known items; approximately 0.1–1.6% invalid IDs across three datasets.	Unique registered IDs do not guarantee that every generated sequence is registered.
LETTER [53], Sections 3.2, 4	Learned item tokenization; tokenizer, regularizer, and backbone comparisons.	A trie restricts generation to valid successor tokens.	Membership in the identifier language does not alone establish one-to-one item credit.
Zhang et al. [25], Sections 3.1, 4	Scientific, Cell, Beauty, Yelp; shared generator across tokenizers; three seeds.	SID Hit/NDCG versus collision-corrected item metrics.	Correcting identifier-level credit changes tokenizer ordering without changing native outputs.

6.2. Valid Outputs Can Receive Ambiguous Item Credit

Dai et al.’s compliance rate checks membership in the supplied set; it does not specify how invalid answers enter NDCG or MRR. GenRec reports generated-name HR and NDCG without a complete matching and failure-aggregation rule in the inspected paper [9,79]. Semantic identifiers pose a different problem: an output can match a valid identifier but refer to several items.

Zhang et al. [25] show the consequence using the same native tokenizer outputs. On Scientific, RK-Means scores 0.1330 against LETTER’s 0.0804 under SID Hit@10, but 0.0654 against 0.0767 under collision-corrected ItemHit@10 (Table 4). The ordering reverses when credit reflects ambiguous item identity. The correction expands collision groups and averages over unresolved within-group order; an actual disambiguator needs its own selection rule. Together, these studies distinguish response compliance, item resolution, and score aggregation as separate operations.

TIGER and LETTER expose the two sides of identifier validity [52,53]. TIGER appends a disambiguating token to known-item semantic IDs, yet reports approximately 0.1–1.6% invalid IDs among top-10 predictions across its three datasets (Section 4.5). LETTER constrains generation with a trie of valid successor tokens (Section 3.2.2). The first controls collisions among registered items; the second controls membership in the registered identifier language. Neither property substitutes for the other. A protocol should document both the decoding constraint and the map from each credited identifier to an item.

Table 8. Generalization and intermediate evidence. The held-out object and the observed behavior determine whether a result concerns transfer, record recoverability, or delivered recommendation quality.

Study	Evaluation object	Generalization or control	Consequence for comparison
UniSRec [54], Section 3.1	Full-catalog item scoring from sequence representations.	Text-only inductive versus text-plus-ID transductive adaptation.	Training negatives are not the test pool; representation transfer differs from a temporal cold-item test.
Recformer [55], Section 3.4.1	Full-catalog text-based scoring; in-set and unseen-item tests.	Cold-token handling for SASRec; text encodings for unseen items.	Baseline access to unseen items is part of the generalization comparison.
OpenP5 [56], Tables 2–4	Indexed-item generation; random, sequential, or collaborative indices.	Seen/unseen prompt templates; T5 and LLaMA backbones.	Held-out prompts do not establish held-out-user or held-out-item performance.
Zhang et al. [26], Sections 3–4	Toys, MicroLens, Steam; temporal item split and separately held-out users.	Full-catalog item-cold targets; Recall/NDCG@10; scale, ID, and training comparisons.	Short history, unseen user, and unseen item are distinct generalization targets.
Di Palma et al. [13], Sections 2–3	MovieLens-1M; recommendation and record-extraction probes.	80/20 leave-n-out; prompt requests 50 titles; HR/nDCG and record coverage.	Recovering benchmark records informs exposure; it does not isolate its causal effect on accuracy.
CLLMR [81], Sections 4–5	Amazon Books, Yelp, Steam; generated user/item side information; 3:1:1 split.	Recall/NDCG at 10, 30, 50; five initializations; singular-value diagnostics and ablations.	Representation collapse is distinct from catalog exposure and stakeholder fairness.

Table 8. Generalization and intermediate evidence. The held-out object and the observed behavior determine whether a result concerns transfer, record recoverability, or delivered recommendation quality.

Study	Evaluation object	Generalization or control	Consequence for comparison
UniSRec [54], Section 3.1	Full-catalog item scoring from sequence representations.	Text-only inductive versus text-plus-ID transductive adaptation.	Training negatives are not the test pool; representation transfer differs from a temporal cold-item test.
Recformer [55], Section 3.4.1	Full-catalog text-based scoring; in-set and unseen-item tests.	Cold-token handling for SASRec; text encodings for unseen items.	Baseline access to unseen items is part of the generalization comparison.
OpenP5 [56], Tables 2–4	Indexed-item generation; random, sequential, or collaborative indices.	Seen/unseen prompt templates; T5 and LLaMA backbones.	Held-out prompts do not establish held-out-user or held-out-item performance.
Zhang et al. [26], Sections 3–4	Toys, MicroLens, Steam; temporal item split and separately held-out users.	Full-catalog item-cold targets; Recall/NDCG@10; scale, ID, and training comparisons.	Short history, unseen user, and unseen item are distinct generalization targets.
Di Palma et al. [13], Sections 2–3	MovieLens-1M; recommendation and record-extraction probes.	80/20 leave-n-out; prompt requests 50 titles; HR/nDCG and record coverage.	Recovering benchmark records informs exposure; it does not isolate its causal effect on accuracy.
CLLMR [81], Sections 4–5	Amazon Books, Yelp, Steam; generated user/item side information; 3:1:1 split.	Recall/NDCG at 10, 30, 50; five initializations; singular-value diagnostics and ablations.	Representation collapse is distinct from catalog exposure and stakeholder fairness.

6.3. Generalization Depends on What Is Held Out

TALLRec predicts binary preference for a supplied target. Its Movie data use preceding interactions, but Book histories are sampled because BookCrossing lacks timestamps. RecRanker also states leave-one-out while reconstructing BookCrossing histories randomly [8,10]. Thus a shared split label does not establish a shared temporal prediction problem.

The cold-start study by Zhang et al. [26] makes entity exposure explicit: item-cold targets first appear after a temporal cutoff, while user-cold evaluation holds out users and supplies short histories. These populations differ from shortening an existing user’s context. TALLRec’s few-shot tuning additionally varies training examples, not the same quantity as inference-history length. Comparisons of generalization need the held-out entity, chronology, and available context, rather than the cold-start label alone.

Holding out an item also raises the question of how it becomes eligible at test time. TIGER’s separate cold-item experiment removes 5% of test items from training, then adds unseen items sharing the generated three-token prefix under a mixture cap. Recformer encodes unseen items from text and gives its SASRec baseline a trained cold-token representation [52,55]. These experiments test different routes into the eligible item set. OpenP5’s unseen results instead hold out prompt templates [56]; interpreting them as unseen-item results would change the generalization claim without changing any score.

6.4. Knowledge and Representations Are Intermediate Objects

Record-extraction probes address a different object from the ranking tasks above. Cross-model associations between extraction success and recommendation accuracy leave model capability and training exposure confounded [13]. The candidate-conditioned exposure findings in Section 5 consequently cannot be explained by extraction success alone. Linking the two would require observing how recoverable knowledge is used when selecting recommendations.

CLLMR supplies a complementary example from the enhancer role [81]. Its singular-value analyses and module ablations examine representation collapse and the contribution of generated side information. These feature-level diagnostics complement the list-level exposure studies and knowledge-extraction probes: the three evaluate representations, accessible model knowledge, and delivered recommendations, respectively. Connecting them requires measuring how changes in features or recoverable knowledge affect the returned lists.

6.5. Selection Begins Before Model Evaluation

The observations used for evaluation are already selected: MovieLens is a collected interaction resource [82], and missing-not-at-random analysis [83] and treatment-based recommendation [84] address the resulting bias. Offline bandit evaluation [85], counterfactual risk minimization [86], and unbiased learning-to-rank [87] provide corrections under assumptions about observation and exposure. Generative recommendation adds selection at two later points: which items become eligible and which outputs survive parsing. Correcting the logged-feedback process leaves these later decisions to be specified separately.

6.6. Two Findings Across the Protocols

• Protocol choices change reported orderings.

The candidate-pool comparison changes which alternatives are evaluated; identifier correction changes which item receives credit; the fairness comparison changes the benefit reference. Each produces a source-reported ordering change through a different operation. The practical question is therefore which comparison matches the intended use: selecting among available candidates, retrieving an identifiable item, or distributing a specified benefit.

• Different measured objects explain apparent disagreement.

Long-tail selection among presented candidates and popular-record recovery concern recommendation behavior and accessible knowledge, respectively. Attribute recoverability and group benefit concern information in a representation and the distribution of an outcome. The distinctions reconcile findings without averaging unlike measurements. They also locate the missing link: a knowledge or representation diagnostic needs additional evidence connecting it to delivered recommendations. The controls below follow from these two findings.

7. Implications for Evaluation Practice

The findings in Section 6.6 lead to two practical tasks: hold the comparison conditions fixed when attributing an improvement to a model, and retain distinct measurements when they concern different objects. Table 9 collects the reporting fields. We focus here on the design and accounting choices needed to use them.

7.1. Fix the Comparison Before Scoring

Framework comparisons [88], rigorous benchmarking guidance [89], and re-evaluations of collaborative-filtering [90] and session-based models [91] motivate strong baselines configured for the task being tested. The same requirement applies when the new component is an LLM.

A reranker comparison needs the same held-out requests, candidate membership, item metadata, and candidate-order permutations for each model. An end-to-end comparison instead allows each system to retrieve its own candidates, but retains the same request population and reports retrieval failures. These two experiments answer different questions. Prompts, matching thresholds, and hyperparameters should be selected on validation data before the test set is scored. Tuning reports should give search spaces, selection criteria, trial counts, and computational cost; equal trial counts alone do not establish equally effective tuning across model families.

For fairness, fix the benefit reference as well as the protected groups: a neutral-prompt list, held-out preferences, probe recoverability, or an exposure allocation. A comparison can then vary history selection while preserving users, history budget, output resolution, and the metric definition. When several references are scientifically relevant, report the strategy ordering under each, as in the CFaiRLLM contrast, rather than collapsing the results into one fairness score. Generalization reports should similarly identify whether the held-out unit is a prompt, user, item, or domain.

7.2. Keep Validity in the Denominator

Consider a single-item recommendation task. Let N be the number of attempted requests, V the number yielding a valid catalog item, and H the number yielding a valid item judged relevant. Validity rate is $V/N$, and conditional hit rate is $H/V$ for $V>0$. If an invalid output counts as a failed request, end-to-end hit rate is

$${\displaystyle \frac{H}{N}}={\displaystyle \frac{V}{N}}{\displaystyle \frac{H}{V}}.$$

(1)

When $V=0$, conditional hit rate is undefined and end-to-end hit rate is zero. This accounting identity makes the denominator choice explicit. Conditional quality measures successful outputs; end-to-end quality measures all attempted requests.

Figure 4 gives an illustrative comparison, not a result from an evaluated model. System A returns 50 valid items and 40 hits across 100 requests; system B returns 100 valid items and 60 hits. A leads on conditional hit rate (80% versus 60%), but B leads on end-to-end hit rate (60% versus 40%). Dropping invalid requests reverses the ordering. The figure motivates reporting both rates with their counts.

Figure 4. Illustrative model-order reversal from the denominator choice. Restricting evaluation to valid requests favors A; counting all attempts, with invalid outputs receiving no hit credit, favors B. Both panels share a zero-based percentage scale. Counts are constructed, not empirical benchmark results.

Figure 4. Illustrative model-order reversal from the denominator choice. Restricting evaluation to valid requests favors A; counting all attempts, with invalid outputs receiving no hit credit, favors B. Both panels share a zero-based percentage scale. Counts are constructed, not empirical benchmark results.

For top-k lists, the evaluator must additionally define missing slots, duplicate items, and ranks after parsing. Removing an invalid first item and shifting later items upward changes discounted gain. A protocol can preserve the original slots and assign zero gain to invalid entries, then report post-repair utility separately when repair is part of the deployed system. Equation 1 should not be transferred directly to listwise NDCG, whose normalization and partial-list validity require their own definitions.

Validity also conditions beyond-utility metrics. Exposure fairness computed only over matched items omits users for whom generation failed. A useful report therefore pairs exposure or novelty with validity by the relevant user or item stratum. The catalog snapshot and popularity reference must stay fixed across systems so that changes in the metric reflect their outputs.

7.3. Test Stability on Paired Requests

Candidate-order tests should reuse the same permutations across models, and history-length tests should reuse the same users. This pairing allows the analysis to compare performance differences under each perturbation. Selecting different users for each history length confounds input truncation with user composition. Report the mean paired difference and its uncertainty, together with the settings under which the sign changes. Repeated generations for one user measure generation variability, not additional independent users. Uncertainty calculations should preserve that grouping, for example by resampling users while retaining their repeated outputs.

Repeated requests within a fixed model version measure generation variability; a later version introduces possible service drift. LLM judges add their own reliability and bias concerns [33]. Judge-swap comparisons should preserve item evidence and use recommendation-specific human or catalog-grounded calibration cases. These checks separate changes in the recommender from changes in the instrument measuring it.

7.4. Make Each Score Traceable

LensKit [92], RiVal [93], and Elliot [94] support reusable recommendation experiments. Accountability guidance [95] and the reproducibility survey [23] emphasize inspectable artifacts. For generative outputs, the crucial addition is a link from each stored prediction to its parsing, catalog resolution, and failure treatment, enabling the same outputs to be rescored.

An inspectable evaluation record links the request and candidate identifiers to the raw output, parsed catalog matches, validity decisions, per-request metrics, prompt, model version, and decoding configuration. These records expose whether a score changed because of generation, matching, or aggregation. When data or API restrictions prevent complete release, authors can still publish the evaluation code, configuration, permissible examples, and aggregate failure counts, stating which portions can be rerun.

8. Open Challenges

The protocol contrasts leave several questions unresolved. They concern the interaction between measurement choices and model behavior, rather than the availability of another metric.

8.1. How Does Item Resolution Redistribute Exposure?

The identifier studies establish that credit rules can change utility rankings. The corresponding effect on beyond-utility outcomes is less well characterized. Exact title matching may disproportionately reject aliases or rare items, while nearest-item repair may concentrate their credit on popular alternatives. A useful comparison would rescore fixed raw outputs under several documented resolution policies, measuring utility, long-tail exposure, and user-group failure rates together. It would distinguish a model’s recommendation behavior from the distribution induced by its evaluator. Catalog editions, translations, and many-to-one identifiers make this question relevant even when every generated string denotes an existing item.

8.2. When Does Benchmark Knowledge Help Generalization?

Extraction probes reveal recoverable benchmark content, but the observed association with accuracy does not isolate its contribution. Comparing familiar and newly released catalogs also changes domain, item difficulty, and metadata quality. A more discriminating design would match these properties while varying access to interaction records and catalog descriptions. Cold-item experiments introduce a related distinction: learning to use an unseen item’s attributes differs from making that item reachable through an index. Separating knowledge access, representation transfer, and retrieval eligibility would clarify when performance reflects generalization rather than benchmark-specific exposure.

8.3. Which Offline Outcomes Predict User Benefit?

User-centered frameworks distinguish system properties from perceived quality and experience [96,97,98]. The fairness studies make a related distinction between list similarity, preference alignment, and attribute recoverability. Whether improvements in these offline measurements translate into perceived relevance, control, or equitable benefit remains a separate empirical question.

Conversational recommendation provides a setting in which to examine that connection. Surveys organize preference elicitation and feedback [32,99]; ReDial supplies recommendation dialogues [100], and CRSLab provides conversational-system tooling [101]. Holding recommended items fixed while varying explanation or dialogue behavior would isolate changes in factual understanding, perceived control, and task completion. Varying items under a fixed dialogue policy would test a different pathway to user benefit.

LLM judges introduce a measurement question within such studies. MT-Bench and Chatbot Arena examine model judgments and human preferences [102]; long-context evaluations document sensitivity to information position [103]; HELM combines multiple metrics with recorded prompts and completions [104]. Recommendation evaluation needs to determine when a judge’s preference tracks catalog-grounded or user-observed quality and when it tracks explanation style. Separate interventions on item relevance, factual support, and wording would help identify that boundary.

8.4. How Do Generated Profiles Alter Feedback Loops?

Algorithmic confounding [105], feedback-loop bias amplification [106], and degenerating recommendation dynamics [107] show why repeated exposure differs from a static held-out test. Generated profiles add a possible feedback path: earlier recommendations can enter a later summary and be interpreted as evidence of user preference. The open question is whether this representation step amplifies or corrects the exposure effects already present in recommendation logs. A sequential comparison would hold the interaction stream and update opportunities fixed while varying how summaries distinguish observed preferences from prior system outputs. Intent-aware recommendation [108] and RS4Good [109] motivate assessing adaptation against changing user purposes and practical outcomes.

8.5. How Stable Are Conclusions Across Evaluators?

The reviewed reversals arise from particular candidate, credit, or reference choices. Their frequency and joint effect across system families remain unknown. Shared raw-output records would permit a crossed comparison: multiple recommenders evaluated under the same alternative candidate protocols, matching policies, and failure-accounting rules. Such a study could identify conclusions that survive evaluator changes and those driven by a specific policy. Fixed request panels and versioned outputs would further separate evaluator effects from stochastic generation and service drift. This moves beyond checking whether a result can be reproduced under one implementation to testing whether its interpretation is stable under scientifically justified alternatives.

9. Conclusion

The reviewed evidence leads to two conclusions. Protocol choices can change reported orderings: candidate pools, identifier credit, and fairness references each provide a concrete instance. Apparent disagreement can also reflect different measured objects: long-tail selection can coexist with popularity-skewed record recovery, while attribute recoverability and user benefit describe separate outcomes. Together, these findings make comparability central to evaluating LLM-based and generative recommenders. Progress is established by an improvement on a defined recommendation problem, with traceable item credit and an explicit outcome reference, rather than by a larger collection of favorable scores.