Large Language Model Alignment with Complex Feedback: A Survey

1. Introduction

Human feedback is central to aligning LLMs with human preferences, values, and task-specific goals, yet it is not a homogeneous signal: it takes multiple forms that differ in expressiveness, annotation cost, and how directly they specify desired improvements. Most large-scale alignment pipelines nonetheless rely mainly on binary preference labels, which are easy to collect but information-sparse (Wu et al., 2023), ambiguous across factual, stylistic, and safety dimensions (Casper et al., 2023; Chaudhari et al., 2025), and outcome-focused, offering little guidance for multi-step reasoning or structured decision-making tasks.

Recent work has demonstrated the benefits of richer feedback: linguistic feedback can be more efficient than scalar rewards (Scheurer et al., 2023), no single modality is optimal across tasks (Yu et al., 2025), and supervision on intermediate reasoning steps improves generalization (Uesato et al., 2022; Lightman et al., 2024). To compare these roles empirically, we conduct a quantitative meta-analysis on HelpSteer3 Wang et al. (2025), quantifying the mutual information contributed by binary preferences, scores, and language toward recovering the aggregated preference (Figure 1; details and a domain-specific breakdown in Appendix C).

Our results show that binary preferences already capture most information needed to recover the preference, while scores and language add only modest value; however, linguistic feedback remains informative for calibration and diagnostics, explaining score magnitude and annotator disagreement and aligning with the semantic direction from rejected to preferred responses. This suggests a trade-off: binary supervision is often efficient for broad alignment, whereas language feedback is more useful for targeted improvement.

We stress that these results do not imply language feedback is uniformly more or less informative than binary preferences; rather, informativeness is objective-dependent: binary comparisons suffice to recover a general preference direction, but more specific objectives, such as localizing where a reasoning chain goes wrong or expressing a stylistic or constraint-based requirement, need richer supervision that scalar or binary labels cannot supply.

To fill this gap, we adopt a feedback-centric perspective, proposing a taxonomy that characterizes feedback by its form and several orthogonal dimensions and shows how these map to corresponding training and evaluation frameworks (Figure 2). Unlike prior reviews organized mainly around algorithms or pipelines, our survey centers complex feedback itself, organizing the literature by its structure and semantics (Table 1).

Beyond offering broader coverage of the alignment pipeline (Table 1), organizing the survey around feedback structure enables a form of analysis that prior, component-centric surveys do not directly support: because feedback connects data collection, training, and evaluation, treating it as the central object exposes mismatches across these stages, as we discuss in Section 5.

Table 1. Comparison with representative surveys. A checkmark (✓) indicates substantial coverage of the corresponding aspect of the alignment pipeline, while a circle (∘) indicates partial discussion.

Survey	Main Perspective	Data	Preference Modeling	Optimization	Evaluation
Fernandes et al. (2023)	Human feedback in NLG	✓			✓
Jiang et al. (2024)	Preference learning		✓	✓
Liu et al. (2025)	Optimization algorithms		✓	✓
Ji et al. (2025)	Reward design and reward modeling	∘	✓	✓
Lai et al. (2025)	Post-training scaling	✓	✓	✓	∘
Ours	Feedback structure and semantics	✓	✓	✓	✓

Table 1. Comparison with representative surveys. A checkmark (✓) indicates substantial coverage of the corresponding aspect of the alignment pipeline, while a circle (∘) indicates partial discussion.

Survey	Main Perspective	Data	Preference Modeling	Optimization	Evaluation
Fernandes et al. (2023)	Human feedback in NLG	✓			✓
Jiang et al. (2024)	Preference learning		✓	✓
Liu et al. (2025)	Optimization algorithms		✓	✓
Ji et al. (2025)	Reward design and reward modeling	∘	✓	✓
Lai et al. (2025)	Post-training scaling	✓	✓	✓	∘
Ours	Feedback structure and semantics	✓	✓	✓	✓

This survey makes the following contributions:

• We introduce a feedback-centric view on LLM alignment and propose a systematic taxonomy of complex feedback by its form and semantics that fundamentally shape learning and evaluation.

• We propose a taxonomy that categorizes existing alignment methods by the feedback types they assume, and introduce a corresponding taxonomy of alignment evaluation based on learned or inferred human preferences.

• We identify mismatches and underexplored regimes among feedback types, optimization objectives, and evaluation designs; and further outline directions for designing more faithful and data-efficient feedback-driven alignment.

We describe our literature search process, inclusion criteria, and scope in Appendix A.

2. A Taxonomy of Complex Feedback

Feedback is central to LLM alignment but varies widely in form, granularity, and cost. We organize feedback by its supervision form—how evaluative information is expressed. Orthogonal dimensions such as interaction structure, feedback source, and granularity further characterize feedback and help situate existing datasets and alignment methods.

Figure 3. Taxonomy of feedback types based on signal form.

Figure 3. Taxonomy of feedback types based on signal form.

2.1. Ordinal Feedback

Ordinal feedback captures relative order between outputs without specifying preference strength.

2.1.1. Binary Preference

Binary preference feedback is the most common form in RLHF. Given a prompt $x\in \mathcal{X}$ and two candidate outputs $y_A,y_B\in \mathcal{Y}$, annotators select the preferred response, which we encode as

$$f(x,y_A,y_B)=\left\{\begin{array}{cc}1,\hfill & \mathrm{if}{y}_A\succ y_B,\hfill \\ 0,\hfill & \mathrm{if}{y}_B\succ y_A,\hfill \end{array}\right.$$

where ≻ denotes preference, with ties typically discarded or mapped to a neutral label. Binary preferences are easy to collect, and integrate naturally with pairwise reward modeling and policy optimization. Yet, they encode only relative order, not preference strength, and can be ambiguous when trade-offs across multiple attributes are involved.

Representative Datasets. Binary preference signals are widely used in RLHF, e.g., summarization (Stiennon et al., 2020), UltraFeedback (Cui et al., 2023), and hh-rlhf (Bai et al., 2022).

2.1.2. List Preference

List preference feedback generalizes binary comparisons by allowing annotators to rank a set of $m\ge 2$ candidate outputs for the same prompt. Though such a total ordering can be decomposed into $m(m-1)/2$ pairwise comparisons, listwise supervision exploits joint judgments in a shared context and encourages globally consistent rankings, which may improve sample efficiency and ranking coherence. Listwise feedback is more informative than binary comparisons but becomes cognitively demanding as m increases. In practice, it is often relaxed to partial orderings, such as top-k or bottom-k selection (marking only the best or worst few outputs) or grouping candidates into quality tiers without ordering within tiers.

Representative Datasets. Few public datasets retain listwise annotations; examples include OpenAssistant Conversations (öpf et al., 2023) and RewardBench v2 (Malik et al., 2025).

2.2. Cardinal Feedback

Cardinal feedback provides absolute assessments of model outputs using discrete labels (e.g., poor–good) or numerical scores (e.g., in $[0,1]$). Unlike ranking-based feedback, it captures preference magnitude and enables smoother rewards and cross-instance comparison. However, cardinal ratings are more subjective and noisy, as annotators may use scales inconsistently and exhibit systematic biases (e.g., central tendency, order effects), requiring careful scale design and calibration at scale.

2.2.1. Scalar Rating

Scalar rating feedback assigns a single numerical or categorical score to each output, typically reflecting overall quality or a specific attribute. For a prompt $x\in \mathcal{X}$ and output $y\in \mathcal{Y}$, the annotator provides $f(x,y)=s,$ where $s\in \mathbb{R}\mathrm{or}s\in \{n_1,\dots ,n_k\},$ with $n_1$ and $n_k$ denoting the lowest and highest possible ratings. Compared to binary or pairwise feedback, scalar ratings capture preference strength and support smoother, potentially more sample-efficient reward modeling.

Representative Datasets. UltraFeedback (Cui et al., 2023) and WebGPT (Nakano et al., 2021) provide scalar human ratings; MultiPref (Miranda et al., 2024) additionally encodes annotator confidence for calibration.

2.2.2. Multi-Aspect Rating

Multi-aspect rating feedback extends scalar ratings to multiple predefined dimensions, providing a vector of scores for each output: $f(x,y)=[s_1,\dots ,s_K],$ where each $s_k$ corresponds to an attribute. A common implementation is rubric-based evaluation, where detailed scoring guidelines specify criteria for each dimension to reduce inter-annotator variance and improve reliability. Attribute-specific supervision decomposes evaluation into interpretable components, enabling multi-objective alignment and fine-grained diagnostics.

Representative Datasets. Examples include HelpSteer2 (Wang et al., 2024), UltraFeedback (Cui et al., 2023), and OpenAssistant Conversations (öpf et al., 2023); other evaluation frameworks Zheng et al. (2023) similarly rely on multi-aspect scales.

2.3. Linguistic Feedback

Linguistic feedback conveys evaluative or corrective information through natural language. We distinguish signals that directly prescribe revised behavior from those that describe and guide it.

2.3.1. Prescriptive Feedback

Prescriptive feedback provides direct supervision on how to improve a model’s response. Given a prompt $x\in \mathcal{X}$ and output $y\in \mathcal{Y}$, the feedback yields an improved target $y^{*}$, typically in one of two forms: (1) Edits: Localized modifications (insertions, deletions, replacements) that specify how to revise the original response; (2) Demonstrations: Rewrites or example outputs that instantiate the desired behavior and can be used as targets for imitation or distillation. Although prescriptive feedback can in principle be converted into binary preference, they go beyond ranking by encoding interpretable transformations that directly guide alignment.

Representative Datasets. Edit-based examples include AVS Edits (Mishra et al., 2024; Yao et al., 2023), WikiPrefs (Majkutewicz and Szymański 2025), and HelpSteer3 (Wang et al., 2025); demonstration-style supervision is exemplified by Anthropic-HH-Golden (Cai et al., 2023).

2.3.2. Descriptive Feedback

Descriptive feedback communicates evaluation and guidance through natural language rather than directly modifying a model output. Given a prompt $x\in \mathcal{X}$ and output $y\in \mathcal{Y}$, we write $f(x,y)=l$, where $l\in \mathcal{L}$ is a linguistic comment on qualitative properties or desired changes (e.g., “the argument lacks evidence,” “make the tone more formal”).

Unlike edits or demonstrations, descriptive feedback does not replace the original output but provides meta-level comments. This supports supervision over high-level attributes such as coherence, politeness, and persuasiveness that are not easily captured through token-level corrections, while requiring the model to interpret and operationalize abstract guidance, which poses challenges.

Representative Datasets. Examples include HelpSteer3 (Wang et al., 2025), TofuEval (Tang et al., 2024), and SLF5K (Scheurer et al., 2023).

2.4. Orthogonal Dimensions of Feedback Taxonomy

The form-centric taxonomy specifies the mathematical representation of supervision but abstracts away operational context. Therefore, it is also useful to analyze signals along several orthogonal dimensions. In practice, feedback with the same form may differ substantially in reliability due to its source. These additional dimensions help disentangle data quality from other factors.

2.4.1. Interaction Structure

This dimension characterizes the temporal scope and dependency of the feedback signal.

•Single-step Feedback: Feedback targets a single turn independently of past or future interactions. This is commonly modeled under an i.i.d. assumption, which simplifies reward learning but may neglect long-term coherence or planning.

•Multi-step and Agentic Feedback: Multi-step feedback captures dependencies across turns, supporting multi-turn consistency and constraint satisfaction, as in OpenAssistant (öpf et al., 2023), LMSYS Chatbot Arena (Chiang et al., 2024), and MT-Bench-101 (Bai et al., 2024). This setting is increasingly relevant to agentic interaction, where feedback covers an entire trajectory of tool calls and actions rather than a single response (Shani et al., 2024; Gao et al., 2024; Shinn et al., 2023).

2.4.2. Feedback Source

This dimension characterizes the origin of the supervision signal, ranging from subjective human judgments to objective verifications.

•Human-Explicit: Human-provided annotations. They remain the gold standard for capturing human values, but are costly to collect at scale and subject to inter-annotator disagreement, which bounds label reliability; Appendix C quantifies this trade-off empirically.

•Human-Implicit: Behavioral signals that indirectly reveal human preferences without requiring explicit annotation, including WildFeedback (Shi et al., 2024) and Liu et al. (2025), which provide supervision from real-world user interaction logs.

•Model-Generated: Signals produced by models rather than by humans, offering improved scalability at the cost of potential bias or drift from human intent. Examples include Constitutional AI (Bai et al., 2022), which generates critiques and preferences guided by human-written constitutions; RFM-RM-as-User (Barreto et al., 2025), which simulates user feedback via trained reward models; and LLM-as-a-Judge approaches.

•Environment-Verifiable: Objective feedback derived from execution environments, test cases, or formal verification. These signals are typically low-noise but are limited to domains with externally checkable correctness, such as coding or instruction following. Examples include EvalPlus (Liu et al., 2023), EvalPerf (Liu et al., 2024), OpenCodeInstruct (Ahmad et al., 2025), IFEval (Dai et al., 2025), and Reveal (Jacovi et al., 2024). Tool-use feedback in agentic settings is a further instance of this source: an agent’s individual tool calls can be checked against environment state, execution results, or task completion criteria.

2.4.3. Feedback Granularity

Feedback granularity specifies the scope of supervision relative to model generation, ranging from holistic judgments to localized annotations.

•Outcome-level: Feedback applies to the final output y and is standard in RLHF. While cost-effective and aligned with user utility, it provides limited localization and exacerbates credit assignment.

•Process-level: Feedback targets intermediate steps or generation trajectories, including PRM800K (Song et al., 2025) and MathShepherd (Wang et al., 2024). In agentic settings, this granularity extends to trajectory-level rewards over an entire episode, per-tool-call feedback on individual actions, and subgoal rewards for intermediate objectives within a longer task, all of which are fine-grained instances of process-level supervision.

2.4.4. Extension to Multimodal Feedback

Although the discussion above centers on text, the same forms and dimensions extend to multimodal generation. Ordinal and cardinal feedback have already been collected over non-textual outputs, e.g., ranked human preferences over generated images (Karthik et al., 2025) and preference-based fine-tuning of diffusion models (Yuan et al., 2024); linguistic feedback in these settings can likewise reference non-textual content, such as visual composition or motion, rather than text alone. Adapting the granularity and source dimensions to multimodal outputs raises distinct challenges, including localization below the level of a whole image or clip and verification with vision-language rather than text-only judges.

Considering these orthogonal dimensions clarifies practical trade-offs central to deploying feedback at scale, including annotation cost, feedback reliability, inter-annotator disagreement, and scalability, alongside the granularity of supervision, and informs the choice of objectives and modeling strategies across alignment settings.

2.5. Scope of the Taxonomy

Category Boundaries: Different types of feedback can convey a judgment about the same concept, but they vary in how that judgment is represented and operationalized for training.

Domains and Languages: The taxonomy is defined by feedback structure rather than task domain or language, so it applies directly to multilingual and domain-specific settings without modification; what varies across settings is which feedback types are practical to collect.

3. Methodology for LLMs Alignment via Complex Human Feedback

After categorizing the feedback data, an important question remains: how can this feedback be effectively leveraged to improve the model?

This section reviews two main categories of alignment methods: (1) Reward-Model-Free Alignment approaches, which directly optimize the language model using feedback without relying on an additional reward model; and (2) Reward-Model-Based Alignment approaches, which either first train a reward model from feedback data, or use a reward model to generate the corresponding feedback and then perform alignment using the reward signal. Our goal is not to provide a deeper algorithmic re-coverage compared with prior studies, but to contribute a feedback-centric view that organizes alignment methods tied to the feedback structure. Figure 4 gives a compact overview of representative methods discussed in this section, organized by feedback type and by whether they are reward-model-free or reward-model-based; Appendix B synthesizes, for each feedback type, the assumptions, suitability, and information exploited by each family of methods.

3.1. Alignment with Ordinal Feedback

Reward-Model-Free Alignment. This line of work (Rafailov et al., 2023; Zhao et al., 2023; Liu et al., 2023; Azar et al., 2024; Meng et al., 2024; Wu et al., 2024; Xiao et al., 2024; Liu et al., 2024; Fisch et al., 2024; Deng et al., 2025) eliminates the need for explicitly training a reward model, and instead directly optimizes the policy from offline preference data by minimizing a pairwise ranking loss defined over preferred and non-preferred responses. Direct Preference Optimization (DPO) (Rafailov et al., 2023), for instance, leverages a closed-form reparameterization that casts the optimal policy as a classifier over preference pairs, enabling stable and efficient fine-tuning without intermediate reward model training. Identity Preference Optimization (IPO) (Azar et al., 2024), derived as a special case of the general $\Psi$-preference optimization framework with $\Psi$ instantiated as the identity function, relaxes the Bradley–Terry assumption underlying DPO. Beyond reliance on pre-collected preference datasets, self-play–based methods (Chen et al., 2024; Gao et al., 2025; Wang et al., 2025) treat model-generated responses as dispreferred and human responses as preferred, enabling automatic data construction and self-improvement.

Reward-Model-based Alignment. Preference annotations are first used to train a reward model with a pairwise ranking objective, such as the Bradley–Terry loss (Stiennon et al., 2020; Bradley and Terry 1952) or its variants (Liu et al., 2024); the language model is then optimized against this reward via reinforcement learning (Williams 1992; Schulman et al., 2017; Ahmadian et al., 2024) or inference-time methods such as Best-of-n (Stiennon et al., 2020; Nakano et al., 2021). Several studies extend this to an iterative setting, using the reward model to construct new pairwise samples for successive policy optimization (Dong et al., 2024; Xiong et al., 2024; Zhang et al., 2024; Rosset et al., 2024; Xie et al., 2024); Razin et al. further argues that effectiveness also depends on preserving sufficient reward variance.

From binary to list preferences. Extending alignment methods from binary to list preferences typically entails generalizing pairwise ranking objectives to listwise formulations, such as ListMLE or contrastive losses (Karthik et al., 2025; Zhu et al., 2024; Song et al., 2024; Liu et al., 2025; Jin et al., 2025). Compared to the binary setting, listwise preference learning jointly models the relative ordering among multiple candidate responses, providing richer signals and enabling hard example mining as well as globally consistent preference modeling (Cao et al., 2007; Zhu et al., 2023).

3.2. Alignment with Cardinal Feedback

Reward-Model-Free Alignment. KTO (Ethayarajh et al., 2024) introduces a human-aware loss framework grounded in prospect theory, which maximizes the utility of model generations using only binary desirable/undesirable feedback. More recently, a growing body of work (Rame et al., 2023; Zhou et al., 2024; Guo et al., 2024; Yang et al., 2024; Zhong et al., 2024) has shifted alignment research toward more flexible, multi-objective, and personalized paradigms, aiming to better capture heterogeneous human values and support controllable preferences. For instance, Controllable Preference Optimization (Guo et al., 2024) and Rewards-in-Context (Yang et al., 2024) incorporate user-specified, multi-dimensional reward signals directly into the model input, enabling dynamic preference adjustment at inference time.

Reward-Model-Based Alignment. Sun et al. (2024) proposed an alternative reward modeling objective, which preserves order consistency without requiring pairwise preference annotations. In addition, several studies (Wang et al., 2024; Adler et al., 2024; Wang et al., 2024; Lin et al., 2025) leverage multi-dimensional scalar feedback (e.g., helpfulness, safety) to train multi-objective reward models that capture diverse and potentially competing human preferences.

3.3. Alignment with Linguistic Feedback

Reward-Model-Free Alignment. Unlike ordinal or cardinal feedback, linguistic feedback is high-dimensional, unstructured, and semantically rich, which makes it substantially more challenging to exploit effectively for learning. A prominent line of research (Scheurer et al., 2023; Bai et al., 2022; Chen et al., 2023,2; Mehri et al., 2025) addresses this challenge by leveraging linguistic feedback for data synthesis, where the original training data are augmented or refined into improved examples guided by the feedback, and the resulting synthesized data are used to fine-tune the model. To reduce reliance on expensive human annotations, a growing body of work (Shinn et al., 2023; Dhuliawala et al., 2024; Besta et al., 2024; Madaan et al., 2023) further explores using LLMs themselves to generate textual feedback via reflection and perform self-refinement.

Reward-Model-Based Alignment. Reward models leverage linguistic feedback in at least two complementary ways, distinguished by whether the critique is a target to imitate or a scaffold for the reward model’s own reasoning. The first treats critique generation as an auxiliary output that the reward model is trained to imitate: the model is trained on critiques that are generated at inference time as part of the judgment itself (Ankner et al., 2024), used as an additional self-supervised training signal (Yu et al., 2025), synthesized to augment scarce critique-annotated data (Ye et al., 2025), or, most directly, produced by casting reward assignment as next-token generation over a critique (Zhang et al., 2024); at inference time, these models first generate an explicit critique before producing the final judgment. The second treats reward assignment itself as a reasoning task, allocating additional training- or test-time computation to deliberate before scoring: examples include reward models fine-tuned to reason step-by-step before a judgment (Chen et al., 2025), models trained to plan an evaluation strategy before acting as a judge (Saha et al., 2025), reward models trained with reinforcement learning to reason toward a score (Guo et al., 2025), and approaches that scale inference-time computation for a generalist reward model (Liu et al., 2025). Compared to the critique-imitation paradigm, this second line of work uses linguistic feedback less as a supervised target and more as a scaffold that shapes how the reward model reasons.

4. Evaluation of Aligned LLMs

As illustrated in the right part of Figure 5, feedback types and evaluation metrics have a many-to-many relationship. Since the goal is to evaluate alignment rather than feedback itself, organizing evaluation methods by feedback type can be ambiguous.

Motivated by these considerations, we categorize evaluation methods according to the type of alignment evidence they rely on, as shown in Figure 5. Direct verification evaluates model outputs against ground truths or other objectively verifiable criteria, whereas proxy verification relies on models or metrics derived from human feedback to approximate human preferences.

Figure 5. Direct and proxy verification metrics and their many-to-many mapping to feedback types.

Figure 5. Direct and proxy verification metrics and their many-to-many mapping to feedback types.

4.1. Direct Verification

Direct verification assumes the existence of a verifiable checker that maps a prompt–response pair to an explicit signal, such as factual consistency with task rules, agreement with labels, or successful execution of generated code. These signals allow alignment to be assessed against objective criteria. This perspective aligns with Reinforcement Learning from Verifiable Rewards (RLVR) (Wen et al., 2025), where the same checker is used as the reward function during training.

Factuality–Based. Factuality-based methods assess whether outputs align with verified facts, typically via accuracy against trusted references or deterministic solutions (e.g., QA, math) (Lightman et al., 2024; Fan et al., 2024; Lai et al., 2024), or objective task-defined rewards (Shani et al., 2024; Wang et al., 2025).

Classification-Based. When objectives are categorical, such as safety or stylistic constraints, standard classification metrics (F1, precision, recall, NDCG) (Ouyang et al., 2022; Mu et al., 2024) quantify consistency with human annotations.

Executable Verification. For objectively testable responses, such as code generation, correctness is validated via execution or test cases, using metrics like Pass@k or pass rate (Chidambaram et al., 2024; Wong and Tan 2024; Li et al., 2025).

Reliability of Direct Verification. Although direct verification grounds evaluation in objective criteria, verifier scores are not always faithful proxies for the intended objective. First, verifiers can be gamed: models may exploit imperfections in the checker or reward function rather than solve the intended task, a failure mode closely related to reward hacking and specification gaming (Helff et al., 2026). Second, when the checker is implemented as an LLM-as-a-judge rather than a fully objective procedure, its judgments may reflect systematic biases or self-preference rather than true task quality (Chen et al., 2024; Ye et al., 2025). Third, in RLVR, higher verifier scores need not correspond to better reasoning: they may arise from spurious rewards that are unrelated to, or even negatively correlated with, correctness (Shao et al., 2026), or from amplifying reasoning paths already present in the base model rather than inducing genuinely new capability (Yue et al., 2025). Verifier scores can therefore saturate or become misleading proxies for the objective they are meant to measure.

4.2. Proxy Verification

When explicit evaluation signals are unavailable, alignment evaluation relies on proxy feedback that approximates human preferences. Proxy verification therefore uses learned evaluators or automated metrics to assess alignment. We refer to this setting as Reinforcement Learning with Unverifiable Rewards (RLUR), a notion introduced in this work.

Reward Model-Based. These methods use trained reward models to score outputs or assess correlation with human preferences, via accuracy or F1 (Wang et al., 2024; Liu et al., 2025), NDCG (Wang et al., 2025), cross-entropy (Liu et al., 2025), or the reward model score itself (Yuan et al., 2023).

Attribute-Based. Attribute-based evaluators use pretrained classifiers, e.g., toxicity or harmfulness detectors, as proxy judges for particular dimensions, such as the safety-focused Perspective API (Wu et al., 2023; Ouyang et al., 2022; Li et al., 2025).

Text-Based. Text-based methods measure similarity to reference responses: lexical metrics such as BLEU Papineni et al. (2002) and ROUGE Lin (2004), and semantic metrics such as BERTScore Zhang et al. (2020) and COMET Williamson et al. (2017), are widely used for fully automatic evaluation of tasks such as summarization and translation (Wu et al., 2023; Stiennon et al., 2020; Ramos et al., 2024; Song et al., 2025).

LLM-as-a-Judge. LLM-as-a-judge uses a large LLM itself as the evaluator, supporting various formats: ranking-based judgments, including pairwise win rates (Shi et al., 2024; Jin et al., 2023; Tucker et al., 2024; Shaikh et al., 2025) and listwise rankings (Yang et al., 2024); scalar ratings, such as attribute-specific scores (Bai et al., 2023; Zhu et al., 2024), preference similarity (Aroca-Ouellette et al., 2025), or numeric scales (Yu et al., 2025; Cui et al., 2023; Liu et al., 2023; Hashemi et al., 2024); and textual critiques (Li et al., 2024; Xie et al., 2024).

Reward Models versus LLM-as-a-Judge. Although both serve as proxies for human preference, the two differ in operational form: a reward model is a trained scoring function that outputs a scalar or ranking directly, while an LLM-as-a-judge relies on prompted evaluation and can additionally produce a natural-language rationale alongside its judgment. They also differ in stability and reproducibility: LLM judges are comparatively more sensitive to prompt design and to the underlying model version, whereas a trained reward model is typically more standardized once fixed.

5. Discussion

5.1. Mismatches and Gaps

The feedback-centric organization adopted throughout this survey surfaces mismatches between data collection, optimization, and evaluation (Section 1) that a component-centric organization would leave implicit. On the training side, rich feedback such as critiques or process-level supervision is often reduced to scalar rewards or pairwise preferences, losing information it originally conveyed, and multi-aspect signals are often aggregated into a single reward, obscuring trade-offs across dimensions such as helpfulness and factuality. On the evaluation side, generic metrics such as win rates or LLM judges often miss improvements along the specific dimensions models were trained on—e.g., process-level supervision evaluated only by outcome-level accuracy leaves reasoning gains unmeasured—and feedback is growing more complex, with preferences varying across contexts (Kim et al., 2025) and rating scales obscuring demographic differences (Ali et al., 2025).

Two directions remain underexplored: principled task–feedback design—which supervision form suits a given deployment and annotation budget—and extending feedback taxonomies and objectives to multimodal models.

5.2. Conclusions

This survey advocates a feedback-centric perspective on LLM alignment, treating the structure of feedback as a primary lens for organizing alignment methods and evaluation, rather than exhaustively cataloging algorithms or metrics. We hope this perspective helps re-center alignment around the structure and semantics of feedback, and supports next-generation alignment pipelines in which data collection, optimization, and evaluation are designed jointly rather than in isolation.

Appendix C.1. Notation and Data

Let $\mathcal{D}={\left\{(P_t,S_t,{\mathbf{f}}_t^{\left(1\right)},{\mathbf{f}}_t^{\left(2\right)},Y_t^{*})\right\}}_{t=1}^N$ denote the annotator-level dataset, where t indexes individual annotations, $P_t\in \{0,1\}$ is the binary preference sign, $S_t\in \mathbb{Z}$ is the scalar score, ${\mathbf{f}}_t^{\left(1\right)},{\mathbf{f}}_t^{\left(2\right)}$ are the linguistic feedback texts for response 1 and response 2 respectively, and $Y_t^{*}\in \{0,1\}$ is the aggregated majority preference for the corresponding item, which serves as the prediction target throughout. Each annotation carries an item identifier $i\left(t\right)$ mapping it to the underlying comparison pair.

• Embedding representations.

We encode all linguistic feedback using a frozen Sentence-BERT model (all-MiniLM-L6-v2; Reimers and Gurevych 2019) to obtain dense representations. Specifically, we embed the reasoning text to obtain ${\mathbf{x}}_t^{\mathrm{reason}}\in {\mathbb{R}}^d$ and the two per-response feedback texts to obtain ${\mathbf{x}}_t^{\mathrm{fb}1},{\mathbf{x}}_t^{\mathrm{fb}2}\in {\mathbb{R}}^d$, where $d=384$. Response texts are similarly encoded: ${\mathbf{r}}_i^{\left(1\right)},{\mathbf{r}}_i^{\left(2\right)}\in {\mathbb{R}}^d$ for item i.

Appendix C.2. Unified Mutual Information Estimation

A central design principle of our analysis is to estimate all mutual information (MI) quantities using the same methodology, so that the resulting numbers are directly comparable across feedback channels.

• Classifier-probe framework.

We estimate MI through the cross-entropy gap of a logistic regression probe. For a generic feature matrix $\mathbf{X}$ and target Y:

$$\widehat{\mathrm{MI}}(Y;\mathbf{X})=\widehat{H}\left(Y\right)-\widehat{\mathrm{CE}}(Y\mid \mathbf{X}),$$

(A1)

where $\widehat{H}\left(Y\right)$ is the plug-in marginal entropy (in bits) and $\widehat{\mathrm{CE}}(Y\mid \mathbf{X})$ is the cross-entropy (in bits) of a logistic regression classifier trained on $\mathbf{X}$ and evaluated on held-out data. For conditional MI we use:

$$\widehat{\mathrm{MI}}(Y;\mathbf{X}\mid \mathbf{Z})=\widehat{\mathrm{CE}}(Y\mid \mathbf{Z})-\widehat{\mathrm{CE}}(Y\mid \mathbf{X},\mathbf{Z}).$$

(A2)

This framework is applied uniformly to all channels: the binary preference P (encoded as a one-dimensional feature), the scalar score S (concatenated with P), and the linguistic feedback embedding (concatenated with P and S). Using the same estimator for every channel ensures that any systematic bias (e.g., under-fitting of logistic regression on high-dimensional inputs) affects all estimates in the same direction, preserving ordinal comparisons.

• Cross-validation and uncertainty.

All cross-entropy estimates are computed via K-fold stratified cross-validation ($K=10$) to reduce variance from a single train/test split. We report the mean and standard deviation over folds. For conditional MI, the standard deviation is propagated as ${\sigma }_{\mathrm{CMI}}=\sqrt{{\sigma }_{\mathrm{CE}\left(Z\right)}^2+{\sigma }_{\mathrm{CE}\left(XZ\right)}^2}$, treating fold-level errors as approximately independent.

Appendix C.3. Mutual Information

Using the framework of Eqs. (A1)–(A2), we compute three quantities:

• $\mathrm{MI}(Y^{*};P)$: information in the binary preference sign alone. The feature matrix is $\mathbf{X}=\left[P_t\right]\in {\mathbb{R}}^{N\times 1}$.
• $\mathrm{MI}(Y^{*};S\mid P)$: incremental information from the score beyond the preference sign. We compare $\mathbf{Z}=\left[P_t\right]$ against $[\mathbf{Z},S_t]\in {\mathbb{R}}^{N\times 2}$.
• $\mathrm{MI}(Y^{*};\mathbf{F}\mid P,S)$: incremental information from the linguistic feedback embedding beyond both P and S. We compare $\mathbf{Z}=[P_t,S_t]$ against $[\mathbf{Z},{\mathbf{x}}_t^{\mathrm{reason}}]\in {\mathbb{R}}^{N\times (2+d)}$.

Appendix C.4. Score Magnitude Prediction

To assess whether linguistic feedback carries information about how strongly an annotator prefers one response, we estimate $\mathrm{MI}\left(\right|S|;\mathbf{F}\mid P)$ using the same classifier-probe approach with $|S_t|$ as a multi-class target. This measures the extent to which reasoning text predicts score magnitude beyond what the preference direction already reveals.

• Permutation null.

We randomly permute the reasoning embeddings across annotators, breaking the correspondence between each annotator’s feedback and their score while preserving the marginal distribution of embeddings. The permuted conditional MI is expected to be near zero; the gap between the real and permuted estimates isolates genuine information content.

Appendix C.5. Disagreement Prediction

For each item i, we define annotator disagreement as the binary entropy of the preference distribution:

$$\begin{array}{cc}\hfill & D_i=-{\overline{P}}_{i}log{\overline{P}}_i-(1-{\overline{P}}_i)log(1-{\overline{P}}_i),\hfill \end{array}$$

(A3)

$$\begin{array}{cc}\hfill & {\overline{P}}_i={\displaystyle \frac{1}{|{\mathcal{A}}_i|}}\sum _{t:i\left(t\right)=i}{P}_t,\hfill \end{array}$$

(A4)

where ${\mathcal{A}}_i$ is the set of annotators for item i, and $D_i=0$ when ${\overline{P}}_i\in \{0,1\}$.

• Response-only features.

To avoid circularity—annotator-produced reasoning is generated jointly with the preference and thus reflects disagreement almost by construction—we predict $D_i$ from response-only features:

$${\mathbf{x}}_i^{\mathrm{resp}}=\left[{\mathbf{r}}_i^{\left(1\right)};{\mathbf{r}}_i^{\left(2\right)};|{\mathbf{r}}_i^{\left(1\right)}-{\mathbf{r}}_i^{\left(2\right)}|\right]\in {\mathbb{R}}^{3d},$$

(A5)

where ${\mathbf{r}}_i^{\left(1\right)},{\mathbf{r}}_i^{\left(2\right)}$ are the Sentence-BERT embeddings of the two response texts. A Ridge regression ($\alpha =1.0$) is trained to predict $D_i$ from ${\mathbf{x}}_i^{\mathrm{resp}}$, and we report $R^2$ averaged over 10-fold cross-validation.

This setup measures whether the content of the response pair itself is predictive of how much annotators will disagree, without leaking annotator-side information.

• Permutation null.

We randomly permute the response embeddings across items, breaking the correspondence between response content and annotator disagreement while preserving the marginal distribution. A negative permuted $R^2$ (as observed) confirms that the model does not extract structure from mismatched response–disagreement pairings.

Appendix C.6. Actionability

The actionability metric tests whether the direction of linguistic feedback (from rejected to preferred) aligns with the corresponding direction in response embedding space.

• Per-annotator alignment.

For annotator t, we define the signed feedback difference and signed response difference as:

$$\begin{array}{cc}\hfill \Delta {\mathbf{f}}_t& =\left\{\begin{array}{cc}{\mathbf{x}}_t^{\mathrm{fb}1}-{\mathbf{x}}_t^{\mathrm{fb}2}\hfill & \mathrm{if}{Y}_t^{*}=1,\hfill \\ {\mathbf{x}}_t^{\mathrm{fb}2}-{\mathbf{x}}_t^{\mathrm{fb}1}\hfill & \mathrm{otherwise},\hfill \end{array}\right.\hfill \end{array}$$

(A6)

$$\begin{array}{cc}\hfill \Delta {\mathbf{r}}_t& =\left\{\begin{array}{cc}{\mathbf{r}}_{i\left(t\right)}^{\left(1\right)}-{\mathbf{r}}_{i\left(t\right)}^{\left(2\right)}\hfill & \mathrm{if}{Y}_t^{*}=1,\hfill \\ {\mathbf{r}}_{i\left(t\right)}^{\left(2\right)}-{\mathbf{r}}_{i\left(t\right)}^{\left(1\right)}\hfill & \mathrm{otherwise}.\hfill \end{array}\right.\hfill \end{array}$$

(A7)

Both vectors point in the “preferred − rejected” direction according to annotator t’s own decision. This per-annotator alignment avoids a directional mismatch that would arise if item-level majority labels were used to sign one vector while annotator-level labels were used to sign the other.

The actionability score is the average cosine similarity:

$$A={\displaystyle \frac{1}{N}}{\displaystyle \sum _{t=1}^N}{\displaystyle \frac{\Delta {\mathbf{f}}_t·\Delta {\mathbf{r}}_t}{\parallel \Delta {\mathbf{f}}_t\parallel \parallel \Delta {\mathbf{r}}_t\parallel }}.$$

(A8)

• Permutation null and effect size.

To assess significance, we construct a null distribution by randomly permuting the annotator indices, breaking the correspondence between feedback and response pairs while preserving marginal distributions. The permuted mean cosine similarity is expected to be near zero; a significant gap confirms that the alignment reflects genuine semantic correspondence rather than distributional artifacts. We report $\Delta =A_{\mathrm{true}}-A_{\mathrm{perm}}$ with a 95% bootstrap confidence interval (5,000 resamples) and Cohen’s d using the pooled standard deviation. Because cosine similarity in high-dimensional spaces concentrates near zero for unrelated vectors, the absolute value of A is expected to be small; Cohen’s d provides a scale-free effect size that is more appropriate for assessing practical significance.

Appendix C.7. Domain-Specific Analysis

HelpSteer3 Wang et al. (2025) provides data from multiple domains, which supports a domain-specific analysis. Across the Code, General, Multilingual, and STEM subsets, the results consistently show that language feedback contains signals beyond binary preferences. First, score magnitude retains substantial mutual information with language feedback in every subset (Table A1), indicating that language feedback encodes preference strength regardless of domain. Second, disagreement predictability varies by domain (Table A3): higher-ambiguity settings such as Multilingual show the strongest predictability ($R^2=10.4\%$), while more structured domains such as Code show a weaker, though still positive, effect ($R^2=1.5\%$); in every domain the real $R^2$ exceeds its permuted null, so the direction of the effect is consistent even as its magnitude varies. Third, actionability shows a stable positive alignment between feedback and response improvements across all domains, with a non-trivial effect size in every subset (Cohen’s d from 0.298 to 0.754; Table A2). Together, these domain-level results support the robustness of the patterns discussed in the main text.

Table A1. Score Magnitude MI: $I\left(\right|S|;F\mid P)$.

Subset	Real	Permuted Null
Code	0.716	-0.013
General	0.861	-0.007
Multilingual	0.834	-0.009
STEM	0.727	-0.016
Overall	0.892	-0.004

Table A1. Score Magnitude MI: $I\left(\right|S|;F\mid P)$.

Subset	Real	Permuted Null
Code	0.716	-0.013
General	0.861	-0.007
Multilingual	0.834	-0.009
STEM	0.727	-0.016
Overall	0.892	-0.004

Table A2. Actionability: $cos(\Delta f,\Delta r)$.

Subset	Real	Permuted Null	Cohen’s d
Code	0.102	0.000	0.754
General	0.071	0.001	0.580
Multilingual	0.030	0.000	0.298
STEM	0.070	-0.001	0.599
Overall	0.070	0.000	0.580

Table A2. Actionability: $cos(\Delta f,\Delta r)$.

Subset	Real	Permuted Null	Cohen’s d
Code	0.102	0.000	0.754
General	0.071	0.001	0.580
Multilingual	0.030	0.000	0.298
STEM	0.070	-0.001	0.599
Overall	0.070	0.000	0.580

Table A3. Disagreement Prediction.

Subset	Score Variance	${\mathit{R}}^2$ (real)	${\mathit{R}}^2$ (permuted null)
Code	0.166	1.5%	-9.6%
General	0.198	0.6%	-5.9%
Multilingual	0.351	10.4%	-10.8%
STEM	0.205	4.3%	-18.9%
Overall	0.222	10.4%	-2.8%

Table A3. Disagreement Prediction.

Subset	Score Variance	${\mathit{R}}^2$ (real)	${\mathit{R}}^2$ (permuted null)
Code	0.166	1.5%	-9.6%
General	0.198	0.6%	-5.9%
Multilingual	0.351	10.4%	-10.8%
STEM	0.205	4.3%	-18.9%
Overall	0.222	10.4%	-2.8%