Seeing the Forest and the Trees: A Survey of Analytic Rubrics for Holistic Reward Modeling in LLMs

1. Introduction

Reinforcement learning has become a mainstream paradigm for aligning AI models through interaction with humans and environments [1,2]. In this paradigm, reward models play a central role by translating human preferences into trainable supervision signals for policy optimization [3], while also serving as scalable proxies for continual adaptation [4]. Yet reward modeling is inherently vulnerable to misalignment at multiple levels [5]. Outer alignment concerns whether the specified reward faithfully captures the intended human objective. Inner alignment concerns whether the learned policy robustly optimizes the specified reward, rather than relying on internally learned proxy objectives (often referred to as mesa-objectives) [6] that achieve high reward only superficially or distributionally. This motivates the study of rubric reward models, which seek to make reward specification more explicit, decomposable, and controllable.

As a bridge between human intentions and model optimization, reward modeling plays an anchoring role in both representing human preferences and enabling models to learn from reward signals. As summarized in Figure 1, we analyze the major paradigms in terms of their trade-offs between scalability and alignment risks. Generative RMs [7] can be more expressive than Scalar RMs [8], as they support preference reasoning, yield more interpretable intermediate judgments, and better exploit inference-time scaling. In contrast, both Scalar RMs and Rule RMs [9] are constrained by human-provided supervision: Sacalar RMs depend on preference annotations, whereas Rule RMs rely on manually specified rules, which may limit their scalability. From the perspective of alignment, Rule RMs reduce some inner misalignment concerns because they do not themselves introduce learned internal objectives, but their manually specified criteria are often too narrow to faithfully capture human intentions [10]. In contrast, Generative and Scalar RMs are better suited to capturing the complexity of human objectives through preference supervision and richer reasoning traces, but they remain less controllable during policy optimization. Beyond these paradigms, Rubric RMs [11,12,13] decompose holistic judgments into multiple explicit and interpretable criteria, providing a more fine-grained interface for aligning reward specification with human objectives. Rubric reward modeling provides a promising way to balance expressiveness, controllability, and scalability by organizing evaluation dimensions into structured and inspectable criteria.

treeRubrics originate from educational assessment and psychometrics, where they are primarily introduced to mitigate the reliance of forestholistic scoring on human raters’ overall impressions, which are prone to cognitive biases such as the Halo Effect [14]. In contrast, treeanalytic scoring decomposes evaluation into multiple relatively independent dimensions, helping standardize the assessment process, improve inter-rater reliability, and provide more precise feedback[15]. Motivated by these insight, Rubric RMs have recently emerged as a new paradigm for reward modeling, extending earlier approaches based on human-written principles or evaluation criteria [10,16]. However, as this line of work is still evolving, there is not yet a detailed survey that provides a systematic account of its terminology, conceptual foundations, and technical development. To fill the existing gaps in the literature, this survey provides a systematic review of rubric reward modeling. Specifically, we discuss the formulation, modeling, and reasoning of Rubric RMs, as well as their role in policy optimization. The main contributions of this survey are as follows:

• To the best of our knowledge, this is the first systematic attempt to survey Rubric RMs, an emerging RM paradigm based on explicit and interpretable evaluation criteria.
• We introduce a unified framework for Rubric RMs from four perspectives: rubric foundations, modeling, reasoning, and their role in policy optimization, As shown in Figure 2.
• Under this framework, we synthesize representative methods, applications, and evaluation practices, trace the technical evolution of the field, and highlight key open challenges and future directions.

2. Overview

In AI alignment [17,18], the RM serves as a bridge between humans and the model, capturing human preferences over states of the world and enabling the model to act consistently with these preferences. In practice, however, human preferences are not directly observable and must be inferred from data. Holistic RMs address this challenge by learning a parameterized reward function $R_{\theta }$ that maps inputs to scalar rewards, thereby implicitly approximating the underlying preference function. Typically, this objective can be effectively operationalized through the Bradley-Terry loss [19], which maximizes the likelihood of observed human preferences. Given a dataset of pairwise comparisons $\mathcal{D}=\left\{(x,y,y^{\prime },l)\right\}$, the loss is defined as:

$$\begin{array}{cc}\hfill {\mathcal{L}}_{\mathrm{BT}}=& -\mathbb{E}[llog\sigma \left(R_{\theta }(x,y^{\prime })-R_{\theta }(x,y)\right)\hfill \\ \hfill & +(1-l)log\sigma \left(R_{\theta }(x,y)-R_{\theta }(x,y^{\prime })\right)],\hfill \end{array}$$

(1)

where $\sigma (·)$ denotes the sigmoid function, $l\in \{0,1\}$ indicates the preference label, with $l=1$ meaning $y^{\prime }$ is preferred over y, and $l=0$ otherwise. However, the theoretical guarantees of Holistic RMs rely on the assumption of “infinite and consistent preference data,” which cannot be satisfied in practice, making them inevitably exposed to the risk of misalignment.

Analytic RMs introduce an intermediate variable to guide preference learning. Drawing upon core concepts from Education and Psychometrics [15,20], and a large body of recently emerging works [11,21], we define a rubric $\mathcal{C}$ as a set of structured, explicit, and fine-grained natural language scoring criteria:

$$\mathcal{C}=\{c_1,c_2,\dots ,c_K\},$$

where each $c_k$ denotes an explicit evaluation criterion. The core paradigm of Rubric RMs consists of two distinct phases: rubric construction and rubric-guided judgment. The rubric construction process is defined as follows: given an input $\mathcal{I}$, a generator $\mathcal{G}$ produces a corresponding set of rubric criteria $\mathcal{C}$. Formally, this is defined as:

$$\mathcal{C}\sim \mathcal{G}(·|\mathcal{I}),$$

Conditioned on $\mathcal{C}$, the judge (RMs) evaluates the candidate responses along multi criteria and aggregates the judgments into an overall reward:

$$R_{\theta }(x,y\mid \mathcal{C})=\mathcal{F}\left(J(x,y,c_1),\dots ,J(x,y,c_K)\right),$$

where J denotes a criterion-level judge and $\mathcal{F}$ is an aggregation function. In contrast, which infer preferences through a single implicit reward signal, Analytic RMs make the evaluation process more explicit by decomposing overall judgments into criteria before aggregation.

Accordingly, this survey is organized into four parts (in Figure 3): We first discuss the foundations (§3) of Rubric RMs , including what rubrics are and how they are constructed. We then review rubric modeling (§4), focusing on how rubrics are learned, synthesized, or parameterized. Next, we examine rubric reasoning (§5), namely how multiple criteria are interpreted and aggregated into reward signals. Finally, we cover optimization (§6), where Rubric RMs are used to support downstream policy learning and decision-making.

3. Foundations

We establish the foundational concepts and methodologies underlying rubric construction. To provide a structured overview, we decompose the rubric formulation pipeline into three primary dimensions: inputs, synthesis, and contents.

3.1. Rubric Inputs

The rubric construction process can be categorized into three settings based on how many candidate responses are considered jointly to extract the criteria: Point-wise: The generator synthesizes a rubric by examining the query [22] or a single response [23] at a time to formulate absolute criteria, optionally accompanied by reference examples [24]. This manner is simple, but often suffers from calibration issues due to the lack of explicit comparison. Pair-wise: The generator considers two responses jointly, extracting criteria based on the contrasting features that determine relative preference [25,26]. This manner naturally produces discriminative criteria and aligns with preference learning frameworks. However, because it is based on local comparisons, it may not fully capture the relative ordering of a larger candidate set. List-wise: The generator analyzes a broader set of responses simultaneously to distill criteria that govern their relative ordering and global quality [27,28,29,30]. This manner can capture richer ranking information than pairwise comparison, though it is typically more complex and costly to implement.

The input to generator may include not only the response itself, but also context for rubric construction. Such context is particularly important in long-horizon decision-making tasks, where responses are often affected by hallucinations and cross-step inconsistencies. To improve rubric quality, prior work enriches the context with signals of different origins, ranging from model-internal knowledge to externally grounded feedback. These contextual signals can be broadly organized into the following categories:

Expert guidelines: A common way to enrich rubric context is to introduce normative signals designed by humans, such as predefined extraction principles [11,31,32] or annotations [33,34]. These guidelines provide explicit standards and can substantially improve quality and consistency of the generated rubrics, especially in tasks requiring structured judgment. However, such context often comes with high construction cost and limited scalability, since the quality of the rubric depends on the availability of domain expertise and human supervision.

Factual information: Another line of work augments rubric context with grounded factual signals, including retrieved knowledge from external environments [27,35], retrieving domain knowledge and cases from semantically similar queries as reference [24,36] or feedback from interactions with external systems such as agents or codebases [37]. Compared with purely intrinsic context, this paradigm can more effectively mitigate hallucinations in open-domain and complex tasks by anchoring rubric construction to verifiable evidence. Its main limitation is that rubric quality becomes highly dependent on the reliability, latency, and coverage of the external feedback source.

3.2. Rubric Synthesis.

Given a target task and input, the corresponding rubric can be constructed and incorporated into the alignment pipeline, which can be characterized along two dimensions: source and adaptability.

3.2.1. Source of Synthesis

Human expert based methods rely on manually designed criteria provided by human experts. Such rubrics often encode subjective yet valuable human judgments, domain-specific standards, and task-specific requirements that are difficult to capture automatically, especially in specialized or high-stakes domains. Early work often focused on universal rubrics designed for a dataset or domain [10,38], while more recent efforts increasingly emphasize instance-specific and fine-grained criteria [21]. Although expert-designed rubrics usually offer strong alignment quality and interpretability, they are inherently limited in scalability, since constructing and maintaining high-quality rubrics requires substantial human effort and expertise.

Synthetic methods construct rubrics automatically, typically using LLMs as the generator. A prevalent mechanism involves directly prompting LLMs to generate rubrics tailored to specific inputs [11,39]. Beyond simple one-shot prompting, recent methods also explore iterative refinement [13,28,40,41] and filtering mechanisms [33,42,43] to extract more discriminative, reliable, and task-aligned criteria. Taking a different approach, Huang et al. [44] introduce a unique Rubric-First strategy, which pre-constructs a high-quality rubric library and subsequently filters or synthesizes queries that align with these rubrics. Benefiting from their broad prior knowledge and generalization ability, LLMs can synthesize reasonably accurate rubrics across a wide range of tasks and domains, substantially improving scalability over purely human-designed approaches. However, the quality of synthetic rubrics still depends on the model’s internal knowledge, prompt design, and external grounding signals when available. Specially, Orcust [34] takes a hybrid approach that integrates human-expert design with automatic synthesis. While human experts define explicit domain requirements and prohibited behaviors, LLMs are leveraged to synthesize articulate additional strategies. It is worth noting that recent work [45] has begun to consolidate these fragmented generation, refinement, and evaluation into a composable pipeline, which enables a more systematic and rigorous comparison of different methodologies.

3.2.2. Adaptability of Synthesis

Beyond the source of synthesis, another important dimension is whether rubrics remain fixed during RL optimization; accordingly, existing approaches can be divided into static and dynamic paradigms.

Static synthesis generates a fixed set of rubrics prior to the RL process or relies exclusively on the query during the RL process, making them entirely independent of the evolving policy distribution or rollout behavior. This paradigm is simple, efficient, and easy to scale, and it remains the default setting in most works [31,33]. However, as the policy model ${\pi }_{\varphi }$ evolves, fixed rubrics may gradually become incomplete or exploitable, failing to cover newly emerged failure modes. As a result, static criteria are especially vulnerable to Goodhart-like reward hacking, where the policy learns to satisfy superficial rubric signals without achieving the intended behavior. Some work attempts to alleviate this issue by manually summarizing observed failure patterns and injecting defensive criteria [39,46], but such patching remains labor-intensive and difficult to sustain under continuous policy shift.

Dynamic synthesis updates, extracts, or regenerates rubrics online according to the current policy distribution or rollout behavior. Compared with static rubrics, this paradigm is better suited to capturing evolving failure modes and maintaining evaluative relevance throughout optimization. Recent methods explore several directions, including online rubric extraction, which dynamically constructs rubrics based on rollout results during the reinforcement learning process [25,27,29,47], and joint optimization frameworks where the policy model co-evolves with a dynamic rubric generator [48,49]. Although dynamic approaches are more promising for robust long-term optimization, they also introduce additional complexity in stability, computational cost, and coordination between rubric generation and policy learning.

3.3. Rubric Contents

Synthesized rubrics specify the criteria used to evaluate a response, action sequence, or decision trajectory. For complex tasks, rubrics are often designed with distinct evaluation foci to assess different aspects of performance separately, such as outcome and process [23], stylistic constraints [50], or positive and negative [27,36]. Furthermore, rubrics can also serve as fine-grained factual constraints on both the content and the reasoning trajectory of response [51,52]. Despite these explorations, there is still no unified consensus on what rubric design is universally optimal; in practice, the most effective formulation often depends on the task complexity, domain requirements, and downstream optimization objective.

4. Modeling

The modeling stage focuses on learning the key components of the rubric evaluation pipeline, including rubric construction, rubric-guided judgment, and, in some cases, their joint optimization.

4.1. What to Learn?

Rubric serves as an intermediate variable in the rubric reward modeling, bridging the designer’s intended evaluation objectives and downstream supervision signals. Accordingly, modeling in Rubric RMs involves two closely related goals: optimizing the generator to construct high-quality rubrics, and optimizing the judge to perform reliable rubric-guided judgment. Some studies focus on the generator, which serves as the core component of Rubric RMs by constructing task-specific evaluation criteria. Lv et al. [22] train a generator aligned with human values using human preference data. In contrast, other studies target the judge, namely the reasoning and aggregation component, to improve its ability to interpret generated rubrics and derive reliable reward signals from them. LLM-RUBRIC [53] aligns the judge’s evaluations with human values by training a personalized calibration network. Beyond such separate optimization, Xu et al. [54] simultaneously optimize the generator and the judge, while Chen et al. [55] and Liu et al. [43] treat rubric construction and rubric-guided judgment as a unified process, enabling end-to-end optimization.

4.2. How to Learn?

In the absence of explicit intermediate supervision for rubric construction and judgment, many studies directly leverage a strong LLMs as the generator or the judge, relying on its pretrained capabilities for rubric synthesis and score aggregation [31]. While this training-free strategy is flexible, it often struggles in complex scenarios that demand hierarchical intent structures, faithful subtask decomposition, and task-specific reasoning. To mitigate these weaknesses, some works adopt SFT [26,46,56], which uses large amounts of high-quality training data to teach the model to follow the desired rubric format and perform more reliable task planning and decomposition. More advanced approaches introduce reinforcement learning based methods, which alleviate the reliance on explicit supervision over intermediate reasoning processes and instead optimize model behavior directly from preference signals. Typical examples include Direct Preference Optimization (DPO) [49] and Group Relative Policy Optimization (GRPO) [22,36]. Specially, Jin et al. [57] and Srivastava et al. [58] inject rubric knowledge into the judge’s parameters.

5. Reasoning

Rubric Reasoning focuses on how multiple criteria are jointly considered by the judge to produce the final reward or supervision signal. This process is inherently challenging because the relationships among criteria can be complex: they may be complementary, partially overlapping, hierarchically dependent, or even conflicting. Therefore, rubric reasoning is not merely a matter of simple aggregation, but of deciding how these criteria should interact, be prioritized, and be balanced in context.

5.1. Explicit Aggregation

Explicit aggregation maps criterion-based judgments over a selected rubric set $\mathcal{R}$ into a final scalar value through an explicit aggregation function. Concretely, the judge first assigns a score $s_i$, usually a binary score, to each criterion $c_i\in \mathcal{R}$, yielding a score vector $\mathbf{s}={[s_1,\dots ,s_K]}^{\top }$, which is then converted into a scalar reward r by applying $\mathcal{F}\left(\mathbf{s}\right)$. Depending on the relational assumptions encoded in $\mathcal{F}$, such aggregation functions can be broadly divided into two primary categories: linear aggregation, which assumes additive and largely independent contributions across criteria, and structure-aware aggregation, which captures richer interactions such as hierarchical dependencies.

Linear Aggregation. 

Linear function provides a simple and interpretable way to combine multiple evaluation dimensions: $\mathcal{F}\left(\mathbf{s}\right)={\displaystyle \frac{{\sum }_i{w}_i{s}_i}{{\sum }_i{w}_i}}$. Here, $w_i$ reflects the relative importance of criterion $c_i$ in the final evaluation. A larger weight indicates a stronger contribution of the corresponding dimension, whereas a smaller weight suggests lower importance. In many cases, such weights must be manually specified or calibrated with additional supervision in order to faithfully reflect the relative importance of different criteria [11,28]. Since such supervision is often unavailable in practice, many approaches adopt uniform weighting [59,60], e.g., $w_i=1$, which implicitly assumes that all criteria are equally important. While this simplification improves usability, it also imposes a strong assumption and may limit the generality of linear aggregation in complex scenrios. Furthermore, some works [34,39,61] target scenarios where hard constraints (e.g., format) and soft constraints (e.g., logic) coexist, thereby achieving a synergy between Rule RMs and Rubric RMs. Overall, linear aggregation is interpretable and flexible, making it suitable for dynamically updating rubrics to mitigate reward hacking. However, it struggles to capture complex non-linear dependencies and its computational cost typically grows linearly with the number of criteria.

Structure-Aware Aggregation. 

In complex settings, criteria may exhibit non-linear trade-offs or hierarchical dependencies (e.g., safety typically possesses veto power), meaning that the contribution of one dimension to the final reward depends on the values of the others. To address this, Huang et al. [44] attempt to employ hand-crafted non-linear rules to capture the complex relationships among various criteria. Similarly, He et al. [46] assign a reward signal if and only if all criteria are strictly satisfied. Furthermore, Bi et al. [23] implement a dynamic strategy that activates the evaluation of subsequent process criteria only upon the successful verification of factual outcomes. Despite these advancements, relying on such manually hard-coded logic still struggles to fully encompass complex interactions.

5.2. Implicit Aggregation

Implicit Aggregation leverages the powerful In-context Learning (ICL) capabilities of LLMs. This method foregoes the intermediate step of item-by-item scoring. Instead, it inputs the entire rubric $\mathcal{R}$ as a context prompt into the model. The model performs internal reasoning to weigh various criteria, directly outputting a final scalar signal. The judge J implicitly subsumes the roles of both the individual criterion scoring and the explicit aggregation function $\mathcal{F}$. Gunjal et al. [11] first coined the term Implicit Aggregation, demonstrating, and demonstrate that Implicit Aggregation outperforms Explicit Aggregation under equivalent conditions by experiment. Following this trajectory, Yuan et al. [32], Bai et al. [62] and Liu et al. [56] have also adopted this approach. Despite its empirical success in capturing complex interactions, Implicit Aggregation presents significant challenges. Its inherent "black-box" nature severely restricts interpretability and fine-grained controllability, as researchers can neither observe intermediate criteria scores nor directly adjust their weights. Furthermore, this approach is hindered by context length constraints, increased prefill costs for extensive rubrics, and susceptibility to position bias, where the ordering of criteria within the prompt can skew the model’s judgment.

6. Rubric RMs as the Core of the Optimization Loop

Rubric RMs play a central role in the optimization loop of downstream tasks by providing both supervision and control signals. Beyond assigning scalar rewards, they improve data quality, guide policy optimization, and strengthen inference-time decision making by translating implicit and holistic task objectives into explicit and analyzable criteria. In the following, we discuss how Rubric RMs contributes to downstream optimization across different stages of the loop.

6.1. Data Synthesis

Compared with traditional data synthesis methods, rubrics enable more fine-grained quality control over both synthesized outputs and their intermediate construction processes via structured criteria. For result quality, RubricHub [31] develop automated coarse-to-fine rubric generation frameworks to construct comprehensive, fine-grained, and highly discriminative rubrics, and further use rubric-based rejection sampling to curate fine-tuning data. To further optimize data synthesis, Optimsyn [63] treat rubric generation as a learnable policy. It utilizes influence-guided optimization to generate rubrics and synthesize corresponding data, thereby maximizing the synthesized data’s training utility on the target model. For process control, rubrics can additionally serve as iterative guidance signals in long-horizon generation. For example, ARISE [64] shows that rubrics not only evaluate intermediate outputs, but also drive step-by-step refinement throughout agentic optimization.

6.2. Policy Learning

In policy learning, rubrics can be transformed into structured supervision signals that improve credit assignment. Compared with traditional outcome feedback, rubric-based supervision provides denser and more structured learning signals, allowing models to more precisely identify which behaviors should be reinforced. At the supervision level, rubrics can guide both outcome-level [22,27,47] and process-level [48,51,52] optimization by decomposing complex objectives into explicit dimensions. Pushing this granularity even further, recent work translates coarse response-level rubric evaluation into fine-grained token-level credit assignment [65]. Furthermore, some methods [66,67] introduce a self-rewarding mechanism, allowing the policy model to serve simultaneously as both the generator and the judge, thereby mitigating reward overoptimization while enabling the quality of the reward signal to evolve synchronously with the policy’s own capabilities. To further enhance the quality and robustness of these supervision signals, Liu et al. [43] demonstrate inference-time scaling for generative reward models, by parallelly sampling multiple diverse criteria to evaluate responses, and employing a meta reward model to score and filter criteria. Specially, RuscaRL [68] demonstrates that rubrics can serve not only as verifiable rewards, but also as explicit scaffolding for exploration, effectively alleviating exploration bottlenecks and improving reinforcement learning on open-ended reasoning tasks.

6.3. Inference-Time Verification

At inference time, Rubric RMs can serve as a verifier to directly improve policy outputs, extending its role beyond training supervision into inference-time decision making. They can support ranking, verification, and revision of candidate outputs, for example in Best-of-N, reranking, self-refinement, and critique-and-revision pipelines, by providing explicit criteria for test-time evaluation. For instance, Wan et al. [69] propose DeepVerifier, a rubric-guided verification framework for Deep Research Agents that enables iterative self-correction through test-time feedback and retry. Beyond iterative refinement, Raghavendra et al. [37] show that context-grounded rubrics constructed by agents can serve as an effective and scalable execution-free verifier for SWE agents, improving test-time selection while providing more interpretable reward signals.

6.4. Domain Adaptation

Rubrics are naturally applicable across a wide range of domains because many downstream tasks rely on implicit expert knowledge that cannot be easily captured by preference signals. The core value of rubric modeling lies in making such domain knowledge explicit by recovering it from task requirements and data, providing a more scalable way to align models with domain-specific objectives. This is particularly important in domains such as medicine [21,24,57], law [52], science [42], mathematics [32], and coding [37], where outputs must satisfy fine-grained constraints over multiple criteria. More broadly, in complex settings such as DeepResearch [22,27,51], and multimodal reasoning [34,60], rubrics can further incorporate expert priors, tool feedback, and environmental interactions as structured constraints, enabling models to better coordinate decision making under real-world task requirements.

7. Evaluation and Benchmarking

In this section, we review existing open-source benchmarks (Table 1) and evaluation methods for Rubric RMs evaluation from two perspectives: Rubric quality evaluates the model’s ability to generate accurate and useful rubrics, which directly measures the quality of rubrics. Rubric-guide judgment evaluates whether the generated rubric can support reliable reward modeling and downstream task judgment. We also provide a collection of related datasets in Appendix B.

Rubric Benchmarks. 

Rubric Benchmarks can be directly used to evaluate the quality of rubrics. RubricBench provides a curated collection of challenging, expert-annotated examples for systematically evaluating reward models under rubric-guided preference alignment. RUBRICEVAL is the first rubric-level meta-evaluation benchmark designed to assess the performance and reliability of RMs in instruction-following tasks. In specific domains, HealthBench introduces expert-defined rubrics for clinical tasks and employs LLM-based judges to assess whether model outputs adhere to medical standards. PaperBench extends this paradigm to scientific research, using multi-step rubrics to evaluate reproducibility in machine learning. ProfBench further broadens the scope to professional domains such as science and business. As a result, this fine-grained evaluation is particularly suitable for high-stakes domains, such as medicine, scientific research, and other settings where evaluation criteria require expert specification.

Reward Benchmarks. 

Reward Benchmarks can be used to evaluate the effectiveness of judgments guided by model-generated rubrics. Among existing benchmarks, RewardBench has become one of the most widely used general-purpose testbeds, covering preference evaluation across a broad range of domains. RewardBench2 further increases the difficulty of this setting by introducing harder examples and multi-candidate selection. Other benchmarks [76,77,78] refine evaluation from complementary perspectives, emphasizing robustness to superficial cues and faithfulness to objective reasoning quality. At the same time, holistic evaluation has an important limitation: correct preference outcomes do not necessarily imply correct reward reasoning. This motivates benchmarks such as RMB and PPE, which adopt best-of-N (BoN) evaluation to more stringently test whether reward models can rank multiple candidates, rather than simply distinguish between two responses. Beyond these general-purpose benchmarks, several specialized benchmarks have been introduced for specific scenarios. These include M-RewardBench for multilingual evaluation, RAG-RewardBench for retrieval-augmented generation, and RewardMath for mathematical reasoning. A similar trend is also visible in multimodal evaluation. Benchmarks such as VL-RewardBench, MultimodalBenchmark, and MJ-Bench extend preference-based reward evaluation to multimodal settings with cross-modal reasoning challenges, typically relying on strong multimodal foundation models to support the evaluation process.

Evaluation Metrics. 

For process-level rubric quality evaluation, model-generated rubrics are typically compared against human-designed ones using metrics such as F1, Recall, N-gram overlap, Hallucination, Missed, and Redundancy [36,70], which capture different aspects of alignment with human rubrics. For outcome-level judgment evaluation, Accuracy is commonly used to measure the agreement between Rubric RMs and human preferences. In addition, some evaluation protocols assess positional bias by examining the consistency of model judgments across repeated evaluations or when the response order is swapped. Beyond direct preference agreement, the capability of Rubric RMs can also be reflected through downstream task performance. For example, PPE evaluates whether Rubric RMs can select the best response from a Best-of-K candidate set on downstream benchmarks such as MMLU-Pro and MATH.

8. Challenges and Future Directions

1. How can we evaluate rubric quality and the faithfulness of rubric-guided judgment under limited human annotation? 

Evaluating rubric quality is itself an open problem for Rubric RMs, especially under limited human supervision [36,70]. In current practice, rubrics are often treated as intermediate content in reasoning, without explicit constraints or independent verification. This increases a typical false-positive risk: a model may reach the correct final judgment even when the rubric it relies on is incomplete, redundant, or otherwise unfaithful. Although prior work has explored various quality dimensions such as discriminability [13];coverage, non-redundancy, and criteria trade-offs [41]; validity [48] and reliability [31] (both of which are defined in education and psychometrics [15]); and value alignment [70,88], there is still no unified definition of rubric quality, nor is there a standardized evaluation framework. Developing a unified framework or designing meta-reward methods remains an important direction for future work. Furthermore, systematic evaluations comparing different rubric construction methodologies are still lacking, and foundational theoretical research regarding the underlying feasibility and mechanisms of rubrics remains extremely scarce [89]. Ultimately, bridging these empirical and theoretical gaps is essential for transitioning Rubric RMs from heuristic tools into rigorous, trustworthy foundations for model alignment.

2. How can common values be identified from human feedback and datasets? 

In many practical secnrios, Rubric RMs rely on fine-grained values to evaluate and improve individual instances. However, instance-specific rubrics are often too local to support robust reuse or transfer. While they may be effective for a particular example, they often fail to generalize across out-of-distribution tasks or datasets, limiting their ability to provide stable supervision at scale. This motivates the need to move beyond instance-level rubrics and identify more persistent dataset-level values across human feedback and data distributions. Recent efforts in this direction, such as the automatic construction of dataset-level rubrics by Xie et al. [41] and Wang and Xiong [59], suggest that common values may be extractable and reusable beyond individual cases. Furthermore, evaluation will extend beyond the single-instance level to dataset analysis. Dataset-level extraction enables researchers to reverse-engineer the implicit human preference distribution from Dataset-level rubric, thereby allowing for a more scientific assessment of the quality and bias inherent in the dataset itself. Nevertheless, an important open question remains: how should such shared values be discovered, abstracted, and organized from human feedback and dataset-level supervision in a principled way?

3. What is the appropriate structured representation of rubrics, and how can such structure be captured or learned effectively? 

Current research has not yet sufficiently addressed how rubrics should be represented as structured objects in complex multi-criteria evaluation. In practice, rubric criteria are rarely fully independent: because it is difficult to construct a completely disentangled rubric. Nevertheless, many existing approaches still implicitly assume that each fine-grained criterion contributes independently to the final judgment. This suggests that an important research direction is to move beyond independent criterion formulations toward more structured representations of rubrics. Rather than treating rubrics as a simple collection of isolated dimensions, future work should explore how to represent and model the relational structure among criteria, including dependency, overlap, priority, and compositionality. Such structured representations are valuable not only for capturing non-linear criterion interactions, but also for improving generalization, interpretability, credit assignment, and context-sensitive evaluation.

4. How should reward models and policies be jointly optimized? 

Current LLM reasoning faces a dual exploration dilemma: sparse solution spaces limit autonomous exploration in out-of-distribution tasks, while the lack of fine-grained guidance hinders capability acquisition in long-horizon reasoning. This suggests that the core challenge is not only exploration, but also guidance. Educational theories such as the Zone of Proximal Development (ZPD) [90] and scaffolding [91] suggest that effective learning requires intermediate support, and rubrics are valuable precisely because they can make such intermediate goals explicit [15,92]. From this perspective, Rubric RMs are better viewed not as static evaluators of response quality, but as structured capability scaffolds that support progressive policy improvement [23,29,68,93,94]. This shifts the focus from post hoc evaluation to joint optimization, where rubric-based rewards serve not only to assess outputs, but also to support capability acquisition during learning.

9. Conclusion

This paper has presented a comprehensive survey of Rubric RMs from four core perspectives: foundations, modeling, reasoning, and their integration into the policy optimization loop. By drawing on established principles from education and psychometrics, Rubric RMs mark an important shift from holistic reward modeling toward more analytic assessment. We have further highlighted key bottlenecks in current research and outlined several promising directions for future work. As the field advances, Rubric RMs may evolve from evaluation tools into foundational mechanisms for guiding reasoning, exploration, and learning in complex, open-ended domains.