Foundation-Model-Assisted Reward Design for Reinforcement Learning: A Review of Reward Program Synthesis, Multimodal Feedback, and Trustworthiness

1. Introduction

Reinforcement learning (RL) enables agents to acquire sequential decision-making policies through repeated interaction with an environment. Unlike supervised learning, RL typically does not provide a correct action for every state. Instead, behavior and its long-term consequences are evaluated through rewards. The reward function therefore plays a dual role. Algorithmically, it provides the numerical signal used for value estimation and policy optimization; from the perspective of task specification, it constitutes a primary interface through which human goals, preferences, and constraints are translated into machine-optimizable objectives [1]. An RL system ultimately optimizes what its reward actually encodes, rather than what a designer intended but failed to formalize.

This interface becomes particularly difficult to engineer in complex tasks. Robotic manipulation may simultaneously involve position, orientation, contact, stability, energy consumption, and safety. Open-world environments often contain long behavioral horizons and many states that are only weakly related to the final objective. Cooperative multi-agent systems introduce additional requirements involving team success, individual contribution, role assignment, and coordination structure. A sparse completion reward may most faithfully represent the final task objective, yet often makes exploration prohibitively difficult. In practice, designers therefore introduce auxiliary terms based on distance, velocity, orientation, collision avoidance, or stage completion. This creates a second problem: as the number of proxy objectives grows, so does the number of ways in which the reward can diverge from the intended task.

Potential-based reward shaping (PBRS) provides an important theoretical foundation for addressing this problem. Ng et al. showed that, under appropriate conditions, shaping based on differences of a state potential preserves the optimal policy of the original task [2]. Wiewiora subsequently established an equivalence between potential-based shaping and a particular form of Q-value initialization [3]. Later work extended the underlying idea to dynamic potentials [4] and episodic settings [5]. These results explain how auxiliary feedback can be introduced without arbitrarily redefining the task objective. They do not, however, solve the more practical reward-engineering problem: how should an appropriate potential, task representation, or set of auxiliary objectives be obtained in the first place?

Traditional reward learning approaches attempt to recover objectives from behavior rather than manually specified formulas. Inverse reinforcement learning (IRL) infers latent rewards from expert policies [6], while apprenticeship learning uses such inferred structure to reproduce expert-like behavior [7]. Maximum-entropy IRL further provides a probabilistic treatment of uncertainty in demonstrations [8]. Preference-based learning reduces dependence on complete expert demonstrations by asking humans to compare trajectory segments and using those comparisons to train reward models [9]. Preferences can also be combined with demonstrations [10]. T-REX recovers rewards from demonstrations of varying and potentially suboptimal quality [11], whereas PEBBLE combines unsupervised pretraining, active preference queries, and experience relabeling to make more efficient use of limited feedback [12].

Human feedback itself has consequently become an independent reward interface. Active preference learning can select behaviors that are expected to be most informative for reward inference [13], while batch preference learning trades information efficiency against the frequency of human interaction [14]. TAMER directly incorporates real-time human evaluative feedback into agent learning [15], and Deep TAMER extends this paradigm to high-dimensional visual states [16]. These methods reduce the need to manually specify complete reward functions, but they do not eliminate supervision cost; moreover, human evaluations remain constrained by observability, cognitive burden, inconsistency, and individual variation.

A deeper difficulty is that an optimizable proxy objective is not necessarily equivalent to the designer's actual objective. Inverse reward design reasons backward from a designed reward to the latent intent of its designer, particularly when the deployment environment differs from the environment in which the reward was specified [17]. Research on AI safety has systematically examined misspecified rewards, negative side effects, safe exploration, and distribution shift [18,19]. Everitt et al. studied incentives for tampering with reward channels through causal influence diagrams [20]. Skalse et al. subsequently formalized reward hacking [21], while more recent work on correlated proxies demonstrates that a proxy reward can be strongly correlated with the intended objective under a reference policy yet lose that relationship after sufficiently strong optimization against the proxy itself [22]. Reward quality must therefore be assessed not only on ordinary trajectories but also after a policy has actively optimized the reward.

Foundation models introduce a qualitatively different source of knowledge into this long-standing problem. Large-scale pretraining produces transferable language, vision, and code capabilities [23]. CLIP established large-scale alignment between natural language and visual representations [24], Flamingo demonstrated multimodal in-context learning [25], and BLIP advanced unified vision–language representation learning [26]. Although these models were not originally trained as RL reward functions, they can interpret natural-language goals, visual scenes, and program structures. Reward design can therefore draw on large amounts of pretrained semantic knowledge rather than relying exclusively on manually designed state features or task-specific labels.

Existing surveys of LLM-enhanced RL usually consider language models in multiple roles—including information processing, reward design, decision making, and generation—such that reward design constitutes only one part of a much broader LLM–RL landscape [27]. Conversely, recent surveys of reward models adopt a broader reward-modeling perspective encompassing IRL, preference learning, proxy rewards, and other mechanisms for obtaining or learning reward signals [28]. There is consequently value in narrowing the unit of analysis to the reward interface between foundation models and external RL agents.

In this review, foundation-model-assisted reward design refers to settings in which outputs from large language models (LLMs), vision–language models (VLMs), video–language models, or related foundation models directly participate in generating, evaluating, learning, selecting, or revising rewards for external RL tasks. Work in which a foundation model is used only as a planner, action generator, or world model without contributing to the reward pathway is not a central focus. Similarly, RLHF, RLAIF, DPO, and related methods whose primary purpose is to train or align the foundation model itself fall outside the main scope.

This article is a structured narrative review. The literature search focused on studies in which foundation-model outputs directly contributed to reward generation, reward evaluation, reward learning, reward selection, or reward revision for external reinforcement-learning agents. Searches combined terms related to foundation models (e.g., “large language model”, “vision-language model”, “foundation model”), reward design (e.g., “reward generation”, “reward function”, “reward shaping”, “reward learning”), and reinforcement learning. Bibliographic databases were complemented by official proceedings repositories, including PMLR, OpenReview/ICLR, NeurIPS, CVF, AAAI, and AAMAS, and the literature was updated through August 2026. Earlier work on reward shaping, inverse reinforcement learning, preference learning, reward machines, and AI safety was included where it provided direct conceptual foundations for the analysis. Studies in which foundation models were used solely for planning, action generation, or policy modeling without contributing to the reward pathway were excluded from the core synthesis, as were RLHF, RLAIF, DPO, and related methods whose principal objective is to train or align the foundation model itself. Publication status was verified where possible through official proceedings or publisher pages; preprints and workshop papers are retained only when they illustrate emerging technical directions and are identified accordingly. Because the reviewed studies differ substantially in task domains, reward scales, policy algorithms, feedback sources, and reward-search budgets, the evidence is synthesized qualitatively rather than through a cross-study meta-analysis.

The contribution of this review lies not in introducing a new reward-learning algorithm, but in providing an analytical structure for a rapidly expanding body of work whose methods, evaluation criteria, and evidential standards remain fragmented. Four contributions are made. First, we define the field through the reward interface between foundation models and external RL agents, rather than through the broader roles that foundation models may play in reinforcement learning. This boundary separates foundation-model-assisted reward design from both conventional reward learning and reinforcement learning used to train foundation models themselves. Second, we organize the literature into three interacting technical pathways—reward program synthesis, multimodal feedback, and feedback-driven reward optimization—and show that these pathways are increasingly converging toward persistent reward-design systems rather than isolated one-shot reward generators. Third, we introduce a five-level trustworthiness framework that distinguishes format validity, execution validity, semantic validity, behavioral validity, and structural assurance. The framework explicitly separates evidence that a reward can be parsed and executed from evidence that it represents the intended task and remains appropriate after policy optimization. Fourth, we propose an end-to-end evaluation perspective in which foundation-model inference, reward search, final-policy training, and human supervision are treated as distinct sources of cost. Together, these contributions shift the unit of analysis from individual generated reward functions toward the complete process by which task intent is converted into reward signals and ultimately into optimized behavior.

These contributions can be understood through a common reward-formation chain. A foundation model first receives task and environmental information, interprets the intended semantics, grounds them to the environment, and produces a reward-related artifact. The resulting reward then influences RL policy optimization, generating new behaviors that may themselves be evaluated and fed back into subsequent reward revision. The object of analysis is thus not merely an isolated reward function, but the entire pathway from task intent to induced behavior, as summarized in Figure 1.

The central change introduced by foundation models is therefore not the mathematical objective of maximizing expected cumulative return. Rather, it is the source of knowledge and degree of automation involved in constructing that return signal. This shift creates a new set of questions: Was the task correctly grounded? Can a generated candidate execute safely? Is feedback reliable? Is the reward-search budget justified? Can the resulting reward knowledge transfer across related tasks? The remainder of this review is organized around these questions.

2. Foundations and Analytical Framework for Reward Design

2.1. From Scalar Rewards to Structured Task Representations

A standard reinforcement-learning task can be represented as a Markov decision process：

$$\mathrm{M}=\left(\mathrm{S},\mathrm{A},\mathrm{P},\mathrm{r},\mathsf{\gamma },{\mathsf{\rho }}_0\right)$$

(1)

where S and A denote the state and action spaces, P is the state-transition distribution, r is the reward function, γ is the discount factor, and ρ₀ is the initial-state distribution. The objective of a policy π is：

$$\mathrm{J}\left(\mathsf{\pi }\right)={\mathbb{E}}_{\mathsf{\pi },\mathrm{P},{\mathsf{\rho }}_0}\left[{\sum }_{\mathrm{t}=0}^{\mathrm{\infty }}{\mathsf{\gamma }}^{\mathrm{t}}{\mathrm{r}}_{\mathrm{t}}\right]$$

(2)

In practical systems, the reward is rarely associated with a single event. Instead, it is commonly constructed as a combination of task-relevant components,

$${\mathrm{r}}_{\mathrm{t}}={\sum }_{\mathrm{k}=1}^{\mathrm{K}}{\mathrm{w}}_{\mathrm{k}}{\mathrm{r}}_{\mathrm{k},\mathrm{t}}$$

(3)

where individual terms may encode task progress, pose, energy consumption, collision avoidance, stability, or safety. This representation is transparent and easy to implement, but shifts considerable responsibility to the designer: appropriate features must be selected, signs must be correct, magnitudes must be calibrated, and conflicts among objectives must be resolved. An auxiliary term that appears reasonable in isolation need not remain beneficial after interaction with the other components.

For tasks with pronounced temporal structure, a single scalar reward is not the only possible representation. Reward machines use finite-state automata to encode non-Markovian rewards and task stages [29], and subsequent work has more systematically characterized how reward-machine structure can support task decomposition, exploitation of reward structure, and transfer [30]. Temporal logic provides another mechanism for translating temporally extended specifications into structured feedback for RL [31]. These approaches share an important property: rather than representing reward as a monolithic and difficult-to-inspect scalar mapping, they expose task-relevant events, states, and transitions as explicit computational structure.

Reward design becomes still more consequential in cooperative multi-agent systems because rewards influence both credit assignment and incentive coordination. Potential-based difference rewards combine team-level shaping with estimates of an individual agent's marginal contribution [32]. Reward machines can likewise decompose team-level objectives among agents while explicitly representing interdependencies among teammates [33]. Multi-agent reward design should therefore not be understood as simply adding an “agent dimension” to single-agent rewards: the resulting local incentives must remain compatible with the collective objective.

Reward representation also affects learning difficulty. Veviurko et al. showed that transformations that are reasonable from the perspective of task optimality can nevertheless exhibit substantially different finite-sample learning behavior [34]. Feature-level preferences decompose holistic trajectory judgments into more interpretable reward-relevant attributes [35], while offline reward-learning formulations can integrate demonstrations and preferences into sets of rewards that satisfy policy-optimality constraints [36]. Meta-Reward-Net jointly optimizes a reward model and policy through bilevel optimization [37], whereas motivation-consistent reward design attempts to construct intrinsic objectives that remain directionally aligned with an external task objective [38].

Two implications are particularly relevant to foundation-model-assisted reward design. First, reward correctness cannot be assessed solely from the final scalar signal: the representation, numerical scale, and subsequent policy-optimization process matter. Second, automated reward design predates foundation models. The distinctive contribution of foundation models is that large-scale language, visual, and code knowledge can now participate directly in reward construction, thereby greatly expanding the space of reward representations that can be produced automatically.

2.2 Visual Representations as Precursors to Multimodal Reward Models

Multimodal foundation-model rewards did not emerge in isolation. Earlier work on robotic visual pretraining investigated whether transferable control representations could be learned from large-scale video. R3M learned general-purpose visual representations from egocentric video and demonstrated transfer across downstream manipulation tasks [39]. VIP went further by encoding temporal relationships in action-free human videos into goal-conditioned value information, allowing the frozen representation itself to provide dense visual rewards for previously unseen robotic tasks [40].

R3M can be viewed primarily as addressing the question of which visual representation is useful for control. VIP begins to address a different question: which visual representation can directly quantify task progress? Later approaches including LIV, RoboCLIP, GVL, and general process reward models follow this second trajectory by attempting to recover progress functions from large-scale visual semantics, temporal structure, or both.

2.3. A Unified View of Foundation-Model-Assisted Rewards

Let x denote a task specification, e the available environment information, D demonstrations, trajectories, or feedback data, and M_θ a foundation model. A reward-related artifact can be represented abstractly as：

$$\mathrm{z}={\mathrm{M}}_{\mathsf{\theta }}(\mathrm{x},\mathrm{e},\mathrm{D})$$

(4)

The output z need not itself be a scalar. It may be a set of reward parameters, executable program code, a reward machine, a visual score, a trajectory preference, a task-completion label, or a language-defined subgoal. To use such a semantic artifact in RL, an additional grounding or compilation process is generally required:

$${\mathrm{r}}_{\mathrm{z}}=\mathrm{C}(\mathrm{z},\mathrm{e})$$

(5)

after which a policy is learned under the resulting reward:

$${\mathsf{\pi }}_{\mathrm{z}}=\mathrm{R}\mathrm{L}\left(\mathrm{e},{\mathrm{r}}_{\mathrm{z}}\right)$$

(6)

This decomposition exposes one of the most important but frequently underemphasized components of foundation-model reward design: the grounding gap between the model's semantic understanding of a task and the reward that can actually be evaluated in an environment. For example, “keep the object stable” is linguistically unambiguous but may correspond computationally to orientation change, contact force, visual motion, or a combination of several signals. Similarly, “cover every target” requires additional decisions regarding entity matching, aggregation over sets, and variations in the number of objects.

If the resulting policy is evaluated and the evaluation is fed back to the model,

$${\mathrm{z}}_{\mathrm{i}+1}={\mathrm{M}}_{\mathsf{\theta }}\left(\mathrm{x},\mathrm{e},{\mathrm{D}}_{\mathrm{i}},\mathrm{E}\mathrm{v}\mathrm{a}\mathrm{l}\left({\mathsf{\pi }}_{\mathrm{i}}\right)\right)$$

(7)

reward design becomes an iterative optimization problem rather than a one-shot generation problem. Eval may include return, success rate, videos, trajectory summaries, preferences, or human comments. Much of the work that followed Eureka can be understood as exploring alternative designs for this feedback loop.

We analyze foundation-model-assisted reward design along three complementary dimensions. The first concerns the reward artifact: does the RL algorithm ultimately receive a program, visual score, preference label, or learned reward model? The second concerns the feedback mechanism: is the reward produced once, manually corrected, revised using policy-training statistics, or optimized through automated search? The third concerns trustworthiness: does the system merely guarantee that the output can be parsed, or does it also verify semantics and policy behavior? Accordingly, the three technical pathways are compared horizontally and five trustworthiness levels vertically in Figure 2.

The purpose of this framework is to separate generative capability from evidential maturity. A model may generate highly sophisticated reward code while providing only compilation-level evidence. Conversely, a relatively simple potential function may enjoy a much stronger policy-preservation guarantee. Expressive reward generation should therefore not be treated as synonymous with methodological maturity.

3. Reward Program Synthesis: From Language Interfaces to Closed-Loop Reward Systems

3.1. From Language-Based Scoring to Executable Reward Programs

One of the earliest ways in which foundation models entered reward design was to use an LLM directly as a proxy reward evaluator. In Reward Design with Language Models, Kwon et al. allowed users to describe desired behavior in natural language and then used a GPT-family model to evaluate agent behavior and output rewards [41]. The appeal of this approach is that goals such as fairness, cooperation, or negotiation preferences can be expressed in language even when they are difficult to encode directly as mathematical functions. At the same time, the approach exposes two early limitations: the foundation model may need to participate frequently in training, and behaviors must be translated into textual representations that the model can interpret.

Language to Rewards adopts a substantially different design philosophy. Instead of allowing the language model to decide the reward at every time step, it defines parameterized motion descriptions and reward interfaces in advance, after which the model maps user instructions to the relevant parameters [42]. Some expressivity is sacrificed, but the resulting environment interface is substantially more stable and the trust boundary is clearer. These approaches illustrate two design choices that continue to characterize the field: using the model itself as an evaluator, or allowing the model to manipulate only a constrained reward space.

Free-form code generation subsequently expanded the expressivity of automated reward design. The Self-Refined LLM Reward Designer generates reward programs from natural-language task descriptions and uses policy-training outcomes for self-refinement [43]. Auto MC-Reward decomposes the system into a Reward Designer, Reward Critic, and Trajectory Analyzer for Minecraft: the critic checks code and task logic, while trajectory analysis identifies behavioral causes of failure [44]. Text2Reward more systematically demonstrated how dense executable reward programs can be generated from language goals and environment interfaces, with further modification through human language feedback [45].

The significance of free-form code is not merely that it saves manual programming effort. It allows a language model to determine which variables should be rewarded and how they should interact, thereby automating reward-structure selection itself. The associated risks, however, also expand from weight tuning to program semantics. A model may produce a syntactically valid, numerically stable function that is physically incorrect—for example, by misinterpreting an axis direction, treating a local observation as global state, or continuing to apply shaping at a terminal state. Such failures cannot be ruled out through execution tests alone.

3.2. Policy Training as a Reward Evaluator

Eureka established the generate–train–feedback–regenerate loop as an influential paradigm for reward design. The system generates multiple reward programs, trains policies under each candidate reward, and feeds reward-component statistics and policy performance back to the language model for further evolution [46]. Its central contribution is not simply the use of better prompting, but the recognition that the properties of a reward are often revealed only after a policy actively optimizes it.

DrEureka extends this principle to sim-to-real transfer. In addition to automated reward design, language models are used to configure domain-randomization ranges based on policy and reward information, improving transfer to physical robots [47]. The study highlights an important point: a reward function is not independent of the environment distribution in which it is optimized. A reward that induces effective behavior under one set of simulated dynamics may not produce the same behavior after the dynamics change.

Closed-loop reward search can nevertheless be expensive. If every candidate requires a full RL run, the reward-design procedure may consume more environment interaction than the final policy training itself. CARD addresses this issue through a Coder and Evaluator that dynamically select among code-level, trajectory-level, and trajectory-preference feedback, reducing the need to repeat full policy optimization at every revision [48]. Self-Alignment adopts a more parameterized design: the LLM proposes reward features and functional forms, while relative rankings among executed trajectories are used to align reward parameters, reducing the complexity of repeatedly rewriting the complete reward structure [49].

Policy training therefore plays at least two roles in foundation-model reward design. One is performance evaluation: does a candidate help a policy learn? The other is semantic diagnosis: why did the policy fail, and what behavior did the reward actually encourage? The former can often be reduced to a scalar score. The latter requires richer information, such as trajectories, videos, or statistics for individual reward components. Feedback becomes more useful for reward improvement as it moves from merely ranking candidates toward diagnosing their behavioral consequences.

3.3. Search Efficiency: From Best-of-N to Stateful Optimization

Once reward generation becomes inexpensive, candidate selection becomes the next bottleneck. The simplest strategy is to generate N rewards, train a policy under each one, and retain the best candidate. Although effective, such Best-of-N procedures can hide substantial cost. If an automated method trains dozens of candidates before selecting one high-performing reward, whereas a hand-designed baseline is trained only once, comparing their final policy-learning curves alone does not constitute a fair comparison of sample efficiency.

ORSO explicitly formulates this problem as online reward selection. Under a fixed interaction budget, training resources are allocated adaptively according to intermediate candidate performance, so clearly inferior rewards need not receive the same budget as promising ones [50]. REvolve introduces human feedback into evolutionary reward search: users evaluate policy behavior, and an LLM uses this feedback to mutate and recombine reward populations [51]. This provides a natural interface for qualitative preferences such as “too aggressive” or “insufficiently smooth,” but it also makes the quality and consistency of human evaluation part of the search process.

R* further separates two reward-search problems of different scales. Reward structure determines which concepts enter the objective, whereas reward parameters determine their relative importance. R* evolves modular reward structures and then aligns their parameters through an LLM critic and preference learning [52]. This separation is often more consistent with practical reward engineering than asking a language model to jointly generate an entire structure and all numerical coefficients.

Automated reward optimization itself does not require foundation models. Lu et al. formulate reward discovery as a bilevel optimization problem in which an upper-level process optimizes the reward and a lower-level process trains the policy [53]. This provides an important control case for LLM-based methods: the distinctive value of language models lies in their semantic knowledge, world knowledge, and code-generation capability, rather than in the mere fact that rewards can be searched automatically.

LaRes improves the efficiency of closed-loop search from the perspective of data reuse. Because candidate rewards share the same environment dynamics, historical transitions can be relabeled under newly generated rewards, and existing policy parameters can be reused rather than restarting every candidate from scratch [54]. RF-Agent instead treats reward design itself as a sequential decision-making problem. It uses a language agent together with Monte Carlo Tree Search to retain and exploit historical candidate information that may otherwise be lost by greedy or simple evolutionary search [55].

Taken together, these studies reveal a clear transition: automated reward design is moving from “an LLM writes a reward function” toward reward-candidate management systems. The generator becomes only one component alongside evaluators, search algorithms, experience buffers, validators, and budget-allocation mechanisms. Stronger foundation models have therefore not eliminated reward engineering; they are transforming hand-crafted reward engineering into system-level reward-search engineering.

3.4. Reintroducing Structure and Theoretical Constraints

As the risks of unrestricted code generation have become more apparent, structured reward representations have again received greater attention. PRM4RL combines high-level task decomposition and progress functions produced with LLM assistance with potential-based reward shaping, yielding dense low-level control rewards with theoretical optimality and convergence guarantees [56]. Compared with unrestricted reward code, this places LLM-derived semantics inside an established RL structure.

ARM-FM translates natural-language task descriptions into reward machines, using formal automata to represent task stages and support compositional transfer [57]. Reward Design Agent (RDA), currently available as a preprint, uses VLM analysis of policy videos to identify specific stages of failure and revise reward code accordingly [58]. Although these studies differ in evidential maturity, they collectively indicate increasing interest in rewards that are structured, diagnosable, and amenable to verification.

The central design trade-off is therefore not whether structured rewards are universally superior to free-form code, but how much freedom a foundation model should have over reward semantics and execution. Fixed templates provide strong constraints but may not express novel task semantics. Free-form programs provide maximal flexibility but are difficult to verify. Typed domain-specific languages, reward machines, and potential-based representations occupy intermediate positions. For deployment-oriented systems, a promising division of responsibilities is to let foundation models propose task semantics and structure, while a trusted execution layer determines how variables are bound, how numerical functions are evaluated, and which operations are permitted. Major technical developments in reward program synthesis are in Table 1.

4. Multimodal Feedback: From Visual Similarity to Process Reward Models

4.1. Foundation Models as Semantic Goal Observers

Programmatic approaches require task semantics to be mapped, at least partially, to computational variables. Multimodal reward approaches relax this requirement by allowing goals to be expressed in forms more natural for humans, including language, images, sketches, and videos.

ZeST uses pretrained foundation models to interpret language, sketches, and goal images and evaluates observations according to their compatibility with a task specification [66]. It established an important alternative source of reward: users can specify desired outcomes multimodally rather than translating the entire objective into internal environment variables.

MineDojo's MineCLIP learns a video–language representation from internet-scale Minecraft videos and associated text, using the alignment between behavioral video and language goals as a reward signal [67]. Compared with single images, video contains temporal information that is more appropriate for activities such as navigation, collection, and crafting. However, internet video–text data are often only weakly aligned, and the precise temporal location of task events is not necessarily labeled.

ELLM uses language knowledge in a different manner: it proposes semantically meaningful exploration goals from the current environment state and provides intrinsic reward when the agent achieves them [68]. Its primary purpose is exploration and pretraining rather than complete replacement of task rewards, but it demonstrates that foundation-model world knowledge can enter RL through decisions about what is worth exploring.

LIV connects language–visual representation learning more directly to task progress. It learns value-like representations from action-free videos paired with language, such that state embeddings aim to capture not only semantic proximity to the goal but also progress toward completion [69]. RoboCLIP uses either text descriptions or a single demonstration video as task specifications and rewards trajectories according to similarity in a video–language representation space [70]. Motif asks an LLM to compare textual descriptions of environment events and distills the resulting AI preferences into an intrinsic reward model suitable for frequent use [71].

The common idea in this stage is that pretrained semantic knowledge can answer a question resembling, “Does the current state look like what the user wants?” The limitation is equally important: visual similarity is not the same as control progress. A robot hand that is merely close to a cup and one that has stably grasped it may be visually similar. A pose can appear correct while containing excessive velocity that causes immediate failure. As applications move from goal recognition toward fine manipulation, surface semantic similarity increasingly becomes insufficient.

4.2. Direct VLM Rewards and Environmental Grounding

VLM-RM directly uses vision–language models as zero-shot reward models, deriving reward from the relationship between natural-language goals and rendered environment observations [72]. Its key advantage is the absence of task-specific reward training data. At the same time, the experiments reveal limitations associated with spatial reasoning, visual-domain shift, and the gap between visible appearance and physical state. A generic VLM may easily recognize that “a cup is on a table” while finding it much harder to distinguish transient contact from stable support or imminent slippage.

Baumli et al. systematically investigated off-the-shelf VLMs as sources of language-conditioned rewards and reported scaling trends between model capability, reward quality, and downstream agent performance in the domains studied [73]. Larger models, however, cannot recover information that is absent from the observation. Increasing model capacity may alleviate failures to interpret an image, but it cannot fundamentally solve cases in which an image does not contain sufficient information to determine control value.

Cachet et al. move this question one level lower: when an environment originally exposes only numerical state, the method used to present that state to a VLM becomes part of reward design. Their use of rendering functions, multiple views, and distillation demonstrates that asking a VLM to evaluate an environment is not a design-free operation [74]. Camera viewpoint, rendering conventions, background appearance, and which objects are visible can all influence the resulting reward.

Multimodal reward design therefore involves at least two grounding steps. The first grounds a linguistic goal into visual concepts. The second grounds the environment's physical state into an observable visual representation. Programmatic rewards can fail through incorrect variable grounding; multimodal rewards can fail through incorrect perceptual grounding. The failure locations differ, but the underlying problem is closely related.

4.3. From Absolute Scores to Preference- and Rating-Based Reward Learning

Foundation models do not always produce well-calibrated absolute reward values. RL-VLM-F therefore asks a VLM to compare two image states according to a language goal and uses the resulting pairwise preferences to train a separate reward function [75]. Expensive VLM inference is concentrated in the preference-labeling stage, while policy optimization can subsequently use a smaller local reward model.

FuRL takes a complementary view by explicitly treating VLM output as a fuzzy reward—a semantically informative prior that remains imperfectly aligned with the control task. Target-domain experience is then used to adapt the representation and reduce the discrepancy between visual semantics and actual task progress [76]. This characterization is more consistent with the available evidence than treating generic VLM scores as ground-truth rewards.

GVL advances the field from static semantic similarity toward temporal progress. Rather than directly asking a VLM to score successive video frames, it reformulates value estimation as temporal ordering over shuffled frames, encouraging the model to use fine-grained semantic and temporal cues to distinguish task progress. The resulting estimator demonstrated zero- and few-shot value prediction over more than 300 real-world tasks across heterogeneous embodiments [77].

VICtoR focuses on long-horizon manipulation. It separates reward modeling into stage detection and motion-progress estimation, allowing the reward to reason about which stage of a task has been reached rather than comparing every observation directly with the final language instruction [78]. This distinction is important in long-horizon tasks because a correct intermediate state may appear visually distant from the final goal while still being a necessary step toward completion.

ERL-VLM revisits absolute rating rather than pairwise comparison. It queries a large VLM for ratings of individual trajectories and develops mechanisms to address imbalance and noise in rating-based reward data [79]. Pairwise preferences and absolute ratings should therefore not be viewed as mutually exclusive alternatives: pairwise judgments are often easier to obtain and calibrate, whereas ratings contain more information but require stronger control of scale and noise.

GoalLadder provides another compromise. Rather than relying on a VLM to produce dense, globally accurate rewards, the model identifies states that represent meaningful progress, and an Elo-style ranking procedure incrementally discovers improved intermediate goals [80]. This illustrates a useful principle for uncertain foundation-model feedback: when precise reward prediction over an entire state space is unreliable, the model can instead be assigned a narrower local comparison problem.

4.4. Toward Process Reward Models

Code as Reward bridges multimodal semantics and programmatic execution. A VLM is used to understand tasks and subtasks, after which the resulting interpretation is converted into reward code that can be evaluated efficiently during RL [81]. This can reduce the need for repeated large-model inference while yielding an explicit executable reward representation. However, an incorrect initial task decomposition can also be compiled into a persistent programmatic error.

ELEMENTAL combines language, visual demonstrations, and inverse reinforcement learning. Language models identify reward-relevant features, visual demonstrations provide evidence of what desirable behavior actually looks like, and IRL estimates the relative importance of those features [82]. The method illustrates a useful division of labor: language is often well suited to expressing what matters, while demonstration provides stronger evidence about how successful behavior is realized.

Reinforcement Learning with Foundation Priors (RLFP) places reward within a broader system of pretrained priors, combining policy, value, and success-reward foundation models to support efficient online exploration and learning in real robotic systems [83]. This suggests that future reward systems may not operate as isolated components; learned reward models may instead interact closely with policy and value priors.

Transfer of reward knowledge across tasks is also becoming an independent research problem. ReWiND learns a language-conditioned reward model from a limited set of demonstrations and subsequently uses the same reward model to adapt policies to new language-specified tasks without requiring new demonstrations for each task; the work was formally published at CoRL 2025 [84]. RoboReward introduces a robotics-specific reward dataset and vision–language reward models trained on large-scale real-robot data. Its negative-example pipeline uses counterfactual relabeling and temporal clipping to construct failures and partial-progress examples; at the time of writing, the work remains a preprint [85]. Large Reward Models similarly specializes foundation VLMs into online reward generators that produce process, completion, and temporal-contrastive feedback, and remains an emerging 2026 preprint [86].

A more mature 2026 result is General Process Reward Modeling for Robotic Reinforcement Learning, presented at CVPR 2026. The method introduces Dopamine-Reward, using large-scale multi-view data to train a step-aware general reward model, and combines the resulting dense progress signal with policy-invariant reward shaping [87]. This directly reconnects two themes reviewed here: foundation models estimate broadly transferable task progress, while classical reward-shaping theory constrains how that progress is injected into RL. Such combinations are a promising direction for future systems.

A similar transition is beginning in multi-agent RL. M³HF uses an LLM to interpret human feedback of mixed quality and modifies multi-agent rewards through predefined reward templates and adaptive weighting [88]. More recent work on LLM-guided incentive-aware reward design generates constrained executable reward programs for cooperative MARL and evaluates them under fixed policy-training budgets, although this work currently remains a preprint [89]. Multi-agent settings impose an additional requirement beyond single-agent reward learning: auxiliary incentives can alter strategic interaction among agents, so reward trustworthiness must also be evaluated at the level of coordination. Major developments in multimodal reward design are in Table 2.

When programmatic and multimodal methods are considered on the same technical timeline, they increasingly appear to converge. Program-centric approaches are moving from unrestricted code toward structured and verifiable reward representations; multimodal approaches are moving from static similarity toward temporal progress and specialized process reward models. Both directions ultimately address the same question: how can a reward representation combine broad semantic coverage with sufficient stability and structure for RL optimization? Figure 3 summarizes this convergence.

Future evaluation should therefore avoid treating “LLM reward-code methods” and “VLM reward methods” as entirely separate categories. More informative comparisons ask what evidence a model ultimately contributes to reward construction, at what level that reward has been validated, and how much interaction, computation, and supervision are required to obtain it.

5. Trustworthiness and Evidence Evaluation

5.1. A Five-Level Trustworthiness Framework

One of the most common interpretive errors in foundation-model reward research is to equate improved policy return with correct reward design. A reward can accelerate learning while inducing undesirable terminal behavior. Conversely, a reward may appear semantically faithful to the task yet be poorly scaled for a particular optimization algorithm. To distinguish these cases, we organize evidence for reward trustworthiness into five levels.

Level 1: Format Validity. The first question is whether a generated program or structure can be parsed correctly by the surrounding system. Programmatic rewards require checks on syntax, variable names, data types, and APIs. Domain-specific languages and reward machines require validation of states, transitions, schemas, and parameter ranges. This level answers only: Can the system interpret the output?

Level 2: Execution Validity. A structurally legal reward must also execute safely and stably on actual environment states. Relevant tests include NaN and Inf detection, division-by-zero checks, numerical overflow, extreme magnitudes, saturation, terminal-state handling, and unintended side effects. Online VLM rewards additionally require handling model timeouts, unavailable services, and missing observations. Execution validity is therefore stronger than format validity, but still does not establish that the reward captures the intended task.

Level 3: Semantic Validity. The next question is whether the reward varies consistently with genuine task progress. Possible evaluations include ranking successful and failed trajectories, constructing counterfactual state pairs, comparing against independent success detectors, and perturbing task-relevant state variables while controlling irrelevant factors. For example, one can change whether an object is actually inside a target region while keeping the visual background approximately unchanged and test whether the reward changes in the correct direction. Such evaluation is particularly important for multimodal rewards because visually salient correlations may be unrelated to task success.

Level 4: Behavioral Validity. A reward must ultimately be tested under optimization. Does a policy that actively maximizes the reward exhibit the desired behavior? Reward hacking, stagnation, cycles, unsafe shortcuts, and conflicts among auxiliary objectives may only become apparent at this level. Correlation between a proxy reward and a task metric on an offline dataset cannot substitute for behavioral validation because policy optimization changes the state distribution and actively searches for high-reward behavior.

Level 5: Structural Assurance. The strongest additional evidence comes from properties that can be audited beyond empirical testing. Examples include policy invariance under fixed potential-based shaping, global reward bounds, legal transitions in reward machines, temporal-logic constraints, or explicit composition with safety mechanisms. Structural assurance does not prove that the chosen semantics are correct, but it can restrict how a misspecified reward is allowed to influence the optimization problem.

The five levels should be interpreted as complementary layers of evidence rather than as a strict logical hierarchy in which evidence at a higher level automatically implies all lower levels. Structural assurance, for example, can establish that a shaping transformation preserves a specified policy objective without establishing that the underlying semantic representation is itself correct. Conversely, a reward may demonstrate strong behavioral validity in a benchmark while lacking any formal guarantee outside the tested distribution. The framework is therefore cumulative in an engineering sense: deployment-oriented reward systems should ideally combine syntactic and runtime checks, semantic tests, behavioral stress tests, and—where applicable—structural guarantees. The value of the framework lies in making explicit which type of evidence a study actually provides, rather than collapsing heterogeneous evidence into a single notion of reward quality.

The framework also prevents qualitatively different notions of “reward correctness” from being conflated. A high compilation-success rate for generated reward programs provides primarily Level-1/Level-2 evidence. Correlation between VLM outputs and human judgments of task progress provides Level-3 evidence. Demonstrating that optimized policies do not exploit reward loopholes under adversarial or stress-test conditions provides Level-4 evidence. Collapsing these measures into a single undifferentiated notion of “reward quality” can conceal important failure modes.

5.2. Major Failure Modes

Programmatic rewards first face environment-grounding errors. An LLM may infer nonexistent variables from similar environments, misunderstand coordinate conventions, interpret local observations as global state, or fail to distinguish natural termination from time-limit truncation. The most dangerous errors are not those that trigger an exception, but those that execute successfully while rewarding the wrong physical quantity.

Reward scale is another frequently underestimated problem. Foundation models can often infer qualitative relationships such as “distance should decrease” or “safety margin should increase,” but are less reliable at determining appropriate numerical magnitudes for a particular RL algorithm. An auxiliary term two orders of magnitude larger than the task reward may effectively redefine the task. A term that is too small may contribute no useful shaping. Programmatic reward systems should therefore examine component distributions, extrema, variance, scale relative to the environment reward, and possible drift in the magnitude of value targets during training.

Conflicts among reward components are still more subtle. Speed, stability, energy efficiency, and safety can all be described linguistically as desirable objectives, yet they necessarily compete in some states. LLMs may tend to translate every positive concept in a task description into an additional reward term, producing unnecessarily complex objectives. Rather than retaining every plausible component, reward-design systems should test marginal contribution and whether removing a component improves the primary task.

The central failure mode of multimodal rewards is the mismatch between visible semantics and control semantics. VLMs are effective at recognizing objects and scenes, whereas RL requires estimating how the current state affects future success. Contact force during grasping, traversability during navigation, and load balance during bimanual manipulation may not be fully observable from a single image. Multimodal reward evaluation should therefore include viewpoint changes, occlusion, background replacement, and counterfactual tests of task-critical physical state.

Preference- and rating-based reward learning can introduce feedback bias. Pairwise comparisons are often easier to obtain but provide only relative ordering. Absolute ratings contain richer information but are more sensitive to prompts and rating scales. More importantly, the learned reward model must extrapolate to states not previously evaluated by the foundation model. Because the RL policy is itself an optimizer that actively searches for high-reward states, such extrapolation error is potentially more consequential than ordinary supervised prediction error.

Feedback-driven search creates a related problem of selection overfitting. If the same environments, random seeds, and short policy-training budgets are repeatedly used to generate and select rewards, the final candidate may become tailored to the evaluation procedure itself. Automated reward design should therefore reserve held-out task configurations, environments, or seeds for final reward evaluation, in the same way that machine-learning pipelines separate model selection from independent testing.

Online reward revision introduces objective non-stationarity. If transitions in an experience buffer retain rewards computed under an earlier version of the objective while the current policy is optimizing a newer one, value learning mixes inconsistent targets. Programmatic rewards can sometimes recompute rewards through relabeling. Human or online VLM feedback may make historical relabeling much more expensive. Dynamic reward systems should therefore explicitly record reward versions and treat reward revision as a change to the learning objective rather than as ordinary hyperparameter tuning. Typical failure modes and recommended validation for foundation-model-assisted rewards are in Table 3.

5.3. Benchmarks Do Not Substitute for Evaluation Design

Much of the current literature on foundation-model-assisted rewards is concentrated in robotic simulation. Meta-World provides multi-task and meta-learning splits and is useful for studying whether rewards transfer among related manipulation tasks [96]. RLBench contains a large set of visually grounded robotic manipulation tasks and naturally supports language- and demonstration-conditioned evaluation [97]. ManiSkill2 provides diverse robots, objects, and manipulation settings, offering a useful platform for studying generalization of visual and programmatic rewards [98].

Reward-code search often requires substantial parallel policy training, making high-throughput GPU simulation systems such as Isaac Gym important infrastructure for automated reward research [99]. The DeepMind Control Suite provides standardized continuous-control tasks with established hand-designed rewards and can serve as a useful comparison platform [100]. D4RL, by contrast, provides fixed offline datasets that can support reward-learning experiments without requiring every reward candidate to collect a new online dataset [101].

Trustworthiness evaluation should additionally incorporate safety and multi-agent benchmarks. Safety Gym separates task objectives from safety costs and can expose cases in which an automatically designed reward improves return by violating safety requirements [102]. PettingZoo provides a standardized API for multi-agent environments [103], while SMAC emphasizes partially observable coordination and credit assignment in cooperative teams [104]. Claims that an LLM-based reward method applies to cooperative MARL are difficult to substantiate through single-agent robotics alone.

Higher-dimensional benchmarks reveal failure modes that are not captured by success rate. HumanoidBench contains whole-body locomotion and manipulation tasks and may reveal unnatural but reward-maximizing behaviors [105]. CALVIN uses natural-language descriptions for long-horizon manipulation and is well suited to evaluating task-stage grounding and language-conditioned rewards [106]. robosuite provides composable robots, objects, controllers, and tasks, enabling analysis of reward transfer across embodiments and configurations [107].

Nevertheless, simply increasing the number of environments does not guarantee stronger evidence. Deep RL results are sensitive to random seeds, implementation details, and hyperparameter choices. Henderson et al. demonstrated how inadequate statistical protocols can yield misleading algorithm comparisons [108]. Agarwal et al. subsequently advocated interval estimation and more robust cross-task aggregation statistics [109]. Patterson et al. provide a broader analysis of experimental design, metrics, and sources of bias in empirical RL research [110].

Foundation-model reward studies should therefore report at least three categories of evidence.

First, policy-level metrics should include final success or return, learning-curve area, interaction required to reach a performance threshold, and safety violations where applicable. Second, reward-level metrics should evaluate trajectory-ranking accuracy, agreement with independent measures of task progress, reward magnitude and variance, sensitivity to counterfactual states, OOD uncertainty, and evidence of reward exploitation. Third, system-level metrics should quantify the number of foundation-model calls, number of reward candidates, RL interactions consumed by reward search, and degree of human involvement.

5.4. Accounting for the Cost of Automated Reward Design

A common source of unfair comparison in automated reward research is to report only the interaction cost of the final policy while omitting the interactions consumed during reward search. A more complete accounting is：

$${\mathrm{C}}_{\mathrm{t}\mathrm{o}\mathrm{t}\mathrm{a}\mathrm{l}}={\mathrm{C}}_{\mathrm{F}\mathrm{M}}+{\mathrm{C}}_{\mathrm{s}\mathrm{c}\mathrm{a}\mathrm{r}\mathrm{c}\mathrm{h}}+{\mathrm{C}}_{\mathrm{p}\mathrm{o}\mathrm{l}\mathrm{i}\mathrm{c}\mathrm{y}}+{\mathrm{C}}_{\mathrm{h}\mathrm{u}\mathrm{m}\mathrm{a}\mathrm{n}}$$

(8)

where ${\mathrm{C}}_{\mathrm{F}\mathrm{M}}$ captures LLM/VLM inference, API, and token cost; ${\mathrm{C}}_{\mathrm{s}\mathrm{c}\mathrm{a}\mathrm{r}\mathrm{c}\mathrm{h}}$ captures the cost devoted specifically to reward candidate generation, validation, selection, and policy probes used only for reward search; ${\mathrm{C}}_{\mathrm{p}\mathrm{o}\mathrm{l}\mathrm{i}\mathrm{c}\mathrm{y}}$ denotes the training cost of the final selected policy after reward selection; and ${\mathrm{C}}_{\mathrm{h}\mathrm{u}\mathrm{m}\mathrm{a}\mathrm{n}}$ includes task description, demonstrations, preference labels, reward inspection, and manual intervention.

This decomposition should be implemented so that the same interaction is not counted in both ${\mathrm{C}}_{\mathrm{s}\mathrm{c}\mathrm{a}\mathrm{r}\mathrm{c}\mathrm{h}}$ and ${\mathrm{C}}_{\mathrm{p}\mathrm{o}\mathrm{l}\mathrm{i}\mathrm{c}\mathrm{y}}$. If a study cannot cleanly separate these quantities, it should at least report the total RL interaction budget and identify which fraction was used for reward selection versus final policy training.

The goal is not to convert all four cost categories into a single monetary unit, but to avoid hiding any of them. Suppose an automated method substantially outperforms a hand-designed reward after 100,000 final-policy steps, but first evaluates 20 reward candidates using 10,000 policy-probe steps each. Its end-to-end interaction cost is at least 300,000 steps, not 100,000. This distinction is especially important in physical robotics.

Claims that foundation models “reduce human reward engineering” also require more precise measurement. If researchers repeatedly modify prompts, inspect policy videos, and manually remove implausible reward candidates, human effort has not disappeared; it has shifted from writing Python reward functions to supervising a generative reward-design system. High-quality studies should document such intervention.

5.5. Minimum Reporting Requirements

Foundation-model-assisted reward design introduces reproducibility variables that are uncommon in conventional RL. Commercial models can change over time, and identical model names may correspond to different serving versions. Temperature, system prompts, candidate count, and the way environment code is presented can all change generated reward functions.

A complete study should therefore disclose the foundation model and precise version where possible, date of use, sampling parameters, prompts, environment information exposed to the model, and the final reward artifact. Program-generation studies should additionally report how many candidates were generated, how many failed to execute, how many were repaired, and why the final reward was selected. Reporting only successful reward code while omitting the generation failure rate can substantially overstate reliability.

Randomness should also be separated into reward-generation randomness and policy-training randomness. Generating one reward once and then training ten policy seeds only demonstrates robustness of policy optimization under that particular reward. It does not demonstrate that repeated calls to the LLM reliably produce rewards of comparable quality.

Evaluation should therefore not collapse to a single final-return curve. Policy effectiveness, reward trustworthiness, automation cost, and transfer jointly constitute the evidence required to assess foundation-model-assisted reward design. Figure 5 presents an end-to-end evaluation framework.

This framework permits programmatic and multimodal methods to be compared within a common logic. Programmatic methods may avoid frequent foundation-model inference but incur substantial candidate-training cost. Direct VLM rewards can eliminate program search while incurring online inference cost. Learned reward models require additional labeling and reward-model training up front but may substantially reduce deployment-time inference. Meaningful comparison should therefore cover the complete reward-design lifecycle rather than only the final segment of policy training.

6. Challenges and Research Directions

6.1. From Free-Form Code Generation to Verifiable Reward Compilation

Free-form reward code is an effective research instrument for testing the capabilities of foundation models, but it is unlikely to be the final architecture for every practical system. Directly executing model-generated Python assigns task interpretation, variable selection, numerical scaling, and execution privileges to the same probabilistic component. Such a trust boundary is difficult to justify in safety-critical applications.

A more robust architecture would place the foundation model primarily at the semantic layer. The model could propose which task relations or progress concepts should matter, a typed intermediate representation could restrict those concepts to an auditable reward language, and a trusted compiler could perform state binding and numerical execution. Related work on intent-constrained LLM planning and minimal-edit repair illustrates a broader engineering pattern in which generative models operate through explicit, auditable structures rather than unrestricted output spaces [111].

The central research question is the trade-off between expressivity and assurance. A domain-specific language that is too small merely allows the model to rearrange reward primitives defined by humans; one that is too general approaches the verification difficulty of unrestricted programming. Object relations, typed sets, role-specific predicates, temporal events, bounded transforms, and verified aggregation operators may provide a useful intermediate design space.

6.2. From Visual-Semantic Correlation toward Causally Informed Task Progress

Vision–language models are effective at answering whether an image is semantically related to a language goal. RL requires a different quantity: whether the current state, and potentially the current action, improves the probability of future success.

Future process reward models should therefore represent temporal order, object interaction, and recoverability more explicitly. World models may provide predictions of action consequences; multi-view observations can reduce single-camera ambiguity; object-centric representations can reduce dependence on background correlations. Recent process reward models already demonstrate the importance of step awareness and multi-view information, but a further step is to incorporate action-conditioned and causally informative notions of progress.

Uncertainty and abstention are equally important. If a reward model is forced to output a confident scalar under occlusion, unfamiliar objects, or clearly out-of-distribution observations, policy optimization may eventually exploit that uncertainty. A trustworthy reward system should be allowed to report insufficient evidence, reduce shaping strength, request another observation, or fall back to the environment's original reward.

6.3. From Per-Task Reward Search to Transferable Reward Knowledge

Many current systems described as “automated reward design” effectively replace human trial-and-error with machine trial-and-error. When a new task arrives, the system still regenerates programs, trains multiple candidates, and selects a reward from scratch. This can save expert engineering time for a single task, but repeated search remains expensive when many related tasks must be solved.

A stronger research objective is to transfer previously acquired reward semantics to new tasks. Programmatic methods require a transition from concrete environment-variable names toward relational representations. Multimodal reward systems require progress models that generalize beyond the tasks used to construct them. Structured reward representations need to support composition and re-instantiation under new task configurations.

This challenge is particularly important in multi-agent settings. The same cooperative semantics may need to apply across different numbers of agents, roles, and objects. Rewards tied to identifiers are intrinsically difficult to reuse. Rewards expressed in terms of relations between entity types—for example, assignment, coverage, communication, or synchronization—are more naturally compatible with permutation and population-size generalization.

6.4. Multi-Agent Reward Design Requires Incentive-Level Validation

Foundation-model-assisted reward design in MARL cannot be validated solely through team-average return [112]. A dense auxiliary reward for individual agents may improve short-term learning while simultaneously changing the strategic incentives among agents and potentially inducing equilibria that differ from the intended team objective.

Multi-agent automated reward systems should therefore answer at least three additional questions. First, does the reward assign individual credit in a way that is compatible with true contribution? Second, does the reward remain aligned with the team objective under different combinations of agent policies? Third, does its scale remain meaningful as the number and roles of agents change?

Language models may be particularly useful for recognizing roles and cooperative semantics because natural language is well suited to expressing who should do what. However, the transition from semantic role descriptions to numerical incentives still requires formal constraints. Combining foundation models with difference rewards, reward machines, game-theoretic analysis, or relational potential functions is therefore likely to be more robust than allowing an LLM to directly write unconstrained team-reward code.

6.5. From Peak Performance to Reliability under Budget Constraints

Future automated-reward benchmarks should impose total design budgets rather than allowing each method to generate and train an arbitrary number of candidates. Under fixed budgets for foundation-model calls, environment interactions, and human feedback, the key question becomes how reliably a method can discover a useful reward.

Benchmarks should also intentionally contain reward traps: states that appear visually close to success while being physically incorrect, behaviors that can obtain high proxy reward by stagnating, and situations in which safety objectives conflict with rapid task completion. A method that exceeds a hand-designed baseline only on tasks whose rewards are easy to specify provides limited evidence that it has solved the reward-design problem.

Budget constraints are even more important in physical robotics. Every unsuccessful rollout consumes time, hardware lifetime, and potentially safety margin. Practical reward systems must eventually move away from a “search until something works” engineering paradigm toward one that rejects invalid candidates before execution, quantifies uncertainty, and degrades safely when reliable reward guidance is unavailable.

6.6. Limitations of This Review

This review has several limitations. First, foundation-model-assisted reward design is evolving rapidly, particularly in robotics and multimodal reinforcement learning, and some 2026 results remain available only as preprints. Publication status and method categorization may therefore change after the literature cutoff. Second, terminology across the field is not yet standardized: closely related systems may describe their outputs as rewards, value functions, progress estimators, preference models, success detectors, or critics, making strict boundaries inherently imperfect. Third, the reviewed studies are highly heterogeneous in environment, policy algorithm, foundation model, interaction budget, and evaluation protocol. Reported numerical improvements should therefore not be interpreted as directly comparable effect sizes across papers. Finally, negative results, failed reward candidates, and human engineering effort are reported inconsistently in the literature, which may bias the available evidence toward successful demonstrations. These limitations further motivate standardized reporting and benchmark protocols for future reward-design research.

7. Conclusions

Foundation models are changing how rewards for reinforcement learning are constructed. Traditionally, rewards were derived primarily from manually written functions, demonstrations, or direct human feedback. Today, LLMs can translate language and environment interfaces into reward parameters, structured specifications, and executable programs; vision–language models can estimate task progress from images and videos; and policy learning itself can become a source of feedback for revising reward definitions.

This increased automation does not remove the reward problem. Instead, it expands the problem from how to write a reward function to how to ground model semantics correctly, how to validate automatically generated rewards, how to prevent policies from exploiting model error, and how to account fairly for reward-design cost.

Accordingly, this review organizes foundation-model-assisted reward design into three interacting directions: reward program synthesis, multimodal feedback, and feedback-driven optimization. It further introduces five complementary levels of trustworthiness—format validity, execution validity, semantic validity, behavioral validity, and structural assurance. The three technical pathways describe how rewards are obtained, whereas the five trustworthiness levels describe what evidence supports their correctness. The end-to-end cost framework additionally emphasizes that automated reward systems cannot be evaluated solely by the return of the final policy; foundation-model inference, candidate search, policy training, and human intervention must all be considered.

The current technical trajectory suggests convergence between the major paradigms. Programmatic approaches are moving from unrestricted code toward structured and verifiable reward representations. Multimodal methods are moving from static semantic similarity toward temporal progress estimation and general-purpose process reward models. The most promising future systems may therefore rely less on a single larger model taking complete control of reward design and more on a division of responsibilities: foundation models interpret tasks and complex observations; structured representations and trusted execution layers constrain computation; optimized policy behavior provides semantic stress tests; and formal constraints or safe fallback mechanisms limit the consequences of reward error.

The central question for the next stage of the field is thus no longer simply whether a foundation model can generate a reward that achieves high return. More consequential questions are whether the resulting reward can be verified, whether the full cost of obtaining it is transparent, and whether the task knowledge encoded in that reward can be reused across tasks, embodiments, and agent populations. Answering these questions will determine whether foundation-model-assisted reward design develops from an attractive automation technique into reusable, auditable, and trustworthy infrastructure for reinforcement learning.