Submitted:
16 October 2026
Posted:
20 October 2026
You are already at the latest version
Abstract
Crafting reward functions remains a challenge in reinforcement learning, particularly in autonomous driving (AD), where safety, efficiency, and task completion must be balanced. We introduce a GPT-5.6-based revision of AD Reward Designer, a framework for generating interpretable dense reward functions from natural language goals and concise Pythonic environment descriptions. The initial reward generation requires no expert demonstrations or model fine-tuning. GPT-5.6 translates task requirements into executable reward code, which is checked before being used by a reinforcement learning algorithm. Policy rollout summaries and human feedback then guide iterative refinement of the reward terms and their weights. An optional demonstration-conditioned extension supplies expert and policy trajectories as additional context for reward design. We retain Highway-env and MetaDrive as the evaluation environments and PPO as the policy learning algorithm. The original experiments provide evidence for the feasibility of language-model-generated rewards; their numerical results are retained as historical evidence and are not reported as GPT-5.6 measurements. The revised GPT-5.6 configuration requires a new evaluation under the same task settings.
Keywords:
artificial intelligence
; reinforcement learning
; machine learning
; autonomous driving
1. Introduction
The reward signal stands as the pivotal information guiding agent learning in Reinforcement Learning (RL) [1,2]. Yet, despite its fundamental significance, crafting high-performing reward functions remains notably challenging [3]. On one hand, articulating human notions of desired behavior via designed reward functions or extensive expert demonstrations proves to be a daunting task. On the other hand, dense rewards, crucial for providing precise learning signals, necessitate either manually decomposing overarching goals into sub-goals or incentivizing auxiliary objectives of interest [4]. However, both of these approaches are hindered by their dependency on expert input and meticulous manual refinement. Harnessing the wealth of pre-trained common-sense knowledge, code generation capabilities, and contextual learning aptitude [5,6,7,8,9,10,11,12,13], LLMs possess the potential to design or refine reward functions for DRL. We achieve this by leveraging natural language instructions and environmental cues to make LLM assist us in designing dense reward functions for the AD realm in auxiliary.
In the realm of autonomous driving [14,15,16,17,18], enhancing the performance of intelligent agents through reinforcement learning algorithms necessitates a well-crafted reward function. Shen and Zhang [19] propose MFTFormer, combining time-frequency decomposition with weather-aware block-aligned fusion to address temporal complexity and meteorological influences in traffic flow prediction. As highlighted in [20], the efficacy of policy learning is heavily contingent upon the design of this function. However, despite its paramount importance, designing reward functions is widely acknowledged as a formidable task [1,21], often requiring a profound understanding of the task and iterative experimentation. Before the advent of LLMs such as GPT or Claude [22,23,24,25], an algorithm called RLprompt [26] emerged as a solution, leveraging GPT’s capabilities to serve as a designer for sparse reward functions. Nevertheless, the advent of LLMs has introduced promising potential, particularly evident in areas like logical reasoning and knowledge acquisition. Moreover, LLMs have demonstrated their capacity to assist in code-writing tasks [27]. Therefore, employing an LLM as a generator for dense reward functions stands as a plausible approach, benefiting from its versatile capabilities.
We achieve this by leveraging natural language instructions and environmental cues [28,29]. AD Reward Designer generates dense reward code from goal descriptions. In the revised framework, GPT-5.6 is the reward generation and refinement model: given an RL goal (e.g., "Reach the destination safely and on time") and a concise Pythonic environment description, it produces reward code for policy training with PPO. The initial generation requires no expert demonstrations, and the symbolic reward terms remain inspectable. To reduce errors caused by ambiguous goals or unavailable simulator APIs, the generated code is checked before policy training. Subsequent prompts combine the current reward code, policy rollout summaries, and human feedback to refine the reward function. We retain two AD environments: Highway-env [30] and MetaDrive [31]. Highway-env includes highway, merge, intersection, and roundabout scenarios, with IDM (Intelligent Driver Model) and MOBIL (Minimizing Overall Braking Induced by Lane Changes) governing background traffic. MetaDrive provides compositional driving scenarios in which the controlled vehicle must reach its destination safely. We also consider an optional extension in which expert and policy trajectories are supplied to GPT-5.6 as context for identifying useful reward terms, without fine-tuning the language model or claiming recovery of a unique underlying reward function.
2. Related Work
Language Models in Reinforcement Learning
With the emergence of advanced capabilities, the potential of Large Language Models (LLMs) to tackle the inherent challenges of Reinforcement Learning (RL) has recently gained traction [32]. The versatile capabilities of LLMs, particularly in natural language understanding, reasoning, and task planning, offer a unique approach to addressing the RL issues mentioned above. For sample inefficiency, [33] proposes a framework wherein LLMs are employed to enhance the sample efficiency of RL agents by providing rich, contextually informed predictions or suggestions, thereby reducing the necessity for extensive interactions with the environment. Regarding reward function design, LLMs can assist in crafting more nuanced and effective reward functions, thereby enhancing the learning process by providing a deeper understanding of complex scenarios [34]. To tackle generalization challenges, [35] introduces a framework that leverages language-based feedback to enhance the generalization of RL policies in unseen environments. For natural language understanding, Pang et al. [36] utilize LLMs to translate complex natural language-based instructions into simple task-specified languages for RL agents. Collectively, these studies demonstrate that LLMs play a promising and powerful role in addressing longstanding challenges in RL.
Explicit Reward Model constructed with LLM
The current focal point of research in this domain predominantly revolves around robotics. Motivated by the self-refinement capabilities of Large Language Models (LLMs) [37,38] introduces a framework featuring a self-refinement mechanism for automated reward function design, encompassing initial design, evaluation, and a self-refinement loop. Their findings suggest that reward functions designed by LLMs can either match or surpass manually crafted ones. Similarly, Eureka [39] devises a reward optimization algorithm incorporating self-reflection. This algorithm utilizes environment source code and task descriptions to sample reward function candidates from a coding LLM, evaluate them via RL simulation, and iteratively enhance the reward outputs based on the results. Their approach demonstrates human-level performance in reward design and successfully tackles dexterous manipulation tasks previously deemed infeasible through manual reward engineering. Another notable contribution, Text2Reward [28], generates finely tailored dense reward functions as executable programs based on environment descriptions. Given the sensitivity of RL training and the inherent ambiguity of language, RL policies may fall short of achieving predefined goals. Text2Reward addresses this challenge by executing the learned policy in the environment, soliciting human feedback, and refining the reward function accordingly.
Building on these reward generation and refinement methods, our revision uses GPT-5.6 as a common backend for initial reward design and feedback-conditioned updates. The main changes concern model selection, explicit simulator interfaces, and execution checks; the driving tasks and PPO training framework are retained.
3. Method
3.1. Evaluation
We retain decision-making tasks in Highway-env and MetaDrive, including highways, intersections, merges, curves, and roundabouts. The proposed GPT-5.6 evaluation compares driving velocity, crash rate, and task success rate with those of policies trained using human-written rewards. Code executability is assessed before RL training. Learning curves are used to examine convergence within a fixed reward function; cumulative rewards from different reward definitions are not directly treated as comparable performance scores.
3.2. Framework
We propose the AD Reward Designer framework for reward shaping in Figure 1. AD Reward Designer takes a goal instruction and an environment description as input and generates a dense reward function for an RL algorithm. The revised framework uses GPT-5.6 as its reward generation and refinement model. We retain PPO [40] and the hyperparameters used with human-written rewards so that changing the language model does not also change the policy learning algorithm. The initial reward generation is zero-shot: the prompt specifies the task and simulator interface but includes no example reward solutions or expert demonstrations. A goal may be provided by a user or arise as a subgoal of a longer task.
To ground reward generation in the simulator, we use a concise Pythonic representation with class definitions, type annotations, and comments [41]. The description specifies available state variables, callable functions, units, and the required reward function signature. GPT-5.6 is instructed to use only these interfaces and to balance collision avoidance, progress toward the goal, driving efficiency, and smooth control when the corresponding quantities are available. It returns reward code and a brief explanation of the reward terms. This preserves the original interpretable code-based design while making the generation requirements explicit.
Before PPO training, the proposed implementation checks syntax, compatibility with the specified simulator interface, and whether the reward returns a finite scalar on sampled transitions. If a check fails, the code and error message are returned to GPT-5.6 for repair within a fixed, recorded repair budget. Passing these checks establishes basic executability; policy rollouts are still required to assess the driving behavior induced by the reward.
To address under-specified instructions, AD Reward Designer incorporates human feedback after each policy training cycle. Users inspect rollout trajectories and summaries of velocity, collisions, and task completion, then describe the desired improvement in natural language. In the revised loop, GPT-5.6 receives the current reward code, these rollout summaries, and the user feedback together. For example, "Maintain a larger gap to the leading vehicle" can motivate a safety-margin term, while "Reach the destination more efficiently" can motivate a progress term. The revised code is checked again before the next PPO training cycle. This approach allows users to guide reward refinement without directly writing reward code, while retaining simulator validation and policy evaluation as necessary checks.
4. Experiment Setting
The proposed GPT-5.6 evaluation retains the original four Highway-env scenarios: highway, intersection, merge, and roundabout, described in Appendix D. Traffic density is 0.1 and the seed is 0. For MetaDrive, the retained scenarios are curve, intersection, T-intersection, and roundabout, with traffic density 0.3, accident probability 0.1, and seed 0. The original single-seed configuration supports a matched comparison, but additional seeds are needed to assess variability. The reward refinement process retains five iterations, incorporating policy rollout summaries and human feedback at each iteration. PPO hyperparameters remain those listed in Appendix A. The training and evaluation budgets must be identical for policies using human-written and GPT-5.6-generated rewards.
GPT-5.6 configuration
The proposed model configuration is gpt-5.6-sol, referred to as GPT-5.6 throughout the revised method, with reasoning.effort=medium and standard reasoning mode. OpenAI documents gpt-5.6 as an alias of this model.1 The same configuration is used for generation, code repair, and reward refinement. The resolved model version, prompt, reasoning settings, and repair budget should be recorded for reproducibility. These are proposed settings, not a claim that the retained experiments have been rerun with GPT-5.6.
Experiment Metric
We compare velocity, crash rate, and task success rate in each applicable scenario. In MetaDrive, success denotes reaching the destination. Reward learning curves are used to inspect training progress within a run, and code validation records whether generated rewards can execute through the specified interface. These measures distinguish the correctness of reward code from the driving performance of the trained policy.
5. Result Analysis
Scope of reported results
The tables, learning curves, and reward examples below are retained from the original GPT-4-based implementation. They provide historical evidence for AD Reward Designer and do not establish the performance of GPT-5.6. A new evaluation using the configuration above is required before making GPT-5.6 performance claims.
5.1. Quantitative Evaluations
5.1.1. Highway
Figure 2 presents cumulative reward curves from the original GPT-4-based implementation. The reward functions differ across scenarios, so their reward magnitudes are not directly comparable. The upward trends suggest that PPO can optimize the corresponding rewards in the evaluated scenarios; driving performance is assessed separately using the metrics in Table 1.
Table 1 compares human-written rewards with the original AD Reward Designer across highway, intersection, merge, and roundabout scenarios. It reports velocity, crash rate, and success rate. In the reported runs, the generated rewards yield higher velocities and comparable or lower crash rates. These observations support the feasibility of automated dense reward design under the original settings; they do not establish that the GPT-5.6 revision will reproduce or improve these values.
An illustrative reward function from the original highway experiment is:

You can see other generated reward functions in other scenarios in Appendix B.
5.1.2. Metadrive
The original implementation also tested generated rewards in MetaDrive [31,42] using PPO. Figure 3 shows the corresponding learning curves. Table 2 compares human-written rewards with the original AD Reward Designer in curve, intersection, roundabout, and T-intersection scenarios, using velocity, crash rate, and success rate. The retained results concern the original implementation; the GPT-5.6 revision requires a matched evaluation.
- 1.
- Velocity: In each scenario, the velocities achieved with the AD Reward Designer are also generally higher than those with the human-written method. Notably, in the T-intersection scenario, there is a significant increase from 20.87 to 27.51. This suggests that the AD Reward Designer method might be optimizing speed performance more effectively.
- 2.
- Crash Rate: The crash rate is 0% across all scenarios, indicating that both the human-written and AD Reward Designer methods achieve high safety without any collisions.
- 3.
- Success Rate: The success rate is consistently 100% across all scenarios, demonstrating that both methods are highly reliable in completing tasks.
Table 2 shows that the AD Reward Designer excels in increasing driving speeds while maintaining high safety and task success rates. This suggests that the AD Reward Designer can design dense reward functions that perform consistently with human expert performance. You can see the generated reward function in all scenarios in Appendix B.
5.1.2.1. Discussion
Reward design must balance safety and efficiency. Strong collision penalties may induce conservative policies, while excessive speed rewards may encourage unsafe behavior. In the proposed GPT-5.6 loop, rollout metrics and human feedback are used to identify these trade-offs and revise individual reward terms. Model replacement alone does not establish an improvement in reward quality; execution checks and matched policy evaluations remain necessary.
5.2. Case Study
The original highway case study supplied expert demonstrations and policy trajectories to GPT-4 in each iteration, replacing explicit human feedback with trajectory context. For the GPT-5.6 revision, we retain this optional demonstration-conditioned reward design setting: GPT-5.6 compares expert and policy behavior to suggest reward terms and weights, using documented state variables and consistent units. The demonstrations are prompt context rather than fine-tuning data. This setting is inspired by inverse reinforcement learning, but it does not by itself recover or validate a unique expert reward function. The following excerpt is retained from the original demonstration format:


Figure 4 retains the learning curve from the original demonstration-conditioned case study. The curve provides a diagnostic of reward optimization in that run, rather than a direct comparison of reward quality across different reward definitions. The original reward code is provided in Appendix C. The same trajectory-conditioned design must be evaluated separately with GPT-5.6 using driving performance metrics.
6. Conclusion
We introduce a GPT-5.6-based revision of AD Reward Designer for generating interpretable dense reward code in autonomous driving. The revision retains the natural language goal interface, Pythonic environment descriptions, PPO training, and human feedback loop, while specifying GPT-5.6 for reward generation and refinement and adding code execution checks. Rollout summaries provide task-specific evidence for adjusting reward terms, and expert trajectories can optionally serve as additional prompt context.
The retained Highway-env and MetaDrive results support the feasibility of the original language-model-based framework. They are not GPT-5.6 measurements. The next empirical step is a matched evaluation of the revised configuration with the same environments, PPO settings, and training budgets, followed by additional seeds to assess variability. This evaluation is needed to determine whether GPT-5.6 improves reward executability, refinement efficiency, or driving performance.
Appendix A. Parameters of PPO Algorithm
Table A1.
Hyper-parameter of PPO algorithm applied to each task.
| Hyper-parameter | Value |
|---|---|
| Discount factor | 0.99 |
| # of epochs per update | 10 |
| Learning rate | |
| # of environments | 4 |
| Batch size | 64 |
| Hidden units per layer | 256 |
| Target KL divergence | 0.02 / None |
| # of layers | 2 |
| # of steps per update | 2048 |
| Rollout steps per episode | 128 / 64 |
Appendix B. Generated Reward Function in Other Scenarios
The human-written reward function in highway scenario of the Highway-Env is:






Appendix C. Generated Reward Function in Case Study

Appendix D. Environment Description
Appendix D.0.0.2. Highway
seen in Figure A1, is a collection of environments for autonomous driving and tactical decision-making tasks. In these tasks, the ego-vehicle is driving on multilane roads populated with other vehicles. The agent’s objective is to reach a high speed while avoiding collisions with neighboring vehicles. Driving on the right side of the road is also rewarded. In different scenarios, there may also be specific rewards. The state space includes the car’s speed, displacement, information about other vehicles, and map information. The action space is divided into two types: discrete and continuous. The discrete action space includes orientation and acceleration or deceleration, while the continuous action space includes acceleration and steering angle.
Figure A1.
These are maps of highway.

Appendix D.0.0.3. MetaDrive
seen in Figure A2, is a driving simulator with the following key features: Compositional: It supports generating infinite scenes with various road maps and traffic settings for the research of generalizable RL. Lightweight: It is easy to install and run. It can run up to 300 FPS on a standard PC. Realistic: Accurate physics simulation and multiple sensory input including Lidar, RGB images, top-down semantic map, and first-person view images. The TopDownMetaDrive is a wrapper class on MetaDriveEnv which overrides observation to pygame top-down renderer. The native observation of this setting is a numpy array with shape [84, 84, 5] and all entries fall into [0, 1]. The above figure shows the semantic meaning of each channel. The action space is composed of acceleration, steering angle, and brake signal.
Figure A2.
These are maps of Metadrive.

References
- Sutton, R.S.; Barto, A.G. Reinforcement Learning: An Introduction. IEEE Trans. Neural Netw. 1998, 9, 1054–1054. [Google Scholar] [CrossRef]
- Gu, X.; Qiao, G.; Jiang, C. Mimicking human intuition: Cognitive belief-driven reinforcement learning. In Proceedings of the 2nd Workshop on Models of Human Feedback for AI Alignment, 2024. [Google Scholar]
- Koohy, B.; Stein, S.; Gerding, E.H.; Manla, G. Reward Function Design in Multi-Agent Reinforcement Learning for Traffic Signal Control. In Proceedings of the Twelfth International Workshop on Agents in Traffic and Transportation co-located with the International Joint Conference on Artificial Intelligence and the European Conference on Artificial Intelligence (IJCAI-ECAI), 2022. [Google Scholar]
- Andreas, J.; Klein, D.; Levine, S. Modular Multitask Reinforcement Learning with Policy Sketches. In Proceedings of the Proceedings of the International Conference on Machine Learning, (ICML), 2017. [Google Scholar]
- Li, S.; Zeng, H.; Luo, X.; Jia, F.; Li, F.; Geng, Z.; Kolter, Z.; Lee, T.S.; Li, T. Witeness Overlap: Directional Provenance Inside Open-Weight Model Families. In Proceedings of the Advances in Neural Information Processing Systems, 2026; p. 2609.31784. [Google Scholar]
- Wang, Y.; Sun, X.; Li, Y.; Fan, Z.; Zhuang, Z. Embedding Foundation Model Predictions in Discrete-Choice Models with Structural Guarantees, 2026; NeurIPS.
- Wang, Y.; Sun, X.; Kong, L.; Gao, W.; Li, Y.; Fan, Z.; Zhuang, Z. Do Time Series Foundation Model Benchmarks Hide Regime-Dependent Failures? Evidence from Traffic Speed Forecasting, 2026. ICML Workshop on Forecast.
- Fan, Z.; Li, Y.; Zhuang, Z.; Sun, X.; Wang, Y. The Moving Target: A Longitudinal Audit of Trust-Benchmark Score Drift Across Open-Source Chat LLM Release Lines, 2026. IEEE International Conference on Machine Learning and Applications (ICMLA).
- Xiao, L.; Bhattacharyya, A.; Kamalzadeh, M.; Gudepu, V.; Horner, L.J.; Fumagalli, A. Opening the Black Box: Zero-Touch Conformance and Security Testing of Commercial O-RUs. In Proceedings of the Proceedings of the 20th ACM Workshop on Wireless Network Testbeds, Experimental Evaluation & Characterization, New York, NY, USA, 2026; WiNTECH ’26, pp. 97–104. [Google Scholar] [CrossRef]
- Xiao, L.; Bhattacharyya, A.; Latha, A.G.; Gudepu, V.; Fumagalli, A. Robotic Process Automation Testing and Validation Platform for Open-source 5G Stack. In Proceedings of the 2025 IEEE 30th International Workshop on Computer Aided Modeling and Design of Communication Links and Networks (CAMAD); IEEE, 2025; pp. 1–2. [Google Scholar]
- Xiao, L.; Latha, A.G.; Bathula, B.; Zhang, T.; Fumagalli, A. Optical network robotic automation platform for multi-vendor optical transport network: accelerating validation, troubleshooting, and proof of concept efforts. In Proceedings of the 2025 International Conference on Optical Network Design and Modeling (ONDM); IEEE, 2025; pp. 1–6. [Google Scholar]
- Xiao, L.; Gudepu, V.; Fumagalli, A. Joint Alarm Correlation and Root Alarm Detection in Optical Transport Networks with Graph-Attention. In Proceedings of the 2025 IEEE 30th International Workshop on Computer Aided Modeling and Design of Communication Links and Networks (CAMAD); IEEE, 2025; pp. 1–6. [Google Scholar]
- Xiao, L.; Latha, A.G.; Gudepu, V.; Fumagalli, A. A YANG-Grounded LLM Agent Supporting Multi-Vendor OpenROADM Optical Transport Networks. In Proceedings of the Proceedings of the ACM SIGCOMM 2026 Conference, 2026; pp. 2109–2115. [Google Scholar]
- Chu, Z.; Qiu, J.; Xu, J.; Zhang, Z.; Zhao, H.; Liu, H. A Collision-Aware Mobile Robot Navigation in Grid Environment using Improved A*. In Proceedings of the 2026 12th International Conference on Control, Automation and Robotics (ICCAR), 2026; pp. 96–101. [Google Scholar] [CrossRef]
- Luo, Y.; Xu, J.; Chen, Y.; Yang, Y.; Zheng, T.; Liu, H. Optimal Hovering Control Strategy for UAVs Using PPO. In In Proceedings of the 2025 7th International Conference on Machine Learning, Big Data and Business Intelligence (MLBDBI), 2025; pp. 247–251. [Google Scholar] [CrossRef]
- Fan, E.; Chen, Y.; Shan, Z.; Nahrstedt, K.; Caesar, M.; Kim, J. Communication-Aware Multi-Agent Reinforcement Learning for Decentralized Cooperative UAV Deployment, 2026. arXiv arXiv:cs.
- Qiao, G.; Quan, G.; Yu, J.; Jia, S.; Liu, G. Trafficgamer: Reliable and flexible traffic simulation for safety-critical scenarios with game-theoretic oracles. arXiv 2024, arXiv:2408.15538. [Google Scholar]
- Qiao, G.; Quan, G.; Qu, R.; Liu, G. Modelling competitive behaviors in autonomous driving under generative world model. In Proceedings of the European Conference on Computer Vision, 2024; Springer; pp. 19–36. [Google Scholar]
- Shen, Q.; Zhang, J. MFTFormer: Meteorological-Frequency-Temporal Transformer with Block-Aligned Fusion for Traffic Flow Prediction. In Proceedings of the 2026 9th International Symposium on Big Data and Applied Statistics (ISBDAS); IEEE, 2026; pp. 248–251. [Google Scholar]
- Knox, W.B.; Allievi, A.; Banzhaf, H.; Schmitt, F.; Stone, P. Reward (Mis)design for Autonomous Driving. Artif. Intell. 2021, 316, 103829. [Google Scholar] [CrossRef]
- Dworschak, F.; Dietze, S.; Wittmann, M.R.A.; Schleich, B.R.; Wartzack, S. Reinforcement Learning for Engineering Design Automation. Adv. Eng. Inform. 2022, 52, 101612. [Google Scholar] [CrossRef]
- Kuang, Z.; Rong, R.; Yuan, Y.; Nie, A. Learning Game-Playing Agents with Generative Code Optimization. In Proceedings of the ICML 2025 Workshop on Programmatic Representations for Agent Learning.
- Nie, A.; Daull, X.; Kuang, Z.; Akkiraju, A.; Chaudhuri, A.; Piasevoli, M.; Rong, R.; Yuan, Y.; Choudhary, P.; Xiao, S.; et al. Understanding the challenges in iterative generative optimization with LLMs. arXiv 2026, arXiv:2603.23994. [Google Scholar]
- Yuan, Y.; Ji, Y.; Li, Z.; Li, R. SP-Mind: An Autonomous Reasoning Agent for Spatial Proteomics Analysis. arXiv 2026, arXiv:2606.24235. [Google Scholar]
- Zhang, P.; Hu, X.; Huang, G.; Qi, Y.; Zhang, H.; Li, X.; Song, J.; Luo, J.; Li, Y.; Yin, S.; et al. aiXiv: A next-generation open access ecosystem for scientific discovery generated by AI scientists. arXiv 2025, arXiv:2508.15126. [Google Scholar]
- Deng, M.; Wang, J.; Hsieh, C.P.; Wang, Y.; Guo, H.; Shu, T.; Song, M.; Xing, E.P.; Hu, Z. RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learning. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, (EMNLP), 2022. [Google Scholar]
- Zhao, W.X.; Zhou, K.; Li, J.; Tang, T.; Wang, X.; Hou, Y.; Min, Y.; Zhang, B.; Zhang, J.; Dong, Z.; et al. A Survey of Large Language Models. ArXiv 2023, abs/2303.18223. [Google Scholar]
- Xie, T.; Zhao, S.; Wu, C.H.; Liu, Y.; Luo, Q.; Zhong, V.; Yang, Y.; Yu, T. Text2Reward: Automated Dense Reward Function Generation for Reinforcement Learning. arXiv 2023, arXiv:2309.11489. [Google Scholar]
- Xin, W.; Yin, S.; Ge, Y.; Chen, S. Document Topic Alignment Metrics for Evaluating Topic Models of Short-Text Public Health Communications on Social Media. arXiv 2026, arXiv:2609.14256. [Google Scholar]
- Leurent, E. An Environment for Autonomous Driving Decision-Making. 2018. Available online: https://github.com/eleurent/highway-env.
- Li, Q.; Peng, Z.; Feng, L.; Zhang, Q.; Xue, Z.; Zhou, B. MetaDrive: Composing Diverse Driving Scenarios for Generalizable Reinforcement Learning. IEEE Trans. Pattern Anal. Mach. Intell. 2023, 45, 3461–3475. [Google Scholar] [CrossRef] [PubMed]
- Du, Y.; Watkins, O.; Wang, Z.; Colas, C.; Darrell, T.; Abbeel, P.; Gupta, A.; Andreas, J. Guiding Pretraining in Reinforcement Learning with Large Language Models. In Proceedings of the International Conference on Machine Learning, (ICML), 2023. [Google Scholar]
- Lin, J.; Du, Y.; Watkins, O.; Hafner, D.; Abbeel, P.; Klein, D.; Dragan, A.D. Learning to Model the World with Language. CoRR 2023, abs/2308.01399. [Google Scholar]
- Li, H.; Yang, X.; Wang, Z.; Zhu, X.; Zhou, J.; Qiao, Y.; Wang, X.; Li, H.; Lu, L.; Dai, J. Auto MC-Reward: Automated Dense Reward Design with Large Language Models for Minecraft. 2023. [Google Scholar] [CrossRef]
- Chakraborty, S.; Weerakoon, K.M.K.; Poddar, P.; Tokekar, P.; Bedi, A.S.; Manocha, D. RE-MOVE: An Adaptive Policy Design Approach for Dynamic Environments via Language-Based Feedback. ArXiv 2023, abs/2303.07622. [Google Scholar]
- Pang, J.C.; Yang, X.; Yang, S.; Yu, Y. Natural Language-conditioned Reinforcement Learning with Inside-out Task Language Development and Translation. ArXiv 2023, abs/2302.09368. [Google Scholar]
- Madaan, A.; Tandon, N.; Gupta, P.; Hallinan, S.; Gao, L.; Wiegreffe, S.; Alon, U.; Dziri, N.; Prabhumoye, S.; Yang, Y.; et al. Self-Refine: Iterative Refinement with Self-Feedback. In Proceedings of the Advances in Annual Conference on Neural Information Processing Systems, (NeurIPS), 2023. [Google Scholar]
- Song, J.; Zhou, Z.; Liu, J.; Fang, C.; Shu, Z.; Ma, L. Self-Refined Large Language Model as Automated Reward Function Designer for Deep Reinforcement Learning in Robotics. CoRR 2023, abs/2309.06687. [Google Scholar]
- Ma, Y.J.; Liang, W.; Wang, G.; Huang, D.A.; Bastani, O.; Jayaraman, D.; Zhu, Y.; Fan, L.; Anandkumar, A. Eureka: Human-Level Reward Design via Coding Large Language Models. arXiv Arxiv-2310.. 2023. [Google Scholar]
- Schulman, J.; Wolski, F.; Dhariwal, P.; Radford, A.; Klimov, O. Proximal Policy Optimization Algorithms. CoRR 2017, abs/1707.06347. [Google Scholar]
- Jia, R.; Xie, W.; Wei, B.; Qiao, G.; Yang, Z.; Lyu, X.; Tang, Z. Molecular formula image segmentation with shape constraint loss and data augmentation. In Proceedings of the 2022 IEEE International Conference on Bioinformatics and Biomedicine (BIBM); IEEE, 2022; pp. 3821–3823. [Google Scholar]
- Li, S.; Hussein, R. Effectiveness of Using Remote Laboratory in Promoting Simulation and Verification Tools. In Proceedings of the Open Science in Engineering, Cham, 2023; pp. 87–95. [Google Scholar]
| 1 | OpenAI model documentation: https://developers.openai.com/api/docs/models/gpt-5.6-sol. |
Figure 1.
An overview of the GPT-5.6-based AD Reward Designer. Natural language goals and typed simulator interfaces guide dense reward generation. Code checks verify syntax, API compatibility, and finite scalar outputs before PPO training; failed checks return the code and error messages to GPT-5.6 for repair. Policy rollout summaries, human feedback, and the current reward code guide subsequent refinement.
Figure 1.
An overview of the GPT-5.6-based AD Reward Designer. Natural language goals and typed simulator interfaces guide dense reward generation. Code checks verify syntax, API compatibility, and finite scalar outputs before PPO training; failed checks return the code and error messages to GPT-5.6 for repair. Policy rollout summaries, human feedback, and the current reward code guide subsequent refinement.

Figure 2.
Reward curves of the original GPT-4-based implementation in Highway-env.

Figure 3.
Reward curves of the original GPT-4-based implementation in MetaDrive.

Figure 4.
Reward curve from the original demonstration-conditioned case study.

Table 1.
Original GPT-4-based results in Highway-env; GPT-5.6 results remain to be evaluated.
| method | velocity | crash rate | success rate |
|---|---|---|---|
| merge | |||
| human-written | 20.00 | 0% | 100% |
| AD Reward Designer | 25.85 | 0% | 100% |
| intersection | |||
| human-written | 9.00 | 20% | 80% |
| AD Reward Designer | 9.17 | 16% | 84% |
| roundabout | |||
| human-written | 14.63 | 0% | 100% |
| AD Reward Designer | 15.99 | 0% | 100% |
| highway | |||
| human-written | 20.17 | 10% | 90% |
| AD Reward Designer | 22.03 | 3% | 97% |
Table 2.
Original GPT-4-based results in MetaDrive; GPT-5.6 results remain to be evaluated.
| method | velocity | crash rate | success rate | |
|---|---|---|---|---|
| curve | ||||
| human-written | 7.15 | 0% | 100% | |
| AD Reward Designer | 14.83 | 0% | 100% | |
| intersection | ||||
| human-written | 16.87 | 0% | 100% | |
| AD Reward Designer | 21.86 | 0% | 100% | |
| roundabout | ||||
| human-written | 10.61 | 0% | 100% | |
| AD Reward Designer | 18.53 | 0% | 100% | |
| T-intersection | ||||
| human-written | 20.87 | 0% | 100% | |
| AD Reward Designer | 27.51 | 0% | 100% |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2026 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).
Copyright: This open access article is published under a Creative Commons CC BY 4.0 license, which permit the free download, distribution, and reuse, provided that the author and preprint are cited in any reuse.