Hierarchical Integration of Large Language Models and Multi-Agent Reinforcement Learning

1. Introduction

Multi-Agent Reinforcement Learning (MARL) has been widely adopted for solving cooperative decision-making problems in domains such as logistics optimization, robotic swarms, and drone control systems [1,2]. In these settings, multiple agents are trained to learn policies that jointly maximize a shared reward under partial observability and dynamic interactions. Recent work on sequential cooperative multi-agent learning demonstrates that explicitly modeling adaptive coordination over extended horizons can significantly improve policy stability and coordination effectiveness in dynamic and uncertain environments [3]. This observation highlights the importance of temporal structure and coordination dynamics in cooperative learning, particularly for long-horizon tasks.

Representative MARL algorithms such as MAPPO and QMIX have shown strong performance in complex games and simulated benchmarks by leveraging centralized training objectives or value decomposition techniques [4,5]. Despite their success, these methods typically adopt a “flat” decision-making structure, where agents select actions at a single temporal scale. As task duration increases, the effective action space grows combinatorially, making it increasingly difficult to associate early decisions with delayed rewards. This credit assignment challenge often leads to unstable learning dynamics and inconsistent strategies, especially in tasks that require long-term planning and coordinated exploration [6,7]. Recent advances in Large Language Models (LLMs) offer new opportunities to address these limitations. LLMs have demonstrated a strong ability to decompose complex objectives into structured sub-tasks and to generate high-level plans from abstract instructions [8,9]. Unlike purely numerical policies, LLMs encode broad world knowledge and semantic structure, enabling them to reason over goals, constraints, and task hierarchies. This capability has motivated a growing body of work that integrates LLMs into interactive agents capable of operating within software environments and simulated worlds [10,11]. In multi-agent settings, language has emerged as an effective medium for coordination, allowing agents to exchange intentions and assign responsibilities in a more interpretable manner than latent numerical vectors [12,13]. Empirical results suggest that language-based coordination can substantially improve cooperation quality in long-horizon and compositional tasks [14]. However, directly combining LLMs with low-level control policies introduces several challenges. A primary limitation lies in the computational latency of LLM inference, which makes them unsuitable for environments that require rapid, fine-grained reactions [15]. In addition, LLMs often lack the precision necessary for continuous control, leading to invalid or unsafe actions when used directly for execution [16]. Many existing approaches address this issue by injecting LLM outputs into the observation space of reinforcement learning agents, without explicitly separating planning from acting. Such designs blur the boundary between high-level reasoning and low-level control, preventing the system from effectively balancing long-term planning objectives with immediate reactive behaviors [17,18].

Motivated by these challenges, this study proposes L2M2, a hierarchical framework that integrates LLMs with MARL through a clear division of labor across temporal scales. The framework adopts a two-level structure in which the LLM functions as a high-level planner that generates abstract instructions at coarse time intervals, while MARL agents operate as low-level executors that learn efficient control policies to fulfill these instructions. By decoupling strategic planning from real-time action execution, the proposed approach mitigates the credit assignment problem in long-horizon tasks and reduces reliance on high-frequency LLM inference. The framework is evaluated on navigation and resource allocation benchmarks using 8,200 training episodes. Experimental results demonstrate that this hierarchical design significantly improves task success rates and accelerates learning compared to flat MARL baselines, highlighting the effectiveness of combining language-based planning with reinforcement learning for scalable multi-agent coordination.

2. Materials and Methods

2.1. Dataset and Environment Description

We used two simulated environments for this study: a Multi-Robot Warehouse and a Cooperative Search Grid. The dataset contains 8,200 episodes. In the warehouse task, 4 to 8 agents must move packages to specific zones without hitting each other. In the search task, agents must explore a map to find hidden targets. We changed the map sizes from $10\text{×}10$ to $50\text{×}50$ to test performance on different scales. Each sample contains the positions of agents, the locations of obstacles, and the status of tasks. We created these environments randomly to ensure the data covers many different situations.

2.2. Experimental Design and Controls

To test the method, we established one experimental group and three control groups. The experimental group used the L2M2 hierarchical framework. Control Group A used a standard “flat” MAPPO algorithm, where agents act directly on raw inputs. Control Group B used a pure LLM planner without low-level training. Control Group C used a random hierarchical method, where the high-level goals were random. This design allowed us to identify whether the improvement came from the hierarchy or the specific advice from the LLM. All groups operated for 5 million time steps.

2.3. Measurement and Quality Control

We measured performance using Task Success Rate (TSR) and Mean Convergence Time (MCT). TSR shows the percentage of episodes where all tasks are finished within the time limit. MCT measures the number of training steps needed to reach stable performance. To ensure quality, we repeated each test five times with different random seeds. We also used a safety check. If the LLM generated a goal that was impossible to reach, the system rejected it. We removed any trials where the simulation failed due to software errors.

2.4. Data Processing and Formulas

We organized the data into two levels. The high-level data contains environment states and goal descriptions. The low-level data contains local observations and actions. We defined a reward function for the low-level agents to encourage them to reach the goal $\mathrm{g}$. We calculated the internal reward${\mathrm{r}}_{\mathrm{in}}$ using Eq. (1):

$${\mathrm{r}}_{\mathrm{in}}({\mathrm{s}}_{\mathrm{t}},\mathrm{g})=-\mathsf{\beta }\text{‖}\mathrm{p}({\mathrm{s}}_{\mathrm{t}})-\mathrm{p}(\mathrm{g}){\text{‖}}_2+\mathrm{I}\left(\mathrm{reached}\right)$$

In this formula, $\mathrm{p}({\mathrm{s}}_{\mathrm{t}})$ is the agent’s position, $\mathrm{p}\left(\mathrm{g}\right)$ is the target position, and $\mathrm{I}$ is a binary value for success. We updated the low-level policy using the standard PPO loss function shown in Eq. (2):

$$\mathrm{L}\left(\mathsf{\theta }\right)=\min ({\mathrm{r}}_{\mathrm{t}}\left(\mathsf{\theta }\right){\widehat{\mathrm{A}}}_{\mathrm{t}},\mathrm{clip}({\mathrm{r}}_{\mathrm{t}}\left(\mathsf{\theta }\right),1-\mathsf{ϵ},1+\mathsf{ϵ}){\widehat{\mathrm{A}}}_{\mathrm{t}})$$

Here, ${\widehat{\mathrm{A}}}_{\mathrm{t}}$ is the advantage function, which measures how good an action is compared to the average.

2.5. Implementation and Statistics

We performed the experiments using Python and the PyTorch library. We used the OpenAI Gym interface for the environments. We trained the models on a server with 4 NVIDIA RTX 3090 GPUs. The learning rate for the low-level agents was $3\text{×}10^{-4}$. We used the GPT-4 model as the high-level planner. To compare the groups, we used a one-way ANOVA test followed by a Tukey post-hoc test. We considered a result to be statistically significant if the $\mathrm{p}$-value was less than 0.05. This confirms that the differences are real and not due to random chance.

3. Results and Discussion

3.1. Improvement in Task Success Rate

We tested the L2M2 framework on the resource allocation task. The results show that the hierarchical method improved the task success rate by 20.5% compared to the flat MAPPO baseline. In complex cases that require long planning, standard agents often failed. They focused on immediate rewards and missed the final goal. In contrast, the L2M2 agents followed the subgoals set by the language model. This allowed them to maintain a steady strategy. Figure 1 compares the average rewards of different reinforcement learning algorithms during training. As shown in the figure, improved methods achieve higher final rewards and learn faster than standard baselines. Our results agree with this pattern. The data confirms that separating high-level planning from low-level control leads to better results in long tasks [19,20].

3.2. Reduction in Convergence Time

We also analyzed the learning speed of the system. The data shows that our method reduced the learning time by 1.6 times compared to the baseline. In flat MARL, the agents must search the entire state space to find a solution. This process is slow when the state space is large. By using the LLM to define subgoals, our method reduced the search area for the low-level policy. The agents only needed to learn how to reach the next nearby subgoal. This made the learning process more efficient. This result suggests that breaking down the time scale is an effective way to speed up training in large systems [21,22].

3.3. Scalability to Large Environments

We tested the performance of the method by increasing the map size from $10 \times 10$ to $50 \times 50$. The performance of the baseline models dropped as the map size increased. On the $50 \times 50$ map, the success rate of the QMIX model fell to below 30%. However, the L2M2 framework maintained a success rate of over 60%. Figure 2 shows the generated paths in a grid environment with obstacles. Similar to the paths shown in the figure, our agents found valid routes by connecting subgoals and avoiding collisions [23]. This visual evidence supports the numerical data. It shows that the hierarchical structure helps agents navigate large spaces without getting stuck.

3.4. Role of the Language Model

Finally, we analyzed the effect of the language model. We compared the full model with a version that used random high-level goals. The results show that the model with random goals failed to solve the task. This shows that the logic provided by the LLM is necessary. The LLM understands the physical limits of the environment and generates logical steps. For example, it directs agents to “pick up the tool” before “repairing the machine.” The low-level policy cannot learn this link easily from rare rewards. Therefore, the combination of logic and motor control is the main reason for the success of the system [24].

4. Conclusions

In this paper, we proposed a method that combines Large Language Models with multi-agent learning. The results show that this method increases the success rate by 20.5% compared to standard baselines. It also reduces the training time by 1.6 times. Unlike previous methods, our system uses language to divide long tasks into small goals. This finding shows that separating planning from control helps solve complex problems. This method is useful for applications like warehouse automation and rescue robots. However, the processing speed is slow. Future studies should use smaller models to increase the speed.