Optimizing Coalition Formation Strategies for Scalable Multi-Robot Task Allocation: A Comprehensive Survey of Methods and Mechanisms

1. Introduction

In the era marked by the emergence of Industry 4.0, there has been a remarkable increase in the demand for autonomous systems and Artificial Intelligence. The field of robotics has experienced widespread popularity across various sectors of society. As a result, the concept of multi-robot systems (MRS) has gained significant attention, surpassing that of individual robots. A multi-robot system comprises a group of robots working together to achieve a predetermined objective. A related term, like MRS, is the multi-agent system (MAS), which consists of a collective of agents collaborating to accomplish a specific goal [1]. The key distinction between MRS and MAS lies in the former involving physical robots as agents, whereas the latter incorporates agents represented as software entities. MRS offer numerous advantages over individual robots:

(a) MRS enables parallel task execution, leading to accelerated goal attainment.

(b) Heterogeneity in robot capabilities can be accommodated within MRS.

(c) MRS effectively handles tasks distributed across large spatial domains.

(d) Inherent robustness in fault tolerance is a characteristic feature of MRS.

The taxonomy of MRS divides into two main categories: homogeneous assemblies, consisting of robots with similar capabilities, and heterogeneous assemblies, comprising robots with varying proficiencies. Additionally, this classification extends to cooperative ensembles, where robots collaborate to achieve goals, and competitive configurations, where robots compete for dominance. In recent times, there has been a significant increase in the demand for MRS in practical applications [2]. This growing demand has led to the integration of MRS into various domains, including mining, warehouse logistics, agricultural field operations, and even recreational and entertainment sectors. However, this widespread adoption of MRS spans across numerous sectors globally [3]. As this trend continues, addressing the existing challenges inherent to MRS becomes increasingly critical.

Figure 1. Real world applications of MRS [101,102,103,104,105,106].

Figure 1. Real world applications of MRS [101,102,103,104,105,106].

MRS pose a range of intricate challenges across technological, logistical, and operational spheres, as depicted in Figure 2. Key challenges include coordinating robot actions to avoid collisions or redundancy, ensuring reliable communication, and efficient task allocation [4]. Moreover, addressing faults, maintaining accurate environmental perception, managing shared resources while upholding privacy, security, and ethical human-robot interactions are significant hurdles. Energy efficiency is crucial, especially for battery-powered robots. These challenges collectively render the development and operation of multi-robot systems multifaceted.

Task allocation stands out as particularly crucial among these challenges due to its pivotal role in optimizing resource utilization, enhancing system efficiency, and facilitating effective collaboration among robots. Efficient task allocation in multi-robot systems offers various benefits. Firstly, it optimizes resource usage, leading to cost savings and improved system performance. Secondly, by ensuring tasks are executed effectively and promptly, task allocation enhances overall system efficiency. Thirdly, it facilitates scalability, enabling seamless integration of additional robots or tasks without compromising efficiency [5]. Moreover, task allocation contributes to system robustness by allowing adaptation to dynamic environments and changing task demands. Lastly, it promotes collaboration among robots, reducing conflicts and redundancy while enhancing system coordination.

1.1. Multi-Robot Task Allocation (MRTA)

Task allocation within a multi-robot system involves determining which robots should perform specific tasks to achieve overarching system objectives, aiming for coordinated team behavior. While some systems, like certain biologically inspired robotic setups, exhibit coordinated team behavior through local interactions among team members and the environment, known as implicit or emergent coordination [6], some other lies on explicit or intentional cooperation. In this approach, tasks are directly assigned to robots or sub-teams, constituting the problem of multi-robot task allocation.

The study reported in [7] proposed a categorization framework for MRTA problems based on three axes. The first axis distinguishes between single-task (ST) robots, capable of executing only one task at a time, and multi-task (MT) robots, capable of handling multiple tasks simultaneously. The second axis categorizes tasks as Single-Robot (SR) tasks, requiring one robot for completion, or Multi-Robot (MR) tasks, necessitating the involvement of multiple robots. The third axis differentiates between Instantaneous Assignment (IA) problems, which involve immediate task allocation without consideration for future assignments, and Time-Extended Assignment (TA) problems, where robots are allocated, tasks based on a predetermined schedule encompassing both current and future allocations [7].The study delineates the MRTA quandary into eight distinct typologies known as two level task allocation taxonomy (iTax) classification, as illustrated in Figure 3 [8]. This scheme rests on the foundation of interdependent resources and constraints. Employing the iTax framework, the MRTA conundrum was segmented into four distinct categories: those devoid of dependencies, those governed by scheduled dependencies, those subjected to cross-scheduled dependencies, and finally, those characterized by complex dependencies [9].

The limitations of the iTax have driven the exploration of alternative strategies like auction-based and optimization-based approaches to tackle MRTA problems. iTax's complexity arises from its division of task allocation into high-level and low-level processes, potentially complicating system design and coordination. Scalability issues may arise with iTax as the number of robots and tasks increases, leading to inefficiencies in allocation and coordination. Additionally, iTax may yield sub-optimal solutions due to the separation of high-level decisions from low-level motion planning considerations. Consequently, auction-based approaches, inspired by auction dynamics, and optimization-based approaches, relying on mathematical formulations, have emerged as alternatives. Auction-based approaches enable dynamic task allocation through robots bidding based on their capabilities, while optimization-based approaches offer systematic solutions considering resource utilization and task efficiency. These alternatives aim to address iTax's limitations and enhance MRTA problem-solving effectiveness.

Considering factors such as solution optimality, allocation timing, and problem-specific constraints, two crucial task allocation strategies have been developed: auction-based approaches [10] and optimization-based approaches [11]. Auction-based approaches draw inspiration from the dynamics of auction-bid systems observed in societal contexts. In this approach, robots submit bids reflecting their current state values to compete for tasks. The task is then assigned to the robot offering the most suitable bid, representing a merit-based selection process. Conversely, optimization approaches involve devising mathematical solutions to tackle task allocation challenges. Additionally, behavior-based methodologies [12] have garnered considerable scholarly attention for their ability to adapt to the dynamic behavior of systems. More recently, innovative dynamic task allocation strategies have emerged to address complex constraints associated with various uncertain conditions. The growing significance of Artificial Intelligence (AI) and learning-based approaches has become prominent in this domain [13]. Researchers are increasingly turning their attention to these methods, motivated by their potential to provide comprehensive autonomous and optimal solutions to MRTA challenges. This trend reflects the evolving landscape of the field.

MRTA optimizes system performance by efficiently distributing tasks among multiple robots, crucial for various applications such as resolving complex challenges like the Multiple Traveling Salesman Problems (mTSP), Vehicle Routing Problems (VRP), Job Scheduling Problems (JSP), Team Orienting Problems (TOP), and Dial-a-Ride Problems (DARP). In mTSP, multiple robots solve a variant of the Traveling Salesman Problem, optimizing routes to minimize travel distance and costs [14]. VRP involves efficiently routing tasks among robots, resembling delivery vehicles, considering constraints like capacities and time windows [15]. JSP focuses on scheduling tasks among robots to minimize make span, critical for industrial automation and warehouse management [16]. TOP dynamically forms and optimizes teams of robots to accomplish tasks collaboratively, crucial for search and rescue missions [17]. DARP coordinates multiple robots to provide transportation services, relevant for ride-sharing and public transportation systems, emphasizing the optimization of passenger routing and vehicle allocation [18].

The coalition formation (CF) remains a crucial but unaddressed aspect in a wide array of multi-robot task allocation problems, such as mTSP, VRP, Job JSP, TOP, and DARP. These problems involve multiple robots or agents tasked with optimizing various objectives. For instance, in mTSP, the challenge is to have multiple salesmen (robots) efficiently visit a set of locations while minimizing the total route length. The establishment of coalitions can facilitate the equitable distribution of tasks, leading to a reduction in the overall travel distance. In VRP, the objective is to deliver goods to multiple customers while minimizing transportation costs. Here, coalitions play a vital role in route and resource sharing to achieve cost-effective solutions. Similarly, in JSP, multiple robots must execute a variety of tasks while optimizing time and resource utilization. The formation of coalitions aids in workload balancing and reduces the make-span. In TOP, the focus is on organizing robots into teams to accomplish specific tasks effectively. Coalitions are instrumental in the formation of teams that maximize task performance. Lastly, DARP involves multiple robots providing transportation services to passengers with pickup and drop-off requests. In this context, coalitions help in passenger sharing and route optimization.

In conclusion, CF is an indispensable aspect of addressing the challenges posed by MRTA across various domains. This paper reviews MRTA algorithms, assesses their support for CF, and seeks to determine the most effective methods for achieving optimal CF.

1.2. Coalition Formation (CF)

In the realm of robotics or multi-agent systems, the assembly of separate groups of robots or agents, each assigned specific tasks, constitutes coalition formation presented in Figure 4 [19]. Task allocation, within this framework, arises as a fundamental aspect of CF, involving the assignment of tasks to robots based on their capabilities. Optimal CF requires maximizing overall team performance while minimizing task completion time and resource consumption [20].

Various algorithms aimed at facilitating efficient collaboration among robots through CF have been devised. Examples include the Contract Net Protocol (CNP) [21], Consensus-Based Bundle Algorithm (CBBA) [22], Task Allocation Via Iterative Bargaining (TAIB) [23], Stable Marriage Algorithm (SMA) [24], Dynamic Distributed Coalition Formation (DDCF), as well as the Merge and Split Approaches, Combinatorial Auctions, Core Selecting Combinatorial Auction (CCA), and the Genetic Algorithm (GA) for CF [25]. These algorithms incorporate negotiation mechanisms, enabling robots to form coalitions based on their individual competencies and preferences.

In the context of the CNP, tasks are disseminated to participating robots by a central coordinator. Interested robots then submit bids for tasks, and the coordinator selects winning bids to form coalitions. CBBA involves robots engaging in communication and negotiations through a consensus-based framework, facilitating deliberation on coalition assignments and task allocation. In TAIB, robots negotiate with other entities to assess potential coalitions, iteratively refining preferences during CF. The SMA entails robots (male) proposing to potential coalitions (females), with coalitions accepting or rejecting proposals based on preferences. DDCF utilizes a distributed negotiation approach, enabling robots to adapt to changes in team composition. The Merge-and-Split algorithm allows existing coalitions to merge or split based on evolving task requirements and robot capabilities. Combinatorial auctions offer bundles of tasks or resources for bidding, with the auctioneer assigning winning combinations to maximize efficiency. Core CCA aims to determine stable and efficient task allocations among coalitions. Lastly, in GA, robot capabilities and preferences are represented as genes, subject to selection, crossover, and mutation operations to converge towards an optimal coalition structure [26].

The domain of multi-robot CF presents various challenges, including managing uncertainty, partial observability [27], non-stationarity [28], communication disruptions, the heterogeneity of robot capabilities, and task interdependencies. Developing efficient coalition algorithms is a significant task, often requiring substantial computational resources and specialized expertise. Maintaining coordination within coalition-based interactions in dynamic environments can be complex, with synchronization of actions emerging as a crucial concern. Additionally, multiple coalitions may converge to undertake similar or overlapping tasks, potentially leading to inefficiencies [29,30]. Nevertheless, CF enables robots to share resources such as energy, computational power, and communication bandwidth, enhancing resource utilization [31]. This sharing facilitates flexibility in team configuration adaptation in response to changing environmental conditions or task requirements. Such adaptive capability enhances system resilience against unforeseen challenges and improves overall robustness.

This article is a revised and expanded version of a paper entitled “Analyzing multi-robot task allocation and coalition formation methods: A comprehensive study,” which was presented at the 1st International Conference on Advanced Robotics, Control, and Artificial Intelligence (ARCAI), Australia, December 2024. The rest of this paper offers a concise taxonomy of various strategies utilized in MRTA and CF. In Section 2, detailed insights into each strategy are provided along with a comparative analysis of different approaches within each strategy and a higher-level comparison of strategies is conducted based on relevant factors. Section 3 focuses on algorithm formulation and presents simulation results, while Section 4 serves as the discussion segment, summarizing the comprehensive review and highlighting the key findings. Section 5 gives our conclusion.

2. MRTA Classification

Figure 5 depicts an arrangement presenting a categorization of strategies pertaining to MRTA. Within this taxonomy, four overarching classifications are discernible: behavior-based, market-based, optimization-based, and learning-based. Each of these classifications introduces a collection of prominent and well-established methodologies to address MRTA challenges, representative of standard paradigms. While certain researchers are modifying or extending these established approaches, others are integrating aspects of these methodologies to engender hybrid strategies. The four main categories of MRTA strategies are Behavior-based MRTA, Market-based MRTA, Optimization-based MRTA and Learning-based MRTA.

2.1. Behavior-Based MRTA

Behavior-based MRTA represents a distinctive strategy characterized by its entirely reactive response to dynamic problem scenarios, as suggested by its nomenclature. This approach is structured upon a dual-tier architecture, comprising a foundational lower level and a supervisory stratum [32]. The lower level includes tasks such as navigation, obstacle avoidance, and task-switching, while the supervisory level handles task identification and inter-robot communication within the team. To enhance resilience, a problem-specific layer augments the lower-level behavior [33]. Figure 5 illustrates that behavior-based MRTA encompasses four methodological approaches: ALLIANCE, vacancy chain scheduling, Broadcast of Local Eligibility (BLE), and Automated Synthesis of Multi-Robot Task Solutions through Software Reconfiguration (ASyMTRe) [6].

2.1.1. Alliance

The Alliance architecture, known for its behavior-based and fully distributed principles, enables MRS to adapt to diverse circumstances by organizing behaviors tailored to high-level tasks [34,35,36]. Key features include robot impatience and acquiescence within coalitions, which impact collaboration by prioritizing individual goals or yielding excessively, respectively. Balancing these behaviors is crucial for effective collaboration [37,38,39]. Alliance supports task allocation, resource sharing, and decision-making among dynamically assembled coalitions, enhancing communication and, information exchange for optimized performance [40,41]. Attributes include distributed decision-making, reactive behavior, emergent coalition behavior, robustness, scalability, and adaptability.

2.1.2. Vacancy Chain Scheduling

The Vacancy Chain System (VCS) models scheduled by creating a cascade effect through job promotions, akin to bureaucratic structures. This method addresses scheduling intricacies and group dynamics using microscopic and macroscopic approaches [42]. While effective in spatial domains, VCS's performance varies with individual robot capabilities. Its strength lies in its versatility across problems, requiring minimal problem-specific information. Key attributes include distributed decision-making, reactive behavior, emergent system behavior, robustness against robot unavailability, scalability, and adaptability through scheduling algorithm adjustments [43,44].

2.1.3. Broadcast of Local Eligibility (BLE)

The Broadcast of Local Eligibility (BLE) mechanism compares a task's local eligibility with the highest eligibility among similar behaviors in other robots, following the Port Arbitrated Behavior (PAB) paradigm [45]. BLE operates by suppressing peer behaviors when a robot's local eligibility surpasses others, signaling its claim to tasks. In the absence of inhibitory interactions, other robots assume task responsibility. BLE's scalability depends on communication bandwidth, with attributes including distributed decision-making, reactive behavior, emergent behavior, robustness against failures, scalability, adaptability, and local communication for sharing eligibility information among neighbors [46,47].

2.1.4. Automated Synthesis of Multi-Robot Task Solutions through Software Reconfiguration (ASyMTRe)

ASyMTRe automatically reconfigures schema connections across and within robots by linking environmental, perceptual, and motor control schemas, facilitating effective multi-robot behaviors aligned with team objectives [48]. It addresses task-related challenges in diverse robot teams, enabling the synthesis of new task solutions and sharing sensory information among networked robots to assist less capable ones [49]. ASyMTRe exhibits distributed decision-making, reactive behavior, emergent behavior, robustness, scalability, and adaptability, allowing robots to autonomously make decisions, adjust to environmental changes, and withstand failures [50].

As previously mentioned, ALLIANCE, Vacancy chain, BLE, and ASyMTRE are four key algorithms falling under the umbrella of behavior-based MRTA. Despite their shared behavior-based nature, they exhibit distinct attributes and advantages and disadvantages. Table 1 and Table 2, below present a comprehensive comparative analysis of these algorithms based on their characteristics. The comparison of these attributes underscores ALLIANCE's proficiency in effectively managing CF in Multi-Robot Systems (MRS).

2.2. Market-Based MRTA

The foundational concept of market based MRTA draws inspiration from the principles of auctions and bidding. Various market-based methodologies are crafted by adapting the core auction-bidding procedure with the incorporation of incentives and penalties. Figure 6 illustrates the fundamental genesis of market based MRTA.

Market-oriented strategies focus on utility functions, quantifying agents' capacity to evaluate their interest in particular tasks for potential exchange. In MRTA systems, these functions clarify the alignment between a robot's skills and task requirements [19]. Market-based approaches in multi-robot task allocation exhibit distinctive attributes that enhance their efficacy. By centering on utility functions, agents can gauge preferences for tasks, facilitating efficient task trading. Auction mechanisms introduce competitive bidding, motivating agents to bid based on perceived task values [51]. Incentive-driven behavior, through rewards and penalties, ensures optimal task allocation and fosters cooperation. Decentralized decision-making empowers agents to autonomously engage in task assignments. The adaptability of market-based strategies allows responses to changing conditions by adjusting utility functions and allocation mechanisms. These methods are flexible, accommodating various tasks and heterogeneous robot capabilities [52]. Through bid submissions and communication, agents share information, enhancing decision-making. The scalability of market-based systems effectively manages large-scale multi-robot scenarios, resulting in robust allocations that maximize overall utility, contributing to resource-efficient task distribution. Collectively, these attributes equip market-based approaches to effectively address multi-robot task allocation challenges.

2.2.1. RACHNA

RACHNA, an ascending auction-based task allocation protocol that allows task preemption, introduces market-based approaches to CF in competitive environments but faces challenges like unnecessary task reassignments, impacting performance due to switching overhead [53]. Designed for CF, RACHNA addresses heuristic-driven methods' limitations by allowing tasks to compete for robot resources and autonomously adjusting sensor values based on supply and demand, without constraints on coalition size [54,55]. It frames CF as a multi-unit combinatorial auction, enabling efficient management of CF through bids on combinations of goods [56].

2.2.2. KAMARA (KAMRO’s Multi-Agent Robot Architecture)

KAMARA combines centralized and distributed frameworks for the Karlsruhe Autonomous Mobile Robot (KAMRO). This architecture enhances control and integration across diverse aspects of distributed intelligent robot systems and their individual components, accommodating various forms of cooperation among interconnected agents, such as closed kinematic chains or camera-manipulator couplings [57,58].

2.2.3. MURDOCH

MURDOCH involves an Auctioneer and Bidder in a swift task allocation method through initial-price, one-round auctions, assigning tasks to the highest bidder based on computed bid values reflecting agents' capabilities [59]. It introduces a robust MRTA algorithm adept at handling robot malfunctions and communication breakdowns, allowing task reassignment and facilitating collaboration among heterogeneous robots [60]. The Auctioneer initiates auctions upon task arrival, selects winning bidders, and monitors task execution, while bidders compute utility metrics, submit bids, and execute tasks upon winning, using a publish-subscribe communication model for coordination [61,62,63,64].

2.2.4. M+

M+ is a distributed multi-robot cooperation scheme integrating mission planning, task refinement, and cooperative mechanisms from the Contract Net Protocol framework within the M+ unified system, designed for integration using the LAAS Architecture [65,66]. It includes M+ task allocation, cooperative response to contingencies, and task execution operations, refining and assigning tasks through negotiation mechanisms. The cooperative reaction activity handles task execution failures by updating the world state, facilitating information exchange, overseeing (re)planning, and coordinating assistance, while the execution function manages task control and synchronization among robots [67,68].

2.2.5. TraderBots

The TraderBots methodology leverages market economies' benefits for effective multi-robot coordination in dynamic scenarios through decentralized decision-making and task allocation within coalitions, demonstrated in simulations and physical implementations [69,70,71]. It excels in CF by employing a market-based framework, facilitating efficient task allocation based on robots' capabilities and task compatibility, adapting to dynamic environments, and fostering information exchange among robots for informed decision-making. TraderBots is robust against failures and communication disruptions, showcasing versatility and scalability for diverse multi-robot scenarios, proving its practicality for effective CF [72].

All these methodologies fall within the realm of market-based task allocation and are founded on the common principle of employing market mechanisms to distribute tasks among robots. However, they diverge in terms of bid generation methods and the precise algorithms employed to optimize the task allocation process. The Table 3 presented below illustrates the comparison of five market-based approaches according to their distinctive attributes.

2.3. Optimization-Based MRTA

Optimization-driven methods strive to address the MRTA challenge by framing it as an optimization dilemma, seeking the best task-to-robot assignment. These methods primarily fall into two categories: conventional optimization methods and evolutionary optimization strategies.

Traditional optimization methods encompass mathematical frameworks, among which are Mixed Integer Linear Programming (MILP) and Quadratic Programming (QP), frequently employed. In contrast, evolutionary optimization approaches such as Particle Swarm Optimization (PSO), Ant Colony Optimization (ACO), GA, and Simulated Annealing (SA) have gained popularity [73]. While MILP and QP are favored for their potential to identify globally optimal solutions, their computational complexity can escalate with larger problem scales [74]. On the other hand, PSO, ACO, GA, and SA serve as metaheuristic optimization methods, capable of managing multiple objectives and non-linear constraints, although they might converge to local optima.

2.3.1. Traditional Optimization

MILP and QP are prevalent traditional methods for solving multi-robot task allocation challenges. MILP involves linear objective functions and constraints, while QP deals with non-linear formulations. A significant challenge with MILP is its scalability due to increased complexity and variable quantities [75]. In contrast, QP controllers are becoming benchmarks for managing complex objectives in robots with multiple joints, such as humanoids, proving effective in position control, torque regulation, multi-robot coordination, and force management [76].

2.3.2. Evolutionary Optimization

PSO operates as a population-centric search algorithm inspired by social behavior, iteratively exploring optimization problems within a designated search domain, making it suitable for robots with limited capabilities [77,78,79,80].

ACO, initiated by Dr. Marco Dorigo, emulates ant foraging behavior and finds applications in routing and scheduling, functioning as a probabilistic strategy for identifying favorable pathways across graphs within swarm intelligence methodologies [81,82,83,84,85].

GA explores optimization challenges through competitive selection, recombination, and mutation within a population of solutions, generating novel solutions biased towards favorable regions of the solution space [86,87].

SA mimics physical annealing transformations in solids, gradually reducing temperature to converge towards the most favorable state, using temperature-dependent random factors to escape local minima and reach lower energy configurations [88,89,90].

Optimization-based MRTA approaches: Methods like PSO, ACO, GA, SA, LP, and QP exhibit diverse attributes shaping their problem-solving dynamics.

• PSO seeks global optima while navigating exploration and exploitation trade-offs.
• ACO emphasizes pheromone-guided exploration and solution construction.
• GA balances diversity through mutation and convergence via crossover.
• SA transitions from high-temperature exploration to low-temperature exploitation.
• LP targets linear relationships, and QP handles quadratic ones, both optimized for resource allocation.

Each approach's distinct characteristics equip them to excel in various problem domains, addressing the intricacies of task allocation scenarios. A comparison of these approaches, considering various attributes, is presented in Table 4 and Table 5 below.

2.4. Learning-Based MRTA

Learning is fundamental to constructing solutions, representing a progression where an AI program enhances its understanding by observing its environment. Technically, AI learning mechanisms involve processing input-output pairs to deduce patterns for a designated function, enabling the anticipation of outputs for novel inputs. This paradigm allows robots to dynamically improve their task allocation strategies through learning from data and experience. By leveraging machine learning techniques, these systems adapt and optimize decisions over time, effectively navigating complex and changing environments [91]. This approach enhances resource utilization, coordination, and overall efficiency in multi-robot systems by allowing them to autonomously refine task allocation behaviors based on real-world interactions and performance feedback.

2.4.1. Machine Learning

Machine Learning, a subset of AI, enables machines to replicate intelligent human actions. It includes four main types: supervised, semi-supervised, unsupervised, and reinforcement learning [92]. These techniques allow computers to learn from data, facilitating informed decision-making and predictions. In multi-robot task allocation, machine learning enhances decision-making and adaptability, enabling dynamic optimization of task allocation strategies based on data-driven insights and historical performance [93]. By learning from interactions between robots, their environment, and tasks, multi-robot systems improve resource utilization, coordination, and system efficiency, enhancing scalability, flexibility, and autonomy in addressing task allocation challenges.

Figure 7. Structure of machine learning.

Figure 7. Structure of machine learning.

Supervised Learning trains algorithms on labeled data to predict specific outputs, iteratively refining models to discern patterns connecting input and output labels. In multi-robot task allocation, supervised learning maps behaviors to optimal allocations, using historical data to predict suitable assignments for new scenarios, optimizing resource utilization and system efficiency [94,95].

Semi-supervised Learning combines supervised and unsupervised learning, leveraging both labeled and unlabeled data to enhance model performance and generalization [96]. In multi-robot task allocation, it blends techniques to optimize resource utilization even with limited labeled data [97].

Unsupervised Learning operates without explicit guidance, uncovering patterns in data without predefined labels, grouping similar data points together to encapsulate datasets more compactly. In multi-robot task allocation, unsupervised learning distills insights from unstructured datasets, improving resource allocation strategies [98].

Reinforcement Learning involves models making sequential decisions to achieve objectives within unpredictable settings, continuously interacting with the environment and receiving rewards or penalties to adapt and refine behaviors over time. It is promising for optimizing resource utilization and improving collaboration in multi-robot task allocation systems [99,100,101,102].

Machine learning encompasses methodologies including supervised, semi-supervised, unsupervised, and reinforcement learning, each offering distinct advantages and applications. Supervised learning excels in prediction tasks, semi-supervised learning enhances understanding with limited labels, unsupervised learning reveals insights from unlabeled data, and reinforcement learning excels in sequential decision-making settings. Each approach uniquely contributes to machine learning challenges, catering to diverse problem domains and objectives. A comparison of these approaches, considering various attributes, is presented in Table 6 and Table 7 below.

In our simulation we are using reinforcement learning with epsilon-greedy algorithm for CF. The epsilon-greedy algorithm is a commonly used strategy in reinforcement learning that balances exploration and exploitation. In our simulation, this algorithm allows the robots to make decisions about which subtasks to pursue based on their learned Q-values while still allowing for some randomness to discover potentially better actions. The action $a$ chosen by a robot (agent) is determined by:

$$a=\left\{\begin{array}{c}randomactionwithprobability\in \\ argma{x}_{a_1^{\text{'}}}Q\left(s,a^{\text{'}}\right)withprobability1-\in \end{array}\right.$$

(1)

$Q\left(s,a^{\text{'}}\right)$ is the action-value function (Q-value) for state $s$ and action $a\text{'}$ and $\in$ is the exploration probability. After taking action $a$ and observing a reward $r$ from the environment, the Q-value is updated according to the Q-learning rule.

2.5. Comparison with Different MRTA Approaches

In order to attain an optimal solution for the MRTA problem, it is imperative to take into account a range of critical factors. Effectively addressing these factors is crucial for obtaining the most favorable results in solving the MRTA problem. The comparative analysis presented in Table 8 highlights the diverse strengths and limitations of various CF methods, including behavior-based, market-based, optimization-based, and learning-based approaches. Each method exhibits distinct characteristics regarding scalability, complexity, optimality, flexibility, and robustness, influencing their applicability in real-world scenarios. Behavior-based methods, while suitable for small to moderate systems, often struggle with optimality and flexibility, primarily relying on local communication and simple task handling. Conversely, market-based methods introduce a more adaptable framework capable of accommodating complex tasks and heterogeneous robot populations, although they may encounter challenges with uncertainty and task reallocation due to their negotiation-based mechanisms.

Optimization-based approaches demonstrate superior performance in achieving optimal solutions, particularly in large systems; however, they may grapple with the complexities of multiple decision variables, impacting their robustness. Learning-based methods, particularly those utilizing reinforcement learning, emerge as a promising alternative, offering significant adaptability and improved performance through experience-based learning. Their capacity to handle dynamic environments and complex tasks positions them as a viable option for future research endeavors in CF.

Despite their individual advantages, it is crucial to consider that these methods may not perform optimally in isolation. A hybrid approach, integrating elements from multiple methodologies, could enhance overall performance by leveraging the strengths of each while mitigating their weaknesses. For instance, combining learning-based algorithms with optimization techniques may yield robust solutions capable of adapting to changing environments and task requirements. Future studies should explore these hybrid methodologies, focusing on their scalability and efficiency in dynamic, real-world scenarios, thereby contributing to the advancement of multi-robot systems and CF strategies.

3. Simulation and Results

In our simulation studies, we use a Dell laptop with an Intel Core i7 processor and 16GB of RAM to run Python-based programs and simulations. The software environment includes necessary libraries such as NumPy, Matplotlib, and custom Python scripts to visualize robot movements and coalitions. The simulations do not involve real hardware robots; instead, we focus on the behavior and coalition dynamics of virtual robots modeled in the software. All robot movements and task assignments appear in real time through graphical plots generated by the Python simulation, providing clear visualization of the coalition-based task allocation process.

We consider a team of N robots, each denoted as robot $i\in \left\{1,2,3,\dots N\right\}$ and their positions are represented by $x_i$ in the two-dimensional space ${\mathbb{R}}^2$ ,$x_i\in {\mathbb{R}}^2$. Similarly, we have objects labeled as $l\in \left\{1,2,3,\dots M\right\}$ and their positions, velocities, and desired positions are represented as $z_l\in {\mathbb{R}}^2,v_l\in {\mathbb{R}}^2,and{z}_l^d\in {\mathbb{R}}^2,$ respectively. Robot $i$ can observe other robots $j$ belonging to the set ${\mathcal{N}}_i$ of neighboring robots. Robot $i$ can observe neighboring robots $j\in {\mathcal{N}}_i$ and obstacle $l\in {\mathcal{L}}_i$. These observed robots and objects are selected based on their proximity to robot $i$, specifically, the $K$ nearest neighbors. We have established the following assumptions for this scenario:

• Robots are aware of the values of M (the total number of objects) and N (the total number of robots).
• Robots possess knowledge of both the current and desired positions of all M objects.
• Robots are capable to communicate with each other as needed.
• We assume that all robots are operating within a workspace where communication between robots is feasible.

We simulate a scenario with a set of 50 robots, denoted as $R=\{r_1,r_2,r_3,\dots .,r_{50}\}$, and a task, $T=\{t_1,t_2,t_3,t_4,t_5\}$, divided into five subtasks. The assignment of robots to these subtasks is based on their capabilities and the specific requirements of each subtask. Although robots are assumed to be identical, they differ in terms of their capabilities, categorized into three groups: high, medium, and low.

Each robot is randomly assigned to evaluate the temporal capabilities of the subtasks, and the optimal CF is determined using various algorithms. We consider the distance of each robot from each subtask and the time required for them to reach the nearest subtask. Since distance and time are directly related, each robot selects the nearest subtask to ensure efficient CF.

In this study, we evaluate the top-performing algorithms within each category. Specifically, we selected four algorithms: Alliance (behavioral-based), M+ (market-based), PSO (optimization-based), and Reinforcement Learning (learning-based). Our simulations involved running these algorithms through a series of tests using identical parameters, with each simulation consisting of 20 iterations.

After 20 iterations, we obtained 50 convergence matrices, each representing the distances of the 50 robots to the nearest subtask over the iterations. The convergence analysis assesses the speed and reliability with which the sequence of approximations, generated by the iterative method, converges toward the actual solution. The convergence matrix plays a crucial role in this analysis.

We deliberately chose 10 convergence matrices for each algorithm to capture a range of distances, spanning from the closest to the farthest robot-subtask configurations. Within these matrices, we analyzed the average distance, and the time required for individual robots to reach convergence with their respective subtasks. Subsequently, we compared the results from all four algorithms to draw conclusions from our findings. The next section will discuss the results of these algorithms.

3.1. Behaviour-Based: Alliance Architecture

The ALLIANCE architecture can be mathematically described in terms of robot capabilities, CF, and robot movement. Robot movement towards a subtask is governed by updating the robot's position based on the direction vector $\underset{\mathit{d}}{\to }$ which points toward the subtask, scaled by the robot's velocity $v$. This is mathematically represented as

$$\mathit{N}\mathit{e}\mathit{w} \mathit{P}\mathit{o}\mathit{s}\mathit{i}\mathit{t}\mathit{i}\mathit{o}{\mathit{n}}_{\mathit{s}}=\mathit{P}\mathit{o}\mathit{s}\mathit{i}\mathit{t}\mathit{i}\mathit{o}{\mathit{n}}_{\mathit{r}}+{\displaystyle \frac{\underset{\mathit{d}}{\to }}{\left|\underset{\mathit{d}}{\to }\right|}}\mathit{*}\mathit{v}.$$

(2)

Each robot's capability for a given subtask is denoted as ${\mathit{C}}_{\mathit{r},\mathit{s}}\in \left[\mathrm{0,1}\right]$, where ${\mathit{C}}_{\mathit{r},\mathit{s}}$ represents the capability of robot $\mathit{r}$ to perform subtask $\mathit{s}$. CF occurs when a robot's capability for a particular subtask is greater than zero, formally expressed as $\mathit{c}\mathit{o}\mathit{a}\mathit{l}\mathit{i}\mathit{t}\mathit{i}\mathit{o}{\mathit{n}}_{\mathit{s}}=\left\{\mathit{r}|{\mathit{C}}_{\mathit{r}}{,}_{\mathit{s}}>0\right\}$, where $\mathit{c}\mathit{o}\mathit{a}\mathit{l}\mathit{i}\mathit{t}\mathit{i}\mathit{o}{\mathit{n}}_{\mathit{s}}$ is the set of robots suitable for subtask $\mathit{s}$.

The implementation of the ALLIANCE architecture in our simulation involves key concepts such as robot capabilities, CF, and autonomous movement. Each robot is assigned distinct capabilities for different subtasks, which are randomly generated and stored in a $robot\_capabilities$ matrix. These capabilities represent the temporal suitability of each robot to perform a given subtask, based on sensor inputs or predefined characteristics.

CF is driven by these capabilities, where robots with non-zero capability values for a specific subtask form coalition. In the code, this process is executed by appending robot IDs to the respective coalitions if their capability exceeds zero.

Once the coalitions are formed, the robots autonomously navigate toward the assigned subtasks. The movement is computed by calculating the direction vector between a robot's current position and the position of the closest subtask.

$$\mathit{d}\mathit{i}\mathit{r}\mathit{e}\mathit{c}\mathit{t}\mathit{i}\mathit{o}\mathit{n}=(\mathit{b}\mathit{e}\mathit{s}{\mathit{t}}_{\mathit{s}\mathit{u}\mathit{b}\mathit{t}\mathit{a}\mathit{s}{\mathit{k}}_{\mathit{p}\mathit{o}\mathit{s}\left[0\right]}}-\mathit{r}\mathit{o}\mathit{b}\mathit{o}{\mathit{t}}_{\mathit{p}\mathit{o}\mathit{s}\left[0\right]},\mathit{b}\mathit{e}\mathit{s}{\mathit{t}}_{\mathit{s}\mathit{u}\mathit{b}\mathit{t}\mathit{a}\mathit{s}{\mathit{k}}_{\mathit{p}\mathit{o}\mathit{s}\left[1\right]}}-\mathit{r}\mathit{o}\mathit{b}\mathit{o}\mathit{t}\_\mathit{p}\mathit{o}\mathit{s}[1\left]\right)$$

(3)

This equation computes the direction vector from a robot’s current position to the position of its most suitable subtask. It does so by subtracting the robot's x and y coordinates from the corresponding coordinates of the best subtask position, selected using a function or method represented by best_(). The resulting vector indicates the direction the robot should move in order to approach the optimal subtask, based on certain criteria such as proximity or task relevance. This directional information is fundamental for navigation and task allocation, helping the robot align its movement towards the assigned goal efficiently.

This dynamic process ensures that each robot continuously updates its movement based on its capability and proximity to the task, reflecting the behavior-based control mechanisms inherent in the ALLIANCE architecture.

The initial simulation employed Alliance architecture with a fleet of 50 robots of differing capabilities, randomly assigned as high, medium, or low. Robots were directed towards five subtasks, selected based on proximity. Over 20 iterations, the results shown in Figure 8 illustrate efficient coalition formation (CF) using the Alliance algorithm. All 50 robots were effectively allocated to the nearest subtasks, promoting active participation and rapid task accessibility. Figure 8 displays the CF outcome after 20 iterations.

The alliance architecture facilitates the convergence of multiple robots into a single point to form a coalition through flexible and decentralized coordination. Robots autonomously organize into alliances based on shared objectives, capabilities, and task compatibility. This adaptability allows robots to dynamically identify suitable partners with aligned interests and abilities, leading to a common goal or convergence point. Consequently, the alliance architecture promotes efficient cooperation among diverse robots, enabling effective collaboration in achieving shared tasks or converging to specific points in their environment, fostering robust and scalable multi-robot systems.

3.2. Market-Based: M+ Algorithm

In our simulation, the M+ algorithm is applied to create optimal CF for robots tasked with subtasks in a warehouse environment. The temporal capabilities of robots, stored in a matrix, are used to rank robots for each subtask. The top-performing robots are selected to form coalitions based on their capabilities. Coalition size is determined by a square root function, simulating optimal task distribution. Once assigned, the robots navigate toward their designated subtasks by calculating the shortest distance and updating their positions iteratively. This code implementation reflects the core mechanics of the M+ algorithm, focusing on distributed decision-making and market-based CF.

Each robot $r$ is assigned a capability score $C_{r,s}\in \left[\mathrm{0,1}\right]$, for each subtask $s$. This score represents the robot's ability to effectively contribute to a specific subtask, and it plays a critical role in determining the robot's eligibility to join a coalition for that subtask. In our implementation, the robot capabilities are represented by a matrix where each element is randomly generated using the following expression:

$${\mathit{C}}_{\mathit{r}}{,}_{\mathit{s}}=\mathit{r}\mathit{a}\mathit{m}\mathit{d}\mathit{o}\mathit{m}.\mathit{u}\mathit{n}\mathit{i}\mathit{f}\mathit{o}\mathit{r}\mathit{m}(0.1,1)$$

(4)

where $C_r{,}_s$ denotes the capability of robot $r$ for subtask $s$, and its value is uniformly drawn from the interval [0.1, 1], ensuring variability in robot performance across subtasks.

$$coalitionsize=\mathrm{m}\mathrm{i}\mathrm{n}(N,|\sqrt{N}\left|\right)$$

(5)

The algorithm selects robots with the highest capabilities to form a coalition for each subtask. The size of the coalition is determined by a function related to the total number of robots N, which approximates the optimal number of robots per coalition. The coalition size is computed as $\left|\sqrt{N}\right|$, reflecting the square root rule commonly used in distributed task allocation.

For each subtask $s$, the robots are ranked based on their capability scores ${\mathit{C}}_{\mathit{r},\mathit{s}}$ and the top-performing robots are selected to form the coalition:

$$coalitio{n}_s=\left\{r|C_r{,}_s\ge threshold\right\}$$

(6)

In our code, the coalitions are formed by selecting the top $\left|\sqrt{N}\right|$ robots from the sorted list of robots based on their capabilities.

Once the coalitions are formed, each robot in a coalition is assigned a specific subtask and moves toward the location of the subtask. The distance $d$ between robot $r$ and subtask $s$ is calculated as:

$$d=\sqrt{{\left(x_s-x_r\right)}^2+{\left(y_s-y_r\right)}^2}$$

(7)

The movement of each robot is governed by minimizing the Euclidean distance between the robot's current position ${(x}_r,y_r)$ and the target subtask's position $(x_s,y_s)$.

The algorithm iteratively updates the robot's position by moving it along the vector direction towards the subtask, ensuring that the robot gradually reduces its distance to the target subtask.

In the simulation with M+ algorithm, we employed the same 50 robots with varying capabilities, randomly assigned as high, medium, or low. These robots were tasked with moving towards 5 subtasks, with their selection based on their current proximity to the subtasks. We conducted a total of 20 iterations, and the conclusive outcome is presented in Figure 9.

The difference between alliance architecture and M+ in CF, concerning their ability to converge to a common point or task, lies in their fundamental mechanisms. Alliance architecture emphasizes a decentralized, self-organizing approach where agents form alliances based on shared interests and capabilities. Convergence in this context relies on the agents' ability to autonomously identify suitable partners and form alliances, gradually moving the system towards a desired task or point of convergence through distributed decision-making. On the other hand, M+ utilizes mathematical programming to globally optimize the allocation of tasks to agents, explicitly defining an objective function. Convergence in M+ occurs through the solution of this optimization problem, aiming for an efficient task allocation. While alliance architecture offers flexibility in dynamic environments, M+ may excel in optimizing task allocations but could be less adaptable to changing circumstances due to its reliance on a fixed optimization framework.

3.3. Optimization-Based: PSO Algorithm

In PSO, each "particle" represents a potential solution, in our case, the position of each robot in the warehouse. The initial positions of robots are randomly generated within the dimensions of the warehouse:

$$x_i\left(0\right)=\left(x_i\left(0\right),y_i\left(0\right)\right),i=\{1,2,3,\dots N\}$$

(8)

$N$ is the number of robots, and $X_I\left(0\right)$ represents the initial position of robot $i$ in the 2D-space. The velocities of the particles are also initialized randomly, and the goal of the algorithm is to iteratively update these velocities and positions to optimize the assignment of robots to subtasks. The objective function that our PSO is optimizing is the ratio of each robot's capability for a subtask to its distance from that subtask. The score $S_i$ of each particle (robot) is defined as:

$$S_i={\sum }_{S=1}^M{\displaystyle \frac{C_i{,}_s}{d_i{,}_s}}$$

(9)

$C_i{,}_s$ is the capability, $d_i{,}_s$ is the Euclidean distance between the position of robot $i$ for subtask $s$, and $M$ is the total number of subtasks. The score is maximized when robots with higher capabilities are closer to their respective subtasks. Each robot maintains its personal best position $P_i,$ means the position where it had the highest score in the past):

$$P_i\left(t\right)=\mathrm{a}\mathrm{r}\mathrm{g}{\max }_i{S}_i\left(t\right)$$

(10)

There is also a global best position $g$ that corresponds to the highest score achieved by any robot across the swarm:

$$g\left(t\right)=\arg {\max }_t{S}_i\left(t\right)$$

(11)

In the PSO algorithm, the velocity and position of each particle, representing a robot, are updated through three key components: inertia, cognitive, and social terms. The inertia term maintains the particle's current momentum, ensuring it continues in its present direction, with the velocity update defined as $v_i\left(t+1\right)=w*v_i\left(t\right)$, where $w$ represents the inertia weight. The cognitive term directs the particle towards its personal best position, $p_i\left(t\right)$, by adjusting its velocity based on the difference between its current position and its best-known position, represented as $c_1{r}_1(p_i\left(t\right)-x_i\left(t\right))$, where $c_1$ is the cognitive weight and $r_1$ is a random scalar between 0 and 1. The social term steers the particle towards the global best position, $g\left(t\right)$, using $c_2{r}_2(g\left(t\right)-x_i\left(t\right))$, where $c_2$ is the social weight and $r_2$ is a random scalar. The overall velocity update equation combines these components as $v_i\left(t+1\right)=w*v_i\left(t\right)+c_1{r}_1\left(p_i\left(t\right)-x_i\left(t\right)\right)+c_2{r}_2(g\left(t\right)-x_i\left(t\right))$, and the particle's position is subsequently updated using $x_i\left(t+1\right)=x_i\left(t\right)+v_i(t+1$).

In this simulation, we applied the PSO algorithm to a fleet of 50 robots with diverse capabilities, randomly classified as high, medium, or low. The robots were tasked with approaching five subtasks, with selection based on their relative distances. Over 20 iterations, the results indicate that the PSO algorithm prioritized the first subtask, allocating all robots towards its completion shown in figure 10. This outcome highlights PSO's strength in converging on near-optimal solutions; however, our objective is to accelerate task completion by enabling simultaneous engagement across all subtasks to minimize total task time.

PSO differs from the Alliance Architecture and the M+ algorithm in CF, particularly in its approach to converging on a common point or task. PSO is a heuristic optimization technique inspired by the collective behavior of birds or fish in a swarm. In CF, PSO models’ agents as particles exploring a solution space to find the optimal task allocation. While the Alliance Architecture and M+ emphasize decentralized coordination and explicit optimization, PSO uses a population-based search mechanism. It doesn't form traditional coalitions but directs agents (particles) towards optimal solutions. PSO's convergence is driven by the optimization process, with particles adjusting their positions based on their experiences and those of their neighbors. This stochastic exploration can effectively find near-optimal solutions, especially in continuous solution spaces, but it may lack the transparency and adaptability of the Alliance Architecture or M+ in multi-agent systems with evolving goals and capabilities.

Figure 10. Coalition formation using PSO.

Figure 10. Coalition formation using PSO.

3.4. Learning-Based: Reinforcement Learning

In the proposed reinforcement learning framework for multi-robot CF, the epsilon-greedy algorithm is utilized to strike a balance between exploration and exploitation during the decision-making process. Each robot maintains a Q-value table $Q(s,a)$, where $s$ represents the current state of the robot, and $a$ represents the actions associated with available subtasks. To select an action, the robot employs the epsilon-greedy strategy, wherein it explores a random action with probability $\in$ and exploits the action with the highest Q-value with probability $1-\in$. This dual approach facilitates the discovery of optimal strategies while ensuring that previously identified effective actions are also utilized.

The Q-value updates are performed following the Q-learning rule, allowing the robots to refine their strategies based on the rewards received from the environment. Specifically, after executing an action and observing a reward, the robot updates its Q-value for the taken action using the formula:

$$Q\left(s,a\right)\leftarrow Q\left(s,a\right)+\alpha [r+\gamma maxQ\left(s^{\text{'}},a^{\text{'}}\right)-Q\left(s,a\right)]$$

(12)

Here, $\alpha$ represents the learning rate, which controls the extent to which new information influences the existing Q-values, while $\gamma$ denotes the discount factor, balancing immediate versus future rewards. This method allows for an effective adaptation of the robots' actions in pursuit of subtasks, enhancing their cooperative performance in the CF process.

In our simulation employing the Reinforcement Learning (RL) algorithm, we utilized the same 50 robots with varying capabilities, randomly assigned as high, medium, or low. These robots were tasked with moving towards 5 subtasks, selecting based on their proximity to the subtasks. We conducted a set of 20 iterations, with the final results displayed in Figures 11a and 11b. A notable limitation of RL is its prolonged learning duration, hindering convergence within the initial 20 iterations. To address this, we extended the number of iterations to a range of 80 to 100, as demonstrated in Figure 11c.

RL differs significantly from alliance architecture, M+, and PSO algorithms in CF due to its agent-centric, trial-and-error learning paradigm. RL agents make decisions based on learned policies, and convergence occurs through individual agent learning. While alliance architecture and M+ emphasize decentralized coordination and explicit CF, RL agents act autonomously to maximize cumulative rewards, often without explicit coalition considerations. Conversely, PSO aims to converge on optimal solutions collectively. RL introduces adaptability and autonomy at the individual agent level, allowing agents to dynamically adapt to changing circumstances and goals. The choice between these approaches depends on factors such as problem complexity, collaboration requirements, desired autonomy and adaptability in the multi-robot system.

From the results depicted in Figure 11, it is evident that RL exhibits behavior similar to alliance architecture, indicating its significant value.

3.5. Statistical Analysis

Figure 12 depicts behavioral bar charts for a set of 10 chosen robots. These charts illustrate the utilization of alliance, M+, PSO, and reinforcement learning algorithms. In the charts, we can observe blue bars representing the average distance covered by each robot, while the yellow bars represent the average time required for convergence. Notably, the starting position of R1 is in proximity to the subtask, whereas R10's initial position is considerably distant from the subtask.

The results make it evident that robots employing alliance and reinforcement learning algorithms exhibit superior performance. However, it's worth noting that the reinforcement algorithm, due to its learning attributes, takes more time to accomplish the task compared to the alliance approach. Yet, as the number of robots increases, the performance of the alliance architecture starts to deteriorate, and this is when reinforcement learning shines. If we can mitigate the time constraints associated with reinforcement learning, it would emerge as the optimal approach for CF and multi-robot task allocation.

In the analysis of CF algorithms, the four approaches, Alliance (behavior-based), M+ (market-based), PSO (optimization-based), and reinforcement learning (Learning-based), exhibit distinct methodologies and effectiveness in promoting collaborative behavior among autonomous agents. The Alliance algorithm emphasizes behavioral dynamics, enabling robots to form coalitions based on mutual capabilities and situational awareness. In contrast, the M+ algorithm leverages market mechanisms to allocate tasks, optimizing efficiency through competitive bidding among agents. The PSO approach, rooted in swarm intelligence, focuses on optimizing coalition structures based on collective experience and positioning, promoting swift convergence towards optimal solutions. While both the Alliance and RL methods yield promising results in CF, the reinforcement learning approach emerges as a robust framework for future studies. This is primarily due to its adaptive learning capabilities, which allow robots to continuously improve their decision-making processes through trial and error, leading to enhanced performance in dynamic environments.

Despite the preliminary findings indicating effective results from the Alliance and RL algorithms, the inclusion of reinforcement learning is particularly compelling for future investigations. It offers the potential for self-optimization and adaptability, accommodating varying task demands and environmental conditions that may not be fully captured by the other methods. Moreover, the comparative analysis conducted through convergence matrices of 10 robots showcases the efficiency of these algorithms, highlighting RL's capability to converge toward optimal coalition structures over time. Such insights pave the way for further exploration of RL in CF, potentially leading to more resilient and flexible robotic systems capable of adapting to complex and evolving scenarios.

4. Discussion

Upon reviewing the literature, it becomes evident that learning-based methods hold the potential to adapt and attain a truly optimal solution for the Multi-Robot Task Allocation (MRTA) problem. Each of these methods, including behavior-based, market-based, and optimization-based approaches, may excel in certain aspects, while others might fall short. Nevertheless, learning-based approaches, when applied to the MRTA problem, have the capability to offer solutions, even when dealing with large teams. When analyzing algorithms against mTSP, VRP, JSP, TOP and DARP; they are not scaled well to many robots or locations, and they may struggle with real-time adaptation to dynamic environments except learning-based algorithms.

Learning-based MRTA approaches are adept at tackling exceedingly intricate problems that conventional techniques struggle to address. Previous research has underscored the competitive nature of reinforcement learning in achieving long-term outcomes in this context. Furthermore, it possesses the capability to rectify errors that may have arisen during the training phase and can adapt when confronted with an insufficient quantity of training data, drawing insights from its experiences. Notably, one of the primary strengths of reinforcement learning lies in its capacity to strike a balance between exploration and exploitation. Exploration involves the testing of novel ideas to discover potential superior solutions, while exploitation entails leveraging the strategies that have previously proven to be effective. The primary challenge associated with learning-based Multi-Robot Task Allocation (MRTA) is the time required for training. Achieving a high number of reward points necessitates a substantial number of trials, which can be time intensive.

Interestingly, an intriguing avenue for addressing the limitations of learning based MRTA is to combine these approaches with one or more alternative methods that can mitigate their drawbacks. Such hybrid approaches hold the promise of yielding viable solutions to the MRTA problem. Additionally, more intricate learning techniques, such as deep reinforcement learning or deep neural networks, have the potential to enhance performance. Moreover, there exists ample room for future research and development in the realm of learning based MRTA, offering exciting prospects for further advancements.

The future research will primarily focus on addressing these challenges, particularly partial observability, and non-stationarity in CF, using a learning-based approach.

5. Conclusion

CF plays a critical role in enhancing the efficiency and effectiveness of MRTA systems. Through an in-depth examination of existing strategies, we highlight that while various methods such as behaviour-based, market-based, optimization-based approaches and learning based offer viable solutions, each comes with its own advantages and limitations. The comparative analysis emphasizes that no single method fits all scenarios; rather, the effectiveness of a CF strategy is heavily dependent on the nature of the task, the capabilities of the robots, and the dynamic environment in which they operate. Our simulation results affirm that strategic grouping of robots based on proximity, capability, and task requirements leads to more reliable and faster convergence toward optimal task execution.

In conclusion, this paper not only provides a structured overview of current MRTA approaches with a focus on CF but also proposes a classification system to better understand and compare these methodologies. The introduced simulation-based evaluation framework further substantiates the potential of the proposed CF approach in solving complex, real-world MRTA problems. Future work may explore hybrid models that dynamically switch between strategies based on environmental changes and robot feedback, paving the way for more adaptive, scalable, and intelligent multi-robot systems.