Vocal Communication between Cobots and Humans to Enhance Productivity and Safety: Review and Discussion

1. Introduction

Cobots operate in shared workspaces with humans, necessitating seamless interaction with their human counterparts [1] (Faccio & Cohen, 2023). This interaction goes beyond mere task execution; it involves a dynamic exchange of information, coordination of efforts, and mutual understanding to optimize the synergies between human skills and robotic precision [2] (Faccio et al. 2023).

Effective communication and collaboration between human workers and cobots are pivotal for maximizing the benefits of cobot integration. The need arises from the intricacies of shared tasks, where human operators and cobots must coordinate their actions to achieve optimal results [3] (Liu et al. 2024). Moreover, as cobots are designed to be more adaptable and responsive to changes in the manufacturing environment, clear communication becomes the essential for successful task allocation, problem-solving, and overall operational efficiency [4] (Papetti et al. 2023). Effective communication fosters a sense of cooperation, trust, and shared purpose, thereby enhancing the overall productivity and safety of the collaborative work environment [5] (Gross & Krenn, 2023).

This paper focuses on human-cobot communication related to cobots’ assistive functions. Vocal communication can free a worker's hands and eyes, allowing them to perform tasks while receiving information or instructions through audio support [6] (Moore & Urakami, 2022). This approach is preferred over augmented reality or head-mounted display-based support in many application areas today [7] (Marklin et al. 2022). So, for assisting a worker performing manufacturing or assembly tasks, the vocal and audio communications are the preferred over other communication forms

This paper delves into strategies for optimizing vocal communication and collaboration between human workers and cobots. It shows that these strategies enhance productivity and safety in assistive work. This is its contribution to the transformative role of collaborative robotics in modern manufacturing. By addressing this critical nexus, we aim to guide future research endeavors and industry practices toward a better collaboration of human and cobots on the factory floor.

2. Literature Review

Human-cobot communication plays a crucial role in task allocation and safety in collaborative assembly systems [8,9] (Heydaryan et al. 2018; Petzoldt et al. 2023). Schmidbauer et al. (2023) [10] also examined static vs. dynamic task allocation preferences of human workers and found that they preferred adaptive task sharing (ATS) to a static predetermined task allocation and reported increased satisfaction with the dynamic allocation. Workers are more likely to assign manual tasks to cobots, while preferring to handle cognitive tasks themselves [10] (Schmidbauer et al. 2023). These dynamic characteristics of collaboration require good communication between humans and cobots. The cobot-human communications could take one of the following forms:

• Text communications: The human user uses text for programming the cobot, and to send it commands, and information [11] (Jacomini et al. 2023) the cobot may also use text to communicate messages, explanations, cautionary warnings and other information types [12,13] (Scalise et al. 2017; Kontogiorgos, 2023).
• Visual communications: Graphical displays, augmented reality (AR), and vision systems are key components of visual interfaces that enable workers to comprehend and interpret information from cobots efficiently [14,15] (Zieliński et al. 2021; Pascher et al. 2022). Graphical displays can provide real-time feedback on cobot actions, task progress, and system status, enhancing transparency and situational awareness [16] (Eimontaite et al. 2022). Augmented reality overlays digital information onto the physical workspace, offering intuitive guidance for tasks and aiding in error prevention [17] (Carriero et al. 2023). Vision systems, equipped with cameras and sensors, enable cobots to recognize and respond to human gestures, further fostering natural and fluid interaction [18] (Sauer et al. 2021).
• Auditory communications: Human-cobot vocal communication has been a topic of intensive research in recent years [19] (Ionescu & Schlund, 2023). Auditory cues are valuable in environments where visual attention may be divided or compromised [20] (Turri et al. 2021). Sound alerts, spoken instructions, and auditory feedback mechanisms contribute to effective communication between human workers and cobots [21] (Su et al. 2023). For instance, audible signals can indicate the initiation or completion of a task, providing workers with real-time information without requiring constant visual focus [22] (Tran, 2020). Speech recognition technology enables cobots to understand verbal commands, fostering a more intuitive and dynamic interaction [23] (Telkes et al. 2024). Thoughtful use of auditory interfaces between humans and cobots, helps create a collaborative environment where information is conveyed promptly, enhancing overall responsiveness and coordination [24] (Salehzadeh et al, 2022). Several recent papers have proposed novel interfaces and platforms to facilitate this type of interaction. Rusan and Mocanu (2022) [25] introduced a framework that detects and recognizes speech messages, converting them into spoken commands for operating system instructions. Carr, Wang, and Wang (2023) [26] proposed a network-independent verbal communication platform for multi-robot systems, which can function in environments lacking network infrastructures. McMillan et al. (2023) [27] highlighted the importance of conversation as a natural way of communication between humans and robots, promoting inclusivity in human-robot interaction. Lee et al. (2023) [28] conducted a user study to understand the impact of robot attributes on team dynamics and collaboration performance, finding that vocalizing robot intentions can decrease team performance and perceived safety. Ionescu & Schlud (2021) [29] found that voice activated cobot programming is more efficient than typing and other programming techniques. These papers collectively contribute to the development of human-robot vocal communication systems and highlight the challenges and opportunities in this field.
• Tactile communications: The incorporation of tactile feedback mechanisms enhances the haptic dimension of human-cobot collaboration [30] (Sorgini et al. 2020). Tactile interfaces, such as force sensors and haptic feedback devices, enable cobots to perceive and respond to variations in physical interactions [31] (Guda et al. 2022). Force sensors can detect unexpected resistance, triggering immediate cessation of movement to prevent collisions or accidents [32] (Zurlo et al. 2023). Haptic feedback devices provide physical sensations to human operators, conveying information about the cobot's state or impending actions [33] (Costes & Le-cuyer, 2023). This tactile dimension contributes to a more nuanced and sophisticated collaboration, allowing for a greater degree of trust and coordination between human workers and cobots.

Human-cobot communication plays a crucial role in task allocation and safety in collaborative assembly systems [34,35] (Keshvarparast et al 2023; Liu et al. 2024). Schmidbauer et al. (2023) also examined static vs. dynamic task allocation preferences of human workers and found that they preferred adaptive task sharing (ATS) to a static predetermined task allocation and reported increased satisfaction with the dynamic allocation. Workers are more likely to assign manual tasks to cobots, while preferring to handle cognitive tasks themselves [10] (Schmidbauer et al. 2023).

Human-robot communication can involve various modes, including verbal communication using speech, multimodal interaction involving head movements, eye gaze, and pointing gestures, and nonverbal cues such as facial expressions and hand gestures [27,36] (McMillan et al. 2023; Schreiter et al. 2023). Multimodal interaction, which combines different modalities, leads to more natural fixation behavior, improved engagement, and faster reaction to instructions in collaborative tasks [37] (Rautiainen et al. 2022). In addition to verbal communication, nonverbal cues play a crucial role in human-robot interaction, allowing for a more natural and inviting experience [38] (Urakami & Seaborn, 2023). The concept of multimodal interaction in human-robot applications acknowledges the synergistic use of different interaction methods, enabling a richer interpretation of human-robot interactions [39] (Park et al. 2021). However, multimodal interaction is not only very costly, it also take the human attention off the task at hand [39,40] (Park et al. 2021; Nagrani et al. 2021).

3. Main Assistive Scenarious

Cobots can assist an industrial worker in many ways and in many types of tasks. Based on the literature, we define main scenarios that cover large part of these tasks (Javaid et al. 2022) [41] , as follows:

• Standard pick and place (known locations): there may be several different such tasks, and their names and locations should be clearly defined for the human and cobot.
• Fetch distinct tool/part/material: visual search may be needed for identifying the object, and corresponding grasping strategy.
• Return tool/part/material: visual search may be needed for identifying the placing location, and corresponding reach and align strategy.
• Dispose defective tool/part/material: Identifying the correct dispose location is necessary.
• Standard “Turn” object”: Identifying the object and pre-defining the meaning of “Turn” may be necessary.
• Turn Object degrees counter/clockwise: Identifying the object for planning the reach, grasp & turn trajectory.
• Move and align
• Drill: location must be pre-defined
• 9Screw: location must be pre-defined
• Solder/glue-point : location must be pre-defined
• Inspect: : location must be pre-defined
• Push/Press: location must be pre-defined
• Pull/Detach: location must be pre-defined
• Hold (reach and grasp): Identifying the object for planning the reach, &grasp trajectory.

These 14 scenarios could benefit form the strategies described in section 4.

4. Main Strategies for Optimizing Vocal Communication

In this section we identify various strategies the enhance vocal cobot related communication.

The relationship between these strategies and the 14 scenarios (from section 3) are summarized in Table 1. Our proposed strategies are listed as follows:

• Workstation map: generate a map of the workstation with location identifying labels of various important points that the cobot arm may need to reach. Store the coordinates of each point in the cobot’s control system and hang the map in front of the worker.
• Dedicated space for placing tools and parts (for both the worker and the cobot): Dedicate a convenient place, close to the worker, for the cobot to place or take tools or parts or materials that the worker asked for. This place could serve several tools or parts to be placed from left to right and top to bottom. Store the coordinates of the place in the cobot’s control system, and the worker must be informed about tis place and understand the expected trajectory of the cobot for reaching this place.
• Dedicated storage place for tool/part: Dedicate a unique storage place for each tool and for part supply, that would be easy to reach for both the human worker and the cobot.
• Define names for mutual use: Make sure both cobot and workers are using the same name for each tool and for each part.
• Define predefined trajectories of the cobot from tool/part areas to the placement area. Thus, the worker knows what movements to expect from the cobot.

These strategies generate a common language for the cobot and the worker, related to locations and objects. They create both clarity and conciseness. Moreover, the trajectory strategy creates worker understanding and anticipation for the cobot moves.

Note that these five strategies appear in Table 1. In abbreviated form as column titles. Each column abbreviated title is explained in the legend under Table 1.

5. Discussion

The results presented in Table 1 underscore the significance of clear communication channels between human workers and cobots in collaborative work environments. Defining names for places, objects, actions, and trajectories emerges as a pivotal factor for efficient communication bidirectionally. The cobot's ability to convey predetermined path trajectories to human workers, elucidating where and how movements are expected to unfold, proves crucial and applicable across all cobot activities. The strategy of constructing a comprehensive workstation map, delineating places and objects for both human and cobot, is identified as another crucial element for successful communication.

Furthermore, the naming strategy generates a common language, fostering a shared understanding of the work environment between human and cobot. This shared understanding is pivotal for the optimization of cobot-assisted tasks. Likewise, the predefined trajectories strategy not only aids in clarity but also facilitates worker anticipation of cobot movements, thereby enhancing overall collaboration efficiency.

The strategies outlined in Table 1 collectively contribute to a structured and efficient communication framework between human workers and cobots. They establish clarity and conciseness, mitigating potential misunderstandings and streamlining the collaborative process. Moreover, these strategies create a sense of order and predictability in the shared workspace, further enhancing the safety and productivity of the collaborative work environment.

While computerized retrieval of names of places and objects, maps, and trajectories is an instantaneous reliable action, the human worker has to rely on his/her memory. This means that worker’s training must precede the work with a cobot. The importance of comprehensive training programs for human workers is evident, ensuring a smooth transition to working alongside cobots. As the data in Table 1 emphasizes, a well-defined communication protocol is crucial, and worker training is integral to this process. Comprehensive training programs should cover not only technical aspects but also the nuances of communication and collaboration in a human-cobot setting.

Augmented Reality (AR) applications could be a preferred alternative to workstation maps (or part of such map) and emerge as promising tools for communication and collaboration. AR could also be used to simulate collaborative scenarios and prepare workers for real-world interactions with cobots. AR technology provides immersive training experiences that can enhance worker preparedness and familiarize them with cobot-assisted tasks in a controlled, virtual environment.

Safety measures are of paramount importance in human-cobot collaboration. Addressing concerns such as collision avoidance, force sensing, and emergency stop mechanisms is crucial to prevent accidents and ensure a secure working environment. Implementing safety standards and regulations is essential for creating guidelines that govern human-cobot interaction, promoting a safe and productive collaborative workspace.

In conclusion, this discussion highlights the critical role of effective communication and collaboration in human-cobot interactions. The identified strategies provide practical insights for optimizing vocal communication, ultimately enhancing productivity and safety in cobot-assisted tasks. Continuous research and development efforts are vital to staying abreast of evolving technologies and refining communication strategies, paving the way for successful integration of cobots into assembly processes. This manuscript contributes to this dynamic field by providing a comprehensive exploration of strategies and insights, serving as a guide for future research and industry practices in the realm of collaborative robotics.

6. Conclusions

In summary, this study illuminates the pivotal role of vocal communication in augmenting productivity and safety in cobot-assisted tasks. The identified strategies, emphasizing clarity through defined names and trajectories, establish a shared language that enhances task efficiency and fosters a secure working environment. Acknowledging the significance of comprehensive worker training and the potential of Augmented Reality (AR) applications, this research underscores the need for a holistic approach to human-cobot collaboration.

This manuscript contributes to the evolving field of human-robot collaboration by providing a comprehensive exploration of strategies to enhance productivity and safety in assembly environments. By synthesizing theoretical frameworks, practical recommendations, and empirical evidence, this work aims to guide future research and industry practices in the dynamic landscape of collaborative robotics.

Future research should delve into refining communication protocols, exploring advanced technologies for immersive worker training, and addressing evolving safety concerns. Investigating the impact of these strategies in diverse industry settings and scaling up their applicability will contribute to the continual evolution of collaborative robotics. As technology advances, understanding the dynamics of human-robot interactions remains crucial for unlocking the full potential of cobots in Industry 5.0. This study serves as a foundation, urging researchers and practitioners to further explore and implement innovative solutions for a seamless integration of cobots into modern manufacturing.