Human Factors Considerations for Quantifiable Human States in Physical Human-Robot Interaction: A Literature Review

1. Introduction

As the global population continues to age rapidly due to increased life expectancy and declining birth rates, the demand for healthcare services and caregivers for older adults is growing [1]. In Japan, over 30% of the population is already over 60, while it is projected that by 2050, one in four individuals in North America and Europe could be aged 65 or older [2]. From 2015 to 2050, the proportion of the global population aged over 60 years is estimated to almost double from 12% to 22% [1].

As a result, there is an increasing demand for solutions that enhance the quality of life and independence of older adults to address the looming shortage of caregivers [3]. To achieve this, the Society 5.0 vision aims to introduce robots as domestic service providers to assist with daily living activities and create novel values in healthcare and caregiving domains [4]. Given that caregiving is closely associated with physical interaction tasks in daily activities such as showering, eating, toileting, and transferring [5], robots introduced into these application domains must be capable of efficient physical support. It is noteworthy that physical support does not involve physical assistance only, but also emotional support through physical comfort such as patting or hugging.

To ensure the successful integration of robots into these fields, a practical approach would be to equip them with the capability to provide multiple services instead of limiting them to a single task (e.g. a robot that only cleans the floor, or only performs feeding), i.e., robotic multiple service providers. The success of this approach has been demonstrated by the widespread of cellphones and computers which has proven that technology can be harnessed to enhance our daily lives by providing a range of functions through a single device. A crucial characteristic of multiple service robots is the possession of dexterity and manipulation capabilities, as it enables them to perform various tasks efficiently, similar to how humans operate.

Furthermore, a seamless assimilation of robots would require them to develop a social understanding of humans, which will help robots make decisions according to their human partners’ state, such as their level of anxiety, emotional state, physical comfort, social distancing or mental workload. This is anticipated to endow robots with the ability to engage with humans in a socially acceptable manner. As a first step towards achieving this goal, robots need to possess the capability of predicting the human state, i.e. quantification of human factors. It’s worth mentioning that this goal is part of a comprehensive vision to develop adaptive control frameworks that can make optimal decisions by taking into account not only the human state but also environmental conditions and a collaborative task goal.

First and foremost, to aid in the development of multiple service robots, it’s necessary to identify research efforts that have: 1) examined quantifiable and measurable factors that can represent the human state, 2) addressed efficient physical interaction, and 3) exploited robots with strong manipulation capabilities. To this end, our goal in this manuscript is to review studies that have quantifiably assessed human factors in physical human-robot interaction (pHRI) scenarios, with a specific focus on robots that are capable of manipulation, meaning those equipped with multi-degree-of-freedom arm(s).

The definitions of pHRI and related human factors in the literature are often ambiguous. The literature on human factors in human-robot interaction (HRI) is characterized by great diversity in its usage and a lack of agreement on unified definitions or terminologies, as noted in [6]. For this review, we characterize human factors as data drawn from humans that can quantifiably interpret the human state to be used in a human-in-the-loop control framework, such as trust, anxiety, mental workload, physical comfort, and perceived safety. As far as the authors are aware, no previous reviews have focused on investigating the studies that have specifically addressed such human factors considerations in the context of pHRI.

Similar to the ambiguity surrounding the definition of human factors, the field of pHRI also grapples with the difficulty of reaching a consensus on a unified definition. In [7], the authors define pHRI as the domain that focuses on the physical aspects of the interaction between humans and robots in a shared workspace, including factors such as robot speed and distance. Whereas in [8], pHRI referred to a broad area of applications that include cooperation, assistance, teleoperation, and entertainment. In [9], the authors categorized the physical connection between humans and robots as either proximally, where there is an exchange of forces between parties, i.e., physical coupling, or remotely, referring to teleoperation. Further restrictions on the area that pHRI refers to are adopted by [10], where authors considered only physical coupling (direct physical contact) between humans and robots, including those mediated by an object. Building upon the perspective of the field presented in [10] on pHRI, we define indirect pHRI as physical coupling that is mediated by an object and direct pHRI as physical coupling that occurs directly between parties without object mediation.

To gain a comprehensive understanding of our topic of interest, it is imperative to discern research gaps as well as evaluate the advancements achieved in the literature. In pursuit of this, we adopted a novel holistic conceptual model for human factors classification in HRI [11], which enabled us to provide an unbiased overview of the most commonly studied human factors and the approaches used to quantify them. However, to facilitate deeper analysis and enhance our understanding of these factors, we first identified the various contexts in which these human factors have been investigated in pHRI, which can serve as a potential mapping scheme for the field. While this review does not aim to provide a strict systematic literature review, it offers an objective summary of the progress made in human factors in pHRI over the last 15 years. The purpose of this summary is to assist in accelerating the development of a streamlined adaptive control framework for multiple service robots.

1.1. Paper Organization

The rest of this review article is organized as follows:

• Section 2, Related Work, presents literature reviews with similar objectives to ours, but with different focus areas.
• Section 3, Research Questions, presents the research questions guiding this review.
• Section 4, Methodology, presents the reviewing process used to develop a potential mapping scheme for the pHRI field and human factors classification.
• Section 5, Results, presents the classification results, which provide answers to the research questions formulated and identify trends and gaps in the literature.
• Section 6, Discussion, proposes suggestions for filling the identified literature gaps.
• Section 7, Limitations of the review, addresses the validity and limitations of this review.
• Section 8, Conclusion, provides a summary of the key findings and their implications for future research in pHRI.

2. Related Work

Remarkable efforts have been expended to comprehensively review HRI studies pertaining to human factors. Hopko et al. [12] conducted a systematic literature review, where they assessed the most frequently studied human factors and quantification methods used in industrial shared spaces. Their findings revealed that trust, cognitive workload, and anxiety were the most commonly investigated human factors. While Hopko et al. dealt with perceived safety as a distinct human factor, Rubagotti et al. [7] incorporated related human factors, such as trust, anxiety, and comfort, as sub-factors under the umbrella of perceived safety. Their primary objective was to understand how physical factors, such as distance and robot speed, and different types of robots, such as industrial manipulators, mobile robots, and humanoids, can influence the perception of safety. Similarly, Ortenzi et al. [13] took a holistic approach to address different human factors by utilizing "user experience" as an umbrella term instead of examining each factor separately. Their study focused on reviewing efforts made in handovers between humans and robots. Likewise, Prasad et al. [14] examined previous studies in handshake scenarios to identify the most commonly addressed human factors. Coronado et al. [11] conducted a literature review to provide an overview of the quantification methods used in HRI industrial applications to evaluate performance and measurable dimensions of human factors. Their work resulted in a taxonomy of performance aspects and a holistic conceptual model for the most commonly addressed human factors in HRI. Their review also revealed that, after physical safety, trust was the most frequently addressed human factor that influences human performance. Simone et al. [15] studied the impact of human-robot collaboration on human factors in industrial settings. Their review highlighted that successful human-robot interaction depends on several crucial factors, including trust, usability, and acceptance. Additionally, they concluded that the interaction can affect stress and mental workload either positively or negatively. Lorenzini et al. [16] conducted a narrative review to identify the most effective assessment methods for physical and cognitive ergonomics in workplace environments. Their review focused on available monitoring technologies that endow robots with the ability to adapt online to the mental states and physical needs of workers. Notably, each of the aforementioned reviews found that questionnaires were the predominant assessment approach utilized in studies across diverse scenarios.

Evidently, a dearth of research has specifically examined human factors in pHRI scenarios or investigated the studies that quantified the impact of human factors within domestic contexts. Moreover, past attempts to classify human factors in this domain have faced numerous challenges, necessitating the use of various methodological workarounds. In light of these gaps, we endeavor to provide an objective overview of the progress made in this field, in order to promote the development of adaptive control frameworks for multiple service robots. Efficiently predicting the human state is a crucial first step in this development.

3. Research Questions

Although the literature on HRI has addressed human factors considerations, there has been a notable lack of dedicated attention to pHRI as a specific area of study requiring a structured analysis of trends and developments. In this review paper, we aim to identify the different contexts in which human factors have been studied in the field, with a specific emphasis on those that involve robots with manipulation capabilities. Therefore, by creating a mapping scheme, we can help researchers navigate and better understand the field. As a result, we formulated the following research question:

• RQ1: What are the existing applications or contexts in the literature, and how can they be categorized?

To answer this question, we will identify the applications and scenarios addressed in the literature on both direct and indirect pHRI.

In Section 2, Related Work, numerous reviews have attempted to overcome the challenge of diverse terminologies used to describe various human factors. Some authors, such as Rubagotti et al. [7], have employed the term "perceived safety," while others, like Ortenzi et al. [13], have used the broader concept of "user experience" to encompass multiple human factors, without delving into specific classifications. Other reviews, like Hopko et al. [12], addressed individual human factors but did not provide clear definitions or outlines.

Similarly, classifying human factors has been a major challenge in this review due to the lack of systematic or standardized definitions in HRI literature, making it difficult to determine the most commonly addressed human factor. For example, some studies use terms like stress [17], strain [18], and frustration [19] without providing clear definitions, making it inaccurate to group them together under one umbrella term. Likewise, legibility [20], and predictability [21] of robot’s motion; and physical ergonomics [22], physical comfort [20], fatigue [23], health and safety [24] are two sets of examples of different human factors that overlap but cannot be lumped under a single comprehensive term within the set. Additionally, perceived enjoyment [19], and enjoyability [25]; and psychological safety, mental safety, and subjective safety [7] are also examples of human factors that are distinct but interconnected. We anticipate that this may have caused an unintentional research bias in favor of human factors that have limited synonyms like trust. Trust has been identified as a common human factor in several HRI reviews and has even been given special attention in dedicated reviews [26].

Furthermore, the challenge of assessing progress in human factors in HRI is exacerbated when studies used generic terms like preference [27] and enjoyment [25] without providing clear definitions for each term. This highlights the importance of defining and classifying human factors in a more precise and standardized way to avoid unintentional research bias. To address this challenge, we explored various classification schemes, such as dividing human factors into cognitive and physical ergonomics, as proposed by Lorenzini [16], or further categorizing them into load and perceptual factors. However, we found that human factors in pHRI, which involved both domestic and industrial scenarios, are more complex and diverse than these schemes can account for, as they may involve emotional factors in addition to cognitive and physical ones.

Therefore, we chose to build upon the holistic conceptual model proposed by Coronado et al. [11] for human factors in HRI. By adopting this model, we aim to develop a standardized approach that can systematically assess the progress of human factors research in HRI, including pHRI scenarios.

Coronado et al. proposed a holistic conceptual model adapted for HRI, based on [28], to reduce confusion between the concepts of usability and user experience in HRI literature. This model illustrates the relationships among usability, user experience, hedonomics, and ergonomics, and identifies four distinct human factor types: cognitive ergonomics, physical ergonomics, belief, and affect. Below we will briefly describe the definition of each concept.

User experience, according to the ISO 9241-11:2018 (ergonomics of human-system interaction) [29], includes "all user’s emotions, beliefs, preferences, perceptions, physical and psychological responses, behaviors, and accomplishments that occur before, during, and after use". Whereas, usability is considered as "the extent to which a system, product or service can be used by specified users to achieve specified goals with effectiveness, efficiency, and satisfaction in a specified context of use".

Ergonomics, which is classified into physical and cognitive ergonomics in this review, aims to mitigate injuries, pain, and suffering, while hedonomics, which is classified into belief and affect in this review, focuses on the pleasant and enjoyable aspects of interactions.

Physical ergonomics encompasses the potential adverse effects on the human body during an interaction, such as postures, repetitive movements, heavy workloads, or forces. Cognitive ergonomics is concerned with designing systems that align with the perceptual and psychological abilities of users. Affect pertains to emotional-related terms, whereas belief refers to cognitive responses that can trigger emotions (i.e., affective responses).

By utilizing this conceptual model, we anticipate addressing the deficiency of unified definitions of human factors by classifying them under one of the four human factor types. Following the naming convention presented in Coronado et al., we use the term measurable dimensions to refer to human factors that can be quantified and measured, such as trust, anxiety, and fatigue. In contrast, we use human factor types to refer to broader categories of human factors, such as cognitive ergonomics, physical ergonomics, belief, and affect. We are confident that this approach will facilitate the investigation of relationships between measurable dimensions within each human factor type. This will allow us to gain a more nuanced understanding of how different aspects of human factors interact with each other.

In order to identify the most common human factor type addressed in pHRI studies that involve robots with manipulation capabilities, we formulated the following research question:

• RQ2: What are the most commonly studied human factors that can be used to develop quantitative measures of the human state?

We hypothesize that belief will be the answer to this question, as it encompasses trust as one of its measurable dimensions, which is consistently identified as one of the most frequently addressed human factors in HRI.

Quantification of human factors based on measurable data plays a key role in the development of adaptive control frameworks that can effectively predict human states and make decisions based on them. As a result, we have formulated the following research question:

• RQ3: What are the quantification approaches employed to evaluate human factors?

To identify the quantification approaches based on measurable data, we will categorize the gathered studies into two categories based on the objectives of the studies that addressed human factors. We call this categorization Human Factor Usage which includes impact and quantification. The impact category explores the effect of manipulating robot parameters on human factors, i.e. the impact of those manipulations, while the quantification category will explore proposed quantification approaches for human factors or their correlation with measurable entities such as physical data (e.g., body posture and interaction forces) and physiological data. We hypothesize that, based on the literature, questionnaires will be the most commonly used approach for quantifying human factors in pHRI. Our objective is to identify promising quantification techniques that have been introduced or adapted specifically for pHRI.

4. Methodology

The methodology used to conduct this review follows the guidelines proposed by Petersen et al. [30], which were adapted to the context of HRI by Coronado et al. [11]. The methodology consists of several steps:

• Justifying the need for conducting a literature review, as described in Section 1, Introduction.
• Defining the research questions, as explained in Section 3, Research Questions.
• Determining a search strategy, assessing its comprehensiveness, establishing selection criteria, and conducting a quality assessment. These steps are detailed in this section.
• Extracting data from the identified studies, organizing and categorizing the information obtained, and visualizing the results. These steps are detailed in Section 5, Results.
• Addressing the limitations of the review by conducting a validity assessment, as described in Section 7, Limitations of the Review.

4.1. Search Strategy

Given that pHRI is a relatively young field and the overall population is not well-known beforehand, conducting a comprehensive database search for human factors is challenging. Unlike more established fields, the literature on pHRI is not yet fully developed, which makes it harder to determine suitable search terms. Additionally, as noted earlier, there is a lack of standardization in the reporting of human factors in HRI, which further complicates the search process. To address this challenge, we employed a combination of search strategies to generate a suitable set of keywords for a comprehensive and effective search string.

As a first step, a combination of manual search [31] and snowballing [32] was used. We identified relevant studies from the references and citations of the following HRI reviews that specifically addressed human factors: [7,12,15,33,34]. In addition to examining the proceedings of the following 2022 conferences which were not indexed yet at the time of writing this manuscript: IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN), ACM/IEEE International Conference on Human-Robot Interaction (HRI), IEEE International Conference on Intelligent Robots and Systems (IROS), and IEEE International Conference on Advanced Robotics and Its Social Impacts (ARSO). Furthermore, we included some prominent papers based on expert recommendations.

Initially, keywords were extracted from the set of papers obtained from the previous step, which formed the initial set of papers. However, it was found that barely half of them used the word "physical" in any of their search fields, despite the physical coupling occurring in the interaction. Moreover, the most common words detected in the papers were too generic for a database search, such as “human-robot interaction”. Additionally, the term "human factors" was not commonly used in the papers, as researchers would not necessarily use it when evaluating factors such as trust [26,35], anxiety [36], or other related concepts [37].

The search string was developed based on combining the generic term detected and the human factors terms identified in the initially considered studies, such as comfort, cognitive load, mental load, fatigue, stress, anxiety, fear, acceptance, perception, emotions, trust, safety, ergonomics, and hedonomics. The string was refined by excluding topics covered under the broad domain of HRI, such as wearables, teleoperation, telepresence, rehabilitation, and autonomous vehicles. The search was conducted on Scopus and Web of Science search engines, and the resulting papers were evaluated using a test-set approach. Each iteration was assessed by checking for the presence or absence of the initial set of papers obtained from the manual and snowballing search approaches. We ended up with 711 papers, after the removal of duplicates.

4.2. Study Selection and Quality Assessment

The study selection process involved the following steps:

• Initial screening: Abstracts of papers were checked against the inclusion criteria, with full texts read for papers that were in doubt. This resulted in 128 papers.
• Full-text assessment: Quality assessment was conducted during this phase to ensure that each paper included information relevant to the research questions. Exclusion criteria were applied, resulting in 99 papers being included in the study.

The following points summarize the characteristics of each study included in this review, i.e., inclusion criteria:

• Physical coupling between humans and robots takes place, direct or indirect pHRI.
• Robot used has manipulation capabilities (i.e. has multi degrees of freedom arm(s)).
• Human factors are evaluated during physical interaction.

While our primary objective is to focus on pHRI in domestic scenarios, we did not discard studies in industrial settings, as the field of pHRI is still young, and studying empirical data from various contexts can provide valuable insights.

Although some papers met the inclusion criteria, they did not directly contribute to the main objectives of this study. As a result, the following exclusion criteria were established:

• Physical coupling with robots for intents other than collaboration [38,39], including rehabilitation.
• Human factor evaluations of individuals with mental disorders were excluded [40], as experimental verification is required to generalize the factors of these special populations to the neurotypical population.
• Unclear identification of physical coupling between agents within a study.
• Manifestly missing methodology of human factors assessment or analysis [41,42].
• Physical interaction with baby-like robots, since this type of robot is not expected to contribute to the main objective of this study [43,44].
• Studies that are not in English [45].
• Inaccessible full text.

5. Results

The following subsections present the results of our analysis, which aim to address the research questions we formulated. We first present a mapping scheme for pHRI in Section 5.1, followed by the classification of human factors based on our conceptual model in Section 5.2. Then, in Section 5.3, we identify the most frequently addressed human factors, and examine both the commonly employed quantification approaches and the proposed quantification approaches.

5.1. Mapping Scheme for pHRI

Due to the vast applications of direct and indirect pHRI categories, it is crucial to provide subcategories that facilitate a more detailed understanding and easier navigation of the field. To address this need, we developed a mapping scheme for pHRI that categorizes the gathered studies based on the common contexts or scenarios identified in direct and indirect pHRI, as shown in Table 1, where each subcategory is described. These subcategories are based on the specific characteristics of the studies in each category, thereby addressing RQ1.

5.1.1. Direct pHRI

Figure 1 shows that direct pHRI has received considerably more attention in human factor studies than indirect pHRI, with over two-thirds of the studies conducted in this area.

The subcategories for direct pHRI proposed in this study are based on the interaction context, including the purpose and duration of contact. These classifications provide a preliminary overview of the research in this area. As the field matures, these subcategories can be further refined to provide a more detailed understanding.

Figure 2 shows that inconsistent contact has been explored in more studies than consistent contact, while Figure 3 indicates that the topic of non-functional contact has been studied almost twice as often as the topic of functional contact.

Many of the studies examined in the context of inconsistent contact were also included in the context of non-functional contact, such as studies that have investigated the NAO robot’s reactive behavior in response to social touch, including one conducted by [46]. Another study, by [49], compared the instrumental and affective touch of the NAO robot to that of a Stretch robot in a caregiving scenario. Moreover, several studies have focused on the use of NAO in caregiving scenarios that involve inconsistent and non-functional touch, such as those by [54,73,75]. These findings are consistent with several reviews in the HRI field that have highlighted the potential of NAO in caregiving scenarios. Fitter and Kuchenbecker have conducted research on different clapping contexts using a Baxter robot. Their studies include teaching [70], and playing with the robot [25,63]. Meanwhile, Shiomi et al. used a female-looking Android to investigate the effects of its subtle reactions to touch [62,74] and the impact of pre-touch reaction distance on humans [66]. Additionally, Hu et al. investigated the impact of unexpected physical actions applied by a Sawyer robot in a gaming context with the aim of helping participants accomplish higher scores [17,36].

In contrast, many studies in the context of consistent contact were also included in the context of the functional touch. Some of those studies are conducted in dancing context [96]; direct manipulation of the arm to follow certain trajectory [90,91,93,103,105], different from kinesthetic teaching; leading robots with mobile platforms using their manipulators in a nursing context [108] and a social context [107].

Fewer studies were identified as common between the consistent contact and non-functional touch categories. These studies primarily focused on hugging and handshake scenarios with humanoids. Some studies on handshake scenarios were conducted with the Meka robot [79] and HRP-2 robot [52]. Studies conducted on hugging scenarios investigated the impact of manipulating visual and tactile appearance using an ARMAR-IIIb robot [77], the duration and style of hugs using a CASTOR robot [71], and the impact of the perceived gender using a Metahug system [57,74].

The least common studies were found between the inconsistent contact and functional touch categories. Many of these studies were conducted in a nursing scenario, where the impact of instrumental and affective touch was investigated using the Cody robot [83] and NAO robot [49]. Additionally, the NAO robot was used to investigate the impact of touch style in a nursing scenario [101], and gaze height and speech timing before and after touches were examined. However, few studies have been conducted in an industrial setting, with the majority focusing on the impact of tapping a robotic manipulator as an indicator to execute a certain command [94,95].

It is worth noting that among the studies that examined the impact of touch on human factors, only two studies ([49,83]) explored the effects of functional and non-functional touch in HRI, both of which were conducted in a nursing context and categorized under inconsistent contact. No studies have examined the effects of consistent versus inconsistent contact on human factors in direct pHRI.

Table 2 shows the common studies between all subcategories of direct pHRI, including functional touch, non-functional touch, consistent contact, and inconsistent contact.

5.1.2. Indirect pHRI

In contrast, to direct pHRI, indirect pHRI has clear classifications based on real-world applications like assembly, handover, co-manipulation, and atypical scenarios. These subcategories can be useful for researchers seeking to understand the potential use cases and challenges of HRI in different practice settings. As depicted in Figure 4, there has been a particular focus on addressing human factors in handovers, which has received the most attention compared to other forms of indirect pHRI applications.

Handover scenarios in industrial settings have been the subject of several studies. For instance, some investigations have used Universal Robot (UR) manipulators to compare comfortable handovers to those that encourage dynamic behaviors [22], while others have employed UR to explore scenarios where a manipulator is completing a preliminary task and an operator requests the handover of a tool through natural communication, such as voice commands [117]. Similarly, some studies have examined industrial contexts that incorporate handovers using a Panda manipulator [118]. Other research has focused on determining the most ergonomic tool orientation for handovers [115,121]. Various handover parameters have also been explored, including physical signs of apprehension [119], the initial position of the robot arm before handover, the robot’s grasp method, and the retraction speed of the arm following handover [116].

In the context of domestic scenarios, there has been limited research into the impact of robot parameters on human factors during handovers. One study examined comfortable robot-to-human handovers in a caregiving context for older adults, using a Pepper robot [122]. Another investigation explored affective and efficient interaction styles in a cooking context, where both faulty and successful handovers were performed during the cooking task [35].

In a general context where handovers may occur in domestic and industrial environments, a study investigated handovers using a mobile manipulator that transferred objects to humans without stopping [21].

In the atypical category, numerous studies have focused on industrial applications of assistive holding, drilling, and cutting, using manipulators [129,130,132,134,135]. A few studies have explored the potential of robots as dressing assistants [133]. Another study investigated the repositioning of patients using robots in a caregiving context [131].

Co-manipulation occurs in both domestic and industrial scenarios and is usually investigated in similar ways. It typically involves investigating ways to reduce joint overload when humans collaborate with robots in lifting heavy objects [104,112,125,127,128]. Some studies also focus on mutual adaptation in this collaboration [124], as well as human intention and behavior [126].

Studies in the assembly category are typically conducted in an industrial setting due to the nature of the application. Some of these studies have explored the use of two robotic arms to hold an object for an operator while they work on it [114]. Other studies have focused on typical assembly tasks where a robot holds an object and the user assembles another part into it [112,113], or welds a wire into it [111].

Only one study in the indirect pHRI category, conducted in an industrial setting, investigated both assembly and co-manipulation tasks [112]. This finding suggests that more research is needed in this area to gain a better understanding of the challenges and opportunities of the indirect pHRI field, since many industrial applications may require more than one type of indirect pHRI.

Among the studies reviewed, only one ([64]) investigated both direct and indirect forms of pHRI. This study incorporated social touch and handover, respectively, into a collaborative task of building a towering toy. This research shows promising directions for addressing contexts where social behavior and physical assistance are combined.

5.2. Human Factors Classification

To address RQ2, we adopted the holistic conceptual model of human factors in HRI proposed by Coronado et al. [11] and adapted it to demonstrate the relationships between the most relevant measurable dimensions found in the pHRI literature. Figure 5 illustrates the relationships between usability, user experience (UX), hedonomics, and ergonomics, based on Coronado et al. definitions, which are discussed in Section 3.

Although the conceptual model used for human factor types does not fully resolve the issue of inconsistent terminologies or definitions in the field, grouping human factors/measurable dimensions, such as trust, robot perception, and anxiety, into subcategories/human factor types, such as belief, can help address this issue. This is because presenting the trends of a category instead of individual human factors can reduce bias that arises from the usage of synonyms or variations in the ways researchers refer to human factors.

On a lower level, to address the lack of unified terminologies in the literature, this study adopts a strategy of lumping some human factors under one measurable dimension, as described in Table 3. To avoid ambiguity for the reader, a brief description of some dimensions is provided. The adopted HRI conceptual model allows for some overlap between dimensions within a human factor type, as it is difficult to define sharp outlines for each dimension. The number of times a dimension is used, whether it is lumped under a broader one, detected individually, or intersects with another in definition, will not affect the overall result of the addressed frequency of a human factor type. However, if two or more human factor types share a dimension, their overall results will be biased. Therefore, in this review, all dimensions considered for each human factor type do not intersect over more than one type. If a dimension intersects over multiple types, it is reviewed and divided into more specific dimensions to avoid bias.

In order to identify the most commonly addressed human factors, i.e., address RQ2, we present two sets of figures: Figure 6 and Figure 7 show the emerged human factor types and measurable dimensions across all gathered studies, with Table 4 showing the characteristics and the studies detected in each human factor type. It is notable how belief and the measurable dimension, robot perception, have drawn the most attention in the pHRI field. This is reflected in Figure 8 to a certain extent, which shows the number of studies that examined human factor types in different direct pHRI categories. Analogous to the considerable overlap between the non-functional touch and inconsistent contact categories, Figure 8 shows that both categories addressed hedonomics more than ergonomics. Similarly, the functional touch and consistent contact categories exhibited a similar trend, with both categories emphasizing belief more than other human factor types. However, Figure 9, which illustrates the number of studies that examined human factor types within subcategories of indirect pHRI, shows that physical ergonomics is the leading human factor type in these scenarios. Similarly, physical ergonomics is the only aspect that receives attention in the atypical category. Likewise, cognitive ergonomics and affect were not addressed in the co-manipulation category. Table 5 shows the studies detected in each measurable dimension and pHRI type.

During the analysis of the results, we observed that several studies evaluated two human factor types simultaneously within the same experiment. Notably, the highest number of studies focused on the evaluation of belief and affect, including [36,46,50,53,65,70,83,90,91,114,135]. Following this, several studies explored the relationship between cognitive ergonomics and belief, such as [17,25,35,36,57,70,90,114,118]. Fewer number of studies examined cognitive ergonomics with physical ergonomics, including [20,22,90,114] and physical ergonomics with belief, such as [55,74,90,114]. Lastly, it was found that the simultaneous consideration of affect and cognitive ergonomics within the same experiment was relatively rare. Only a few studies, such as [36,70], specifically explored this combination. Notably, there were no studies found that examined measurable dimensions from physical ergonomics and affect in the same experiment.

Additionally, some studies, such as Amanhoud et al. [114] and Wang et al. [90], assessed three human factor types at once, namely cognitive ergonomics, physical ergonomics, and belief. Similarly, Fitter et al. [70] and Hu et al. [36] investigated the effects of cognitive ergonomics, belief, and affect.

5.3. Quantification Approaches of Human Factors

Less than half of the studies included in this analysis correlated dimensions with measurable entities or proposed quantification approaches for different human factors, as revealed by the human factor usage classification, as depicted in Figure 10. Some studies validated their proposed quantification approaches by manipulating robot parameters and observing their impact on the human state, including studies by [36,46,53,92,104,106,109,116,118,119,122,129,130,134]. As a result, these studies were also included in the impact category, which contained more studies than the quantification category. Table 5 includes the studies that emerged in each human factor usage category, organized according to human factor types and measurable dimensions.

Confirming the hypothesis for RQ3, questionnaires were found to be the most commonly used approach for human factors quantification in pHRI, as shown in Figure 11. However, the remaining approaches were used with much lower frequencies compared to questionnaires. To determine the role of questionnaires in pHRI, either proposed as a quantification approach or employed as an assessment approach, further analysis was conducted. Despite the fact that questionnaires were heavily employed for assessment, Figure 12 indicates that only three studies proposed or validated standardized questionnaires for pHRI. On the other hand, the proposed quantification approaches heavily relied on physical data (e.g., force, pressure, body posture, gaze direction, etc.), either through direct correlation or mathematical and machine learning models. Table 6 shows the studies that employed quantification approaches for each measurable dimension and highlighted the studies that proposed quantification approaches. For a comprehensive understanding and easier navigation of the quantification approaches in the field, we organized the studies that utilized mathematical models and machine learning models in Table 5 and Table 6 based on the input requirements of each model, including task parameters, physical data, and physiological signals.

Consistent with the overall trend, the questionnaire approach was found to be the leading method for evaluating human factors in all human factor types, as depicted in Figure 13. Although belief was heavily assessed using questionnaires and physiological signals, no studies developed machine learning models to quantify any of its measurable dimensions. Cognitive ergonomics did not see any studies that addressed physical data, while in affect there was a moderate reliance on physiological signals.

6. Discussion

In light of the literature gaps identified in the preceding section, this section proposes suggestions on how to address these gaps and highlights the importance of addressing them in order to enhance the development of adaptive controllers for multiple service robots.

6.1. Mapping Scheme for pHRI

The proposed mapping for pHRI was essential to draw conclusions about human factors considerations in the literature. However, one cannot disregard the fact that the papers gathered by this review are limited to the ones that have addressed human factors, meaning that mapping for pHRI as a domain would require a broader range of studies. Thus, this mapping can be considered a starting point that is extensible to include more pHRI subcategories. For instance, as the field develops and more experiments are conducted, it is foreseen that the consistent subcategory in direct pHRI could be further broken down to have passive and active interaction, where robots may comply with human motion or not by applying forces against them (e.g. imposing the robot’s task as prioritized with respect to the human intention/action). It is expected that such a category would impact robot perception, and usability [36].

6.1.1. Direct pHRI

The difference in the number of studies between direct and indirect pHRI domains reflects the research community’s greater focus on evaluating the human state during direct interaction conditions compared to indirect interaction conditions. This emphasis on direct pHRI may be attributed to the fact that many of the applications that have emerged are socially oriented, as evidenced by the greater number of studies on non-functional touch compared to functional touch. For example, handshaking, [52,58,67,68,72,79,80,82,85]; hugging [51,59,71,74,77]; clapping [25,63,70]; nursing [49,83,100,101,108]; and playing virtual games [17,36,105,106] have received significant attention. This trend is consistent with the idea that touch can serve important social and emotional functions in HRI, which in turn contributes to the realization of the Society 5.0 vision to a certain extent. However, it is important to note that the Society 5.0 vision encompasses assistive applications as well. Similar to the ongoing efforts to realize Society 5.0, the emergence of Industry 5.0, which emphasizes collaboration between humans and robots instead of full automation, is expected to drive the attention given to human factors in industrial settings. This includes the adoption of direct pHRI scenarios in industrial environments.

The identification of common studies between consistent contact and non-functional touch, such as handshaking and hugging, emphasizes the importance of investigating social behaviors in a realistic and broader context that includes more types of interactions. Conversely, the considerable overlap of studies between the categories of inconsistent contact and non-functional touch, especially in nursing scenarios, suggests a maturation of the field of pHRI in which researchers design experiments that encompass a wider range of interaction types beyond touch. Additionally, the studies that have investigated the purpose of direct pHRI (i.e., functional touch vs. non-functional touch) have mainly focused on nursing scenarios, indicating the relatively greater attention given to this area. However, the common studies detected between inconsistent contact and non-functional touch, which mainly comprise studies addressing social touches, may be due to the fact that touches occur spontaneously in human interactions and do not necessarily trigger or stop them if not maintained, unlike functional touches where many studies were exploratory and used the robot as a tool to follow certain paths.

The significant overlap between consistent contact and functional touch, and between inconsistent contact and non-functional touch suggest that they may be redundant, indicating a need to consider either the purpose or duration categories for future study classification in reviews. Further refinement of these classifications could enhance our understanding of direct pHRI as the field matures.

Our main objective is to bring robotic multiple service providers to life, and it is promising that most direct studies of pHRI focus on non-industrial applications. However, this also highlights a gap in current research for Industry 5.0. While non-industrial applications have explored social aspects of pHRI, such as handshaking, hugging, and playing, they have not adequately investigated physical assistance applications in daily life activities such as toileting, sit-and-stand, transferring, and others. Without proper physical assistance, the potential benefits of social aspects are limited, and pHRI may not be useful in domestic or institutional environments. Therefore, it is important to investigate more physical assistive applications, while taking into consideration the human state, to fully realize the potential of robotic multiple service providers.

6.1.2. Indirect pHRI

Although human factors in handover scenarios have been explored in various contexts, there has been insufficient attention given to handovers in domestic settings. Handovers in the industrial context are being thoroughly investigated, including the orientation of tools and the human body posture. However, the objective of comfortable and efficient handovers is likely similar in both industrial and domestic settings. This suggests that results from studies conducted in one setting could be applicable to the other.

However, most studies on atypical scenarios and assembly tasks are industrial-focused, highlighting the need to address domestic applications beyond those currently studied. Atypical scenarios in domestic settings, such as dressing, grooming, showering, or household chores, are complex and technically challenging. Similarly, assembly contexts in domestic settings, such as putting pillows into pillowcases or performing certain cleaning duties, could also pose technical challenges.

Studies on co-manipulation scenarios are limited, and the focus is mainly on reducing human joint overload during collaboration. Few studies have investigated mutual adaptation to achieve a common goal. While addressing joint overload is a logical first step in co-manipulation, mutual adaptation between humans and robots is also crucial in domestic scenarios where robots can assist with tasks such as furniture reorganization.

The scarcity of research in the indirect pHRI category underscores the insufficient attention that human factors researchers have paid to this field. Additionally, The lack of studies that addressed different applications simultaneously suggests that more research is needed in this area to gain a better understanding of the challenges and opportunities of indirect pHRI, since many industrial settings may require multiple types of applications. Notably, domestic environments have received minimal to no attention in any of the subcategories of indirect pHRI. It could be argued that the term pHRI is more commonly used in industrial settings than in domestic contexts in the indirect pHRI domain.

6.2. Human Factor Classification

The holistic human factors model adapted to pHRI and proposed in Figure 5 is focused on our specific research questions and therefore differs from the original in some respects, i.e., in [11]. In particular, we did not include measurable dimensions related to "performance" as a human factor type under "ergonomics" since it was not a primary focus of our analysis. Based on user experience definition, we found it appropriate to categorize all measurable dimensions related to physical and cognitive ergonomics under the "user experience" umbrella, whereas the original model depicts "user experience" overlapping with these human factor types but not entirely encompassing them. It is worth noting that for a more comprehensive understanding of the conceptual model, we recommend readers to refer to the original adapted HRI model.

Given the objectives of our review, we focused on factors that are affected by the interaction. Therefore, our review did not detect factors that appear out of the scope of the definition of user experience, such as workplace design and awareness, which are not impacted during an interaction [11].

We ensured that each measurable dimension we identified was classified under only one human factor type, as overlapping dimensions between types would have contradicted the holistic approach of the conceptual model we adopted. To achieve this, we based our classification on the definitions of usability, user experience, hedonomics, ergonomics, and their respective sub-categories.

Similarly, based on the definition of usability and the main objectives of this review, all of the measurable dimensions detected in usability belong to the hedonomics category of human factors. This is because the objective aspects of usability, which are more related to performance, were not a primary focus of the review.

Although defining each measurable dimension in detail was not our primary focus, we classified them based on the provided descriptions of human factor types and the usability definition. This approach allowed us to present an accurate and unbiased overview of the most relevant human factors that have been addressed in pHRI literature.

Our primary goal is to expedite the development of adaptive control frameworks for multiple service robots. We believe that investigating the most frequent human factor types evaluated together in a single experiment, along with exploring the relationships between measurable dimensions within each human factor type, will aid in this development by identifying the human factors that may have significant correlations for researchers to consider when designing adaptive control frameworks.

Since affect and belief address humans’ emotions from different perspectives, it is not surprising that most of the studies that have evaluated more than 1 human factor type belonged to belief and affect categories. Similarly, cognitive ergonomics and belief come next in the most common number of studies between human factor types, since both evaluate cognitive aspects but from different perspectives. In contrast, no studies have evaluated humans’ emotions during an assessment of their physical ergonomics. Therefore, it is anticipated that there could be an unrevealed relationship between the 3 human factor types and their measurable dimensions that is worth investigating. On a broader view, unrevealed relationships between measurable dimensions within a human factor type are very likely to exist. However, a lack of definitions may not support conducting such a study, unlike correlating human factor types to each other, which can be perceived as more feasible.

No wonder direct pHRI studies addressed a wider variety of human factor types in comparison with indirect pHRI studies since many more papers have emerged in the former than the latter. As expected, affect is the most addressed human factor type among the non-functional touch studies, and accordingly, the inconsistent studies, given the big number of common studies among both categories. Along the same lines, physical ergonomics is the least addressed human factor type among them since those type of studies is more socially directed. It was foreseeable that belief would be highly addressed among most of the studies since HRI literature has pointed out that dimensions such as trust were the most frequently addressed in comparison with any other dimensions. In contrast, cognitive ergonomics neither attracted most of the research community nor was completely ignored by it.

As previously mentioned, the majority of indirect pHRI applications are focused on industry. Consequently, it is not surprising that physical ergonomics is the most frequently addressed category, with limited attention given to affect. In fact, affect was hardly mentioned in any of the indirect categories. Belief, however, which takes emotions into account, received more attention than affect due to its evaluation being influenced by cognitive aspects.

To fully realize the potential of pHRI in domestic scenarios, it is essential to address various gaps in the literature. In direct pHRI scenarios, the robotics research community must prioritize assessing physical assistance experiments, such as toileting, and sit-and-stand, from a human factors perspective. While social interaction experiments like hugging, handshaking, and playing are important, it is crucial to combine both physical interaction types to understand their influence on user perception during physical assistance. Additionally, applications that may be considered industrial or non-industrial, such as handovers and co-manipulation, should also prioritize hedonomics and cognitive ergonomics, in addition to physical ergonomics. Moreover, atypical and assembly applications such as eating assistance, grooming, showering, and household chores require attention to various human factors. Addressing these literature gaps will enhance the overall HRI experience and increase the usefulness of pHRI in domestic settings.

6.3. Human Factors Quantification

Due to the lack of consensus on human factors terminology and definitions in the domain of HRI, the large number of experiments conducted to understand the impact of human factors, as shown in Figure 10, is not expected to yield efficient and uniform progress in the field. In other words, without reliable and agreed-upon quantification approaches, the results of experiments on the impact of human factors may not be reliable enough to facilitate the development of the field. However, A promising aspect is that a significant number of studies proposed quantification approaches and validated them, demonstrating the research community’s awareness of the importance of such validation. The development of standardized and reliable quantification methods has the potential to facilitate the detection of progress in the field of pHRI, thus accelerating its advancement.

It is clear that the human factor types addressed in Figure 6 follow a trend similar to the measurable dimensions addressed in Figure 7. However, the human factor types are comprehensive, reflect the research direction in a broader view and give all measurable dimensions, within a human factor type, the same level of importance despite the terminology used. Thus, adopting the conceptual model is expected to yield efficient uniform progress in the field.

Figure 11 and Figure 12 demonstrate that subjective approaches are the primary evaluation method used across all human factor types. Questionnaires are commonly used due to their simplicity and fast administration, whereas other assessment approaches may require specialized equipment and complex data processing. However, questionnaires should be considered a preliminary assessment approach that provides an initial understanding of human states or validates other objective approaches. Otherwise, the development of adaptive control frameworks for human states is unlikely to become a reality in the near future. Hedonomics relies primarily on questionnaires, whereas ergonomics has investigated objective approaches more frequently than subjective ones. Physical ergonomics parameters are quantifiable, which explains why objective measures have been used in physical ergonomics studies more than in cognitive ergonomics. Future efforts should focus on developing objective measures for the other three human factor types to enable the implementation of real-time adaptive control frameworks in the future. Nevertheless, developing real-time objective quantification approaches for HRI, including its subcategory pHRI, is challenging, which explains why these approaches are rarely addressed in these fields.

7. Limitations of the Review

This section is dedicated to demonstrating the validity of this review by discussing its limitations. It should be noted that limitations and future work of the field are discussed within Section 6, Discussion. In order to evaluate the validity of this review, 3 validity factors proposed by [30] can be applied: theoretical validity, interpretive validity, and descriptive validity.

A common limitation in literature reviews is the possibility of existing studies that are related to the main topic but could not be detected due to unfortunate choices of keywords either from the review’s authors or the related studies. However, the adopted guidelines, which included the test-set step, could make us argue that this type of threat is eliminated to a decent extent. Petersen et al. [30] have referred to this validity as theoretical validity which they defined as “our ability to be able to capture what we intend to capture”. Therefore, theoretical validity includes data extraction validation. In order to restrain data extraction bias, full-text checkout over all the gathered papers was performed iterative, until a relatively proper lumping of measurable dimensions is achieved.

Another validity factor is interpretive validity, which “is achieved when the conclusions drawn are reasonable given the data" [30]. To limit interpretive validity threats, HRI experts got involved during each step of conducting this review, including the interpretation of the extracted data. However, one can argue that broad dimensions, such as robot perception, emotions, and physical comfort are misinterpreted as they can be divided into many smaller dimensions. For instance, robot perception can be divided into attitudes, acceptance, perceived performance, etc. However, according to the HRI conceptual model adopted, measurable dimensions that do not intersect between human factor types cannot bias our results as previously explained in Section 5.2. Nevertheless, breaking down factors can provide a clearer understanding of the measurable dimensions addressed within a human factors type. Therefore, it is highly encouraged for future reviewers to consider this approach. Furthermore, it can be argued that one common human factor, such as perceived safety, did not emerge as a measurable dimension within any human factor type, which might be regarded as another misinterpretation. However, perceived safety is a broad term that is shared between different human factor types as it includes dimensions such as stress, anxiety, trust, fear, and psychological comfort[7]. Therefore, considering it as a measurable dimension is not reasonable.

To maintain descriptive validity, which “is the extent to which observations are described accurately and objectively”, the adapted model of human factors in HRI is considered. The conceptual model provides clear outlines for usability, user experience, and each of the human factors types, while still allowing for accurate classification of measurable dimensions within each category. However, it can be noticed that performance as a human factor type within ergonomics is discarded, which can rise a descriptive validity threat by arguing that ergonomics is not accurately conducted throughout this review. As discussed in Section 5.2, performance, as defined by Coronado et al. [11], includes the evaluation of the performance of the robot, the user, and the overall system. However, since our main objective is to quantify the human state, these performance evaluations are not directly relevant to our review. Especially that perceived performance, which contributes to the human state, is included in robot perception, as shown in Table 3. As this is a human-centered review, research has shown that in domestic contexts, humans tend to prefer robots that exhibit human-like characteristics over high-performance robots [35]. Therefore, since our primary objective is to quantify the human state, discarding performance evaluations do not negatively impact the main focus of this review. Therefore, we are confident that the conclusions of this review can serve as a reliable foundation for future research in this field.

8. Conclusions

For multiple service robots to coexist with humans, they must meet users’ physical and social needs. Socially, robots must be able to predict and quantify human states to interact appropriately. Physically, manipulation is a key capability. This article focused on studies of robots with manipulation abilities and their impact on human factors in pHRI. The goal is to advance the development of pHRI control frameworks that can predict human states and make decisions accordingly.

To achieve the objective of this review, we identified the most common human factors in pHRI, noted the factors typically addressed together, and determined the most commonly used assessment approaches. We also collected and classified proposed assessment approaches for each human factor and created an initial map of the common contexts and applications in pHRI. Our aim was to provide an unbiased overview of the field, identify research gaps, and facilitate the search for adaptive control framework advancements.

Most studies in the direct pHRI category focus on social behaviors, with belief being the most commonly addressed human factor type. Task collaboration is moderately investigated, while physical assistance is rarely studied. In contrast, indirect pHRI studies often involve industrial settings, with physical ergonomics being the most frequently investigated human factor. More research is needed on human factors in direct and indirect physical assistance applications, including studies that combine physical social behaviours (e.g., comforting touch) with physical assistance tasks (e.g., sit-to-stand). This will enable robots to exhibit appropriate social skills while providing physical assistance. Expectedly, the predominant approach in most studies involves the use of questionnaires as the main method of quantification. However, it is worth noting a recent trend that seeks to address quantification approaches based on measurable data.

To advance service robots and Society 5.0, future reviews should include a wider range of HRI scenarios beyond physical collaboration. These reviews should prioritize studies that use measurable data to quantify human factors, as well as those that explore correlations between demographic and personal information and human factors. By identifying relationships between measurable dimensions of each human factor, we can create more robust adaptive control frameworks for various pHRI scenarios.