A Concise Overview of the Vision-based Human Action Recognition Field

1. Introduction

Artificial intelligence (AI) redefines our understanding of the world by enabling high-impact applications such as intelligent video surveillance systems [1], self-driving vehicles [2], assisted living [3], and human-computer interface systems [4]. These applications frequently require a single core task: video-based human action recognition, an active research field with continuous improvements. Its principal purpose is to acquire insight into what a subject is doing in a video [5,6,7].

On the one hand, current research points out numerous directions, including effectively combining multi-modal information [8,9], learning without annotated labels [10], training with reduced data points [10,11], and exploring novel architectures [12,13].

On the other hand, recent surveys focused on providing an in-depth review of a specific contribution. For example, [14] categorized standard vision-based human action recognition datasets, whereas [15] analyzes the classification performance of standard action recognition algorithms. [16] was one of the first surveys to incorporate deep learning algorithms, providing a comprehensive overview of the datasets employed. [17] offers a comprehensive taxonomy centered on deep learning methodologies. While [18] and [19] concentrate on its applicability. Finally, [20] delved into future directions of the field.

Given the enormous knowledge and many directions, it can take time to introduce oneself to the subject thoroughly. As a result, in this work, we focus on offering an intuitive picture of the vision-based human action recognition field and describing how the approaches evolved and the intuitive notions that underpin them. Also, we gathered a collection of abstracts utilizing the papers-with-code portal services [21] and analyzed them to determine the most prevalent keywords by year. Thus, we can evaluate the most-discussed themes, the nature of their evolution, and potential prospects for the near future.

We divide the rest of the document as follows: in Section 2.1, we define the concept of human action. Then, we speak about the vision-based human action recognition taxonomy in Section 2.2. In Section 3, we explore the common terms in the paper’s abstract to find future directions. Finally, we jump to conclusions in Section 4.

2. Action Recognition Taxonomy

The aim of this section is two-fold. First, in Section 2.1, we explain what this work understands as action. Finally, we introduce the action recognition taxonomy in Section 2.2.

2.1. What is an Action?

To understand the idea behind an action, picture the image of a person greeting another. Probably, the mental image constructed involves the well-known waving hand movement. Likewise, if we create a picture of a man sprinting, we may build a more dynamic image by focusing on the person’s legs, as shown in Figure 2. We unconsciously associate a particular message with a sequence of movements, which is what we call "an action" [22]. The human action recognition goal is to build approaches that can understand the encoded message in the sequence of gestures.

Although it is a natural talent for a person to recognize what others do, it is not an easy assignment for a computer since it faces numerous challenges [16], including the considerable variability in how a person can act. For example, an older person’s running movement will differ from that of a younger person. In addition, it is challenging for an individual to perform the same activity in precisely the same way. Finally, actions can be influenced by cultural context, as in the case of greeting.

Similarly, there are restrictions on the camera’s capabilities [16]: first, various camera placements achieve infinite action perspectives, which may generate object occlusions or limited viewpoints. Another limitation is the video resolution, which may not provide enough pixel information to identify the target. External factors, such as rain or wind, can also harm the camera’s image quality.

2.2. Human Action Recognition Taxonomy

In this section, we describe the approaches to recognizing human actions. First, in Figure 1, we provide a formal description of the human action recognition taxonomy and summarize the evolution of the methods and some keypointsld. We categorized the methods into two main branches. On the one hand, we have the family of procedures whose feature engineering is performed manually, knowns as "handcrafted methods." On the other hand, there are methods whose representation is learned, as in the case of deep learning techniques.

2.2.1. handcrafted Approaches

As illustrated in Figure 1, handcrafted approaches entail manually engineering features, i.e., we must develop characteristics that support a computer to understand the human action concept, analogous to creating a translator app for those who do not speak the same language.

Global representations [16], shown in Figure 1 (A), are the first attempt to recognize actions whose intuition is to represent the content as a global feature. For example, as individuals, we can discern a person’s actions by looking at their silhouette. However, this approach proved inadequate in addressing the numerous challenges posed by videos or images, such as different viewpoints and occlusions. Consequently, global representations could not fully capture the variability of an action. Hence, a novel approach was needed. Among the most relevant methods are Motion Energy Image (MEI) [23], Motion History Image (MHI) [24], silhouettes [25], and Space-Time Volume (STV) [26].

The world is full of little details that are difficult to capture using the "big picture." Intuitively, as humans, to discover those little secrets, we need to explore, focus on the details, and zoom in on the regions of interest, which is the idea behind local representations [17,27], shown in Figure 1 (B). Local representations seek to extract descriptors from multiple regions of the video to get insights into the details. Local approaches break down into a sequence of steps: a) detection of points of interest, b) extraction of video descriptors, and c) aggregations of local descriptors. As a consequence, the researcher’s contributions can be three-fold.

As the name suggests, the first step is to detect which regions of the video to analyze. Nevertheless, determining the significance of a region can be a relatively tricky undertaking. Applying edge detection algorithms is one method, such as Space-Time Interest Points (STIPs) [28] and hessian detector [29]. However, its application could lead to noise and lousy performance due to the extraction of edges that belong to something other than the target subject. To assess the regions’ relevance and eliminate noisy information, Liu et al. [30] propose to use of statistical properties as a pruning method.

Camarena et al. [31,32] suggest that pose estimation can be used as the regions of interest, resulting in a method that has a fixed and low number of processing areas, which ensures a consistent frame processing rate. However, the approach is dependent on the subject body’s visibility.

Another solution is to apply dense sampling [33], which consists of placing points without semantics. Dense sampling increases the classification accuracy, but it is computationally expensive [31]. Besides, noise injected by other motion sources can affect the classifier’s performance [31,32].

Once we have determined which regions to analyze, we must extract the corresponding region description. Visual and motion data are essential for accurately characterizing an action [31]. In this regard, the typical approach combines several descriptors to have a complete perspective of the target action. Regarding the visual information, we have histogram of oriented gradients 3D (HOG3D) [34], speed-up robust features (SURF) [35], 3D SURF [29], and pixel pattern methods [36,37,38]. On the other hand, descriptors that focus on motion information include histogram of Oriented Flow (HOF) [39], Motion Boundaries Histogram (MBH) [33], and MPEG flow [40].

Capturing motion information is a complex task; videos are composed of images in which the target person moves or changes location over time [33]. The naive method uses cuboids, which utilize static neighborhood patterns throughout time. However, cuboids are not suitable for modeling the temporal information of an object. its natural evolution were trajectory-based approaches [33,41,42] that rapidly becoming one of the most used methods [17,32].

Trajectory-based methods use optical flow algorithms to determine the position of the object of interest in the next frame, which helps to improve the classification performance [17]. Although there are several efficient optical flow algorithms, their application at different points of interest can be computationally expensive [32]. To reduce the computational time is essential to know that there are several motion sources besides the subject of interest, including secondary objects, camera motions, and ambient variables. Focusing on the target motion may reduce the amount of computation required. On the one hand, we can use homographies [17] for reducing the motion’s camera; on the other hand, pose estimation [32] can be used to remove the optical flow process thoroughly.

Descriptor aggregation is the final stage in which the video descriptor is constructed using the region descriptors acquired from the preceding processes. There are several methods, including Bag-of-visual-words (BoVW) [43], Fisher Vectors [44], stacked fisher vector (SFV) [45], vector quantization (VQ) [46], vector of locally aggregated descriptors (VLAD) [47], super vector encoding (SVC) [48]. Among the handcrafted approaches, it is popularly referred to that FV and SFV, along with dense trajectories, achieve the best classification performance [16].

2.2.2. Deep Learning Approaches

Due to their strong performance in various computer vision tasks [1,2,3], convolutional neural networks (CNNs) have become increasingly popular. Hence, its application to vision-based human action recognition appeared inevitable.

Andrej et al. [49] developed one of the first approaches, which involved applying a 2D CNN to each frame and then determining the temporal coherence between the frames. However, unlike other computer vision problems, using a CNN does not outperform handcrafted approaches [20]. The main reason was that human actions are defined by spatial and temporal information, and using a standalone CNN does not fully capture the temporal features [20]. Therefore, subsequent deep learning research for human action recognition has focused on combining temporal and spatial features.

As a common practice, biological processes inspire computer vision and machine learning approaches. For example, as individuals, we use different parts of our brain to process the appearance and motion signals we perceive[50,51]. This understanding can be used for human action recognition, as suggested by [50]. The concept is straightforward. On the one hand, a network extracts spatial characteristics from RGB images. On the other hand, a parallel network extracts motion information from the optical flow output [50]. By combining spatial and temporal information, the network can effectively process visual information.

Due to the comparable performance of two-stream networks to trajectory-based methods, [20], interest in these approaches grows, leading to novel research challenges such as how to merge the output of motion and appearance features. The most straightforward process, referred to as late fusion [52], is a weighted average of the stream’s predictions. More sophisticated solutions considered that interactions between streams should occur as soon as possible and proposed the method of early fusion [52].

Because of the temporal nature of videos, researchers investigated the use of recurrent neural networks (RNN) [53] and long-term short-term memory (LSTM) [54,55] as the temporal stream for two-stream approaches. As proven by Ma et al. [56], pre-segmented data is necessary to explode the performance of an LSTM in videos thoroughly, eventually leading to temporal segment networks (TSN), which has become a popular configuration for two-stream networks [20].

A generalization of two-stream networks is multi-stream networks [20], which describe actions using additional modalities like pose estimation [57], object information [58], audio signals [59], text transcriptions [60], and depth information [61].

One factor that impacts the performance of deep neural networks is the amount of data used to train the model. In principle, the more data we have, our network’s performance will be higher. However, the datasets employed in vision-based human action recognition [62,63,64] do not have the scale that requires a deep learning model [65]. Do not dispose of enough data has various implications, one of which is that it is difficult to determine which Neural network architecture is optimal. Carreiera et al.[65] introduced the Kinetics dataset as the foundation for re-evaluated state-of-the-art architectures and proposed a novel architecture called Two-Stream Inflated 3D ConvNet (I3D) architecture, based on 2D ConvNet inflation. I3D [65] demonstrates that 3D convolutional networks can be pre-trained, which aids in pushing state-of-the-art action recognition further. Deep learning methods work under a supervised methodology implicating considerable high-quality labels [66]. Nevertheless, data notation is a time-intensive and costly process [66]. Pretrained is a frequent technique to reduce the required processing time and amount of labeled data [66]. Consequently, researchers explored the concept of 2D CNN inflation further [67,68], yielding innovative architectures like R(2+1)D [69].

Current research in vision-based human action recognition has several directions. First, novel architectures such as visual transformers have been ported to action recognition [69,70]. Second, there is a need for novel training methods like self-supervised learning (SSL) [71], which is a novel training technique that generates a supervisory signal from unlabeled data, thus eliminating the need for human-annotated labels. Third, few-shot learning action recognition is also being investigated [72].

Most of the architectures described are known as discriminative approaches [73], but there is another family of deep learning methods based on generative techniques [73]. Its core idea is based on the popular phrase "if I cannot create it, then I do not understand it" [74]. auto-encoders [75], variational autoencoders [76], and adversarial networks (GAN) [77] are examples of this approach.

3. Human Action Recognition Trends: Metadata Analysis

To gain a deeper understanding of the vision-based human action recognition discipline, we scanned the paper’s abstracts to identify the most frequently used terms. Identifying the commonly used keyword allowed us to evaluate yearly trends regarding topics, techniques, and potential research directions. The extraction of the paper’s abstracts could be accomplished in two ways: through the use of web scraping techniques [78] and web services such as programmatic interfaces (APIs) [79].

Web scraping techniques attempt to replicate human behavior by reading and parsing websites [78]. The significance of this type of technique [80] is that it permits the extraction of data even when there is no programmatic interface, such as in the case of Google Scholar [81]. However, a disadvantage of web scraping [78] is that some websites apply security measures like low-number of requests or the use of captchas, making it challenging to collect large-dimensional information. Also, there is a site structure dependency, which means that if the design of the website changes, the developed program could be affected.

Application programming interfaces (APIs) [79] are another method for obtaining information because they enable integration and give capabilities for extracting pre-defined attributes. Another advantage over web scraping is that the quality of structure and cleanliness is believed to be higher and allows for greater modularity and scalability. There are several web services to extract research data, including arXiv [82], papers-with-code [21], Scopus API [83], and Science Direct API [84]. We found low-bias information while using the papers-with-code service, hence using it in this research. However, in the future, we might combine more than one service to make a more robust information base.

To extract the paper-related information, we request the query "action-recognition-in-video" through the task retrieval module of the service, which returned 1600 paper instances. Then, we split the abstracts by year and selected the 30 most frequently used words. The visualization is shown in Figure 3. The analysis helps us identify the key terms used to gather information about the topics that researchers face.

As expected, "CNN", "performance", "ConvNet", "architecture", and "handcrafted" were the most often used in 2014 and 2015. The keywords suggest that finding the optimal architecture for human action recognition was a substantial issue. We previously mentioned that the first attempts to recognize human actions using neural networks did not perform as well as their handcrafted counterpart due to the high dependence on temporal information. Therefore, finding the architecture that would take advantage of the neural network’s ability to extract representations without manual feature engineering makes sense as the main issue. This intuition is aligned with the keywords presented in 2016 and 2017, where "optical flow", "3D CNN", "pre training", and "two-stream" are techniques that enable deep learning methods to outperform handcrafted approaches.

This tendency continued in 2018 and 2019, with the keywords strongly aligned to model the temporal information. Also, some relevant insights are that "cross-modal" and "self-supervised learning" are gaining popularity, which is one of the most promising research paths in our days [85]. Then, in 2020 and 2021, the terms "self-supervised learning" and "few-shot learning" increased in popularity, indicating a desire to reduce the amount of labeled data required. Finally, in 2022, some keywords appear that may show future directions, like ’graph convolutional neural networks, "understanding," and "reasoning". The keywords may indicate that current research not only focuses on the learning process but also ensures that the model truly understands what it is learning, opening the door to a more robust generalization and capabilities to learn new concepts. Furthermore, the granularity of actions and long-term behavior becomes a focused path for research. Finally, we see the application of vision transformers to vision-based human action recognition.

4. Conclusions

In this work, we thoroughly examine the field of study of human action recognition, delving into its history and evolution, and providing an in-depth overview of the method, from traditional handcrafted techniques to modern deep learning architectures. We present this information comprehensively, easy-to-understand manner, emphasizing how ideas in the field have evolved.

Although neural networks enabled a breakthrough in recognition task performance and removed the need for manual feature engineering, the process still has some limitations. Neural networks work under a supervised methodology, which requires a vast number of high-quality labels. Assuming unlabeled data is available. Also, the process of annotating data is a time-consuming and expensive operation.

To understand the current directions of the vision-based human action recognition field, we examined the scientific publication abstracts in which we found frequent terms among them. The keywords help to understand the yearly issues and provide clues to future directions.

To reduce the dependency on human-annotated labels, one of the research lines to pay attention to is self-supervised learning, which aims to generate a natural supervisory signal derived directly from the unlabeled data. In the same direction, few-shot learning techniques aim to generalize novel object classes in addition to new instances, reducing the amount of data required.

Reasoning instead of only learning is a novel trend in human action recognition, which can be helpful in a few-shot environment and when facing fine-grained activities. Creating models that perform well with fine-grained actions is a challenging problem because datasets cannot represent every possible variation of an action. Nevertheless, due to the increasing demand for specialized action comprehension in real-world applications, interest in this method has been growing.