Action Recognition via Adaptive Semi-Supervised Feature Analysis

1. Introduction

Effective feature representation of videos is key to action recognition. Spatiotemporal features [1,2], subspace features [3,4], and label information [5] have been investigated for action recognition. Correlations between multiple features may provide distinctive information; hence, feature correlation mining has been explored to improve the recognition results when labeled data are scarce [4,6]. However, these approaches may have limitations in learning discriminant features, they have limitations. First, although existing algorithms evaluate the common shared structures among different actions, they do not take inter-class separability into account. Second, current semi-supervised approaches solve the nonconvex optimisation problem by impressive derivation, but the global optimum may not be computed mathematically through alternating least squares (ALS) iterative method.

To overcome the limitations of using multiple features for training, we propose modelling intra-class compactness and inter-manifold separability simultaneously, then capturing high-level semantic patterns via Multiple feature analysis. Considering the optimisation process, we introduce the PBB algorithm because of its effectiveness in obtaining an optimal solution [7]. The PBB method is a non-monotone line-search technique considered for the minimisation of differentiable functions on closed convex sets [8].

Inspired by the research using multiple features [5,6], our framework was extended in a multiple-feature-based manner to improve recognition. We proposed the characterisation of high-level semantic patterns through low-level action features using multiple-feature analysis. Multiple features were extracted from different view of labeled and unlabeled action videos. Based on the constructed graph model, pseudo information of unlabeled videos can be generated by label propagation and feature correlations. For each type of feature, nearby samples preserve the consistency separately, while unlabeled training data perform the label prediction by jointly global consistency of multiple features. Thus, an adaptive semi-supervised action classifier was trained. The main contributions can be summarized as follows:

(1) This work first simultaneously consider manifold learning and Grassmannian kernels in semi-supervised action recognition, as we assume that action videos samples may be found in a Grassmannian manifold space. By modelling a embedding manifold subspace, both inter-class separability and intra-class compactness were considered.

(2) To solve the unconstrained minimisation problem, we incorporate PBB method to avoid matrix inversion, and apply globalisation strategy via adaptive step sizes to render the objective functions non-monotonic, leading to improved convergence and accuracy.

(3) Extensive experiments verified that our method is better than other approaches on three benchmarks in a semi-supervised setting. We believe that this study presents valuable insights in adaptive feature analysis for semi-supervised action recognition.

2. Related Work

We review the related researches on semisupervised action recognition, multiple feature analysis, and embedded subspace representation in this section.

2.1. Semisupervised Action Recognition

Unlabeled samples are valuable for learning data correlations in semi-supervised manner [3,4,9,10]. Although it tends to achieve remarkable performance even with very limited labeled data, there are still many issues in semi-supervised learning techniques, such as suboptimal due to without utilizing the temporal dynamics and inherent multimodal attributes, or obtained pseudo-labels using confident predictions from the model to teach itself [11,12].

Si et al.[13] tackle the challenge of semi-supervised 3D action recognition for effectively learning motion representations from unlabeled data. Singh et al.[14] maximize the similarity of same video at two different speeds, and recognize actions by training a two-pathway temporal contrastive model. Kumar and Rawat[15] detect action video ation via end-to-end semi-supervised learning, which develop a spatio-temporal consistency based approach with two regularization constraints: temporal coherency and gradient smoothness.

2.2. Multiple Feature Analysis

Because we can describe an object by different features which provide different discriminative information, multiple-feature analysis have gained increasing interest in many applications. In the early and late-fusion strategies, multistage fusion schemes have recently been investigated [4,16,17,18]. While the correlations of each feature type have not been considered in most late-fusion approaches.

Wang et al.[19] apply shared structural analysis to characterize discriminative information and preserve data distribution information from each type of feature. Chang and Yang [20] discover shared knowledge from related multi-tasks, take various correlations into account then select features in a batch mode. Huynh-The et al.[21] capture multiple high-level features at image-based representation by fine-tuning pre-trained network, transfer skeleton pose to encoded information and depict an action through spatial joint correlations and temporal pose dynamics.

2.3. Embedded Subspace Representation

Previous studies have shown that manifold subspace learning can mine geometric structure information by considering the space of probabilities as a manifold [22,23,24]. Recent researches focus on graph embedded subspace or distance metric learning to measure activities similarity [25,26,27,28,29].

Rahimi et al.[30] build neighborhood graphs with geodesic distance instead of euclidean distance, and project high-dimensional action to low-dimensional space by kernelized Grassmann manifold learning. Yu et al.[31] propose an action matching network to recognize open-set actions, construct an action dictionary and classifies an action via distance metric. Peng et al.[32] alleviate the over-smoothing issue of graph representation, when multiple GCN layers are stacked by flexible graph deconvolution technique.

The two aforementioned studies [3,4] are similar to ours. They assumed that the visual words in different actions shared a common structure in a specific subspace. A transformation matrix is introduced to characterise the shared structures. They solved the constrained nonconvex optimisation problem by ALS–like iterative approach and matrix derivation. Nevertheless, the deduced inverse matrix is poorly scaled during optimisation or close to singular, that may lead to inaccurate results.

To address these problems, we hypothesise that manifold mapping can preserve the local geometry and maximise discriminatory power. However, we did not aim to mine shared structures. Therefore, we ignored shared-structure regularisation and modelled the manifold by creating two graphs. As the optimisation solution in [3,4] may be mathematically imprecise, Karush-Kuhn-Tucker (KKT) conditions and PBB are introduced to improve algorithm convergence and avoid matrix inversion.

Different from another related research named semisupervised discriminant multimanifold analysis(SDMM) [10], we try to make modifications in two main aspects: multiple feature analysis through manifold subspace projection with combined Grassmannian kernels, unconstrained convex optimisation through non-monotone line search strategy with adaptive step sizes.

3. Proposed Approach

3.1. Formulation

To leverage the multiple feature correlation, n training sample points $\mathbb{X}=[X_1,...,X_n]\in {\mathbb{R}}^{d\times n}$ are defined from the underlying Grassmannian manifold, where $X_i\in {\mathbb{R}}^{d\times 1}$. We aim to uncover a new manifold while preserving the local geometry of data points, that is, $\alpha :X_i\to F_i$. Since we should demonstrate data distribution on manifold, a predicted label matrix $\mathbb{F}=[F_1,...,F_n]\in {\mathbb{R}}^{n\times n}$ is defined, where the predicted vector of the i-th datum $X_i\in \mathbb{X}$ is $F_i\in {\mathbb{R}}^{n\times 1}$.

Figure 1. An illustration of our method. (a) Video-sets can be represented in ${\mathbb{R}}^D$. We can use the principal angles between them, to compare two actions. (b) Data points on the Grassmannian manifold M can be described as linear subspaces in ${\mathbb{R}}^D$. When points on the manifold having a proper geodesic distance, the video-set matching problem may be converted to a points distance measurement problem. (c) By employing proper Grassmannian kernel, data points can be mapped into another Grassmannian manifold $M^{\prime }$ where same actions become closer while different actions are well separated.

Figure 1. An illustration of our method. (a) Video-sets can be represented in ${\mathbb{R}}^D$. We can use the principal angles between them, to compare two actions. (b) Data points on the Grassmannian manifold M can be described as linear subspaces in ${\mathbb{R}}^D$. When points on the manifold having a proper geodesic distance, the video-set matching problem may be converted to a points distance measurement problem. (c) By employing proper Grassmannian kernel, data points can be mapped into another Grassmannian manifold $M^{\prime }$ where same actions become closer while different actions are well separated.

We assume that a similarity measurement of data points on manifold subspace is available through a Grassmannian kernel [22] $k_{i,j}=\langle X_i,X_j\rangle$. By confining the solution to a linear function, that is, ${\alpha }_i={\sum }_{j=1}^n{a}_{ij}{X}_j$, we define the prediction function f as $f\left(X_i\right)=F_i={(\langle {\alpha }_1,X_i\rangle ,\langle {\alpha }_2,X_i\rangle ,...,\langle {\alpha }_r,X_i\rangle )}^T$. By denoting $A_l={(a_{l1},...,a_{ln})}^T$ and $K_i={(k_{i1},...,k_{in})}^T$, it can be shown that $\langle {\alpha }_l,X_i\rangle =A_l^T{K}_i$, and thus, $f\left(\mathbb{X}\right)=\mathbb{F}={\mathbb{A}}^T\mathbb{K}\approx \mathbb{Y}$, where $\mathbb{A}=\left[A_1\right|A_2|...|A_r]$ and $\mathbb{K}=\left[K_1\right|K_2|...|K_n]$. As mentioned in [33], the performance of least square loss function is comparable to hinge loss or logistic loss. This is associated with its diagonal matrix $\mathbb{Y}=[Y_1,...,Y_n]\in {\{0,1\}}^{n\times n}$, where $Y_i\in {\{0,1\}}^{n\times 1}$ is the label matrix. We employed least squares regression to solve the following optimisation problem, then obtain the projection matrix $\mathbb{A}$:

$$\underset{\mathbb{A}}{min}\parallel {\mathbb{A}}^T{\mathbb{K}-\mathbb{Y}\parallel }_F^2+\eta {\parallel {\mathbb{A}}^T\parallel }_F^2,$$

(1)

where $\eta$ is the regularisation parameter. ${\parallel ·\parallel }_F^2$ denotes Frobenius norm. $\parallel {\mathbb{A}}^T{\parallel }_F^2$ controls the model complexity to prevent overfitting.

3.2. Manifold Learning

In contrast to [4], which utilises a graph model to estimate data distribution on manifold, we model the local geometrical structure by generating between-class similarity graph $G_b$ and within-class similarity graph $G_w$ , where $G_w(i,j)=1$, if $x_i\in N_w\left(x_j\right)$ or $x_j\in N_w\left(x_i\right)$, otherwise $G_w(i,j)=0$. $G_b(i,j)$ applies the same method, although it selects $x_i\in N_b\left(x_j\right)$ or $x_j\in N_b\left(x_i\right)$, where $N_b\left(x_i\right)$ contains neighbours with different labels, $N_w\left(x_j\right)$ is the set of neighbours $x_j$ sharing the same label as $x_i$. Notably, the intra-class and inter-class distances be mapped on a manifold by similarity graphs [24].

Inspired by manifold learning [10,22,24], we maximised inter-class separability and minimised intra-class compactness simultaneously. An ideal transform pushes the connected points of $A_b$ to the extent possible while moveing the connected points of $A_w$ closer. The discriminative information can be represented as follows:

$$\begin{array}{cc}\hfill f& =\frac{1}{2}\sum _{i,j=1}^n{(F_i-F_j)}^2{G}_w(i,j)-\frac{1}{2}\beta \sum _{i,j=1}^n{(F_i-F_j)}^2{G}_b(i,j)\hfill \\ \hfill & =tr\left({\mathbb{F}}^T(L_w-\beta L_b)\mathbb{F}\right),\hfill \end{array}$$

(2)

where $\beta$ is a regularisation parameter, which controls the trade-off between inter-class separability and intra-class compactness. $tr(·)$ denotes the trace operator and $L_w=D_w-G_w$ denotes the Laplacian matrix. Furthermore, $D_b$ is a diagonal matrix with $D_b(i,i)={\sum }_{j=1}^n{G}_b(i,j)$, and $D_w$ is a diagonal matrix with $D_w(i,i)={\sum }_{j=1}^n{G}_w(i,j)$.

3.3. Multiple Feature Analysis

Multiple features imply combining kernelized embedding features, data-point manifold subspace learning (1st term in Eq.(4)), label propagation (2nd term in Eq.(4)) with low-level feature correlations (3rd term in Eq.(4)) for labeled and unlabeled data.

We modify the aforementioned function to leverage both labeled and unlabeled samples. First, the training dataset is redefined as $\mathbb{X}={[{\mathbb{X}}_l^T,{\mathbb{X}}_u^T]}^T$, where ${\mathbb{X}}_l={[X_1,...,X_m]}^T$ is the labeled data subset, and ${\mathbb{X}}_u={[X_{m+1},...,X_n]}^T$ is the unlabeled data subset. The label matrix $\mathbb{Y}={[{\mathbb{Y}}_l^T,{\mathbb{Y}}_u^T]}^T$, where ${\mathbb{Y}}_l={[Y_1,...,Y_m]}^T\in {\left\{1\right\}}^{m\times m}$. The unlabeled matrix ${\mathbb{Y}}_u={[Y_{m+1},...,Y_n]}^T\in {\left\{0\right\}}^{(n-m)\times (n-m)}$. According to [3,34], diagonal label matrix $\mathbb{Y}$ and the similarity graphs $G_w,G_b$ should be consistent with the label prediction matrix $\mathbb{F}$. We generalised the graph-embedded label consistency as follows:

$$\underset{\mathbb{F}}{min}tr\left({\mathbb{F}}^T(L_w-\beta L_b)\mathbb{F}\right)+{\parallel \mathbb{F}-\mathbb{Y}\parallel }_F^2,$$

(3)

In contrast to previous shared-structure learning algorithms, we did not consider shared-structure learning within a semi-supervised learning framework. Alternatively, we proposed a novel joint framework that incorporates the multiple-feature analyses of multiple manifolds. As discussed in the problem formulation section, by employing the Frobenius norm regularised loss function, we can reformulate the objective:

$$\begin{array}{cc}\hfill \underset{\mathbb{F},\mathbb{A}}{min}& tr\left({\mathbb{F}}^T(L_w-\beta L_b)\mathbb{F}\right)+{\parallel \mathbb{F}-\mathbb{Y}\parallel }_F^2\hfill \\ \hfill & +\mu \left(\parallel {\mathbb{A}}^T{\mathbb{K}-\mathbb{Y}\parallel }_F^2+\eta {\parallel {\mathbb{A}}^T\parallel }_F^2\right),\hfill \end{array}$$

(4)

where $\beta >0$, $\mu >0$ and $\eta >0$ are regular parameters.

The presented function (4) is an unconstrained convex optimisation problem, hence, we can obtain the global optimum by performing ALS or the projected gradient method. Although the correlation matrix can only be singular under specific circumstances, the projected gradient method can handle the aforementioned issues without matrix inversion [7], and therefore leads to a better optimum than ALS. Notably, the convergence conditions in [3,4] merely depend on a monotone decrease, which may result in mathematically improper convergence; therefore KKT conditions is utilized to consider this problem.

3.4. Grassmannian Kernels

The similarity between two action sample points $X_i$ and $X_j\in {\mathbb{R}}^{d\times 1}$ can be measured by projective kernel combination:

$$k_{i,j}^{\left[proj\right]}={\Vert X_i^T{X}_j\Vert }_F^2.$$

(5)

One attempt to solve the point matching problem was the notion of principal angles [22]. Given $X_i$ and $X_j$, we can define the canonical correlation kernel as

$$k_{i,j}^{\left[cc\right]}=\underset{a_p\in span\left(X_i\right)}{max}\underset{b_q\in span\left(X_j\right)}{max}{a}_p^T{b}_q,$$

(6)

subject to $a_p^T{a}_p=b_p^T{b}_p=1$ and $a_p^T{a}_q=b_p^T{b}_q=0,p\ne q$.

We create a combined Grassmannian kernel through existing Grassmannian kernels [22].

$$k^{[A+B]}={\delta }^{\left[A\right]}{k}^{\left[A\right]}+{\delta }^{\left[B\right]}{k}^{\left[B\right]},$$

(7)

where ${\delta }^{\left[A\right]},{\delta }^{\left[B\right]}\ge 0$. Notably, $k^{\left[A\right]}+k^{\left[B\right]}$ defines a new kernel based on the theory of reproducing kernel Hilbert space as described in [22].

3.5. Optimisation

According to [7,8], a general unconstrained minimisation problem can be solved by trace operator and PBB method. Hence, a new objective function $\mathrm{g}(\mathbb{F},\mathbb{A})$ instead of (4) is defined:

$$\begin{array}{cc}\hfill \mathrm{g}(\mathbb{F},\mathbb{A})& =tr\left({\mathbb{F}}^T(L_w-\beta L_b)\mathbb{F}\right)+tr{(\mathbb{F}-\mathbb{Y})}^T(\mathbb{F}-\mathbb{Y})\hfill \\ \hfill & +\mu tr{({\mathbb{A}}^T\mathbb{K}-\mathbb{Y})}^T({\mathbb{A}}^T\mathbb{K}-\mathbb{Y})+\mu \eta tr\left(\mathbb{A}{\mathbb{A}}^T\right).\hfill \end{array}$$

(8)

If $({\mathbb{F}}^{*},{\mathbb{A}}^{*})$ is an approximate stationary point in (8), it must satisfy the KKT conditions in (8). Then, we have a iteration-stopping criterion

$$\parallel \nabla {\mathrm{g}}_{\mathbb{F}}({\mathbb{F}}^{*},{\mathbb{A}}^{*}){\parallel }^2+{\parallel \nabla {\mathrm{g}}_{\mathbb{A}}({\mathbb{F}}^{*},{\mathbb{A}}^{*})\parallel }^2\le \epsilon ,$$

(9)

where $\epsilon$ is a non-negative small constant.

Algorithm 1: Kernel Grassmann Manifold Analysis (KGMA)

3.6. Projected Barzilai-Borwein

Similar to [7], a sequence of feasible points $({\mathbb{F}}^t,{\mathbb{A}}^t)$ are generated by the gradient method:

$$\begin{array}{cc}\hfill & d{\mathbb{F}}^t=-{\lambda }^t\nabla {\mathrm{g}}_{\mathbb{F}}({\mathbb{F}}^t,{\mathbb{A}}^t),{\mathbb{F}}^{t+1}={\mathbb{F}}^t+{\sigma }_{t}d{\mathbb{F}}^t,\hfill \\ \hfill & d{\mathbb{A}}^t=-{\lambda }^t\nabla {\mathrm{g}}_{\mathbb{A}}({\mathbb{F}}^t,{\mathbb{A}}^t),{\mathbb{A}}^{t+1}={\mathbb{A}}^t+{\sigma }_{t}d{\mathbb{A}}^t,\hfill \end{array}$$

(10)

where ${\sigma }_t$ denotes the non-monotone line search step size and ${\lambda }^t=min\{{\lambda }_{max},max\{{\lambda }_{min},{\lambda }_{ABB}^t\}\}>0$ is another step size, that is determined through an appropriate selection rule. Following [8], we have two choices for step size

$$\begin{array}{cc}\hfill & {\lambda }_{BB1}^{t+1}=\frac{\langle s_1^t,s_1^t\rangle +\langle s_2^t,s_2^t\rangle }{\langle s_1^t,y_1^t\rangle +\langle s_2^t,y_2^t\rangle },\hfill \\ \hfill & {\lambda }_{BB2}^{t+1}=\frac{\langle s_1^t,y_1^t\rangle +\langle s_2^t,y_2^t\rangle }{\langle y_1^t,y_1^t\rangle +\langle y_2^t,y_2^t\rangle },\hfill \end{array}$$

(11)

where

$$\begin{array}{cc}\hfill & s_1^t={\mathbb{F}}^{t+1}-{\mathbb{F}}^t,s_2^t={\mathbb{A}}^{t+1}-{\mathbb{A}}^t,\hfill \\ \hfill & y_1^t=\nabla {\mathrm{g}}_{\mathbb{F}}({\mathbb{F}}^{t+1},{\mathbb{A}}^{t+1})-\nabla {\mathrm{g}}_{\mathbb{F}}({\mathbb{F}}^t,{\mathbb{A}}^t),\hfill \\ \hfill & y_2^t=\nabla {\mathrm{g}}_{\mathbb{A}}({\mathbb{F}}^{t+1},{\mathbb{A}}^{t+1})-\nabla {\mathrm{g}}_{\mathbb{A}}({\mathbb{F}}^t,{\mathbb{A}}^t),\hfill \end{array}$$

(12)

The characteristic of the adaptive step sizes (11) can render the objective functions non-monotonic; hence, $\mathrm{g}({\mathbb{F}}^t,{\mathbb{A}}^t)$ may increase in some iterations. Alternatively, using (11) is better than merely using one of them [8]; the step size is expressed by

$${\lambda }_{ABB}^t=\left\{\begin{array}{cc}{\lambda }_{BB1}^t,\hfill & \mathrm{for}\mathrm{odd}\mathrm{number}t\hfill \\ {\lambda }_{BB2}^t,\hfill & \mathrm{for}\mathrm{even}\mathrm{number}t\hfill \end{array}\right.$$

(13)

To guarantee the convergence of $({\mathbb{F}}^t,{\mathbb{A}}^t)$, a globalisation strategy based on the non-monotone line-search technique is described as [7]

$$\begin{array}{cc}\hfill \mathrm{g}({\mathbb{F}}^{t+1},{\mathbb{A}}^{t+1})\le & C_t+\gamma {\sigma }_t\{\langle \nabla {\mathrm{g}}_{\mathbb{F}}({\mathbb{F}}^t,{\mathbb{A}}^t),d{\mathbb{F}}^t\rangle \hfill \\ \hfill & +\langle \nabla {\mathrm{g}}_{\mathbb{A}}({\mathbb{F}}^t,{\mathbb{A}}^t),d{\mathbb{A}}^t\rangle \}\hfill \end{array}$$

(14)

where $\tau \in (0,1],C_t$ are the parameters of the Armoji line-search method [8]. Following [7], in order to overcome some drawbacks of non-monotone techniques, the traditional largest function value is converted by the weighted average function value:

$$C_t=\frac{\tau ·min\{t-1,M\}{C}_{t-1}+\mathrm{g}({\mathbb{F}}^t,{\mathbb{A}}^t)}{\tau ·min\{t-1,M\}+1},$$

(15)

4. Experiments

The proposed method, called the Kernel Grassmann Manifold Analysis (KGMA), is summarised in Algorithm 1. The conventional method that uses SPG [10] and ALS method instead of PBB, called kernel spectral projected gradient analysis (KSPG) and kernel alternating least squares analysis (KALS), respectively, was also adopted to solve the objective function (8) for comparison in our experiments.

Features. For handcrafted features, we follow [10] to extracted improved dense trajectories (IDT) and Fisher vector (FV), as shown in Figure 2. For deep-learned features, we retrained the temporal segment network (TSN) [2] models of 15×c, and then extracted the global pool features of 15×c using pretrained TSN model, concatenating rgb+flow into 2048 dimensions with power L2-normalisation, as listed in Table 1.

We verified the proposed algorithm using three kernels: projection kernel $k^{\left[proj\right]}$, canonical correlation kernel $k^{\left[CC\right]}$, and combined kernel $k^{[proj+CC]}$. In some cases, $k^{\left[proj\right]}$ is better than $k^{\left[CC\right]}$, whereas vice versa, suggesting that the kernels combination is more suitable for different data distributions. For $k^{[proj+CC]}$, the mixing coefficients ${\delta }^{\left[proj\right]}$ and ${\delta }^{\left[CC\right]}$ were fixed at one. We obtain better results by combining ${\delta }^{[proj+CC]}$ two kernels.

Datasets. Three datasets were used in the experiments: JHMDB, HMDB51, and UCF101 [1]. The JHMDB dataset has 21 action categories. The average recognition accuracies over three training–test splits are reported. The HMDB51 dataset records 51 action categories. We reported the MAP over three training–test splits. The UCF101 dataset includes 101 action categories, containing 13,320 video clips. The average accuracy of the first split was reported.

For the JHMDB dataset, we followed the standard data partitioning (three splits) provided by the authors. For other datasets, we used the first split provided by the authors, and applied the original testing sets for fair comparison. Because the semi-supervised training set contained unlabeled data, we performed the following procedure to reform the training set for each individual dataset. the class number c was denoted for each dataset (c = 21, 51, and 101 for JHMDB, HMDB51, and UCF101, respectively).

Using JHMDB as an example, we first randomly selected 30 training samples per category to form a training set ( $30\times c$ samples) in our experiment. From this training set, we randomly sampled m videos (m = 3, 5, 10, and 15) per category as labeled samples. Therefore, if $m=10$, $10\times c$ labeled samples will be available, leaving ($30\times c-10\times c$) videos as unlabeled samples for the semi-supervised training setting. We used a standard test set as the test set. Owing to the random selected training samples, the experiments were repeated 10 times to avoid bias.

To demonstrate the superiority of our approach (KGMA), we adopted 8 methods for comparison: SVM, SFUS [35], SFCM [3], MFCU [4], KSPG, and KALS. Notably, SFUS, SFCM, MFCU, KSPG, and KALS are semi-supervised action recognition approaches. Using the available codes, we can facilitate a fair comparison.

Table 1. Comparison with deep-learned features (average accuracy ± std) when $15\times c$ training videos are labeled

	JHMDB	HMDB51	UCF101
SFUS	0.6942 ± 0.0121	0.5217 ± 0.0114	0.7910 ± 0.0087
SFCM	0.7125 ± 0.0099	0.5394 ± 0.0108	0.8070 ± 0.0101
MFCU	0.7154 ± 0.0088	0.5556 ± 0.0098	0.8429 ± 0.0085
SVM-${\chi }^2$	0.6931 ± 0.0106	0.5190 ± 0.0095	0.8138 ± 0.0108
SVM-linear	0.7140 ± 0.0086	0.5385 ± 0.0077	0.8450 ± 0.0087
KSPG	0.7287 ± 0.0114	0.5697 ± 0.0833	0.8552 ± 0.0111
KALS	0.7218 ± 0.0087	0.5607 ± 0.0098	0.8411 ± 0.0095
KGMA	0.7361±0.0096	0.5762±0.1040	0.8673±0.0087

Table 1. Comparison with deep-learned features (average accuracy ± std) when $15\times c$ training videos are labeled

	JHMDB	HMDB51	UCF101
SFUS	0.6942 ± 0.0121	0.5217 ± 0.0114	0.7910 ± 0.0087
SFCM	0.7125 ± 0.0099	0.5394 ± 0.0108	0.8070 ± 0.0101
MFCU	0.7154 ± 0.0088	0.5556 ± 0.0098	0.8429 ± 0.0085
SVM-${\chi }^2$	0.6931 ± 0.0106	0.5190 ± 0.0095	0.8138 ± 0.0108
SVM-linear	0.7140 ± 0.0086	0.5385 ± 0.0077	0.8450 ± 0.0087
KSPG	0.7287 ± 0.0114	0.5697 ± 0.0833	0.8552 ± 0.0111
KALS	0.7218 ± 0.0087	0.5607 ± 0.0098	0.8411 ± 0.0095
KGMA	0.7361±0.0096	0.5762±0.1040	0.8673±0.0087

For the semi-supervised parameters $\eta ,\beta ,\mu$ for SFUS, SFCM, MFCU, KSPG, KALS, and KGMA, we follow the same settings utilised in [3,4], ranging from {${10}^{-4},{10}^{-3},{10}^{-2},{10}^{-1},1,{10}^1,{10}^2,{10}^3,{10}^4$}. Because the PBB parameters were not sensitive to our algorithm, we initialised the parameters as in [7], as indicated in Algorithm 1. Notably, since KGMA applied PBB to solve the optimal value of objective function (8), it resulted in non-monotonic convergence with oscillating objective function values, as shown in Figure 3. Thus, using only the absolute error made it difficult to determine when to stop iterating, relative error of objective function values was better than absolute error, which may be mathematically improper convergence. We chose constant $\epsilon ={10}^{-4}$ as the iteration-stopping criterion in (9).

Mathematical Comparisons. The recognition results with handcrafted features on three datasets were demonstrated in Figure 2. We compared our method with deep-learned features in Table 1.

Regarding the presented objective function 8, Figure 3 summarized the computational results of the three optimization methods. When we used the 2048-dimensional deep-learned features TSN on JHMDB dataset, the model was trained with only 15 labeled samples and 15 unlabeled samples per class, setup the same semi-supervised parameters $\eta ,\beta ,\mu$, then the performance differences during the solving of the same objective function could be compared in terms of running time, number of iterations, absolute error, relative error, and objective function value. Figure 3 shown the convergence curves of three optimization methods. Since both SPG and PBB were non-monotonic optimization methods with relatively large fluctuations in objective function values, we omitted the first 29 iterations of SPG and PBB in Figure 3, and only displayed the data starting from the 30th iteration, so as to better illustrate the monotonic convergence process of ALS.

As shown in Table 2, for a randomly selected video data sample, ALS exhibited the fewest iterations, shortest running time and fastest computation speed of 0.1220 seconds after extracting the deep features by TSN. In contrast, PBB exhibited the most iterations, longest running time and slowest computation speed of 0.4212 seconds; while SPG’s performance were intermediate between ALS and PBB. Considering Figure 3 and Table 2, it is evident that despite using the PBB optimization method, our KGMA algorithm still achieves the highest accuracy on the kernelized Grassmann manifold space. Nevertheless, the equation 9 using SPG results in marginal improvement over ALS, which likely attributable to our novel kernelized Grassmann manifold space.

Performance on Action Recognition. A linear SVM was utilised as the baseline. Based on the comparisons, we observe the following:1) KGMA achieved the best performance, our semi-supervised algorithm was better than linear SVM which is widely-used supervised classifiers; 2) all methods achieved better performances using more labeled training data, as shown in Figure 2, or enlarging semi-supervised parameter (i.e., $\eta ,\beta ,\mu$) range such as Figure 4; 3) we averaged an accuracy of $3\times c$, $5\times c$, $10\times c$, and $15\times c$ cases, and the recognition of KGMA on JHMDB, HMDB51, and UCF101 improved by 2.97%, 2.59%, and 2.40%, respectively. When using TSN features, the recognition of our KGMA on above-mentioned datasets improved by 2.21%, 3.77%, and 2.23%, respectively. Evidently, our semi-supervised method can improve recognition by leveraging unlabeled data compared to linear SVM with labeled data merely. Figure 2 illustrated that our algorithm benefits from the multiple-feature analysis, kernelized Grassman space and iterative skills of PBB method.

These results can be attributed to several factors. First, our method not only leverages semi-supervised approaches, but also leverages intra-class action variation and inter-class action ambiguity simultaneously. Therefore, ours gain more significant performance than other approaches when there are few labeled samples. Second, we uncover the action feature subspace on Grassmannian manifold by incorporating Grassmannian kernels, and solve the objective function optimisation by adaptive line-search strategy and PBB method mathematically. Hence, the proposed algorithm works well in few labeled case.

Convergence Study. According to the objective function (4), we conducted experiments with the TSN feature, fixed the semi-supervised parameters $\eta ,\beta ,\mu$, and then executed both the ALS and PBB methods 10 times. The results of the study are listed in Table 2. Although no oscillation exists in the convergence of the ALS and it requires fewer iterations, the PBB method can outperform the ALS for three reasons. First, the PBB method uses a non-monotone line-search strategy to globalise the process [8], which can obtain the global optimal objective function value rather than being trapped in local optima using the monotone ALS method. Second, the character of adaptive step sizes is an essential characteristic that determines efficiency in the projected gradient methodology [8], whereas the iteration step skill has not been considered in ALS. Finally, the efficient convergence properties of the projected gradient method have been demonstrated because the PBB is well defined [8].

Computation Complexity. In the training stage, we computed the Laplacian matrix L, the complexity of which was $O\left(n^2\right)$. To optimise the objective function, we computed the projected gradient and trace operators of several matrices. Therefore, the complexity of these operations was $O\left(n^3\right)$.

Parameter Sensitivity Study. We verified that KGMA benefits from the intra-class and inter-class by manifold discriminant analysis, as shown in Figure 4. We analysis the impact of manifold learning on JHMDB and HMDB51, set $\eta ={10}^3$ and $\mu ={10}^{-1}$ at optimal values over split2, for $15\times c$-labeled training data. As $\beta$ varied from ${10}^{-4}$ to ${10}^4$, the accuracy oscillated significantly and reached a peak value when $\beta ={10}^4$. Since $\beta$ controls the proportion of the intra-class local geometric structure and the inter-class global manifold structure, as shown in Figure 4. when the intra-class local geometric structure is treated as a constant 1, $\frac{\beta }{1}$ can be considered that the inter-class global manifold structure has a larger proportion in the objective function, and vice versa. When $\beta =0$, no inter-manifold structure is utilised; thus, if $\beta \to +\infty$, no intra-class structure is present. When the Grassmann manifold space leverages an adequate balance of intra-class action variation and inter-class action ambiguity, the proposed algorithm can further enhance the discriminatory power of the transformation matrix.

5. Conclusion

This study proposed a new approach to categorise human action videos. With Grassmannian kernels combination and multiple-feature analysis on multiple manifolds, our method can improve recognition by uncovering the intrinsic features relationships. We evaluated the presented approach on three benchmark datasets, and experiment results show ours outperformed all competing methods, particularly when there are few labeled samples.