A Global-Local Differencing Network for Lightweight Point-Cloud Human Action Recognition

1. Introduction

Human action recognition supports applications such as human–computer interaction, rehabilitation assessment, ambient intelligence, and security surveillance. Depth sensors provide a practical compromise between RGB appearance and skeletal abstraction. They are insensitive to illumination changes and capture metric scene geometry, without requiring texture-rich images to be stored or relying on a pose estimator. Converting a depth sequence produces a dynamic point cloud, in which each frame is an unordered set of three-dimensional points. Early depth-based action-recognition methods mostly operated on depth images and represented temporal changes using two-dimensional representations, including histograms of oriented normals, dynamic images, and structured depth images [1–3]. With advances in deep learning for point clouds [4,5], recent research has shifted markedly toward action-recognition methods that model three-dimensional point clouds directly [6–13], thereby recasting action recognition as a point-cloud sequence modeling problem.

The lack of stable point correspondences is a central obstacle in dynamic point-cloud modeling. Most current methods address this issue by constructing local spatiotemporal neighborhoods. MeteorNet performs feature aggregation directly on dynamic point sets [6]; P4Transformer and PSTNet combine spatial grouping with temporal aggregation [7,8]; and PST-Transformer enhances the representation using displacement-aware attention [9]. More recent studies have explored feature-level surfaces [10], minimalist pointwise architectures [11], state-space models [12], spectral-domain mixing [13], point-relationship guidance [33], and hierarchical collaboration [34]. Despite following different technical routes, most of these methods retain the same underlying paradigm of local geometric construction. They rely on intensive farthest point sampling, anchor grouping, and overlapping neighborhood queries, resulting in high overall computational cost. A single descriptor obtained by global pooling can substantially reduce computation, but discriminative information from subtle movements of body parts such as the arms and legs can be obscured. Existing methods essentially couple spatial locality with motion modeling by relying on detailed local-neighborhood searches to capture motion implicitly. This raises two questions: if motion cues can be obtained explicitly and more directly, can the requirement for spatial locality be substantially relaxed, and how far can the model be simplified?

To address these questions, this paper proposes the Global–Local Differencing Network (GLD-Net). Its central idea is to decouple motion extraction from spatial locality by replacing implicit local-geometric learning with explicit motion measures and fine-grained neighborhood searches with coarse body partitions. The model has three main components. For feature extraction, it uses global and local branches. The global branch represents the complete point cloud in each frame, whereas the local branch uniformly divides the body along the vertical direction into several semantic regions and encodes each region with an independent pointwise network. This coarse decomposition along the body axis retains discriminative cues from important regions such as the arms and legs while avoiding the cost of farthest point sampling and ball queries. The local features are aggregated by attention and paired frame by frame with the global features, after which a temporal Transformer models them jointly. For motion representation, the distances from each point to its nearest neighbors in the preceding and subsequent frames directly describe its motion state: stationary regions produce small matching distances, whereas moving limbs produce large residuals. This representation requires neither point correspondences nor learnable parameters. For temporal modeling, bidirectional differencing is applied to frame-level features to incorporate action changes explicitly. The temporal Transformer can therefore focus on recognizing action patterns rather than deriving motion from static frames. The complete framework requires no farthest point sampling, ball queries, or learnable four-dimensional neighborhood construction.

The main experiments are conducted on MSR-Action3D [55]. GLD-Net achieves 95.82% recognition accuracy with 0.579 M parameters and 1.42 G operations. Compared with PvNeXt [11], a recent model of similar scale, GLD-Net achieves 1.05 percentage points higher accuracy with 19.6% fewer parameters. Figure 1 shows the positions of the selected models and representative methods in the accuracy–parameter plane. To facilitate reproducibility, the implementation code is publicly available at https://github.com/daidai321/GLD-Net.

The main contributions of this paper are as follows:

1.

A two-branch global and local feature extraction scheme is proposed. The global branch represents the complete point cloud in each frame, whereas the local branch uniformly divides the body along the Y-axis and extracts features from each segment independently. Attention-based fusion combines holistic context with local body evidence without requiring any neighborhood-construction operation.

2.

A fast nearest-neighbor motion representation is proposed. It represents motion using the distances from each point to its nearest neighbors in the preceding and subsequent frames. The representation is parameter-free, requires no point correspondences, and can be attached directly to any point-cloud sequence model.

3.

Bidirectional differencing is applied to frame-level features to separate action changes explicitly from static frame features, allowing the temporal Transformer to focus on action-pattern recognition. A constant-parameter ablation protocol is used to verify the independent contribution of each design.

2. Related Work

2.1. Point-Cloud Feature Extraction

PointNet pioneered static point-cloud feature learning by combining a shared pointwise network with symmetric pooling to learn permutation-invariant set representations [4]. Deep Sets and Set Transformer subsequently provided a theoretical formalization of invariance and attention-based set aggregation, respectively [14,15]. PointNet++ introduced local geometry into point-cloud learning through hierarchical sampling and grouping [5]. In parallel with pointwise methods, voxel-based approaches first quantize point clouds into regular grids and then extract features using three-dimensional convolutions [16], at the cost of quantization errors and memory consumption that increases with resolution. Subsequent studies have focused on how to model neighborhoods, including learnable permutations [17], dynamic graphs [18], continuous convolutions [19], deformable kernels [20], and vector attention [21]. Another line of research has shown that residual architectures and carefully designed training strategies can also be effective without introducing complex neighborhood operators [22–24]. Masked point modeling has further extended point-cloud representation learning to the pretraining paradigm [25,26].

Dynamic point-cloud models add temporal associations to these spatial representations. Sparse four-dimensional convolutions process spatiotemporal coordinates on voxel grids [27]. MeteorNet, P4Transformer, PSTNet, and PST-Transformer established the prevalent paradigm of combining spatial grouping with temporal aggregation [6–9], while Kinet and PSTNet++ further improved feature-level surfaces and operator decomposition [10,28]. Recent action-recognition models have evolved along several directions, including explicit action structures [29], minimal pointwise designs [11], state-space models [12], spectral-domain modeling [13,30], adaptive decoupling [31], KAN enhancement [32], point-relationship guidance [33], and hierarchical collaboration [34]. Unlike these methods, GLD-Net performs neither iterative nor learnable point-neighborhood construction and organizes spatial information using only one complete point set and three fixed-height segments.

2.2. Motion Representation in Action Recognition

The central issue in action recognition is how to represent motion information. Early depth-based methods mainly made motion explicit in two-dimensional representations, including histograms of normals [1], dynamic images [2], structured depth images [3], HHA encoding [35], and depth-skeleton sequences [36]. Spatial encodings based on orthographic projection have also provided an efficient approach to depth-based human analysis [37–38]. Although these representations are intuitive and effective, they regularize or project the original three-dimensional samples. After the transition to point clouds, one line of research directly estimated dense scene flow between adjacent point sets [39]. For action recognition, however, dense flow constitutes an additional prediction problem and incurs considerable computational and supervision costs. Another line of research characterizes changes at the feature level: rather than requiring a network to infer motion implicitly from static frame features, inter-frame differences can be provided explicitly to the model. The two designs in this paper provide lightweight implementations of these two directions. Nearest-neighbor distances provide only a scalar motion cue without requiring pointwise correspondences; they neither move points, alter labels, nor introduce parameters. Bidirectional feature differencing makes ’action changes’ explicit at the feature level, so that the temporal model does not need to infer the motion process by itself from static frame features.

2.3. Temporal Modeling in Action Recognition

One class of methods that directly models temporal information in raw point-cloud sequences uses pointwise spatiotemporal operators [6–9,27]. These methods maintain high-resolution spatiotemporal feature maps at both point and frame granularities, and pointwise neighborhood searches result in persistently high computational and memory costs. Transformers provide a global receptive field for sequence modeling [40], but their attention cost grows quadratically with the number of tokens. Video Transformers generally control this cost by factorizing spatial and temporal attention or reducing token redundancy [41,42]. For point-cloud sequences, treating every point as a token would make this cost prohibitive. Representing each complete frame as a token instead makes the scale of temporal modeling independent of the number of points per frame, providing a natural choice that balances efficiency and global dependencies. Following this idea, GLD-Net compresses each frame into one global token and one local token. The attention-weighted aggregation of local features is related to attention-based multiple-instance learning [43], except that the ’instances’ here are differenced segment-level frame features rather than image patches.

3. Methods

The complete pipeline of GLD-Net is shown in Figure 2. For every point in each frame point cloud ($N\times 3$), the network first computes its nearest-neighbor (NN) distances $d^{-}$ and $d^{+}$ to the preceding and succeeding frame point clouds. After frame-wise normalization, these distances are concatenated with the three-dimensional coordinates $(x,y,z)$ of the point itself and a normalized temporal feature to form a six-dimensional input ($N\times 6$). Global and local networks then extract features separately. The global network directly encodes the complete $N\times 6$ input using PointNet, whereas the local network partitions the human point cloud into K segments along the Y-axis ($K=3$ on MSR-Action3D in this paper) and extracts features from them using mutually independent PointNets. The features in each stream are bidirectionally differenced from the corresponding-stream features of the preceding and succeeding frames, making ’action changes’ explicit at the feature level. The K differenced local features are then aggregated into one local token through attention weighting. Finally, each frame retains two independent tokens, one global and one local. A temporal Transformer captures temporal associations throughout the complete sequence through self-attention and maps them to an action prediction. The complete network constructs no point-level neighborhoods, while integrating three complementary cues: global structure, local structure, and motion changes.

3.1. Input Representation and Preprocessing

Action recognition on point-cloud sequences can be formulated as follows: given a point-cloud sequence back-projected from consecutive depth frames, predict the action category to which the sequence belongs. For a clip containing T frames, the tth frame contains N three-dimensional points, denoted by $P_t=\left\{p_{t,i}\right\}$, where $p_{t,i}\in {\mathbb{R}}^3$. All coordinates are uniformly divided by 300 for scale normalization. This normalization does not involve centroid alignment, and the relative displacement between frames is therefore fully preserved. Each frame is resampled to 2048 points. When more than 2048 points are available, points are randomly sampled without replacement; when fewer points are available, the points are cyclically repeated and the remainder is filled by random sampling.

3.2. Point-Level Bidirectional Nearest-Neighbor Motion Representation

Points are unordered, so inter-frame motion cannot be computed by direct index-wise subtraction as in grid-based videos. For each point $p_{t,i}$, its minimum distances to the preceding and succeeding point sets are therefore defined as the backward and forward residuals, respectively, as shown in Equation (1):

$$d_{t,i}^{-}=\underset{j}{min}{\parallel p_{t,i}-p_{t-1,j}\parallel }_2,d_{t,i}^{+}=\underset{j}{min}{\parallel p_{t,i}-p_{t+1,j}\parallel }_2.$$

(1)

Here, $p_{t,i}$ is the ith point in the tth frame, the index j traverses all points in the adjacent-frame point set, ${\parallel ·\parallel }_2$ denotes the Euclidean distance, and $d_{t,i}^{-}$ and $d_{t,i}^{+}$ are the minimum distances from this point to the preceding and succeeding point sets, respectively. At temporal boundaries, only the available one-sided adjacent frame is used. The two distance fields are separately normalized within each frame, as shown in Equation (2):

$$a_{t,i}^{\pm }={\displaystyle \frac{d_{t,i}^{\pm }}{{max}_l{d}_{t,l}^{\pm }+ϵ}}.$$

(2)

Here, the index l traverses all points in the tth frame, $ϵ$ is a very small constant that prevents division by zero, and the normalized value $a_{t,i}^{\pm }$ is the nearest-neighbor motion representation of the point. This operation contains no parameters and requires neither optical-flow ground truth nor point correspondences. A larger distance indicates that the point is more difficult to explain using the adjacent point set and is therefore more likely to lie on a moving surface, a newly visible region, or a boundary affected by motion.

The nearest-neighbor search in Equation (1) is implemented by brute force. For each frame, the $N\times N$ distance matrix between it and an adjacent frame is directly computed, and the minimum is then taken along the adjacent-frame point dimension. Its computational complexity is $O\left(T{N}^2\right)$, making it the only quadratic term in the complete model. This step contains no parameters, requires no backpropagation, and is implemented as a single batched matrix operation on a GPU. The measured per-clip inference latency reported in Section 4.2 fully includes this computation. Compared with iterative neighborhood-construction operations such as farthest point sampling and ball query, it has a higher asymptotic complexity but requires only a single execution and can be fully parallelized.

The final six-dimensional point descriptor is given by Equation (3):

$$q_{t,i}=[x_{t,i},y_{t,i},z_{t,i},a_{t,i}^{-},a_{t,i}^{+},{\tau }_t],$$

(3)

where $q_{t,i}$ is the six-dimensional descriptor of point $p_{t,i}$; $x_{t,i}$, $y_{t,i}$, and $z_{t,i}$ are its three-dimensional coordinates; and ${\tau }_t\in [0,1]$ is the normalized temporal feature, namely, the relative position of the frame index within the clip. The nearest-neighbor search and construction of the six-dimensional descriptor are illustrated in Figure 3.

3.3. Global and Local Feature Extraction

The complete frame is encoded by a global pointwise network ${\varphi }_g$. Because actions occur primarily in body parts such as the arms and legs, body-part-level local information is introduced by uniformly dividing the interval $[y_{min},y_{min}+H]$ along the Y-axis, corresponding to the human height direction, into $K=3$ non-overlapping segments. In ascending order of the Y-coordinate, these segments are denoted by $R_1$, $R_2$, and $R_3$, and points outside the interval are assigned to the nearest boundary segment. Here, $y_{min}$ and H are constants defined in the normalized coordinates described in Section 3.1. On MSR-Action3D, $y_{min}=-0.5$ and $H=1.25$, covering the height range of the human bodies in this dataset. The segment boundaries are therefore fixed throughout the complete dataset and do not adapt to individual frames or samples, thereby avoiding segment jitter introduced by frame-wise statistics. The specific values of K, D, and other parameters presented in this section correspond to the MSR-Action3D configuration; the corresponding values for NTU RGB+D 60 are given in Section 4.5. Each segment uses an independent encoder ${\varphi }_k$ without parameter sharing because the upper, middle, and lower segments differ markedly in body structure and motion patterns. All encoders operate directly on the six-dimensional point attributes, as shown in Equation (4):

$$f_t^g=\underset{i}{max}{\varphi }_g\left(q_{t,i}\right),f_t^k=\underset{i:p_{t,i}\in R_k}{max}{\varphi }_k\left(q_{t,i}\right).$$

(4)

Here, max denotes channel-wise max pooling, $R_k$ is the set of points in the kth segment, and $f_t^g$ and $f_t^k$ are the global feature and the local feature of the kth segment in the tth frame, respectively. The layer widths of the global encoder are 6–64–128–128. The layer widths of each local encoder are only 6–32–32–128, directly producing a 128-dimensional feature vector rather than first concatenating several narrow vectors and then expanding their dimensionality. Both types of encoder consist of three linear layers. The first two layers are followed by GELU activations, the final layer has no activation, and batch normalization is not used at any stage. If a segment contains no points in a frame, its feature vector is set to zero. Unlike PointNet++-style grouping, the three segments are fixed and non-overlapping, requiring neither anchor-point sampling nor radius-based search. Local structure is therefore introduced with almost no additional cost.

3.4. Feature-Level Bidirectional Differencing

After the encoding described in Section 3.3, each frame yields $K+1$ streams of 128-dimensional features: one global feature $f_t^g$ and K local features $f_t^1,\dots ,f_t^K$. These features describe only the static form of a single frame, and changes between frames have not yet been represented explicitly. This section applies bidirectional temporal differencing to each feature stream, directly computing ’action changes’ for the subsequent Transformer rather than requiring it to infer motion from static frame features by itself. Specifically, let $s\in \{g,1,\dots ,K\}$ index the $K+1$ feature streams. Each stream is subtracted from the corresponding-stream features $\gamma$ frames earlier and $\gamma$ frames later, as shown in Equation (5):

$${\Delta }^{-}{f}_t^s=f_t^s-f_{t-\gamma }^s,{\Delta }^{+}{f}_t^s=f_{t+\gamma }^s-f_t^s,$$

(5)

where ${\Delta }^{-}{f}_t^s$ and ${\Delta }^{+}{f}_t^s$ are the backward and forward differences of the sth feature stream, respectively, and $\gamma$ is the temporal interval. We use $\gamma =2$ throughout the paper, so that differencing is performed over a longer temporal range than adjacent frames while the interval remains much shorter than the 24-frame window. This value is fixed in all experiments and is not tuned separately for individual datasets. Unavailable differences at the boundaries are set to zero. For each stream, the current feature, backward difference, and forward difference are concatenated into a 384-dimensional vector and then mapped back to 128 dimensions as the final frame feature of that stream. The global stream uses a direct linear transformation, as shown in Equation (6):

$${\tilde{f}}_t^g=W_g[f_t^g\parallel {\Delta }^{-}{f}_t^g\parallel {\Delta }^{+}{f}_t^g]+b_g.$$

(6)

Here, ∥ denotes channel-wise concatenation, $W_g\in {\mathbb{R}}^{128\times 384}$ and $b_g\in {\mathbb{R}}^{128}$ are learnable weights and biases, and ${\tilde{f}}_t^g$ is the final frame feature of the global stream. The three local streams do not directly apply this transformation but instead use two steps, compression followed by restoration. A $384\to 32$ linear layer first compresses the feature, followed by GELU activation and dropout [56], and a $32\to 128$ linear layer then restores the feature dimension. This structure is referred to as a low-rank bottleneck, and its output is denoted by ${\tilde{f}}_t^k$ ($k=1,2,3$), which serves as the final frame feature of the kth local stream. The local streams do not use the same one-step $384\to 128$ transformation as the global stream because experiments showed that such a transformation was difficult to optimize on the small dataset, whereas introducing an intermediate 32-dimensional compression substantially improved training stability. At this point, the network contains explicit motion information at two levels. The nearest-neighbor distances in Section 3.2 measure the geometric non-correspondence of individual points at the input level, whereas the differences in this section measure temporal changes in pooled features at the feature level. The two operate at different levels and are not redundant. The structures used for differencing and mapping in the individual streams are shown in Figure 4.

3.5. Temporal Encoding and Classification

The K local feature streams output by Section 3.4 must be merged into one stream. Rather than using simple concatenation or summation, learnable scalar attention is introduced to compute a scalar weight for each local feature. The weights are normalized across the segments using softmax and are then used to form a weighted sum of the features, as shown in Equation (7):

$${\alpha }_t^k={softmax}_k\left(w_a^{\top }{\tilde{f}}_t^k\right),{\tilde{f}}_t^r={\displaystyle \sum _{k=1}^K}{\alpha }_t^k{\tilde{f}}_t^k.$$

(7)

Here, $w_a\in {\mathbb{R}}^{128}$ is a learnable scoring vector, ${softmax}_k$ denotes normalization along the segment dimension, ${\alpha }_t^k$ is the weight of the kth local feature, and ${\tilde{f}}_t^r$ is the weighted sum. The weights are learned by the network, allowing different actions to focus on different segments. For a segment that contains no points in a frame in Section 3.3, its feature vector is zero. After the low-rank bottleneck, it still produces a constant output determined by the bias and participates in the normalization in Equation (7). Because the human body is continuously distributed along the Y-axis in both MSR-Action3D and NTU RGB+D 60, empty segments do not occur in the actual data, and no additional mask is applied for this case. The vector ${\tilde{f}}_t^r$ is the local feature of the frame. It is paired with the global feature frame by frame, yielding one global token and one local token for each frame and forming the token sequence $X={\{{\tilde{f}}_t^g,{\tilde{f}}_t^r\}}_{t=1}^T\in {\mathbb{R}}^{T\times 2\times D}$, which is then passed to the temporal Transformer.

Self-attention itself does not distinguish the order of its inputs, so the model must be explicitly informed of the frame to which each token belongs. A set of learnable temporal embeddings is therefore introduced, and both tokens from the tth frame receive the same vector $e_t\in {\mathbb{R}}^D$: $U_{t,s}^0=X_{t,s}+e_t$, where $X_{t,s}$ is the token of type s in the tth frame, $s\in \{g,r\}$, and $D=128$ is the feature dimension. No additional identity embedding is introduced to distinguish the global and local tokens. Temporal information appears in the network at two levels. The scalar ${\tau }_t$ in Equation (3) enters the pointwise descriptor, allowing every point to perceive its frame position during encoding, whereas $e_t$ provides frame order to self-attention at the token level. Because the two operate at different granularities, both are retained. Finally, the 24 frames × 2 token types, for a total of 48 tokens, are arranged into a single sequence in temporal order as the encoder input.

The temporal Transformer consists of $L=3$ pre-normalization layers with $h=8$ attention heads and a head dimension of $d_h=D/h=16$. The layer-wise computations are shown in Equations (8)–():

$$\begin{array}{cc}\hfill H^{\ell }& =LN\left(U^{\ell }\right),[Q,K,V]=H^{\ell }[W_Q,W_K,W_V],\hfill \end{array}$$

(8)

$$\begin{array}{cc}\hfill A_j^{\ell }& =softmax\left({\displaystyle \frac{Q_j{K}_j^{\top }}{\sqrt{d_h}}}\right),O^{\ell }={Concat}_{j=1}^h\left(A_j^{\ell }{V}_j\right),\hfill \end{array}$$

(9)

$$\begin{array}{cc}\hfill Y^{\ell }& =U^{\ell }+Dropout\left(GELU\left(O^{\ell }{W}_O\right)\right).\hfill \end{array}$$

(10)

Here, $U^{\ell }$ is the input token sequence of the ℓth layer, LN denotes layer normalization, $W_Q$, $W_K$, $W_V$, and $W_O$ are learnable projection matrices, $Q_j$, $K_j$, and $V_j$ are the query, key, and value of the jth head, respectively, and Concat denotes concatenation of the outputs of the h heads. Equation () includes a GELU activation after the attention output projection $W_O$. This differs from a standard Transformer encoder, which directly adds the residual after the output projection, and is a design retained in the implementation of this paper. After the tokens are arranged into a single sequence, self-attention computes pairwise associations among all 48 tokens. Each head therefore produces a $48\times 48$ attention matrix $A_j^{\ell }$, whose element in row i and column m represents the degree to which the ith token attends to the mth token. Because these 48 tokens span all 24 frames and both global and local token types, this matrix simultaneously covers long-range inter-frame dependencies and interactions between global and local information. After the attention sublayer, each token independently passes through a small multilayer perceptron that first expands the 128-dimensional feature to 256 dimensions, applies a GELU activation, and then compresses it back to 128 dimensions, as shown in Equation (11):

$$U^{\ell +1}=Y^{\ell }+Dropout\left(W_{2}Dropout\left(GELU(W_{1}LN\left(Y^{\ell }\right))\right)\right).$$

(11)

Here, $W_1\in {\mathbb{R}}^{256\times 128}$ and $W_2\in {\mathbb{R}}^{128\times 256}$ are the learnable weights of the feed-forward network, and $Y^{\ell }$ is the output of the attention sublayer. The complete process is shown in Figure 5.

The output of the temporal Transformer remains a set of 48 tokens with 128 dimensions. To obtain a single representation for the complete video clip, the maximum among the 48 tokens is taken for each feature channel and aggregated into a 128-dimensional vector. After layer normalization, dimensionality expansion, and activation, a classification layer maps the vector to scores for the 20 action categories. During evaluation, the softmax probabilities of all windows from the same video are accumulated before the final category is produced. The classification process is given by Equation (12):

$$z=\underset{t,s}{max}{U}_{t,s}^L,q=Dropout\left(GELU(W_{p}LN\left(z\right))\right),o=W_{c}q+b_c.$$

(12)

Here, z is the pooled clip-level feature, $W_p$, $W_c$, and $b_c$ are the learnable weights and bias of the classification head, q is the intermediate hidden vector, and o contains the scores for the 20 action categories.

4. Experiments

4.1. Datasets

Experiments are conducted on the MSR-Action3D dataset [55]. The dataset contains 20 action categories performed by 10 subjects and is recorded as depth-map sequences. We convert the depth maps into point clouds following the procedure described in Section 3.1. Following the common setting used in previous studies [7,8], we adopt a cross-subject split: sequences from subjects 1–5 are used for training, and sequences from subjects 6–10 are used for testing. To ensure comparability, we use the same fixed cross-subject split and benchmark-level model analysis protocol as P4Transformer, PSTNet, PST-Transformer, and PvNeXt [7–9,11]. In addition, Section 4.5 evaluates generalizability on NTU RGB+D 60.

4.2. Implementation Details

Each input clip consists of 24 consecutive frames, with 2048 points sampled from each frame. Following the protocols used in previous studies [7–9], sliding windows are extracted from each sequence during training. During testing, all consecutive 24-frame windows are enumerated, and the softmax probabilities of all windows from the same video are accumulated to obtain the video-level prediction. We report the highest video-level accuracy achieved during each complete 50-epoch training run. Unless otherwise stated, all comparisons, ablations, and parameter sweeps below use the same protocol: the data split, clip length, number of points per frame, window enumeration procedure, and video-level voting remain unchanged, and only the factor under investigation is varied.

Four types of random augmentation are applied to the input clips during training. The first is independent random scaling along each axis, with scaling factors uniformly sampled between 0.9 and 1.1. The second is random rotation around the Y-axis of the human subject, with angles between $-{15}^{\circ }$ and ${15}^{\circ }$. The third is independent random point dropout in each frame. Between one half and all of the points in each frame are randomly retained, with a lower limit of 64 points to prevent individual frames from becoming excessively sparse. The fourth is coordinate jittering, in which zero-mean Gaussian noise with a standard deviation of 0.01 is added to each point. The random scaling and rotation parameters are shared across all frames in a clip, whereas point dropout and coordinate jittering are applied independently to each frame. No augmentation is applied during testing.

The model uses a feature dimension of $D=128$, three local segments, and a three-layer, eight-head temporal Transformer. The dropout rates at four locations in the network are 0.1 in the low-rank bottleneck of the local stream, 0.2 after local-token attention aggregation, 0.1 within the temporal Transformer, and 0.2 in the classification head. No dropout is used within the pointwise encoders. The model is optimized using SGD with a momentum of 0.9 and a weight decay of ${10}^{-4}$. The learning rate is linearly warmed up to 0.01 during the first ten epochs and reduced to one tenth of its previous value at epochs 20 and 30. Training lasts for 50 epochs with a batch size of 24. The complete model contains 578,613 trainable parameters (0.579 M).

On a single NVIDIA RTX 3090, each training epoch takes approximately 45 seconds. Including a complete test-set evaluation at the end of each epoch, each epoch takes approximately 84 seconds, and all 50 epochs require approximately 70 minutes.

Inference efficiency is evaluated using FP32 on a single NVIDIA RTX 3090. With a batch size of 1, the model processes 107 input clips of size $24\times 2048$ per second, with a mean latency of 9.3 ms and peak GPU memory usage of 112 MiB. When the batch size is increased to 24, throughput increases to 145 clips per second. The latency, throughput, and memory measurements above all correspond to the actual 24-frame input. To ensure consistency with the computation values directly reported for previous methods, Table 1 uses the measured result of the same network configuration with a $16\times 2048$ input: the total computation is 1.42 G.

4.3. Comparison on MSR-Action3D

As shown in Table 1, GLD-Net achieves an accuracy of 95.82% with 0.579 M parameters and 1.42 G computation. The recently proposed PRG-Net and EchoNet are included in the table. EchoNet achieves the highest accuracy, but it contains 6.42 M parameters. Among the methods for which parameter counts are reported, PvNeXt is the only other sub-million-parameter model in the table. Compared with PvNeXt, GLD-Net improves accuracy by 1.05 percentage points while reducing the parameter count by 0.141 M, or 19.6%; its computation increases from 0.55 G to 1.42 G. These results show that the advantage of the proposed method lies in achieving higher accuracy with fewer parameters.

Point-level visualizations of four actions over five adjacent frames are shown in Figure 6. In the nearest-neighbor motion representations in the upper row, the large responses for arm waving and hammering are concentrated on the moving arm, whereas those for side kicking are concentrated on the kicking leg after the kick begins. During jogging, the entire body moves, and the responses spread over the whole body. This occurs because the within-frame normalization in Equation (2) removes the absolute motion magnitude. Although this representation indicates where motion occurs, body-part-level localization still requires local segmentation and feature differencing. The lower row marks the input points that attain the maximum value in channel-wise max pooling. These points are sparsely distributed over locations such as the head, hands, and feet and do not completely coincide with the large-distance regions in the upper row. The nearest-neighbor distance is only an input-level motion measurement; the learned features determine which points support the pooled representation.

4.4. Ablation Experiments

This section evaluates each design of GLD-Net through ablation experiments. We examine the contributions of global and local feature extraction, nearest-neighbor motion representation, and feature differencing, as well as the effects of the number of layers and heads in the temporal Transformer. The component ablations in Table 2 use a constant-parameter protocol. When a component is removed, only its corresponding input is set to zero, or its output is set directly equal to its input; no parameters are deleted. All 0.579 M trainable parameters are therefore retained in the model, allowing the reported performance changes to be attributed to the removed information rather than differences in model capacity. The sweeps over the number of local segments and the temporal Transformer architecture in Table 3 and Table 4 necessarily change the parameter count. Their results are used only to discuss trends in the hyperparameter settings and do not constitute capacity-controlled comparisons. All variants use the same optimizer, data split, sampling procedure, and evaluation protocol.

Global and local features. The accuracy is 92.68% when only global features are used. Adding local features increases the accuracy by 3.14 percentage points to 95.82%, indicating that local motion information and global context are complementary. The effect of the number of segments is shown in Table 3. Three segments produce the highest accuracy, followed closely by four segments at 95.47%. Two segments are too coarse to preserve sufficient local evidence. When the number of segments exceeds four, the accuracy does not continue to improve despite the linear increase in the parameter count. The six- and seven-segment settings degrade substantially, whereas the eight-segment setting partially recovers to 93.03%. This non-monotonic variation indicates that the alignment between uniform segments and body parts interacts with the number of points within each segment. The three-segment setting provides the best balance among localization capability, the number of points within each segment, and model complexity. Following the established evaluation practice of previous point-cloud action-recognition studies on MSR-Action3D, all sensitivity analyses and ablation experiments are conducted using the fixed cross-subject benchmark split. Because the dataset does not provide an official validation-set split, these results should be interpreted as benchmark-level sensitivity analyses rather than independent validation-set estimates.

Nearest-neighbor motion representation. Setting both nearest-neighbor distance channels to zero reduces the accuracy from 95.82% to 88.15%. The resulting decrease of 7.67 percentage points is the largest single-component decrease, indicating that the point-level nearest-neighbor motion representation is the primary source of the model’s discriminative ability.

Feature differencing. Setting the bidirectional feature differences to zero reduces the accuracy to 91.99%, a decrease of 3.83 percentage points. This result indicates that explicitly represented action changes provide information beyond the frame-wise features.

Temporal Transformer. Removing the temporal Transformer reduces the accuracy from 95.82% to 92.33%, a decrease of 3.49 percentage points, indicating that self-attention modeling of inter-frame dependencies is an important source of recognition performance. The effects of depth and number of heads are shown in Table 4. With eight heads, the one-layer encoder achieves 95.47%, which is 0.35 percentage points lower than the three-layer encoder, while using 45.7% fewer parameters. The two-layer encoder achieves 94.08%. The four-layer encoder collapses to 11.85% when trained with the learning rate of 0.01 used for the other configurations. When the learning rate is reduced to 0.001 and all other settings remain unchanged, its accuracy reaches 89.55%, but it remains 6.27 percentage points below the three-layer result. With the depth fixed at three layers, the accuracy increases from 91.99% with two heads to 94.77% with four heads and 95.82% with eight heads. Therefore, the final model uses a three-layer, eight-head temporal Transformer.

4.5. Generalization Validation on NTU RGB+D 60

All preceding experiments are conducted on MSR-Action3D. To examine the feasibility of the proposed design on a larger dataset, we conduct a validation experiment on NTU RGB+D 60 [57]. The dataset was recorded by three cameras in 17 acquisition setups and contains 56,880 videos of 60 action categories performed by 40 subjects. Following the official cross-subject protocol, 40,320 videos are used for training and 16,560 videos are used for testing. The point clouds are back-projected from the officially provided masked human depth maps, consistent with previous studies [7,8,11].

The official masked depth maps still contain residual ground points around the feet and body boundaries. Because each combination of acquisition setup and camera corresponds to a fixed camera position, we calibrate one ground plane for each of the 17 setups × 3 cameras, giving 51 viewpoints in total. For each viewpoint, several spatially distributed ground points are manually selected in one depth frame, and a plane is fitted using RANSAC. The fitted plane is then automatically applied to all sequences from that viewpoint. It is used both to remove ground points and to transform the point cloud into a ground-referenced world coordinate system [38], so that the Y-coordinate directly represents height above the ground. Because the comparison methods do not perform this step, we report results under two input conditions: using only the officially provided masked depth maps, which are the inputs used by the comparison methods, and removing the ground points from these inputs.

Implementation setting. Because NTU RGB+D 60 contains substantially more classes and data than MSR-Action3D, the feature dimension is increased to $D=256$, and the number of local segments is increased to $K=5$. The remaining architecture is unchanged, with a three-layer, eight-head temporal Transformer and $\gamma =2$. The complete model contains 2,200,861 trainable parameters (2.20 M). Because the point clouds have been transformed into a ground-referenced coordinate system, the segmentation interval is no longer defined using constants in normalized coordinates. Instead, it is set directly according to height above the ground, with $y_{min}=0$ and $H=2$ m. Each of the five segments covers 0.4 m, giving the correspondence with body parts a clear physical meaning. Each clip still contains 24 frames × 2048 points. Unlike the sliding-window protocol in Section 4.2, four randomly segmented samples are drawn from each video during training, giving 161,280 clips in total. During testing, one deterministic segmented sample is drawn from each video, giving 16,560 clips in total. The prediction of this clip is used directly as the video-level result without multi-window voting. Training augmentation includes frame-wise point dropout after sampling, with a minimum retention ratio of 0.5; rotation of $\pm {15}^{\circ }$ around the Y-axis; mirroring with a probability of 0.5; tilting by $\pm 5^{\circ }$ around the X- and Z-axes; and random scaling. These augmentations are stronger overall than those used on MSR-Action3D. The classification-head dropout is set to 0.25, and no dropout is used in the pointwise encoders. The optimizer and learning-rate schedule are exactly the same as those in Section 4.2: SGD with a momentum of 0.9 and a weight decay of ${10}^{-4}$; the learning rate is linearly warmed up to 0.01 during the first ten epochs and reduced to one tenth of its previous value at epochs 20 and 30. Training lasts for 40 epochs with a batch size of 24.

As shown in Table 5, GLD-Net achieves 83.27% accuracy under exactly the same input conditions as the other methods. Removing only the residual ground points from the input increases the accuracy to 86.63%. This experiment quantifies a structural property of the proposed method. Because GLD-Net does not construct point neighborhoods, its frame-level descriptor depends on whole-frame pointwise max pooling, while the fixed-height segmentation provides only a coarse spatial partition. It therefore cannot effectively separate background points from human-body points, causing background noise to compete directly with evidence of human motion. In contrast, dense-neighborhood methods naturally exclude distant background points from local regions through anchor grouping and radius search. The cost in accuracy comes with lower computation: the per-clip computation of GLD-Net is 2.23 G, the lowest in the table, at approximately $1/2.4$ of that of PvNeXt, $1/6$ of that of PSTNet, and $1/15$ of that of P4Transformer and PST-Transformer.

4.6. Discussion

The ablation results show that global features alone achieve 92.68% accuracy. Introducing local features increases the accuracy by 3.14 percentage points, demonstrating the importance of local body-part evidence for action discrimination. The nearest-neighbor motion representation and feature differencing explicitly encode motion information in the input and feature spaces, respectively. Removing the nearest-neighbor motion representation alone reduces the accuracy by 7.67 percentage points, whereas removing feature differencing alone reduces it by 3.83 percentage points. Compared with the lightweight model PvNeXt at the same level, GLD-Net reduces the parameter count by 19.6% while improving the accuracy by 1.05 percentage points, thereby reducing model complexity while improving recognition performance. In addition, GLD-Net provides efficient training and inference: complete training takes approximately 70 minutes on a single RTX 3090, and the per-clip inference latency for a 24-frame input is less than 10 ms.

5. Conclusions

This paper proposed GLD-Net, a lightweight model for action recognition from point-cloud sequences. The model uses the distances from each point to its nearest neighbors in the preceding and subsequent frames as a parameter-free motion representation that requires no point correspondences. A global branch and three local branches defined by vertical partitions extract frame-level features. Bidirectional differencing then represents action changes explicitly, and a temporal Transformer performs the final sequence modeling. The complete process requires neither farthest point sampling nor neighborhood queries. On MSR-Action3D, GLD-Net achieves 95.82% accuracy with 0.579 M parameters and 1.42 G operations. Compared with the lightweight PvNeXt, the proposed method achieves 1.05 percentage points higher accuracy with 19.6% fewer parameters. Constant-parameter ablation experiments show that removing the local features, nearest-neighbor motion representation, and feature differencing decreases accuracy by 3.14, 7.67, and 3.83 percentage points, respectively.

These results show that explicit nearest-neighbor motion representation and global–local feature fusion can replace dense spatiotemporal neighborhood computations with little parameter overhead. The experiments on NTU provide preliminary evidence that this computational advantage can be retained on a larger dataset.