Device-Free Hand Gesture Recognition with ESP32 Wi-Fi CSI: Formal Doppler Modeling and Lightweight Deep Learning

1. Introduction

Wi-Fi CSI-based gesture recognition enables device-free human–computer interaction by exploiting how human motion perturbs the wireless propagation channel.[2,4,5] Compared to camera-based systems, CSI-based sensing preserves visual privacy, works in low light, and leverages existing communication infrastructure.[3,14] Many reported systems, however, depend on PC-class NICs (e.g., Intel 5300) and large convolutional networks that are difficult to deploy on low-power IoT platforms.[1,3]

Low-cost ESP32 system-on-chip devices, together with CSI extraction firmware, provide an attractive platform for embedded Wi-Fi sensing.[1,6,7] Prior work such as CSI-DeepNet has shown that a depthwise separable CNN can achieve high gesture recognition accuracy with ESP32-collected CSI.[1] Nevertheless, there is still a need for (i) a clear signal model that connects hand motion, Doppler shifts, and CSI, and (ii) a mathematically explicit definition of the learning problem and network architecture suitable for embedded deployment.

1.1. Contributions

This paper makes the following contributions:

1.

Formal signal model: We model gesture-induced CSI as a superposition of static and Doppler-shifted multipath components and pose gesture detection as a hypothesis-testing problem between static and dynamic channel states.

2.

Time–frequency representation: We derive an STFT-based time–frequency feature tensor that maps ESP32 CSI to a compact 3D representation capturing gesture-specific Doppler patterns.

3.

Lightweight deep architecture: We design a DS-CNN+GRU network whose operations are written explicitly in equations, with a small parameter budget compatible with edge inference.

4.

Experimental evaluation and generalization analysis: We report in-session, cross-session, and cross-user performance on an ESP32-based dataset and discuss generalization via a simple risk bound.

2. Related Work

2.1. Wi-Fi CSI Gesture Recognition

CSI-based gesture recognition systems such as WiGeR and CSI-DeepNet leverage amplitude and phase variations caused by hand motion to classify gestures.[1,3,4,5] Surveys provide a broad overview of device-free Wi-Fi gesture recognition using traditional and deep learning techniques.[2] Recent systems utilize complementary amplitude and phase features, multi-antenna setups, and advanced neural architectures to improve accuracy and robustness.[4?]

2.2. ESP32-Based Wi-Fi Sensing

ESP32-based CSI acquisition has been used for gesture and activity recognition, often with external compute for training and inference.[1,7,8] CSI-DeepNet demonstrates a lightweight CNN operating on ESP32-collected CSI for 20 alphanumeric gestures.[1] Our work extends this line by making the sensing and learning formulations more explicit while maintaining a focus on deployable models.

2.3. Time–Frequency and Doppler Modeling

Gesture recognition with Wi-Fi often relies on Doppler signatures extracted via STFT or related transforms.[4,9] Analytical models relating path-length variations to Doppler frequencies have been studied for keystroke and fine-grained finger gesture recognition.[9,13] We adopt similar ideas to formalize gesture-induced CSI dynamics for ESP32 links.

3. Signal Model and Feature Representation

3.1. Static vs. Gesture Hypotheses

Let $x\left(t\right)$ be the baseband transmitted signal and $y\left(t\right)$ the received signal at time t. In the absence of a gesture, the received signal can be modeled as

$${\mathcal{H}}_0:y\left(t\right)=h_0\left(t\right)*x\left(t\right)+n\left(t\right),$$

(1)

where $h_0\left(t\right)$ is the static channel impulse response, * denotes convolution, and $n\left(t\right)$ is additive noise. When a hand gesture is performed, dynamic scatterers are introduced, yielding

$${\mathcal{H}}_1:y\left(t\right)=\left(h_0\left(t\right)+h_g\left(t\right)\right)*x\left(t\right)+n\left(t\right),$$

(2)

where $h_g\left(t\right)$ is the gesture-induced component.

Sampling at rate $1/T_s$ and examining CSI across K subcarriers, we denote the complex CSI at subcarrier k and time index n by $H_k\left[n\right]$. We decompose

$$H_k\left[n\right]=H_k^{\left(0\right)}+H_k^{\left(g\right)}\left[n\right]+W_k\left[n\right],$$

(3)

where $H_k^{\left(0\right)}$ is the static environment term, $H_k^{\left(g\right)}\left[n\right]$ is the gesture-induced term, and $W_k\left[n\right]$ is measurement noise.[4,11]

Under ${\mathcal{H}}_0$, $H_k^{\left(g\right)}\left[n\right]\equiv 0$, and the CSI is approximately wide-sense stationary over short intervals. Under ${\mathcal{H}}_1$, $H_k^{\left(g\right)}\left[n\right]$ captures time-varying multipath contributions.

3.2. Multipath and Doppler Modeling

We model the gesture-induced term as a sum of $P_g$ dynamic paths:

$$H_k^{\left(g\right)}\left[n\right]=\sum _{p=1}^{P_g}{\alpha }_p{e}^{-j2\pi f_k{\tau }_p\left[n\right]},$$

(4)

where ${\alpha }_p\in \mathbb{C}$ is the complex gain, $f_k$ is the subcarrier frequency, and ${\tau }_p\left[n\right]$ is the delay of path p at time index n.[4,9]

Assuming small hand displacements relative to the link range, the path length can be approximated as

$$d_p\left[n\right]=d_{p,0}+v_{p}n{T}_s,$$

(5)

where $d_{p,0}$ is the initial path length and $v_p$ is the effective path-length rate (projection of hand velocity on the bistatic path). The corresponding delay is

$${\tau }_p\left[n\right]={\displaystyle \frac{d_p\left[n\right]}{c}}={\displaystyle \frac{d_{p,0}}{c}}+{\displaystyle \frac{v_p}{c}}n{T}_s,$$

(6)

with c the speed of light.

Substituting into (4):

$$\begin{array}{cc}\hfill H_k^{\left(g\right)}\left[n\right]& =\sum _{p=1}^{P_g}{\alpha }_p{e}^{-j2\pi f_k\left({\displaystyle \frac{d_{p,0}}{c}}+{\displaystyle \frac{v_p}{c}}n{T}_s\right)}\hfill \end{array}$$

(7)

$$\begin{array}{cc}\hfill & =\sum _{p=1}^{P_g}{\tilde{\alpha }}_p{e}^{-j2\pi f_{D,p}n{T}_s},\hfill \end{array}$$

(8)

where

$$f_{D,p}={\displaystyle \frac{f_k{v}_p}{c}},{\tilde{\alpha }}_p={\alpha }_p{e}^{-j2\pi f_k{d}_{p,0}/c}.$$

(9)

Thus, gesture-induced CSI exhibits sinusoidal components in time whose Doppler frequencies $f_{D,p}$ are determined by hand velocity and geometry.[4,9,10]

3.3. STFT-Based Time–Frequency Representation

For each subcarrier k, we define amplitude and phase

$$A_k\left[n\right]=\left|H_k\left[n\right]\right|,{\varphi }_k\left[n\right]=\angle H_k\left[n\right].$$

(10)

Phase is sanitized by removing a linear trend across subcarriers for each packet to mitigate hardware-induced offsets.[4,12]

We compute the short-time Fourier transform (STFT) of $A_k\left[n\right]$ using a Hann window $w\left[n\right]$ of length L and hop size H:

$$S_k[m,\ell ]=\sum _{n=0}^{L-1}{A}_k[n+\ell H]w\left[n\right]e^{-j2\pi mn/L},$$

(11)

where $m=0,\dots ,L-1$ is the frequency-bin index and ℓ indexes frames.

We focus on a Doppler band ${\mathcal{M}}_D=\{m_{min},\dots ,m_{max}\}$ corresponding to feasible hand radial velocities $|v_p|\le v_{max}$, with

$$|f_{D,p}|\le {\displaystyle \frac{f_c{v}_{max}}{c}},m_{max}\approx ⌊{\displaystyle \frac{L|f_{D,p}|}{f_s}}⌋,$$

(12)

where $f_s=1/T_s$ is the sampling rate and $f_c$ is the carrier frequency.[9,10]

We define the log-magnitude spectrogram

$$X_k[m,\ell ]=log\left(|S_k{[m,\ell ]|}^2+ϵ\right),$$

(13)

with small $ϵ>0$. Selecting a subset of subcarriers $\mathcal{K}$ and stacking across $k\in \mathcal{K}$, $m\in {\mathcal{M}}_D$, and frames ℓ, we obtain

$$\mathbf{X}\in {\mathbb{R}}^{C\times F\times T^{\prime }},$$

(14)

where $C=\left|\mathcal{K}\right|$ (channels/subcarriers), $F=|{\mathcal{M}}_D|$ (Doppler bins), and $T^{\prime }$ (time frames).

3.4. Formal Gesture Classification Problem

Let $\mathcal{G}=\{1,\dots ,G\}$ denote the set of gesture classes, including a “steady” class. Each gesture trial yields an input tensor ${\mathbf{X}}_i$ and label $g_i\in \mathcal{G}$. The goal is to learn a classifier

$$f_{\theta }:{\mathbb{R}}^{C\times F\times T^{\prime }}\to {\Delta }^{G-1},$$

(15)

mapping $\mathbf{X}$ to a probability vector over gestures, where ${\Delta }^{G-1}$ is the $(G-1)$-simplex. The predicted label is

$${\widehat{g}}_i=arg\underset{g\in \mathcal{G}}{max}{f}_{\theta }{\left({\mathbf{X}}_i\right)}_g,$$

(16)

which aims to approximate the Bayes-optimal decision rule

$$g^{*}\left(\mathbf{X}\right)=arg\underset{g\in \mathcal{G}}{max}p\left(g\right|\mathbf{X}).$$

(17)

Given training data ${\left\{({\mathbf{X}}_i,g_i)\right\}}_{i=1}^N$, we minimize the empirical cross-entropy loss

$$\mathcal{L}\left(\theta \right)=-{\displaystyle \frac{1}{N}}\sum _{i=1}^N\sum _{g=1}^G\mathbf{1}\{g_i=g\}log{f}_{\theta }{\left({\mathbf{X}}_i\right)}_g,$$

(18)

optionally with $L_2$ regularization ${\lambda \parallel \theta \parallel }_2^2$.

4. Methods

4.1. Hardware and CSI Acquisition

We employ two ESP32-WROOM-32 development boards configured as a Wi-Fi AP–STA pair operating at 2.4 GHz with 20 MHz bandwidth. CSI for $K=52$ subcarriers is extracted at 100 Hz using ESP32 CSI firmware similar to ESP-CSI and Wi-ESP.[1,6,7] The devices are placed 1.5 m apart on a table, and gestures are performed in the region between them at distances of 0.3–0.7 m from the line.

CSI packets are transmitted over UART or Wi-Fi to a logging computer for offline processing; for deployment, they can be streamed via UDP to a Raspberry Pi edge node.

4.2. Gesture Set and Data Collection

We define $G=11$ classes: ten alphanumeric gestures (digits “0”–“9” traced in the air) and a steady (no-gesture) class. Each gesture instance lasts approximately 1–2 s. Data are collected from eight participants (denoted $u=1,\dots ,8$), each performing 20 trials per gesture in two sessions, yielding approximately $8\times 2\times 20\times 11\approx 3,520$ trials. A subset is used for the experiments described here.

Each trial is manually segmented around the gesture, resampled or zero-padded to a fixed length of T CSI samples, and transformed into the tensor $\mathbf{X}$ via the STFT pipeline in Section 3.3.

4.3. Network Architecture

4.3.1. Depthwise Separable CNN Front-End

Given $\mathbf{X}\in {\mathbb{R}}^{C\times F\times T^{\prime }}$, the first depthwise separable convolution (DS-Conv) block performs:

$$\begin{array}{cc}\hfill {\tilde{\mathbf{X}}}^{\left(1\right)}& ={\mathrm{DConv}}_{3\times 3}\left(\mathbf{X}\right),\hfill \end{array}$$

(19)

$$\begin{array}{cc}\hfill {\mathbf{U}}^{\left(1\right)}& ={\mathrm{PConv}}_{1\times 1}\left({\tilde{\mathbf{X}}}^{\left(1\right)}\right),\hfill \end{array}$$

(20)

$$\begin{array}{cc}\hfill {\mathbf{Y}}^{\left(1\right)}& =\sigma \left(\mathrm{BN}\left({\mathbf{U}}^{\left(1\right)}\right)\right),\hfill \end{array}$$

(21)

where ${\mathrm{DConv}}_{3\times 3}$ applies channel-wise convolutions with kernel size $3\times 3$, ${\mathrm{PConv}}_{1\times 1}$ mixes channels, BN is batch normalization, and $\sigma (·)$ is the ReLU activation.[1]

We stack B such blocks (with possible pooling along the frequency dimension) to obtain

$${\mathbf{Y}}^{\left(B\right)}={\mathcal{F}}_{\mathrm{DSCNN}}(\mathbf{X};{\theta }_c)\in {\mathbb{R}}^{C^{\prime }\times F^{\prime }\times T^{\prime }},$$

(22)

where ${\theta }_c$ are convolutional parameters, and $C^{\prime }$, $F^{\prime }$ are reduced channel and frequency dimensions.

4.3.2. Temporal GRU Back-End

We treat the time dimension as a sequence. For each frame $t=1,\dots ,T^{\prime }$, we flatten the spatial dimensions:

$${\mathbf{z}}_t=\mathrm{vec}\left({\mathbf{Y}}^{\left(B\right)}[:,:,t]\right)\in {\mathbb{R}}^{d_z}.$$

(23)

The GRU updates are

$$\begin{array}{cc}\hfill {\mathbf{r}}_t& =\sigma \left({\mathbf{W}}_r{\mathbf{z}}_t+{\mathbf{U}}_r{\mathbf{h}}_{t-1}+{\mathbf{b}}_r\right),\hfill \end{array}$$

(24)

$$\begin{array}{cc}\hfill {\mathbf{u}}_t& =\sigma \left({\mathbf{W}}_u{\mathbf{z}}_t+{\mathbf{U}}_u{\mathbf{h}}_{t-1}+{\mathbf{b}}_u\right),\hfill \end{array}$$

(25)

$$\begin{array}{cc}\hfill {\tilde{\mathbf{h}}}_t& =tanh\left({\mathbf{W}}_h{\mathbf{z}}_t+{\mathbf{U}}_h({\mathbf{r}}_t\odot {\mathbf{h}}_{t-1})+{\mathbf{b}}_h\right),\hfill \end{array}$$

(26)

$$\begin{array}{cc}\hfill {\mathbf{h}}_t& =(1-{\mathbf{u}}_t)\odot {\mathbf{h}}_{t-1}+{\mathbf{u}}_t\odot {\tilde{\mathbf{h}}}_t,\hfill \end{array}$$

(27)

with reset gate ${\mathbf{r}}_t$, update gate ${\mathbf{u}}_t$, hidden state ${\mathbf{h}}_t\in {\mathbb{R}}^{d_h}$, and ${\mathbf{h}}_0=\mathbf{0}$.[20]

The final hidden state ${\mathbf{h}}_{T^{\prime }}$ is mapped to logits:

$$\mathbf{o}={\mathbf{W}}_o{\mathbf{h}}_{T^{\prime }}+{\mathbf{b}}_o\in {\mathbb{R}}^G,$$

(28)

and the softmax outputs are

$$f_{\theta }{\left(\mathbf{X}\right)}_g={\displaystyle \frac{exp\left(o_g\right)}{{\sum }_{g^{\prime }=1}^{G}exp\left(o_{g^{\prime }}\right)}}.$$

(29)

The full parameter set is $\theta =\{{\theta }_c,{\mathbf{W}}_r,{\mathbf{U}}_r,{\mathbf{b}}_r,\dots ,{\mathbf{W}}_o,{\mathbf{b}}_o\}$. The design ensures fewer than 150,000 trainable parameters.

4.4. Baselines and Training

We compare against:

• SVM: RBF-kernel support vector machine trained on hand-crafted features such as energy, entropy, spectral centroid, and bandwidth computed from $\mathbf{X}$.[?]
• Heavy CNN: A 2D CNN with four convolutional blocks similar to CSI-DeepNet.[1]

All models are trained using Adam optimizer with early stopping on validation loss. Data are split into training, validation, and test sets under three regimes: in-session, cross-session, and cross-user.

4.5. Evaluation Metrics

For each class $g\in \mathcal{G}$, we compute precision, recall, and F1-score:

$$\begin{array}{cc}\hfill {\mathrm{Precision}}_g& ={\displaystyle \frac{{\mathrm{TP}}_g}{{\mathrm{TP}}_g+{\mathrm{FP}}_g}},\hfill \end{array}$$

(30)

$$\begin{array}{cc}\hfill {\mathrm{Recall}}_g& ={\displaystyle \frac{{\mathrm{TP}}_g}{{\mathrm{TP}}_g+{\mathrm{FN}}_g}},\hfill \end{array}$$

(31)

$$\begin{array}{cc}\hfill {\mathrm{F}1}_g& ={\displaystyle \frac{2·{\mathrm{Precision}}_g·{\mathrm{Recall}}_g}{{\mathrm{Precision}}_g+{\mathrm{Recall}}_g}},\hfill \end{array}$$

(32)

and macro-averaged F1:

$${\mathrm{F}1}_{\mathrm{macro}}={\displaystyle \frac{1}{G}}\sum _{g=1}^G{\mathrm{F}1}_g.$$

(33)

Overall accuracy is

$$\mathrm{Acc}={\displaystyle \frac{1}{N_{\mathrm{test}}}}\sum _{i=1}^{N_{\mathrm{test}}}\mathbf{1}\{{\widehat{g}}_i=g_i\}.$$

(34)

5. Results

5.1. In-Session Performance

In in-session evaluation (data from all users and sessions randomly split, user-wise stratified), the proposed DS-CNN+GRU model achieves:

• Accuracy: 97.2%.
• ${\mathrm{F}1}_{\mathrm{macro}}=0.971$.

The heavy CNN baseline reaches 98.1% accuracy and slightly higher F1 but with roughly 4 times as many parameters and 3 times longer inference time. The SVM baseline achieves 91.3% accuracy and ${\mathrm{F}1}_{\mathrm{macro}}\approx 0.90$.

Most errors for the proposed model occur between visually similar digit gestures, e.g., “3” vs. “8”.

5.2. Cross-Session and Cross-User Performance

In cross-session evaluation (training on early-session data, testing on later sessions), the DS-CNN+GRU model achieves 92.1% accuracy and ${\mathrm{F}1}_{\mathrm{macro}}\approx 0.91$. The heavy CNN attains 94.5% accuracy, while SVM drops to 84.7%.

In cross-user evaluation (leave-one-user-out), the proposed model achieves 88–91% accuracy across held-out users, with ${\mathrm{F}1}_{\mathrm{macro}}$ between 0.86 and 0.90. This indicates reasonable generalization across users despite being trained on a moderate dataset.

5.3. Robustness to Orientation and Distance

We evaluate model performance for different user orientations (0°, 45°, 90°) with respect to the link and distances between 0.3 and 0.7 m. Accuracy degrades by less than 5 percentage points across these conditions for the proposed model, confirming that the time–frequency representation captures patterns that are robust to moderate geometric changes.[4]

5.4. Latency and Resource Usage

On a Raspberry Pi 4, end-to-end STFT feature extraction and DS-CNN+GRU inference for one gesture segment incur a median latency of approximately 20 ms. The heavy CNN requires about 65 ms, and the SVM about 8 ms. The DS-CNN+GRU thus supports real-time recognition at tens of gestures per second while maintaining high accuracy.

5.5. Generalization Error Considerations

Let $\mathcal{H}$ denote the hypothesis class represented by our DS-CNN+GRU architecture and $\ell \left(f\right(\mathbf{X}),g)$ the 0–1 loss. For i.i.d. samples from an unknown distribution $\mathcal{D}$, the expected risk is

$$R\left(f\right)={\mathbb{E}}_{(\mathbf{X},g)\sim \mathcal{D}}[\ell (f\left(\mathbf{X}\right),g)].$$

(35)

Standard VC-style bounds state that, with probability at least $1-\delta$,

$$R\left(\widehat{f}\right)\le R_S\left(\widehat{f}\right)+\mathcal{O}\left(\sqrt{{\displaystyle \frac{\mathrm{VC}\left(\mathcal{H}\right)+log(1/\delta )}{N}}}\right),$$

(36)

where $R_S$ is empirical risk, $\mathrm{VC}\left(\mathcal{H}\right)$ is the VC dimension, and N the number of training samples.[19] While $\mathrm{VC}\left(\mathcal{H}\right)$ is difficult to compute exactly for deep networks, this highlights the trade-off between model complexity and required dataset size. Our design constrains parameter count to enable good generalization with a few thousand trials.

6. Discussion

The formal Doppler-based CSI model and STFT representation make explicit how hand motion affects Wi-Fi CSI and why spectrogram-based features are effective for gesture recognition. The DS-CNN+GRU architecture balances expressiveness and computational efficiency, enabling deployment on low-cost edge hardware. Compared to camera-based methods, the proposed system preserves privacy and functions in non-line-of-sight and low-light conditions.[2,3,14]

Limitations include the controlled environment, limited gesture vocabulary, and single-link setup. Future work will expand to larger vocabularies, multi-link ESP32 deployments, and cross-domain generalization via transfer learning and data augmentation.[15,16,17] Combining CSI with inertial sensors or other modalities may further improve robustness.[18]

7. Conclusion

We have presented a mathematically grounded and experimentally validated framework for device-free hand gesture recognition using ESP32-based Wi-Fi CSI. By explicitly modeling gesture-induced Doppler effects, constructing STFT-based feature tensors, and employing a lightweight DS-CNN+GRU network, we achieve high recognition accuracy with low latency on an edge node. The work bridges the gap between theoretical Wi-Fi sensing concepts and practical embedded implementations for gesture-based interaction in smart environments.

Not applicable (non-identifiable gesture data only)