EfficientMixer: A Lightweight Vision Architecture via Complementary Design Principles

1. Introduction

Lightweight vision networks have become increasingly important for deploying deep learning models on resource-constrained devices [1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37]. Unlike large-scale models, they operate under strict constraints on parameter count, memory footprint, and computational cost, making naive scaling strategies ineffective [10,12,14,38,39]. A central challenge is therefore to maximize representational capacity under a fixed resource budget.

To address this challenge, a wide range of architectural strategies have been proposed, including depthwise separable convolutions [2], inverted bottlenecks [3], channel shuffling [5,6], cheap feature generation [7], neural architecture search [12,38], and hybrid convolutional–transformer designs [20,21,22,23,40]. Despite their success, these methods are often developed independently and evaluated under differing assumptions, which makes it difficult to isolate the contribution of individual design choices.

Attention mechanisms constitute another key direction for improving feature representation in vision networks [41,42,43,44,45,46,47]. Lightweight variants such as channel attention [41,44], spatial and channel–spatial attention [42,43,48], coordinate attention [45], and global context modules [49,50,51] have been widely explored. In parallel, Transformer-based models have demonstrated strong performance in vision tasks [26,52,53], but their computational cost often limits applicability in mobile settings [33,37]. This raises a practical question: can lightweight attention mechanisms provide similar contextual benefits when integrated into efficient convolutional backbones? In particular, we investigate whether lightweight channel–spatial attention can recover some of the contextual modeling benefits typically associated with Transformer architectures while preserving the computational efficiency of convolutional networks [41,42,44,45,54].

Motivated by these observations, we propose EfficientMixer, a lightweight architecture that integrates three complementary design principles: (i) hierarchical feature scaling inspired by EfficientNet [10], (ii) isotropic spatial–channel mixing from ConvMixer [55], and (iii) lightweight attention refinement motivated by MobileViT [20]. Instead of relying on Transformer blocks, EfficientMixer employs a lightweight nonlinear variant of CBAM [42] that increases feature selectivity while introducing minimal computational overhead.

Beyond its architectural design, EfficientMixer serves as a controlled framework for analyzing how different design principles interact under fixed parameter budgets. Specifically, hierarchical scaling increases semantic abstraction across stages, spatial–channel mixing strengthens local token interaction within stages, and attention selectively enhances informative features. Because these mechanisms operate at different levels of representation learning, they provide complementary inductive biases that can be systematically combined within a unified lightweight architecture.

Experimental evaluation is conducted using a multi-dataset protocol consistent with recent lightweight model evaluation frameworks [56], enabling controlled comparison under matched training settings and similar parameter budgets. Results show that EfficientMixer consistently outperforms EfficientNet, ConvMixer, and MobileViT across multiple datasets. Ablation studies further demonstrate that hierarchical scaling, spatial–channel mixing, and lightweight attention each contribute complementary improvements, highlighting compositional architectural design as an effective strategy for enhancing representation learning under constrained computational budgets.

Our major contributions can be summarized as follows:

• Complementary Architectural Design Principles. We demonstrate that hierarchical feature scaling, isotropic spatial–channel mixing, and lightweight attention provide complementary representational capabilities whose interaction improves lightweight vision models under fixed computational budgets [15,19].
• EfficientMixer Architecture. We propose EfficientMixer, a composed lightweight convolutional architecture that integrates these complementary principles through EfficientNet-style hierarchical scaling, ConvMixer-style spatial–channel mixing, and lightweight CBAM-based attention [41,42].
• Comprehensive Multi-Dataset Evaluation. Controlled experiments and ablation studies show that EfficientMixer consistently outperforms strong lightweight baselines under matched parameter budgets, and that these improvements arise from the interaction of complementary architectural components rather than increased model capacity [10,20,57].

The implementation and training code, including data processing, training pipelines, and evaluation scripts, are publicly available in the project repository [58] (commit 58a1eae).

2. Materials and Methods

2.1. Architectural Principles of Lightweight Vision Networks

Modern lightweight vision architectures improve representational efficiency under strict computational constraints through combinations of efficient convolutional processing, scaling strategies, and lightweight attention mechanisms [10,33,35]. Although these architectures differ substantially in implementation, many successful designs share common structural principles. To motivate the design of EfficientMixer, we examine representative principles underlying ConvMixer, EfficientNet, and MobileViT.

2.1.1. ConvMixer Principles

ConvMixer [55] exhibits three main design patterns: (A) Patch-based tokenization, where input images are converted into patch-level representations using strided convolutions, mimicking Transformer-style tokenization while remaining fully convolutional (Eq. (2) [53]; (B) Isotropic processing, where a constant feature dimensionality is maintained across layers and representations are refined through repeated application of identical blocks (Eq. (1)) [59]; and (C) Decoupled spatial–channel mixing, where spatial and channel interactions are separated via depthwise and pointwise convolutions, respectively (Eq. (3)), enabling efficient feature transformation [2,3].

$$z^{\left(0\right)}=\varphi \left(x\right),z^{(l+1)}=z^{\left(l\right)}+g_{\theta }\left(z^{\left(l\right)}\right),$$

(1)

$$\varphi \left(x\right)={\mathrm{Conv}}_{k,s}\left(x\right),$$

(2)

$$g_{\theta }\left(z\right)=W_p*(W_d*z),$$

(3)

where $W_d$ and $W_p$ denote depthwise and pointwise convolutions, respectively.

$$g_{\theta }=g_{\mathrm{channel}}\circ g_{\mathrm{spatial}},$$

(4)

where $g_{\mathrm{spatial}}\left(z\right)=W_d*z$ and $g_{\mathrm{channel}}\left(z\right)=W_p*z$.

$$z^{\left(l\right)}\in {\mathbb{R}}^{H^{\prime }\times W^{\prime }\times C},\forall l.$$

(5)

2.1.2. EfficientNet Principles

From EfficientNet [10], we identify three design patterns: (A) stem embedding, where a convolutional stem maps RGB inputs into feature space, as defined in Eq. (6); (B) MBConv-based feature extraction, where inverted bottleneck blocks combine channel expansion, depthwise convolution, and squeeze-and-excitation operations [3,41] (Eq. (7)); and (C) hierarchical scaling, where spatial resolution is progressively reduced while channel capacity increases via compound scaling rules [10] (Eq. (8)).

$$x^{(i+1)}=h_i\left(x^{\left(i\right)}\right),$$

(6)

$$h_i(·)=\mathrm{MBConv}(·),\mathrm{MBConv}=\mathrm{PWConv}\circ \mathrm{DWConv}\circ \mathrm{SE},$$

(7)

$$H_{i+1}={\displaystyle \frac{H_i}{s_i}},W_{i+1}={\displaystyle \frac{W_i}{s_i}},C_{i+1}>C_i.$$

(8)

2.1.3. MobileViT Principles

MobileViT [20] introduces a global aggregation operator $\mathcal{A}(·)$ that unfolds feature maps into tokens, applies self-attention, and projects them back to spatial form. This enables long-range dependencies through (A) convolution–attention coupling via QKV projections (Eq. (9)) and (B) global contextual modeling via scaled dot-product attention, originally introduced in Vision Transformers [53] (Eq. (10)).

$$Q,K,V=Unfold\left(x\right)W_{Q,K,V},$$

(9)

$$\mathcal{A}\left(x\right)=Fold\left(Softmax\left({\displaystyle \frac{Q{K}^{\top }}{\sqrt{d}}}\right)V\right).$$

(10)

Collectively, these design patterns define the construction space of EfficientMixer. EfficientNet contributes hierarchical feature extraction and compound scaling [10], ConvMixer contributes isotropic patch-level mixing with convolutional tokenization [55], and MobileViT contributes global contextual modeling through lightweight attention mechanisms [20].

2.2. EfficientMixer Architecture

Rather than optimizing a single architectural family or introducing isolated building blocks, EfficientMixer investigates whether complementary design principles extracted from independently developed lightweight architectures can be systematically composed within a unified framework. The key contribution is not a standalone module, but the empirical validation that principled architectural composition yields consistent performance gains under controlled computational budgets.

EfficientMixer integrates complementary architectural principles from EfficientNet [10], ConvMixer [55], and MobileViT [20] within a unified lightweight architecture. It is constructed by selectively composing these patterns into a hierarchical design, enabling controlled analysis of their individual and combined effects on lightweight vision performance. Specifically, it combines hierarchical feature scaling, intra-stage isotropic mixing, and lightweight attention refinement to improve representation learning under fixed computational budgets. The architecture can be interpreted as the composition of four design patterns (Figure 1):

(A) Hierarchical stem embedding. A convolutional stem maps the input RGB image into a feature representation while preserving local spatial structure prior to downsampling, following EfficientNet-style design principles.

(B) Hierarchical spatial–channel scaling. The network adopts EfficientNet’s stage-wise scaling, progressively reducing spatial resolution while increasing channel capacity (e.g., $H\times W\to {\displaystyle \frac{H}{2}}\times {\displaystyle \frac{H}{2}}\to {\displaystyle \frac{H}{4}}\times {\displaystyle \frac{H}{4}}$, $C:64\to 128\to 256\to 512$), enabling a transition from local spatial detail to high-level semantic features.

(C) Intra-stage isotropic mixing. Following the ConvMixer design, features are refined at a fixed spatial resolution within each stage using repeated depthwise–pointwise convolutional blocks with residual connections. Depthwise convolutions capture local spatial dependencies, pointwise convolutions enable channel mixing, and residual connections improve optimization stability.

(D) Learnable Residual Attention Fusion. Motivated by the stage-level contextual refinement employed in MobileViT [20], we replace Transformer-based refinement with a lightweight trainable Nonlinear Attention Fusion (NAF) module. The proposed module combines CBAM-based channel–spatial attention with nonlinear residual fusion to improve feature selectivity with minimal computational overhead. Given an input feature map $x\in {\mathbb{R}}^{C\times H\times W}$, CBAM produces an attention-refined representation:

$$x_{cbam}=\mathrm{CBAM}\left(x\right),$$

(11)

The original and attention-refined features are then combined through the NAF-attention module:

$$y=\mathrm{NAF}(x,x_{cbam})=\alpha x+\beta x_{cbam}+\gamma (x\odot x_{cbam}),$$

(12)

where ⊙ denotes element-wise multiplication. The fusion coefficients are derived from a learnable parameter vector $w\in {\mathbb{R}}^3$:

$$(\alpha ,\beta ,\gamma )={\displaystyle \frac{\mathrm{ReLU}\left(w\right)}{{\sum }_{i=1}^3\mathrm{ReLU}\left(w_i\right)+ϵ}},w=(w_1,w_2,w_3),\alpha +\beta +\gamma =1,$$

(13)

In summary, EfficientMixer composes hierarchical scaling, intra-stage isotropic mixing, and lightweight attention into a unified lightweight architecture. Hierarchical scaling captures increasingly abstract semantic representations across stages, isotropic mixing strengthens local spatial and channel interactions within each stage, and attention selectively emphasizes informative features. Together, these complementary design patterns improve representation learning while maintaining computational efficiency. The resulting architectural patterns are summarized in Table 1.

2.3. Model and Dataset Selection

To evaluate the proposed architecture under realistic mobile constraints, we instantiate a baseline EfficientMixer model containing approximately 2.5 million parameters (Table 2). Matching parameter budgets across models ensures that performance differences primarily reflect architectural design rather than model scale or hyperparameter tuning. This setting reflects practical edge-deployment scenarios, where memory, computation, and latency constraints favor compact models [2,3,37]. Maintaining a fixed parameter budget also enables a fair comparison of architectural efficiency, ensuring that observed performance differences arise from design choices rather than increased model capacity [6,10,33].

We compare EfficientMixer against representative lightweight architectures with similar parameter counts. EfficientNet and ConvMixer are included because EfficientMixer is derived from architectural principles extracted from these models and because they achieved the strongest cross-domain performance among lightweight convolutional architectures in our previous evaluations [56,60]. MobileNet serves as a representative baseline based on depthwise separable convolutions, while MobileViT represents hybrid convolution–attention architectures employing lightweight Transformer blocks. In contrast, EfficientMixer combines hierarchical scaling, isotropic feature mixing, and lightweight CBAM-based attention within a fully convolutional framework. Parameter counts are summarized in Table 3.

We evaluate all models on four image classification benchmarks with complementary characteristics: CIFAR-10 [61], Stanford Dogs [62], HAM10000 [63], and MIT Indoor-67 [64]. These datasets span general object recognition, fine-grained classification, medical imaging, and scene recognition, providing a compact benchmark for assessing cross-domain generalization under resource-constrained settings [2,33,34,35,36,37]. Dataset statistics are summarized in Table 4.

CIFAR-10 [61] consists of low-resolution images from 10 object categories, challenging models to learn discriminative features under limited spatial detail and strong visual overlap between classes. For consistency, we preprocess this dataset to 224 by 224px.

Stanford Dogs [62] is a fine-grained recognition benchmark with 120 visually similar dog breeds, testing a model’s ability to capture subtle inter-class differences from limited training data.

HAM10000 [63] contains dermatoscopic images from seven diagnostic categories and exhibits severe class imbalance, challenging models to represent rare classes without overfitting dominant ones.

MIT Indoor-67 [64] focuses on indoor scene classification, emphasizing global spatial layout and contextual reasoning amid high intra-class variability.

All architectures are trained using identical hyperparameters, including the learning rate schedule, batch size, optimizer, loss function, and data augmentation, to eliminate model-specific tuning effects (Table 5). We report both baseline and Self-Competitive Distillation (SCD) results in terms of average accuracy. Under matched parameter budgets and training conditions, differences in average performance reflect how effectively each architecture converts representational capacity into cross-domain performance, while improvements under SCD indicate more complete utilization of that capacity.

3. Experiments and Results

3.1. Baseline Evaluation of EfficientMixer

We hypothesize that combining these complementary design patterns will provide stronger representations than any individual architectural motif alone, resulting in improved performance across diverse visual domains without increasing model complexity. To evaluate whether complementary architectural design patterns can be effectively combined within a lightweight model, we compare EfficientMixer against EfficientNet, ConvMixer, MobileNet, and MobileViT under matched parameter budgets of approximately 2.5 million parameters.

All models are trained for 100 epochs with and without SCD, using identical hyperparameters and evaluated on four datasets spanning object recognition, fine-grained classification, medical imaging, and scene recognition. This controlled evaluation isolates architectural design from model scale and training effects, allowing us to assess whether integrating EfficientNet-inspired hierarchical scaling, ConvMixer-style spatial–channel mixing, and lightweight attention mechanisms yields improved predictive performance and cross-domain generalization under fixed computational constraints. Table 6 shows the test accuracy of each model at the end of 100 training epochs from scratch, with and without SCD.

As shown in Figure 2, EfficientMixer achieves consistently strong performance compared to other lightweight architectures under both training settings, with and without Self-Competitive Distillation (SCD). Notably, SCD improves all evaluated models, with a larger relative gain observed for EfficientMixer. This indicates that EfficientMixer benefits more effectively from the distillation process, suggesting a stronger underlying representation capacity and better utilization of the additional supervisory signal provided by SCD. Interestingly, ConvMixer also exhibits strong gains under SCD, and given that EfficientMixer inherits ConvMixer-style isotropic intra-stage feature mixing, this observation suggests that such isotropic feature filtering may be a beneficial design choice for enhancing distillation effectiveness in lightweight architectures.

3.2. Ablation Study

To quantify the contribution of individual architectural components, we conduct ablation studies using EfficientMixer variants with approximately matched parameter counts. Specifically, we analyze the effects of the stem design, hierarchical staging, attention, and dropout. By modifying one component at a time while maintaining a comparable computational budget, we isolate the impact of each design choice and evaluate whether performance gains arise from individual components or their interactions. We compare a baseline EfficientMixer model with eight architectural variants, as detailed in Table 7.

All models are trained for 100 epochs under identical settings and evaluated on four datasets: CIFAR-10, HAM10000, Stanford Dogs, and Indoor-67. Each configuration is trained with three random seeds to account for optimization variability.

We report mean test accuracy and standard deviation over three runs per dataset. A single cross-dataset score is obtained by averaging per-dataset means and standard deviations, yielding an overall mean ± standard deviation. This provides a compact summary while preserving per-dataset results. Best-performing values are highlighted in bold in Table 8.

Figure 3 shows the per-component deltas from the ablation study, reported as deviations in mean test accuracy relative to the baseline EfficientMixer configuration (75.78%). This enables a direct comparison of how each architectural modification influences performance. The ablation variants are grouped into four categories—stem design, stage depth, attention mechanisms, and dropout configurations—facilitating a structured analysis of each architectural component.

Among the eight variants, only Stem 2 and No Attention consistently underperform the EfficientNet baseline, indicating that EfficientMixer’s hierarchical staging and isotropic feature mixing form a strong architectural backbone. Overall, performance is most sensitive to changes in the stem and attention modules, highlighting the importance of early feature embedding and global context modeling. In contrast, variations in stage depth and dropout induce more moderate fluctuations around the baseline, suggesting that EfficientMixer is relatively robust to these design and regularization choices. This structured comparison clarifies the relative influence of each component on final model performance.

The convolutional stem was found to be one of the most influential architectural components. The baseline two-layer stem (3–32–64) achieved the highest overall xScore of 75.78, outperforming both the wider single-stage stem (3–64) and the narrower stem (3–32). While the 3–32 variant achieved the highest CIFAR-10 accuracy (95.47%), its performance dropped substantially on Stanford Dogs and MIT Indoor-67, resulting in the lowest overall xScore. These results suggest that progressively increasing channel capacity within the stem improves the quality of low-level feature representations and enhances transferability across diverse visual domains. Overall, the hierarchical stem contributes more to cross-dataset generalization than either a shallow or aggressively widened alternative.

Alternative stage width and depth configurations produce only modest changes in performance. Stage 1 and Stage 2 reduce accuracy by 0.33 and 0.55 points, respectively, relative to the baseline, indicating that EfficientMixer is relatively insensitive to moderate variations in channel allocation and block depth. Although Stage 2 achieved the highest CIFAR-10 accuracy (95.50%), its performance on the other datasets was consistently lower than the baseline. Similarly, Stage 1 yielded a minor improvement on MIT Indoor-67 but did not improve overall accuracy. These results suggest that the effectiveness of EfficientMixer is not primarily driven by specific width or depth scaling choices, but rather by the overall architectural design and feature processing strategy.

The attention module provides a consistent contribution to overall performance. Replacing the baseline NAF attention with a linear attention module reduces xScore from 75.78 to 75.38, while removing attention entirely leads to a larger decline to 74.41. The largest performance degradation is observed on Stanford Dogs, where accuracy decreases from 65.30% to 62.05% when attention is removed. This suggests that attention is particularly beneficial for fine-grained recognition tasks that require enhanced contextual modeling. Furthermore, the baseline NAF attention consistently outperforms linear attention across the benchmark suite, indicating that lightweight contextual refinement remains an important component of EfficientMixer despite its small computational overhead.

The impact of dropout regularization is comparatively small. Introducing dropout rates of 0.25 and 0.30 yields accuracy of 75.48% and 75.66%, respectively, both close to the baseline of 75.78%. Although the 0.30 configuration slightly improves performance on Stanford Dogs, differences across datasets are generally within one standard deviation of the baseline. These findings suggest that EfficientMixer already benefits from strong implicit regularization through its architecture and training strategy, including data augmentation and label smoothing. Consequently, additional dropout provides only marginal gains and does not substantially affect cross-domain robustness.

Overall, the ablation study indicates that the hierarchical convolutional stem and lightweight attention modules are the primary contributors to EfficientMixer’s cross-domain performance, beyond the core convolutional architecture. In contrast, variations in stage width/depth and dropout have only marginal effects, suggesting that model performance is driven more by feature extraction and contextual refinement than by capacity scaling alone.

4. Conclusion

This work investigates a central question in lightweight vision design: which architectural components most effectively improve representation learning under strict parameter constraints. Motivated by the complementary strengths of hierarchical scaling, isotropic spatial–channel mixing, and lightweight attention mechanisms, we proposed EfficientMixer as a unified architecture for studying and combining these design principles.

Extensive experiments across multiple visual domains demonstrate that EfficientMixer consistently outperforms representative lightweight architectures, including EfficientNet, ConvMixer, and MobileViT, under matched computational budgets. In addition, EfficientMixer benefits more effectively from Self-Competitive Distillation (SCD), suggesting that its design facilitates stronger representation learning under enhanced supervision.

Ablation studies provide further insight into the relative importance of architectural components. The hierarchical convolutional stem and lightweight attention modules contribute additional performance gains for cross-domain generalization, while variations in stage configuration and dropout have comparatively limited impact. These results indicate that improved feature extraction and contextual refinement are more critical than capacity scaling alone in determining performance under constrained settings.

Overall, our findings support the hypothesis that complementary architectural motifs can be effectively combined to improve representational capacity without increasing model complexity. This suggests that future lightweight model design should prioritize the principled integration of feature hierarchy, spatial–channel interaction, and efficient attention mechanisms, rather than relying primarily on scaling depth, width, or isolated architectural innovations. More broadly, these results indicate that lightweight architecture design may benefit from systematically composing complementary design principles, rather than relying on increasingly specialized building blocks or large-scale architecture search.