SA-MAP: Similarity-Attention Boosted Token Compression via Merging-and-Pruning for Efficient Speech LLMs

1. Introduction

Benefiting from the advances in Large Language Models (LLMs) [1,2,3], Speech Large Language Models (Speech LLMs) [4,5,6,7] have achieved remarkable progress, demonstrating exceptional capabilities in both Automatic Speech Recognition (ASR) and complex audio understanding. A typical Speech LLM comprises three core components: (1) an audio encoder that converts raw waveforms into discrete or continuous representations; (2) a backbone LLM that generates responses autoregressively (which constitutes the primary computational bottleneck); and (3) an optional speech synthesizer for waveform reconstruction.

Most existing SpeechLLMs remain predominantly constrained to short-duration audio (e.g., 30s) due to prohibitive computational costs and latency. Given that the backbone LLM constitutes the primary inference bottleneck, the high density of audio tokens in the input sequence accounts for the vast majority of the inference overhead. While high-frame-rate acoustic embeddings afford fine-grained resolution, they introduce substantial redundancy, particularly during silent intervals or stationary noise. Furthermore, the quadratic complexity of self-attention mechanisms imposes a significant bottleneck on long-form audio scalability, escalating both inference latency and memory overhead. Consequently, efficient compression of audio tokens is imperative to enhance end-to-end throughput and facilitate long-context speech processing.

Current compression strategies generally fall into two categories: merging and pruning. In speech acceleration, merging-based methods (e.g., A-ToMe [8], FastAdaSP [9]) predominate due to the sequential nature of speech, progressive merging adjacent audio tokens during the prefill phase, whereas pruning method [10] selects the top-k salient audio tokens based on attention weights. However, these compression modules are primarily integrated into the LLM transformer blocks, which often introduces a compatibility bottleneck: direct access to internal attention scores is frequently precluded by highly optimized kernels like FlashAttention [11], which prioritize throughput via fused operations.

Based on the core features leveraged, compression methods can be divided into similarity-based and attention-based paradigms. The former leverages inter-token correlations to retain a maximally diverse token set but risks omitting critical fine-grained details, whereas the latter exploits attention-sparsity to extract salient tokens but often retains redundant duplicates. Hybrid approaches like VisPruner[12] and VisionZip [13] attempt to combine these metrics through multi-stages. however, as similarity and attention are applied in isolation, they fail to achieve a synergistic convergence of these complementary features.

Notably, as shown in Figure 1, an inherent task divergence between Vision Language Models (VLMs) and Speech LLMs exacerbates the incompatibility of direct method migration. VLMs rely on sparse visual cues for reasoning and context comprehension, whereas Speech LLMs execute multiple tasks within a single model: they address sequence-intensive tasks (e.g., ASR) that require leveraging all audio tokens to ensure transcription accuracy, while simultaneously handling sparsity-oriented tasks (e.g., Emotion Recognition (ER), Spoken Question Answering (SQA)) where only a few tokens carry key predictive information. This duality necessitates a more prudent token reduction strategy that accounts for temporal correlations and varying semantic densities. Existing VLM works focus primarily on local spatial redundancy and instruction relevance of visual information, failing to capture these speech-specific temporal dynamics; thus, direct migration of such techniques to Speech LLMs cannot yield optimal performance and gains.

To address these limitations, we introduce SA-MAP, a plug-and-play, similarity-attention synergistically driven framework for joint token mergingandpruning. To maximize hardware efficiency and ensure compatibility with optimized inference kernels, we deploy the SA-MAP module before the LLM. Different from methods requiring deep integration into the Transformer layers, our approach only utilizes attention scores from a single audio encoder layer, significantly reducing architectural coupling and memory overhead. Tailored to the temporal dependencies and redundancy of speech tokens, we propose a two-part merging-pruning pipeline. In the first merging stage, we group adjacent tokens based on intra-group global feature similarity and perform attention-guided weighted merging to collapse redundant segments while preserving critical semantic information. In the next Pruning Stage, we implement a pruning kernel that integrates both attention and similarity information for diversity-driven pruning. By incorporating a similarity thresholding mechanism, SA-MAP adaptively calibrates the ratio between merging and pruning for each individual sample, achieving a dynamic equilibrium between compression aggressiveness and task-specific accuracy.

Extensive experiments validate that SA-MAP effectively reduces the computational overhead of various Speech LLMs, with evaluations performed on three representative models: Qwen2-Audio, Kimi-Audio, and GLM-ASR. For dense ASR tasks, at a 60% token retention ratio, SA-MAP only incurs a marginal absolute WER degradation of 1.33%–2.26%. Notably, compared to existing methods, SA-MAP facilitates a further 40.5%–68.7% reduction in the performance gap relative to the uncompressed models. Furthermore, for general audio understanding tasks, SA-MAP achieves a 50% lossless token reduction, effectively halving the sequence length with negligible impact on semantic accuracy.

In summary, the contributions of our work are as follows:

• We introduce an adaptive, hardware-friendly, plug-and-play framework driven by the synergy of similarity and attention for joint token Merging and Pruning.
• Similarity and Attention are synergistically integrated, merging employs intra-group global similarity with attention-guided weighted aggregation; pruning uses a similarity-attention integrated kernel for diversity-driven pruning.
• Extensive validation on multiple public speech datasets and mainstream Speech LLMs confirms the method’s effectiveness and generalization across ASR and audio understanding tasks.

2. Related Work

Speech Large Language Models. Breakthroughs in Large Language Models (LLMs) [1,2,3] have driven research to bridge the gap between Natural Language Processing (NLP) and speech signal understanding, spawning Speech LLMs[4,5,6,14]. Diverging from conventional single-task systems like ASR or TTS, Speech LLMs integrate acoustic perception and language reasoning capabilitie to support complex tasks such as spoken QA and cross-modal retrieval. A standard Speech LLM comprises an audio encoder, a modality adapter, and a pre-trained LLM backbone. To preserve fine-grained temporal dynamics, most models retain high resolution during tokenization, resulting in extreme token density [4,14]. However, this high temporal fidelity incurs heavy computational overhead. In long-form scenarios, token sequences expand sharply: 30s for ∼ 750 tokens and 10min for over 15,000 [4]. Due to the quadratic complexity of self-attention, this raises inference latency and memory overhead and hinders edge deployment. Thus, developing cost-efficient techniques to address these challenges is critical.

Audio Token Reduction. Token compression in Speech LLMs is dominated by merging-based approaches, stemming from the inherent temporal dependencies of speech signals. Most existing frameworks embed compression modules directly within Transformer layers of LLM. For instance, FastAST [15] incorporates token merging and cross-modal distillation into the Spectrogram Transformer to optimize audio classification efficiency. Other representative merging methods, such as A-ToMe [8] iteratively merges pairwise adjacent audio tokens across stacked layers to achieve adaptive compression, while FastAdaSP [9] leverages intermediate-layer attention scores to guide the merging weights of adjacent token clusters. Additionally, pruning-based SpeechPrune [16] performs audio token pruning within Transformer layers using a two-stage strategy based on token-text similarity and attention. However, these methods exhibit several critical shortcomings. First, a sole reliance on pairwise adjacent similarity ignores the correlations between arbitrary tokens within a group, potentially compromising representation accuracy; Second, by embedding modules within transformer layers and requiring layer-wise stacking for high compression ratios, these approaches introduce excessive hyperparameters and elevate deployment difficulty.Third, the dependency on intermediate-layer attention scores (as seen in FastAdaSP and SpeechPrune renders these methods incompatible with optimized attention kernels like FlashAttention, leading to suboptimal hardware efficiency.

Visual Token Reduction. Compared to speech, visual inputs in multimodal large language models (MLLMs) exhibit even more pronounced redundancy [17,18]. Evidence suggests that over 50% of tokens in typical MLLM sequences exert negligible influence during inference [19,20]. Consequently, compression techniques are more mature in VLMs, where modules are typically decoupled from the backbone and deployed between the Vision Transformer (ViT) and the LLM. These methods leverage cross-modal information fusion to perform token pruning or merging based on attention or similarity. Specifically, similarity-based methods [18,21,22] typically calculate the cosine similarity between features, merging or pruning tokens with high similarity thresholds to mitigate redundancy, though they risk blurring critical fine-grained details. Attention-based methods [23,24] utilize intermediate attention weights to gauge importance and select top-K salient tokens, which tends to retain duplicate tokens with high attention. Notably, hybrid strategies like VisionZip [13] and VisPruner [12] attempt to adopts a two-stage pruning/merging strategy to exploit importance and similarity phasedly. Yet, a fundamental limitation persists: by applying similarity and attention in isolation, these methods fail to resolve potential conflicts between diversity and saliency, precluding a truly synergistic optimization of token information. Yet, a common limitation persists: applying similarity and attention in isolation may lead to potential conflicts between diversity and importance, precluding a truly synergistic optimization of token information.

3. Method

This section details the design principles and core components of our SA-MAP. Section 3.1 covers attention mechanisms and token compression in Speech LLMs as preliminaries. Section 3.2 and 3.3 examine two core modules: the attention-enhanced token merging module and attention-similarity based token pruning module. Section 3.4 presents the adaptive merging-pruning fusion framework, followed by a trainable tuning scheme proposed in Section 3.5.

3.1. Preliminaries

Architecture of Speech LLMs. Existing Speech LLMs typically consist of three core components: audio encoder, modality projector, and a LLM. Audio encoder, such as Whisper large-v3 [25] (adopted in Qwen2-Audio and Kimi-Audio) and audio tokenizer (utilized in Kimi-Audio), transform raw audio waveforms into continuous or discrete tokens that are interpretable and processable by the LLM. The projection module aligns these audio tokens with the word embedding space of the large language model. The LLM then fuses the aligned audio and text information to generate corresponding responses.

Audio Token compression. Most audio token compression modules are embedded within the backbone of LLMs. High compression ratio is achieved via multi-layer stacking, a design that entails a high level of hardware customization. Notably, some of these modules demand direct access to the attention scores of intermediate layers, rendering them generally incompatible with highly optimized acceleration libraries (e.g., FlashAttention). To maximize hardware efficiency, we draw inspiration from relevant studies in VLMs and deploy the proposed SA-MAP compression module upstream of the LLM, only utilizing attention scores from a single audio encoder layer.

Attention in Audio encoder. Typical audio encoders, e.g., Whisper v3 is based on transformer architecture. Typically, we assess the importance of each audio token by analyzing the attention scores within the audio encoder. Specifically, we compute the attention score according to Eq. 1.

$${\mathit{A}}_h=Softmax\left({\displaystyle \frac{Q_h{K}_h^T}{\sqrt{D_h}}}\right),$$

(1)

within the context of multi-head attention, $A_h$ stands for the attention score of each individual head, $D_h$ signifies the head dimension, and $Q_h$ and $K_h$ refer to the query and key respectively.

3.2. Attention-Enhanced Token Merging

Merging similar audio tokens can eliminate redundant audio information while preserving critical content. To this end, we propose a novel token merging method: Intra-Group Global Similarity-based Attention-Weighted Token Merging (G-SAM), deployed upstream of the LLM, with two sequential implementation steps: First, audio token grouping, which leverages the similarity between audio features. We cluster adjacent, highly similar audio tokens into groups for subsequent merging. Furthermore, to maximize the information retention of tokens post-merging, we incorporate attention information from the audio encoder to introduce an importance metric, and perform weighted merging on intra-group tokens.

Audio Token Grouping. Given the audio features ${\mathit{H}}_a\in {\mathbb{R}}^{N\times D}$ generated by the projection module preceding the LLM, we compute the pairwise cosine similarity across tokens to derive the similarity matrix $\mathit{L}$:

$${\mathit{L}}_{i,j}={\displaystyle \frac{{\mathit{H}}_a^i·{\mathit{H}}_a^j}{\parallel {\mathit{H}}_a^i\parallel ·\parallel {\mathit{H}}_a^j\parallel }}.$$

(2)

Unlike the strategies adopted by A-ToMe and FastAdaSP, which only leverage the similarity between adjacent tokens for clustering, our method considers the similarity among any tokens within the group. This design can capture more comprehensive correlation patterns between tokens in the group, effectively avoiding the locality bias caused by relying solely on adjacent relationships, thereby enhancing the rationality of clustering results and the completeness of feature representation, and better preserving the key semantic information in audio data.

As illustrated in Figure 2, we fix the merging similarity threshold $\lambda$ and iterate through the audio token list $S=\{1,...,N\}$ to sequentially incorporate multiple adjacent indices into a merged cluster. Let $s_i=\{k,...,k+m-1\}$ denote the set of audio tokens in the original merged cluster, where the cardinality of $s_i$ is m. We then compute the average similarity between the next adjacent token (with index $k+m$) and the tokens in $s_i$ , which is formally defined as:

$$l_{k+m,s_i}={\displaystyle \frac{1}{m}}{\displaystyle \sum _{t=k}^{k+m-1}}{\mathit{L}}_{k+m,t},$$

(3)

if $l_{k+m,s_i}\ge \lambda$ we incorporate the token indexed by $k+m$ into $s_i$ ; otherwise, we initialize a new cluster $s_{i+1}=\{k+m\}$. The above procedure is repeated iteratively until all audio tokens are assigned to corresponding clusters. Ultimately, we yield K merged clusters, formally denoted as $S^{*}=\left\{s_i\right|i=1,\dots ,K\}$.

Weighted Token Merging. Specifically, we extract attention scores from the audio encoder’s -2 layer to calculate the importance of each audio token, as this layer retains the fine-grained semantic information of audio features. We observe that attention maps across heads exhibit significant discrepancies, with each head capturing distinct audio semantic dimensions. To maximize the discriminative importance information encoded by attention, we compute the maximum value across the head dimension, and then derive the important score $W_k\in {\mathbb{R}}^N$ of each audio token:

$${\mathit{W}}_k={\displaystyle \frac{1}{N}}{\displaystyle \sum _{j=0}^N}\underset{i}{max}{\mathit{A}}_{i,j,k},$$

(4)

where N denotes the length of the audio token sequence, H stands for the number of attention heads, and attention scores $\mathit{A}\in {\mathbb{R}}^{H\times N\times N}$.

Then, based on Eq. 5, we aggregate the audio features of each subset $s_i$ in the cluster set $S^{*}$ (derived from the aforementioned clustering process) into a single token feature , ultimately yielding K compressed tokens denoted as ${\tilde{\mathit{H}}}_a=\left\{{\tilde{\mathit{H}}}_a^i\right|i=1,...,K\}$. and the corresponding index list set is updated to $\tilde{S}=\{1,...,K\}$, in which:

$${\tilde{\mathit{H}}}_a^i={\displaystyle \frac{{\sum }^{|s_i|}{\mathit{W}}_j·{\mathit{H}}_a^j}{{\sum }^{|s_i|}{\mathit{W}}_j}},$$

(5)

where $j\in s_i$, and ${\tilde{\mathit{H}}}_a^i$ denotes the merged feature corresponding to the cluster subset $s_i$.

3.3. Attention-Similarity Based Token Pruning

Furthermore, we design an attention-enhanced similarity-driven pruning module denoted as ADPruner, integrating attention into the similarity matrix to optimize the DPP kernel and maximize the diversity of audio tokens as well as preserve their critical semantic information.

DPP [26] was initially introduced to model fermion repulsion in quantum physics and has since been widely applied in list-wise diversity modeling. Formally, it is a probability measure defined on the power set of a discrete set S, characterized by a positive semi-definite (PSD) kernel matrix $\mathit{L}\in {\mathbb{R}}^{N\times N}$ indexed by elements of S. The probability of sampling a subset $\widehat{S}\subseteq S$ is:

$$\mathcal{P}\left(\widehat{S}\right)={\displaystyle \frac{det\left({\mathit{L}}_{\widehat{S}}\right)}{det(\mathit{L}+\mathit{I})}}\propto det\left({\mathit{L}}_{\widehat{S}}\right),$$

(6)

in which, ${\mathit{L}}_{\widehat{S}}$ represents the principal submatrix of $\mathit{L}$ corresponding to $\widehat{S}$.

Via the DPP sampling procedure, the optimal subset ${\widehat{\mathit{H}}}_a$ is determined as:

$${\widehat{S}}^{*}=\underset{\begin{array}{c}\widehat{S}\subseteq S,\left|\widehat{S}\right|=m\end{array}}{argmax}det\left({\mathit{L}}_{\widehat{S}}\right),{\widehat{\mathit{H}}}_a=\left\{{\mathit{H}}_a^i\mid i\in {\widehat{S}}^{*}\right\}.$$

(7)

In the context of VLM token pruning, CDpruner [22] leverages DPP to model the diversity of the retained subset of visual tokens. Given a set of visual tokens $H_v$, the kernel matrix $\mathit{L}$ is defined based on the pairwise cosine similarity of visual features. Similarly, when applied to audio token pruning, the default kernel matrix $\mathit{L}$ is constructed using the pairwise cosine similarity of audio features, see Eq. 2. If the construction of the DPP kernel matrix relies solely on feature similarity (a single dimension), it tends to overlook fine-grained critical information when preserving diversity. To address this issue, we propose an attention-enhanced DPP kernel matrix, enabling the simultaneous consideration of both the feature similarity and importance of audio tokens during pruning. We similarly extract attention scores from the penultimate layer of the audio encoder to compute the importance matrix $\widehat{\mathit{A}}\in {\mathbb{R}}^{N\times N}$ for audio tokens, $\widehat{\mathit{A}}={\displaystyle \frac{1}{H}}{\sum }_{i=0}^H{\mathit{A}}_{i,j,k}$.

Finally, we integrate feature similarity and token importance for audio token pruning, yielding our proposed attention-enhanced DPP-based pruning method (ADPruner), as illustrated in Figure 2. Specifically, we weight the original kernel matrix by the derived importance scores to construct a novel conditional kernel matrix:

$$\widehat{\mathit{L}}=diag\left(\widehat{\mathit{A}}\right)·\mathit{L}·diag\left(\widehat{\mathit{A}}\right).$$

(8)

The updated log-probability of the subset S for DPP is:

$$logdet\left({\widehat{\mathit{L}}}_{\widehat{S}}\right)={\displaystyle \sum _{i\in \widehat{S}}}log\left({\widehat{\mathit{A}}}_i^2\right)+logdet\left({\mathit{L}}_{\widehat{S}}\right),$$

(9)

which jointly considers both feature similarity and importance of the retained audio tokens. We then obtain the optimal subset via MAP inference [27].

3.4. Adaptive Merging-Pruning Fusion Framework

Merging combines highly similar tokens, reducing redundancy while preserving key information. However, under high compression rates, using merging alone may force the fusion of dissimilar tokens, which can distort speech representations. In contrast, pruning keeps only the most important tokens and removes the others, which may lead to the loss of useful information. To address these issues, SA-MAP integrates improved merging and pruning modules, as shown in Figure 2. We introduce a threshold that adaptively adjusts the ratio of merging to pruning for each sample, thereby achieving high compression rates while robustly maintaining model performance. Changing the threshold leads to different accuracy, as illustrated in Figure 3. In practice, the threshold is selected through a manual search on a small dataset, which keeps the tuning cost low. As described in Appendix 1, the proposed compression process consists of two stages. In the first stage, adjacent tokens are selectively merged based on the threshold to preserve as much information as possible. In the second stage, pruning is applied to the merged tokens to improve representation diversity and reduce redundancy. By jointly optimizing merging and pruning, our method achieves an effective balance between compression efficiency and information preservation.

3.5. Optional: Trainable Compression Module

We introduce a trainable compression module to enhance token pruning within the merging-pruning joint framework, which integrates two lightweight, end-to-end optimized components: a learnable merging weight generator and a trainable importance scorer. Each consists of two linear projection layers, similar to the key and query transformations in standard self-attention. The former adaptively determines token fusion weights based on input-dependent attention patterns, while the latter estimates token importance through simplified self-attention interactions for refining the DPP kernel. The details are shown in Appendix C.

Tuning merging weights. As shown in Eq. 4 and Eq. 5, we perform weighted fusion of similar tokens in the merger module. In our joint framework, the attention scores from the penultimate layer of the audio encoder are used for this computation. Furthermore, the weighting coefficients in Eq. 5 can be derived from a trainable weighting module. Specifically, we design a lightweight module that generates these coefficients dynamically in a manner analogous to computing attention scores. This module consists of two linear projection layers, similar to the key and query transformations in standard self-attention. We first project the audio features using two separate learnable weight matrices, ${\mathrm{\Theta }}_q$ and ${\mathrm{\Theta }}_k$:

$$\mathbf{Q}={\mathit{H}}_a{\mathrm{\Theta }}_q,\mathbf{K}={\mathit{H}}_a{\mathrm{\Theta }}_k,$$

(10)

where ${\mathrm{\Theta }}_q,{\mathrm{\Theta }}_k\in {\mathbb{R}}^{D\times D}$. The weighting coefficient ${\mathit{W}}_i$ for each token i is then computed as the scaled dot-product attention scores over the input elements:

$$\mathit{A}=\mathrm{Sigmoid}\left({\displaystyle \frac{\mathbf{Q}{\mathbf{K}}^{\top }}{\sqrt{d_k}}}\right),{\mathit{W}}_i={\displaystyle \frac{1}{N}}{\displaystyle \sum _{j=0}^{N-1}}{\mathit{A}}_{ij}.$$

(11)

Subsequently, these coefficients are used in Eq. 5 to perform a weighted merging of components, allowing the model to adaptively determine the contribution of each part based on the input context.

Tuning importance scorer. We introduce a lightweight, learnable importance scorer to calculate the importance scores of tokens in DPP kernel. The design of this scorer draws inspiration from the Token Selector module proposed in VisionSelector[28]. Specifically, the input representation is projected into queries and keys via two separate linear layers, after which a simplified self-attention matrix is computed to capture pairwise token interactions, as expressed in Eq.10. The importance score for each token is then derived by averaging its interaction scores with all other tokens, as follows:

$$\mathit{P}={\displaystyle \frac{\mathbf{Q}{\mathbf{K}}^{\top }}{\sqrt{d_k}}},{\mathit{A}}_{\mathit{i}}={\displaystyle \frac{1}{N}}{\displaystyle \sum _{j=0}^{N-1}}{\mathit{P}}_{ij}.$$

(12)

Finally, the resulting importance score vector is incorporated into the similarity matrix to construct a refined DPP kernel, as described by Eq. 8, which guides the subsequent pruning of tokens.

4. Experiments

To validate the effectiveness of our method, we conduct a series of experiments. This section details our experimental setup, and the evaluation benchmarks used.

4.1. Experiment Setting

We choose open speech datasets for evaluation, covering ASR and speech understanding tasks. For speech recognition tasks, we select LibriSpeech[29], Fleurs[30], AISHELL1[31], AISHELL2-ios[32] and WenetSpeech[33]. For speech understanding tasks, we choose MELD[34], VocalSound[35], TUT2017[36], Nonspeech7k[37] and MMAU[38]. And we conduct experiments on Qwen2-Audio, Kimi-Audio, GLM-ASR. We select A-ToMe and FastAdasp for comparison. In addition, we migrate methods from the visual domain to audio models, including the pure pruning approaches VisPruner and CDPruner, as well as VisionZip, which combines both pruning and merging. Our experiments are conducted using the Kimi-Audio-Evakit toolbox[7]. For ASR and speech understanding tasks, we employ WER and accuracy metrics for evaluation, respectively.

Model Configuration: We consistently integrate SA-MAP after the encoder and the projection module of the model. For Kimi-Audio, which utilizes both discrete and continuous features, we select the continuous features for calculating the similarity matrix. For a fair comparison, we modified the two methods, A-ToMe and FastAdasp, which perform merging within the LLM block, after the projector instead. Unlike the toolbox, we do not employ GPT-4o-mini as the judge model for the sparsity task. Instead, we use a string matching approach to ensure a fair comparison.

4.2. Main Results

In this section, we compare our method with other methods on both dense and sparse tasks. For full experiments results, please refer to Appendix B.

Performance on Dense ASR Tasks. As shown in Table 1, on dense ASR benchmarks, preserving fine-grained acoustic details is critical. SA-MAP’s two-stage merging-pruning approach demonstrates clear superiority over methods that rely solely on a single metric or local dependencies. For instance, at a 60% retention rate, on Qwen2-Audio, SA-MAP reduces the absolute average WER by 1.53% and 0.96% compared to the best-performing pure merging method (FastAdaSP), and the strongest pure pruning method (CDPruner) respectively, narrowing the performance gap between these method and the uncompressed baseline by 53.4% and 41.9%. Furthermore, when compared with the hybrid approach VisionZip, SA-MAP yields a substantially higher reduction of 5.83% in absolute average WER, effectively closing the performance gap by 81.4%.

Across different compression ratios, our approach consistently exhibits superior performance compared with existing methods, particularly under high compression settings. For example, on Qwen2-Audio with retention rates ranging from 70% to 50%, SA-MAP further narrows the performance gap with the unpruned model by 41.84%–45.76% compared with the best-performing existing pruning method. Similar trends are observed across different backbone. On Kimi-Audio, SA-MAP achieves an additional reduction of 17.92%–41.20% at retention rates between 70% and 50%. Likewise, on GLM-ASR-Nano with retention rates from 70% to 60%, SA-MAP further reduce by 40.45%–52.48%. These results indicate that SA-MAP maintains stable and superior performance across a wide range of compression levels and backbones. Details are reported in Table A1–Table A3.

Performance on Sparse Understanding Tasks. For sparse understanding tasks, SA-MAP emphasizes adaptively compression to balance efficiency and semantic preservation as shown in Table 2. On Qwen2-Audio at 50% compression, it achieves an average accuracy of 64.82%, representing a relative improvement of 0.4% over the unpruned baseline. Compared with hybrid methods like VisionZip and FastAdaSP, SA-MAP further delivers relative gains of 0.64% and 1.5%, respectively, demonstrating its superior robustness under aggressive compression. On Kimi-Audio, SA-MAP achieves the highest overall average accuracy of 73.23%, outperforming all compared methods and achieving top performance on 4 out of 7 sub-tasks. Notably, on the challenging MMAU benchmark, it surpasses the unpruned baseline by 0.90% on music and by 0.6% on sound, demonstrating that our joint attention–similarity modeling effectively removes acoustic redundancy while preserving critical semantic information. Across other subsets, SA-MAP maintains stable recognition performance, highlighting its robustness in complex acoustic environments.

Table 1. Performance comparison of different pruning methods on ASR dense tasks (Qwen2-audio / Kimi-Audio / GLM-ASR)

Method	LibriSpeech				Fleurs		AISHELL-1	AISHELL-2	WenetSpeech		Average
	dev_clean	dev_other	test_clean	test_other	zh	en			test-meeting	test-net
Qwen2-Audio	1.67	3.65	1.74	4.03	3.63	5.20	1.52	3.08	8.40	7.64	4.06
Retain 60% Tokens (40% Compression Ratio)
VisionZip	7.31	10.35	7.08	10.10	8.02	6.85	6.99	13.85	17.88	23.75	11.22
VisPruner	7.42	9.74	7.20	9.69	7.00	8.39	6.91	9.67	13.37	14.07	9.35
CDPruner	4.22	6.05	4.18	6.53	4.88	7.17	2.70	4.62	12.29	10.90	6.35
A-ToMe	4.12	6.81	4.20	6.98	8.00	8.13	4.18	5.56	14.05	14.15	7.62
FastAdaSP	4.91	7.26	4.95	7.51	5.47	7.31	3.28	4.69	11.51	12.30	6.92
SA-MAP	2.59	5.00	2.72	5.02	4.37	5.94	2.69	4.42	11.05	10.11	5.39
Kimi-Audio	1.23	2.39	1.38	2.45	2.87	4.92	0.61	2.57	6.33	5.39	3.01
Retain 60% Tokens (40% Compression Ratio)
VisionZip	6.35	7.94	5.93	7.71	7.63	8.91	6.84	10.01	14.65	19.51	9.55
VisPruner	5.36	6.90	5.07	6.75	5.73	7.82	4.36	7.65	13.70	18.56	8.19
CDPruner	5.70	6.96	5.44	6.78	6.73	8.66	5.07	9.21	14.95	18.41	8.79
A-ToMe	4.13	6.04	3.92	5.87	6.26	7.57	3.58	6.88	12.97	14.99	7.22
FastAdaSP	4.49	6.15	4.28	6.02	4.27	7.64	2.22	5.67	12.62	14.75	6.81
SA-MAP	2.32	3.67	2.49	3.75	4.37	6.26	2.05	3.99	11.47	12.35	5.27
GLM-ASR-Nano	2.14	4.05	2.18	4.53	3.44	4.11	2.47	3.48	8.43	6.65	4.15
Retain 70% Tokens (30% Compression Ratio)
VisionZip	9.18	12.75	8.27	13.27	8.17	8.08	13.9	20.55	19.36	20.45	13.40
VisPruner	9.22	12.33	8.34	12.13	6.95	8.33	11.68	14.57	18.11	16.56	11.82
CDPruner	5.38	7.75	5.28	7.76	4.99	5.93	4.88	5.94	14.21	11.70	7.38
A-ToMe	4.55	8.04	4.66	7.83	5.08	5.77	5.40	5.95	13.16	12.08	7.25
FastAdaSP	6.45	10.34	6.50	10.41	4.34	6.70	6.42	7.02	16.10	14.10	8.84
SA-MAP	3.41	5.53	3.43	5.55	3.76	4.83	3.53	4.52	12.07	9.60	5.62

Table 1. Performance comparison of different pruning methods on ASR dense tasks (Qwen2-audio / Kimi-Audio / GLM-ASR)

Method	LibriSpeech				Fleurs		AISHELL-1	AISHELL-2	WenetSpeech		Average
	dev_clean	dev_other	test_clean	test_other	zh	en			test-meeting	test-net
Qwen2-Audio	1.67	3.65	1.74	4.03	3.63	5.20	1.52	3.08	8.40	7.64	4.06
Retain 60% Tokens (40% Compression Ratio)
VisionZip	7.31	10.35	7.08	10.10	8.02	6.85	6.99	13.85	17.88	23.75	11.22
VisPruner	7.42	9.74	7.20	9.69	7.00	8.39	6.91	9.67	13.37	14.07	9.35
CDPruner	4.22	6.05	4.18	6.53	4.88	7.17	2.70	4.62	12.29	10.90	6.35
A-ToMe	4.12	6.81	4.20	6.98	8.00	8.13	4.18	5.56	14.05	14.15	7.62
FastAdaSP	4.91	7.26	4.95	7.51	5.47	7.31	3.28	4.69	11.51	12.30	6.92
SA-MAP	2.59	5.00	2.72	5.02	4.37	5.94	2.69	4.42	11.05	10.11	5.39
Kimi-Audio	1.23	2.39	1.38	2.45	2.87	4.92	0.61	2.57	6.33	5.39	3.01
Retain 60% Tokens (40% Compression Ratio)
VisionZip	6.35	7.94	5.93	7.71	7.63	8.91	6.84	10.01	14.65	19.51	9.55
VisPruner	5.36	6.90	5.07	6.75	5.73	7.82	4.36	7.65	13.70	18.56	8.19
CDPruner	5.70	6.96	5.44	6.78	6.73	8.66	5.07	9.21	14.95	18.41	8.79
A-ToMe	4.13	6.04	3.92	5.87	6.26	7.57	3.58	6.88	12.97	14.99	7.22
FastAdaSP	4.49	6.15	4.28	6.02	4.27	7.64	2.22	5.67	12.62	14.75	6.81
SA-MAP	2.32	3.67	2.49	3.75	4.37	6.26	2.05	3.99	11.47	12.35	5.27
GLM-ASR-Nano	2.14	4.05	2.18	4.53	3.44	4.11	2.47	3.48	8.43	6.65	4.15
Retain 70% Tokens (30% Compression Ratio)
VisionZip	9.18	12.75	8.27	13.27	8.17	8.08	13.9	20.55	19.36	20.45	13.40
VisPruner	9.22	12.33	8.34	12.13	6.95	8.33	11.68	14.57	18.11	16.56	11.82
CDPruner	5.38	7.75	5.28	7.76	4.99	5.93	4.88	5.94	14.21	11.70	7.38
A-ToMe	4.55	8.04	4.66	7.83	5.08	5.77	5.40	5.95	13.16	12.08	7.25
FastAdaSP	6.45	10.34	6.50	10.41	4.34	6.70	6.42	7.02	16.10	14.10	8.84
SA-MAP	3.41	5.53	3.43	5.55	3.76	4.83	3.53	4.52	12.07	9.60	5.62

Table 2. Performance comparison of different pruning methods on Audio Understanding tasks (Qwen2-Audio / Kimi-Audio)

Method	MELD	mmau-test-mini			Nonspeech7k	TUT2017	VocalSound	Average
		music	sound	speech
Qwen2-Audio	51.19	60.48	72.07	60.36	86.48	32.35	88.00	64.42
Retain 50% Tokens (50% Compression Ratio)
VisionZip	51.73	60.48	72.07	61.56	85.38	31.73	86.33	64.18
VisPruner	50.88	61.68	73.87	62.16	84.69	30.74	86.22	64.32
CDPruner	51.61	61.38	71.77	61.26	86.62	31.73	87.50	64.55
A-ToMe	50.15	61.38	71.17	56.46	83.45	29.20	84.32	62.30
FastAdaSP	50.81	59.88	72.07	59.46	83.86	31.23	85.96	63.32
SA-MAP	51.96	61.38	72.07	63.06	85.10	32.53	87.64	64.82
Kimi-Audio	57.48	63.77	79.28	65.77	93.10	61.11	94.24	73.54
Retain 50% Tokens (50% Compression Ratio)
VisionZip	57.78	63.47	79.88	61.86	90.21	60.56	93.23	72.43
VisPruner	56.67	65.57	79.88	63.66	90.21	60.25	93.21	72.78
CDPruner	57.40	63.17	79.58	65.17	89.52	60.31	93.15	72.61
A-ToMe	57.25	64.67	79.28	63.66	92.14	60.68	93.40	73.01
FastAdaSP	56.79	63.47	79.88	66.37	90.48	60.37	93.34	72.96
SA-MAP	56.40	65.57	79.88	66.37	90.76	60.19	93.43	73.23

Table 2. Performance comparison of different pruning methods on Audio Understanding tasks (Qwen2-Audio / Kimi-Audio)

Method	MELD	mmau-test-mini			Nonspeech7k	TUT2017	VocalSound	Average
		music	sound	speech
Qwen2-Audio	51.19	60.48	72.07	60.36	86.48	32.35	88.00	64.42
Retain 50% Tokens (50% Compression Ratio)
VisionZip	51.73	60.48	72.07	61.56	85.38	31.73	86.33	64.18
VisPruner	50.88	61.68	73.87	62.16	84.69	30.74	86.22	64.32
CDPruner	51.61	61.38	71.77	61.26	86.62	31.73	87.50	64.55
A-ToMe	50.15	61.38	71.17	56.46	83.45	29.20	84.32	62.30
FastAdaSP	50.81	59.88	72.07	59.46	83.86	31.23	85.96	63.32
SA-MAP	51.96	61.38	72.07	63.06	85.10	32.53	87.64	64.82
Kimi-Audio	57.48	63.77	79.28	65.77	93.10	61.11	94.24	73.54
Retain 50% Tokens (50% Compression Ratio)
VisionZip	57.78	63.47	79.88	61.86	90.21	60.56	93.23	72.43
VisPruner	56.67	65.57	79.88	63.66	90.21	60.25	93.21	72.78
CDPruner	57.40	63.17	79.58	65.17	89.52	60.31	93.15	72.61
A-ToMe	57.25	64.67	79.28	63.66	92.14	60.68	93.40	73.01
FastAdaSP	56.79	63.47	79.88	66.37	90.48	60.37	93.34	72.96
SA-MAP	56.40	65.57	79.88	66.37	90.76	60.19	93.43	73.23

4.3. Trainable Compression Results

In Table 3, we evaluate SA-MAP’s trainable compression on LibriSpeech using Qwen2-Audio under multiple compression ratios. Results show that it consistently mitigates performance loss, especially at high compression ratios. This enables SA-MAP to maintain strong recognition accuracy, confirming the effectiveness of the trainable compression strategy.

4.4. Ablation Study

We conduct ablation studies on LibriSpeech to validate SA-MAP’s key components and optimal configurations. Experiments are performed across Qwen2-Audio, Kimi-Audio, and GLM-ASR, evaluating performance WER (% ↓). Results are detailed in Figure 4, Figure 5 and Figure 6.

Impact of Attention-Weighted Merging.Figure 4 demonstrates that our attention-weighted merging (lighter bars) consistently outperforms standard average merging across all models. For instance, on Qwen2-Audio with a 60% retention ratio, the WER drops significantly by 1.42% like A-ToMe. Similar trends are observed in Kimi-Audio and GLM-ASR. This confirms that weighting tokens by attention scores effectively preserves critical acoustic information that are otherwise diluted by background noise in simple averaging.

Impact of Attention-Enhanced Pruning. We compare ADPruner against the baseline CDPruner in Figure 5. ADPruner exhibits superior robustness, particularly at high compression rates. Notably, on Kimi-Audio with 50% token retention, ADPruner maintains a low WER of 7.67%, achieving a 6.17% reduction compared to CDPruner. This indicates that attention-guided pruning successfully retains semantically rich tokens that magnitude-based metrics often discard.

Impact of GSA-Merging (Combined with ADPruner)Figure 6 illustrates that GSA-Merging achieves the optimal efficiency-accuracy trade-off. Unlike A-ToMe or FastAdaSP which rely on adjacent token similarity, GSA captures comprehensive correlation patterns among all tokens within a group. This global perspective mitigates locality bias, ensuring more rational clustering and complete feature representation. Notably, on Kimi-Audio with 50% token retention, SA-MAP achieves a 22.2% relative WER improvement over FastAdaSP, confirming its ability to better preserve key semantic information in audio data.

Impact of Threshold Selection Taking Qwen2-Audio as an example, we selected LibriSpeech-dev_clean as the reference sample, and Figure 3 presents WER of SA-MAP under different thresholds. As can be seen from the figure, setting different thresholds affects the final accuracy. We determined the thresholds corresponding to various compression rates through a search process on the sample set. For instance, a compression rate of 30% can be achieved by setting the threshold to a specific value (e.g., 0.72), as demonstrated in the results.

5. Conclusion

This paper introduces SA-MAP, an adaptive, hardware-friendly, plug-and-play framework for joint audio token merging and pruning in Speech LLMs, driven by the synergy of similarity and attention. The first stage merges adjacent tokens to maximize semantic fidelity, while the second executes pruning to preserve token diversity. By setting a similarity threshold and jointly optimizing merging and pruning, our method effectively balances compression efficiency and information retention, maintaining reliability even under high compression settings. Experiments demonstrate that our method achieves a 50% lossless token reduction in understanding tasks and a 40% compression ratio in ASR with only marginal degradation evaluated on representative Speech LLMs.