A Survey on Efficient Protein Language Models

1. Introduction

Inspired by the success of large language models (LLMs) in natural language processing (NLP) [1,2], protein language models (pLMs) have emerged as a transformative force in computational biology [3]. By treating Amino Acid (AA) sequences as a “language of life,” [4] pLMs apply self-supervised learning to vast protein databases to learn the fundamental grammar of protein structure, function, and evolution [5,6]. A comparative overview between LLMs and pLMs is shown in Figure 3, providing intuitive context for their methodological parallels and distinctions. This paradigm shift enables protein analysis at an unprecedented scale of hundreds of millions of sequences [7], offering a powerful alternative to traditional experimental methods like A foundational experimental technique used to determine the precise three-dimensional atomic structure of a protein by analyzing the diffraction pattern of X-rays passing through a protein crystal. (X-ray crystallography), Nuclear Magnetic Resonance Spectroscopy (NMR) spectroscopy, and Cryo-Electron Microscopy (Cryo-EM) which, while foundational, are often slow and costly [8,9]. The power of pLMs lies in their ability to bridge the enormous gap between the over 200 million known protein sequences and the fewer than 200 thousand experimentally determined structures [10,11], thereby accelerating drug discovery and synthetic biology [9]. A historical overview of key milestones in pLM development is summarized in Figure 2.

The remarkable capabilities of pLMs stem from their deep neural architectures, predominantly transformer-based networks [1], which are trained extensively on massive protein sequence datasets [12] to learn intrinsic biological patterns and representations. The entire pipeline of pLMs consists of dataset construction, model architecture design, training methodologies (pre-training, fine-tuning, and distillation), and inference strategies, each influencing both performance and efficiency [13,14]. Powered by large-scale computational resources, these models integrate evolutionary and structural information from massive-scale evolutionary protein corpora into parameter spaces reaching up to tens of billions [7], thereby achieving state-of-the-art accuracy on structure-prediction benchmarks such as CAMEO [15] and CASP14 [16], a level not previously reached by single-sequence models.

Despite their transformative potential, the effectiveness of large-scale pLMs comes at a substantial computational and memory cost, posing significant practical challenges. Typical large pLMs contain billions of parameters—such as the 15B parameter ESM-2 [7]—and require extensive resources; for instance, training models of this scale conventionally demands tens of thousands of GPU hours, translating to immense financial and environmental burdens [7,17,18]. This heavy reliance on compute limits accessibility for resource-constrained research groups [19]. Furthermore, deploying such extensive models in real-world scenarios, such as proteome-wide homology searches, becomes computationally formidable, severely restricting their broader applicability and practical utility [20].

Motivated by these challenges, a growing body of research has focused explicitly on improving the efficiency of pLMs across various stages of their lifecycle [21,22,23]. These efficiency-focused methods aim to achieve competitive or even superior performance while reducing resource consumption by orders of magnitude. Specifically, efficiency strategies in pLMs encompass several critical categories, the reviewed methods of which are listed in Figure 1:

• Dataset Efficiency: Enhancing pLM efficiency through optimizing the allocation and utilization of datasets [21], to maximize learning from limited or expensive experimental data [22].
• Architecture Efficiency: Developing compact or lightweight architectures, such as quantized transformer models [24] or convolutional architectures [27,28], reducing parameter counts by up to 10× while maintaining predictive performance.
• Training Efficiency: Optimizing the training procedures, including computational resource allocation guided by scaling laws [21], efficient fine-tuning techniques [22,30,31], and knowledge distillation [39] approaches that transfer knowledge from larger, teacher models to compact, efficient student models [33,34,35,36].
• Inference Efficiency: Employing techniques such as post-training quantization (PTQ) [38], dense embedding-based retrieval [37], and alignment-free inference strategies to accelerate search speeds by over two orders of magnitude compared to traditional alignment methods [20].

While recent surveys have provided broad overviews of protein language models (pLMs) and their applications [9,40], our work particularly focuses on the efficiency dimension of pLMs across dataset, architecture, training, and inference. We systematically review these efficiency-focused methodologies, highlighting their core principles, underlying mechanisms, and performance implications. While recent surveys have provided broad overviews of protein language models (pLMs) and their applications [9,40], our work particularly focuses on the efficiency dimension of pLMs across dataset, architecture, training, and inference. We systematically review these efficiency-focused methodologies, highlighting their core principles, underlying mechanisms, and performance implications. The remainder of this survey is organized as follows: Section 2 details dataset strategies, Section 3 reviews architectural innovations, Section 4 discusses training methodologies, Section 5 covers inference acceleration, and Section 6 outlines future directions.

Figure 2. The evolution history of Protein Language Models (pLMs) from 2019 to 2025. The timeline illustrates the development and divergence of major pLM families, such as Masked pLMs and Conditional Inverse Folding pLMs, highlighting key models like the ESM series, ProtT5, and ProGen, along with their originating research labs.

Figure 2. The evolution history of Protein Language Models (pLMs) from 2019 to 2025. The timeline illustrates the development and divergence of major pLM families, such as Masked pLMs and Conditional Inverse Folding pLMs, highlighting key models like the ESM series, ProtT5, and ProGen, along with their originating research labs.

Figure 3. A comparative overview of general Large Language Models (LLMs) and Protein Language Models (pLMs). While both are based on the Transformer architecture, LLMs (left) typically use causal attention for autoregressive text generation. In contrast, pLMs (right) often use masked language modeling (with full attention) to learn sequence representations from amino acid inputs, which can then be used for downstream tasks like structure prediction. The folding blocks denote structure-prediction components used for structure prediction, e.g., as in ESMFold [7]; their internal architectural details are omitted here for brevity.

Figure 3. A comparative overview of general Large Language Models (LLMs) and Protein Language Models (pLMs). While both are based on the Transformer architecture, LLMs (left) typically use causal attention for autoregressive text generation. In contrast, pLMs (right) often use masked language modeling (with full attention) to learn sequence representations from amino acid inputs, which can then be used for downstream tasks like structure prediction. The folding blocks denote structure-prediction components used for structure prediction, e.g., as in ESMFold [7]; their internal architectural details are omitted here for brevity.

Table 1. Summary of representative dataset-related methods for pLMs.

Category	Method	Brief Description
Dataset	scaling-law [21]	Empirical study on optimal data/model allocation for pLM training.
	FSFP [22]	few-shot with meta-learning and LoRA [41] under severe data scarcity.

Table 1. Summary of representative dataset-related methods for pLMs.

Category	Method	Brief Description
Dataset	scaling-law [21]	Empirical study on optimal data/model allocation for pLM training.
	FSFP [22]	few-shot with meta-learning and LoRA [41] under severe data scarcity.

2. Datasets for Efficient pLMs

2.1. Background and Challenges

The training paradigm for modern language models is fundamentally data-driven, predicated on learning statistical and semantic regularities from vast corpora [42]. For Large Language Models (LLMs) like BERT [2] and the GPT series [43,44], this involves processing immense textual datasets that are rigorously deduplicated, filtered, and ultimately discretized into a finite vocabulary via tokenization [45,46]. Transposing this paradigm to computational biology, Protein Language Models (pLMs) operate on datasets of AA sequences drawn from large-scale biological repositories such as UniRef [12] and expansive Relating to the study of genetic material recovered directly from environmental samples, which often contain a vast collection of uncultured and unknown organisms. (metagenomic) collections [47]. These biological datasets undergo analogous preprocessing, including stringent sequence identity filtering and task-specific curation, to construct a suitable training corpus [21].

Within both NLP and computational biology, an efficiency-centric approach to dataset construction seeks to maximize model utility and generalization performance under a constrained computational-budget. For LLMs, this often involves curating data for optimal scale and diversity to avoid diminishing returns on compute [48,49]. In the context of pLMs, efficiency-driven data strategies have emerged to address distinct biological challenges. Key approaches include meta-learning frameworks for few-shot adaptation, which are critical in scenarios where experimental labels are prohibitively expensive or scarce [22]. Additionally, researchers have formulated empirically-derived scaling laws that govern the co-scaling of model and data size to optimize the training trajectory within a fixed computational envelope [21]. Such methods directly confront data-related bottlenecks, aiming to enhance model performance within the practical constraints of bioinformatics research.

The development of these efficiency-focused strategies is motivated by several fundamental challenges inherent to biological data utilization:

• Paucity of Labeled Data: A critical bottleneck in many real-world applications, such as protein engineering and A field of molecular biology that aims to understand the function of genes and proteins, often at a genome-wide scale. (functional genomics), is the extreme scarcity of experimentally validated labels. Often, only a few dozen labeled protein variants are available, posing a fundamental impediment to traditional supervised fine-tuning and demanding highly data-efficient adaptation methods [22].
• Absence of Principled Scaling Guidelines: Given a finite computational-budget, determining the optimal allocation between model parameter count and training data volume remains a significant challenge. An unprincipled approach, such as aggressively scaling model size while repeatedly cycling through a small dataset, can lead to suboptimal training dynamics, including premature convergence, overfitting, and a failure to capitalize on the model’s architectural capacity [21].

To surmount these obstacles, recent work has focused on developing sophisticated meta-learning protocols and establishing a principled, theoretical foundation for balancing model and data scaling. The subsequent section reviews these methodologies in detail, with an overview provided in Table 1. See Figure 4 for an illustration of these efficient dataset strategies.

2.2. Efficient Dataset Methods

Few-Shot Learning for Protein Fitness Prediction (FSFP) [22] addresses the challenge of efficiently fine-tuning pLMs under severe data scarcity [50], enabling fitness-prediction with only tens of experimentally labeled mutants per target protein. The key innovation of FSFP lies not in increasing data volume, but in its auxiliary-task construction, evolutionary-informed pseudo-labeling, and meta-learning-driven adaptation, providing a robust solution for pLM fine-tuning in low-data scenarios.

FSFP frames fitness-prediction as a ranking task, employing a listwise-loss [51] throughout both meta-learning and fine-tuning, which better reflects practical needs in protein engineering research and applications [52]. FSFP consists of the following major components:

• Auxiliary Task Construction: FSFP constructs three auxiliary tasks: it first retrieves two labeled DMS datasets [53,54] from proteins similar to the target, and additionally generates a third pseudo-labeled dataset using GEMME [55], which leverages evolutionary information from MSA [56,57].
• Meta-Learning: These auxiliary tasks are used for meta-training via MAML [58], equipping the pLM for rapid adaptation.
• Parameter-Efficient Adaptation: To prevent overfitting under few-shot settings [59], FSFP freezes the backbone pLM and trains LoRA [41] parameters during meta-training and when transferring to target task; meta-learning yields a better initialization of the LoRA parameters, and the subsequent adaptation continues to update these parameters.

Benchmarking on 87 protein datasets [60], FSFP demonstrates superior performance compared to strong baselines. It outperforms unsupervised zero-shot inference (including ESM-1v [61], ESM-2[7], and Saprot [62]) and significantly surpasses the state-of-the-art supervised few-shot baseline, Ridge Regression [63]. Specifically, FSFP boosts the average spearman correlation by up to 0.1 and raises positive hit rates in real-world engineering by 25% [22].

Scaling laws for dataset allocation in pLM training [21] have been empirically established to maximize training efficiency under fixed compute. Similar to LLMs in NLP tasks, simply increasing model size or compute is insufficient [42,48]; the size and diversity of training data are critical. Empirical analysis shows that masked language models (MLMs) [2] and causal language models (CLMs) [44,64] follow distinct power-law relationships between model size and required data, with sublinear data scaling [65]. Repeating small datasets across many epochs yields overfitting in MLMs [66] and diminishing returns in CLMs.

To address these issues, the study introduces the UniMeta200B dataset [47,67], aggregating diverse metagenomic protein sequences with strict deduplication. This improves out-of-domain perplexity and yields more stable independent and identically distributed validation curves with reduced variance across families.

The resulting scaling laws provide explicit formulas for allocating parameters and dataset size under a compute constraint, enabling principled, data-centric decisions in pLMs development. For fitted relations and resource-allocation details, see Section 4.2. These results highlight that enhancing dataset diversity—rather than simply expanding model size or repeating data—is essential for efficient protein language modeling.

3. Efficient Architecture

3.1. Background and Challenges

Model architecture fundamentally determines the representational capability and computational complexity of language models. LLMs in NLP predominantly employ transformer architectures [1], characterized by self-attention mechanisms that effectively capture long-range-dependencies. However, the quadratic scaling of transformers in computational complexity and memory consumption with respect to input sequence length poses significant efficiency challenges [1,48]. Alternatively, convolutional neural networks (CNNs) [68,69], leveraging local receptive fields, scale linearly with sequence length and inherently incorporate positional information through sliding window operations. These distinct architectural paradigms have informed similar developments in pLMs, where transformers remain dominant due to their robust performance on capturing long-range residue interactions [5], while CNN-based methods have emerged as efficient alternative [27].

Efficiency-focused model architectures in LLMs typically include strategies such as weight quantization, embedding dimension reduction, and structural sparsity to mitigate computational overheads [70,71,72]. Analogously, recent advancements in pLM architectures have pursued quantization of weights [24], embedding compression [26], and cross-attention [1] modules conditioned on homologous sequences [25] to integrate evolutionary information efficiently [73,74]. Convolution-based models, with inherent linear scaling, further enhance computational efficiency, presenting competitive alternatives for large-scale protein modeling tasks [27].

The emergence of efficiency-focused model architectures in pLMs is largely driven by several challenges in model architecture, including:

• High Computational and Memory Cost: Standard transformer-based pLMs require quadratic time and memory with respect to sequence length, leading to prohibitive resource demands for long proteins or large models. The use of full-precision weights and high-dimensional embeddings further increases GPU and energy costs, particularly during large-scale training and deployment [1,5,24,26,27].
• Redundancy and Massive Activations in Embeddings: Protein embeddings in large models are highly compressible, indicating over-parameterization, and also exhibit massive activations in certain channels, leading to inefficient memory use [26].
• Limited Inductive Bias and Information Integration: Existing architectures may not efficiently leverage important biological information, such as evolutionary relationships or sequence homology, constraining their performance and flexibility in specialized tasks [25].

The following section reviews recent architectural strategies developed to overcome these challenges. Table 2 provides an overview of these methods, and Figure 5 illustrates representative efficient architectural designs.

3.2. Efficient Transformer

2Bits of Protein [24] is a ternary-quantization architecture (based on [75]) which offers an efficient solution to the computational and memory bottlenecks of large pLMs. In this approach, all linear layers except the embedding and output head are quantized to ternary precision, with weights constrained to $\{-1,0,1\}$ by scaling and rounding each weight matrix [76,77,78]. Specifically, for a weight matrix W, the quantized matrix $\tilde{W}$ is defined as:

$$\tilde{W}=\mathrm{RoundClip}\left({\displaystyle \frac{W}{\gamma +ϵ}},-1,1\right),$$

where $\gamma$ is the mean absolute value of W’s elements, and $\mathrm{RoundClip}(x,a,b)$ rounds x to the nearest integer within $[a,b]$.

This design achieves substantial reductions in GPU memory and energy usage—prior NLP work reports up to 3.55× lower memory and 21.7× less energy consumption compared to full-precision models [76]. The ternary pLM preserves the standard transformer encoder architecture [1,7] and remains trainable from scratch with stable performance [79]. On ProteinGym [80], ternary ESM2-8M attains a mean spearman-correlation of 0.181 across 217 zero-shot tasks, competitive with efficient baselines such as ProtGPT2 [14] and UniRep [81]. Under matched compute (24 GPU-hours, ‘crammed’ training [82]), the performance gap becomes statistically insignificant, whereas under the standard training schedule, ESM2-8M is about 25% higher and statistically significant ($p<0.05$). Thus, ternary-quantization provides a principled and highly efficient design for pLMs, substantially reducing computational cost while maintaining competitive predictive performance.

Compressed Hourglass Embedding Adaptations of Proteins (CHEAP) [26] provide an architectural solution for drastically reducing the dimensionality and memory footprint of pLMs embeddings, while retaining both sequence and structure information. CHEAP employs a transformer-based hourglass-autoencoder  [83,84], which applies linear downsampling and projection to the ESMFold [7] latent embedding. Its key innovations can be divided into:

• Continuous Compression: CHEAP enables up to $128\times$ channel compression and $8\times$ sequence length compression through a learnable bottleneck, allowing backbone structure reconstruction with less than $1.34$ Å root-mean-squared distance [85] at $32\times$ channel compression, and maintaining sequence recovery above $99\%$ until fewer than 8 channels remain. To address abnormal massive activations in certain embedding channels, CHEAP introduces per-channel normalization [86,87], further improving compressibility.
• Discrete (Tokenized) Compression: CHEAP supports discrete embeddings via finite scalar quantization [88], which outperforms VQ-VAE [89,90] for large codebooks on latent and structure reconstruction, and it also provides a representation of protein structure obtainable from sequence alone.

These compressed embeddings enable efficient downstream applications—such as protein design, similarity search, and functional prediction—making advanced pLM representations accessible for resource-limited scenarios while preserving high-resolution information.

RAG-ESM [25] introduces a parameter-efficient encoder-decoder architecture that augments pretrained ESM2 [7] with a minimal number of cross-attention layers and extensive parameter-sharing. The encoder embeds a context (homologous) sequence using the original ESM2 model, while the decoder reuses pretrained ESM2 layers, inserting a few cross-attention layers to transfer information from the context embedding to the masked input sequence. Critically, self-attention and feed-forward layers in both modules share weights, so the number of new parameters is negligible compared to the backbone model. Formally, the total parameter count of RAG-ESM is

$$N_{\mathrm{RAG}-\mathrm{ESM}}=N+M(M\ll N),$$

where N is the number of parameters in the pretrained ESM2 backbone, and M is the number of newly added cross-attention parameters. Conditioning on homologs during inference allows RAG-ESM to extract evolutionary information [91] without requiring MSAs [74,92,93] or significantly increasing model size.

Empirically, RAG-ESM models with 12M and 165M parameters achieve 48% and 43% lower perplexity, respectively, than their ESM2-8M and ESM2-150M baselines on masked AA prediction. Both are trained under the discrete-diffusion [94,95] using the closest homolog as context [96], with modest compute budgets (50–120 GPU-hours). Cross-attention heads naturally learn alignment [97]; several heads reach $\rho >0.6$, and a logistic regression trained on all 60 heads achieves an average $\rho =0.76$ against Needleman–Wunsch [98] alignment matrices.

This design enables efficient and scalable homology-aware pLMs that achieve state-of-the-art performance among sequence-based models for homolog-conditioned generation and remain competitive on motif scaffolding [99], while retaining flexibility.

Cramming Protein Language Model Training in 24 GPU Hours [23] introduces an efficient transformer-based architecture specifically optimized for rapid, resource-constrained pre-training. The corresponding pre-training setup is detailed in Section 4.2.

The core approach adapts the HuggingFace ESM2 [7] backbone by removing: (1) all query, key, and value biases in attention blocks, (2) biases in intermediate linear layers, thereby reducing computational overhead while retaining model capacity to maximize per-token throughput under a stringent compute budget: model weights are initialized from scratch, and training is limited within 24 hours on a single GPU.

These architectural and implementation choices collectively allow a 67M-parameter model to achieve competitive downstream performance—on tasks such as protein fitness landscape inference [60] and protein–protein interaction classification [100]—with models trained for over 15,000× greater GPU hours. It demonstrates that by simplifying architectural details, it is feasible to obtain expressive pLMs suitable for practical research and deployment under extreme computational constraints.

3.3. Convolutional Layer Based Methods

Convolutional Autoencoding Representations of Proteins (CARP) [27] demonstrates that convolutional architectures can match or exceed transformer-based pLMs [5,7] in both predictive power and computational efficiency. CARP replaces transformer self-attention with ByteNet-style [68] dilated convolutional blocks [101,102], exponentially expanding the receptive-field while ensuring linear scaling in computation and memory with sequence length. Formally, the computational complexity per layer is:

$$\left\{\begin{array}{cc}\mathcal{O}\left(L^2\right)\hfill & \mathrm{for}\mathrm{transformer}\mathrm{self}-\mathrm{attention}\hfill \\ \mathcal{O}\left(\mathbf{L}\right)\hfill & \mathrm{for}\mathrm{convolution}\mathrm{in}\mathrm{CARP}\hfill \end{array}\right.$$

where L is the input sequence length.

CARP-640M (640M parameters) achieves a masked language modeling loss of $2.02$, close to ESM-1b’s [5]$1.96$, and attains higher average zero-shot mutation effect prediction (Spearman $0.49$ vs. $0.46$) on 41 datasets. Unlike ESM-1b, CARP processes sequences up to 4,096 residues without memory overflow and maintains stable loss and runtime for longer sequences. These results support convolutional architectures as efficient, scalable, and competitive alternatives to transformers for large protein sequence modeling.

ProtHyena [28] introduces the Hyena operator [103] as an efficient alternative to self-attention and standard convolution in pLMs. The per-layer computational complexity is:

$$\left\{\begin{array}{cc}\mathcal{O}\left(L^2\right)\hfill & \mathrm{Transformer}\mathrm{self}-\mathrm{attention}\hfill \\ \mathcal{O}\left(L\right)\hfill & \mathrm{Standard}\mathrm{CNN}\hfill \\ \mathcal{O}\left(\mathbf{L}{log}_{\mathbf{2}}\mathbf{L}\right)\hfill & \mathrm{The}\mathrm{Hyena}\mathrm{operator}\hfill \end{array}\right.$$

where L is the input sequence length.

Compared to transformer models whose self-attention layers require quadratic complexity, the Hyena operator enables substantially improved scalability; compared to standard CNNs, whose long-range modeling is limited by kernel size, Hyena leverages implicit long convolutions [104] and element-wise gating [105,106] to efficiently capture both local and global dependencies. The core block applies:

$$y=x_N·(h_N*(x_{N-1}·(h_{N-1}*\dots x_1·(h_1*v))))$$

where N is block depth, $h_i$ are long convolution filters, v is the input projection, and · denotes gating.

This enables ProtHyena to process sequences up to a million residues and achieve accuracy competitive with or better than larger transformer-based pLMs, such as ESM-1b [5], ESM-2 [7], and ProteinBERT [107], using about 10% of their parameters and up to $60\times$ speedup.

4. Efficient Training

4.1. Background and Challenges

Training LLMs generally follows two stages: pre-training and post-training. During pre-training, models are trained on large unlabeled corpora to learn general representations [2,43]. post-training then adapts these models to downstream objectives (including, for example, supervised fine-tuning (SFT) [49] and reinforcement learning (RL)-based tuning such as RLHF [108] or GRPO [109]). In this paper, we mainly focus on fine-tuning [110] within post-training. In parallel, distillation transfers knowledge from high-capacity teachers to compact students for efficiency [39,111,112].

Training pLMs adopt a conceptually similar pipeline to LLMs but with several domain-specific characteristics: (1) Pre-training: pLMs are trained on large corpora of unlabeled protein sequences at the amino-acid level to learn general biological representations [3,5,6,81]. (2) Post-training: pLMs are further adapted on task-specific datasets for specialized bioinformatics objectives [113], for example, function prediction [30,114]. (3) Distillation: knowledge is transferred from large teacher models to smaller students for efficient deployment, which closely mirrors the approach used in LLMs [34,36,112,115].

Given these characteristics, improving computational and data efficiency has become a key objective in pLMs training. Efficiency-focused pLMs training therefore focuses on the following bottlenecks:

• Challenges in Pre-training:

  -

  Lack of Resource Allocation Guidelines: It is challenging to optimally balance model size and training dataset size under fixed compute, with suboptimal allocation leading to overfitting [116] or diminished returns [21].

  -

  Limited Structural Integration: Most pre-training pipelines rely solely on sequence data, whereas structural information remains underutilized, which may ultimately limit both the efficiency and generalization capabilities of the models [29].

• Challenges in Fine-tuning:

  -

  Lack of Comprehensive Empirical Studies: Compared to NLP [117], systematic evaluations of parameter-efficient fine-tuning (PEFT) [41,118,119,120]methods for pLMs across diverse tasks has been limited, leaving practical guidelines unclear [30].

  -

  High Computational Demands: Full-model fine-tuning of large pLMs requires substantial memory and compute, limiting accessibility and scalability, especially under limited hardware situations [30,31].

  -

  Low Data Efficiency: Adapting models to new tasks with limited labeled data can be inefficient, as conventional fine-tuning is prone to overfitting and resource waste [22].

• Challenges in Distillation:

  -

  Teacher-student Discrepancy: Differences in size, architecture, or modality (e.g., sequence, structure, function) between teacher and student models can hinder efficient knowledge transfer and limit student performance [33,34,121].

  -

  Knowledge Transfer Bottlenecks: Distilling large teacher models or integrating multi-modal knowledge (e.g., structure, function) into compact students can reduce predictive power or require extensive tuning [34,35,36].

Following sections review methods addressing challenges through efficient pre-training, fine-tuning, and distillation.

4.2. Efficient Pre-Training Methods

To orient the discussion, Figure 6 summarizes the efficient pre-training methods reviewed in this section.

Table 3. Summary of representative training-stage efficiency methods for pLMs.

Category	Method	Brief Description
Pre-training	Scaling Law [21]	Empirical study on pLMs optimal data/model allocation.
	Structure Integration [29]	Joint sequence-structure pre-training (e.g., ProstT5).
	Cramming Challenge [23]	Train a performant pLM in 24 hours on a single GPU.
Fine-tuning	A study of PEFT Methods [30]	LoRA for parameter-efficient tuning.
	QLoRA [31]	4-bit quantization + LoRA for efficient fine-tuning.
	SES-Adapter [32]	Structure-aware adapters for cross-modal representation.
	FSFP [22]	few-shot with meta-learning and LoRA.
Distillation	Cross-distillation in AlphaFold3 [33]	Distillation from a single large teacher-model.
	ProtGO [34]	Transferring functional and structural knowledge.
	VespaG [35]	Distillation via evolutionary expert supervision.
	MTDP [36]	Aggregating knowledge from multiple teacher models.

Table 3. Summary of representative training-stage efficiency methods for pLMs.

Category	Method	Brief Description
Pre-training	Scaling Law [21]	Empirical study on pLMs optimal data/model allocation.
	Structure Integration [29]	Joint sequence-structure pre-training (e.g., ProstT5).
	Cramming Challenge [23]	Train a performant pLM in 24 hours on a single GPU.
Fine-tuning	A study of PEFT Methods [30]	LoRA for parameter-efficient tuning.
	QLoRA [31]	4-bit quantization + LoRA for efficient fine-tuning.
	SES-Adapter [32]	Structure-aware adapters for cross-modal representation.
	FSFP [22]	few-shot with meta-learning and LoRA.
Distillation	Cross-distillation in AlphaFold3 [33]	Distillation from a single large teacher-model.
	ProtGO [34]	Transferring functional and structural knowledge.
	VespaG [35]	Distillation via evolutionary expert supervision.
	MTDP [36]	Aggregating knowledge from multiple teacher models.

Scaling law-guided pre-training for pLMs [21] empirically establishes optimal resource allocation between model size and data for transformer-based pLMs. On a large dataset of 939M protein sequences, over 300 models were trained to reveal that both masked (MLM) and causal (CLM) objectives follow distinct power-law scaling similar to Kaplan et al. [42] and Hestness et al. [65]. For a $10\times$ increase in compute, optimal MLM size increases $6\times$ with $1.7\times$ more data, while CLM size grows $4\times$ with $3\times$ more data. The fitted scaling relations for MLMs are:

$$\left\{\begin{array}{cc}{N}_{\mathrm{MLM}}^{\mathrm{opt}}=6.19\times {10}^{-8}{C}^{0.776}\hfill & \mathrm{MLMs}\text{’}\mathrm{model}\mathrm{parameters}\hfill \\ \\ D_{\mathrm{MLM}}^{\mathrm{opt}}=2.02\times {10}^6{C}^{0.230}\hfill & \mathrm{MLMs}\text{’}\mathrm{training}\mathrm{tokens}\hfill \end{array}\right.$$

and for CLMs are:

$$\left\{\begin{array}{cc}{N}_{\mathrm{CLM}}^{\mathrm{opt}}=1.26\times {10}^{-3}{C}^{0.578}\hfill & \mathrm{CLMs}\text{’}\mathrm{model}\mathrm{parameters}\hfill \\ \\ D_{\mathrm{CLM}}^{\mathrm{opt}}=1.23\times {10}^2{C}^{0.422}\hfill & \mathrm{CLMs}\text{’}\mathrm{training}\mathrm{tokens}\hfill \end{array}\right.$$

where C represents the total pre-training FLOPs, N is the number of forward-activated non-embedding parameters, and D represents the number of training tokens.

Repeating data leads to overfitting for MLMs and diminishing returns for CLMs [122], which is mitigated by introducing a diverse, deduplicated dataset (UniMeta200B). Transfer learning—pre-training with CLM and then fine-tuning with MLM—reduces the compute needed for optimal MLMs by up to $7.7\times$. Large-scale validations confirm that these scaling laws yield models with better generalization and downstream performance under fixed compute budgets, particularly in practical, resource-constrained environments.

ProstT5 [29] is a T5-based model [49] that integrates structure information [123] into pLM pretraining to boost efficiency and versatility. ProstT5 unifies protein sequence (Amino Acid, AA) and structural (Three-Dimensional Interaction (token) (3Di) token) modalities within a single encoder-decoder architecture. Its key innovations in the pretraining stage can be divided into:

• Multimodal Pre-training: By encoding 3D structures from AlphaFoldDB [124] as 3Di-token sequences, and expanding the vocabulary with 3Di tokens, ProstT5 applies span-based denoising objectives to both AA and 3Di sequences, teaching the model new structural tokens while avoiding catastrophic forgetting of sequence information.
• Bi-directional Translation Objectives: ProstT5 is trained to translate between AA and 3Di representations using direction tags (<fold2AA>, <AA2fold>), enabling both “folding” (AA→3Di) and “inverse folding” (3Di→AA), thus robustly linking sequence and structure information.

This bilingual setup allows ProstT5-predicted 3Di strings to be used for structure-level similarity search (e.g., with Foldseek [125]), achieving The task of identifying distant evolutionary relationships between proteins, typically when their sequence similarity is very low but their 3D structures are still related. (remote homology detection) [126] sensitivity nearly matching experimental structures, but with a speedup of over three orders of magnitude (e.g., 43 seconds vs. 48 hours for full The complete set of proteins expressed by an organism, cell, or tissue at a given time. (proteome) annotation) [57,125]. Embedding-based annotation transfer further shows improved CATH fold classification [127] accuracy over ProtT5 [6] and ESM-1b [5], and The task of predicting the local secondary structures of a protein (e.g., alpha-helices, beta-sheets) from its primary amino acid sequence. (secondary structure prediction) with ProstT5 matches or exceeds prior state-of-the-art. In inverse folding [63,99], ProstT5 generates diverse sequences with predicted structures closely matching targets, demonstrating both enhanced efficiency and versatility.

Overall, integrating structure into pLM pretraining via bilingual modeling delivers substantial gains in inference speed and structural generalization, advancing protein design and large-scale annotation capabilities.

Cramming Protein Language Model Training in 24 GPU Hours [23] proposes a pre-training paradigm specifically tailored for rapid and resource-constrained scenarios. In this framework, a transformer-based pLM is trained from scratch under a strict 24 GPU-hour budget, with no use of pre-trained models at any stage. The pre-training pipeline consists of several key components:

• Random initialization: All model weights are initialized randomly.
• On-the-fly masked language modeling: Masked language modeling is performed directly on UniRef50 [12] splits, with all data processing (e.g., tokenization, filtering, sorting) occurring on-the-fly during training, and no offline pre-processing.
• Large effective batch size: Training employs a batch size of 128, sequence length of 512, and gradients are accumulated over 16 steps for an effective batch of 2048 sequences.
• Critical hyperparameter tuning: The optimizer is AdamW [128] with carefully tuned parameters. The learning rate schedule is central: the optimal regime employs a fast warmup [129,130] to a peak learning rate of $1\times {10}^{-3}$ over 1,000 steps, followed by a slow linear decay [131] to near zero over the remaining steps, with training capped at 50,000 updates.

These pre-training choices allow the resulting 67M-parameter model to match or approach the downstream performance of much larger models such as ESM2-3B [7]—trained for over 15,000× more GPU hours—on tasks including protein fitness landscape inference and PPI classification. The architectural design supporting these results is detailed in Section 3.2. This work demonstrates that, through systematic pre-training design and hyperparameter tuning [132,133], performant pLMs can be obtained at a fraction of the conventional computational cost, enabling broader accessibility for rapid biological modeling.

4.3. Efficient Fine-Tuning Methods

Fine-tuning in pLMs involves adapting a pre-trained model to specific downstream biological tasks using relatively small amounts of labeled protein data [30]. This step is inherently efficiency-focused, enabling rapid model adaptation without extensive retraining. Advanced PEFT techniques, such as LoRA [134] and structure-aware adapters [32], further minimize computational and memory demands while maintaining high performance. Subsequent subsections review these PEFT methods and associated efficiency gains.

To summarize the discussion, Figure 7 provides an overview of the key techniques covered for efficient fine-tuning.

A study of Parameter-efficient fine-tuning (PEFT) methods systematically compared full-model fine-tuning with several PEFT approaches—including LoRA [41], DoRA [135], IA3 [136], and prefix tuning [137]—across diverse pLMs and protein prediction tasks [30]. Its main findings include:

• Efficiency and accuracy: By freezing most model weights and only updating a small fraction (e.g., 0.25% for LoRA), PEFT methods, such as LoRA, achieved nearly equivalent accuracy to full fine-tuning.
• Training speedup and memory savings: For larger models, LoRA offered up to a 4.5× training speedup with comparable GPU memory requirements, and required saving only adapter weights, further improving efficiency.
• Compatibility and resource efficiency: LoRA, DoRA, and similar methods are compatible with mixed precision, gradient accumulation, and CPU-offloading, enabling fine-tuning even on 8GB or 16GB GPUs.
• Method comparison: Among PEFT methods, LoRA was generally most compute-efficient for large pLMs, with DoRA sometimes up to 30% slower, and all PEFT methods yielded average prediction gains of 61.3% compared to using static embeddings.

PEFT—especially LoRA—provides a practical solution for adapting large pLMs in resource-constrained environments, with all tested PEFT methods demonstrating substantial improvements over static, pre-trained embeddings.

Quantization with LoRA (QLoRA) [31] combines 4-bit quantization with LoRA to enable highly efficient fine-tuning of pLMs while maintaining strong performance [138]. By quantizing most model weights to 4 bits and updating only a small set of adapter parameters, QLoRA substantially reduces memory and computation costs.

Clancy et al. [31] systematically evaluated QLoRA across multiple pLMs, including ESM-2 [7], ESM C [139], ProtBERT [107], ProtT5-half [6], and Ankh-base [140], and diverse tasks, including Green Fluorescent Protein (GFP) fluorescence [141], protease stability [142], and protein secondary structure prediction [16,143,144]. Experiments show that QLoRA achieves an average GPU memory reduction of 46.7%, with generative models such as ProtGPT2 and ProLLaMA reaching up to 76.4% reduction.

Performance remains largely preserved: for ESM C 600M, fluorescence prediction SpearmanR is 0.850 for QLoRA versus 0.863 for full precision, and secondary structure accuracy remains identical at 0.870. In generative tasks, 4-bit quantization yields negligible differences in predicted protein quality, with metrics such as Foldseek pass rate, local distance difference test, and predicted local distance difference test showing no significant decline.

Overall, QLoRA lowers hardware barriers for fine-tuning and inference of large pLMs, making efficient protein modeling feasible on modest computing infrastructure.

Few-Shot Learning for Protein Fitness Prediction (FSFP) [22] builds on the dataset construction and meta-learning strategies described in Section 2.2, and focuses on PEFT for robust fitness prediction with minimal labeled data.

After meta-training on auxiliary tasks constructed from related proteins and MSA-based pseudo-labels, FSFP fine-tunes only a small subset of parameters (e.g., via LoRA with rank $r=16$) to mitigate overfitting. Fitness prediction is formalized as a ranking problem, with ListMLE loss [145] used to optimize the ordering of variants. On 87 deep mutational scanning datasets, FSFP outperforms both zero-shot and regression-based baselines, improves spearman-correlation by up to 0.1 with just 20 labeled mutants, and demonstrates strong extrapolation to unseen mutations. Wet-lab validation further highlights its efficiency and real-world utility.

SES-Adapter [32] is a structure-aware, PEFT framework for pLMs, integrating sequence and structural information via cross-modal multi-head attention. In SES-Adapter, the pLM backbone is frozen, and only the adapter module is trained. Structural features representations are serialized via, such as, FoldSeek [125] or DSSP [146], into embeddings and serve as queries (Q), while pLM sequence embeddings serve as keys and values (K, V). Uniquely, both Q and K are first embedded with rotary position embedding (RoPE). For an embedding vector $x\in {\mathbb{R}}^{d_k}$ at position p, RoPE performs a rotation on each coordinate pair $(2i,2i+1)$:

$$\left[\begin{array}{c}{(RoPE(x,p))}_{2i}\\ {(RoPE(x,p))}_{2i+1}\end{array}\right]=\left[\begin{array}{cc}cos{\theta }_{p,i}& -sin{\theta }_{p,i}\\ sin{\theta }_{p,i}& cos{\theta }_{p,i}\end{array}\right]\left[\begin{array}{c}{x}_{2i}\\ x_{2i+1}\end{array}\right],i=0,\dots ,\frac{d_k}{2}-1,$$

where ${\theta }_{p,i}$ is a position-dependent rotation angle varying with p and the dimension index i.

This fusion enables SES-Adapter to consistently outperform vanilla pLMs and prior tuning strategies: on nine diverse benchmarks, it yields up to 11% (avg. 3%) accuracy improvement, convergence acceleration up to 1034% (avg. 362%), and maintains robustness to structural noise (performance variation <0.6% across different structure predictors). SES-Adapter thus offers an efficient, scalable, and model-agnostic solution for fine-tuning pLMs in protein research.

4.4. Efficient Training via Distillation

knowledge distillation [39] in pLMs is a widely adopted model compression technique that transfers knowledge from larger, high-capacity teacher models to smaller, efficient student models. It typically involves guiding student models to mimic teacher outputs, intermediate representations, or other informative signals[147]. Distillation inherently emphasizes efficiency, enabling deployment of lightweight models that substantially reduce computational and memory requirements [148,149] while retaining predictive capabilities comparable to larger models [150,151]. Recent literature highlights diverse distillation strategies—including single-teacher [33], expert-guided [35], cross-modal [34], and multi-teacher adaptive methods [36]—consistently demonstrating significant efficiency improvements.

Cross-distillation of AlphaFold 3 [33] is a strategy for mitigating hallucinations and improving data efficiency in diffusion-based protein structure prediction [152]. Its core methodological steps are:

• Teacher-augmented training data: The training set is augmented by including high-confidence structures predicted by AlphaFold-Multimer v2.3 [153,154], in addition to experimental structures. Teacher-model predictions act as additional supervisory targets during training [155,156].
• Joint supervision with ground-truth and teacher outputs: AlphaFold 3 is explicitly optimized to align its outputs not only with experimental ground-truth structures, but also with reliable teacher-generated structures. This enables the model to learn more physically realistic disorder regions, as teacher-generated structures tend to represent unstructured segments as extended loops, reducing hallucinated compact order in intrinsically disordered regions.

On the CAID 2 benchmark [16], cross-distillation led to better alignment between predicted confidence (predicted local distance difference test) and actual disorder, effectively suppressing generative artifacts. By increasing both the diversity and quantity of usable training data, intra-chain metrics reach 97% of their maximum within the first 20,000 training steps, reflecting rapid and efficient learning. Overall, cross-distillation exemplifies how leveraging both ground-truth and high-quality model-generated supervision can simultaneously reduce hallucinations and accelerate robust protein structure learning in next-generation diffusion-based models.

ProtGO [34] is a multimodal protein representation learning framework that distills functional knowledge from a teacher to a student network via distribution alignment in latent space. The teacher leverages sequence, structure, and Gene Ontology (GO) (GO) [157,158] function annotations, while the student uses only sequence and structure, making it practical for annotation-scarce scenarios. Instead of aligning individual embeddings, ProtGO minimizes the KL-divergence [159] between the batch-level Gaussian-distributed graph embeddings of teacher and student:

$$L_{\mathrm{kd}}=\mathrm{KL}\left[p_S\left(z_S\right)\parallel p_T\left(z_T\right)\right]$$

where $z_S$ and $z_T$ are graph-level embeddings of the teacher and student models, and $p_S(·)$, $p_T(·)$ denote their estimated batch-wise distributions (Gaussian with mean and variance computed over each batch). The overall loss for training the student is:

$$L=L_{\mathrm{student}}+\beta L_{\mathrm{kd}}$$

where $L_{\mathrm{student}}$ is the cross entropy loss for downstream classification and $\beta$ is a trade-off hyperparameter.

This strategy enables the student to inherit rich functional information, achieving state-of-the-art results: e.g., 60.5% accuracy for fold classification (3.8 points higher than prior methods), 89.4% for enzyme reaction prediction, and top $F_{max}$ values on GO term and A numerical classification code for enzymes, based on the chemical reactions they catalyze. (Enzyme Commission (EC) number) prediction [160]. Ablation studies confirm the distillation module is critical for performance (fold accuracy drops from 60.5% to 57.8% when removed). ProtGO thus demonstrates the efficiency and effectiveness of distribution-based multimodal distillation for robust protein representation learning.

Expert-guided distillation in VespaG [35] offers an efficient strategy for protein The task of predicting the functional impact (e.g., on fitness, stability) of a mutation or a set of mutations on a protein. (variant effect prediction) by training a lightweight neural network (student) to imitate an evolutionary expert model (teacher). Here, the teacher is GEMME [55]—a MSA (MSA)-based method that generates mutational effect scores—while the student is VespaG, a feed-forward neural network with a single hidden layer. VespaG takes embeddings from a pretrained pLM (ESM-2 [7]) as input and learns to predict GEMME’s variant effect scores for each possible mutation. The student is trained by minimizing the mean squared error between its predictions $\widehat{y}$ and the GEMME scores $y^{*}$:

$$L={\displaystyle \frac{1}{N}}\sum _{i=1}^N{({\widehat{y}}_i-y_i^{*})}^2$$

This distillation approach removes the need for log-odds calculations and MSAs at inference time, enabling VespaG to rapidly predict mutational landscapes across entire proteomes on standard hardware. On the ProteinGym [80] benchmark (over 3 million variants), VespaG achieves a mean spearman-correlation of $0.48\pm 0.02$, matching state-of-the-art methods and sometimes surpassing its expert teacher in certain protein families. Importantly, VespaG is over three orders of magnitude faster than MSA-based methods. This demonstrates that expert-guided distillation with pLM embeddings yields robust, scalable, and practical variant effect predictors.

Multi-teacher distillation in MTDP [36] enables efficient and high-fidelity embedding by aggregating knowledge from multiple large pre-trained models, specifically ESM2-33 and ProtT5-XL-UniRef50.MTDP student, a T5-based model with ∼20M parameters, is trained via an adaptive teacher selection mechanism that dynamically assigns the most suitable guidance to each sample using RL strategy [161]. The training objective jointly minimizes a masked language modeling loss ($L_{\mathrm{MLM}}$) and a distillation loss ($L_{\mathrm{Distill}}$) defined by the KL divergence between student and teacher output distributions:

$$L=\alpha L_{\mathrm{MLM}}+(1-\alpha )L_{\mathrm{Distill}},\alpha =0.2$$

where,

$$\begin{array}{cc}\hfill L_{\mathrm{MLM}}& =-\sum _{i=1}^N\sum _{j=1}^{V}I(y_i=j)logp(x_i=j)\hfill \\ \hfill L_{\mathrm{Distill}}& =\sum _{i=1}^N\mathrm{KL}\left(p_T\left(x_i\right)\parallel p_S\left(x_i\right)\right)\hfill \end{array}$$

where N is sequence length, V is vocabulary size, $y_i$/$x_i$ are the true/predicted amino acids at position i, $I(·)$ is the indicator function, $p_T\left(x_i\right)$/$p_S\left(x_i\right)$ are the teacher/student output probabilities, and $\mathrm{KL}$ denotes KL divergence. At each step, selected teacher T for each sample is chosen via a RL-based scheduling mechanism [161].

Compared to its teachers (650M and 120M parameters), the MTDP student is dramatically more efficient, achieving ±1.5% accuracy of large models on tasks like function and The specific location or compartment within a cell where a protein resides and performs its function (e.g., nucleus, mitochondria, cytoplasm). (subcellular localization) prediction, and requiring ∼70% less encoding time for 10,000 protein sequences. Ablation studies confirm multi-teacher distillation outperforms single-teacher setups. MTDP can be deployed on modest GPU hardware, providing a scalable, resource-efficient solution for large-scale, real-time protein informatics applications.

5. Efficient Inference

5.1. Background and Challenges

Inference in language models refers to the process of generating outputs, such as predictions or embeddings, based on trained models given new input data. Autoregressive models (e.g. GPT series [44]) generate text by predicting subsequent tokens, while masked language models (e.g. BERT [2]) infer masked tokens or produce contextual embeddings for various tasks. This inference typically involves passing inputs through multiple transformer layers to capture contextual relationships and generate meaningful, task-relevant representations.

Analogously, pLMs infer structural, functional, or evolutionary properties from protein sequences. Given a protein sequence, pLM inference generates embeddings that encode rich biological information, facilitating downstream tasks like The computational task of identifying proteins that share a common evolutionary ancestor (homologs), which is often challenging when sequence similarity is low (remote homologs). (homolog detection), structure prediction, and functional annotation [5].

Efficiency-focused inference methods in LLMs encompass techniques such as quantization [162,163] and embedding-based dense retrieval, aiming to reduce computational costs and memory usage while preserving model performance [164,165]. Similarly, efficient inference in pLMs integrates analogous strategies, including PTQ [38], embedding-based retrieval via dual-encoder architectures [20], and structure-informed similarity scoring [37]. An overview of these efficient inference strategies is provided in Figure 8.

The emergence of efficiency-focused inference methods in pLMs is largely motivated by several challenges related to inference process, including:

• Resource Constraints: The inference process of large pLMs require substantial computation and memory, hindering deployment in resource-limited settings [38].
• Inference Latency: Traditional alignment-based and MSA methods are inherently slow and cannot efficiently scale to large databases, which is critical for many practical applications, such as proteome-wide A bioinformatics method used to identify sequences with a shared evolutionary ancestry, typically by comparing a query sequence against a database. (homology search) [20,37].
• Limited Sensitivity: Sequence-based retrieval may miss remote homologs and deeper structural relationships [20,37].

The following section reviews methods that address these issues, with an overview provided in Table 4.

5.2. Efficient Inference

PTQ4Protein [38] enables efficient inference of large pLMs such as ESMFold [7] by applying PTQ [79] to both weights and activations. Baseline experiments show that standard 8-bit uniform quantization [162] substantially reduces memory usage ($4\times$) but leads to significant accuracy loss—mainly due to the asymmetric and wide activation distributions before LayerNorm layers. To address this, PTQ4Protein employs piecewise linear quantization for activations, splitting the value range $[r_l,r_u]$ at breakpoints $p_l$ and $p_u$, and quantizing each region separately:

$$\widehat{r}=\left\{\begin{array}{cc}\mathrm{uni}(r_1;b-1,p_l,p_u,p_l)\hfill & r_1\in [p_l,p_u]\hfill \\ \mathrm{uni}(r_2;b-1,r_l,p_l,r_l)\hfill & r_2\in [r_l,p_l)\hfill \\ \mathrm{uni}(r_3;b-1,p_u,r_u,p_u)\hfill & r_3\in (p_u,r_u]\hfill \end{array}\right.$$

where $[r_l,r_u]$ denotes the full activation range, $p_l$ and $p_u$ are the central breakpoints, b is the total bit width, and $\mathrm{uni}(·)$ denotes uniform quantization within each segment. Each $r_1,r_2,r_3$ represents the activation value in the central and two tail regions, respectively.

Experiments demonstrate that PTQ4Protein achieves nearly lossless accuracy: for 8-bit quantization, TM-score [166] drops by only 0.36% (CASP14) [16] and 0.08% (CAMEO) [167], with memory usage reduced to 25% of baseline. Even at 6 bits, accuracy loss remains below 0.5%. Ablation studies show ESM-2 [7] is particularly robust, and the visual fidelity of predicted structures is preserved. Overall, PTQ4Protein provides a practical solution for deploying large pLMs on resource-limited hardware with minimal accuracy degradation.

PLMSearch [37] is an efficient inference framework for large-scale remote homology search that leverages pLM embeddings and structure-informed similarity prediction. Its pipeline consists of:

• PfamClan-based pre-filtering: PfamClan [168] assigns proteins to evolutionarily related superfamilies (clans); this stage retains only protein pairs sharing the same clan, ensuring high recall while greatly reducing the candidate search space.
• SS-predictor: A bilinear neural network is trained to predict structural similarity (TM-score [169]) between protein pairs based on pLM embeddings; it integrates predicted TM-scores with cosine similarity, capturing global and local sequence features for remote homolog detection.
• PLMAlign: For top-scoring pairs, per-residue alignment is performed using pLM embeddings to refine hit precision and enhance biological relevance.

A key efficiency innovation is the precomputation and caching of all target protein embeddings, enabling scalable, million-level searches via a single forward pass through the SS-predictor. By restricting downstream alignment to high-confidence pre-filtered pairs, PLMSearch dramatically reduces computation while maintaining high sensitivity. On standard benchmarks, PLMSearch achieves orders-of-magnitude faster search and higher sensitivity than conventional structure or sequence-based methods. These results establish PLMSearch as a scalable, structure-aware solution for accurate and ultra-fast remote protein homology inference.

Dense Homolog Retriever (DHR) [20] is an structure-aware inference framework for large-scale protein homolog detection, integrating pLMs with deep dense retrieval. Its workflow consists of:

• Dual-encoder architecture: Both query and candidate sequences are embedded into dense vectors using a pair of ESM-initialized encoders, allowing alignment-free similarity calculation.
• Contrastive learning: The dual-encoders are trained to embed homologous pairs close together and nonhomologous pairs far apart, enabling effective discrimination via dot-product scoring.
• Scalable dense retrieval: All database embeddings are precomputed and cached, allowing rapid, large-scale retrieval of homologs through efficient vector similarity search [170,171] on standard hardware.

As an MSA prefilter, DHR accelerates MSA (with JackHMMER [172]) by up to 93-fold, increases MSA diversity (log Meff [173]), and improves AlphaFold2 [174] structure prediction—boosting TM-score [169] by up to 0.03 and lowering root-mean-squared distance [85] by 0.15 Å on hard targets. DHR also maintains robust performance on massive datasets such as BFD [175] or [47] MGnify (over 500 million sequences) [57], demonstrating its practicality for structure-aware, scalable, and accurate protein inference at unprecedented speed.

6. Future Directions

Quantum Algorithms for Protein Structure Prediction

A central challenge in classical, physics-driven approaches to protein folding (like A computational simulation method that models the physical (molecular dynamics (MD))) is their immense computational complexity. These methods simulate the intricate A multidimensional surface representing the potential energy (energy landscape) of a protein, a task that scales poorly on classical hardware. This has motivated the exploration of quantum algorithms, which are naturally suited for simulating complex physical systems. An overview of quantum–classical learning pipeline is provided in Figure 9.

The core physical principle of protein folding is that a protein chain will configure itself into its most stable, lowest-energy state. In physics, the total energy of a system is described by a mathematical operator known as a Hamiltonian. Therefore, a protein’s most stable 3D structure can be reframed as finding the lowest possible energy value of this The energy operator of a system, representing the total energy (Hamiltonian). This lowest-energy solution is known as the ground state.

Quantum Protein Structure Prediction (QPSP) attempts to solve this problem by first simplifying it [176,177,178,179]. The vast, continuous space of all possible protein folds is too complex to model directly. Instead, the protein is “discretized” by placing its residues onto the points of a 3D grid, or lattice. Common choices include cubic lattice [180,181,182] and tetrahedral lattice, which build upon earlier 2D models [183,184,185].

With the problem now on a lattice, the Hamiltonian is assembled from terms that encode the The potential energy associated with a given molecular or (structural energy): terms that capture the interactions between nearby An individual amino acid unit within a protein chain; in lattice (residue) and terms that act as constraints to enforce a structurally valid chain. Finding the ground state of this lattice-based Hamiltonian becomes an optimization task for a quantum computer. Research in this emerging field is focused on improving the realism of these models—progressing from 2D to 3D lattices—and developing more efficient encoding (e.g., turn-based rather than coordinate-based) to reduce the number of qubits required.

A fundamental limitation remains: native protein structures are not confined to a discrete lattice. This mismatch limits the fidelity of such models. Consequently, the accuracy of any lattice-based prediction is ultimately constrained by the resolution of the lattice, thereby necessitating models with higher degrees of freedom to yield more realistic fits [186].

Hybrid quantum–classical Transformers as a path toward quantum PLMs.

A more immediate and pragmatic future direction lies in creating hybrid quantum-classical models. Instead of attempting to replace the entire classical pipeline, this approach seeks to identify computationally expensive sub-components of existing models (like PLMs) and "enhance" them with a quantum subroutine.

The self-attention mechanism is a prime target. At its core, the attention score is a similarity measure—typically a simple dot product—between a Query (Q) vector and a Key (K) vector. This score determines how much influence one token has on another. Recent work on hybrid transformers [187] proposes the replacement of this classical dot product.

In this hybrid model, the Q and K vectors are used to prepare quantum states. A quantum circuit then measures the fidelity (or overlap) between these two states, and this measurement becomes the new attention score. This is not just a faster way to compute a dot product; it is a fundamentally different, quantum-native similarity metric.

This pathway is promising for several reasons. As a new similarity metric, this quantum-derived similarity is a specialized nonlinear function that may be able to capture complex relationships between token embeddings that a simple dot product misses, potentially improving model generalization in the low-data regimes common to protein engineering. It also offers theoretical efficiency: while the primary benefit may be representative power, this quantum subroutine also offers a theoretical cost reduction for the scoring portion, from $O\left(n^{2}d\right)$ to $O(n^{2}logd)$, where n is the sequence length and d the embedding dimension. Furthermore, it has An acronym for Noisy Intermediate-Scale Quantum, referring (NISQ)-era (Noisy Intermediate-Scale Quantum) feasibility, as the model remains "mostly classical," retaining standard tokenization, embedding, and feed-forward blocks. Crucially, the quantum subroutine only requires a The fundamental unit of quantum information, represented by (qubit) count that scales as $logd$, making its parameter-efficiency a viable candidate for NISQ hardware. Finally, it represents a practical prototyping path that can be prototyped today. Using frameworks like CUDA-Q [188,189], the quantum circuit can be simulated on classical CPU/GPU hardware and integrated directly into a standard model’s training loop, which allows for rapid, end-to-end validation of the concept.

Positive results from such simulations would position hybrid attention as a credible and practical first step toward future, quantum-enhanced PLMs for protein modeling.

7. Conclusions

In this survey, we systematically reviewed advancements in efficient pLMs across four dimensions: dataset optimization, model architecture, training strategies, and inference. We highlighted a spectrum of techniques—from meta-learning and scaling law-guided design to PEFT, knowledge distillation, and quantization—that collectively address the formidable computational demands of modern pLMs. These approaches enable substantial cost reductions, democratize accessibility, and lay the groundwork for scalable, practical deployment in computational biology.

Despite this substantial progress, significant challenges remain. While efficient pLMs achieve competitive results on standard benchmarks, a performance gap often emerges in complex tasks such as de novo structure prediction or out-of-distribution generalization. The optimal strategies for downstream adaptation, such as fine-tuning protocols and representation pooling in extreme resource-limited settings, also remain underexplored. Future research must not only extend these efficiency paradigms to more diverse biological tasks but also systematically benchmark adaptation strategies and develop even more scalable solutions through advances in optimization, quantization, and architectural innovation.

We anticipate that continued innovation in these areas, coupled with broader community engagement, will be essential for unlocking the full potential of efficient pLMs and accelerating biological discovery.