Advances in Parameter-Efficient Fine-Tuning: Optimizing Foundation Models for Scalable AI

1. Introduction

Foundation models, large-scale neural networks pre-trained on vast amounts of data, have revolutionized the field of artificial intelligence by achieving state-of-the-art performance across a wide range of tasks [1]. These models, such as GPT, BERT, T5, and CLIP, exhibit remarkable generalization capabilities, enabling their application to numerous downstream tasks with minimal adaptation [2]. However, fine-tuning these models for specific applications poses significant challenges, particularly due to their enormous size, high computational costs, and memory-intensive requirements [3]. As a result, researchers have increasingly explored parameter-efficient fine-tuning (PEFT) techniques, which aim to adapt foundation models to new tasks with significantly reduced computational and storage overhead. Traditional fine-tuning approaches involve updating all model parameters to optimize performance on a specific task [4]. While effective, this method is prohibitively expensive for modern foundation models, which often contain billions of parameters. Additionally, full fine-tuning leads to storage inefficiencies, as a separate copy of the model must be maintained for each downstream task. In contrast, PEFT techniques modify only a small subset of parameters while keeping the majority of the pre-trained model fixed [5]. This approach not only reduces computational and memory requirements but also facilitates efficient multi-task learning and continual learning by minimizing catastrophic forgetting [6]. Several PEFT methods have emerged, each leveraging different strategies to achieve efficient adaptation. Low-rank adaptation (LoRA) injects trainable low-rank matrices into specific layers of the model, enabling efficient weight updates with minimal parameter overhead [7]. Adapter-based methods introduce lightweight, task-specific layers that are fine-tuned while keeping the original model frozen. Prompt tuning and prefix tuning leverage soft prompt embeddings or learnable prefix vectors to steer model behavior without modifying its internal weights [8]. BitFit, a simpler yet effective approach, fine-tunes only the bias terms of the model, significantly reducing the number of trainable parameters [9]. The adoption of PEFT methods has been particularly beneficial in resource-constrained environments, where full fine-tuning is infeasible due to hardware limitations [10]. By reducing the computational cost of adaptation, PEFT enables democratized access to foundation models, empowering researchers and practitioners with limited resources to leverage state-of-the-art AI systems [11]. Moreover, PEFT methods facilitate model deployment in edge devices, where memory and processing power are restricted, thereby broadening the applicability of foundation models beyond traditional cloud-based infrastructure [12]. Despite the advantages of PEFT, several open challenges remain. The trade-offs between efficiency and performance must be carefully considered, as reducing the number of trainable parameters can sometimes lead to suboptimal task adaptation [13]. Moreover, the interpretability and robustness of PEFT methods require further investigation to ensure reliable and consistent outcomes across diverse applications. Additionally, as foundation models continue to scale, new strategies for efficient adaptation must be developed to keep pace with their increasing complexity. This survey provides a comprehensive overview of parameter-efficient fine-tuning techniques for foundation models, examining their methodologies, advantages, limitations, and practical applications [14]. We categorize existing approaches, analyze their effectiveness across various benchmarks, and highlight emerging trends in the field. By consolidating the current state of PEFT research, this survey aims to guide researchers and practitioners in selecting appropriate fine-tuning strategies for their specific needs and to inspire future advancements in efficient model adaptation.

2. Background

The emergence of foundation models has significantly transformed the landscape of artificial intelligence, enabling the development of powerful, general-purpose systems that can be adapted to diverse downstream tasks [15]. These models, typically based on deep neural networks, are pre-trained on large-scale corpora using self-supervised or unsupervised learning techniques. Once trained, they serve as a foundation for various applications, requiring only minor modifications or additional fine-tuning to achieve optimal performance on specific tasks [16].

2.1. Foundation Models and Transfer Learning

Foundation models leverage large-scale pre-training to learn generalizable representations of language, vision, and multimodal data [17]. Prominent examples include transformer-based architectures such as BERT [18], GPT [19], T5 [19], and CLIP [17]. These models have demonstrated exceptional performance across a wide range of natural language processing (NLP), computer vision (CV), and multimodal tasks [20]. Their success is largely attributed to their ability to learn hierarchical feature representations from massive datasets, enabling efficient knowledge transfer to new tasks with minimal supervision. Transfer learning plays a crucial role in adapting foundation models to specific applications [21]. In traditional transfer learning, a pre-trained model is fine-tuned on a target dataset by updating its parameters through supervised learning [22]. This approach allows models to retain general knowledge while specializing in domain-specific tasks [23]. However, fine-tuning large-scale models presents significant computational and memory challenges, necessitating the development of more efficient adaptation techniques [24].

2.2. Challenges of Full Fine-Tuning

While full fine-tuning has been the standard approach for adapting foundation models to downstream tasks, it is often impractical for several reasons:

• Computational Complexity: Large foundation models contain billions of parameters, making full fine-tuning computationally expensive and requiring high-end hardware resources such as TPUs or GPUs [25].
• Memory Constraints: Storing multiple fine-tuned versions of a foundation model for different tasks leads to excessive memory consumption, limiting scalability in real-world applications.
• Catastrophic Forgetting: When fine-tuning on new tasks, models often overwrite previously learned knowledge, hindering their ability to retain general capabilities [26].
• Deployment Challenges: Fine-tuned models must be stored separately for each task, increasing the cost of deployment and maintenance, particularly in edge computing scenarios [27].

To address these challenges, researchers have developed parameter-efficient fine-tuning (PEFT) methods that reduce the number of trainable parameters while preserving model performance [28]. These methods aim to maintain the benefits of transfer learning while minimizing computational and storage costs [29].

2.3. Parameter-Efficient Fine-Tuning: A New Paradigm

Parameter-efficient fine-tuning (PEFT) has emerged as a promising alternative to full fine-tuning, offering substantial efficiency gains without compromising performance [30]. Instead of updating all model parameters, PEFT techniques modify only a small subset, significantly reducing the number of learnable parameters [31]. This approach offers several advantages:

• Reduced Computational Cost: By fine-tuning only a fraction of the model’s parameters, PEFT methods lower the training cost, enabling adaptation on resource-constrained devices [32].
• Improved Storage Efficiency: Since PEFT methods require storing only a small number of task-specific parameters, they facilitate scalable multi-task learning and efficient model deployment.
• Modular and Reusable Components: Many PEFT methods allow task-specific modifications to be applied in a modular fashion, enabling fast adaptation to new domains without extensive retraining [33].
• Better Transferability: By preserving the majority of the pre-trained weights, PEFT techniques maintain generalization capabilities, mitigating catastrophic forgetting.

In the following sections, we systematically explore various PEFT methods, categorizing them based on their underlying techniques and assessing their effectiveness across different tasks. We provide a comparative analysis of their computational efficiency, parameter savings, and task performance, highlighting key trends and future research directions in parameter-efficient adaptation [34].

3. Taxonomy of Parameter-Efficient Fine-Tuning Methods

Parameter-efficient fine-tuning (PEFT) methods can be broadly categorized based on the underlying strategies they employ to adapt foundation models [35]. These techniques aim to strike a balance between efficiency and task-specific performance by modifying only a small subset of model parameters while leveraging the knowledge stored in the pre-trained weights [36]. In this section, we present a taxonomy of PEFT approaches, classifying them into four major categories: (1) Adapter-Based Methods, (2) Low-Rank Adaptation Methods, (3) Prompt-Based Tuning, and (4) Selective Parameter Tuning [37].

3.1. Adapter-Based Methods

Adapter-based methods introduce additional task-specific modules into the pre-trained model while keeping the original parameters mostly frozen [38]. These methods insert small, trainable neural network layers—known as adapters—into specific parts of the architecture, allowing the model to learn task-relevant transformations without modifying its core parameters [39]. The key advantages of adapter-based methods include modularity, reusability, and efficiency in multi-task learning.

3.1.1. Standard Adapters

The concept of adapters was first proposed in NLP tasks to facilitate efficient transfer learning [40]. A standard adapter consists of a lightweight feed-forward network inserted between layers of a transformer model [41]. These adapters are typically composed of a down-projection layer that reduces the feature dimension, a non-linearity (e.g., ReLU), and an up-projection layer that restores the original dimension [42]. This bottleneck structure ensures that the number of trainable parameters remains minimal.

3.1.2. Compacter and HyperNetwork-Based Adapters

Variants of adapters, such as Compacter [44], further reduce parameter count by leveraging parameterized weight sharing [43]. Instead of maintaining independent adapter weights for each task, these methods use a low-rank reparameterization through hypernetworks, which generate adapter weights dynamically [44]. This approach significantly enhances parameter efficiency while maintaining adaptation flexibility.

3.2. Low-Rank Adaptation Methods

Low-rank adaptation (LoRA) [45] is a highly effective PEFT technique that approximates weight updates using low-rank matrices. Instead of modifying the original model weights, LoRA injects trainable low-rank decomposition matrices into selected layers, capturing essential task-specific variations while keeping the main parameters frozen [46].

3.2.1. LoRA and Its Variants

LoRA operates by decomposing weight updates into a pair of low-rank matrices, reducing the number of trainable parameters by orders of magnitude [47,48]. It allows efficient adaptation with minimal computational overhead [49]. Variants of LoRA, such as QLoRA [52], integrate quantization techniques to further enhance efficiency, making them suitable for large-scale models with memory constraints [50].

3.3. Prompt-Based Tuning

Prompt-based tuning methods leverage additional input tokens or soft embeddings to steer the model’s behavior without modifying its internal weights. These approaches include prompt tuning, prefix tuning, and P-Tuning.

3.3.1. Soft Prompt Tuning

Prompt tuning [51] optimizes a small set of trainable prompt vectors that are prepended to the input sequence [52]. These soft prompts act as learnable task-specific instructions, conditioning the model without requiring modifications to its core parameters [53].

3.3.2. Prefix Tuning

Prefix tuning [54] extends soft prompt tuning by introducing learnable prefix embeddings that modulate attention mechanisms at every layer [55]. This technique enables more effective control over the model’s internal representations while maintaining a minimal number of trainable parameters.

3.4. Selective Parameter Tuning

Instead of adding new modules or embeddings, selective parameter tuning methods fine-tune only a small subset of existing model parameters [56]. Examples include BitFit [60], which fine-tunes only the bias terms, and Layerwise Fine-Tuning, which selectively updates certain layers.

3.4.1. BitFit

BitFit is an extremely lightweight PEFT method that freezes all weight matrices except for the bias terms [57]. Despite its simplicity, BitFit achieves competitive performance on many NLP tasks while requiring orders of magnitude fewer trainable parameters than full fine-tuning [58].

3.4.2. Layerwise Fine-Tuning

Layerwise fine-tuning approaches selectively update only certain layers of the model, often focusing on higher layers that encode more task-specific knowledge [59]. This strategy balances efficiency and effectiveness, ensuring that the model retains most of its pre-trained knowledge while adapting to new tasks [60].

3.5. Comparison of PEFT Methods

Each PEFT method offers distinct trade-offs in terms of parameter efficiency, computational cost, and task-specific performance. Table 1 summarizes the key characteristics of the major PEFT approaches.

In the next section, we delve into an in-depth analysis of the performance of these PEFT techniques across various benchmarks and tasks [61].

4. Analysis and Evaluation of PEFT Methods

To understand the effectiveness of parameter-efficient fine-tuning (PEFT) methods, it is essential to evaluate their performance across various benchmarks, considering trade-offs between efficiency and accuracy [62]. This section presents a comparative analysis of PEFT approaches based on multiple criteria, including performance on NLP and vision tasks, computational efficiency, parameter reduction, and generalization capabilities [63].

4.1. Performance Across NLP and Vision Benchmarks

PEFT techniques have been extensively evaluated on a variety of natural language processing (NLP) and computer vision (CV) benchmarks [64]. These evaluations measure the ability of each method to achieve competitive results while significantly reducing the number of trainable parameters.

4.1.1. Natural Language Processing Benchmarks

For NLP tasks, PEFT methods have been assessed on datasets such as:

• GLUE [65]: A collection of language understanding tasks including sentiment analysis, textual entailment, and paraphrase detection.
• SuperGLUE [70]: An extension of GLUE with more challenging reasoning tasks.
• SQuAD [71]: A popular reading comprehension benchmark.
• MT-Bench [66]: A benchmark evaluating multi-task generalization across various NLP tasks [67].

Across these benchmarks, adapter-based methods and LoRA have shown strong performance, often achieving results comparable to full fine-tuning while requiring significantly fewer parameters [68]. Prompt tuning methods, while efficient, sometimes exhibit reduced effectiveness on tasks requiring deep syntactic understanding [69].

4.1.2. Computer Vision Benchmarks

In computer vision, PEFT techniques have been evaluated on tasks such as:

• ImageNet [76]: A large-scale image classification benchmark [70].
• COCO [78]: A dataset for object detection and segmentation [71].
• CIFAR-10/100 [80]: Standard benchmarks for image classification [72].
• VTAB [82]: A comprehensive benchmark for evaluating transfer learning in vision tasks [73].

PEFT approaches such as LoRA and adapter-based tuning have demonstrated strong transferability in vision models like ViTs [74] and CLIP [17,75]. Selective parameter tuning methods, such as BitFit, have been less effective in vision tasks due to the importance of spatial representations that require deeper adaptation [76].

4.2. Computational Efficiency and Memory Footprint

One of the primary motivations for PEFT methods is reducing computational and memory overhead. Table 2 compares different PEFT techniques in terms of their training and inference efficiency [77].

As shown in Table 2, prompt tuning and BitFit have the lowest computational costs, making them highly efficient for real-time adaptation. However, they sometimes underperform on complex reasoning tasks compared to LoRA and adapter-based methods.

4.3. Generalization and Transferability

Another critical factor in evaluating PEFT methods is their ability to generalize across diverse tasks. Key findings include:

• Adapters and LoRA exhibit strong cross-task generalization, making them ideal for multi-task and continual learning [78].
• Prompt tuning is highly effective in zero-shot and few-shot learning scenarios but may struggle in domain adaptation [79].
• BitFit works well for classification tasks but is less effective for structured prediction tasks such as dependency parsing or named entity recognition.

4.4. Summary of Key Findings

Based on our analysis, we summarize the strengths and limitations of different PEFT methods:

• Adapters: Strong task performance and modularity but require additional inference computation.
• LoRA: Highly efficient and effective for large models, but may require careful layer selection [80].
• Prompt Tuning: Extremely parameter-efficient but less robust for complex tasks [81].
• BitFit: Simple and effective for classification but limited in complex reasoning tasks.

In the next section, we explore real-world applications of PEFT methods and discuss their practical deployment in industry and research [82].

5. Real-World Applications and Deployment

Parameter-efficient fine-tuning (PEFT) methods have gained significant traction in real-world applications, enabling the adaptation of large foundation models in resource-constrained environments while maintaining high performance. This section explores various domains where PEFT methods have been successfully deployed, highlighting their impact on natural language processing, computer vision, multimodal learning, and edge computing. We also discuss key deployment considerations, including latency, scalability, and energy efficiency [83].

5.1. Natural Language Processing Applications

PEFT methods have revolutionized NLP applications by allowing the fine-tuning of large language models (LLMs) such as GPT-3, T5, and BERT with minimal computational overhead [84]. Some notable applications include:

5.1.1. Conversational AI and Chatbots

Large-scale chatbots and virtual assistants, such as ChatGPT and Google Assistant, rely on PEFT methods to efficiently adapt to new domains, languages, and user preferences [85]. Adapter-based and LoRA-based fine-tuning techniques enable these models to specialize in customer support, healthcare interactions, and financial services while keeping storage and compute costs low [86].

5.1.2. Machine Translation

In machine translation, PEFT methods facilitate efficient adaptation of multilingual models, such as mT5 and XLM-R, to low-resource languages. Instead of retraining the entire model for a new language, adapter-based approaches and prompt tuning allow for effective cross-lingual transfer with minimal parameter updates [87].

5.1.3. Text Summarization and Information Extraction

News summarization, legal document processing, and biomedical text mining benefit from PEFT by enabling rapid customization of large-scale transformer models for domain-specific tasks [88]. LoRA and prefix-tuning approaches have shown strong results in summarization benchmarks while keeping computational costs manageable [89].

5.2. Computer Vision Applications

In computer vision, PEFT methods have enabled scalable fine-tuning of vision transformers (ViTs) and CLIP-based models for diverse applications, including:

5.2.1. Medical Image Analysis

PEFT methods such as LoRA and adapter tuning have been successfully applied in medical imaging tasks, including disease diagnosis and radiology report generation. For example, fine-tuning pre-trained ViTs on MRI or CT scan datasets with adapter modules allows for efficient deployment in healthcare settings without excessive retraining costs [90].

5.2.2. Autonomous Driving

PEFT techniques help adapt large vision models for real-time perception in autonomous vehicles [91]. Selective parameter tuning methods, such as BitFit, enable efficient updates to object detection and scene segmentation models, allowing adaptation to new environments with minimal retraining.

5.2.3. Retail and E-commerce

Visual recommendation systems and automated product tagging in e-commerce platforms leverage PEFT to customize pre-trained vision models to brand-specific datasets [92]. For instance, LoRA-based fine-tuning allows companies to personalize search and recommendation algorithms without incurring high computational costs.

5.3. Multimodal Learning and Vision-Language Models

PEFT plays a crucial role in fine-tuning multimodal models such as CLIP, Flamingo, and BLIP, which integrate vision and language understanding [93]. Some key applications include:

5.3.1. Content Moderation and Hate Speech Detection

Social media platforms employ multimodal models for detecting harmful content, combining textual and visual cues. PEFT enables these models to adapt to evolving moderation policies and emerging linguistic patterns efficiently [94].

5.3.2. AI-Generated Art and Image Captioning

Creative AI applications, such as DALL·E and Stable Diffusion, benefit from PEFT by allowing users to customize image generation models with style-specific or domain-specific fine-tuning [95]. Prompt tuning and LoRA-based techniques enable rapid adaptation to new artistic styles without modifying the entire model [96].

5.4. Edge Computing and Low-Power Devices

One of the most significant advantages of PEFT is its applicability in edge computing, where deploying full-scale fine-tuned models is infeasible due to hardware constraints. Some real-world examples include:

5.4.1. Smartphones and IoT Devices

Voice assistants, camera enhancements, and real-time language translation on smartphones leverage PEFT to deliver high-performance AI features without excessive power consumption [97]. For instance, prompt tuning allows language models to personalize voice recognition systems on mobile devices [98].

5.4.2. On-Device Personalization

Recommendation engines and predictive typing systems on smartphones use PEFT methods such as BitFit and prefix tuning to adapt to user behavior while ensuring privacy by keeping models locally fine-tuned [99].

5.4.3. Industrial Automation

In manufacturing, PEFT methods are used to fine-tune anomaly detection models for predictive maintenance [100]. Instead of retraining entire models on new machinery data, LoRA-based adaptation enables efficient model updates with minimal computational overhead.

5.5. Deployment Considerations and Challenges

Despite their advantages, deploying PEFT methods in real-world applications requires careful consideration of several factors:

• Latency and Inference Speed: While PEFT reduces training costs, some methods (e.g., adapter-based tuning) introduce additional computational overhead at inference time [101]. Strategies such as pruning and quantization can help mitigate this issue.
• Scalability Across Tasks: PEFT enables multi-task adaptation, but managing multiple fine-tuned modules (e.g., task-specific adapters) can lead to complexity in large-scale deployments [102].
• Energy Efficiency: Reducing the number of trainable parameters lowers energy consumption, making PEFT ideal for sustainable AI development [103]. However, some methods may still require optimization for real-time applications.
• Security and Privacy: Deploying PEFT models on personal devices or enterprise environments raises security concerns [104]. Techniques such as federated learning and encrypted fine-tuning can enhance privacy [43].

5.6. Summary and Future Directions

The real-world adoption of PEFT methods has demonstrated their effectiveness in reducing computational costs while maintaining competitive performance. Key takeaways include:

• PEFT methods are widely used in NLP, CV, and multimodal applications, enabling efficient fine-tuning of foundation models.
• Edge computing benefits significantly from PEFT by enabling AI-driven applications on resource-constrained devices.
• Deployment challenges, such as inference overhead and scalability, require further research to optimize PEFT for production use.

In the next section, we discuss emerging trends and future research directions in PEFT, exploring novel techniques and potential advancements [105].

6. Emerging Trends and Future Directions

As foundation models continue to grow in scale and complexity, parameter-efficient fine-tuning (PEFT) remains a crucial area of research [106]. Recent advancements in model architectures, training strategies, and optimization techniques are shaping the future of PEFT [107]. In this section, we explore emerging trends and outline potential future directions, including advanced low-rank adaptation methods, dynamic fine-tuning strategies, cross-modal adaptation, and efficient deployment in decentralized and federated settings [108].

6.1. Advanced Low-Rank Adaptation Techniques

Low-rank adaptation (LoRA) has proven to be one of the most effective PEFT methods, but ongoing research aims to enhance its flexibility and efficiency [109]. Key developments include:

• Dynamic Low-Rank Updates: Instead of using a fixed low-rank decomposition, recent approaches propose dynamically adjusting the rank of adaptation matrices based on task complexity, leading to more efficient fine-tuning [110].
• Sparse Low-Rank Decomposition: Hybrid approaches that combine sparsity with low-rank updates aim to further reduce the number of trainable parameters while maintaining model expressiveness [111].
• Rank Selection Optimization: Methods such as AutoLoRA explore automatic selection of optimal rank values, reducing the need for manual hyperparameter tuning.

6.2. Meta-Learning and Dynamic Fine-Tuning

One of the limitations of current PEFT techniques is that they require separate tuning for each new task [112]. Emerging approaches in meta-learning and dynamic adaptation aim to mitigate this:

• Task-Agnostic Fine-Tuning: Instead of learning separate parameters for each task, meta-learning-based PEFT approaches enable models to generalize to unseen tasks with minimal updates [113].
• Adaptive Parameter-Freezing: Future PEFT techniques may incorporate automated freezing and unfreezing of parameters based on task complexity, optimizing the trade-off between efficiency and performance [114].
• Few-Shot and Continual Learning: Integrating PEFT with few-shot and continual learning paradigms allows models to efficiently acquire new knowledge while avoiding catastrophic forgetting.

6.3. Cross-Modal and Multilingual Adaptation

Foundation models are increasingly trained on diverse modalities, necessitating efficient adaptation techniques that extend beyond a single modality or language:

• Unified Multimodal PEFT: Research is exploring ways to integrate PEFT across vision-language, speech-text, and other multimodal domains, allowing a single fine-tuned module to operate across different inputs [115].
• Cross-Lingual Adaptation: PEFT methods that enable seamless adaptation of multilingual models across new languages, especially for low-resource settings, are gaining attention [116].
• Domain-Agnostic PEFT: Instead of learning separate PEFT modules for each domain (e.g., medical vs [117]. legal text), future approaches may develop universal PEFT methods that generalize across domains [118].

6.4. Federated and Decentralized Fine-Tuning

With growing concerns over data privacy and computational efficiency, federated and decentralized fine-tuning approaches are becoming critical areas of research:

• Federated PEFT: Techniques such as federated LoRA allow models to be fine-tuned across decentralized devices while preserving user privacy [119].
• On-Device Adaptation: Lightweight PEFT models optimized for deployment on smartphones, IoT devices, and edge servers reduce reliance on centralized cloud computing [120].
• Privacy-Preserving Fine-Tuning: Encryption-based techniques such as homomorphic encryption and secure multi-party computation are being integrated with PEFT to ensure data security during adaptation [121].

6.5. Scalable and Hardware-Aware PEFT

Efficient fine-tuning must also consider hardware constraints, particularly for deployment on accelerators such as GPUs, TPUs, and specialized AI chips:

• Quantized and Pruned PEFT Models: Applying quantization and pruning to PEFT methods can further reduce memory and inference costs without compromising accuracy [122].
• PEFT for Sparse Models: Sparse models such as mixture-of-experts (MoE) architectures require specialized PEFT techniques that adapt only active model components [123].
• Accelerator-Aware Fine-Tuning: Future PEFT approaches will need to optimize for hardware-specific characteristics, such as tensor core efficiency on GPUs or reduced precision arithmetic on edge AI chips.

6.6. Bridging PEFT with Neural Architecture Search

Neural Architecture Search (NAS) has revolutionized the discovery of efficient deep learning architectures [124]. A promising future direction is integrating NAS with PEFT to automatically identify the most efficient fine-tuning configurations:

• Task-Specific PEFT Search: Automating the selection of PEFT methods based on task characteristics to optimize efficiency and accuracy [125].
• Hybrid NAS-PEFT Approaches: Exploring how NAS can guide the design of efficient adapters, LoRA layers, or prompt tuning strategies for foundation models [126].
• Energy-Efficient NAS-PEFT Pipelines: Combining NAS with PEFT to identify configurations that minimize energy consumption while maintaining high performance [127].

6.7. Challenges and Open Questions

Despite the rapid progress in PEFT research, several open challenges remain:

• Optimal PEFT Selection: How can models automatically determine which PEFT method is best suited for a given task?
• Interpretability: How do PEFT modifications affect the internal representations of foundation models, and can they be made more interpretable [128]?
• Scalability to Extremely Large Models: As models surpass trillions of parameters, will current PEFT techniques remain viable, or will entirely new methods be needed [129]?
• Generalization Across Domains: Can a single PEFT module generalize across multiple domains without requiring separate fine-tuning?

6.8. Summary and Outlook

PEFT has become an essential component of modern AI, enabling efficient adaptation of foundation models for a wide range of applications. Looking ahead:

• Research is moving toward more dynamic and adaptive PEFT methods that optimize parameter selection and task generalization [130].
• Multimodal, multilingual, and federated fine-tuning approaches will expand the reach of PEFT beyond traditional NLP and vision tasks.
• Hardware-aware and NAS-integrated PEFT methods will drive new breakthroughs in efficiency and deployment scalability.

As AI systems continue to evolve, PEFT will remain a key enabler of practical and efficient foundation model adaptation, ensuring that powerful AI capabilities remain accessible across diverse domains and computational environments.

7. Conclusion

The rapid evolution of foundation models has necessitated the development of parameter-efficient fine-tuning (PEFT) techniques, allowing for scalable adaptation without incurring prohibitive computational and memory costs. This survey has explored the fundamental principles, methodologies, real-world applications, and emerging trends in PEFT, highlighting its critical role in democratizing access to large-scale AI models.

7.1. Key Takeaways

Throughout this survey, we have identified several key insights:

• PEFT methods such as adapters, LoRA, prefix tuning, and BitFit offer efficient alternatives to full fine-tuning, significantly reducing the number of trainable parameters while maintaining competitive performance.
• The application of PEFT extends beyond natural language processing to computer vision, multimodal learning, and edge computing, demonstrating its versatility across different AI domains.
• Real-world deployment of PEFT requires careful consideration of inference latency, scalability, energy efficiency, and privacy, motivating ongoing research in optimized architectures.
• Emerging trends such as dynamic fine-tuning, federated adaptation, and hardware-aware optimization are paving the way for more flexible and scalable PEFT strategies.
• Despite its advantages, PEFT still faces challenges related to optimal method selection, interpretability, and generalization across domains, indicating promising directions for future research.

7.2. The Future of PEFT

Looking ahead, PEFT is expected to play an increasingly important role in the AI ecosystem. As models grow even larger, the need for efficient fine-tuning strategies will become even more pronounced, especially in decentralized and privacy-sensitive applications. Future research will likely focus on:

• Hybrid PEFT Approaches: Combining different fine-tuning techniques (e.g., LoRA with prompt tuning) to optimize both efficiency and generalization.
• Task-Agnostic Adaptation: Developing PEFT methods that allow for seamless cross-domain and cross-modal transfer without requiring extensive task-specific fine-tuning.
• Neural Architecture Search (NAS) for PEFT: Automating the discovery of optimal fine-tuning configurations tailored to specific tasks and computational constraints.
• Sustainable AI: Further reducing the carbon footprint of model adaptation through lightweight PEFT methods that minimize energy consumption.

7.3. Final Thoughts

As AI continues to integrate into various aspects of society, ensuring the accessibility and efficiency of foundation models remains a crucial challenge. PEFT methods provide a promising pathway to making these models more adaptable, cost-effective, and widely deployable. By addressing the open questions and exploring novel approaches, future research in PEFT will contribute to the broader goal of making AI both powerful and sustainable.