CoDES: A Context-Efficient Framework for Enhancing Small Language Models via Domain-Specific Adaptation and Model Ensembling

Efficiently adapting large language models (LLMs) to specialized domains remains challenging due to substantial computational and memory requirements. In this work, we introduce CoDES (Context-efficient Domain Ensemble System), a framework designed to enhance small language models through context-efficient domain adaptation and weighted parameter ensembling. CoDES integrates context-specific fine-tuning, parameter-efficient adaptation using Low-Rank Adaptation (LoRA), and completion-only supervision to focus training on answer tokens while preserving pretrained capabilities and reducing computational cost. To further improve performance and robustness, the framework combines multiple fine-tuned models through weighted parameter ensembling. We evaluate CoDES on biomedical multiple-choice question answering using the MedMCQA benchmark. Experimental results show that the ensemble of tuned small models achieves 74.8% accuracy, approaching the performance of a much larger 72B-parameter model (77.1%). While requiring substantially fewer computational resources. The proposed framework offers several practical advantages, including achieving comparable performance, lower energy consumption, faster inference, and flexible adaptation to specialized domains. By reducing the reliance on extremely large models, CoDES provides a scalable and resource-efficient pathway for deploying high-performing language model systems in knowledge-intensive environments where models must be frequently updated with evolving domain information.