Natural Language Processing in the Era of Large Language Models: Foundations, Integration, and Low-Resource Frontiers

1. Introduction

The relationship between Natural Language Processing and Large Language Models is one of both inheritance and transformation. NLP, as a field, developed over decades a rich repertoire of techniques: part-of-speech tagging, named entity recognition, syntactic parsing, coreference resolution, and semantic role labeling, among others [1,2]. These were crafted largely as modular components in processing pipelines, each consuming linguistically annotated data and producing structured representations of language.

The arrival of the transformer architecture [3] and its subsequent scaling into billion-parameter models such as GPT [4], BERT [5], and T5 [6] did not render classical NLP obsolete; rather, it radically changed its role. Pre-trained LLMs implicitly encode rich linguistic knowledge that emerges from self-supervised objectives over massive corpora, capturing morphological, syntactic, and semantic regularities without explicit annotation [7]. Yet the theoretical frameworks of NLP remain indispensable for understanding, evaluating, and auditing LLM behavior — and for building systems in data-scarce settings where brute-force scaling is not a viable strategy.

This preprint is organized as follows. Section 2 revisits foundational NLP components and traces their integration into LLM architectures. Section 3 examines the multilingual challenge, focusing on low-resource settings. Section 4 surveys state-of-the-art approaches and benchmarks. Section 5 presents targeted case studies. Section 6 discusses open problems and future directions.

2. NLP Foundations Inside LLM Architectures

2.1. Tokenization and Subword Modeling

Tokenization is the first — and arguably most consequential — NLP decision embedded in every LLM. Classical approaches relied on whitespace segmentation or hand-crafted rules; modern LLMs universally employ subword tokenization algorithms such as Byte-Pair Encoding (BPE) [8], WordPiece [9], and SentencePiece [10]. These methods balance vocabulary coverage against sequence length, enabling models to handle morphologically rich languages with limited out-of-vocabulary exposure.

Nevertheless, tokenization remains an Achilles’ heel for linguistic diversity. Languages with complex morphology — such as Turkish, Finnish, or Tagalog — are systematically over-segmented by vocabularies optimized for English, yielding longer token sequences, higher inference cost, and diminished representational fidelity [11]. Recent work by Rust et al. [12] demonstrates that vocabulary mismatch is a primary driver of performance degradation in multilingual BERT variants for low-resource languages.

2.2. Syntactic and Semantic Representations

Probing studies have extensively demonstrated that transformer hidden states encode syntactic information, including dependency parses and constituency structures, despite receiving no explicit syntactic supervision [13,14]. Work by Tenney et al. [14] showed that lower layers of BERT capture POS and chunking information while higher layers encode semantic role labels and coreference — mirroring the classical NLP pipeline in depth.

This emergent syntactic structure has practical implications: it suggests that LLMs can serve as general-purpose NLP backends, reducing the need for task-specific annotated datasets. However, this implicit encoding is fragile under distribution shift, performing poorly on non-standard dialects, code-switched text, and languages not represented in pre-training [15].

2.3. Discourse and Pragmatics

Beyond sentence-level processing, LLMs have absorbed capabilities traditionally associated with discourse NLP: coherence modeling, anaphora resolution, and pragmatic inference. The attention mechanism, with its capacity to relate arbitrary token pairs across long contexts [3], provides a structural basis for discourse-level reasoning. Long-context models such as Longformer [16] and extended context variants have further pushed these boundaries, enabling document-level summarization and multi-turn dialogue management.

3. The Low-Resource Language Challenge

3.1. The Data Imbalance Problem

Of the approximately 7,000 living languages documented globally, fewer than 100 have meaningful representation in the corpora used to train contemporary LLMs [17]. The Common Crawl, which underlies much of modern pre-training data, is estimated to be approximately 46% English, with the remaining share distributed heavily toward high-resource European languages [18]. Languages such as Tagalog, Yoruba, Swahili, Amharic, and dozens of others have token counts orders of magnitude smaller, producing severely undertrained representations.

This disparity has practical consequences: LLMs deployed in multilingual settings exhibit substantial performance gaps between high- and low-resource languages on benchmarks including XNLI [19], TyDiQA [20], and XTREME [21]. These gaps are not merely quantitative; they reflect genuine failures of understanding in morphology, pragmatics, and cultural knowledge that scale alone cannot easily remedy.

3.2. Cross-Lingual Transfer and Multilingual Pre-Training

Multilingual pre-trained models such as mBERT [5], XLM-R [22], and mT5 [23] represent the dominant strategy for extending NLP capabilities to low-resource languages through shared multilingual representations. XLM-R, trained on 100 languages with a two-trillion-token corpus, demonstrated that sufficiently large multilingual models achieve competitive performance even on languages with limited training data, via cross-lingual transfer [22].

Zero-shot and few-shot cross-lingual transfer — the ability to fine-tune on high-resource language data and generalize to low-resource targets — has emerged as the primary practical strategy for NLP development in resource-scarce contexts [24]. Instruction-tuned LLMs such as mT0 [25] and BLOOMZ [25] further extend this paradigm, demonstrating that multilingual instruction following can be achieved with surprisingly few target-language examples.

3.3. Data Augmentation and Synthetic Data Strategies

In the absence of large labeled corpora, data augmentation has proven essential for low-resource NLP. Techniques include back-translation [26], cross-lingual data projection, template-based generation, and LLM-assisted corpus synthesis. Back-translation, originally developed for neural machine translation, has been repurposed to generate pseudo-labeled training data for classification and named entity recognition tasks [26].

More recently, LLMs themselves have been used as data generators for low-resource NLP, prompting high-resource-language models to produce training instances that are then projected or translated into target languages [27]. While promising, this approach risks propagating hallucinations and culturally inappropriate content, underscoring the need for human-in-the-loop validation, especially for safety-sensitive applications.

4. State-of-the-Art Approaches and Benchmarks

4.1. Instruction Tuning and RLHF for Multilingual Settings

Instruction tuning — fine-tuning LLMs on diverse task descriptions paired with target outputs — has become a standard technique for eliciting zero-shot generalization [28]. When applied in multilingual settings, instruction tuning has been shown to substantially improve performance on low-resource languages, even when the instruction dataset itself is predominantly English [25]. This phenomenon, termed ‘multilingual task arithmetic,’ suggests that instruction-following capabilities transfer across languages more readily than raw language modeling performance.

Reinforcement Learning from Human Feedback (RLHF) [29], while predominantly studied in English, presents both opportunity and risk for low-resource languages. Human preference data is expensive to collect for every language, and crowd-sourced preferences may not reflect culturally specific notions of helpfulness or harmlessness. Approaches such as preference data projection and culturally-aware RLHF are active areas of inquiry [30].

4.2. Benchmark Landscape

Evaluating multilingual NLP progress requires benchmarks that go beyond high-resource languages. Key benchmarks include XNLI (cross-lingual natural language inference, 15 languages) [19], TyDiQA (typologically diverse question answering, 11 languages) [20], XTREME and XTREME-R (cross-lingual transfer, 40+ languages) [21], and AfriSenti [31] for African language sentiment analysis. These benchmarks reveal consistent performance hierarchies strongly correlated with training data volume, emphasizing the need for targeted low-resource interventions beyond simply scaling general-purpose models.

5. Case Studies in Low-Resource NLP Toolkits

5.1. CalamanCy: A Tagalog NLP Toolkit

Tagalog, spoken by over 90 million people as a first or second language in the Philippines, has historically been severely underrepresented in NLP research. CalamanCy [32,35] is a spaCy-based NLP toolkit developed to address this gap, providing production-ready pipelines for Tagalog including tokenization, POS tagging, morphological analysis, dependency parsing, and named entity recognition. The toolkit leverages transformer backbones pre-trained on Filipino web corpora and demonstrates that focused, community-led NLP development can produce high-quality tools for low-resource languages outside the standard English-centric ecosystem.

A subsequent implementation and evaluation presented at the 2025 IEEE International Conference on Industrial Technology & Computer Engineering (ICITCE) [35] further validated CalamanCy’s practical utility in industrial and academic settings, confirming its robustness across diverse Tagalog text domains. CalamanCy’s design philosophy emphasizes interoperability with existing NLP infrastructure, enabling downstream integration with modern LLM pipelines. The toolkit highlights both the potential and the effort required for community-driven low-resource NLP: achieving competitive performance requires careful curation of training data, attention to morphological complexity, and sustained community engagement [32,35].

5.2. AfroNLP and African Language Models

The AfroNLP initiative and associated models — including AfriBERTa [33] and Afro-XLMR — represent coordinated efforts to build NLP infrastructure for African languages at scale. Africa hosts over 2,000 languages, the vast majority of which have no digital NLP resources whatsoever. AfriBERTa, trained on 11 African languages using a compact corpus aggregated from Common Crawl and curated sources, demonstrates that even modest pre-training can yield meaningful NLP capabilities when combined with careful multilingual fine-tuning [33]. These efforts underscore the importance of community-centered data collection and the limitations of relying solely on web-scraped corpora for languages with limited digital footprints.

5.3. Lessons Across Case Studies

Across these case studies, several common themes emerge: (1) community and domain expertise is irreplaceable in corpus curation for low-resource languages; (2) subword tokenization must be adapted to the morphological typology of the target language; (3) cross-lingual transfer from high-resource relatives provides a strong baseline but cannot substitute for native-language pre-training data; and (4) evaluation benchmarks must reflect the specific linguistic properties and use cases of the target community rather than simply translating English benchmarks.

6. Open Problems and Future Directions

Despite significant progress, fundamental challenges remain at the intersection of NLP and LLMs, particularly for linguistically diverse settings. We identify five priority research directions:

Linguistically-Informed Tokenization: Next-generation tokenizers should incorporate morphological analysis to better serve agglutinative and polysynthetic languages, moving beyond frequency-based BPE toward linguistically motivated segmentation.

Efficient Low-Resource Adaptation: Parameter-efficient fine-tuning methods — including LoRA [34], prefix tuning, and adapter layers — offer promising paths for adapting large multilingual models to low-resource languages with minimal data and computation.

Culturally-Aware Alignment: RLHF and instruction tuning methodologies must be adapted to account for cultural variation in language use, pragmatics, and user expectations, requiring diverse human feedback collection beyond English-speaking populations.

Evaluation Ecosystem: There is an urgent need for evaluation benchmarks that cover a substantially broader set of languages, particularly for underrepresented language families in sub-Saharan Africa, the Pacific, and indigenous communities globally.

Interpretability and Linguistic Grounding: As LLMs become de facto NLP engines, interpretability research must maintain rigorous connection to linguistic theory, enabling diagnosis and correction of model failures rooted in genuine linguistic misunderstanding rather than superficial pattern matching.

7. Conclusion

This paper has examined the deep and evolving relationship between Natural Language Processing and Large Language Models. LLMs have absorbed and extended core NLP capabilities — from tokenization to discourse modeling — through scale and self-supervision, while classical NLP frameworks remain essential for evaluation, linguistic grounding, and low-resource development. The challenge of linguistic inclusivity is both an ethical imperative and a scientific frontier: the billions of speakers of low-resource languages deserve NLP tools of comparable quality to those available in English, and building such tools will require novel approaches to data, modeling, and community engagement. We hope this survey serves as a useful reference for researchers and practitioners working to make language technology genuinely universal.

Submitted as a preprint. This work has not undergone peer review. Comments welcome.