Preprint
Review

This version is not peer-reviewed.

Synthetic Data Generation in the Era of Large Language Models: A Comprehensive Survey

Submitted:

14 November 2026

Posted:

17 November 2026

You are already at the latest version

Abstract
Background: The development of Large Language Models (LLMs) has dramatically advanced the capabilities of Natural Language Processing (NLP), enabling breakthroughs in text understanding, generation, and reasoning. However, the success of these models hinges on access to massive, diverse, and high-quality labeled datasets. In many real-world scenarios, such as specialized domains (e.g., healthcare, law), low-resource languages, and privacy-sensitive applications, curating such data remains prohibitively expensive, time-consuming, or infeasible due to regulatory or ethical constraints. Moreover, as LLMs continue to scale in size and complexity, the demand for training and evaluation data grows disproportionately, exacerbating annotation bottlenecks and domain coverage gaps. Synthetic Data Generation (SDG) has emerged as a powerful paradigm to address these challenges by programmatically producing task-specific or domain-adapted examples that complement or replace manually labeled data. Initially grounded in statistical modeling and deep generative architectures, SDG has evolved into a central strategy for improving data efficiency, mitigating distributional shifts, and enabling scalable supervision in modern NLP pipelines, especially in conjunction with LLM-driven prompt-based learning and self-supervised generation. Objectives: This survey provides a comprehensive overview of SDG techniques up to November 2026, with a particular focus on LLM-oriented pipelines. We categorize generative approaches for synthetic text, identify trends in controllability, verification and automation, and highlight open challenges in evaluation, fairness, privacy and scalability. Methods: We review the model families used to generate synthetic text, including probabilistic models, autoencoders and variational autoencoders, Generative Adversarial Networks (GANs), Recurrent Neural Networks (RNNs), transformers, diffusion language models and mixtures of experts, as background for LLM-based pipelines such as instruction synthesis, Retrieval Augmented Generation (RAG), prompt evolution, feedback loops and multi-agent generation, together with toolkits such as PromptSource [1], Self-Instruct [2] and LM-Debugger [3]. The original literature was collected informally; we extended it with a documented search run on 14 November 2026 (49 queries, 78 candidate records, 47 works added) covering 2025–2026 work on reasoning traces and verifiable rewards, model collapse and data quality, agentic and multi-agent pipelines, and evaluation, privacy and multilingual generation. The scope is text and language data, with other data types only where an LLM generates or directs their generation, and every reference was checked against arXiv, Crossref or the publisher. Results: Our synthesis traces an evolution from static, single-model pipelines to modular, verifier-centred and agentic systems in which LLMs plan, generate, critique and check data. Since 2025, synthetic data has become central to teaching models to reason: verifiable rewards move the bottleneck from preference labels to problems with checkable answers, which procedural generators, rephrased web text and self-play now supply. Studies of model collapse show that accumulating real and synthetic data avoids collapse while replacing real data does not, but also that even small synthetic fractions and recursive training can degrade models and their fairness. Synthetic text can leak membership information about its source data, and differentially private generation methods are narrowing the utility gap. Evaluation practice remains fragmented, and multilingual generation increasingly relies on culturally grounded rather than translated data. Conclusions: LLM-driven SDG is rapidly transforming data creation, making it more scalable, controllable and adaptive, but its reliability now depends on verification, provenance and privacy safeguards. This survey offers a taxonomy and a roadmap for building trustworthy synthetic data systems, and it motivates future work on verifiers beyond checkable domains, standardized evaluation of data generators, collapse-aware data practice and formal privacy guarantees.
Keywords: 
;  ;  ;  ;  ;  ;  ;  
Copyright: This open access article is published under a Creative Commons CC BY 4.0 license, which permit the free download, distribution, and reuse, provided that the author and preprint are cited in any reuse.