Survey on Coresets for Deep Learning: Methods and Applications

1. Introduction

In general, a coreset is a compact data summary designed to approximate a large dataset, enabling one to use such a data summary to complete challenging computational tasks with much lower complexities. The first formal definition of coresets dates back to Agarwal et al. [1], where they introduce this concept in computational geometry for solving geometric covering and clustering problems. Over the past 20 years, the research on coresets has expanded tremendously, ranging from classical machine learning to modern deep learning. Figure 1 illustrates the technological evolution of coresets. In classical machine learning, coresets have been applied to optimization problems like clustering [2,3,4,5], logistic regression [6,7], and linear regression [8,9]. More recently, in modern deep learning (especially in the recent 5 years), coresets have found applications in robust optimization [10,11], active learning [12,13], robust training [14,15], continual learning [16,17], and large language models [18,19,20].

Coresets have gained increasing importance in the deep learning era. As shown by the scaling laws [21], a power-law relationship exists between the improvements in model performance, intelligence and the increases in dataset size, model parameters, and computational resources [22,23]. Therefore, training deep learning models often suffers from significant computational cost. For instance, it usually takes several months to train a large language model from scratch. In this context, algorithms that enable effective data compression and efficient data utilization, such as coreset-based methods, are expected to substantially reduce the computational cost [24,25]. Specifically, we consider two major application scenarios for coresets:

1.

Reducing the computational cost, where coresets are used for compressing the original large input data. In particular, we expect that the model trained on the coreset maintains a comparable accuracy to the model trained on the original data.

2.

Improving the data utilization under resource constraints, such as active learning [26] (where labeling effort and domain knowledge are limited) and continual learning [27] (where memory and storage constraints prevent retaining all previously seen data).

To provide some intuition, we present one of the most commonly used formulations for the first scenario, i.e., using a small coreset to compress the input dataset. Let $X={\left\{x_i\right\}}_{i=1}^n$ denote the original input dataset of size n, $Y={\left\{y_i\right\}}_{i=1}^n$ denote the corresponding label vector, and $w={\left\{w_i\right\}}_{i=1}^n$ denote the corresponding positive weight vector. Also, let $\Theta$ denote the parameter space of the model. Suppose $L(\theta ;X,w)={\sum }_{i=1}^n{w}_i𝓁(x_i,y_i;\theta )$ is the weighted empirical risk over X for the model with parameter vector $\theta \in \Theta$ (𝓁 is the per-sample loss). Typically, we assume that $w_i=1$ for all is, then we have the standard empirical risk, which can be simply written as $L(\theta ;X)={\sum }_{i=1}^n𝓁(x_i,y_i;\theta )$. The objective of coreset construction is to identify a small, weighted subset $S={\left\{{\widehat{x}}_i\right\}}_{i=1}^k\subseteq X$, where the size $k\ll n$, and the corresponding positive weight vector $\widehat{w}={\left\{{\widehat{w}}_i\right\}}_{i=1}^k$, serving as a compact proxy for the full dataset. Informally, we hope that the weighted loss on S can approximate the loss on X for all parameters in $\Theta$:

$$\begin{array}{c}\hfill L(\theta ;S,\widehat{w})\approx L(\theta ;X),\forall \theta \in \Theta .\end{array}$$

(1)

This approximation property ensures that solving the target problem on the coreset yields a solution of quality comparable to that obtained on the full dataset. Figure 2 provides a t-SNE visualization of the coreset selection process, demonstrating how a relatively small yet representative subset can preserve the structure of the full dataset.

We should emphasize that as the rapid developments of coresets in multiple areas, their formulations become more and more diverse and it is not realistic to give a unified definition for them. The purpose of introducing (1) is simply to provide an illustrative example. The various task-specific definitions of coresets will be discussed in later sections.

Here, we should also clarify that coreset methods are related to, but distinct from, several other data reduction and selection strategies. First, coresets can be regarded as a specific approach falling under the umbrella of a broader topic called data compression [31]. In this area, another widely studied tool is sketching, which aims to reduce the number of features or dimensionality [32]; the most common sketching methods include “PCA” [33] which uses top principal components to build low dimensional representation, and “Johnson-Lindenstrauss transforms” [34] which projects data onto a random lower-dimensional subspace.

Besides sketching, coreset methods also share certain similarity with two recently proposed data reduction techniques. The technique of dataset distillation [35,36] also serves as a proxy or summarization for the original dataset in many deep learning problems. The major difference between coresets and dataset distillation is that, the former one often relies on selecting a representative subset of actual samples from the original data, but the later technique typically synthesizes a small dataset from the original training dataset which can be potentially large. The second similar technique is data curation [37], which aims at selecting high-quality samples from the input dataset. This process typically involves evaluating each data item using a reference model trained on a reference dataset, which specifies the target distribution to be curated. However, we need to remind the reader that there is no strict boundary between coresets and these two similar techniques in reality. In some cases, a data distillation or curation technique can also be regarded as a generalized coreset method in the stage of data processing.

In this article, we provide a comprehensive review of recent developments in coreset techniques and their applications in deep learning. We are aware of several excellent surveys on coreset methods, but they primarily focus on classical optimization problems [38,39,40,41], including clustering, geometric optimization (such as the smallest enclosing ball), linear and logistic regression, and subspace approximation. To the best of our knowledge, the only survey dedicated to coresets in deep learning is by Guo et al. [42], which mainly focuses on the applications of coresets for image classification tasks.

This survey is organized as follows. Section 2 introduces four categories of commonly used coreset construction techniques for efficient model training. Section 3 explores the applications of coresets in deep learning for enhanced data utilization, with Section 4 specifically addressing their use in pretraining and fine-tuning large language models. Finally, Section 5 discusses several key challenges and promising research directions to advance the study and application of coresets in deep learning.

2. Coresets for Efficient Training

As we know, the Stochastic Gradient Descent (SGD) algorithm and a number of its variants [43,44,45,46] are widely employed for training deep learning models. Despite their effectiveness, training large-scale models still entails substantial computational cost. In this section, we overview existing coreset methods for improving the efficiency of training deep learning models. Based on their underlying construction principles, we group these methods into four categories: greedy selection, importance sampling, filtering, and distribution matching. Each category employs a distinct strategy to identify the most informative training examples as the coreset. It is worth noting that these methods are not mutually exclusive. In practice, they are often combined as a hybrid coreset approach to handle complicated tasks.

2.1. Greedy Selection Methods

A greedy selection method iteratively builds the coreset by adding data points that maximize a predefined utility function at each step. The core of this principle lies in designing an effective utility function that quantifies the marginal benefit of adding a new data item, which is often based on criteria like information gain, diversity, or model improvement [16,47,48,49,50,51,52,53,54]. Below, we introduce the details for this type of coreset construction methods.

Recall that $X={\left\{x_i\right\}}_{i=1}^n$ represents the original dataset. For any subset of X, let $f:2^X\to \mathbb{R}$ be a utility function that quantifies some measure of interest (e.g., coverage, or informativeness). A greedy selection based coreset of cardinality $k\in {\mathbb{Z}}^{+}$ is constructed via a generic iterative procedure, which is described in Algorithm 1. The algorithm is initialized with a set $S_0$, and at each iteration, it appends to the current set the element that yields the largest marginal gain with respect to the utility f. The marginal gain of adding $x\in X\setminus S$ to the current subset S is defined as:

$${\Delta }_f(x\mid S)=f(S\cup x)-f\left(S\right).$$

(2)

The procedure terminates until $\left|S\right|=k$. The resulting set S is then taken as the final coreset. In deep learning, the utility function f used for coreset selection is often deliberately designed to be submodular [48], or to closely approximate a submodular function [55], so that the greedy selection rule in (2) can be applied efficiently with theoretical support.

A set function f, defined over a finite ground set X, is said to be submodular if it satisfies the following property for all subsets $A\subseteq B\subseteq X$ and any element $x\notin B$:

$$f(A\cup x)-f\left(A\right)\ge f(B\cup x)-f\left(B\right).$$

(3)

Inequality (3) expresses the well-known diminishing returns property [56], which indicates that when an element x is added to a larger set B, the resulting gain is smaller than when it is added to a smaller set A. Moreover, when f is monotone (i.e., $f\left(A\right)\le f\left(B\right)$ for all $A\subseteq B$) and normalized (i.e., $f(Ø)=0$), the seminal result of Nemhauser et al. [57] (as shown in Theorem 1) ensures that the greedy algorithm achieves a provable approximation guarantee.

Theorem 1 

([57]). Let $f:2^X\to {\mathbb{R}}^{+}$ be a non-negative monotone submodular function, k be the target set size, and ${\left\{S_i\right\}}_{i\ge 0}$ denote the sequence of subsets constructed by the greedy algorithm (Algorithm 1). Then, for all non-negative integers $l\in {\mathbb{Z}}^{+}$, the following guarantee holds:

$$f\left(S_l\right)\ge \left(1-exp\left(-{\displaystyle \frac{l}{k}}\right)\right)\underset{S:\left|S\right|\le k}{max}f\left(S\right),$$

(4)

and in particular, when $l=k$, we have

$$f\left(S_k\right)\ge \left(1-{\displaystyle \frac{1}{e}}\right)\underset{S:\left|S\right|\le k}{max}f\left(S\right).$$

(5)

Remark 1.

Following Theorem 1, submodular maximization has been extensively studied from both complexity-theoretic and algorithmic perspectives. Exact maximization of a submodular function is NP-hard, and the $(1-1/e)$ approximation ratio achieved by the greedy algorithm (Algorithm 1) is provably optimal unless $\mathrm{P}=\mathrm{NP}$ [58]. Subsequent research has focused on improving computational efficiency through approximation algorithms such as lazy greedy [59] and stochastic greedy [60]. These methods often yield substantial speedups, typically one to two orders of magnitude for lazy greedy and up to three orders of magnitude for stochastic greedy, while retaining the same or nearly the same approximation guarantees.

Algorithm 1 Generic greedy coreset construction

Input: Dataset X, utility function f, target size k 1: $S_0\leftarrow Ø$ 2: for  $i=0,\dots ,k-1$do 3:       $x^{*}={\mathrm{argmax}}_{x\in X\setminus S_i}{\Delta }_f(x\mid S_i)$ 4:       $S_{i+1}\leftarrow S_i\cup x^{*}$ 5: end for 6: return  $S_k$

A representative submodularity-based coreset construction method is gradient matching, which aims to align the gradient computed on the coreset with that of the full dataset [48,61,62]. The rationale behind this idea is that, if the gradient of the loss function over the selected subset can approximate that of the full dataset, then updating the model using the coreset should yield similar optimization dynamics. For example, Mirzasoleiman et al. [48] propose CRAIG (Coresets for Accelerating Incremental Gradient Descent), an algorithm designed to select a weighted subset of training data for accelerating the training of large-scale deep learning models. Specifically, the goal in CRAIG is to find the smallest subset $S\subseteq X$ and corresponding per-element stepsizes ${\gamma }_j>0$ that approximate the full gradient with an error at most $ϵ>0$ for all possible values of the optimization parameters:

$$\underset{S\subseteq X,{\gamma }_j\ge 0\forall j}{min}\left|S\right|\mathrm{s}.\mathrm{t}.\underset{\theta \in \Theta }{max}\parallel \sum _{x_i\in X}{g}_i\left(\theta \right)-\sum _{x_j\in S}{\gamma }_j{g}_j\left(\theta \right)\parallel \le ϵ,$$

(6)

where $g_i\left(\theta \right)$ denotes the gradient contribution of sample $x_i$ at parameter $\theta$, and $\Theta$ is the parameter space. They define the utility function $f\left(S\right)$ as

$$f\left(S\right)\triangleq \underset{\theta \in \Theta }{min}\parallel \sum _{i\in X}{g}_i\left(\theta \right)-\sum _{j\in S}{\gamma }_j{g}_j\left(\theta \right)\parallel ,$$

(7)

which measures the approximation error of the subset S with respect to the full gradient over the entire parameter space. Then, they apply the greedy Algorithm 1 to approximately solve problem (6) by iteratively selecting the element that provides the largest marginal decrease in $f\left(S\right)$, until the approximation error is below $ϵ$.

Besides the aforementioned submodularity-based methods, there also exist several other greedy selection approaches for coreset construction. Killamsetty et al. [49] propose “Grad-Match”, a method that employs the orthogonal matching pursuit method [63] to efficiently identify coresets. Pooladzandi et al. [50] introduce “AdaCore”, a method that leverages the geometry of data distribution to extract coreset from given training examples.

Another representative greedy selection method is loss reduction, where the selection criterion relies on the decrease in training loss incurred when a data point is added to the coreset [51,52,55]. For instance, Killamsetty et al. [51] introduce the method “RETRIEVE”, which is a coreset selection framework designed for efficient and robust semi-supervised learning. The approach formulates the coreset selection as a mixed discrete continuous bilevel optimization problem, where the objective is to select a labeled subset that minimizes the supervised loss. Xu et al. [55] propose a robustness-aware coreset selection (RCS) method to accelerate adversarial contrastive learning. Without relying on label information, RCS selects an informative subset by minimizing the representational divergence between natural data and their virtual adversarial variants. As exhaustively searching all possible subsets is computationally infeasible, the authors reformulate RCS as a surrogate submodular maximization problem. This transformation enables the use of a greedy algorithm, which offers an efficient solution with theoretical optimality guarantee.

2.2. Importance Sampling Methods

Importance sampling [64] is a popular and long-history algorithmic technique in computer science. For example, it is often used for the low-rank approximation problem [65,66,67], where the goal is to approximate a given matrix by a matrix with a lower rank. Such an approximation can be explicitly represented by a small subset of the columns and/or rows of the original matrix. These columns or rows can be found using an importance sampling based approach by first calculating an “influence score” for each one, and then sampling them with probabilities proportional to the scores. The importance sampling idea has also been introduced as an effective approach for coreset construction [3]. First, it assigns an importance score to each data point; then, one can sample the data items according to the probability distribution derived from these scores. In contrast to uniform sampling, which treats all examples equally, this approach prioritizes data points deemed more valuable for the learning task.

Similar to the greedy selection methods introduced in Section 2.1 which rely on a predefined utility function, the key idea of importance sampling is to design a proper scoring function f that assigns an importance score to each data point in the full dataset $X={\left\{x_i\right\}}_{i=1}^n$. This score defines a proposal distribution p from which samples are drawn, with the sampling probability for each $x_i\in X$ given by

$$p\left(x_i\right)={\displaystyle \frac{f\left(x_i\right)}{{\sum }_{j=1}^{n}f\left(x_j\right)}}.$$

(8)

A coreset $S={\left\{{\widehat{x}}_i\right\}}_{i=1}^k$ of size k is then obtained by sampling from X according to p. To correct for the bias introduced by this non-uniform sampling, each sampled point ${\widehat{x}}_i\in S$ is assigned a weight by

$${\widehat{w}}_i={\displaystyle \frac{q\left({\widehat{x}}_i\right)}{p\left({\widehat{x}}_i\right)}},$$

(9)

where q is the probability density in the original dataset X. When X is uniformly distributed, $q\left(x\right)={\displaystyle \frac{1}{n}}$ and the weight is simply set to ${\widehat{w}}_i={\displaystyle \frac{1}{np\left({\widehat{x}}_i\right)}}$. Actually, uniform sampling can be regarded as a special case where $f\left(x_i\right)$ is constant for all i. That is, $p\left(x_i\right)={\displaystyle \frac{1}{n}}$.

The theoretical foundation of this importance sampling based coreset framework was established by Feldman and Langberg [3]. This framework has since been extended to numerous classical machine learning problems, including k-means clustering [4,5,68], logistic regression [8,69], and SVM [8]. Let $ϵ\in (0,1)$ be a prespecified parameter, and we use “$\mathtt{dim}$” to denote the measure to evaluate the complexity of the optimization objective. For example, for a clustering problem, the $\mathtt{dim}$ can be defined as its combinatorial complexity [3]. For a deep neural network, the $\mathtt{dim}$ can be defined as its pseudo-dimension [70,71,72]. The value of $\mathtt{dim}$ is often deeply related with the VC dimension in learning theory [3,70,71,72,73,74,75]. According to the theory introduced in Feldman and Langberg [3], when we construct a weighted coreset S with the size $|S|=\mathtt{Poly}({\displaystyle \frac{1}{ϵ}},\mathtt{dim})$ via the importance sampling, we can guarantee that

$$\begin{array}{c}\hfill L(\theta ;S,\widehat{w})\in (1\pm ϵ)L(\theta ;X),\forall \theta \in \Theta \end{array}$$

(10)

with high probability. For more details, we refer the reader to the surveys [38,39,40,41].

In this article, we mainly focus on the applications of coresets for deep learning models. However, the theory for the aforementioned classical machine learning problems cannot be strictly guaranteed in the context of deep learning, due to its highly complicated optimization objective and non-convex loss landscapes. To shed some light, we can consider a deep neural network with piecewise linear activation functions, such as the ReLU [76]. It has been proven that its VC dimension has an upper bound of $O(rllogr)$ and a lower bound of $\Omega (rllog{\displaystyle \frac{r}{l}})$ [72], where r and l denote the numbers of parameters and layers, respectively. Therefore, the value of $\mathtt{dim}$ of a deep neural network is typically very large. Nevertheless, the idea of importance sampling has still been shown to be effective for improving the training efficiency of deep learning models in practice [77,78,79,80,81,82]. Below, we introduce several examples.

Several methods compute the importance of each data item based on gradients. The method “Importance Sampling Stochastic Gradient Descent (ISSGD)” [77] speeds up the training procedure by reducing gradient variance in SGD. Specifically, ISSGD employs a distributed implementation that utilizes multiple worker machines to compute the gradient norms for disjoint data subsets in parallel, and then takes a master machine to collect these norms; finally, it performs importance sampling based on these gradient norms. Compared with regular SGD, which samples each batch uniformly, ISSGD can minimize training loss more quickly. For example, in the experiments on the SVHN dataset [83], ISSGD achieves convergence in less than 1 hour, compared to about 6 hours by regular SGD [77]. Instead of resorting to distributed implementations, Katharopoulos and Fleuret [84] propose a more efficient importance sampling method through estimating upper bounds for the gradient norms. The experiments show that this approach achieves not only faster convergence but also superior model performance compared to uniform sampling.

The importance sampling idea has also been successfully adapted to accelerate the training of deep generative models, such as the Generative Adversarial Networks (GANs) [85]. The method “GAN with Flow-based Importance Sampling (FIS-GAN)” [78] performs importance sampling by prioritizing important regions in the latent noise distribution of GANs. The latent noise distribution is dynamically updated based on the Jacobian norms of the noise samples fed into the generator. The regions of noise samples with higher Jacobian norms are identified as “harder” regions, and then are assigned higher densities in the updated distribution. When training on the Fashion-MNIST dataset [86], FIS-GAN achieves a Fréchet Inception Distance (FID) score [87] below 3.00 in just 2000 training steps, while training without importance sampling needs 5000 training steps to reach comparable performance.

In addition to the gradient-based metrics, some methods adopt alternative criterion for measuring importance, such as prediction uncertainty [79] or training loss values [80]. Chang et al. [79] propose to prioritize training examples exhibiting high classification uncertainty, which is quantified as the variance of the predicted probability for the correct class across different training iterations. Intuitively, if the prediction for an example fluctuates significantly across iterations (high variance), it likely lies near the model’s decision boundary, thereby exhibiting greater uncertainty; such uncertain examples are more important to refine the decision boundaries. InfoBatch [80] proposes measuring importance using training loss values, which can be obtained without incurring additional computational cost. Training examples with high loss values are typically not well-learned, making them informative for further training. In each training epoch, high-loss examples are retained, whereas low-loss ones are discarded according to a predefined pruning probability. In practice, InfoBatch delivers substantial training acceleration while maintaining competitive model performance. For instance, when training ResNet-50 [88] on the ImageNet [89] using 8 NVIDIA A100 GPUs, InfoBatch achieves 76.5% average accuracy in just 84.0 GPU hours, which is nearly 40% faster than full-data training that achieves 76.4% average accuracy in 140.0 GPU hours.

Another line of research is applying importance sampling to neural network pruning [90,91,92,93], with the aim of compressing models rather than reducing datasets. In this setting, network parameters are treated as a large set of points, and the objective is to find a small subset that approximates the output of the original model. Unlike the methods reviewed in this section, which focus on efficient training, the coreset-based pruning methods target faster inference and broader deployment. Nevertheless, they also contribute to making deep learning more scalable and accessible for real-world applications.

2.3. Filtering Methods

Filtering methods construct coresets by discarding data points with lower importance. Unlike importance sampling introduced in Section 2.2, which introduces stochasticity through probability sampling by (8), the filtering methods are deterministic with hard selection rules. Typically, they set thresholds or compute rankings, to retain only those examples that meet specified standards of informativeness or relevance [94,95,96,97,98,99,100,101,102,103,104,105].

Formally, the coreset S is defined as

$$S=\{x\in X\mid f(x)\ge \tau \},$$

(11)

where f is a scoring function (which could measure the uncertainty, loss, or influence) similar to the one employed by importance sampling methods, and $\tau$ is a predefined threshold. Alternatively, in the top-k selection variant, the coreset consists of the k highest-scoring examples.

Similar to the importance sampling methods, the choice of filtering criterion is also highly task-dependent. Toneva et al. [94] introduce a concept called “forgetting event”, which occurs when a training example, correctly classified in the previous training step, is misclassified in the current step. In their experiments on CIFAR-10, up to 35% of the dataset who have fewer forgetting events can be removed and there is only a less than 0.2% decrease in the final test accuracy. Paul et al. [95] introduce the “Gradient Normed (GraNd)” score to quantify the importance of each individual data point. Theoretically, they show that at any given training step, the contribution of a training example to reducing the loss on any other example is bounded by its loss gradient norm (up to a multiplicative constant). This insight motivates the authors to use the GraNd score as the filtering metric, which is defined as the expected norm of the loss gradient. “Moving-one-Sample-out (MoSo)” [96] measures the importance of a training example by estimating how its removal affects the empirical risk over the remaining dataset. Intuitively, removing a critical example from the training set would significantly increase the empirical risk, while an unimportant one would have negligible impact. Directly calculating this effect is computationally prohibitive, as it requires repeatedly retraining the model with each example removed. To address this, the authors try to approximate the change in empirical risk by computing the expected inner product between the loss gradient of a candidate example and the loss gradient of the remaining dataset. An example receives a high MoSo score if its gradient strongly aligns with the gradient of the remaining dataset, indicating that training on it produces a similar effect to training on the remaining dataset.

“Top-k GAN” [97] employs a filtering strategy for GAN training by leveraging outputs from the discriminator as importance scores. In the experiments, when training on generated images deemed least realistic by the discriminator, the resulting gradient updates tend to push them away from their nearest mode, ultimately degrading model performance and reducing training efficiency. Based on this observation, when updating the generator, Top-k GAN discards such unrealistic samples by zeroing out the gradients from them. Compared to regular GAN training, Top-k GAN mitigates mode dropping and enhances model performance without introducing additional computational overhead, thereby successfully improving the training efficiency.

Beyond those gradient-based and error-based importance metrics, some methods adopt different criterion, such as distance-based [98] or density-based [99] measures. Xia et al. [98] address the challenge of designing a scenario-agnostic scoring criteria that is robust across different task scenarios. They propose “Moderate-DS”, which selects samples with scores closest to the median score, since the median is a proxy for the score distribution in statistics and could be more robust across different scenarios. Specifically, Moderate-DS first computes class centers by averaging the representations of all samples within each class, and then calculates the distance between each point and its corresponding class center. After sorting these distances, Moderate-DS constructs its coreset by selecting the points with distances closest to the median distance.

DeVries et al. [99] accelerate GAN training by selecting data from high-density regions in the data manifold. To identify these regions, all images are first projected into an embedding space of semantical representation. A scoring function (such as the log likelihood under a standard Gaussian, or the distance to the k-th nearest neighbor) is then applied to calculate the density in the neighborhood of each embedded data point. Finally, only the samples with high density scores are retained for training. This sampling strategy yields remarkable training acceleration as demonstrated in their experiments. For instance, when training BigGAN [106] on the ImageNet dataset using 8 NVIDIA V100 GPUs with 16GB of RAM, the method can achieve about 4× acceleration while maintaining comparable generation performance.

Filtering methods offer a direct and easily interpretable mechanism for coreset selection. However, their deterministic nature could be a limitation since a fixed threshold has the risk of excluding underrepresented regions in the data space. A possible solution to remedy this issue could be developing adaptive filtering strategies to balance selectivity with diversity [107].

2.4. Distribution Matching Methods

The idea of distribution matching methods is to construct a coreset whose statistical properties are close to those of the full dataset. By minimizing the discrepancy between these two distributions, this method ensures that the selected subset retains the essential characteristics of the original data [108,109,110,111,112], thereby supporting both model training and generalization.

A central challenge of distribution matching lies in selecting an appropriate distributional distance metric $d(p_X,p_S)$, where $p_X$ and $p_S$ denote the empirical distributions of the full dataset X and the coreset S, respectively. The coreset is then obtained by solving:

$$S=\underset{S\subseteq X,\left|S\right|=k}{\mathrm{argmin}}d(p_X,p_S).$$

(12)

A common distance metric is the Kullback-Leibler (KL) divergence:

$$d_{\mathrm{KL}}(p_S\parallel p_X)=\int p_S\left(x\right)log{\displaystyle \frac{p_S\left(x\right)}{p_X\left(x\right)}}dx.$$

(13)

“Bayesian Pseudo-coresets Exemplar VAE (ByPE-VAE)" [110] employs a distribution matching method based on the KL-divergence (13) to speed up the training of Exemplar VAE [113]. Exemplar VAE is a variant of Variational Autoencoders (VAEs) [114] with a mixture of exemplar-based priors on the latent space. Specifically, a set of exemplars is chosen from the training set, and an exemplar-based prior is learned for each exemplar. The size of the exemplar set could be potentially as large as the training set, which makes Exemplar VAE computationally expensive to train. To address this limitation, ByPE-VAE employs Bayesian pseudo-coresets [115] to create a small-scale pseudo-coreset of pseudo-data points, which is constructed by minimizing the KL-divergence (13) between the posterior distributions induced by the pseudo-coresets and the true data points. By using these pseudo-data points as exemplars, ByPE-VAE achieves significant training accelerations without sacrificing model performance. For instance, on Fashion-MNIST and CIFAR-10 using a single NVIDIA 1080Ti GPU, ByPE-VAE delivers approximately 3× faster training compared to standard Exemplar VAE while maintaining comparable performance.

Besides the approaches that try to directly minimize a distributional distance metric, there also exist several “indirect” distribution matching methods. They usually employ other techniques to implicitly minimize the discrepancy between the coreset and the input distribution. One common indirect approach is clustering-based methods. These methods use clustering techniques, such as k-center clustering [116], to try to minimize the discrepancy by selecting the most representative data items [111,112]. In each iteration of GAN training, “Small-GAN” [111] tries to increase the effective batch size by formulating it as a k-center problem, with the Euclidean distance as the distance metric. Coresets are constructed by first randomly sampling a larger batch, and then using the Gonzalez’s algorithm [116] to select a smaller batch from the larger one. “Fréchet Descriptor Distance based Coreset (FDD-Coreset)” [112] also formulates coreset selection as a k-center problem, but employs the Fréchet distance [117] as the distance metric.

2.5. Summary

We have introduced four categories of coreset construction methods for efficient model training. Submodularity-based greedy selection methods enjoy strong theoretical guarantees. They can produce quality guaranteed coresets that provide a $(1-1/e)$-approximation to the optimal solution, as shown in Theorem 1. Filtering methods strictly discard low-score examples and focus exclusively on challenging or informative ones. As a result, coresets constructed by them tend to lack diversity. In contrast, importance sampling methods naturally introduce diversity since they select probabilistically rather than deterministically. However, they may select less informative examples from unimportant regions in the data distribution. Another advantage of importance sampling is that it can provide an unbiased estimator of the loss on the full dataset through reweighting by (9). Distribution matching methods can select representative examples since they aim to preserve the overall data distribution. However, similar to importance sampling methods, they may select examples that are easy or already well learned.

It is also worth noting their computational complexities. Greedy selection methods, such as the generic greedy coreset construction in Algorithm 1, incur a computational complexity of $O(nk·T_f)$, where n is the dataset size, k is the pre-specified coreset size, and $T_f$ is the cost for evaluating the marginal gain of the utility function f. As mentioned in Remark 1, several approximation algorithms have been proposed to reduce the complexities [59,60]. The computational complexities of both importance sampling and filtering methods are dominated by the cost of computing the selection scores. Consider, for example, the filtering criterion “Error L2-Norm (EL2N)" [95], which is a practical approximation to the GraNd score (as introduced in Section 2.3). EL2N measures the expected $l_2$-norm of the difference between the predicted class-probability vector and the one-hot ground-truth label. To estimate this score, the neural network needs to be trained from $R\in {\mathbb{Z}}^{+}$ independent random initializations, each for $E\in {\mathbb{Z}}^{+}$ epochs over all n training examples. The training phase requires $O\left(nrRE\right)$ time, where r represents the number of model parameters. After training, computing the EL2N scores only requires forward passes to obtain the predicted probability vectors, followed by evaluation of their distances to the true labels and averaging over the R runs. This stage costs $O\left(nR\right(r+C\left)\right)$, where C is the number of classes. Finally, selecting the top-k examples according to EL2N can be done in $O\left(n\right)$ time. Hence, the overall time complexity is $O(nrRE+nRC)$. For importance sampling methods, once the scores have already been computed, the sampling step also only takes linear time. For distribution matching methods, the main bottleneck lies in repeated distance computations. An example is the Gonzalez’s algorithm [116], whose time complexity is $O\left(nkd\right)$ where d is the feature dimension and k is the number of selected points, since each of the k iterations requires scanning all n points and updating distances in d-dimensional space.

In addition to the coresets for efficient training introduced in this section, these four categories of methods are also applied to other scenarios, including coresets for enhancing data utilization introduced in Section 3 and for large language models in Section 4. For example, the “SAS” [118] and the “SimCore” [119] methods for semi/self-supervised learning introduced in Section 3.3.3, and the “CoLM” method [20] for large language models in Section 4.2, all formulate coreset selection as a submodular optimization problem. Xie et al. [120] use the idea of importance sampling to select data for pretraining large language models, as introduced in Section 4.1. While many active learning algorithms introduced in Section 3.1 act as filtering methods since they assign scores to examples and select those with the highest scores [121,122,123,124,125,126], some algorithms adopt alternative strategies such as importance sampling or distribution matching. For instance, the “DACS” method [127] introduced in Section 3.1.3 selects more samples from the sparse regions in the data distribution and less from the dense regions, which is similar to the idea of importance sampling. And the “WAAL” method [128] in Section 3.1.3 explicitly formulates active learning as a distribution matching problem. Please see the corresponding sections for more details. Overall, these cases demonstrate that the four categories of methods introduced in this section have broad applicability across different coreset selection scenarios.

3. Coresets for Enhancing Data Utilization with Limited Resource

In this section, we survey the coreset selection techniques developed for enhancing data efficiency. In particular, we review the methods for three typical scenarios with limited resource: active learning, continual learning and semi/self-supervised learning. In active learning, coreset selection is used to identify the most informative unlabeled data points for annotation, so as to avoid spending too much human labeling effort [26]. In continual learning, a central challenge is the “catastrophic forgetting”, where the model could lose knowledge from previously learned tasks when trained on new ones [129,130]. A common approach for addressing this problem is to store a small subset (i.e., the coreset) of past data in a replay buffer [131,132]. Self-supervised learning [133] and semi-supervised learning [134] consider the scenario that we are given a large pool of unlabeled data, and there is no labeled data or the amount of labeled data is quite limited. For semi/self-supervised learning, a key difference with the aforementioned active learning is that we are not allowed to draw support from human effort to add labels.

3.1. Active Learning

In this section, we focus on pool-based active learning, which is the most widely adopted type of active learning in deep learning. Pool-based active learning assumes that there is an initial small labeled dataset and a large pool of unlabeled data available for sampling. It iteratively performs the following four steps: (1) training the target model on the labeled dataset, (2) selecting important unlabeled examples based on the trained model, (3) having the selected examples labeled by annotators, and (4) adding the newly labeled data into the labeled dataset, as shown in Figure 3. Typically, a labeling budget is imposed, which is much smaller than the size of the unlabeled dataset. In this procedure, only a small number of important examples are labeled within the budget, and it is natural to consider it from the perspective of coreset construction. For instance, Sener and Savarese [13] have demonstrated that the problem of active learning for convolutional neural networks can be formulated as a coreset selection problem.

Next, we treat the data selection strategies of active learning algorithms as coreset selection methods and provide a detailed review of them. Based on the criteria for evaluating the importance of unlabeled examples, we categorize active learning approaches into three types: loss-based, coverage-based, and hybrid methods. A summary of several representative methods from each category is presented in Table 1.

Note that our goal is not to provide an exhaustive survey of active learning, but to shed light on the key strategies for assessing and identifying important unlabeled examples from the perspective of coresets construction. For more comprehensive introductions on active learning, we refer the reader to the surveys [26,145,146,147].

3.1.1. Loss-Based Methods

Loss-based methods prioritize unlabeled examples that are expected to yield high training loss if added to the training set. Such examples typically include those with high predicted training loss [121,148] or high prediction uncertainty [122,123,124,135,136,149,150,151,152,153,154,155,156]. By concentrating on the examples where the model performs poorly or exhibits high uncertainty, these methods try to improve the model by training it on the most challenging or informative examples. Instead of focusing on training loss, some other loss-based approaches consider generalization performance, prioritizing unlabeled examples that are predicted to maximally reduce the test-time error [125,126,157,158]. Next, we introduce these methods in more detail.

To identify unlabeled examples that are likely to produce high training loss, Yoo and Kweon [121] design a loss prediction model to estimate the loss for unlabeled examples and select those with the highest predicted losses for annotation. Instead of relying on an auxiliary loss prediction model, Huang et al. [148] demonstrate that the training loss for an unlabeled example can be estimated via the difference between two consecutive active learning iterations.

Rather than directly estimating training loss, many methods select unlabeled examples for which the model exhibits the highest uncertainty. These uncertain examples usually result in high training loss because the model lacks sufficient knowledge to predict their labels confidently. In image classification tasks, common uncertainty measures include least confidence, margin, and entropy. Given an unlabeled dataset X, least confidence based methods select the example with the lowest confidence in its top predicted class:

$$x^{*}=\underset{x\in X}{\mathrm{argmax}}1-P_{\theta }\left(\widehat{y}\right|x),$$

(14)

where $P_{\theta }\left(y\right|x)$ is the conditional probability of class y given input x, under the model parameterized by $\theta$, and $\widehat{y}={\mathrm{argmax}}_y{P}_{\theta }\left(y\right|x)$. One limitation of least confidence is that it only considers the class with the highest probability, ignoring the probability distribution over other classes. To alleviate this problem, margin based methods take into account the probabilities of the top two most probable classes:

$$x^{*}=\underset{x\in X}{\mathrm{argmin}}{P}_{\theta }\left({\widehat{y}}_1\right|x)-P_{\theta }\left({\widehat{y}}_2\right|x),$$

(15)

where ${\widehat{y}}_1={\mathrm{argmax}}_y{P}_{\theta }\left(y\right|x)$ and ${\widehat{y}}_2={\mathrm{argmax}}_{y\ne {\widehat{y}}_1}$$P_{\theta }\left(y\right|x)$. Entropy is a more general measure that leverages the information from all classes:

$$x^{*}=\underset{x\in X}{\mathrm{argmax}}-\sum _i{P}_{\theta }\left(y_i\right|x)log{P}_{\theta }\left(y_i\right|x).$$

(16)

The method “Cost-Effective Active Learning (CEAL)” [135] uses those three uncertainy measures (defined in (14), (15), and (16), respectively) to select unlabeled examples for annotation, and incorporates low-entropy unlabeled examples into the training set by assigning pseudo-labels to them. Ranganathan et al. [149] not only employ the entropy (16) to select unlabeled examples but also integrate it into the training loss of the target model. The method “Adversarial Sampling for Active Learning (ASAL)” [122] employs the entropy (16) in a different way by optimizing a generator to produce a synthetic example that maximizes classification entropy. Then, the unlabeled example closest to the synthetic example is selected and annotated. “Look-Ahead Data Acquisition (LADA)” [151] integrates data augmentation into active learning by learning an augmentation policy through maximizing the classification entropy (16). “Training Dynamics for Active Learning (TiDAL)” [153] computes the margin (15) and entropy (16) based on the training dynamics of unlabeled examples, which is defined as their averaged classification results across training iterations. An et al. [154] propose to consider the reduction of classification entropy (16) for the remaining unlabeled examples when a specific unlabeled example is labeled. The entropy and its reduction are combined through a convex combination, which is then used to select the most informative unlabeled examples for annotation.

An alternative measure of uncertainty for an unlabeled example is its distance to the decision boundary of the model. Examples closer to the decision boundary exhibit higher uncertainty, since they lie in regions where even small perturbations could lead to a change in their predicted class. Since calculating the exact distance is intractable, the approach “DeepFool Active Learning (DFAL)” [123] tries to approximate the distance by determining the smallest perturbation applied to an unlabeled example that is needed to reclassify it into a different category. Gao et al. [136] prioritize unlabeled examples with inconsistent predictions under data augmentations. “Active Learning by FeAture Mixing (ALFA-Mix)” [155] selects unlabeled examples exhibiting prediction inconsistency when their feature embeddings are interpolated with classwise average features of labeled examples. The method “Multi-classifier Adversarial Optimization for Active Learning (MAOAL)” [124] employs adversarial training to simultaneously train a feature generator and multiple classifiers, and selects unlabeled examples with the highest classification discrepancy across the classifiers for annotation.

While most approaches rely on training loss for identifying important unlabeled examples, there are also some methods that select examples based on their predicted contribution to reducing the generalization error of the target model. “Batch Active learning via Information maTrices (BAIT)” [157] selects unlabeled examples by minimizing the expected log-likelihood error on the whole unlabeled dataset through optimizing an objective involving the Fisher information matrix (which is the Hessian of the negative log-likelihood function) [159]. Wang et al. [125] illustrate that selecting unlabeled data with the highest gradient norm can maximally boost the testing performance of the target model. The method “dynamicAL” [158] leverages the training dynamics of unlabeled examples, which is defined as the derivative of the training loss with respect to the training iteration, to select valuable examples. “Sharpness-Aware Minimization (SAM)” [160] demonstrates that the generalization error is upper bounded by the maximal training loss sharpness in addition to a parameter regularization term. Therefore, SAM trains models by simultaneously minimizing the loss value and loss sharpness. Inspired by this, another method “Sharpness-Aware Active Learning (SAAL)” [126] estimates the loss sharpness for unlabeled examples by assigning pseudo-labels and selects those with the highest estimated sharpness for annotation.

3.1.2. Coverage-Based Methods

Coverage-based methods aim to select unlabeled examples that best represent the underlying data distribution. This is often achieved by prioritizing examples that either maximize coverage of the data space [139,140,161,162] or enhance diversity of the selected set [13,137,138,163,164,165,166,167].

Coverage maximization methods. These methods select unlabeled examples for covering different regions of the data space, where the goal is to capture a broad range of the overall data distribution. “Probability Coverage (ProbCover)” [139] formulates it as a Max Probability Cover problem. Based on this formulation, ProbCover selects unlabeled examples maximizing the probability of the union of balls centered at each selected point with a predefined radius. “MaxHerding” [140] generalizes ProbCover by introducing a notion of generalized coverage, which is defined as the expected maximum similarity between the labeled set and the overall data distribution. Conceptually, it encourages to select the unlabeled examples that maximize their similarity to the entire unlabeled dataset. One limitation of MaxHerding is that it exclusively focuses on coverage but neglects uncertainty. To remedy this issue, the recent method “Uncertainty Herding (UHerding)” [161] extends the generalized coverage concept by integrating it with an uncertainty measure. More hybrid methods that simultaneously consider coverage and uncertainty are introduced in Section 3.1.3.

Diversity maximization methods. Rather than directly optimizing for coverage, many methods seek to select unlabeled examples that are distinct from each other. Intuitively, a diverse set of examples can reduce information redundancy, ensuring that the target model encounters a broader range of features or patterns during training. “Farthest First Active learning (FF-Active)” [163] selects the next point as the one farthest from the already selected points in the feature space. Sener and Savarese [13] formulate active learning for convolutional neural networks as a k-center clustering problem in feature space, and use the Gonzalez’s algorithm [116] to greedily select an initial subset of data points. Then, a mixed integer program is employed to further refine the subset. The method “Contextual Diversity based Active Learning using Core-Sets (CDAL-CS)” [164] also uses the Gonzalez’s algorithm, but employs a notion of “pairwise contextual diversity” as the distance metric (rather than the distance induced in feature space).

Instead of relying on explicit distance metrics to select diverse examples, several active learning methods employ auxiliary classifiers [137] or adversarial training [138,165] to implicitly quantify their dissimilarities. “Discriminative Active Learning (DAL)” [137] trains a binary classifier to differentiate between unlabeled and labeled examples. The unlabeled examples classified as unlabeled with the highest confidence are then selected. Another approach “Variational Adversarial Active Learning (VAAL)” [138] identifies important unlabeled examples through an adversarial game between a VAE and a discriminator. The VAE is trained to deceive the discriminator into classifying both labeled and unlabeled examples as labeled, while the discriminator is trained to differentiate between them. After this training procedure, unlabeled examples classified as unlabeled with the highest confidence by the discriminator are selected. One limitation of VAAL is its task-agnostic nature, which overlooks task-specific information. To address this limitation, an improved approach “Task-Aware Variational Adversarial Active Learning (TA-VAAL)” [165] integrates VAAL with a task-aware method which trains a ranking model to prioritize examples by their loss rankings.

Recall that in Section 2.1, we introduced the definition of a utility function $f:2^X\to \mathbb{R}$ defined over a given dataset X, with the property of submodularity. A submodular utility function f can be used for measuring similarity due to the diminishing returns property (3). This property can be interpreted from the perspective of information gain by viewing $f\left(S\right)$ as the amount of information contained in S. From this perspective, the property (3) states that the information gained by adding a new element to a set decreases as the set grows. This is because a larger set is more likely to already contain information that is similar to that of the new element. Therefore, maximizing a proper submodular utility function naturally encourages the selection of diverse and representative examples and discourages redundant ones. “Submodular Information Measures based actIve LeARning (SIMILAR)” [166] employs submodular information measures [168,169] to deal with the bias issue in real scenarios, including datasets with imbalanced classes, out-of-distribution data points, or high redundancy. For example, the submodular mutual information (SMI) is used to select unlabeled examples when datasets contain rare classes [166]. Given an unlabeled dataset X, a set R consisting of a small number of examples from the rare classes, and a submodular function f that measures the diversity of an input set, a set $S\subseteq X$ can be constructed by maximizing the SMI, which is defined as:

$$I_f(S;R)=f\left(S\right)+f\left(R\right)-f(S\cup R).$$

(17)

Since R is a fixed set, $f\left(R\right)$ is also fixed. Therefore, maximizing (17) is equivalent to selecting a set S that is not only diverse itself (high $f\left(S\right)$) but also similar to R (low $f(S\cup R)$). In other words, it encourages to select a diverse set that contains examples similar to the known rare ones.

3.1.3. Hybrid Methods

There also exist numerous “hybrid” methods that jointly optimize uncertainty and diversity, aiming at identifying examples that are both informative and representative [127,128,141,142,143,144,161,170,171,172,173,174,175,176]. That is, the joint optimization enables an effective trade-off between exploitation (refining the model in uncertain regions) and exploration (probing regions with unseen patterns).

Yin et al. [170] take a two-step approach: first a set of points is selected based on the classification entropy (16) and dissimilarity, then a set of additional points are selected based solely on dissimilarity. Similar with the coreset idea introduced in Section 2.4, “Wasserstein Adversarial Active Learning (WAAL)” [128] formulates the active learning problem as a distribution matching problem between the underlying data distribution and the distribution of queried data.

Clustering algorithms play an important role in active learning as they are often used to simultaneously optimize both uncertainty and diversity. The approach “Batch Active learning by Diverse Gradient Embeddings (BADGE)” [141] runs k-means++ [177] to select unlabeled examples, where the distance metric is based on their gradient embeddings computed from pseudo-labels and last-layer network parameters. “Cluster-Margin” [143] performs hierarchical agglomerative clustering on unlabeled examples and focuses on the clusters containing examples with low margin scores (15). These clusters are sorted in ascending order according to their size. Then, Cluster-Margin samples from the sorted clusters in a round-robin scheme until the labeling budget is exhausted. “Density-Aware Core-Set (DACS)” [127] shows that examples from lower-density regions exhibit higher prediction entropy (16) and loss, suggesting that unlabeled examples from sparse regions are more informative. Therefore, DACS clusters unlabeled examples by their estimated densities and samples from each cluster inversely proportional to its size (which is similar to the idea of importance sampling introduced in Section 2.2). “Least Disagree Metric based Sampling (LDM-S)” [144] introduces Least Disagree Metric (LDM) to quantify the distance of an example to the decision boundary. Then, it incorporates a modified version of k-means++ to promote diversity, ensuring that the selected examples are both sufficiently informative and representative.

3.1.4. Experimental Comparisons

The performance of active learning algorithms is typically evaluated by examining test accuracy as a function of coreset size, that is, the number of labeled training samples acquired during the active learning process. This evaluation metric captures how efficiently a method improves predictive performance as additional labeling budget is allocated. In image classification, test accuracy is defined as the proportion of correctly classified instances among all examples in the test set. Widely used benchmarks for image classification in active learning include MNIST [178], SVHN [83], CIFAR-10 [29] and CIFAR-100 [29]. These datasets vary significantly in difficulty and complexity. MNIST is a relatively simple dataset composed of grayscale handwritten digits, whereas SVHN introduces more realistic variability through colored street-view digit images. CIFAR-10 represents a further increase in difficulty, containing natural images from ten categories with greater intra-class variation. CIFAR-100 is more challenging as it involves one hundred classes with finer-grained distinctions and increased visual diversity.

Performance trends across benchmarks of varying complexities reveal important differences between active learning algorithms. On relatively simple datasets like MNIST and SVHN, loss-based methods usually outperform coverage-based approaches [141,142,143,144,166,173]. This is likely because these datasets exhibit lower intrinsic variability, so selecting informative samples near the decision boundary is often sufficient. As dataset complexity increases, however, the relative advantage of loss-based methods tends to diminish. On more difficult benchmarks such as CIFAR-10 and CIFAR-100, coverage-based methods become increasingly competitive [141,142,143,144,153,154,173]. This suggests that, in richer and more complex data distributions, ensuring diversity and adequate coverage of the data distribution becomes more critical. Finally, hybrid methods, which combine loss-based and coverage-based criteria, are frequently among the best-performing approaches across datasets [141,143,144,155,161,173]. Their strong empirical performance highlights the importance of balancing uncertainty and diversity during sample selection. Overall, these comparisons underscore that the relative effectiveness of active learning strategies is closely tied to dataset complexity. Simpler datasets tend to favor methods that emphasize uncertainty, while more complex datasets increasingly benefit from strategies that also account for diversity and coverage. Consequently, hybrid approaches often provide the most reliable performance across benchmarks. This makes them a particularly attractive choice when the characteristics of the dataset are not known in advance.

3.2. Continual Learning

Continual learning [27,179] is a learning paradigm in which artificial neural networks are trained on a sequence of tasks, aiming to acquire new knowledge while preserving performance on previously learned tasks. This paradigm faces a fundamental challenge known as catastrophic forgetting [129,130], where adapting to new tasks leads to significant degradation in performance on earlier ones. Replay-based methods [131,132,180] have been widely studied for mitigating catastrophic forgetting. These methods typically maintain a small buffer that contains a subset of past training samples (see Figure 4 for an illustration). Given the limited storage capacity, the key challenge lies in how to construct the memory buffer. It is natural to consider this problem from the perspective of coreset selection, which is to identify the most representative samples (similar with active learning in Section 3.1). Formally, replay-based methods operate under the constraint of a fixed-size memory buffer M. For each $t\ge 1$, let $X_t$ be the data of the t-th task, and the goal is to select a coreset $S_t\subseteq X_t$ that is able to approximate $X_t$. The selected coreset $S_t$ is stored in the memory buffer M. When training on a new task, the parameters $\theta$ of the model are updated by optimizing a composite objective function [181], which incorporates both the current task data and the stored coresets:

$$L\left(\theta \right)=L(\theta ;X_{t+1})+\lambda L(\theta ;M),$$

(18)

where $\lambda >0$ controls the rehearsal strength. In this section, we categorize the coreset selection methods in continual learning into two major types: coverage-based methods and gradient-based methods (see Table 2 for an illustration).

3.2.1. Coverage-Based Methods

The methods in the first category usually maintain predefined selection mechanisms and do not adapt dynamically during training procedure (in contrast to the gradient-based methods discussed in Section 3.2.2). Representative methods include reservoir sampling [131,182,183], ring buffer [184,185,186], and k-means clustering [187].

Reservoir sampling [190] is a foundational streaming algorithm for maintaining fixed-size uniform samples from a data stream with unknown cardinality. The canonical implementation initializes a reservoir M with the first k observed samples. For the i-th incoming sample ($i>k$), the algorithm retains it with probability $P_i=k/i$, replacing a uniformly random element in M if selected. This probabilistic replacement rule ensures that every observed sample has an equal probability of $k/n$ to be selected, where n denotes the total number of elements in the stream. The reservoir sampling algorithm achieves $O\left(n\right)$ time complexity and $O\left(k\right)$ space complexity, making it asymptotically optimal for single-pass scenarios. Due to these properties, it is particularly suitable for the applications involving large-scale data streams.

The integration of reservoir sampling into continual learning originated with Riemer et al. [182], who pioneered its use for task-agnostic memory buffering. Their method investigates the continual learning problem from the balance between learning plasticity and memory stability, which is achieved through gradient alignment, i.e., by encouraging gradients of different tasks to point in similar directions so that learning on one task facilitates rather than interferes with others. They introduce the method “Meta Experience Replay (MER)”, which integrates experience replay with meta-learning to enhance knowledge transfer while mitigating interference.

In some continual learning scenarios, we often encounter multi-label datasets that exhibit long-tailed distributions, where a few dominant classes are overrepresented while many minority classes have limited samples. To address this challenge, Kim et al. [191] propose a sampling strategy termed “Partitioning Reservoir Sampling (PRS)”, which is designed to allocate a proportional share of the memory buffer to moderate and minority classes, thereby preserving a more balanced representation for both current and past experiences. Moreover, Chrysakis and Moens [192] introduce a strategy called “Class-Balancing Reservoir Sampling (CBRS)” to mitigate the imbalance issue.

In addition to employing reservoir sampling for storing exemplars, recent works [131,183] propose to retain some intermediate products of training, e.g., the logits. Specifically, Buzzega et al. [131] introduce “Dark Experience Replay (DER)”, which leverages reservoir sampling to match the current logits with those stored from past experiences so that the model remains consistent with previously learned knowledge. However, Boschini et al. [183] demonstrate that DER may suffer from several limitations. For instance, it can introduce classification bias and overlook important semantic relationships between previously learned and newly introduced classes. To overcome these drawbacks, they propose eXtended-DER (X-DER), a novel continual learning method that embraces memory update and future preparation. In particular, they first introduce a future preparation strategy, which exploits past and present data to prepare future classification heads so that they can better encode meaningful information. Then, they propose a memory update procedure that keeps the memory buffer up to date by inserting secondary information from the present into memories of the past.

The ring buffer strategy was first introduced by Lopez-Paz and Ranzato [184] and subsequently adopted in later studies on continual learning [185,186]. Broadly speaking, the ring buffer maintains a fixed-size memory that stores a subset of past samples and updates this set in a first-in-first-out (FIFO) manner as new data arrive. In this context, the strategy is applied to efficiently manage and refresh stored examples during training, which allows the model to rehearse previous knowledge while keeping memory usage constant. Specifically, given C total classes and a memory buffer M, each class $c\in \{1,2,\dots ,C\}$ is assigned a buffer $M^c$ with a quota of

$$|M^c|=⌊{\displaystyle \frac{\left|M\right|}{C}}⌋$$

under a FIFO replacement policy. Formally, for each incoming batch B of data points, let $B^c=\{x\in B\mid y=c\}$ denote the set of points in B belonging to class c, where y is the label of x. Also, let $m^c$ denote the number of currently stored points in the buffer $M^c$. Then, each $M^c$ is updated as follows:

$$M^c=\left\{\begin{array}{cc}\mathrm{FIFO}\left(M^c\cup B^c\right)\hfill & \mathrm{if}|M^c|-m^c<\left|B^c\right|,\hfill \\ M^c\cup B^c\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

This deterministic allocation guarantees equal buffer capacity across classes, ensuring a balanced representation of all classes in memory.

The k-means algorithm, which is adopted by Chaudhry et al. [187], provides a geometric framework for coreset construction in continual learning. For each class $c\in \{1,\dots ,C\}$, the method performs an online variant of the Lloyd’s algorithm (a popular implementation for k-means [193]) to iteratively estimate $k^c=\lfloor \left|M\right|/C\rfloor$ centroids ${\left\{{\mu }_i^c\right\}}_{i=1}^{k^c}$ in the latent feature space (preceding the classification layer of the model). This strategy shares similar advantages and limitations with the ring buffer approach, but yields improved coverage of the constructed coreset in feature space.

3.2.2. Gradient-Based Methods

The second category of approaches for continual learning is gradient-based, which leverages the information from gradients to guide coreset selection. A pioneering work in this direction is “Gradient-based Sample Selection (GSS)” by Aljundi et al. [132], who formulate the sample selection problem as the following optimization problem:

$${\theta }_t=\underset{\theta }{\mathrm{argmin}}L(\theta ;X_t)\mathrm{s}.\mathrm{t}.L(\theta ;X_i)\le L({\theta }_{t-1};X_i);\forall i\in [0\dots t-1],$$

(19)

where ${\theta }_{t-1}$ is the model trained on task $t-1$, t denotes the index of the current task, and i ranges over the previously seen tasks. The objective is to minimize the loss on the current examples without increasing the losses on the previously learned examples, which effectively constrains parameter updates to preserve performance on past tasks and thereby mitigates catastrophic forgetting. Meanwhile, the authors demonstrate that this selection process is equivalent to maximizing sample diversity in the replay buffer and propose a lightweight greedy algorithm to solve it. However, this method does not consider the imbalance and noisy issues which are common in real-world data streams.

To address these issues, Yoon et al. [188] propose “Online Coreset Selection (OCS)”, a simple yet effective method which selects the most representative and informative samples at each iteration, and trains the model on them in an online fashion. Specifically, they introduce three gradient-based selection criterion for coresets: (1) minibatch similarity, which selects the samples that are most representative of the current task distribution; (2) sample diversity, which encourages selecting the samples with minimal redundancy to improve coverage of the input space; (3) coreset affinity, which minimizes interference between selected samples and previously acquired knowledge, thereby promoting the stability across different tasks.

The gradient-based idea is also adopted by several bilevel formulations for coreset selection. For instance, Borsos et al. [16] introduce a nested cardinality-constrained optimization model via the bilevel framework. Recall that we use w to denote the weight vector of the given dataset $X={\left\{x_i\right\}}_{i=1}^n$, $L(\theta ;X,w)={\sum }_{i=1}^n{w}_i𝓁(x_i,y_i;\theta )$ the weighted empirical risk over X for a model with the parameter vector $\theta \in \Theta$, and $L(\theta ;X)$ the case where $w_i=1$ for all i. Then, the bilevel formulation is

$$\begin{array}{c}\hfill \widehat{w}\in \underset{\begin{array}{c}w\in {\mathbb{R}}_{+}^n,{\parallel w\parallel }_0\le m\end{array}}{\mathrm{argmin}}L({\theta }^{*}\left(w\right);X)\mathrm{s}.\mathrm{t}.{\theta }^{*}\left(w\right)\in \underset{\theta \in \Theta }{\mathrm{argmin}}L(\theta ;X,w).\end{array}$$

(20)

The solution of (20) can be obtained through the greedy forward selection method “Matching Pursuit (MP)” [194]. Hao et al. [53] introduce another bilevel formulation which seeks to learn a probability distribution over a low-dimensional manifold by incorporating a smoothed top-K loss as the regularizer. Another method closely related to bilevel optimization is “Gradient Coreset Replay (GCR)” [181]. In this approach, a coreset is constructed and maintained to approximate the aggregate gradient over all previously observed data items.

There also exist several other gradient-based approaches that focus on diversity and uncertainty. For example, Bang et al. [189] highlight the importance of maintaining diversity within the episodic memory, since a memory buffer dominated by redundant or highly similar samples fails to provide sufficient coverage of the data distribution, which in turn limits the model’s ability to retain previously learned knowledge. Then they propose “Rainbow Memory (RM)”, a strategy to enhance diversity in continual learning.

3.2.3. Experimental Comparisons

In this section, we review the datasets and evaluation metrics used in coreset-based continual learning, and summarize the performance of prior approaches on these benchmarks to provide a more comprehensive overview. In general, the performance of continual learning can be evaluated from two aspects [27]: (i) overall performance on all tasks learned so far, typically measured by average accuracy (AA) [184,195]; and (ii) the degree of forgetting on previously learned tasks, typically quantified by forgetting measure (FM) [195].

We evaluate coreset-based continual learning methods on three commonly used benchmarks. Multiple Datasets [188] consists of five distinct datasets: MNIST [178], Fashion-MNIST [86], NotMNIST [196], Traffic Sign [197], and SVHN [83]. It contains five tasks, each formed by randomly sampling 1000 training instances from a single dataset. Split CIFAR-100 [54] is based on the original CIFAR-100 [29] and is split into 20 tasks, each consisting of five disjoint classes. Split Tiny-ImageNet [198] contains 200 classes of colored images resized to $64\times 64$, with 500 training images, 50 validation images, and 50 test images per class. The dataset is divided into 20 tasks, each comprising 10 disjoint classes. Together, these benchmarks cover a diverse range of continual learning settings, including heterogeneous datasets, fine-grained class-incremental scenarios, and large-scale image classification tasks.

Across these benchmarks, the two categories of coreset selection methods (i.e., coverage-based methods [131,182,183,184,185,186,187] and gradient-based methods [16,53,132,181,188]) exhibit distinct empirical behaviors. Specifically, coverage-based methods provide competitive baselines and achieve moderate mitigation of forgetting; however, their static and gradient-agnostic selection strategy limits their ability to maximize overall accuracy. In contrast, gradient-based methods consistently yield superior AA across all benchmarks, with BCSR [53] achieving the best overall performance (59.89, 61.60, and 44.13 on Multiple Datasets, Split CIFAR-100, and Split Tiny-ImageNet, respectively). Meanwhile, the methods such as OCS [188] and SPR [199] focus on mitigating forgetting, attaining the lowest FM on Multiple Datasets (0.062) and Split Tiny-ImageNet (0.102), respectively, while exhibiting a slight drop in average accuracy. Overall, the results suggest that different methods exhibit distinct strengths across different benchmarks. Some approaches achieve higher overall accuracy, while others are more effective at mitigating forgetting. Therefore, improving the trade-off between performance and memory stability remains an important direction for future research.

3.3. Semi/Self-Supervised Learning

In this section, we introduce the coreset methods for two important scenarios of deep learning, self-supervised learning and semi-supervised learning. In general, self-supervised learning [133] utilizes only unlabeled data, while semi-supervised learning [134] combines a small amount of labeled data with a much larger pool of unlabeled data. In particular, self-supervised learning has been a rapidly developing area in recent years, especially in its applications to large language models and multimodal models [200,201]. A representative self-supervised learning technique is contrastive learning [202,203]. A prominent example is CLIP (Contrastive Language-Image Pretraining) [204], which extends contrastive learning to align images and texts in a shared embedding space, and has become a cornerstone framework for modern vision-language applications. The primary advantage of semi/self-supervised learning methods is their ability to learn without extensive labeled data. Nevertheless, a significant drawback is their demand for large-scale datasets and substantial computational resources. For alleviating this issue, coreset techniques have attracted a great amount of attention in recent years. These techniques contribute to improved computational efficiency [51,205], minimizing data redundancy [206], and boosting robustness to distributional shifts [55].

Similar to the other scenarios discussed earlier, coreset selection in this setting is typically formalized as an optimization problem that aims at minimizing the performance gap between the models trained on the full dataset and a selected subset. Roughly speaking, the coreset should be as informative as the full unlabeled data in terms of the learned representations [206,207] or training dynamics [51,119,205]. Since finding the exact optimal subset is NP-hard in general cases, most practical approaches rely on approximate or heuristic algorithms. We categorize existing approaches by the primary proxy used to assess “data quality”: loss-based [206,208], gradient-based [51,205] and coverage-based [209,210,211,212,213,214] methods. A summary of several representative methods from each category is presented in Table 3.

3.3.1. Loss-Based Methods

Loss-based methods exploit the training loss or related signals to identify informative samples. The key intuition is that samples which incur higher loss values or provide stronger signals are more valuable for representation learning.

A typical example is the “s-CLIP-Loss” metric, proposed by [206]. It generalizes the original CLIP training loss by incorporating a batchwise normalization term. This modification penalizes redundant samples whose image-text embeddings exhibit high similarity with incorrect text embeddings, thereby prioritizing specific and informative data samples. From a geometric perspective, s-CLIP-Loss refines the selection criterion by adjusting cosine similarity with a log-sum-exp aggregation of negative-pair similarities, acting as a soft redundancy-aware regularizer in the embedding space.

Another related concept is the dynamic leveraging of challenging samples, as implemented in the method called “hard negative mining” [208,215], which has been a widely used strategy in contrastive learning. Hard negatives are samples (or pairs) that the current model finds highly similar to the query sample despite being non-matching; in other words, these samples usually can provide a strong training signal to our coreset construction. Mining these hard negatives constitutes a form of dynamic data selection: at each training stage, the model selects the most informative pairs or samples to guide the learning process.

3.3.2. Gradient-Based Methods

Gradient-based methods aim to construct subsets that preserve the gradient statistics of the full dataset, so that optimization on the subset closely approximates the learning trajectory of the original training process. In this sense, the selected coreset captures the training dynamics of the model.

A well-studied approach is gradient matching. In Section 2.1, we have already introduced the idea of gradient matching for coreset construction. Roughly speaking, the gradient matching method tries to select a subset whose induced gradient could approximately represent those computed from the full dataset, thereby maintaining similar training dynamics. As mentioned before, RETRIEVE [51] applies this idea to improve both efficiency and robustness for semi-supervised learning.

Another important challenge in semi/self-supervised learning pipelines is how to select informative data for efficient fine-tuning after semi/self-supervised pretraining. Such fine-tuning is commonly required in downstream tasks, including linear probing or head tuning for classification, text-image retrieval, and lightweight adaptation for detection or segmentation, particularly under limited computational resources. The recent method “Sketchy Moment Matching (SkMM)” [205] addresses this problem by controlling the variance-bias trade-off that arises in high-dimensional fine-tuning. It first applies gradient sketching to project the full dataset’s gradients onto a low-dimensional subspace that retains essential information of the model. Within this subspace, it then performs moment matching to select a coreset whose sketched gradient moments closely approximate those of the full dataset. This procedure is label-agnostic and thus it is well suited for resource-constrained fine-tuning after unsupervised pretraining.

3.3.3. Coverage-Based Methods

In contrast to the methods that focus on training dynamics, coverage-based methods construct coresets that preserve the geometry or distribution of the dataset in the representation space. Their core principle is that a good coreset should preserve the geometric or statistical properties of the full dataset’s representation, such as its distribution, diversity, and coverage.

Distribution matching. A large part of coverage-based approaches follow the idea of distribution matching, which is previously introduced in Section 2.4. In the context of semi/self-supervised learning, a notable example is “ClipCov” [207], a theoretically grounded coreset selection method tailored for CLIP [204]. Unlike prior methods developed for supervised or unimodal contrastive learning, ClipCov specifically addresses the multimodal nature of CLIP by selecting image-caption pairs that can well preserve the cross-modal covariance structure. This distributional alignment is crucial for maintaining CLIP’s generalization performance.

When the information of downstream tasks is available, the focus of distribution matching can align with the target distribution. A prime example of this approach is the “NormSim” method proposed by [206]. This method guides the selection of semantically aligned subsets by evaluating the p-norm similarity between candidate samples and the target task distribution in the visual embedding space. This forward-looking selection strategy can yield a tighter coupling between the pretraining and fine-tuning stages. Specifically, it ensures that the coreset is constructed not only to preserve information from the original dataset, but also to achieve promising performance on specific downstream applications.

Diversity and coverage. Similar to the coreset construction methods introduced in Section 3.1.2 for active learning, there also exists a large body of work on semi/self-supervised learning that focuses on selecting a subset that maximally covers the data’s semantic space. An example is from [118], who introduce “Subsets that maximize Augmentation Similarity (SAS)”, a method that identifies examples with preserving both augmentation alignment and class-center divergence among the input data. Similar to the coreset idea introduced in Section 2.1, this method formulates the subset selection as a submodular optimization problem. Furthermore, it provides a theoretical guarantee that the alignment and divergence measures computed on the selected subset uniformly approximate those computed on the full dataset, thereby ensuring that representations learned from the subset achieve comparable downstream generalization performance. The principle of maximizing coverage is also used in “SimCore” [119], a coreset-based framework for fine-grained visual recognition under limited labels and abundant unlabeled open-set data. SimCore selects a subset from the open-set by maximizing its semantic similarity to a small target set, which is also formulated as a submodular optimization problem. The selection is performed in a learned representation space and is compatible with a variety of self-supervised methods, including the well-known approaches such as SimCLR [202], BYOL [218], DINO [219], and MAE [220]. By sampling from the embedding space, SimCore ensures that the selected examples are both semantically relevant and diverse, which is particularly effective for adapting large-scale unlabeled data to specialized downstream tasks.

Other heuristics for achieving sufficient coverage in the representation space include clustering-based selection (e.g., k-center or k-means on pretrained embeddings) to pick cluster prototypes that span high-density regions [211], and redundancy-aware pruning, which removes overly similar samples to reduce representation overlap [212]. [211] introduce “Effective Label-Free Coreset Selection (ELFS)”, a method for coreset selection without relying on ground-truth labels. ELFS utilizes deep clustering technique [221] to infer pseudo-labels and approximates difficulty scores based on training dynamics such as the model’s behavior across epochs measured by indicators including the Area Under the Margin (AUM) [222], forgetting events [94], and early-epoch loss (EL2N) [95]. To mitigate noise and distributional shifts introduced by pseudo-labels, it adopts a double-ended pruning strategy that prunes both trivially easy and overly hard samples, thereby improving the representativeness and overall utility of the selected coreset.

Mayilvahanan et al. [212] conduct a systematic study of pruning in CLIP’s embedding space to examine whether its strong out-of-distribution (OOD) generalization mainly stems from high train and test similarity. They define a perceptual similarity metric, computed as the cosine similarity between CLIP image embeddings, to quantify visual closeness in both content and style, and progressively remove training samples that are overly similar to specific test distributions. Even after removing these high-similarity samples, CLIP’s zero-shot accuracy on OOD benchmarks such as ImageNet-Sketch [223] and ImageNet-R [224] decreases only moderately and remains far higher than that of models trained solely on ImageNet [89]. They further show that a 100M subset obtained through far-pruning achieves nearly the same accuracy as the full 400M dataset, suggesting that large-scale diversity and cross-modal supervision, rather than sample-level overlap, are the main factors driving CLIP’s generalization. These findings offer useful insights for designing data-efficient pretraining and diversity-preserving coreset strategies.

The aforementioned approaches, also broadly categorized as “data filtering” or “data curation”, highlight a growing consensus: data quality is often more critical than sheer quantity. Li et al. [209] introduce “DeCLIP”, a framework designed to address redundancy in large-scale image-text datasets. By integrating additional self-supervised objectives, multi-view consistency, and nearest neighbor supervision, DeCLIP extracts richer training signals from smaller, curated subsets. Such implicit coreset selection mechanisms leverage model-based filtering strategies to curate high-quality subsets. This strategy embodies the coreset principle of achieving high learning efficacy using minimal yet informative data. Similarly, a number of recent works also underscore the importance of subset quality over sheer dataset size [207,210,213,214,216,217].

3.3.4. Experimental Comparisons

Despite differences in setup, the results consistently show that much of the training data can be removed with little impact on performance, often leading to significant efficiency improvements. In semi-supervised learning, RETRIEVE keeps VAT on CIFAR-10 at 87.02% using only 30% of the unlabeled data, compared with 87.8% on the full set, while reducing training time from 30.41h to 10.27h; on CIFAR10-OOD, a 50% subset reaches 79.0%, slightly above DS3L’s 77.6%, with 17.03h rather than 88.91h [51]. For post-pretraining adaptation, SkMM [205] raises CIFAR-10 linear-probing accuracy from 91.68% to 92.96% at a budget of 1000 examples, and improves Stanford Cars fine-tuning from 40.35% to 41.77% at 4000 examples [205].

In contrastive self-supervised learning, SAS shows that removing about 20% of CIFAR100 and 40% of STL10 or TinyImageNet does not hurt downstream accuracy, and the selected subsets outperform random subsets by more than 3%; on STL10 with BYOL [218], removing 20% of the data even yields about a 2% gain [118]. ELFS [211] reports comparable gains for label-free selection, achieving improvements of up to 10.2% over D2 [225] with a SwAV [226] encoder, and 3.9% over FreeSel [227] with a DINO encoder on ImageNet-1K. Notably, it approaches the performance of supervised coreset selection at 30% and 50% pruning rates. In multimodal pretraining, ClipCov shows that up to 50% of the data can be removed without loss in performance, while retaining only 30% of the data recovers over 90% of full-data performance; with only 10% of CC3M [228], it reaches 11.33% zero-shot accuracy on ImageNet, versus 6.48% with CLIPScore and 3.95% with random filtering [207]. NormSim [206] further improves CLIPScore-based filtering by 5.3% on ImageNet-1k and 2.8% on average across 38 downstream tasks. Related work on caption quality further shows that a small number of highly descriptive captions can be as informative as many noisier ones [216]. More broadly, a range of filtering-based methods and data-quality analyses confirm the same trend: carefully curated subsets are often more effective than substantially larger but noisier corpora [210,213,217].

3.4. Summary

We have introduced the coreset selection methods for enhancing data efficiency in three scenarios with limited resource. Generally speaking, the coreset methods developed for these scenarios can be divided into four categories, that is, loss-based (see Section 3.1.1 and Section 3.3.1), gradient-based (see Section 3.2.2 and Section 3.3.2), coverage-based (see Section 3.1.2, Section 3.2.1 and Section 3.3.3), and hybrid methods (see Section 3.1.3). Loss-based methods aim to select examples that are likely to incur higher loss values and prioritize those about which the model is most uncertain [122,136,151,155]. However, relying on uncertainty alone risks selecting redundant or similar examples that contain highly overlapping information [127,141,143,144]. Gradient-based methods primarily select examples that could preserve the gradient statistics of the full dataset [51]. Coverage-based methods, on the other hand, emphasize diversity by selecting representative and varied examples that enable broad coverage of the input space [118,119,139,140]. Therefore, these categories prioritize examples that are diverse and dissimilar [132,188]. But focusing exclusively on diversity leads to annotating examples that are trivial or already well learned [127,141,143]. Hybrid methods are proposed to mitigate these shortcomings by simultaneously considering both uncertainty and diversity [127,141,143,144]. The goal is to select examples that not only provide novel information but also adequately represent the underlying data distribution. As a result, hybrid methods could offer a more effective trade-off between uncertainty and diversity.

4. Coresets for Large Language Models

Large language models (LLMs), such as DeepSeek-v3 [248], GPT-4 [249], and LLaMa [250], exhibit remarkable proficiency across a broad spectrum of language understanding and generation tasks. In their training procedures, LLMs typically follow two key stages [251]: pretraining on large scale corpora and fine-tuning on instruction following datasets. Pretraining constitutes a fundamental stage in the development of LLMs, a stage that equips the models with core knowledge and capabilities. LLMs acquire a deep understanding of language syntax, world knowledge, and reasoning abilities through self-supervised learning on extensive textual corpora. This stage lays the groundwork for fine-tuning, a process in which models are further trained on a carefully annotated set of (instruction, response) pairs so that their capabilities are enhanced and their controllability is improved.

Similar with the studies on semi/self-supervised learning in Section 3.3, recent studies suggest that high-quality public text may be exhausted by 2026–2028, a phenomenon known as the Data Wall [252]. The quality of training data plays a more critical role than its quantity for LLMs [253]. Consequently, a growing body of works focus on selecting high-quality subsets from large-scale pretraining or fine-tuning datasets. In this section, we provide a comprehensive overview of recent coreset-based data selection techniques for the pretraining and fine-tuning stages of LLMs. Table 4 provides a summary of several representative methods from each construction type introduced in Section 2.

4.1. Coresets for Pretraining

The pretraining of a large language model is significantly influenced by the characteristics of the training corpus [254,255]. Existing research shows that model performance can be substantially improved through the careful curation of high-quality data, and pretraining is shifting from more tokens to better tokens [236,252,256]. In particular, data quality, diversity, and coverage are widely recognized as the key factors in enhancing the training efficiency and generalization ability of LLMs [250,255,257]. Existing works on coreset selection for LLM pretraining can be broadly categorized as static selection including representative rule- and discriminator-based methods, and dynamic selection like influence-based and importance-sampling methods.

To improve data quality, rule-based filtering techniques have been widely adopted [250,258,259,260]. For example, the widely used web corpora FineWeb and its educational subset FineWeb-Edu [234] are constructed by applying a series of rule-based filters to the Common Crawl derived data. These methods rely on manually designed heuristics, such as removing terminal symbols, detecting repetitive sentences, or enforcing length constraints, to filter out low-quality samples from the training corpus. While being effective in eliminating superficial noise, such approaches lack the capacity to capture semantic-level information, which is critical for fine-grained and content-aware data selection.

Several studies have been proposed to address the above limitation by the introduction of discriminators [22,235,261,262,263]. These methods compare candidate data to predefined high-quality corpora, such as Wikipedia or instruction-tuning datasets, using trained discriminators to identify the samples that resemble the reference distribution. But such approaches heavily depend on the qualities of the reference datasets. So, a number of recent studies [236,256,260] turn to find different solutions, e.g., leveraging existing large language models as the data evaluators. By employing carefully designed prompts to assess data quality from various dimensions (e.g., factuality, coherence, and task relevance), these methods enable more nuanced and flexible filtering criterion for data selection. This LLM-assisted evaluation approach offers a promising alternative to discriminator-based approach, facilitating more robust and scalable pretraining data selection.

Similar to the importance-based coreset idea introduced in Section 2.2, a line of works formalize this intuition via online batch selection and importance sampling [84,264]. To make this idea applicable to large-scale data selection in pretraining, Xie et al. [120] propose “Data Selection with Importance Resampling (DSIR)”, an efficient and scalable framework that estimates importance weights in a reduced feature space. The data items are then selected via importance resampling based on these weights. To ensure efficiency, the authors instantiate DSIR using hashed n-gram features, enabling the selection of 100 million documents from the full Pile dataset within 4.5 hours on 4 Titan RTX GPUs.

Influence function is another approach to estimate the impact of training data on the final model performance [265]. For instance, Wang et al. [229] introduce a method called “Influential Subset Selection (ISS)”, which explicitly leverages knowledge of the end task to guide pretraining data selection. Specifically, ISS selects the samples that have the greatest positive influence on the performance of the end task. Furthermore, the authors design a gradient-matching-based influence estimation method, which drastically reduces the time required to estimate influence. Using only 0.45% of the data and achieving a computational cost that is three orders of magnitude lower, ISS outperformed the pretrained models (e.g., RoBERTa) on eight datasets covering four domains. Moreover, a line of works utilize one-step Taylor expansion to estimates the sample influence on the validation set [230,266,267]. To overcome the high computational cost of influence estimation, they dive in the inner-product between the gradients, and utilize the “ghost-product” technique that allows for the efficient computation of pairwise gradient inner-products without the need to instantiate any model-sized vectors, achieving a speedup of 20x compared to directly computing the influence. Wang et al. [231] further speedups the influence function estimation by utilizing CountSketch projections [268] to reduce the inner products computations into a low-dimensional space.

Complementary to the influence-based selection, recent research Thrush et al. [237] demonstrate that high-quality pretraining data can be selected by exploiting the correlation between language model perplexity and downstream task performance. Specifically, the “perplexity” is computed as the normalized sequence negative log-likelihood. The authors propose to select the domains of pretrained corpus with largest correlations between perplexity and target benchmark scores. Such correlation can be estimated by the output logits computed from a sample of 90 LLMs taken from the “Hugging Face Open LLM Leaderboard”1, without the need for additional LLM training. Marion et al. [269] perform a rigorous comparison between the simple data quality estimator of perplexity and more computationally intensive estimates of the Error L2-Norm [95] and Memorization [270], and find that the simple perplexity-based method outperforms the more computationally expensive approaches. Lin et al. [238] introduce a novel language model named “RHO-1”, which is different from the conventional LLMs who predict every next token in the corpus. Instead, RHO-1 focuses on learning from tokens aligned with the target distribution. This is achieved by assigning scores to tokens using a reference model and applying a loss function that prioritizes higher-scoring (i.e., more informative or relevant) tokens during training.

Beyond the importance- or influence-based methods, several recent works focus on optimizing the distributional characteristics of the training corpus through clustering [243,244]. These approaches aim to reduce redundancy while preserving semantic diversity. For example, Abbas et al. [243] propose the method “SemDeDup”, which leverages embeddings from pretrained models to detect and eliminate semantic duplicates, which are pairs of examples that are highly similar in meaning but not necessarily identical. Shao et al. [244] develop the technique “ClusterClip” to balance the text distribution of training data for better model training. Specifically, ClusterClip utilizes clustering to capture the data distribution of the training set and balances the common and rare samples during the training procedure.

4.2. Coresets for Fine-Tuning

Fine-tuning plays a critical role in aligning LLMs with human instructions. By training on (instruction, output) pairs, fine-tuning helps bridge the gap between pretrained models and diverse human intents. Specifically, it enables LLMs to generate outputs that better reflect human preferences, thereby improving their controllability and safety. Moreover, fine-tuning can facilitate the adaptation to specific domains or acquisition of task-specific knowledge, without requiring substantial computational resources or architectural modifications.

Recent study [253] demonstrates that fine-tuning with only 1,000 carefully curated examples can yield strong performance comparable to the models trained on datasets that are several orders of magnitude larger. Very recently, several model-dependent data selection methods for efficient fine-tuning have been proposed. Joaquin et al. [19] provide the method “In2Core”, which leverages internal model gradients to quantify the influence of each training instance. Nguyen et al. [20] introduce “CoLM”, a memory-efficient training method for LLMs, where it identifies compact mini-batches to approximate the gradients. Similar to the coreset idea introduced in Section 2.1, they model the problem as a submodular maximization problem with leveraging pairwise gradient similarities. Liu et al. [245] quantitatively assess data examples from three key dimensions:complexity, quality, and diversity. They propose an enhanced and finer-grained data measurement and train a scorer to predict these scores to select the good samples in a score-first, diversity-aware manner. Similarly, Lu et al. [246] defines instruction diversity and complexity regarding tags to select samples. Du et al. [247] selects instruction data based on three criteria: quality, coverage and necessity. These methods share the similar idea of coverage-based methods discussed earlier in Section 3.1.2.

To overcome the high computation to compute the score or influence, a proxy model is typically used. For example, Zhang et al. [239] introduce “STAFF”, an efficient coreset selection strategy designed for task-specific fine-tuning of LLMs. The method first employs a smaller model from the same architecture family to estimate data importance scores, which are then verified and refined in the target LLM. This two-stage process can yield relatively accurate identification of high-impact data regions while preserving overall diversity. Lin et al. [240] consider the potential gap between the surrogate model and LLMs, and propose an effort score to prioritize some hard samples specifically for LLMs. They conducted experiments on two competitive LLM-based recommender models, and empirical results on three real-world datasets suggest they can use 2% samples to surpass the full data fine-tuning, reducing time cost by 97%. Yang et al. [233] propose “SmallToLarge (S2L)”, a data selection method that first trains a small reference model, clusters examples based on their loss trajectories, and then samples data from these clusters to guide the training of larger models.

Given the widely used next-token prediction paradigm that fairly treats each token during pretraining, fine-tuned models often face the challenge of overlooking the critical task-specific tokens [271], being susceptible to token noises [272,273], and inheriting biases in Zipf’s law [274]. To overcome the above problems, a line of works dive in the token-level coreset selection. Pang et al. [275] investigates token quality from a noisy-label perspective and proposes a generic token cleaning pipeline for SFT tasks. They measure token influence in a single pass with a fixed reference model or iteratively with self-evolving reference models. Chen et al. [232] borrow the spirit of bi-level coreset selection [16] and propose Quadratic Coreset Selection (QCS) that reconciles sequence-level and token-level influence contribution. Moreover, they use hierarchical policy gradients without requiring back-propagation, meanwhile achieving provable convergence and certified asymptotic equivalence to the original objective.

Beyond LLM-specific data selection, token-level coreset principles can also be extended to other tokenized architectures and modalities, such as Vision Transformers. Recent studies have leveraged token reduction to enable efficient fine-tuning of Vision Transformers, with representative methods including EViT [276], which reduces image tokens based on the token attentiveness measured by the class token attention scores. DynamicViT [241] utilizes a lightweight prediction module to estimate the importance score of each token given the current features, and proposes an “attention masking” strategy which eliminates attention values for the pruned tokens. DiffRate [242] enables propagating the loss function’s gradient onto the compression ratio, which is considered as a non-differentiable hyperparameter in previous work. They concurrently find top-K importance tokens and optimize the compression ratio as an optimization problem. Lei et al. [277] analyze the limitations of current token reduction techniques and highlight the importance of attention diversity and high-frequency information representation. They perform a Hadamard product between image tokens and a learnable mask to address the perturbation issue caused by the FFN bias. Their experimental results suggest outstanding performance across three real-world tasks with minimal FLOPs.

4.3. Summary

This section provides an overview of recent advances in coreset-based data selection methods for both the pretraining (in Section 4.1) and fine-tuning (in Section 4.2) stages of large language models. During pretraining, high-quality data selection plays a crucial role in improving model efficiency and generalization. Existing approaches include rule-based filtering [250,258,259,260], discriminator-based selection [22,235,262,263] using pretrained models as evaluators, and distributional optimization techniques that better align training corpora with target distributions [243,244]. Representative methods such as DSIR [120] and GREATS [230] enable efficient large-scale selection, while ISS [229] identifies influential subsets through gradient-based influence estimation. Additional techniques such as clustering (e.g., SemDedup [243], ClusterClip [244]) further enhance data quality by reducing redundancy and maintaining diversity.

For fine-tuning, the focus shifts toward selecting informative (instruction, response) pairs that can maximize downstream performance while reducing annotation and computational cost. The methods such as In2Core [19], CoLM [20], STAFF [239], and S2L [233] leverage model-internal signals or smaller proxy models to estimate data importance, cluster examples, and identify high-impact regions of the data space. Token-level selection methods such as [232,275,277] provide a finer-grained selection beyond the example level. These techniques enable models to achieve competitive performance with only a small fraction of the original data.

5. Challenges and Future Work

This survey reviewed the coreset methods in deep learning, which enhance data efficiency and preserve performance by selecting small, representative data subsets. We outlined their foundational principles and roles: reducing computational cost for large-scale training (as introduced in Section 2) and improving data utilization in resource-constrained scenarios (as introduced in Section 3) such as active learning, continual learning, and semi/self-supervised learning. We also explored the growing relevance of coreset techniques in large language models in Section 4. Despite the significant advances in coreset techniques, several critical challenges and promising future directions still remain in this field. At the end of this article, we highlight three key open issues that are pivotal to advancing both the theory and practical applications of coresets.

5.1. Dynamic and Adaptive Coreset Selection

Traditional coreset techniques often assume the given datasets are static. However, real-world scenarios frequently involve dynamic data streams or continual learning settings, where data distributions evolve over time. Although coresets have been successfully designed for fully dynamic data settings in models like SVMs and linear regression [278], extending these capabilities to deep neural networks remains a significant challenge. In particular, designing algorithms that can adaptively update coresets in response to streaming data or distributional shifts is still an open problem [131,132].

Recent findings reveal that in large-scale pretraining scenarios, especially with contrastive learning models such as CLIP [204], the utility of high-quality data can decay rapidly with repetition [279]. This highlights a critical limitation of static or heuristic-based coreset strategies, which often ignore the interplay between data utility, training repetitions, and compute budgets (such as available GPU hours or memory size).

Future coreset algorithms should not only track data distributional shifts but also estimate and model repetition-induced utility decay across heterogeneous data pools. Recent advances in scaling-law analysis have begun to formalize this idea by explicitly modeling how the marginal utility of data decays with repeated exposures, and how the optimal data-filtering strength depends on available compute resources. In particular, Goyal et al. [279] demonstrate that data curation cannot be compute-agnostic: they derive a scaling law that predicts performance across heterogeneous datasets by jointly modeling data quality, repetition, and compute budget. This framework provides a principled foundation for predicting the performance of different coreset configurations under varying compute budgets.

Moreover, adapting coreset strategies to the model architecture and training dynamics (e.g., ViT vs. CNN, contrastive vs. generative objectives) is becoming increasingly important [214]. For instance, large vision transformers exhibit greater performance gains from data diversity than smaller models [214]. This effect arises because larger models possess higher representational capacity and are often trained with fine-tuning techniques that enable them to leverage more informative samples. In contrast, smaller models tend to saturate earlier and gain less from additional diversity. Consequently, the coreset construction should be considered jointly with model scaling to ensure that the selected subset matches the model’s capacity and fine-tuning methods.

Furthermore, the development of automated coreset construction represents a promising direction. A practical challenge in deploying coreset methods lies in determining the appropriate coreset size [280] and selection criteria, which currently depend on manual tuning or domain-specific heuristics. This tuning often requires balancing trade-offs among accuracy, robustness, fairness, and computational efficiency. Automated coreset construction, potentially leveraging AutoML or meta-learning paradigms, could more efficiently discover minimal coresets tailored to specific objectives. Early efforts in this direction, such as RETRIEVE [51], demonstrate that meta-learned selection strategies can adaptively construct coresets aligned with downstream task requirements.

5.2. Balancing Fidelity and Diversity

Existing coreset methods frequently depend on single, isolated selection criteria, such as maximizing diversity or prioritizing regions of high data density, potentially leading to suboptimal trade-offs between fidelity and diversity. For instance, the methods emphasizing dense, high-quality regions risk neglecting rare but critical modes, thus reducing generalization capabilities and diversity in generative tasks. On the other hand, the methods that solely optimize for diversity may include noisy or irrelevant data points, diminishing the overall performance.

Recent findings further highlight that diversity and fidelity cannot be treated as independent or uniformly beneficial dimensions. In large-scale multimodal pretraining, data from heterogeneous sources often exhibits complementary but non-additive robustness properties [213]. Moreover, simply increasing dataset diversity without assessing semantic quality or target-task relevance can worsen performance under distribution shifts. This underscores the importance of not only balancing but also contextualizing fidelity-diversity trade-offs based on source characteristics and alignment with downstream tasks.

To address these challenges, recent studies propose more metric-informed data selection strategies for multimodal pretraining. For instance, the classical CLIPScore [281] measures the cosine similarity between visual and textual embeddings from CLIP, serving as a simple proxy for sample quality. However, it often overestimates generic or weakly aligned pairs. To mitigate this, s-CLIP-Loss [206] refines quality estimation by incorporating a contrastive-pair normalization term inspired by the original CLIP training loss, yielding a more faithful assessment of multimodal consistency. In parallel, NormSim [206] defines a vision-only p-norm similarity between pretraining data and target domains, allowing models to prioritize samples that are semantically relevant to downstream tasks. Together, these complementary metrics suggest that data fidelity and task relevance should be optimized jointly but with different priorities, that is, rare yet informative examples are retained, while redundant or mismatched samples are pruned more aggressively.

Future research should therefore pursue hybrid coreset frameworks that incorporate both structural diversity and semantically-informed fidelity signals. Such systems could dynamically tune sampling strategies using proxy metrics like task-targeted NormSim, or decay-based models of utility [279], allowing coresets to remain compact yet highly expressive across different tasks and training phases. This direction holds promise for building coreset selection methods that are robust, efficient, and capable of generalizing across diverse tasks.

5.3. Ethics and Privacy

As coreset selection becomes increasingly prevalent, it raises significant concerns regarding algorithmic fairness and data privacy. The selection process itself, while designed for efficiency, can inadvertently amplify systemic biases encoded in the original data. For example, selection criteria may disproportionately favor majority groups or samples that align with spurious correlations, thereby degrading the group robustness of downstream models [282]. Moreover, the resulting coreset, being a concentrated and highly representative summary, could inadvertently leak sensitive information or make individuals more identifiable, posing a privacy risk. While early explorations into private coresets have been studied for tasks like clustering [283,284], ensuring these properties in broader applications remains a challenge. Thus, developing coreset selection frameworks that integrate formal guarantees, such as differential privacy principles and explicit fairness criteria, represents a vital and promising research direction for trustworthy machine learning [285].