From Entropy and Beyond: A Comprehensive Survey of Probability-Space Unsupervised Objectives

1. Introduction

The growth of compute, model scale, and optimization infrastructure has outpaced the growth of labeled data. In many application domains, annotation remains the dominant bottleneck rather than computation. When target-domain labels are scarce [1], a common strategy is to transfer models pre-trained on abundant labeled source-domain data [2]. The limitation is well known: domain shift between source and target distributions can substantially reduce performance on out-of-distribution (OOD) inputs [3]. A disease diagnostic system trained on one distribution, for example, may fail when the target population or acquisition process changes [4]. These constraints have renewed interest in unsupervised and zero-shot adaptation methods that update a model, or an attached module, without target labels [5,6,7]. Many such methods operate without source data and sometimes without additional supervised training, instead optimizing objectives defined directly in the probability space [8].

Figure 1. Overview of probability-space unsupervised objectives for label-free learning. The survey organizes 17 objectives into four families: information-theoretic, probabilistic dispersion, geometric, and regularization-based, and analyzes their behavior, relationships, and practical applications using a unified predictive-distribution perspective. By enabling optimization directly from model predictions, these objectives offer a versatile framework for adaptation, robustness, and learning in data-scarce environments.

Figure 1. Overview of probability-space unsupervised objectives for label-free learning. The survey organizes 17 objectives into four families: information-theoretic, probabilistic dispersion, geometric, and regularization-based, and analyzes their behavior, relationships, and practical applications using a unified predictive-distribution perspective. By enabling optimization directly from model predictions, these objectives offer a versatile framework for adaptation, robustness, and learning in data-scarce environments.

1.1. Motivation

A broad class of unsupervised objectives acts on a single predictive distribution rather than on paired labels, paired samples, or paired domains. Shannon entropy is the most familiar example, but it is only one member of a larger design space spanning information-theoretic, dispersion-based, geometric, and regularization-based formulations. These objectives have appeared in applications ranging from feature selection and anomaly detection to test-time adaptation and source-free adaptation. Their role in modern AI systems, including large language models such as GPT [9] and BERT [10], and vision-language models such as CLIP [11] and Flamingo [12], is increasing but remains scattered across subfields [13]. In data-scarce settings, objectives such as Shannon and Tsallis entropy can improve robustness, reduce overfitting, and respond to domain shift [8,14,15]. More broadly, single-distribution objectives are a natural fit for unlabeled optimization because they can be computed directly from model predictions [16,17].

In this review, a single probability distribution refers to the conditional predictive distribution associated with one input, typically written as $P(y_i\mid x_i)$. The survey focuses on objectives whose primary argument is this single predictive distribution. We therefore exclude pairwise distribution losses, contrastive objectives, and divergences between two distributions except when they are needed to clarify relations among the surveyed objectives.

1.2. Objective & Impact

This survey reviews 17 foundational unsupervised objectives defined on single probability distributions. Our goal is not simply to catalogue formulas, but to clarify what each objective measures, which assumptions it makes, how it relates to nearby objectives, and where it has been used in practice. In contrast to prior reviews centered on tasks or application domains, we focus on the objective itself as the unit of analysis, with emphasis on theory, implementation, and practical selection [8].

The impact of this scope extends beyond classical unsupervised learning. In large language models, such objectives can support adaptation and fine-tuning under limited supervision, including settings involving low-resource languages [10]. In vision-language systems, they can regularize confidence, encourage diversity, or help align representations in unlabeled target domains. More generally, probability-space objectives expose compact levers for controlling uncertainty, concentration, diversity, and structure, which makes them relevant to robustness, fairness, efficiency, and interpretability in modern AI systems.

1.3. Contribution and Outline

This review consolidates probability-space unsupervised objectives into a single framework and evaluates them from both a theoretical and a practical perspective. Compared with the original draft, the revised version is more synthesis-oriented: it reduces repetition, highlights algebraic relations across objectives, and makes explicit which objectives are truly distinct and which are reparameterizations or reinterpretations of the same primitive.

Our contributions are as follows:

• Taxonomy and synthesis. We organize 17 objectives into four families and describe the design principles that distinguish them.
• Theoretical clarification. We make explicit the relations among closely related objectives, including normalized, exponential, quadratic, and moment-based variants.
• Practical guidance. We summarize computational properties, implementation considerations, and task-level suitability, and we preserve direct pointers to implementation through the existing project assets.
• Empirical comparison. We retain the small-scale controlled experiments to illustrate how different unsupervised objectives behave when used directly for unlabeled optimization.
• Forward-looking discussion. We connect the surveyed objectives to current adaptation settings in large pre-trained models and identify promising directions for future work.

The remainder of the paper follows the same backbone as the original manuscript with minor structural tightening. Section 2 defines the scope, introduces the single-probability setting, and positions this survey relative to prior reviews. Section 3 presents the objective families. Section 4 reports the comparative experiments. Section 5 synthesizes practical lessons, comparative behavior, and future directions. Section 6 and Section 7 conclude the paper and discuss validity threats.

2. Background

2.1. Evolution of Learning Objectives

The history of learning objectives mirrors the history of machine learning itself. Early work, especially in statistics and information theory, introduced measures of uncertainty, surprise, spread, and concentration. As machine learning moved toward more complex predictive tasks, these measures became useful not only for analysis but also for optimization. Entropy-based objectives formalized uncertainty. Dispersion measures quantified variability. Moment-based quantities such as skewness and kurtosis described shape. Regularization terms provided a way to control model complexity and prevent brittle overconfident predictions.

The deep learning era amplified all of these needs. High-capacity models required stronger regularization, better confidence control, and principled ways to adapt under domain shift. At the same time, the cost of annotation motivated unsupervised and self-supervised methods. The recent rise of source-free adaptation, test-time adaptation, and adaptation of large pre-trained models has therefore brought single-distribution objectives back to the foreground. Figure 2 summarizes this evolution and shows that many “new” objectives are modern uses of older statistical principles rather than entirely new inventions.

2.2. Single Probability Distribution (SPD)

The common object underlying this survey is the conditional predictive distribution $P(y_i\mid x_i)$, which assigns a probability to each possible outcome for a given input. This perspective is especially natural in zero-shot, transfer, and adaptation settings because the model already produces probabilities even when labels are unavailable.

Definition 1. 

Let $x_i\in \mathcal{X}$ be a sample from a distribution X, and let $y_i\in \mathcal{Y}$ denote the corresponding label. The conditional probability distribution $P(y_i\mid x_i)$ is given by Bayes’ rule:

$$P(y_i\mid x_i)={\displaystyle \frac{P(x_i\mid y_i)P\left(y_i\right)}{P\left(x_i\right)}}.$$

Here, $P(x_i\mid y_i)$ is the likelihood, $P\left(y_i\right)$ is the prior, and $P\left(x_i\right)={\sum }_{y_i\in \mathcal{Y}}P(x_i\mid y_i)P\left(y_i\right)$ is the marginal likelihood. In discriminative models, the same object is usually parameterized directly. For multiclass prediction,

$$P(y_i\mid x_i;\theta )={\displaystyle \frac{exp\left(f_{\theta }(x_i,y_i)\right)}{{\sum }_{y_j\in \mathcal{Y}}exp\left(f_{\theta }(x_i,y_j)\right)}},$$

(1)

where $f_{\theta }(x_i,y_i)$ is the logit for class $y_i$ and $\theta$ are model parameters. Prediction is typically obtained by

$$y^{*}=arg\underset{y_i\in \mathcal{Y}}{max}P(y_i\mid x_i).$$

(2)

This formulation provides the substrate on which the surveyed objectives operate. Some objectives treat the predictive distribution as a categorical object and are invariant to class permutations; others require an ordered or numeric support and therefore depend on how outcomes are encoded.

2.3. Related Works

Several prior surveys review loss functions or unsupervised learning more broadly. [18] surveys loss functions across supervised, unsupervised, and reinforcement learning. [19] focuses on clustering and dimensionality reduction. [20] provides a broad review of loss functions across domains. [21] studies unsupervised losses for homography estimation in geometric computer vision. [22] surveys unsupervised methods in supply-chain settings. These works are useful but they do not focus specifically on objectives whose input is a single probability distribution.

That omission matters. Objectives such as Shannon entropy [23] provide direct control over uncertainty, diversity, or concentration in predictive outputs, and they are now central to modern adaptation settings including large pre-trained models [11,13,14]. The present survey therefore complements prior reviews by taking the predictive distribution itself as the organizing principle.

Table 1. Comparison of related review works on unsupervised loss (UL) functions and their focus. Single Distribution indicates whether the UL function inputs a single probability distribution, making it more applicable in state-of-the-art settings, while Comments indicates a positive or negative aspect of each review work.

Ref	Year	UL	Single Distribution	Focus of Review	Comments
[18]	2023	✓	✗	Broad survey of loss functions across tasks	Task-specific taxonomy
[19]	2022	✓	✗	Clustering and dimensionality reduction	Descriptive without analysis
[20]	2020	✓	✗	Theoretical analysis and taxonomy	Naively discussed 4 UL functions
[21]	2021	✓	✗	Applications in geometric computer vision	Briefly analyzed just 2 UL functions
[22]	2024	✓	✗	Optimization in supply chain systems	A certain use-case focused
Ours	2024	✓	✓	Single Probability Distribution Losses	Demonstrated extensive experiments

Table 1. Comparison of related review works on unsupervised loss (UL) functions and their focus. Single Distribution indicates whether the UL function inputs a single probability distribution, making it more applicable in state-of-the-art settings, while Comments indicates a positive or negative aspect of each review work.

Ref	Year	UL	Single Distribution	Focus of Review	Comments
[18]	2023	✓	✗	Broad survey of loss functions across tasks	Task-specific taxonomy
[19]	2022	✓	✗	Clustering and dimensionality reduction	Descriptive without analysis
[20]	2020	✓	✗	Theoretical analysis and taxonomy	Naively discussed 4 UL functions
[21]	2021	✓	✗	Applications in geometric computer vision	Briefly analyzed just 2 UL functions
[22]	2024	✓	✗	Optimization in supply chain systems	A certain use-case focused
Ours	2024	✓	✓	Single Probability Distribution Losses	Demonstrated extensive experiments

2.4. Taxonomy of Objectives

We organize the surveyed objectives into four families: Information Theory-based losses, Probabilistic Dispersion-based losses, Geometric Distribution-based losses, and Regularization-based losses. The taxonomy reflects both mathematical form and functional role.

Information-theoretic objectives measure uncertainty, surprise, or coding length. Probabilistic-dispersion objectives quantify spread, impurity, dominance, or complexity. Geometric objectives capture higher-order distributional shape and therefore require an ordered support. Regularization objectives are best interpreted as inductive biases: they do not simply measure a property of the predictive distribution, but actively prefer smoother, more concentrated, or sparser outputs. Figure 3 provides the resulting hierarchy.

3. Unsupervised Objectives for SPDs

3.0    Preliminary

3.0.1. Assumption

Unless stated otherwise, we assume a discrete predictive distribution

$$\sum _{i}P(y_i\mid x)=1,P(y_i\mid x)\ge 0\forall i.$$

(3)

When an objective requires numeric support, we write the associated support values as $x_i$. This distinction matters because some objectives are permutation-invariant over class labels, while others are not.

3.0.2. Exemplary Illustration

The curve plots used throughout this survey evaluate each objective on a simple one-dimensional probability sweep. Concretely, we consider probabilities $p\in [0.01,0.99]$ and visualize how each objective varies with concentration, uncertainty, or shape. These plots are not substitutes for full optimization analysis, but they provide an interpretable first view of monotonicity, symmetry, and extreme-value behavior.

3.0.3. Notations

Unless stated otherwise, logarithms are natural, in which case values are expressed in nats; base-2 logarithms yield bits [24]. We use the term objective rather than loss when the sign convention depends on implementation, since several quantities in this survey are maximized in analysis but minimized in optimization by negating them.

3.1. Information Theory Losses

Information-theoretic objectives are the most widely used probability-space objectives in modern adaptation pipelines. They directly quantify uncertainty, coding length, or surprise. Many are closely related: perplexity is the exponential form of Shannon entropy; collision entropy is a special case of Renyi entropy; normalized entropy rescales Shannon entropy; and information content is the pointwise term whose expectation gives Shannon entropy.

3.1.1. Shannon Entropy

Shannon entropy measures the uncertainty of a predictive distribution [23,25]. For a discrete SPD,

$$H\left(P\right)=-\sum _{i}P(y_i\mid x)logP(y_i\mid x)$$

(4)

with maximum value under the uniform distribution and minimum value at a point mass. The computation is linear in the number of outcomes, $O\left(n\right)$, and the quantity is invariant to permutations of the class labels [26].

In machine learning, Shannon entropy underlies information-gain criteria for tree induction [27], clustering criteria [28], and entropy-minimization schemes for adaptation [6,29]. It also appears in language modeling and generation [30,31], anomaly detection [32], domain generalization [33], and zero-shot or test-time adaptation of vision-language models [13,14]. Its appeal comes from the fact that it is simple, stable, and often acts as a reliable confidence-control signal.

3.1.2. Perplexity Loss

Perplexity measures the effective number of equally likely outcomes and is the exponential form of base-2 Shannon entropy [23,34]:

$$\mathit{Perplexity}=2^{-{\sum }_{i}P(y_i\mid x){log}_{2}P(y_i\mid x)}$$

(5)

Lower perplexity corresponds to more concentrated and predictable predictions. Because it is computed from entropy, its complexity is also $O\left(n\right)$.

Perplexity is standard in generative modeling and natural language processing. It is widely used to evaluate and guide recurrent language models [35], large generative models [31,36], and transformer-based pre-trained models such as BERT [10]. Relative to raw entropy, its main advantage is interpretability: the value can be read as an effective support size.

3.1.3. Rényi Entropy

Rényi entropy generalizes Shannon entropy by introducing an order parameter $\alpha$ that controls how strongly the objective emphasizes large or small probabilities [25]:

$$H_{\alpha }\left(P\right)={\displaystyle \frac{1}{1-\alpha }}log\sum _{i}P{(y_i\mid x)}^{\alpha },\alpha >0,\alpha \ne 1$$

(6)

As $\alpha \to 1$, it converges to Shannon entropy. Larger $\alpha$ values emphasize dominant probabilities; smaller values emphasize the tail. The cost remains $O\left(n\right)$, with minor overhead from exponentiation.

This tunability makes Rényi entropy useful in settings where one wants to control sensitivity to concentration or rarity. Representative uses include regularization in neural networks [37,38], quantum information and entanglement analysis [39,40], adaptive learning in dynamic neural systems [41,42], and variational estimation of distributions and divergences [43].

3.1.4. Tsallis Entropy

Tsallis entropy is another one-parameter generalization of Shannon entropy, designed for non-extensive settings in which classical additivity is not the preferred assumption [44,45]:

$$S_q\left(P\right)={\displaystyle \frac{1-{\sum }_{i}P{(y_i\mid x)}^q}{q-1}},q\ne 1$$

(7)

It converges to Shannon entropy as $q\to 1$. Like Rényi entropy, it remains an $O\left(n\right)$ objective, but its parameterization often yields a different practical sensitivity to concentration and outliers.

Tsallis entropy has been used in imbalanced decision trees and robust learning [46], anomaly detection [47], ecological diversity analysis [48], text-domain adaptation [49], and quantum-information settings [50]. In practice, it is attractive when the target domain exhibits heavy tails, imbalance, or atypical concentration patterns.

3.1.5. Minimum Description Length (MDL)

The Minimum Description Length principle occupies a boundary between information-theoretic objective and model-selection criterion. Its core idea is to minimize total code length by trading data fit against model complexity [51,52,53]:

$$\mathit{MDL}\left(P\right)=-logP(y_i\mid x)+\mathit{complexity}\mathit{penalty}$$

(8)

Unlike the other information-theoretic objectives in this section, MDL is not determined solely by the predictive distribution; it also depends on how model complexity is encoded. This distinction is important when comparing MDL to purely distributional quantities.

MDL appears in few-shot and transductive inference [54], reinforcement learning [55], spiking neural networks [56], dimension reduction [57], prompt and parameter-efficient tuning [58], and analyses of intrinsic dimensionality in language-model fine-tuning [59]. Its main strength is principled control of overfitting through compression.

3.1.6. Collision Entropy

Collision entropy is the order-2 case of Rényi entropy and measures concentration through pairwise collisions [60,61,62]:

$$H_c\left(P\right)=-log\sum _{i}P{(y_i\mid x)}^2$$

(9)

The quadratic term makes the objective especially sensitive to dominant probabilities. Computation is $O\left(n\right)$, and the objective is permutation-invariant over categories.

Collision entropy is widely used in cryptography and security analysis [60,63,64]. It also appears in ecological dominance analysis [65] and as a sharpness-sensitive diagnostic in machine learning [66]. Relative to Shannon entropy, it emphasizes overlap and concentration rather than average uncertainty.

3.1.7. Normalized Entropy

Normalized entropy rescales Shannon entropy to the interval $[0,1]$, making comparisons across different support sizes more direct [23,67]:

$$H_n\left(P\right)={\displaystyle \frac{H\left(P\right)}{logn}}$$

(10)

This simple normalization preserves the ordering induced by Shannon entropy while eliminating dependence on the number of categories. The computational cost is again $O\left(n\right)$.

Normalized entropy is useful when support size varies across tasks or datasets. Representative applications include clustering evaluation [68], biodiversity analysis [69], and feature ranking based on relative uncertainty reduction [70]. In survey terms, it is best viewed as a reporting and comparison layer over Shannon entropy rather than a fundamentally different primitive.

3.1.8. Information Content

Information content is the pointwise surprise associated with a single event [23,67,71]:

$$I\left(y_i\right)=-logP(y_i\mid x)$$

(11)

Its expectation under P recovers Shannon entropy. For a single event the computation is $O\left(1\right)$; evaluating it over all outcomes is $O\left(n\right)$.

Because it is event-specific rather than distribution-averaged, information content can be useful when rare outcomes must be emphasized. It is central to coding theory [72], appears in language and text modeling [68], and can be used in decision-making or reinforcement learning to value informative outcomes [73,74]. The experiments in this paper also show that this emphasis on rare events can be destabilizing when used naively as an adaptation objective.

Table 2. Summary of information theory-based unsupervised objectives that take a single probability distribution with n outcomes as input. Min and Max represent the minimum and maximum possible values for n outcomes, AUC represents the possible area under the curve of the objective’s value across distributions, and Monotonicity describes the behavior of the objective as probabilities $p_i$ approach 1, where $p_i$ represents the probability assigned to the i-th outcome of a discrete probability distribution P.

Unsupervised Objective	Min	Max	AUC	Monotonicity	Key Characteristics
Shannon Entropy [Eq. 4]	0	$log\left(n\right)$	Finite	Decreasing for $p_i\to 1$	Measures uncertainty in distributions.
Perplexity Loss [Eq. 5]	1	n	Finite	Decreasing for $p_i\to 1$	Measures effective number of outcomes.
Rényi Entropy ($\alpha >0$) [Eq. 6]	0	$log\left(n\right)$	Finite	Varies with $\alpha$	Generalizes Shannon Entropy.
Tsallis Entropy ($q>0$) [Eq. 7]	0	${\displaystyle \frac{n^{1-q}-1}{1-q}}$	Finite	Varies with q	Generalizes Shannon Entropy with q.
Minimum Description Length [Eq. 8]	$\ge 0$	Unbounded	N/A	Varies	Measures complexity of data encoding.
Collision Entropy [Eq. 9]	0	$log\left(n\right)$	Finite	Decreasing for $p_i\to 1$	Based on pairwise collisions in P.
Normalized Entropy [Eq. 10]	0	1	Finite	Decreasing for $p_i\to 1$	Normalized measure of uncertainty.
Information Content [Eq. 11]	0	∞	Infinite	Decreasing for $p_i\to 1$	Negative log-probability of events.

Table 2. Summary of information theory-based unsupervised objectives that take a single probability distribution with n outcomes as input. Min and Max represent the minimum and maximum possible values for n outcomes, AUC represents the possible area under the curve of the objective’s value across distributions, and Monotonicity describes the behavior of the objective as probabilities $p_i$ approach 1, where $p_i$ represents the probability assigned to the i-th outcome of a discrete probability distribution P.

Unsupervised Objective	Min	Max	AUC	Monotonicity	Key Characteristics
Shannon Entropy [Eq. 4]	0	$log\left(n\right)$	Finite	Decreasing for $p_i\to 1$	Measures uncertainty in distributions.
Perplexity Loss [Eq. 5]	1	n	Finite	Decreasing for $p_i\to 1$	Measures effective number of outcomes.
Rényi Entropy ($\alpha >0$) [Eq. 6]	0	$log\left(n\right)$	Finite	Varies with $\alpha$	Generalizes Shannon Entropy.
Tsallis Entropy ($q>0$) [Eq. 7]	0	${\displaystyle \frac{n^{1-q}-1}{1-q}}$	Finite	Varies with q	Generalizes Shannon Entropy with q.
Minimum Description Length [Eq. 8]	$\ge 0$	Unbounded	N/A	Varies	Measures complexity of data encoding.
Collision Entropy [Eq. 9]	0	$log\left(n\right)$	Finite	Decreasing for $p_i\to 1$	Based on pairwise collisions in P.
Normalized Entropy [Eq. 10]	0	1	Finite	Decreasing for $p_i\to 1$	Normalized measure of uncertainty.
Information Content [Eq. 11]	0	∞	Infinite	Decreasing for $p_i\to 1$	Negative log-probability of events.

Figure 4. Objective curves for the information-theoretic family.

Figure 4. Objective curves for the information-theoretic family.

3.2. Probabilistic Dispersion Losses

Probabilistic-dispersion objectives quantify spread, impurity, dominance, or structural complexity. Here the support matters. Gini impurity and Simpson’s index depend only on the probabilities and are permutation-invariant. Variance and mean absolute deviation require numeric support values, so their meaning changes if categories are relabeled. Statistical complexity combines entropy with a second structural term and sits between pure dispersion and structural analysis.

3.2.1. Variance Loss

Variance measures average squared deviation from the mean and remains one of the basic spread measures in statistics [75,76]:

$$\mathrm{Var}\left(P\right)=\mathbb{E}\left[{(x-\mu )}^2\right]=\int {(x-\mu )}^{2}P\left(x\right)dx$$

(12)

with $\mu =\mathbb{E}\left[x\right]$. For discrete support, the same definition becomes a weighted sum over support values. The cost is $O\left(n\right)$.

Variance is central to regression and statistical learning [77,78], as well as risk-sensitive modeling in finance [79,80,81,82]. In the SPD setting, however, it should be used only when support values have semantic meaning, because arbitrary class indices make the quantity hard to interpret.

3.2.2. Gini Impurity

Gini impurity measures the probability of misclassification if labels were sampled from the predictive distribution itself [27,83,84]:

$$G\left(P\right)=1-\sum _{i}P{(y_i\mid x)}^2$$

(13)

The computation is $O\left(n\right)$. Algebraically, Gini impurity is the complement of both Simpson’s diversity index and the quadratic regularizer discussed later, which is why these objectives often behave similarly in practice.

Its classical use is in decision trees and related splitting criteria, where lower impurity corresponds to cleaner partitions. In probability-space optimization more broadly, Gini acts as a diversity-promoting alternative to entropy with a simpler quadratic form.

3.2.3. Mean Absolute Deviation (MAD)

Mean absolute deviation measures average absolute deviation from the mean and is a more robust alternative to variance under outliers or heavy tails [85,86]:

$$\mathit{MAD}\left(P\right)=\sum _{i}P\left(x_i\right)|x_i-\mu |$$

(14)

Its complexity is $O\left(n\right)$, and, like variance, it depends on the chosen numeric support.

MAD appears in robust regression [85], multivariate outlier analysis [86], anomaly detection [87], image analysis [88], forecast evaluation [89], and exploratory data analysis [90]. In practice it is attractive when variance is too sensitive to a few extreme probabilities or support values.

3.2.4. Statistical Complexity Loss

Statistical complexity attempts to capture not only uncertainty but also structural organization. A common form multiplies Shannon entropy by Fisher information [91,92,93,94]:

$$C\left(P\right)=H\left(P\right)·\mathit{Fisher}\mathit{Information}$$

(15)

The exact Fisher-information term depends on parameterization, so implementations vary, but the computational burden is typically linear in the number of support points once the parameterization is fixed.

This objective has been used in time-series analysis [95], neural information-bottleneck studies [96], feature selection and compression [97], and ecological-network analysis [98]. It is best suited to settings where one wants to reward informative but nontrivial structure rather than mere uncertainty.

3.2.5. Simpson’s Diversity Index

Simpson’s Diversity Index (SDI) measures dominance by summing squared probabilities [99]:

$$D\left(P\right)=\sum _{i}P{(y_i\mid x)}^2$$

(16)

A reciprocal form, $1/D\left(P\right)$, is also common. The raw index is identical to the L2 distribution regularizer discussed in Section 3; only the interpretation changes. In ecology and retrieval, high $D\left(P\right)$ means dominance and low diversity, while in regularization the same quantity measures concentration.

SDI appears in diversified retrieval [100], microbiome clustering and diversity analysis [101], and ecological modeling [102,103]. Its usefulness in machine learning comes from the fact that the quadratic form is cheap, differentiable, and easy to interpret as concentration.

Figure 5. Objective curves for the probabilistic-dispersion family.

Figure 5. Objective curves for the probabilistic-dispersion family.

Table 3. Summary of probabilistic dispersion-based unsupervised objectives that take a single probability distribution with n outcomes as input. Min and Max represent the minimum and maximum possible values for n outcomes, AUC represents the possible area under the curve of the objective’s value across distributions, and Monotonicity describes the behavior of the objective as probabilities $p_i$ approach 1, where $p_i$ represents the probability assigned to the i-th outcome of a discrete probability distribution P.

Unsupervised Objective	Min	Max	AUC	Monotonicity	Key Characteristics
Variance Loss [Eq. 12]	0	${\displaystyle \frac{{(n-1)}^2}{n^2}}$	Finite	Decreasing for $p_i\to 1$	Measures dispersion from the mean.
Gini Impurity [Eq. 13 ]	0	$1-{\displaystyle \frac{1}{n}}$	Finite	Decreasing for $p_i\to 1$	Measures probability of misclassification.
Mean Absolute Deviation [Eq. 14]	0	${\displaystyle \frac{2(n-1)}{n^2}}$	Finite	Decreasing for $p_i\to 1$	Measures average absolute deviation from mean.
Statistical Complexity Loss [Eq. 15]	$\ge 0$	Unbounded	N/A	Varies with distribution	Measures structure and unpredictability.
Simpson’s Diversity Index [Eq. 16]	${\displaystyle \frac{1}{n}}$	1	Finite	Decreasing for $p_i\to 1$	Measures lack of diversity in distributions.

Table 3. Summary of probabilistic dispersion-based unsupervised objectives that take a single probability distribution with n outcomes as input. Min and Max represent the minimum and maximum possible values for n outcomes, AUC represents the possible area under the curve of the objective’s value across distributions, and Monotonicity describes the behavior of the objective as probabilities $p_i$ approach 1, where $p_i$ represents the probability assigned to the i-th outcome of a discrete probability distribution P.

Unsupervised Objective	Min	Max	AUC	Monotonicity	Key Characteristics
Variance Loss [Eq. 12]	0	${\displaystyle \frac{{(n-1)}^2}{n^2}}$	Finite	Decreasing for $p_i\to 1$	Measures dispersion from the mean.
Gini Impurity [Eq. 13 ]	0	$1-{\displaystyle \frac{1}{n}}$	Finite	Decreasing for $p_i\to 1$	Measures probability of misclassification.
Mean Absolute Deviation [Eq. 14]	0	${\displaystyle \frac{2(n-1)}{n^2}}$	Finite	Decreasing for $p_i\to 1$	Measures average absolute deviation from mean.
Statistical Complexity Loss [Eq. 15]	$\ge 0$	Unbounded	N/A	Varies with distribution	Measures structure and unpredictability.
Simpson’s Diversity Index [Eq. 16]	${\displaystyle \frac{1}{n}}$	1	Finite	Decreasing for $p_i\to 1$	Measures lack of diversity in distributions.

3.3. Geometric Distribution Losses

Geometric-distribution objectives rely on higher-order moments. Unlike entropy or Gini-type quantities, they are not permutation-invariant and are meaningful only when outcomes lie on an ordered or numeric support. For categorical labels without order, they are generally not appropriate.

3.3.1. Skewness Loss

Skewness measures asymmetry around the mean [104,105]:

$$\mathrm{Skew}\left(P\right)=\mathbb{E}\left[{\left({\displaystyle \frac{x-\mu }{\sigma }}\right)}^3\right]$$

(17)

It is positive for right-tailed distributions and negative for left-tailed distributions. Computing it requires the mean, standard deviation, and third centered moment, which is again $O\left(n\right)$.

Skewness-aware objectives have been used to regularize neural activations [106], detect anomalous asymmetry [107], analyze financial-return asymmetry [108], and adapt learning under imbalance [109]. The key practical caveat is that skewness is support-dependent; it should not be interpreted on arbitrarily indexed classes.

3.3.2. Kurtosis Loss

Kurtosis measures tail heaviness or outlier sensitivity relative to a reference shape [110,111]:

$$\mathit{Kurt}\left(P\right)=\mathbb{E}\left[{\left({\displaystyle \frac{x-\mu }{\sigma }}\right)}^4\right]-3$$

(18)

The fourth power amplifies extreme deviations, which makes kurtosis highly sensitive to outliers. The cost is $O\left(n\right)$, but numerical stability may require care when the predictive distribution becomes extremely concentrated.

Applications include anomaly detection in multivariate time series [112], financial risk analysis [113], signal denoising [114], and robust adaptive neural models [115]. As with skewness, the quantity is meaningful only when the support has geometry.

Figure 6. Objective curves for the geometric family.

Figure 6. Objective curves for the geometric family.

Table 4. Summary of geometric distribution-based and regularization-based unsupervised objectives that take a single probability distribution with n outcomes as input. Min and Max represent the minimum and maximum possible values for n outcomes, AUC represents the possible area under the curve of the objective’s value across distributions, and Monotonicity describes the behavior of the objective as probabilities $p_i$ approach 1, where $p_i$ represents the probability assigned to the i-th outcome of a discrete probability distribution P.

Unsupervised Objective	Min	Max	AUC	Monotonicity	Key Characteristics
Skewness Loss [Eq. 17]	$-\infty$	$+\infty$	Undefined	Depends on distribution	Measures asymmetry of probability distributions.
Kurtosis Loss [Eq. 18]	$-3$	$+\infty$	Undefined	Varies with tail weight	Measures “peakedness” and tail behavior.
L2 Regularization Eq. 19	0	1	Finite	Decreasing for $p_i\to 1$	Penalizes sharp distributions.
Sparsity Loss [Eq. 20]	0	n	Finite	Increasing with non-zeros	Counts non-sparse dimensions (above a threshold).

Table 4. Summary of geometric distribution-based and regularization-based unsupervised objectives that take a single probability distribution with n outcomes as input. Min and Max represent the minimum and maximum possible values for n outcomes, AUC represents the possible area under the curve of the objective’s value across distributions, and Monotonicity describes the behavior of the objective as probabilities $p_i$ approach 1, where $p_i$ represents the probability assigned to the i-th outcome of a discrete probability distribution P.

Unsupervised Objective	Min	Max	AUC	Monotonicity	Key Characteristics
Skewness Loss [Eq. 17]	$-\infty$	$+\infty$	Undefined	Depends on distribution	Measures asymmetry of probability distributions.
Kurtosis Loss [Eq. 18]	$-3$	$+\infty$	Undefined	Varies with tail weight	Measures “peakedness” and tail behavior.
L2 Regularization Eq. 19	0	1	Finite	Decreasing for $p_i\to 1$	Penalizes sharp distributions.
Sparsity Loss [Eq. 20]	0	n	Finite	Increasing with non-zeros	Counts non-sparse dimensions (above a threshold).

3.4. Regularization Losses

Regularization objectives are best understood as preferences over predictive distributions. Rather than estimating uncertainty or shape for its own sake, they bias optimization toward smoother or sparser outputs. This family overlaps algebraically with other families, which is why interpretation matters.

3.4.1. L2 Distribution Regularization

L2 distribution regularization penalizes concentrated predictive distributions [116,117]:

$$R\left(P\right)=\sum _{i}P{(y_i\mid x)}^2$$

(19)

This is the same quadratic form as Simpson’s diversity index, but here it is interpreted as an anti-overconfidence regularizer. The objective is cheap to compute, $O\left(n\right)$, and easy to differentiate.

Representative uses include knowledge distillation and smoothing [118], label smoothing in deep networks [117], and regularization of sequence models to reduce peaked output distributions [119]. In adaptation settings, it often serves as a practical alternative when entropy is numerically inconvenient or when one wants a purely quadratic penalty.

3.4.2. Sparsity Loss

Sparsity objectives encourage predictive distributions with only a few active entries. In idealized form,

$$S\left(P\right)={\parallel P\parallel }_0$$

(20)

where ${\parallel P\parallel }_0$ counts nonzero entries. The formal complexity is $O\left(n\right)$, but the more important issue is conceptual: for dense softmax outputs, exact zeros rarely occur, so raw $L_0$ sparsity is often ill-posed unless one uses thresholds, relaxed surrogates, sparse alternatives to softmax, or discrete latent variables.

Even with that caveat, sparsity remains influential in feature selection and representation learning [120,121,122]. Recent uses include sparse autoencoders for model interpretability [123], unsupervised learning with sparse priors [124], and sparse latent-variable models [125]. In adaptation problems, the main question is not whether sparsity is useful, but which relaxed formulation best matches dense predictive outputs.

Figure 7. Objective curves for the regularization family.

Figure 7. Objective curves for the regularization family.

4. Experiments and Analysis

4.1. Experimentation

The experiments are retained as a controlled, small-scale comparison of how the surveyed objectives behave when used directly for unlabeled optimization. The goal is illustrative rather than benchmark-driven: we use the same backbone, the same adaptation recipe, and multiple datasets to expose broad patterns rather than to claim state-of-the-art results.

4.1.1. Problem Formulation

Following [13,14], we study image classification under unlabeled optimization. Given an image x, a model $f_{\theta }$ produces predictive probabilities $\widehat{y}=f_{\theta }\left(x\right)$. Instead of using ground-truth labels, we compute an unsupervised objective $\mathcal{L}\left(\widehat{y}\right)$ directly from the predictive distribution and update the parameters through

$${\theta }^{\prime }=\theta -\eta {\nabla }_{\theta }\mathcal{L}\left(\widehat{y}\right).$$

The comparison is therefore between the original model $f_{\theta }$ and the model after objective-driven optimization, $f_{{\theta }^{\prime }}$. This setup mirrors common test-time and source-free adaptation recipes [13,14,29] while remaining simple enough for controlled comparison.

4.1.2. Experimental Setup

We evaluate on CIFAR-10, MNIST, Fashion-MNIST, and SVHN, each with 10 classes. For tractability, 2,000 samples are drawn from each dataset and split 80:20 for training and testing. Two initialization settings are used. In the first, ResNet-18 is initialized from ImageNet and evaluated without supervised fine-tuning. In the second, the same backbone is first fine-tuned for 10 epochs using cross-entropy with Adam and learning rate ${10}^{-3}$. Unsupervised objective-based optimization then runs for 20 epochs with Adam and learning rate ${10}^{-4}$.

These two settings are useful because they separate two practical scenarios: adapting a generic pre-trained model and adapting a task-specialized model. The contrast also helps explain why some objectives are helpful before fine-tuning but harmful after fine-tuning, or vice versa.

4.2. Analysis

4.2.1. Comparison of Loss Functions

Table 5 and Table 6 show that the objectives are not interchangeable. In the pre-trained setting, Shannon entropy, variance, and Gini impurity improve average performance relative to the base model, while collision entropy shows notable gains on CIFAR-10 and SVHN. In the fine-tuned setting, Renyi entropy, statistical complexity, and sparsity are among the strongest performers, with Renyi entropy and statistical complexity yielding the best average results across the four datasets.

The negative cases are equally informative. MDL and information content consistently underperform in the fine-tuned setting, suggesting that objectives emphasizing coding length or rare-event surprise can conflict with already specialized classifiers. The experiments therefore reinforce a central survey lesson: objective choice encodes inductive bias, and the same bias can help or hurt depending on calibration, initialization, and domain.

4.2.2. Effect of Unsupervised Optimization

Across both initialization settings, unlabeled optimization is often beneficial, but not uniformly so. Confidence-promoting or smoothness-oriented objectives tend to improve performance more reliably than objectives that aggressively amplify rare events or high-order support structure. In the pre-trained regime, the gains are modest but consistent for several objectives. In the fine-tuned regime, gains are larger for a smaller subset of objectives, indicating that a better starting representation does not eliminate the need for objective selection; it makes the choice sharper.

4.2.3. Prediction Insights

The confusion matrices in Figure 8 and Figure 9 make this contrast visible. When Shannon entropy is used, off-diagonal errors generally shrink and class predictions become more consistent across both initialization settings. This is the expected behavior of a confidence-control objective that sharpens predictions without forcing them into degenerate modes too quickly.

Information content behaves differently. Because it emphasizes low-probability events at the pointwise level, it can amplify unstable gradients and degrade class separability. The resulting confusion matrices show more persistent off-diagonal structure, matching the lower accuracies in Table 6. In other words, the experiments do not simply rank objectives; they reveal why certain objectives are easier to optimize than others in unlabeled settings.

5. Discussion

5.1. Key Takeaways

The main lessons from the survey are summarized below.

• Many named objectives reduce to a small number of primitives. Perplexity is an exponential re-expression of entropy; collision entropy is Renyi entropy at $\alpha =2$; normalized entropy is scaled Shannon entropy; and Gini impurity, Simpson’s diversity index, and L2 distribution regularization are algebraically tied through the quadratic term ${\sum }_i{p}_i^2$.
• Support dependence is a first-order design issue. Entropy, Gini, collision entropy, normalized entropy, and L2-type objectives are permutation-invariant over categorical labels. Variance, MAD, skewness, and kurtosis are not, and therefore require meaningful support geometry.
• Objective families correspond to different inductive biases. Information-theoretic objectives primarily control uncertainty; dispersion objectives control spread or dominance; geometric objectives emphasize shape; and regularization objectives impose preferences such as smoothness or sparsity.
• Simple objectives often transfer better. In the experiments, entropy-like or quadratic objectives were easier to optimize robustly than pointwise surprise or higher-moment objectives.
• Implementation details matter. Stable logarithms, temperature scaling, thresholded sparsity surrogates, and support encoding can change behavior as much as the nominal objective choice itself.
• Application fit should drive selection. The same objective can be appropriate in one setting and problematic in another; there is no universally best probability-space objective.

Table 7. Summarizing the focus of applications and respective references for Information Theory-based, Probabilistic Dispersion-based, Geometric Distribution-based, and Regularization-based Unsupervised Objectives. Also see Figure ?? for detailed references to respective applications.

Unsupervised Objective	Applications
Shannon Entropy [Eq. 4]	Decision tree splits [27], clustering [28], entropy minimization for confident predictions [13], NLP for sequence unpredictability [30,31].
Perplexity Loss [Eq. 5]	Language model evaluation [35], improving fluency [31], machine translation, text generation, speech recognition.
Rényi Entropy [Eq. 6]	Neural network regularization [37], quantum entanglement [39], adaptive learning for dynamic models [41].
Tsallis Entropy [Eq. 7]	Statistical mechanics [44], clustering robust to outliers [46], anomaly detection [47], ecological diversity [48], quantum entanglement [50].
Minimum Description Length [Eq. 8]	Few-shot learning [54], reinforcement skill acquisition [55], dimensionality reduction in PCA [57], compact language model representations [59].
Collision Entropy [Eq. 9]	Cryptography key randomness [63], species dominance [65], class probability analysis for overfitting [66], quantum key distribution [60].
Normalized Entropy [Eq. 10]	Clustering quality evaluation [68], ecological biodiversity [69], feature selection [70].
Information Content [Eq. 11]	Data compression [72], text entropy in NLP [68], reinforcement learning decision-making [73,74].
Variance Loss [Eq. 12]	Regression error minimization [77,78], robust machine learning models, portfolio optimization [79], market trend prediction [81,82].
Gini Impurity [Eq. 13]	Decision tree splits [83,84], random forests [78], feature selection [126], multi-class classification [127].
Mean Absolute Deviation [Eq. 14]	Robust regression [85], outlier detection [86,88], anomaly detection [87], time series error analysis [89], exploratory data analysis [90].
Statistical Complexity Loss [Eq. 15]	Time series analysis [95], neural network information bottleneck [96], feature selection [97], ecological AI predictions [98], generative model diversity [98].
Simpson’s Diversity Index [Eq. 16]	Search result diversity [100], microbiome clustering [101], AI biodiversity modeling [102,103], synthetic data generation, social network diversity analysis.
Skewness Loss [Eq. 17]	Neural network generalization [106], anomaly detection [107], financial risk modeling [108], adaptive learning in imbalanced datasets [109].
Kurtosis Loss [Eq. 18]	Outlier detection in time series [112], financial fat-tail analysis [113], signal noise suppression [114], robustness in neural networks [115].
L2 Distribution Regularization [Eq. 19]	Probabilistic balance in classification [118], label smoothing in deep learning [117], interpretable predictions in NLP [119].
Sparsity Loss [Eq. 20]	Sparse autoencoders for feature interpretation [123], unsupervised learning with sparse priors [124], learning sparse latent representations [125].

Table 7. Summarizing the focus of applications and respective references for Information Theory-based, Probabilistic Dispersion-based, Geometric Distribution-based, and Regularization-based Unsupervised Objectives. Also see Figure ?? for detailed references to respective applications.

Unsupervised Objective	Applications
Shannon Entropy [Eq. 4]	Decision tree splits [27], clustering [28], entropy minimization for confident predictions [13], NLP for sequence unpredictability [30,31].
Perplexity Loss [Eq. 5]	Language model evaluation [35], improving fluency [31], machine translation, text generation, speech recognition.
Rényi Entropy [Eq. 6]	Neural network regularization [37], quantum entanglement [39], adaptive learning for dynamic models [41].
Tsallis Entropy [Eq. 7]	Statistical mechanics [44], clustering robust to outliers [46], anomaly detection [47], ecological diversity [48], quantum entanglement [50].
Minimum Description Length [Eq. 8]	Few-shot learning [54], reinforcement skill acquisition [55], dimensionality reduction in PCA [57], compact language model representations [59].
Collision Entropy [Eq. 9]	Cryptography key randomness [63], species dominance [65], class probability analysis for overfitting [66], quantum key distribution [60].
Normalized Entropy [Eq. 10]	Clustering quality evaluation [68], ecological biodiversity [69], feature selection [70].
Information Content [Eq. 11]	Data compression [72], text entropy in NLP [68], reinforcement learning decision-making [73,74].
Variance Loss [Eq. 12]	Regression error minimization [77,78], robust machine learning models, portfolio optimization [79], market trend prediction [81,82].
Gini Impurity [Eq. 13]	Decision tree splits [83,84], random forests [78], feature selection [126], multi-class classification [127].
Mean Absolute Deviation [Eq. 14]	Robust regression [85], outlier detection [86,88], anomaly detection [87], time series error analysis [89], exploratory data analysis [90].
Statistical Complexity Loss [Eq. 15]	Time series analysis [95], neural network information bottleneck [96], feature selection [97], ecological AI predictions [98], generative model diversity [98].
Simpson’s Diversity Index [Eq. 16]	Search result diversity [100], microbiome clustering [101], AI biodiversity modeling [102,103], synthetic data generation, social network diversity analysis.
Skewness Loss [Eq. 17]	Neural network generalization [106], anomaly detection [107], financial risk modeling [108], adaptive learning in imbalanced datasets [109].
Kurtosis Loss [Eq. 18]	Outlier detection in time series [112], financial fat-tail analysis [113], signal noise suppression [114], robustness in neural networks [115].
L2 Distribution Regularization [Eq. 19]	Probabilistic balance in classification [118], label smoothing in deep learning [117], interpretable predictions in NLP [119].
Sparsity Loss [Eq. 20]	Sparse autoencoders for feature interpretation [123], unsupervised learning with sparse priors [124], learning sparse latent representations [125].

5.2. Pairwise Comparison

Pairwise comparisons help distinguish genuine differences from mere reparameterizations. In Figure 10, Shannon entropy, perplexity, normalized entropy, and Tsallis entropy exhibit strongly aligned monotonic relationships, confirming that they often encode closely related uncertainty preferences. MDL and information content behave differently: their curves emphasize low-probability regions more sharply, which explains their distinctive optimization behavior.

Figure 11 shows a similar pattern among the dispersion objectives. Variance, Gini impurity, and MAD correlate strongly when evaluated on simple supports, while statistical complexity departs because it mixes uncertainty with a second structural term. Figure 11 highlights a different kind of separation: skewness and kurtosis respond nonlinearly to support asymmetry and tails, whereas L2 and sparsity express direct optimization biases. The pairwise plots therefore validate the taxonomy while also revealing where family boundaries overlap.

5.3. Behavioral Analysis

Derivative-based views expose how aggressively an objective responds to changes in predictive confidence. Figure 12 suggests five practical observations.

• Entropy-like objectives are smooth near balanced predictions. Shannon entropy, normalized entropy, and Tsallis entropy exhibit stable symmetry around $p=0.5$, which helps explain their favorable optimization behavior.
• Pointwise-surprise objectives can be steep near the boundary. Information content and MDL rise sharply as probabilities approach zero, making them sensitive to rare events but also easier to destabilize.
• Quadratic objectives are simple and predictable. Gini impurity, SDI, and L2 regularization have smoother derivatives and therefore tend to produce more stable updates.
• Higher-order moments are highly sensitive. Skewness and kurtosis respond strongly to extreme support configurations, which is useful when support geometry matters but risky for generic categorical outputs.
• Sparsity needs careful relaxation. The behavior of $L_0$-style sparsity is not naturally compatible with dense softmax outputs, reinforcing the need for thresholds or surrogate formulations.

5.4. Future Directions

The survey points to several promising directions for future work.

• Objective composition. Rather than using a single objective, future work should study principled combinations of uncertainty, concentration, and regularization terms.
• Support-aware design. Shape-sensitive objectives such as skewness and kurtosis need versions adapted to categorical outputs, ordinal labels, or learned supports.
• Relaxed sparsity for dense predictors. Practical sparsity objectives for modern neural outputs remain underdeveloped despite strong motivation from efficiency and interpretability.
• Evaluation beyond accuracy. For adaptation and foundation models, calibration, robustness, fairness, and energy efficiency should accompany classification metrics.
• Large-model adaptation. Probability-space objectives are especially attractive for source-free, test-time, and low-resource adaptation because they can be computed without labels.
• Benchmark standardization. A common benchmark suite for single-distribution objectives would make future comparisons more reproducible and more useful than isolated task-specific reports.

6. Conclusion

This survey reviewed 17 unsupervised objectives defined on single predictive distributions and organized them into four families. The study emphasizes a central point: although the literature contains many names, the underlying design space is structured by a small number of recurring ideas such as uncertainty, concentration, shape, and regularization. Making these relations explicit helps clarify when objectives are interchangeable, when they are support-dependent, and when they encode genuinely different inductive biases.

The theoretical review and the retained experiments lead to a practical conclusion. Probability-space objectives are useful tools for unlabeled optimization, but they should be selected by matching their bias to the task and the model state rather than by treating them as generic drop-in losses. This is especially relevant for modern adaptation settings in which labels are absent but predictive distributions are readily available.

7. Validity Threat

A review paper is only as reliable as the choices it makes about scope, inclusion, and interpretation [128]. As in other systematic and critical reviews, selection bias, heterogeneity of source studies, and publication bias are real concerns [129,130]. We mitigate these threats by focusing on foundational objectives rather than an exhaustive list of modified variants, by making algebraic relations explicit where possible, and by separating theoretical claims from empirical observations.

A second threat concerns the experimental section. The retained experiments are intentionally small-scale and illustrative; they should not be read as definitive rankings of the surveyed objectives. Their value lies in exposing qualitative differences under a controlled protocol. More comprehensive benchmarks across tasks, models, and adaptation settings remain necessary for stronger empirical generalization.