Information-Weight Interpretation of Probability: A Novel Information-Theoretic Perspective

1. Introduction

Probability is virtually ubiquitous, playing a role in almost all scientific disciplines (Hájek, 2002). Mathematically, probability is defined as a function satisfying Kolmogorov’s axioms, providing a framework for quantifying uncertainty. However, as D’Agostini (2017) argues, probability is not just a number between 0 and 1 that satisfies the axioms, “because such a ‘definition’ says nothing about what we are talking about.” Probability requires a practical interpretation to connect mathematical formalism to real-world contexts. The question of what probability represents in real-world contexts has sparked decades of philosophical and scientific debate and remains an open issue (Galavotti, 2017). For comprehensive surveys of these interpretive debates, readers may refer to Bunge (1981), Seidenfeld (2015), D’Agostini (2017), Hennig (2024), and Spiegelhalter (2024).

Existing interpretations of probability generally fall into two categories: subjective and objective. In the subjective interpretation (or Bayesian interpretation), probability reflects personal beliefs about a parameter (treated as a constant) or hypothesis, often referred to as the “degree of belief” (e.g., D’Agostini, 2017; Willink, 2025). In contrast, objective interpretations view probability as an inherent physical property of a system. Objective interpretations include frequency interpretation and propensity interpretation. The frequency interpretation defines probability as the limiting relative frequency in repeated trials. The propensity interpretation describes probability as a system’s tendency to produce an outcome, and therefore probability can be called the “degree of propensity” (Huang, 2023). While both subjective and objective interpretations provide valuable insights, they often fail to fully capture the intuitive ways people perceive probability in terms of information and informativeness.

In this paper, we propose an information-theoretic perspective, which defines probability as “information weight” to align with the intuitive and familiar concept “weight”. The following sections are structured as follows. Section 2 describes the proposed information-theoretic perspective on probability and probability density. Section 3 describes the relations of the proposed perspective with Shannon information theory (Shannon, 1948) and informity theory (Huang, 2025). Section 4 introduces information-theoretic terminology for probability concepts. Section 5 provides discussion and conclusion.

2. The Proposed Information-Theoretic Perspective on Probability and Probability Density

2.1. Discrete Systems

In the context of this paper, a discrete system is a system with a countable number of states. The state variable of a discrete system changes only at specific, distinct points (states). For example, tossing a coin is a discrete system with two specific, distinct states: "heads" and "tails."

Let X denote the state variable. Suppose X has n multiple states: $\{x_1,\dots x_n\}$, which is its state space. Each state x_i occurs multiple times m_i. Mathematically, this discrete system can be modeled by a multiset (X, m): $\{x_1^{m_1},x_2^{m_2},\dots x_n^{m_n}\}$, where $X=\{x_1,,\dots x_n\}$ is the underlying set of distinct elements (states), n is the number of elements, $m_i$ is the multiplicity, i.e., the number of times each element $x_i$ appears in the multiset, and $\sum _{j=1}^n{m}_j$ is the size (i.e., the total multiplicities) of the multiset.

Definition 1.

Each state $x_i$ is a piece of information about the state variable X (or the discrete system). For instance, the state “heads” or “tails” is a piece of information about the coin-tossing system.

Definition 2.

The multiplicity $m_i$ is defined as the information content of $x_i$, and the size of the multiset $\sum _{j=1}^n{m}_j$ is defined as the total information content of the discrete system.

Definition 3.

The information weight of $x_i$, denoted by ${W(x}_i)$, is defined as the ratio of $m_i$ to $\sum _{j=1}^n{m}_j$

$$W\left(x_i\right)={\displaystyle \frac{m_i}{\sum _{j=1}^n{m}_j}}.$$

(1)

That is, information weight is the relative information content of $x_i$. Note that the sum of the information weights of all the states of the discrete system is

$$\sum _{i=1}^n\left({\displaystyle \frac{m_i}{\sum _{j=1}^n{m}_j}}\right)=1.$$

(2)

That is, the total relative information content of the discrete system is 1.

Furthermore, the information weight function, denoted by $W\left(X\right)$, represents the distribution of information weight across all states of the discrete system.

Definition 4.

Assume that the discrete system has a mean state and each state $x_i$ is numerical. (e.g., measurement errors are real numbers). Let $\overline{x}$ denote the mean state, which is defined as

$$\overline{x}=\sum _{i=1}^n{\displaystyle \frac{m_i{x}_i}{\sum _{j=1}^n{m}_j}}=\sum _{i=1}^n{W(x}_i)x_i.$$

(3)

Therefore, the mean state $\overline{x}$ is a weighted average of all states $x_i$ of the discrete system, explaining why we term ${W(x}_i)$ the information weight.

Now, we relate the discrete system modeled by the multiset to a discrete random variable X with a probability mass function (PMF) $P\left(X\right)$. Each state $x_i$ in the discrete system corresponds to a possible outcome of X and $P\left(x_i\right)$ represents its probability. From this, it is easy to see that the probability $P\left(x_i\right)$ is equal to the information weight ${W(x}_i)$

$$P\left(x_i\right)=W\left(x_i\right)={\displaystyle \frac{m_i}{\sum _{j=1}^n{m}_j}},$$

(4)

and

$${\sum }_{i=1}^{n}P\left(x_i\right)=\sum _{i=1}^n\left({\displaystyle \frac{m_i}{\sum _{j=1}^n{m}_j}}\right)=1.$$

(5)

Equation (4) establishes the equivalence between information weight and probability, while Eq. (5) confirms the normalization of probabilities. Therefore, the PMF $P\left(X\right)$ and the information weight function $W\left(X\right)$ are equivalent, representing the informational structure of a discrete system or discrete distribution.

The information weight ${W(x}_i)$ quantifies how strongly a state contributes to the system’s informational structure. Higher information weight means the state (or outcome) is more prevalent or predictable. For example, consider a fair six-sided die rolled 60 times, with outcomes {1, 2, 3, 4, 5, 6} and multiplicities {10, 10, 10, 10, 10, 10}, the information weight of each outcome is ${W(x}_i)=10/60\approx 0.167$. Therefore, each outcome has equal informational weight, reflecting a uniform distribution where no single outcome dominates the system’s informational structure.

2.2. Continuous Systems

In the context of this paper, a continuous system is a system where its state variable varies continuously, taking any value within a range. For example, sediment particle sizes, ranging from pebbles to microscopic clay, form a continuous system (i.e., a continuous distribution of particle sizes).

Let Y denote the state variable, taking values $y\in [a,b]$, where $[a,b]$ is a continuous state space. To extend the discrete framework, we divide [a, b] into $n(∆y)$ number of small bins of size $∆y={\displaystyle \frac{b-a}{n(∆y)}}$, each bin i ($i=1,2,...n(∆y))$ contains the element ${\delta }_i\in [y_i-{\displaystyle \frac{∆y}{2}},y_i+{\displaystyle \frac{∆y}{2}}]$ centered at $y_i$. Then, this discretized continuous system can be modeled by a multiset (Y, m): $\{{\delta }_1^{m_1(∆y)},{\delta }_2^{m_{2(∆y)}},\dots {\delta }_{n(∆y)}^{m_{n(∆y)}(∆y)}\}$, where $Y=\{{\delta }_1,,\dots {\delta }_{n(∆y)}\}$ is the underlying set of distinct elements, $m_i(∆y)$ is the multiplicity, i.e., the number of data points that fall into the bin i, and $\sum _{j=1}^{n(∆y)}{m}_j(∆y)$ is the size of the multiset. This multiset corresponds to a histogram commonly used to visualize a set of continuous data, with ${\delta }_i$ on the x axis and $m_i(∆y)$ on the y axis, such as data for sediment particle sizes. Smaller bin size $∆y$ increases the bin number $n(∆y)$, but reduces the multiplicity $m_i(∆y)$ of each ${\delta }_i$, thereby refining the histogram’s approximation to the continuous distribution.

Definition 5.

Each value $y_i\in [a,b]$ is a piece of information about the continuous state variable Y. For example, the sediment particle size $y_i$=0.003mm is a piece of information about Y.

Definition 6.

The multiplicity $m_i(∆y)$ divided by the bin size $∆y$ is the information content per unit state space, and $\sum _{j=1}^{n(∆y)}{m}_j(∆y)$ is the information content of the discretized continuous system. Then, the information weight per unit state space is given by

$${\displaystyle \frac{W\left({\delta }_i\right)}{∆y}}={\displaystyle \frac{m_i(∆y)/∆y}{\sum _{j=1}^{n(∆y)}{m}_j(∆y)}}.$$

(6)

Definition 6.

The information density at $y_i$, denoted by $D\left(y_i\right)$, is defined as

$$D\left(y_i\right)=\underset{∆y\to 0}{\lim }\left({\displaystyle \frac{W\left({\delta }_i\right)}{∆y}}\right)=\underset{∆y\to 0}{\lim }\left({\displaystyle \frac{m_i(∆y)/∆y}{\sum _{j=1}^{n(∆y)}{m}_j(∆y)}}\right),$$

(7)

which reflects the informational contribution per unit state space to the informational structure of the continuous system. Furthermore, the information density function, denoted by $D\left(Y\right)$, represents the distribution of information density across the values $y\in [a,b]$.

Now, we relate the continuous system to a continuous random variable Y with probability density function (PDF) $p\left(Y\right)$ with the support $[a,b]$. Then, it is easy to see that the probability density $p\left(y_i\right)$ equals the information density $D\left(y_i\right)$

$$p\left(y_i\right)=D\left(y_i\right)=\underset{∆y\to 0}{\lim }\left({\displaystyle \frac{m_i(∆y)/∆y}{\sum _{j=1}^{n(∆y)}{m}_j(∆y)}}\right),$$

(8)

with the normalization

$${\int }_a^{b}p\left(y\right)dy=\underset{∆y\to 0}{\lim }\left({\sum }_{i=1}^{n(∆y)}p\left(y_i\right)∆y\right)=1.$$

(9)

Thus, the PDF $p\left(Y\right)$ and the information density function $D\left(Y\right)$ are equivalent, representing the informational structure of a continuous system.

Moreover, the information weight of an interval [$y_{1,}{y}_2]$ is given by

$$W{(y}_{1,}{y}_2)={\int }_{y_1}^{y_2}yD\left(y\right)dy,$$

(10)

which corresponding to the coverage probability of the interval.

3. Relations with Shannon Information Theory and Informity Theory

This section explores how the information-weight interpretation relates to Shannon information theory (Shannon, 1948) and informity theory (Huang, 2025), highlighting correspondences and distinctions.

3.1. Relations with Shannon Information Theory

In discrete systems, each state $x_i$ has information weight $D\left(x_i\right)=P\left(x_i\right)$, representing its relative information content. This should not be confused with Shannon information content, defined as (Shannon, 1948)

$$I_S\left(x_i\right)=-{\mathrm{l}\mathrm{o}\mathrm{g}}_{2}P\left(x_i\right),$$

(11)

which is the logarithm of the probability’s reciprocal. We term that reciprocal

$$R\left(x_i\right)={\displaystyle \frac{1}{P\left(x_i\right)}}={\displaystyle \frac{1}{W\left(x_i\right)}}$$

(12)

the information rarity. $R\left(x_i\right)$ quantifies how uncommon the state $x_i$ is relative to the total occurrence of all states in the discrete system. Notably, information rarity and information weight are inversely related: as information rarity increases, information weight decreases, and vice versa.

In the multiset framework, information rarity can be expressed as

$$R\left(x_i\right)={\displaystyle \frac{\sum _{j=1}^n{m}_j}{m_i}}.$$

(13)

Then, Shannon information content can be written in terms of information rarity

$$I_S\left(x_i\right)={\mathrm{l}\mathrm{o}\mathrm{g}}_{2}R\left(x_i\right).$$

(14)

Accordingly, the discrete entropy can be written as

$$H\left(X\right)=\sum _{i=1}^{n}P\left(x_i\right){\mathrm{l}\mathrm{o}\mathrm{g}}_{2}R\left(x_i\right),$$

(15)

which means that the discrete entropy is the expectation of the logarithm of information rarity.

Intuitively, information rarity is related to uncertainty. Higher rarity indicates greater uncertainty, explaining Shannon entropy as a measure of the uncertainty of a system.

Moreover, in continuous systems, the differential entropy (which is also called continuous entropy) can be written as

$$H\left(Y\right)=-{\int }_a^{b}p\left(y\right){\mathrm{l}\mathrm{o}\mathrm{g}}_{2}p\left(y\right)dy={\int }_a^{b}p\left(y\right){\mathrm{l}\mathrm{o}\mathrm{g}}_{2}R\left(y\right)dy,$$

(16)

where $R\left(y\right)=1/p\left(y\right)$, which can be called information rarity density. Thus, the differential entropy is the expectation of the logarithm of information rarity density.

3.2. Relations with Informity Theory

Informity theory, proposed by Huang (2025), is an information-theoretic framework where probability is viewed as relative information content, and informity quantifies the mean relative information content of a system. The proposed information-weight interpretation operationalizes relative information content using the multiset framework, equating probability, information weight, and relative information content as $P\left(x_i\right)=W\left(x_i\right)={\displaystyle \frac{m_i}{\sum _{j=1}^n{m}_j}}$. Accordingly, the discrete informity $\beta \left(X\right)$ can be expressed in terms of information weight

$$\beta \left(X\right)=\sum _{i=1}^n{P\left(x_i\right)}^2=\sum _{i=1}^{n}P\left(x_i\right)W\left(x_i\right)=\mathrm{E}\left[W\left(X\right)\right].$$

(17)

That is, the discrete informity is the expectation of the information weight.

Moreover, the continuous informity $\beta \left(Y\right)$ can be expressed in terms of information density

$$\beta \left(Y\right)={\int }_a^b\int {p\left(y\right)}^{2}dy={\int }_a^b\int p\left(y\right)D\left(y\right)dy=\mathrm{E}\left[D\left(Y\right)\right].$$

(18)

That is, the continuous informity is the expectation of the information density.

4. Information-Theoretic Terminology for Probability Concepts

This section introduces terminology for probability concepts within the proposed information-theoretic framework, building on the information-weight and information-density interpretations established in Sections 2.1 and 2.2. By comparing the information-theoretic perspective with Bayesian and propensity perspectives, we define “degree of information,” “information spectrum,” and “information interval” to reflect the philosophical emphasis on informational structure of a system, contrasting these with subjective belief and causal propensity.

In various perspectives on probability, the mathematical concepts of probability, probability distribution, and probability interval are assigned distinct terminologies that reflect their philosophical underpinnings. In the Bayesian perspective, probability is termed the “degree of belief” (e.g., D’Agostini, 2017; Willink, 2025), the probability distribution can be named the “belief spectrum,” and the probability interval is often known as the “credible interval.” In the propensity perspective, these are referred to as the “degree of propensity,” “propensity spectrum,” and “propensity interval,” respectively (Huang, 2023). Similarly, in the proposed information-theoretic perspective, probability can be called the "degree of information," the probability distribution can be termed the "information spectrum," and the probability interval can be called the "information interval." These terminologies are not mere labels; they align with distinct philosophical views. The "degree of belief" emphasizes subjective confidence in a parameter or hypothesis, the "degree of propensity" highlights the causal tendency of an outcome, and the "degree of information" underscores the informational weight of a state (or outcome). Similarly, the "belief spectrum” describes the distribution of personal beliefs, the "propensity spectrum” describes the distribution of propensity, and the "information spectrum" describes the distribution of information weight or density.

Furthermore, from the proposed information-theoretic perspective, a probability interval with a specified coverage probability can be termed the “information interval” with a specified “degree of information”, representing a range of values supported by the information weight equal to the coverage probability. For example, consider the sample mean (statistic) $\overline{Y}$ of n observations drawn from a normal distribution $Y~N(\mu ,{\sigma }^2)$, where μ is the mean and σ is the standard deviation. Its sampling distribution, $N(\mu ,{\displaystyle \frac{{\sigma }^2}{n}}),$serves as the information spectrum, describing the informational structure of the continuous system. The information interval covering sample means supported by an information weight of 0.95 is given by

$$\left(\mu -1.96{\displaystyle \frac{\sigma }{\sqrt{n}}},\mu +1.96{\displaystyle \frac{\sigma }{\sqrt{n}}}\right),$$

(19)

which is equivalent to the probability interval with a coverage probability of 0.95.

When $\mu$ and σ are unknown, the sample mean $\overline{y}$ estimates $\mu$ and $s/c_{4,n}$ estimates σ, where s is the sample standard deviation and $c_{4,n}$ is the bias-correction factor for s. This factor is given by (e.g., Wadsworth, 1989)

$$c_{4,n}=\sqrt{{\displaystyle \frac{2}{n-1}}}{\displaystyle \frac{\Gamma \left(\frac{n}{2}\right)}{\Gamma \left(\frac{n-1}{2}\right)}},$$

(20)

where Г(.) represents the Gamma function.

Using these estimates, we can obtain an empirical information spectrum for the sample mean $\overline{Y}$, which follows a normal distribution $N(\overline{y},{\displaystyle \frac{s^2}{c_{4,n}^{2}n}})$ (Huang, 2018). Then, the empirical information interval covering sample means supported by a nominal information weight of 0.95 (i.e., nominal coverage probability of 0.95) can be constructed using this empirical information spectrum. It is given by

$$\left(\overline{y}-1.96{\displaystyle \frac{s}{c_{4,n}\sqrt{n}}},\overline{y}+1.96{\displaystyle \frac{s}{c_{4,n}\sqrt{n}}}\right).$$

(21)

The half-width of this empirical information interval is $1.96{\displaystyle \frac{s}{c_{4,n}\sqrt{n}}}$, which corresponds to the unbiased expanded uncertainty in measurement uncertainty analysis (e.g., Huang, 2018).

5. Discussion and Conclusion

The proposed information-weight interpretation of probability is intuitive because it maps probability onto the familiar concept of weight. This aligns with D'Agostini’s (2017) view that we can understand probability statements by mapping them in “some ‘categories’ of our mind, as we do with space and time.” For example, stating “the probability of rain tomorrow is 0.8” can be rephrased as “raining tomorrow has an information weight of 0.8”. Thus, probability, as information weight, can be viewed as a “property” of a piece of information (e.g., a state, outcome, or hypothesis). This is analogous to “strength” is a “physical property” of a material, such as “AISI 302 stainless steel has a tensile strength of 585 MPa.”

The proposed information-weight interpretation of probability provides a novel philosophical perspective. Rather than linking probability to personal beliefs (Bayesian), system tendencies (propensity), or long-run frequencies (frequentist), this framework emphasizes probability as the informational weight that supports a state, outcome, or hypothesis and reflects its contribution to the informational structure of a system. In addition, this interpretation holds regardless of how probabilities are derived, whether from professional judgment, physical models, or empirical frequencies, making it independent of probability assignment methods. Moreover, by defining probability distributions as information spectra and probability intervals as information intervals, this framework unifies the treatment of probability and probability density under a single information-theoretic perspective.