On the Optimal Point of the Weighted Simpson Index

1. Introduction

The Simpson index concerning a population distributed among k categories or classes is defined as

$$S=\sum _{i=1}^k{p}_i^2,$$

where $p_i$ denotes the probability (or proportion of occurrences) of class i. So, one has $p_i\ge 0$ and ${\sum }_{i=1}^k{p}_i=1$, and therefore S is defined on a $k-1$ simplex. This index equals the probability that two elements taken at random from the population of interest belong to the same category or class. The value of Simpson’s index ranges from $1/k$ to 1, with 1 representing no diversity, so, the larger the value of $S$ the lower the diversity. The name “Simpson index” roots from the influential 1949 paper by Edward Hugh Simpson entitled “Measurement of Diversity” [1], wherein he introduced what he called a measure of concentration defined in terms of population constants, with the minimum concentration equaling the maximum diversity. The Simpson index became a widely used quantitative metric in ecological and biodiversity studies as a tool for assessing and quantifying the diversity and evenness of species within ecological communities. It also applies to other biological problems, including biomedical sciences, such as measuring diversity concerning immunity in response to viral infections (e.g., [2]).

However, it is also acknowledged that the original mathematical concept formulation was used in cryptanalysis as far back as the 1920s and 30s — therein named probability of monographic coincidence — by the American cryptanalysts William Friedman and Solomon Kullback (e.g., [3]). It is relevant to note that the Italian statistician Corrado Gini had already applied the quantity $1-{\sum }_{i=1}^k{f}_i^2$ as early as 1912. He defined the index with relative frequencies $f_i$ computed from large samples, referring to it as an index of mutability for disconnected (qualitative) variables [4]. This quantity became later known as the “Gini-Simpson index”, a name adopted in the 1980s by the eminent statistician C. R. Rao (e.g., [5,6]), who restated it with probabilities as $G={\sum }_{i=1}^k{p}_i(1-p_i)=1-{\sum }_{i=1}^k{p}_i^2$. For instance, Jian et al. [7] consider a “Livelihood Simpson Index” which in fact is a Gini-Simpson index. Obviously, given a probability distribution $\left(p_1, \dots ,p_k\right)$, the Simpson and Gini-Simpson indices correspond to complementary events, verifying $S+G=1$.

The use of a weighted version of the Simpson index appears to have been first reported in 1992, when Nowak and May [8] conceived the effective immune response against the virus population composed of different strains in the context of HIV infections, then revisited two years later [9].

Some refer to the weighted Simpson index when they are actually dealing with the weighted Gini-Simpson index (e.g., [10,11]). The weighted Gini-Simpson index is defined as ${GS}_w={\sum }_{i=1}^k{w}_i{p}_i(1-p_i)$, a concave function, differentiable in the interior of a simplex, with an identifiable maximum value for which a method to determine the optimal point (maximizer) was framed based on the fact that one is dealing with feasibility values associated with the constraints of the simplex [12] – namely, that the optimal coordinates must verify $p_i^{*}\ge 0$ – what was not taken into account in [11] and may lead to miscalculations.

Yet, Kasulo and Perrings used a price-weighted Simpson index [13] to assess scenarios relative to the connection between the diversity of catch in a multi-species fishery and profit maximizing regimes. And one can still find an inverse form of the weighted Simpson index in [14], which the authors clarify could be interpreted as a weighted version of Hill’s number $N_2$, concerning the unifying notation that was proposed therein [15]. Also, one can see that J. Ma used a symmetric form of the weighted Simpson index, building what he named a “comprehensive weighted Gini-Simpson index” [16].

In recent years, there has been a panoply of diversity, phylogenetic, and dissimilarity indices with a focus on biology (e.g., [17,18]), most of them associated with developments related to Rényi measures of entropy [19], including general reviews on the subject (e.g.,[20]), and a plethora of recent applications (e.g., [21]). There are also many publications within the scope of diversity in the social sciences – a process that had a cornerstone with the reference work of Patil and Taillie [22] addressing linguistic diversity, industrial concentration and income inequality – with applications either in economics (e.g., [23]) or demographics, such as the weighted Rényi’s entropy for lifetime distributions [24].

However, with regard to the weighted Simpson index, it seems that an analytical study addressing the optimal point (minimizer) and the optimal value (minimum), has not yet been published.

2. The weighted Simpson index

Herein, we define the weighted Simpson index, concerning a population distributed among k categories or classes, as:

$$S_w=\sum _{i=1}^k{w}_i{p}_i^2$$

(1)

where $p_i$ denotes the probability (or proportion of occurrences) of class i and $w_i>0$ is a weight assigned to that class, altogether defining a vector of positive real values ${w=\left(w_1,\dots ,w_k\right)}^t$. For now, we have decided not to impose any extra conditions on the weights, leaving this matter to be discussed later. Our current focus is on understanding the broader context.

Weights allow to consider various features for the classes. In the context of biodiversity, these features may be related for example to environmental benefits, conservation importance, vulnerability or economic value of species – a subject that was already emphasized at least since the beginning of the 1980s, exemplified with biomass and other importance values [25]. The environmental benefits may include the ecological roles and contributions of a species to its ecosystem. Conservation importance further adds depth to the evaluation, addressing the urgency and priority assigned to preserving certain species to maintain ecosystem stability. Vulnerability is another important factor that weights help account for. Vulnerable species—those in danger of going extinct—need to be given particular attention during the assessment process. Economic value represents yet another dimension where weights come into play. The assessment of species in terms of their economic contributions, whether through ecosystem services, medicinal properties, or commercial value, demands a careful weighting to reflect their overall importance. Yet, one should be aware of the complexities and entanglements associated with a community structure when dealing with several trophic levels, where, in addition to competition, there may occur symbiosis and predator-prey interactions conveying a hierarchical structure in a networked ecological community (e.g., [26]).

2.1. Optimizing the weighted Simpson index

The weighted Simpson index (1) is defined on the simplex ${\Delta }^{k-1}=\left\{\left(p_{1,}\dots ,p_k\right)\in {\mathbb{R}}^k: p_i\ge 0, {\sum }_{i=1}^k{p}_i=1\right\}$ and is a convex function. So, $S_w$ attains its (global) maximum at the boundary of ${\Delta }^{k-1}$ and its (global) minimum at the interior of ${\Delta }^{k-1}$. Clearly, the maximum is attained when all except one of the $p_i$ are zero, and therefore one has $\mathit{max}{S}_w=\underset{i}{\mathit{max}}{w}_i$.

The minimum can be assessed with the method of Lagrange multipliers for finding the minima of $S_w$ subject to the equality constraint ${\sum }_{i=1}^k{p}_i=1$. As $S_w$ is differentiable in the interior of ${\Delta }^{k-1}$, one can build the Lagrangian function

$$L_w\left(p_1,\dots ,p_k;\right)=S_w\left(p_1,\dots ,p_k\right)+\left({\sum }_{i=1}^k{p}_i-1\right),$$

and find its extrema. Equating partial derivatives to zero, we get:

$$\left\{\begin{array}{c}\frac{\partial L_w}{\partial p_i}=0\\ \frac{\partial L_w}{\partial }=0\end{array}\right.⟺\left\{\begin{array}{c}2w_i{p}_i+=0\\ {\sum }_{i=1}^k{p}_i-1=0 \end{array}\right.$$

(2)

From (2) we conclude that for any specific j the relationship $w_j{p}_j=-\lambda /2 ⟺ p_j=-\frac{\lambda }{2w_j}$ holds. Now, using the constraint ${\sum }_{j=1}^k{p}_j=1$ one has the following equivalence $1=-\frac{\lambda }{2}$ ${\sum }_{i=1}^k\frac{1}{w_i} ⟺ 1=$ $w_j{p}_j$ ${\sum }_{i=1}^k\frac{1}{w_i}$ and we get the optimal coordinates of the minimum point given by:

$$p_j^{*}=\frac{1}{w_j\left({\sum }_{i=1}^k \frac{1}{w_i}\right)} \mathrm{for} j=1,\dots k.$$

(3)

Also, using (3) one can evaluate the minimum value of the weighted Simpson index (1) as follows:

$$S_w^{*}=S_w\left(p_1^{*}, \dots , p_k^{*}\right)={\sum }_{j=1}^k{w}_j{\left(\frac{1}{w_j\left({\sum }_{i=1}^k\frac{1}{w_i}\right)}\right)}^2=\frac{1}{{\left({\sum }_{i=1}^k\frac{1}{w_i}\right)}^2}{\sum }_{j=1}^k\frac{1}{w_j}=\frac{1}{{\sum }_{i=1}^k\frac{1}{w_i}}$$

(4)

2.2. Some further comments on the minimizer

Note that the minimum value of the weighted Simpson index (4) is related to the harmonic mean of the weights $\mathrm{H}\left(w\right)$ by $S_w^{*}=\mathrm{H}\left(w\right)/k$. The name “harmonic mean” is said to have been proposed by Archytas and Hippasus and adopted by Nicomachus in accordance with the view of the geometrical harmony of the cube, because it has 12 edges, 8 vertices, and 6 faces, and 8 is the mean of 12 and 6 according to the theory of harmonics [27] (pp. 85-86), meaning the harmonic mean of those quantities. In general, the harmonic mean of $k$ nonzero numbers is the reciprocal of the arithmetic mean of the reciprocals of those numbers, and is appropriate for averaging rates over constant numerator units [28]. As a typical example, if a set of investments are invested at different interest rates, and they all give the same income, the unique rate at which all of the capital tied up in those investments must be invested to produce the same revenue as given by the set of investments, is equal to the harmonic mean of the individual rates [29] (p. 240).

The special case of all weights being equal to 1 leads to $p_i^{*}=1/k, \forall i$ and to the minimum value $S_w^{*}=1/k$ as can be seen from expressions (3) and (4), and also because in this case $S_w=S$ with $S$ being the (unweighted) Simpson index whose minimum is $1/k$.

Rewriting the optimal coordinates in (3) as

$p_j^{\mathrm{*}}=\frac{1}{{1+w}_j\left(\sum _{i\ne j}\frac{1}{w_i}\right)}$it is straightforward to conclude that the weights are driving forces of the values of the coordinates of the minimizer, operating reciprocally: when the weight attached to a specific class increases and all the others keep invariant, the corresponding optimal coordinate decreases; and when a weight attached to another class increases, all the others being invariant, the original class increases its optimal coordinate.

If one considers the valuation of the classes of a distribution in the usual sense of importance assessment with an ordering of positive real numbers, expecting that $v_m>v_n$ would promote the result $p_m^{*}>p_n^{*}$, then one should be aware that the weights associated with the optimal point (3) would not be the values $\left\{v_m,v_n\right\}$ but could possibly be conceived like $w_m=1/v_m$ and $w_n=1/v_n$ instead.

2.3. Normalization

The use of normalized indices in applications is important for several reasons. Normalized indices provide a standardized scale, usually ranging from 0 to 1, or 0% to 100%, irrespective of the specific scale or magnitude of the data. This standardization allows for direct comparisons between different datasets or populations and remains consistent across different contexts and scales.

In the case of the weighted Simpson index this can be done in a classic way defining the normalized weighted Simpson index as

$s_w=\frac{S_w-\mathrm{m}\mathrm{i}\mathrm{n}\left(S_w\right)}{\mathrm{m}\mathrm{a}\mathrm{x}\left(S_w\right)-\mathrm{m}\mathrm{i}\mathrm{n}\left(S_w\right)}=\frac{S_w{\sum }_{i=1}^k\frac{1}{w_i}-1}{\mathrm{m}\mathrm{a}\mathrm{x}\left(w_i\right){\sum }_{i=1}^k\frac{1}{w_i}-1}$However, this normalization eliminates the effect of the number of classes in the distribution (e.g., species in a community). For example, in the case of all weights equal to 1, the normalized weighted Simpson index of a population with $k$ species uniformly distributed is always 0 and thus independent of the number of species. The fact that normalized indices of diversity can be misleading has already been mentioned by several authors (e.g., [11]).

In the case of a weighted index, it may be relevant to normalize the weights so that the index becomes dimensionless and independent of the order of magnitude of the weights. For the weighted Simpson index, this normalization can be done, for example, by imposing the condition ${\sum }_{i=1}^k{w}_i=1$. This corresponds to dividing each non-normalized weight $w_i$ by the sum of all the weights. As $S_{\mathsf{\beta }w}=\mathsf{\beta }{S}_w$ for $\mathsf{\beta }>0$, this normalization procedure does not affect the results of the previous sections, namely, for the weighted Simpson index with normalized weights $S_{\mathit{W}}$, the optimal coordinates of the minimum point are as in (3) and the value of the minimum of $S_{\mathit{W}}$ is as in (4).

2.4. The inverse problem

The inverse procedure relative to the weighted Gini-Simpson was formulated recently [30]. Now, we consider the analogous problem concerning the weighted Simpson index, stated as: given a minimum point of $S_w$ denoted by $\left(p_1^{\mathrm{*}},\dots , p_k^{\mathrm{*}}\right)$, verifying both $0<p_j^{*}<1$ and ${\sum }_{j=1}^k{p}_j^{*}=1$, what would be a set of weights able to generate that solution? The answer to this question follows straightforward: recalling (3), the weights must be chosen to be inversely proportional to the optimal coordinates $p_j^{*}$ with the proportionality constant equal to $S_w^{*}$ (4), as we can see rewriting:

$$p_j^{\mathrm{*}}=\frac{1}{w_j\left({\sum }_{i=1}^k\frac{1}{w_i}\right)}⟺\frac{1}{p_j^{\mathrm{*}}}=w_j\left({\sum }_{i=1}^k\frac{1}{w_i}\right)⟺\frac{1}{p_j^{\mathrm{*}}}=w_j\frac{1}{S_w^{*}}⟺w_j=\frac{S_w^{*}}{p_j^{\mathrm{*}}}.$$

For non-normalized weights, there are infinitely many solutions to the inverse problem, parameterized by the minimum $S_w^{*}$. For normalized weights, using the condition ${\sum }_{j=1}^k{w}_j=1$, one gets ${\sum }_{j=1}^k\frac{S_w^{*}}{p_j^{\mathrm{*}}}=1$ $⟺$ $S_w^{*}=1/\left({\sum }_{j=1}^k\frac{1}{p_j^{\mathrm{*}}}\right)$ and so the weights must be chosen as:

$$w_i=\frac{1}{p_i^{*}\left({\sum }_{j=1}^k\frac{1}{p_j^{\mathrm{*}}}\right)} \mathrm{for} i=1,\dots ,k.$$

(5)

It should be noted that from the condition that the sum of the weights equals 1 it follows that ${\sum }_{i=1}^k\frac{1}{p_i^{*}}={\sum }_{i=1}^k\frac{1}{w_i}$.

3. Final remarks

We have presented a detailed analytical study of the optimization problem associated with the weighted Simpson index. The core result is that at the optimal point one has $w_i{p}_i^{*}=w_j{p}_j^{*}$ for $i,j=1,\dots ,k$, and also, that for all i, one gets $w_i{p}_i^{*}=S_w^{*}$, the minimum value of the index. So, there is a trade-off between the weights and the optimal probabilities (or proportions of occurrences) in what could be seen as an equilibrium condition. The fact that Nowak [8,9] has used the weighted Simpson index as a Lyapunov function to assess an antigenic diversity threshold, seems compatible with an equilibrium point perspective, which could also be used in a broad sense concerning different problems within several scientific fields.

Furthermore, for a random variable $W$ with values corresponding to the previous weights ${\left\{w_i\right\}}_{i=1,\dots ,k}$, and probability function given by $\mathrm{Pr}\left[W=w_i\right]=p_i^{*}$, with $p_i^{*}$ defined as in (3), computing the mean value of $W$ entails $E\left[W\right]={\sum }_{i=1}^k{w}_i{p}_i^{*}=k{S}_w^{*}$, which equals the harmonic mean of the weights, meaning $E\left[W\right]=\left(w\right)$.