The Self-Inconsistent Nature of the h-Index Determination

1. Introduction

Since its original proposal the h-index (Hirsch, 2005) has become in the more popular method for the evaluation of the impact of individual academic researchers. It been even applied to academic journals or institutions (Braun et al., 2006; Sidiropoulos et al., 2007). The definition of the h-index is (Hirsch, 2005; p. 16569) “A scientist has index h if h of his or her N_p papers have at least h citations each and the other (N_p – h) papers have ≤h citations each”, where N_p is the number of the a researcher over n years”. Besides, the h-index must be corrected for the time of research activity in order to permit the comparison of researcher in different career stage (Hirsch, 2005).

Certainly, the h-index is currently highly influential in the academic world and in scientific thought. There have been proposed a multitude of modified/extended versions or alternative indexes somewhat related (e.g., Egghe, 2006; Egghe and Rousseau, 2006, 2008; Sidiropoulos et al., 2007; Hirsch, 2010, 2021; Barnes, 2016). There have also been criticisms from detractors (Brito and Navarro, 2021). For example, it has been considered to give inconsistent results when comparing the evolution of different researchers in specific situations (Waltman and van Eck, 2012). In this brief note I show in a simple way that, when considering the deep significance of impact of individual citations, the h-index is self-inconsistent. Much of the present-day manner to evaluate academic researcher is therefore greatly relying on an index with an inaccurate definition. I also show that a more rigorous definition, but following a similar rationale spirit is cumbersome, and not easy to use.

A scientific document is only cited, at least in most cases, in other documents which consider it as relevant for their own topic, and therefore their potential citation number is conditioned in a complex way by the specific field size (number of researchers, research groups, journals, issues, references by documents, etc) and dynamics. Although h-indexes should only be compared for researchers in the same research field (Hirsch, 2005; Hirsch and Buela-Casal, 2014), it is frequently used, greatly in private situations, for comparing researchers in different fields, and there have been some attempts to find relative h-indexes (e.g., Dias, 2012). Thus, here I also present an evaluation of the consistency of the use of the h-index for comparing researcher performances in different research fields.

2. Rationale

As recognized by Hirsch (2005) the h-index value of a given author corresponds to a exact number of citations, such that

$$h^2=r_h$$

(1)

where r_h is the minimum number of citations of an author with a given h-index value. In other words, the minimum number of citations defining the h-index value of that author. The ratio between r_h and the total number R of citations in the scientific literature (which includes the citations made by all the documents by whichever author whose references are counted) is

$$P={\displaystyle \frac{r_h}{R}}={\displaystyle \frac{h^2}{R}}$$

(2)

P is the proportion of the total reference citation represented by the h-index value of an author, and it could be taken as a proxy for the total impact of an author. Equation (2) does not suppose any advantage on the total citation counting for that author, but serves to make clear the very limitation of the h-index concept. Obviously, the real weight of an individual citation is

$$W=R^{-1}$$

(3)

The big question for evaluating the deep significance of the weight of each individual citation, and therefore the consistency of the h-index determination, is how properly defining R. For example, R cannot be taken as an all-times value, because it would imply that the estimates of authors impact would be influenced by the amount of past scientific production. R cannot be measured from author publication debut, because P would continuously decrease for retired researchers. R must therefore be taken therefore for a time interval (i.e. the n years time in the definition by Hirsch), but when considering the well-known fact that the numbers of published documents per year (and hence R) increases with time (Bornmann and Mutz, 2015; Savage and Olejniczak, 2022), whichever long interval produces a bias between the real value of the citations obtained in distinct moments of this interval. Thus, the real weight of an individual citation is dependent on the citation moment. Since the rational of the h-index implies that whichever citation to documents published by the author have the same value, the inescapable consequence is that the h-index is self-inconsistent.

For the sake of simplicity, we can use D, the number of potentially citatory documents, instead R. We then obtain more handily, but mathematically less rigorous (some scientific journals limit the number of citations per document), versions of the Equations (2) and (3)

$$P*={\displaystyle \frac{h^2}{D}}$$

(4)

and

$$W*=D^{-1}$$

(5)

where the asterisk denotes the alternative calculation as a function of D instead R. However, the self-inconsistency problem of the h-index remains unchanged, due to that the definition of D, and therefore the calculation of W*, continue to be time-dependent.

As an alternative to citation, we could consider use a “citation value”, for example

$$V_R={\displaystyle \frac{R_{py}}{R_{cy}}}$$

(6)

or

$$V_D={\displaystyle \frac{D_{py}}{D_{cy}}}$$

(7)

where R_py and D_py, and R_cy and D_cy are the total number of citations and documents published in, respectively, the publication and citation year of the cited document. Equations (6) and (7) account for the progressive publication increasing, and consequently its effect on W and W* described by Equations (3) and (5) respectively. If again by the sake of simplicity we use the document number, the total value of the citations obtained by a given document is

$$V_{Dtotal}={\displaystyle \sum _i\frac{D_{pyi}}{D_{cyi}}}$$

(8)

Paraphrasing Hirsch (2005) a V-index can be defined as: A scientist has index V if V of his or her N_p papers have at least a citation value of V each and the other (N_p – V) papers have ≤V citation value. This V-index is more rigorously defined than the h-index, but it is not easy to be managed, and therefore it does not satisfy the original “simplicity spirit” of the proposal of Hirsch (2005).

3. Comparison of h-Index Values in Different Research Fields

Consider the h-index of two researchers, denoted by 1 and 2, working in different research fields. If both researchers have a same h-index value, then from Equation (2) we would have

$$P_1{R}_1=P_2{R}_2$$

(9)

which implies a inverse relation between the probability of a document to be cited and the total number of citations in the respective research fields. Thus, the h-index of two researchers of different fields cannot be straightforwardly compared (unless we find a reason to justify the condition imposes by equation (9) in the real world).

In theory, we can still compare the performance of two researcher through the ratio between the respective h-index values, such

$${\displaystyle \frac{h_2}{h_1}}=\sqrt{{\displaystyle \frac{P_2{R}_2}{P_1{R}_1}}}\propto \sqrt{{\displaystyle \frac{R_2}{R_1}}}$$

(10)

This implies that the relative performance between both researcher as measured by their respective h-indexes is proportional to the square root of the ratio of respective field citation sizes. If, for the sake of simplicity, we use D, the number of potentially citatory documents published in a given research field, instead R, we obtain a more handily, but mathematically less rigorous, version of the Equation (6)

$${\displaystyle \frac{h_2}{h_1}}=\sqrt{{\displaystyle \frac{P_2^{*}{D}_2}{P_1^{*}{D}_1}}}\propto \sqrt{{\displaystyle \frac{D_2}{D_1}}}$$

(11)

Thus, it is clear that the comparison between researcher of two different fields must account for the respective values of R or D.

However, as shown in the previous section, the determination of the h-indexes in both research field is self-inconsistent. Following the same rationale as in the previous section, a more rigorous comparison between the scientific impact of authors could consider the ratio of their V-indexes (as defined in section 2) in the respective research fields, but it also must account for the productivity of both fields

$${\displaystyle \frac{V_2}{V_1}}\propto \sqrt{{\displaystyle \frac{{\displaystyle \sum _{i(2)}{\left(D_{py2}/D_{cy2}\right)}_{i(2)}}}{{\displaystyle \sum _{i(1)}{\left(D_{py1}/D_{cy1}\right)}_{i(1)}}}}}$$

(12)

Does not seem therefore that this kind of comparisons is very useful. The situation is worse considering that a research field may include several topics with its own publication and citation dynamics.

4. Conclusions

I have above shown that the determination procedure of the h-index is self-inconsistent, does not consider the true weight of individual citations, and the change in the number of published documents and citations with time. Also, the simple analysis here presented demonstrates that the h-index of researchers publishing in different research fields cannot be directly compared (neither in personal nor academic situations). More rigorous alternative indexes based in related rationales are cumbersome to be implemented. Thus, the use of the h-index for the evaluation of academic researchers, journal or institutions should be eliminated or at least alleviated.