F*, an interpretable transformation of the F measure, equates to the critical success index

1. Introduction

Many measures may be derived from the data in a 2x2 contingency table.¹ One of these is the F measure, defined as the harmonic mean of precision (or positive predictive value, PPV) and recall (or sensitivity, Sens).² This corresponds to the coefficient described by Dice³ and independently by Sørensen,⁴ sometimes known as the Dice coefficient or the Sørensen-Dice coefficient, and to the approach advocated by van Rijsbergen.⁵

In terms of the base data from a 2x2 contingency table containing N elements with four degrees of freedom (where TP = true positive, FP = false positive, FN = false negative, TN = true negative):

F = 2.TP/(2.TP + FP + FN)

or in terms of PPV and Sens:

F = 2.PPV.Sens/(PPV + Sens)

= 2/[1/Sens + 1/PPV]

that is, F is the harmonic mean of PPV and Sens.

More recently, Hand et al. (2021) have described “F*” as “an interpretable transformation of the F measure,”⁶ where:

F* = F/(2 – F)

As will be shown, these authors have in fact redescribed an already existing binary classification measure, first reported in the late nineteenth century as the ratio of verification in the context of forecasting tornadoes,⁷ and subsequently as the Jaccard index or similarity coefficient (J),⁸ the threat score,⁹ the Tanimoto index,¹⁰ and later still as the critical success index (CSI).^11,12 Here we use the latter terminology.

2. Mathematical proofs of identity of F* and CSI

The identity of F* and CSI may be shown in several ways using elementary mathematical methods.

• 2a. From the base data of a 2x2 contingency table.

Hand et al.⁶ showed that:

F* = TP/(N – TN)

This also holds for CSI, since in terms of the base data:

CSI = TP/(TP + FP + FN)

= TP/(N – TN)

Hence F* = CSI, QED.

• 2b. From the monotonic relationship of F to CSI.

The monotonic relationship between F and CSI, as shown for example by Jolliffe¹³ (modified), is given by:

F = 2.CSI/(1 + CSI)

The equivalence of F* and CSI may thus be shown. Since Hand et al.⁶ showed that:

F* = F/(2 – F)

Then rearranging:

F = F*.(2 – F)

= 2.F* – F*.F

Dividing through by F and rearranging:

(2.F*/F) – F* = 1

F* + 1 = 2.F*/F

Hence:

F = 2.F*/(F* + 1)

Hence F* = CSI, QED.

• 2c. From the combination of PPV (or precision) and Sens (or recall).

Like F, CSI may be characterised in terms of PPV and Sens:

CSI = 1/[(1/PPV) + (1/Sens) – 1]

Again, the equivalence of F* and CSI may be shown. Hand et al.⁶ found that:

F* = (PPV x Sens)/PPV + Sens – (PPV x Sens)

Dividing through by (PPV x Sens) gives:

F* = 1/[(1/Sens) + (1/PPV) – 1]

Hence F* = CSI, QED.

• 2d. From the combination of Sens, PPV, P, and Q.

In the 2x2 contingency table, prevalence or base rate P = (TP + FN)/N, and bias or threshold Q = (TP + FP)/N. Thus, from Powers²:

F = 2.Sens.P/(Q + P)

= 2.PPV.Q/(Q + P)

For CSI the equations are¹:

CSI = 1/[(Q + P)/Sens.P] – 1

= 1/[(Q + P)/PPV.Q] – 1

Since from Hand et al.⁶:

F* = F/(2 – F)

Then substituting and rearranging:

F* = [2.Sens.P/(Q + P)]/(2 – [2.Sens.P/(Q + P)])

= 1/[(Q + P)/Sens.P) – 1

F* = [2.PPV.Q/(Q + P)]/(2 – [2.PPV.Q/(Q + P)])

= 1/[(Q + P)/PPV.Q) – 1

Hence F* = CSI, QED.

3. Conclusion

Hand et al. noted that “researchers may recognise this [i.e. F*] as the Jaccard coefficient widely used in areas where TN may not be relevant”⁶ and they cite Jaccard’s 1908 paper,¹⁴ although others² cite his 1901 paper¹⁵ as the forerunner of the 1912 English translation.⁸

We suggest that this is a parameter which, like F, has undergone periodic redescriptions (or convergent evolution). The first report of which we are aware is Gilbert’s “ratio of verification” of 1884,⁷ predating the Jaccard similarity coefficient.⁸ This latter measure is equivalent in set theory to union over intersection, which was also proposed by Tanimoto in 1958 when working for IBM,¹⁰ without reference to either Gilbert or Jaccard. The same measure has also been described by Palmer & Allen in 1949 as the threat score,⁹ and as the critical success index by Donaldson et al.¹¹ in 1975 and by Schaefer¹² in 1990, and now as F* by Hand et al.⁶ These multiple redescriptions may reflect use of this measure by researchers in different disciplines (weather forecasting, ecology, machine learning) unaware of prior authors and unbeknownst to later authors.

The critical success index has recently been exported to the domain of clinical medicine, for example to evaluate the accuracy of instruments used in day-to-day clinical practice for screening cognitive function in patients with possible dementia or mild cognitive impairment,¹⁶ as well as in diagnostic accuracy studies of administrative epilepsy data.¹⁷ In these studies the identity of F* and CSI has been confirmed using the respective datasets. We have also suggested possible application of CSI in assessing both NICE criteria for 2-week-wait suspected brain and CNS cancer referrals¹⁸ and polygenic hazard scores.¹⁹ These are all situations in which large numbers of TN may complicate the interpretation of more traditional measures such as PPV and Sens.