Converging Information-Theoretic Measures of Metacognitive Efficiency

1. Introduction

Assessing the validity of our own decisions is a self-reflective act of immense survival value. For example, while learning for a test, we have to constantly monitor the level of our knowledge. If we feel that a satisfactory level of knowledge has been accomplished we can halt our preparation, otherwise we have to continue our learning [4,5]. This self-reflective ability is a form of metacognition [6,7,8,9,10,11,12]. Metacognition plays a central role in various domains, including: learning [13], memory [14,15], self-awareness [16], action [17,18], psychiatric disorders [19], and social interaction [20]. Several neural substrates are involved in metacognitive assessments and monitoring of behavior [22]. These include the right anterior dorsolateral prefrontal cortex and the bilateral parahippocampal cortex [10]. A prominent method of assessing metacognition behaviorally is based on collecting confidence judgments from participants [12]. Measuring the quality of these confidence ratings is consequential for understanding the underlying mechanisms of metacognition. The goals of the present study are threefold: (a) to introduce several recently developed information-theoretic measures of metacognitive efficiency that are based on confidence judgments [3,4], (b) to demonstrate how these measures can be applied to a concrete behavioral task of face-matching [21], and (c) to show that these information-theoretic measures converge on a single theoretical construct in this empirical context [22].

A primary objective of research on metacognition is that of measuring the quality of confidence judgments. Three quantities stand out as crucial aspects of measurement and modeling: bias, sensitivity, and efficiency. To motivate our understanding of these concepts, consider a simple task by which an observer is asked to detect whether a signal is present or absent by emitting one of two possible responses, and then immediately she is asked to rate her confidence in the correctness of her decision. It is often customary to call the first task type 1 task, and the second type 2 task [23,24]. It is also theoretically convenient to separate the actor, who performs in the type 1 task, from the rater, who engages in the type 2 task, although the two entities might be the same person [25]. Sensitivity plays a role in both the actor’s and the rater’s decisions. The actor’s sensitivity is reflected in her ability to separate between noise and noise plus signal. In the language of Signal Detection Theory [SDT, [26,27] this quantity is captured by the parameter $d^{\prime }$. The rater’s sensitivity reflects his ability to distinguish between the actor’s correct and incorrect decisions. A metacognitively sensitive rater tends to provide high confidence ratings for the actor’s correct decisions, and low confidence ratings for the actor’s incorrect decisions. However, the sensitivity of the rater cannot serve as a viable measure of the rater’s quality of decision. Trivially, a rater can be highly sensitive if the actor is giving correct decisions most of the time. In other words, the rater’s job becomes much easier as the actor becomes more sensitive. Thus, a genuine measure of metacognitive efficiency should correct the rater’s sensitivity for the actor’s performance. Moreover, metacognitive efficiency can be affected by bias. An overconfident rater, one who gives high confidence ratings even when the actor’s decisions are incorrect, is effectively both insensitive and inefficient.

Model-free [28] and model-based [1,2,24,25,29] approaches have been developed to quantify metacognitive bias, sensitivity and efficiency. One of the most influential model-based frameworks in the field is called the $meta-d^{\prime }$ model [1,2]. This model (see Figure 1) relies on principles of SDT [26,27], and as such incorporates parametric assumptions [23,24,30]. The model’s objective is to predict the rater’s hypothetical sensitivity in the type 1 task based on the rater’s performance in the type 2 task. The value of this theoretical quantity is dubbed $meta-d^{\prime }$. Because this quantity and the actor’s type 1 $d^{\prime }$ are expressed in the same measurement units [2], they can be directly compared. Metacognitive efficiency can then be assessed in two possible ways: either as an M-ratio ($meta-d^{\prime }/d^{\prime }$) or as a difference ($meta-d^{\prime }-d^{\prime }$). The rater’s efficiency is optimal when $M-ratio=1$, or when $meta-d^{\prime }-d^{\prime }=0$. The rater is hyposensitive when $M-ratio<1$ or when $meta-d^{\prime }-d^{\prime }<0$. There are situations in which the rater is hypersensitive, namely when $M-ratio>1$, or when $meta-d^{\prime }-d^{\prime }>0$ [1]. This can occur when the rater has access to additional information accumulated after the actor’s decision. Such a scenario can be modeled, for example, by the second-order model [25]. According to this model, the actor and the rater are sampling evidence from two separate distributions. Hypersensitivity in this model can be modeled by making the noise (i.e.,variance) in the rater’s distribution much lower than the noise (i.e.,variance) in the actor’s distribution.

Several measures emanating from the $meta-d^{\prime }$ model [1,2], such as $meta-d^{\prime }-d^{\prime }$ and $M-ratio$, have been used extensively by researchers to evaluate the quality of confidence judgments [32]. However, these measures are not without their limitations [3]. First, it is not immediately clear why empirical confidence ratings should necessarily accommodate to a type 1 decision process [3]. Second, strong evidence suggests that these measures carry-over a residual dependency on type 1 performance [33], and on metacognitive bias [34]. Beyond these two reasons, these measures: (a) require distributional assumptions, (b) are not bias-free, (c) are applicable only to ratio-scale variables, (d) do not scale with the number of stimuli, responses, or confidence rating alternatives, (e) do not have definite upper and lower bounds, and (f) are applicable only to experimental designs with two stimuli and two responses. Dayan [3] and Fitousi [4] have recently proposed information-theoretic indexes of metacognitive efficiency that can ameliorate most of these problems, and serve as additional means of assessment.

Dayan [3] has succinctly summarized the utility of an information-theoretic approach to judgments of confidence: ”the problem of confidence is inherently one of information – that the actor has about the true state of the stimulus; and that the rater has about the same quantity and about what the actor used. It therefore seems appropriate to use the methods of information theory to judge the relationship between the stimulus, the actor and the rater” (p.11). Indeed, the information theoretic measures have many desirable characteristics: (a) they are relative measures, (b) accommodate to type 1 performance, (c) bias- free (e.g., do not depend on criterion shifts), (d) do not require any parametric assumptions, (e) scale with the number of confidence rating points, (f) do not require the assumption of ratio or order of confidence ratings, and (g) relatively easy to compute. Both Dayan [3] and Fitousi [4] performed simulation with canonical Bayesian models to support the validity of the information-theoretic measures they developed. Both authors demonstrated convergence of their information-theoretic measures based on those SDT and related Bayesian models. However, neither of them computed the measures in real empirical data sets, nor did they examine the convergence validity of those measures is such cases. The present study sought to accomplish these objectives.

2. An information-Theoretic Approach to Metacognitive Efficiency

Recent years have seen a renewed interest [35] in information-theoretic approaches to perception and cognition [36,37,38,39,40,41,42,43,44]. Many researchers have began to realize the potential strength of information theory [45,46] in providing powerful tools for addressing the uncertainty that characterizes brain and behavior [47,48]. In two recent studies, Dayan [3] and Fitousi [4] have developed complementary information-theoretic approaches to metacognition that build on the idea of confidence judgments as a communication system. An illustration of such a system is presented in Figure 2. The communication model consists of three entities: Stimuli which are set by the Experimenter, Actor who responds to the stimuli, and Rater who provides confidence judgments of the Actor’s response to the Stimuli.

2.1. Dayan’s $Meta-\mathcal{I}$

Dayan [3] outlined an information-theoretic measure of metacognitive efficiency he called $meta-\mathcal{I}$. Consider an actor who performs in a two-choice signal detection task with consequences of a =1, or a = -1. The actor’s response is r, and she is correct when $r=a$ and incorrect when $r\ne a$. The confidence-accuracy discrete distributions can be used to assess the metacognitive efficiency of the rater as a difference between two entropies.

$$\begin{array}{ccc}\hfill meta-\mathcal{I}& =\hfill & \hfill H_2\left(r\right|a=-1)-H_2\left(r\right|c,a=-1)\end{array}$$

(1)

where

$$\begin{array}{ccc}\hfill H_2\left(r\right|a=-1)& =\hfill & \hfill h_2\left[P\left(r\right|a=-1)\right]\end{array}$$

(2)

and

$$\begin{array}{ccc}\hfill H_2\left(r\right|c,a=-1)& =\hfill & \hfill \sum _{c}P\left(c\right|a=-1)h_2\left[P\left(r\right|c,a=-1)\right]\end{array}$$

(3)

where

$$\begin{array}{ccc}\hfill h_2\left[P\left(x\right)\right]& =\hfill & \hfill \sum _{x=0;1}-P\left(x\right){log}_{2}P\left(x\right)\end{array}$$

(4)

is the entropy of a Bernoulli random variable. The first entropy $H_2\left(r\right|a=-1)$ is the overall uncertainty about the actor’s accuracy. In the SDT model it is determined by the value of the parameter $d^{\prime }$. When the actor is perfectly accurate and $d^{\prime }\to \infty$ or when the actor is perfectly wrong, this entropy amounts to 0 $bits$; when the actor is completely uncorrelated with $d^{\prime }$, which occurs when $d^{\prime }\to 0$, this entropy amounts to 1 $bits$ . The second entropy $H_2\left(r\right|c,a=-1)$ is the weighted average uncertainty about the residual accuracy after observing the confidence rating c where the weights arrive from the probability of observing that rating c.

The $meta-\mathcal{I}$ measure has many desirable characteristics: (a) it is a relative measure, (b) accommodates to type 1 performance, (c) does not require any parametric assumptions, (d) is bias-free, (e) scales with the number of confidence rating points, (f) does not require the assumption of ratio or order of confidence ratings, and (g) has definite bounds from above and from below.

The $meta-\mathcal{I}$ is an ingenious measure on its right. But it can be modified to become a more general index of metacognitive efficiency. Note that the measure is contingent on accuracy (i.e.,it depends on a specific partition, r = a, $r\ne$ a), while there is strong evidence that participants do not necessarily rate confidence based on the accuracy of response [49]. For example, Koriat [52,53] has demonstrated the operation of a consensuality principle in judgments of confidence, whereby participants give the highest confidence rating to the most popular option, not necessarily to the one that is objectively correct. Moreover, $meta-\mathcal{I}$ was originally developed to assess metacognitive efficiency in the two-choice SDT task (i.e., it assumes two mutual choices, a = 1 or a=-1). However, $meta-\mathcal{I}$ does not apply to tasks of multi-alternative choice [49]. For example, assume an experimental design with four stimuli that are categorized into four responses, and are then rated on a 1-4 confidence scale. In that case, $meta-\mathcal{I}$ is not readily applicable. Thus, a more general version of $meta-\mathcal{I}$, I call $meta-{\mathcal{I}}^{\prime }$, is needed. This version is not contingent on accuracy, and makes no assumptions about the structure of the experimental design (i.e., number of stimuli or/and responses). The new $meta-{\mathcal{I}}^{\prime }$ proposed here meets these requirements. It is computed as the mutual information between response and the joint confidence-stimuli variable:

$$\begin{array}{cc}\hfill meta-{\mathcal{I}}^{\prime }=I(r;c,s)& =H\left(r\right|s)-H(r|s,c)\hfill \\ \hfill & =H\left(r\right)+H(c,s)-H(s,r,c)\hfill \end{array}$$

(5)

This quantity amounts to 0 bits when the actor shares no information with the stimuli and rater; it is maximal when the actor shares all the information with the stimuli and the rater. An upper-bound on $meta-{\mathcal{I}}^{\prime }$ is determined by the variable with the lowest number of alternatives. For example, if the stimuli variable has the lowest number of alternatives, say four, then the upper bound is 2 bits. In the case of a two-choice SDT task, the upper bound is 1 bit. Decomposition of $meta-{\mathcal{I}}^{\prime }$ into its entropy terms can provide more insights. According to one possible decomposition, it reflects the degree to which uncertainty in the actor’s response to the stimuli $H\left(r\right|s)$ is reduced after observing the rater’s confidence judgments $H\left(r\right|s,c)$. According to another decomposition, $meta-{\mathcal{I}}^{\prime }$ reflects the reduction in the combined uncertainty in the actor’s response and the confidence-stimuli variables $H\left(r\right)+H(s,c)$ after removing the uncertainty in the joint stimuli-response-confidence variable $H(s,r,c)$. In its new form, $meta-{\mathcal{I}}^{\prime }$ can be applied to complex experimental designs, with several stimuli and responses, and without the need to define the correctness of the response.

2.2. Fitousi’s $Meta-\mathcal{U}$, $Meta-\mathcal{KL}$, and $Meta-\mathcal{J}$

Fitousi [4] has recently developed three complementary information-theoretic measures of meatcognitive efficiency he called: $meta-\mathcal{U}$, $meta-\mathcal{KL}$, and $meta-\mathcal{J}$. The first measure bears close affinity with Dayan’s $meta-\mathcal{I}$, and its generalized form $meta-{\mathcal{I}}^{\prime }$. The measure quantifies metacognitive efficiency as the mutual information exchanged between the rater’s confidence judgments and the stimuli-response variable:

$$\begin{array}{cc}\hfill meta-\mathcal{U}=I(c;s,r)& =H\left(c\right|s)-H(c|s,r)\hfill \\ \hfill & =H\left(c\right)+H(s,r)-H(s,r,c)\hfill \end{array}$$

(6)

The index expresses the amount of information gained about confidence from knowing the actor’s response, after knowing the stimuli. When the rater’s metacognitive efficiency is null, $meta-\mathcal{U}$ amounts to zero bit, setting a lower bound on efficiency. An upper-bound on $meta-\mathcal{U}$ is placed by the variable with the lowest number of alternatives in the communication system. Decomposition of $meta-\mathcal{U}$ into entropy terms can explicate the logic of this measure. In the first decomposition, $meta-\mathcal{U}$ reflects the degree to which the conditional uncertainty of confidence after knowing the stimulus is reduced after observing the actor’s response. In the second decomposition, we note that $meta-\mathcal{U}$ gauges the degree to which uncertainty in the confidence and stimulus-response variables $H\left(c\right)+H(s,r)$ is reduced after observing the actor’s response $H(s,r,c)$. This measure corrects for type 1 performance by taking the difference between type 1 uncertainty $H(s,r)$ and the type 1 and type 2 uncertainty $H(s,r,c)$, in a similar way to $meta-d^{\prime }-d^{\prime }$.

Importantly, it can be shown that $meta-\mathcal{U}$ equals $meta-{\mathcal{I}}^{\prime }$ due to the symmetry of mutual information:

$$\begin{array}{c}\hfill meta-\mathcal{U}=meta-{\mathcal{I}}^{\prime }\\ \hfill I(c;s,r)=I(r;c,s)\end{array}$$

(7)

In the case of a two-choice SDT task, $meta-\mathcal{U}$ and $meta-{\mathcal{I}}^{\prime }$ are equal to Dayan’s $meta-\mathcal{I}$.

The other two measures advanced by Fitousi [4], $meta-\mathcal{KL}$, and $meta-\mathcal{J}$, are based on the Kullback-Leibler (KL) [50] and Jeffrey’s [51] divergences. These divergences are applied to the normalized confidence-accuracy distributions $P\left[\right(confidence=y\left)\right|Correct\left)\right]$ and $Q\left[\right(confidence=y\left)\right|Incorrect\left)\right]$. It should be noted though that, in principle, these divergence measures can be derived for any required partition of the stimuli-response joint variable, irrespective of accuracy. Accuracy is defined by labels (i.e., ”correct”, ”incorrect”) placed by the experimenter on the actor’s responses. But information-theoretic quantities are agnostic to such labels. A rater can be metacognitively efficient even when she flips the ”correct” and ”incorrect” labels on the actor’s responses.

The new measure proposed by Fitousi, $meta-\mathcal{KL}$, is based on a KL divergence $D_{KL}(Q\Vert P)$ that measures the degree of separation between correct and incorrect accuracy-confidence distributions:

$$\begin{array}{ccc}\hfill meta-\mathcal{KL}& =D_{KL}(P\Vert Q)\hfill & \hfill =\sum P(c=y|Correct)log\left({\displaystyle \frac{P(c=y|Correct)}{Q(c=y|Incorrect\left)\right)}}\right)\\ \hfill \end{array}$$

(8)

It amounts to 0 bits when the rater cannot distinguish between correct and incorrect responses of the actor, and increases as the rater becomes more metacognitively efficient. An upper bound is met when the two distributions are at their maximal separability. An analogous expression of $meta-\mathcal{KL}$ gauges the difference between the entropy in the joint correct and incorrect distribution and the entropy in the correct distribution:

$$\begin{array}{cc}\hfill meta-\mathcal{KL}& =D_{KL}(P\Vert Q)\hfill \\ \hfill & =H(P,Q)-H\left(P\right)\hfill \\ \hfill & =H(c=y|Correct,c=y\left|Incorrect\right)-H(c=y|Correct)\hfill \end{array}$$

(9)

Metacognitive efficiency is null when uncertainty in the joint correct-incorrect confidence-accuracy distribution is equal to the uncertainty in the correct confidence-accuracy distribution. Metacognitive efficiency increases as the uncertainty in the correct distribution decreases. It reaches an upper bound when the latter is 0 $bits$, namely, the rater eliminates all uncertainty in the actor’s correct (incorrect) responses to the stimuli.

The $meta-\mathcal{KL}$ measure does not satisfy the triangle inequality, and is also not symmetric in the two distributions, since $D_{KL}(P\Vert Q)\ne D_{KL}(Q\Vert P)$. In this sense, $meta-\mathcal{KL}$ does not qualify as a metric [52]. To address these constraints, Fitousi [4] has also proposed $meta-\mathcal{J}$ based on the Jeffrey’s divergence [51,53], which offers a symmetric measure of divergence:

$$\begin{array}{cc}\hfill meta-\mathcal{J}=J(P,Q)& =D_{KL}(P\Vert Q)+D_{KL}(Q\Vert P)\hfill \\ \hfill & =\sum P(c=y|Correct)log\left({\displaystyle \frac{P(c=y|Correct)}{Q(c=y|Incorrect\left)\right)}}\right)+\hfill \\ \hfill & \sum Q(c=y|Incorrect)log\left({\displaystyle \frac{Q(c=y|Incorrect)}{P(c=y|Correct\left)\right)}}\right)\hfill \end{array}$$

(10)

This measure functions in a similar way to $meta-\mathcal{KL}$. It equals 0 bits when the rater cannot distinguish between the correct and incorrect accuracy-confidence distributions, and increases as the rater becomes more efficient. But unlike $meta-\mathcal{KL}$, it takes into consideration the two possible KL divergences, $D_{KL}(P\Vert Q)$ and $D_{KL}(Q\Vert P)$ between the correct and incorrect confidence-accuracy distributions. Another way of expressing the same measure:

$$\begin{array}{cc}\hfill meta-\mathcal{J}& =J(P,Q)\hfill \\ \hfill & =2\times H(P,Q)-H\left(P\right)-H\left(Q\right)\hfill \\ \hfill & =2\times H(c=y|Correct,c=y\left|Incorrect\right)-H(c=y|Correct)-H(c=y\left|Incorrect\right)\hfill \end{array}$$

(11)

suggests that metacognitive efficiency gauges the degree to which the rater reduces uncertainty in the actor’s response by eliminating uncertainty in the correct accuracy-confidence distribution, and/or in the incorrect accuracy-confidence distribution. The $meta-\mathcal{J}$ takes into consideration two sources of uncertainty, weighting them as two independent contributions to metacognitive efficiency. This is in contrast to the $meta-\mathcal{KL}$, which gives all the weight to the uncertainty in the correct accuracy-confidence distribution.

To validate these novel measures, Fitousi [4] has performed simulations with the $meta-d^{\prime }$ model [1,31]. He varied the values of $d^{\prime }$ and $meta-d^{\prime }$ incrementally in small steps, and in an orthogonal fashion and generated the corresponding confidence-accuracy distribution for each pair of values. He then computed the information-theoretic measures $meta-\mathcal{U}$, $meta-\mathcal{KL}$, and $meta-\mathcal{J}$ for each confidence-accuracy distribution, and plotted those values as a heatmap, with $d^{\prime }$ in the abscissa and $meta-d^{\prime }$ in the ordinate. Figure 3 presents the results of these simulations. As can be noted, all three heatmaps reveal one of the most important properties of an efficiency measures, namely that the rater’s sensitivity is corrected for type 1 performance. To see why, note that the highest values of metacognitive efficiency are achieved when the actor’s sensitivity ($d^{\prime }$) is low and the rater’s sensitivity ($meta-d^{\prime }$) is high. In all three heatmaps, as the actor’s sensitivity increases, there is less and less room for the rater to accomplish high metacognitive efficiency. Also notable are the differences between the mutual information measure ($meta-\mathcal{U}$), and the divergence measures ($meta-\mathcal{KL}$, $meta-\mathcal{J}$) with respect to the maximal values they can take, and their overall dynamics.

The information theoretic measures have many valuable characteristics. These are discussed in detail in the papers by Dayan [3] and Fitousi [4]. Here I give a brief summary of the most important features. They are: (a) relative measures, (b) accommodate type 1 performance, (c) bias-free, (d) do not require any parametric assumptions, (e) scale with the number of confidence rating points, (f) do not require the assumption of ratio or order of confidence ratings, (g) easy to compute, and (h) offer definite lower and upper bounds on performance.

One important question that remains to be address is whether these measures of metacognitive efficiency converge on the same psychological construct. Previous efforts by Dayan [3] and Fitousi [4] have emulated various simulations with theoretical models to show that the measures converge. But there is still the need to assess these measures’ convergence with real data sets. If these different indexes capture the same psychological construct, they should correlate in real empirical data. This hypothesis is subjected to scrutiny in the next section.

3. Empirical Validation

3.1. The Face-Matching Task

Here I provide an empirical validation of the information-theoretic measures of metacognitive efficiency. To do so, I take advantage of two data sets collected in a recent face-matching study in my lab. The face-matching task [54,55,56,57,58] has become an important tool in assessing sundry aspects of face recognition. In this task participants are presented with two face images and are asked to decide whether these belong to the same identity, or alternatively to two different identities. The images are taken under various lighting conditions, with different cameras, and at different times. It turns out that with unfamiliar facial identities, this task poses serious challenges for both humans [21,54,55,56,57,58] and machines [59]. This is mainly because under variable ambient conditions, images of the same personal identity can look completely different; whereas images of different identities may appear similar.

Figure 4 presents examples of pairs of face images from the Glasgow Face Matching Task [GFMT [54], and from the Kent Face Matching Task [KFMT 60], two of the most popular standard tests, which were also deployed in the present investigation. Extensive research on the face-matching task [61,62] has greatly enhanced our understanding of the underlying mechanisms that govern this task. A recent paper by Fitousi [21] has introduced an unequal-variance signal detection model, one that can account for many aspects of this task, including type 1 confidence judgments, similarity ratings, and accuracy.

However, despite extensive research on the face-matching task [55,63,64,65], only little is known about the underlying metacognitive capabilities with which participants perform this task. The application of the $meta-d^{\prime }$ indexes, along with the novel information-theoretic measures can shed new light on this issue. But most importantly, the joint application of these measures is needed to garner strong evidence for their convergence.

3.2. The Data Sets

The data sets were collected in my lab. In data set 1, seventy participants engaged in a face-matching task with the long version of the GFMT [54]. In data set 2, seventy new participants were tested with the KFMT [60]. There were 168 images in the GFMT and 40 images in the KFMT. More details about these tests can be found here [21]. The general method in both experiments was identical. Participants were sitting in front of a computer screen. On each trial they were presented with a randomly selected face-pair, and were asked to decided as accurately and as speedily as they can, whether the pair constituted a ”match” or a ”mismatch” by pressing one of two computer keys (type 1 task). Immediately after responding, the images were removed from the screen, and a a four-point confidence scale appeared on the screen with 1=”not confident ” and 4 = ”highly confident.” In data set 1 participants completed two blocks, which amounted to a total of 336 trials. In data set 2 participants completed four blocks, which resulted in a total of 160 trials.

3.3. Results

The face-matching task administrated in data set 1 was relatively easier ($d^{\prime }$ = 2.51, sd = 0.65) than the one in data set 2 ($d^{\prime }$ = 1.01, sd = 0.41). However, the results from both data sets converged on the conclusion that, on average, participants did exhibit meta-knowledge about their face-matching performance. The mean of the $meta-d^{\prime }-d^{\prime }$, measure amounted to 0.86 (sd = 1.31) in data set 1, and 1.09 (sd = 0.97) in data set 2. Recall that for $meta-d^{\prime }-d^{\prime }$ a value greater than zero entails hypersensitivity. Similarly, the mean of the $M-ratio$ measure was equal to 1.44 (sd = 0.66) in data set 1, and 2.83 (sd = 3.25) in data set 2. Recall that for $M-ratio$ a value greater than 1 entails hypersensitivity. One-sided student-tests supported these conclusions, with all t-statistics being significant at the $0.05$ level. The novel information-theoretic measures also supported the conclusion that participants in both data sets acquired meta-knowledge about their decisions.The average $meta-\mathcal{U}$ values were 0.22 bits (sd = 0.17) and 0.37 bits (sd = 0.21) in the two data sets, respectively. Recall that in the two-choice task $meta-\mathcal{U}$ is quantitatively comparable to Dayan’s [3]$meta-\mathcal{I}$. The average $meta-\mathcal{KL}$ values amounted to 1.54 bits (sd = 0.95) and 1.11 bits (sd = 0.74) in the two data sets, respectively. The average $meta-\mathcal{J}$ values were 4.15 bits (sd = 2.37) and 2.54 bits (sd = 1.74) in the two data sets, respectively.

I then computed the SDT and information-theory measures for each participant, and used linear regression on each pair of measures to quantify the coefficient of determination $R^2$. Figure 5 and Figure 6 present the results of these analyses, respectively, for the GFMT and KFTM data sets.

As can be noted in Figure 5 and Figure 6, significant positive and strong relations ensued between each pair of measures in both data sets. Overall, higher $R^2$ obtained for the KFMT than the GFMT data. In the former, all coefficient of determination values were larger than $0.47$. In the latter, all those values exceeded $0.29$. The strongest relations were observed between $meta-\mathcal{KL}$ and $meta-\mathcal{J}$. This is quite expected given their mathematical affinity. Taken collectively, these results: (a) support the conclusion that participants do have meta-knowledge about their face-matching performance, and (b) confer validation support in the convergence of the information-theoretic measures on a common psychological structure – metacognitive efficiency.

4. Conclusions

I introduced various information-theoretic measures of metacognitive efficiency recently developed by Dayan [3] and Fitousi [4]. These indexes consist of: the $meta-\mathcal{I}$ [3], and the $meta-{\mathcal{I}}^{\prime }$, $meta-\mathcal{U}$, $meta-\mathcal{KL}$, and $meta-\mathcal{J}$ [4]. These measures were originally developed within a common theoretical framework, assuming a communication system with stimuli (experimenter), actor, and rater as encoders/decoders. I provided a succinct account of each measure, along with its possible interpretations. I then applied these information indexes, along with the $M-ratio$ and $meta-d^{\prime }-d^{\prime }$ indexes, to data sets from the face-matching task [54]. The results strongly supported the convergence of these measures on a single psychological construct.

One source of motivation for developing information-theoretic markers of metacognitive efficiency has to do with the limitations of existing model-based measures, such as $meta{d}^{\prime }-d^{\prime }$ and $M-ratio$. These indexes are built on the assumption that they should fit exactly to a type 1 decision process. But this assumption is dubious [3]. Moreover, these measures are subjected to decisional bias [1,2], have a remaining dependency on type 1 performance [33], and on metacognitive bias [34]. Information theory measures are free of such limitations.

Another crucial point is that both model-based and model-free measures of metacognitive efficiency [66] do not accommodate changes in the granularity of type 2 confidence ratings, nor do they scale with the number of type 1 stimuli or responses. Information-theoretic measures can be computed relatively easily, without making any parametric assumptions, can be applied to tasks which incorporate multi-alternative choices, are compatible with existent SDT-based measures, and can be applied to data without labeling the responses as correct or incorrect. Moreover, unlike SDT-based measures, they can accommodate the granularity of the confidence judgments (e.g., number of rating points), and can provide definite upper- and lower-bounds on performance. All of these valuable characteristics have been previously demonstrated by Dayan [3] and Fitousi [4].

Another reason for advancing the information-theoretic approach to meatcognition has to do with the suitability of such a theory to characterize capacity limitations, information-bottlenecks, rate-distortions, and other quantities that are not readily available in current approaches (e.g. SDT). Future work on confidence and metacognition should seek to expand our theoretical understanding by building on these powerful notions.