An Information-Theoretic Framework for Understanding Learning and Choice Under Uncertainty

1. Introduction

Information theory has been used widely across different domains of cognitive and systems neuroscience, ranging from analyzing spike trains to estimating uncertainty in sensory environments. For example, Shannon entropy and related metrics have been used to study information flow [1,2], functional and effective connectivity [3,4,5,6,7], and variability in neural response [8,9,10,11]. These quantitative approaches have provided significant insight into brain functions at the cellular and system levels.

Despite this widespread use of information-theoretic metrics in analyzing neural data, their application in investigating behavior has been surprisingly limited. Although several studies have employed entropy-based metrics to quantify uncertainty in stimuli [12,13,14,15] and outcomes [16,17,18], there are only a few studies that have utilized information theory to directly examine the underlying decision-making and learning mechanisms [19,20,21]. This is despite the fact that many behavioral measures—such as binary choices and reward feedback—are ideally structured for analysis using information-theoretic tools.

A common tool for studying learning and choice behavior is reinforcement learning (RL) models [22,23,24,25,26,27], due to their simplicity and interpretability. In this approach, various RL models are constructed and fit to the choice data. The best-fitting model is then identified through model selection, and its components form the basis for inferences about underlying cognitive and/or neural mechanisms. However, these models often need to be augmented with additional components to capture empirical data in specific tasks (e.g., [28,29,30,31]), while it remains unclear how many components are necessary—or sufficient—to define the “best” model. Such extensions include separate learning rates for positive versus negative prediction errors (differential learning rates), time-varying (dynamic) learning rates, modulations of learning by reward harvest rate, and arbitration between alternative models of the environment, among others [32,33,34,35,36,37,38,39,40]. Nevertheless, currently there is no systematic method to identify the critical components required in RL models. This lack of methodology can lead to important mechanisms being overlooked, even in models that provide the best fit among the tested models.

Interestingly, choice behavior under uncertainty is inherently stochastic, making it well-suited for analysis using information theory. Recently, a set of information-theoretic metrics was proposed as a model-agnostic approach to uncover the hidden learning mechanisms underlying behavior. For example, Trepka et al. [29] have suggested that behavioral metrics based on conditional entropy can quantify the consistency in local choice strategies and predict undermatching. Moreover, Woo et al. [41] have shown that mutual information, alongside other measures, can capture the influence of high-order reward statistics. These results highlight the potential of information-theoretic metrics to probe learning and decision-making processes beyond what traditional model-fitting techniques can reveal.

Here, we extended this approach by applying information-theoretic metrics to simulated choice data from different RL models—serving as ground truth— across four learning tasks. We show that this method captures keys aspects of learning and decision making without relying on model fitting. To that end, we used existing metrics and developed new ones to detect a range of learning and decision-making mechanisms and their dynamic adjustments. We begin by identifying higher learning rates following positive prediction errors (rewarded outcomes) than following negative prediction errors (unrewarded outcomes)—a phenomenon referred to as positivity bias [42,43,44,45]. We also examine changes in learning rates over time as a result of metaplasticity [46,47]. Next, we investigate the influence of reward harvest rate, which has been shown to modulate learning and decision making [34,35,48,49]. Finally, in naturalistic reward environments, choice options often possess multiple attributes, each potentially predictive of reward outcomes. Previous studies have demonstrated that humans and other animals tackle this challenge by simultaneously learning about alternative reward contingencies (i.e., models of the environment), arbitrating between these models, and deploying selective attention to guide differential learning and decision making [38,50,51,52,53,54,55,56,57,58]. Therefore, we test how information-theoretic metrics can be used to detect the presence of distinct learning strategies in complex reward environments. Overall, our results demonstrate that the patterns of information-theoretic metrics provide useful summary statistics of the behavioral signatures generated by different learning and choice mechanisms, thereby offering a complementary approach to model fitting (both discovery and recovery).

2. Materials and Methods

2.1. General Experimental Paradigm: Probabilistic Reversal Learning

We simulated choice data using different reinforcement learning (RL) agents performing two variants of the probabilistic reversal learning task (PRL), a widely used paradigm to assess cognitive flexibility across various species [59]. In a typical probabilistic reversal learning paradigm, subjects choose between two options based on reward feedback they receive on each trial, with selection of each option associated with a different probability of reward. Choice options can take various forms, such as distinct visual stimuli on a computer screen, identical stimuli presented at different spatial locations, or physical levers. One option yields reward with a higher probability (the better option) than the other (the worse option), but these reward contingencies switch at fixed or random times within an experimental session, creating ’reversals’. Crucially, reversals are not signaled and thus, subjects have to adjust their choice solely based on reward feedback to maximize their chance of winning a reward. Commonly used rewards include drops of juice or water (e.g., [60,61]), sucrose pellets (e.g., [62,63]), and monetary incentives for humans (e.g., [56,64]), which are delivered probabilistically at the end of each trial. For most simulations, we set the reward probabilities at 80 and 20 (corresponding to 80/20% reward schedule) with each block consisting of 80 trials, unless stated otherwise. Each block contains a reversal in the middle of the block, where reward probabilities on the two options switch. In the final set of simulations (Section 3.4), we also considered a generalized PRL task in which reward probabilities depended on one feature of each choice option (e.g., its color or its shape), while the other feature carried no information about reward.

2.2. Information-Theoretic Metrics

In this study, we utilized and extended the information-theoretic metrics introduced previously [29,41]. These measures—–grounded in conditional entropy, mutual information, and outcome-specific decompositions—–quantify how past rewards and choices shape the uncertainty surrounding an agent’s decision to stay with or switch from its previous choice (Figure 1b).

In general, for discrete random variables X and Y, the conditional entropy $H\left(Y\right|X)$ represents the remaining uncertainty in Y given the information about the variable X. Formally, it is defined as:

$$\begin{array}{cc}\hfill H\left(Y\right|X)& =\sum _{x\in X}P\left(x\right)·H\left(Y\right|X=x)\hfill \\ \hfill & =-\sum _{x\in X}\sum _{y\in Y}P(x,y){log}_2{\displaystyle \frac{P(x,y)}{P\left(x\right)}}\hfill \end{array}$$

(1)

Lower values of conditional entropy indicate that knowing the values of X reduces the uncertainty in Y, suggesting a strong dependence between the two variables.

Similarly, mutual information, denoted as $I(X;Y)$, quantifies the information shared between discrete variables X and Y. Higher values of $I(X;Y)$ indicate a greater dependency between variables, such that knowledge of Y would make X more predictable (less uncertain). This relationship is expressed using the difference between Shannon entropy for Y, $H\left(Y\right)$, and the conditional entropy, $H\left(Y\right|X)$:

$$\begin{array}{cc}\hfill I(X;Y)& =H\left(Y\right)-H\left(Y\right|X)\hfill \\ \hfill & =\sum _{x\in X}\sum _{y\in Y}P(x,y){log}_2{\displaystyle \frac{P(x,y)}{P\left(x\right)P\left(y\right)}}\hfill \end{array}$$

(2)

Building on these general formulations, we define behavioral metrics to quantify uncertainty in the agent’s choice strategy, specifically in terms of whether the agent "stays" with or "switches" from the previous choice option, given certain outcomes (Figure 1a). Specifically, we aim to quantify how uncertainty in choice strategy is reduced by certain task-related information: the previous reward outcome (R), and the previously chosen option ($Opt$). The resulting metrics include the conditional entropy of reward-dependent strategy (ERDS), the conditional entropy of option-dependent strategy (EODS), and the conditional entropy of reward and option-dependent strategy (ERODS). These are paired with mutual information metrics including mutual information between reward outcome and choice strategy (MIRS), mutual information between previous choice and choice strategy (MIOS), and mutual information between reward outcome, choice options, and choice strategy (MIROS).

More specifically, ERDS measures the influence of previous reward outcomes—reward vs. no reward (referred to as win vs. loss for simplicity)—on the uncertainty of the subsequent choice strategy (stay vs. switch), as follows:

$$\begin{array}{cc}\hfill \mathrm{ERDS}& =H(St{r}_t\mid R_{t-1})\hfill \\ \hfill & =H\left(Str\right)-I(R;Str)\hfill \\ \hfill & =-\sum _{R\in \left\{\mathrm{win},\mathrm{loss}\right\}}\sum _{Str\in \left\{\mathrm{stay},\mathrm{switch}\right\}}P(R,Str){log}_2{\displaystyle \frac{P(R,Str)}{P\left(R\right)}}.\hfill \end{array}$$

(3)

where $St{r}_t$ or $Str$ denotes the choice strategy between two subsequent trials (stay = 1, switch = 0) , $R_{t-1}$ is the reward outcome on the previous trial (reward or win $=1$ and no reward or loss $=0$), $P(R_{t-1},St{r}_t)$ is the joint probability of choice strategy given a reward outcome on the previous trial, and $P\left(R\right)$ is the probability of reward. In the equation above, $H\left(Str\right)$ is the Shannon entropy of strategy, measuring the randomness in choice strategy in terms of stay or switch:

$$H\left(Str\right)=-\sum _{Str\in \left\{\mathrm{stay},\mathrm{switch}\right\}}-P\left(Str\right){log}_{2}P\left(Str\right)=\left(P\left(\mathrm{stay}\right)·{log}_{2}P\left(\mathrm{stay}\right)+P\left(\mathrm{switch}\right)·{log}_{2}P\left(\mathrm{switch}\right)\right),$$

(4)

where $P\left(Str\right)$ is the probability of stay or switch ($P\left(stay\right)=1-P\left(switch\right)$). $I(R_{t-1};St{r}_t)$, which we refer to as MIRS, is the mutual information between reward outcome and strategy, equal to:

$$\begin{array}{cc}\hfill \mathrm{MIRS}& =I(R_{t-1};St{r}_t)\hfill \\ \hfill & =\sum _{R\in \left\{\mathrm{win},\mathrm{loss}\right\}}\sum _{Str\in \left\{\mathrm{stay},\mathrm{switch}\right\}}P(R,Str){log}_2{\displaystyle \frac{P(R,Str)}{P\left(R\right)P\left(Str\right)}}.\hfill \end{array}$$

(5)

Analogously, the conditional entropy of option-dependent strategy (EODS) represents the remaining uncertainty in the agent’s strategy after accounting for the choice made on the previous trial. It is defined as the difference between $H\left(Str\right)$ and the mutual information between the previous choice and strategy (MIOS), as follows:

$$\begin{array}{cc}\hfill \mathrm{EODS}& =H(St{r}_t\mid Op{t}_{t-1})\hfill \\ \hfill & =H\left(Str\right)-I(Opt;Str)\hfill \\ \hfill & =-\sum _{Opt\in \left\{\mathrm{better},\mathrm{worse}\right\}}\sum _{Str\in \left\{\mathrm{stay},\mathrm{switch}\right\}}P(Opt,Str){log}_2{\displaystyle \frac{P(Opt,Str)}{P\left(Opt\right)}},\hfill \end{array}$$

(6)

where $Op{t}_{t-1}$ indicates the option chosen on the previous trial, with 1 indicating the better option and 0 indicating the worse option, as defined by the assigned reward probabilities, and P(Opt) is the probability of choosing the better option. $I(Op{t}_{t-1};St{r}_t)$ denotes the mutual information between the chosen option and the agent’s strategy, referred to as MIOS, and is calculated as follows:

$$\begin{array}{cc}\hfill \mathrm{MIOS}& =I(Op{t}_{t-1};St{r}_t)\hfill \\ \hfill & =\sum _{Opt\in \left\{\mathrm{better},\mathrm{worse}\right\}}\sum _{Str\in \left\{\mathrm{stay},\mathrm{switch}\right\}}P(Opt,Str){log}_2{\displaystyle \frac{P(Opt,Str)}{P\left(Opt\right)P\left(Str\right)}}.\hfill \end{array}$$

(7)

To consider the combined effect of reward outcome and chosen option (i.e., winning or losing after choosing the better or worse option), we also considered a generalized metric that quantifies the combined effects of the two variables. Specifically, we define the conditional entropy of reward- and option-dependent strategy (ERODS), as follows:

$$\begin{array}{cc}\hfill \mathrm{ERODS}& =H(St{r}_t\mid R_{t-1},Op{t}_{t-1})\hfill \\ \hfill & =H\left(Str\right)-I(R,Opt;Str)\hfill \\ \hfill & =-\sum _{R\in \left\{\mathrm{win},\mathrm{loss}\right\}}\sum _{Opt\in \left\{\mathrm{better},\mathrm{worse}\right\}}\sum _{Str\in \left\{\mathrm{stay},\mathrm{switch}\right\}}P(R,Opt,Str){log}_2{\displaystyle \frac{P(R,Opt,Str)}{P(R,Opt)}}.\hfill \end{array}$$

(8)

Here, $I(R,Opt;Str)$ denotes the mutual information between the combination of the previous reward and choice outcomes ($R,Opt$) and subsequent choice strategy (MIROS):

$$\begin{array}{cc}\hfill \mathrm{MIROS}& =I(R_{t-1},Op{t}_{t-1};St{r}_t)\hfill \\ \hfill & =\sum _{R\in \left\{\mathrm{win},\mathrm{loss}\right\}}\sum _{Opt\in \left\{\mathrm{better},\mathrm{worse}\right\}}\sum _{Str\in \left\{\mathrm{stay},\mathrm{switch}\right\}}P(R,Opt,Str){log}_2{\displaystyle \frac{P(Opt,Str)}{P(R,Opt)P\left(Str\right)}}.\hfill \end{array}$$

(9)

2.2.1. Decomposition of Information-Theoretic Metrics

For the metrics described above, we also analyzed their decompositions into component values associated with each specific outcome of the conditioning variable. In the case of conditional entropy measures, this corresponds to the conditional entropy of Y given a specific value of $X=x$, weighted by the probability $P\left(x\right)$. For instance, ERDS can be decomposed into two components based on previous reward outcomes: $ERD{S}_{+}$ and $ERD{S}_{-}$, which reflect the uncertainty reduction following rewarded (win) and unrewarded (loss) trials, respectively [29]:

$$\begin{array}{cc}\hfill {\mathrm{ERDS}}_{+}& =H(St{r}_t\mid R_{t-1}=\mathrm{win})·P\left(\mathrm{win}\right)\hfill \\ \hfill & =-\left(P(\mathrm{stay},\mathrm{win})·{log}_2{\displaystyle \frac{P(\mathrm{stay},\mathrm{win})}{P\left(\mathrm{win}\right)}}+P(\mathrm{switch},\mathrm{win})·{log}_2{\displaystyle \frac{P(\mathrm{switch},\mathrm{win})}{P\left(\mathrm{win}\right)}}\right),\hfill \end{array}$$

(10)

$$\begin{array}{cc}\hfill {\mathrm{ERDS}}_{-}& =H(St{r}_t\mid R_{t-1}=\mathrm{loss})·P\left(\mathrm{loss}\right)\hfill \\ \hfill & =-\left(P(\mathrm{stay},\mathrm{loss})·{log}_2{\displaystyle \frac{P(\mathrm{stay},\mathrm{loss})}{P\left(\mathrm{loss}\right)}}+P(\mathrm{switch},\mathrm{loss})·{log}_2{\displaystyle \frac{P(\mathrm{switch},\mathrm{loss})}{P\left(\mathrm{loss}\right)}}\right).\hfill \end{array}$$

(11)

This decomposition guarantees that ${\mathrm{ERDS}}_{+}+{\mathrm{ERDS}}_{-}=\mathrm{ERDS}$ (Equation (3)).

Similarly, EODS can be decomposed into $EOD{S}_B$ and $EOD{S}_W$ based on the choice of the better or worse option on the previous trial:

$$\begin{array}{cc}\hfill {\mathrm{EODS}}_B& =H(St{r}_t\mid Op{t}_{t-1}=\mathrm{better})·P\left(\mathrm{better}\right)\hfill \\ \hfill & =-\left(P(\mathrm{stay},\mathrm{better})·{log}_2{\displaystyle \frac{P(\mathrm{stay},\mathrm{better})}{P\left(\mathrm{better}\right)}}+P(\mathrm{switch},\mathrm{better})·{log}_2{\displaystyle \frac{P(\mathrm{switch},\mathrm{better})}{P\left(\mathrm{better}\right)}}\right),\hfill \end{array}$$

(12)

$$\begin{array}{cc}\hfill {\mathrm{EODS}}_W& =H(St{r}_t\mid Op{t}_{t-1}=\mathrm{worse})·P\left(\mathrm{worse}\right)\hfill \\ \hfill & =-\left(P(\mathrm{stay},\mathrm{worse})·{log}_2{\displaystyle \frac{P(\mathrm{stay},\mathrm{worse})}{P\left(\mathrm{worse}\right)}}+P(\mathrm{switch},\mathrm{worse})·{log}_2{\displaystyle \frac{P(\mathrm{switch},\mathrm{worse})}{P\left(\mathrm{worse}\right)}}\right).\hfill \end{array}$$

(13)

Finally, ERODS is decomposed into four components based on the combination of the previous reward and the choice option:

$$\begin{array}{cc}\hfill {\mathrm{ERODS}}_{B+}& =H(St{r}_t\mid R_{t-1}=\mathrm{win},Op{t}_{t-1}=better)·P\left(\mathrm{win},\mathrm{better}\right)\hfill \\ \hfill & =-\left(P(\mathrm{stay},\mathrm{win},\mathrm{better})·{log}_2{\displaystyle \frac{P(\mathrm{stay},\mathrm{win},\mathrm{better})}{P(\mathrm{win},\mathrm{better})}}\right.\hfill \\ \hfill & +\left.P(\mathrm{switch},\mathrm{win},\mathrm{better})·{log}_2{\displaystyle \frac{P(\mathrm{switch},\mathrm{win},\mathrm{better})}{P(\mathrm{win},\mathrm{better})}}\right),\hfill \end{array}$$

(14)

$$\begin{array}{cc}\hfill {\mathrm{ERODS}}_{B-}& =H(St{r}_t\mid R_{t-1}=\mathrm{loss},Op{t}_{t-1}=better)·P\left(\mathrm{loss},\mathrm{better}\right)\hfill \\ \hfill & =-\left(P(\mathrm{stay},\mathrm{loss},\mathrm{better})·{log}_2{\displaystyle \frac{P(\mathrm{stay},\mathrm{loss},\mathrm{better})}{P(\mathrm{loss},\mathrm{better})}}\right.\hfill \\ \hfill & +\left.P(\mathrm{switch},\mathrm{loss},\mathrm{better})·{log}_2{\displaystyle \frac{P(\mathrm{switch},\mathrm{loss},\mathrm{better})}{P(\mathrm{loss},\mathrm{better})}}\right),\hfill \end{array}$$

(15)

$$\begin{array}{cc}\hfill {\mathrm{ERODS}}_{W+}& =H(St{r}_t\mid R_{t-1}=\mathrm{win},Op{t}_{t-1}=worse)·P\left(\mathrm{win},\mathrm{worse}\right)\hfill \\ \hfill & =-\left(P(\mathrm{stay},\mathrm{win},\mathrm{worse})·{log}_2{\displaystyle \frac{P(\mathrm{stay},\mathrm{win},\mathrm{worse})}{P(\mathrm{win},\mathrm{worse})}}\right.\hfill \\ \hfill & +\left.P(\mathrm{switch},\mathrm{win},\mathrm{worse})·{log}_2{\displaystyle \frac{P(\mathrm{switch},\mathrm{win},\mathrm{worse})}{P(\mathrm{win},\mathrm{worse})}}\right),\hfill \end{array}$$

(16)

$$\begin{array}{cc}\hfill {\mathrm{ERODS}}_{W-}& =H(St{r}_t\mid R_{t-1}=\mathrm{loss},Op{t}_{t-1}=\mathrm{worse})·P\left(\mathrm{loss},\mathrm{worse}\right)\hfill \\ \hfill & =-\left(P(\mathrm{stay},\mathrm{loss},\mathrm{worse})·{log}_2{\displaystyle \frac{P(\mathrm{stay},\mathrm{loss},\mathrm{worse})}{P(\mathrm{loss},\mathrm{worse})}}\right.\hfill \\ \hfill & +\left.P(\mathrm{switch},\mathrm{loss},\mathrm{worse})·{log}_2{\displaystyle \frac{P(\mathrm{switch},\mathrm{loss},\mathrm{worse})}{P(\mathrm{loss},\mathrm{worse})}}\right),\hfill \end{array}$$

(17)

where the subscripts refer to winning after choosing the better option ($B+$), losing after the better option ($B-$), winning after the worse option ($W+$), and losing after the worse option ($W-$). Note that ERODS can be alternatively decomposed based on either reward outcome or choice option alone, by adding the relevant components. For example, this metric can be decomposed based on whether the previous trial resulted in reward or no reward, as follows:

$$\begin{array}{c}\hfill {\mathrm{ERODS}}_{+}={\mathrm{ERODS}}_{B+}+{\mathrm{ERODS}}_{W+},\\ \hfill {\mathrm{ERODS}}_{-}={\mathrm{ERODS}}_{B-}+{\mathrm{ERODS}}_{W-}.\end{array}$$

(18)

This metric can also be decomposed based on whether the better or worse option was selected on the previous trial:

$$\begin{array}{c}\hfill {\mathrm{ERODS}}_B={\mathrm{ERODS}}_{B+}+{\mathrm{ERODS}}_{B-},\\ \hfill {\mathrm{ERODS}}_W={\mathrm{ERODS}}_{W+}+{\mathrm{ERODS}}_{W-}.\end{array}$$

(19)

For decomposing mutual information metrics, we utilize the general formulation given by:

$$\begin{array}{cc}\hfill I(X=x;Y)& =H\left(Y\right)-H\left(Y\right|X=x)\hfill \\ \hfill & =-\sum _{y\in Y}P\left(y\right)·{log}_{2}P\left(y\right)+\sum _{y\in Y}P\left(y\right|x)·{log}_{2}P\left(y\right|x),\hfill \end{array}$$

(20)

This quantity represents the information that a specific value $X=x$ provides about Y, and is known as pointwise mutual information. The expected value of pointwise mutual information over all values of X is equal to the mutual information between X and Y (Equation (2)):

$$\begin{array}{cc}\hfill I(X;Y)& =H\left(Y\right)-H\left(Y\right|X)\hfill \\ \hfill & =\sum _{x\in X}P\left(x\right)H\left(Y\right)-\sum _{x\in X}P\left(x\right)H(Y\mid X=x)\hfill \\ \hfill & =\sum _{x\in X}P\left(x\right)\{H\left(Y\right)-H(Y\mid X=x)\}\hfill \\ \hfill & =\sum _{x\in X}P\left(x\right)I(X=x;Y)\hfill \end{array}$$

(21)

Unlike the mutual information $I(X;Y)$, which is always nonnegative, the pointwise mutual information for a specific event x, $I(X=x;Y)$, can take negative values when $H\left(Y\right)<H\left(Y\right|X=x)$[65]. Conceptually, negative values indicate that the outcome x is misleading—or misinformative—about Y [66].

Using this definition, we compute two components of MIRS, ${\mathrm{MIRS}}_{+}$ and ${\mathrm{MIRS}}_{-}$, corresponding to previously rewarded and unrewarded outcome, respectively:

$$\begin{array}{cc}\hfill {\mathrm{MIRS}}_{+}& =I(R_{t-1}=\mathrm{win};St{r}_t)·P\left(\mathrm{win}\right)\hfill \\ \hfill & =\{H\left(Str\right)-H(St{r}_t\mid R_{t-1}=\mathrm{win})\}·P\left(\mathrm{win}\right),\hfill \end{array}$$

(22)

$$\begin{array}{cc}\hfill {\mathrm{MIRS}}_{-}& =I(R_{t-1}=\mathrm{loss};Str)·P\left(\mathrm{loss}\right)\hfill \\ \hfill & =\{H\left(Str\right)-H(St{r}_t\mid R_{t-1}=\mathrm{loss})\}·P\left(\mathrm{loss}\right).\hfill \end{array}$$

(23)

These include the decomposition terms for conditional entropy (Equations (10)-11), which satisfy the identity ($\mathrm{MIRS}={\mathrm{MIRS}}_{+}+{\mathrm{MIRS}}_{-}$). Similarly, the decompositions for mutual information between the option chosen on the previous trial and strategy, MIOS, are defined as:

$$\begin{array}{cc}\hfill {\mathrm{MIOS}}_B& =I(Op{t}_{t-1}=\mathrm{better};St{r}_t)·P\left(\mathrm{better}\right)\hfill \\ \hfill & =\{H\left(Str\right)-H(St{r}_t\mid Op{t}_{t-1}=\mathrm{better})\}·P\left(\mathrm{better}\right),\hfill \end{array}$$

(24)

$$\begin{array}{cc}\hfill {\mathrm{MIOS}}_W& =I(Op{t}_{t-1}=\mathrm{worse};St{r}_t)·P\left(\mathrm{worse}\right)\hfill \\ \hfill & =\{H\left(Str\right)-H(St{r}_t\mid Op{t}_{t-1}=\mathrm{worse})\}·P\left(\mathrm{worse}\right).\hfill \end{array}$$

(25)

These terms satisfy $\mathrm{MIOS}={\mathrm{MIOS}}_B+{\mathrm{MIOS}}_W$. Lastly, the decompositions for MIROS are given as:

$$\begin{array}{cc}\hfill {\mathrm{MIROS}}_{B+}& =I(R_{t-1}=\mathrm{win},Op{t}_{t-1}=\mathrm{better};St{r}_t)·P\left(\mathrm{win},\mathrm{better}\right)\hfill \\ \hfill & =\{H\left(Str\right)-H(St{r}_t\mid R_{t-1}=\mathrm{win},Op{t}_{t-1}=\mathrm{better})\}·P\left(\mathrm{win},\mathrm{better}\right),\hfill \end{array}$$

(26)

$$\begin{array}{cc}\hfill {\mathrm{MIROS}}_{B-}& =I(R_{t-1}=\mathrm{loss},Op{t}_{t-1}=\mathrm{better};St{r}_t)·P\left(\mathrm{loss},\mathrm{better}\right)\hfill \\ \hfill & =\{H\left(Str\right)-H(St{r}_{t-1}\mid R_{t-1}=\mathrm{loss},Op{t}_{t-1}=\mathrm{better})\}·P\left(\mathrm{loss},\mathrm{better}\right),\hfill \end{array}$$

(27)

$$\begin{array}{cc}\hfill {\mathrm{MIROS}}_{W+}& =I(R_{t-1}=\mathrm{win},Op{t}_{t-1}=\mathrm{worse};St{r}_t)·P\left(\mathrm{win},\mathrm{worse}\right)\hfill \\ \hfill & =\{H\left(Str\right)-H(St{r}_t\mid R_{t-1}=\mathrm{win},Op{t}_{t-1}=\mathrm{worse})\}·P\left(\mathrm{win},\mathrm{worse}\right),\hfill \end{array}$$

(28)

$$\begin{array}{cc}\hfill {\mathrm{MIROS}}_{W-}& =I(R_{t-1}=\mathrm{loss},Op{t}_{t-1}=\mathrm{worse};St{r}_t)·P\left(\mathrm{loss},\mathrm{worse}\right)\hfill \\ \hfill & =\{H\left(Str\right)-H(St{r}_t\mid R_{t-1}=\mathrm{loss},Op{t}_{t-1}=\mathrm{worse})\}·P\left(\mathrm{loss},\mathrm{worse}\right).\hfill \end{array}$$

(29)

These terms quantify the shared information between each instance of reward-option combination and the agent’s strategy. Alternative decompositions can be computed similarly to ERODS as in Equations (18) and (19). Notably, these decompositions reveal how specific combinations of trial outcomes influence subsequent strategy, providing additional information beyond the MIRS and MIOS metrics.

2.2.2. Normalization of Mutual Information Metrics

The mutual information metrics defined thus far quantify the total amount of shared information between the random variables X and Y. However, one might be interested in the fraction of uncertainty in Y that is reduced by the knowledge of X, rather than the absolute amount of shared information [67,68,69]. In our setting, if the agent’s strategy is largely deterministic (i.e., $H\left(Str\right)$ is low), then the mutual information will also be low due to small value of $H\left(Str\right)$ itself (e.g., see the area of gray circle specifying $H\left(Str\right)$ in Figure 1b). Therefore, to account for the baseline uncertainty in the strategy, we also considered normalized mutual information metrics, obtained by dividing the mutual information by $H\left(Str\right)$.

More specifically, we define normalized mutual information metrics for MIRS, MIOS, MIROS as follows:

$$\begin{array}{cc}\hfill \mathrm{n-MIRS}& =I(R_{t-1};St{r}_t)/H\left(Str\right),\hfill \end{array}$$

(30)

$$\begin{array}{cc}\hfill \mathrm{n-MIOS}& =I(Op{t}_{t-1};St{r}_t)/H\left(Str\right),\hfill \end{array}$$

(31)

$$\begin{array}{cc}\hfill \mathrm{n-MIROS}& =I(R_{t-1},Op{t}_{t-1};St{r}_t)/H\left(Str\right).\hfill \end{array}$$

(32)

These metrics quantify the proportion of uncertainty in $H\left(Str\right)$ that is explained by each corresponding mutual information term. The normalized decompositions for each metric are similarly obtained by dividing each component by $H\left(Str\right)$. For example, normalized mutual information between reward and strategy is computed as:

$$\begin{array}{cc}\hfill \mathrm{n-MIRS}& ={\mathrm{n-MIRS}}_{+}+{\mathrm{n}-\mathrm{MIRS}}_{-},\hfill \\ \hfill & ={\displaystyle \frac{P\left(\mathrm{win}\right)·I(R_{t-1}=\mathrm{win};St{r}_t)}{H\left(Str\right)}}+{\displaystyle \frac{P\left(\mathrm{loss}\right)·I(R_{t-1}=\mathrm{loss};St{r}_t)}{H\left(Str\right)}},\hfill \end{array}$$

(33)

and analogously, normalized mutual information between choice option and strategy is defined as:

$$\begin{array}{cc}\hfill \mathrm{n-MIOS}& ={\mathrm{n-MIOS}}_B+{\mathrm{n-MIOS}}_W,\hfill \\ \hfill & ={\displaystyle \frac{P\left(\mathrm{better}\right)·I(Op{t}_{t-1}=\mathrm{better};St{r}_t)}{H\left(Str\right)}}+{\displaystyle \frac{P\left(\mathrm{worse}\right)·I(Op{t}_{t-1}=\mathrm{worse};St{r}_t)}{H\left(Str\right)}}.\hfill \end{array}$$

(34)

Finally, the normalized mutual information between reward-choice combination and strategy can be computed as follows (dropping subscripts for trials for simplicity):

$$\begin{array}{cc}\hfill \mathrm{n-MIROS}& ={\mathrm{n-MIROS}}_{B+}+{\mathrm{n-MIROS}}_{B-}+{\mathrm{n-MIOS}}_{W+}+{\mathrm{n-MIOS}}_{W+},\hfill \\ \hfill & ={\displaystyle \frac{P\left(\mathrm{better},\mathrm{win}\right)·I(R=\mathrm{win},Opt=\mathrm{better};Str)}{H\left(Str\right)}}+{\displaystyle \frac{P\left(\mathrm{better},\mathrm{loss}\right)·I(R=\mathrm{loss},Opt=\mathrm{better};Str)}{H\left(Str\right)}}\hfill \\ \hfill & +{\displaystyle \frac{P\left(\mathrm{worse},\mathrm{win}\right)·I(R=\mathrm{win},Opt=\mathrm{worse};Str)}{H\left(Str\right)}}+{\displaystyle \frac{P\left(\mathrm{worse},\mathrm{loss}\right)·I(R=\mathrm{loss},Opt=\mathrm{worse};Str)}{H\left(Str\right)}}.\hfill \end{array}$$

(35)

2.3. Reinforcement Learning Models and Simulations

We used multiple reinforcement learning (RL) models to simulate two variants of the probabilistic reversal learning task in order to illustrate that information-theoretic metrics can detect: (1) the differential learning rates for rewarded versus unrewarded trials, (2) changes in learning rates due to metaplasticity, (3) the influence of reward harvest rate, and (4) the use of alternative learning strategies in multi-dimensional reward environments.

For each task, we generated synthetic choice data using a range of model parameters and computed a set of relevant behavioral metrics from the resulting data. Below, we outline the general architecture of the RL models used, which are based on the standard Q-learning algorithm [22,25,26,27]. In this algorithm, the reward values of the two choice options, $Q_A$ and $Q_B$, are updated on a trial-by-trial basis using reward feedback, and the difference between these values is used to make a choice (Figure 2a). Specifically, the value of the chosen option on a given trial t, $Q_c$, is updated according to the following rule:

$$\begin{array}{c}\hfill Q_c(t+1)=Q_c\left(t\right)+\alpha (R\left(t\right)-Q_c\left(t\right)),\end{array}$$

(36)

where $R\left(t\right)$ is the binary reward outcome (1 if rewarded, 0 if unrewarded), and $\alpha$ denotes the learning rate, which determines the magnitude of the value update by scaling the reward prediction error, $RPE=R\left(t\right)-Q_c\left(t\right)$. The value of the unchosen option remains unchanged. All value estimates are initialized at $0.5$ for the first trial.

On each trial, the choice is determined stochastically based on the probability of selecting each option, $P_j$, computed using the softmax function as follows:

$$\begin{array}{c}\hfill P_j\left(t\right)={\displaystyle \frac{e^{\beta Q_j\left(t\right)}}{e^{\beta Q_j\left(t\right)}+e^{\beta Q_k\left(t\right)}}}={\displaystyle \frac{1}{1+e^{-\beta (Q_j\left(t\right)-Q_k\left(t\right))}}},\end{array}$$

(37)

where the indices j and k correspond to the two choice alternatives, and $\beta$ is the inverse temperature parameter that controls the sensitivity to value differences (i.e., slope of the softmax function). We selected the softmax function because it is widely adopted in empirical research and follows naturally from normative rationality principles [70,71]. In its simplest form, the RL model has only two free parameters: $\alpha$ and $\beta$. Below, we describe the variants of the RL models used in each set of simulations.

2.3.1. Simulations of Positivity Bias

In this set of simulations, we examined the positivity bias—also known as the optimism bias—which refers to the tendency of an agent to learn more from feedback with positive valence than from feedback with negative valence [42,43,44,45]. This bias is often quantified by estimating separate learning rates for the outcomes that either are “better” or “worse” than expected, or equivalently, for positive and negative prediction errors, in the context of error-driven learning. In the case of binary reward outcomes, these correspond to the learning rates following rewarded and unrewarded trials, respectively. Therefore, the learning rule in Equation (36) can be generalized as:

$$Q_c(t+1)=\left\{\begin{array}{cc}\hfill & Q_c\left(t\right)+{\alpha }_{+}(R\left(t\right)-Q_c\left(t\right)),\mathrm{if}R\left(t\right)=1\hfill \\ \hfill & Q_c\left(t\right)+{\alpha }_{-}(R\left(t\right)-Q_c\left(t\right)),\mathrm{if}R\left(t\right)=0,\hfill \end{array}\right.$$

(38)

where ${\alpha }_{+}$ and ${\alpha }_{-}$ denote the learning rates used to update the value of the chosen option on rewarded and unrewarded trials, respectively. Positivity bias is thus formally characterized by the condition where ${\alpha }_{+}>{\alpha }_{-}$.

We used the above RL model to simulate the choice behavior in a probabilistic reversal learning task. Each block consisted of 80 trials with an 80/20% reward schedule, and a reversal in reward contingencies occurred after trial 40. To examine how positivity bias can be detected using information-theoretic metrics, we simulated choice behavior using different combinations of (${\alpha }_{+}$, ${\alpha }_{-}$) values. We set the inverse temperature at $\beta =10$, consistent with values observed in previous experimental studies [42,72]. To explore a plausible range of learning rates while avoiding extreme values, we varied each learning rate ${\alpha }_i\in [0.1,0.9]$ in increments of 0.05, resulting in a 17-by-17 grid of (${\alpha }_{+}$, ${\alpha }_{-}$) combinations. For each point in this parameter space, choice behavior was simulated over 10,000 blocks.

To test whether the information-theoretic measures contain sufficient information to determine the presence of positivity bias (${\alpha }_{+}>{\alpha }_{-}$), we utilized a linear decoder based on logistic regression, implemented using the fitclinear function in MATLAB). Decoders were trained on metrics computed from the choice behavior of training samples and tested on a separate held-out set, with the goal of predicting whether the underlying RL model used to generate the data had ${\alpha }_{+}>{\alpha }_{-}$.

More specifically, in each decoding experiment, we randomly sampled RL agents, each assigned a true (${\alpha }_{+}$, ${\alpha }_{-}$) pair drawn independently from uniform distributions over ${\alpha }_i\in [0.1,0.9]$ (Figure 3e). Twenty RL agents were assigned to each group, labeled as either "optimistic" (${\alpha }_{+}$ > ${\alpha }_{-}$) or "pessimistic" (${\alpha }_{+}$ < ${\alpha }_{-}$). For each agent, every simulated block (out of total of 100) used a distinct (${\alpha }_{+}$, ${\alpha }_{-}$) pair sampled from gaussian distributions ($\sigma =0.03$) centered on that agent’s true (${\alpha }_{+}$, ${\alpha }_{-}$) values. For each block, we obtained theoretical averages of the information-theoretic metrics by averaging across 10 repeated simulations. To evaluate decoding performance, we employed a leave-one-out procedure at the agent level: the decoder was trained on the information-theoretic metrics computed from the choice behavior of all agents except the one being tested. The training data were balanced to include an equal number of optimistic and pessimistic agents. Decoding accuracy was averaged across 100 independent decoding experiments, each using a unique set of RL agents.

Finally, to benchmark our decoder against behavioral features beyond the information-theoretic metrics, we trained a separate decoder on the coefficients of a logistic regression predicting choice from past choice and reward history. More specifically, we fit the following logistic regression model [30] to the same segment of choice data:

$$log\left({\displaystyle \frac{P\left(C\right(t)=j)}{P\left(C\right(t)=k)}}\right)={\beta }_0+\sum _{i=1}^5{\beta }_i^R(R_j(t-i)-R_k(t-i))+\sum _{i=1}^5{\beta }_i^U(U_j(t-i)-U_k(t-j)),$$

(39)

where $C\left(t\right)$ indicates the choice made on trial t, j and k index the two choice options, $R_i$ indicates a rewarded choice (1 if rewarded, 0 otherwise), and $U_i$ denotes an unrewarded choice (1 if unrewarded, 0 otherwise). We used the resulting ten regression coefficients (${\beta }^R$ and ${\beta }^U$) to decode whether a given agent is optimistic or pessimistic.

2.3.2. Simulations of Reward-Dependent Metaplasticity

In this set of simulations, we examined whether adjustments in the learning rates—predicted by reward-dependent metaplasticity [46,47]—can be detected using information-theoretic metrics. Importantly, reward-dependent metaplasticity enables dynamic adjustments of synaptic plasticity over time and can account for stimulus- or action-specific learning rates, including asymmetries such as positivity bias. Using the same probabilistic reversal learning task with an 80/20% reward schedule as in the positivity bias simulations, we compared the choice behavior of the metaplastic model to that of the ’plastic’ model, which is equivalent to the standard RL model described earlier. Below, we briefly describe the metaplastic model (see [46] for more details).

Importantly, the standard RL algorithm can be implemented through binary synapses that encode and update the value of each choice option through reward-dependent plasticity: transitioning from a "weak" to a "strong" state following reward (reward-dependent potentiation), and from a "strong" to a "weak" state following no reward (reward-dependent depression) [47,73,74] (Figure 4a). Importantly, the proportion of synapses in the "strong" state provides an estimate of the value of a given choice option ($Q_j$), as this quantity increases and decreases following reward and no reward, respectively [73]. We assume that only the synapses associated with the chosen option undergo state transitions according to the reward outcome (reward-dependent plasticity), while those associated with the unchosen option remain unchanged—consistent with the assumption of standard Q-learning model.

The metaplastic model generalizes the above reward-dependent plasticity mechanism by introducing multiple meta-states for each level of synaptic efficacy, with synapses occupying deeper meta-states being more resistant—or stable—with respect to future changes (Figure 4b). More specifically, synapses in the metaplastic model undergo both plastic and metaplastic transitions. During plastic transitions, synaptic efficacy shifts between "weak" and "strong" states. During metaplastic transitions, synaptic efficacy remains unchanged, but the stability is modified as the synapse transitions to a deeper or shallower meta-state (Figure 4b).

Formally, higher stability of deeper meta-states is captured by decreasing transition probabilities $q_i$ for the $ith$ meta-state, governed by a power law as follows:

$$q_i=q_1^{{\displaystyle \frac{i(m-2)+1}{m-1}}}\mathrm{for}2\le i\le m,$$

(40)

where m is the number of meta-states, $q_1$ is the (baseline) transition probability between the most unstable weak and most unstable strong meta-states, and $q_i$ is the transition probability from weak (or strong) meta-state $i+1$ to the most unstable strong (or weak) state in response to positive (or negative) reward feedback (reward vs. no reward), as indicated by the diagonal arrows in Figure 4b.

Additionally, metaplastic synapses can undergo transitions that do not alter their synaptic efficacy (vertical arrows in Figure 4b), with the probability of such transitions decreasing for deeper meta-states:

$$p_i={p_1}^i\mathrm{for}2\le i\le m-1,$$

(41)

where $p_i$ indicate the transition probability between the $ith$ and the ${(i+1)}^{th}$ meta-states. After positive reward feedback (potentiation events), "weak" synapses undergo transition toward less stable meta-states while "strong" synapses transition toward more stable, deeper meta-states. Conversely, after negative reward feedback (depression events) "weak" synapses undergo transition toward more stable, deeper meta-states while "strong" synapses transition toward less stable meta-states (vertical arrows in Figure 4b).

The value of each option in the metaplastic model is computed by summing over the fractions of synapses in the strong states:

$$Q_j\left(t\right)=\sum _{i=1}^m{S}_{j,i}\left(t\right),$$

(42)

where j indexes the choice option, and $S_{j,i}\left(t\right)$ indicates the fraction of synapses in the $ith$ strong meta-state on a given trial.

Because the metaplastic model includes multiple meta-states with different transition rates, its update in response to reward feedback—and thus its rate of learning—changes over time. This dynamic can be quantified using two "effective" learning rates computed on each trial. The effective learning rate following reward (${\alpha }_{+}^{eff}$) is defined as the fraction of synapses that transition from weak to strong states, while the effective learning rate following no reward (${\alpha }_{-}^{eff}$) reflects the fraction transitioning from strong to weak states, as follows:

$$\begin{array}{cc}\hfill {\alpha }_{+}^{eff}\left(t\right)& ={\displaystyle \frac{Q_c(t+1)-Q_c\left(t\right)}{R\left(t\right)-Q_c\left(t\right)}}\mathrm{if}R\left(t\right)=1,\hfill \\ \hfill {\alpha }_{-}^{eff}\left(t\right)& ={\displaystyle \frac{Q_c(t+1)-Q_c\left(t\right)}{R\left(t\right)-Q_c\left(t\right)}}\mathrm{if}R\left(t\right)=0,\hfill \end{array}$$

(43)

where the numerator indicates the change in the overall value of the chosen option, while the denominator corresponds to the prediction error.

To simulate choice behavior, we set $\alpha =0.3$ and $\beta =10$ for the plastic model with a single learning rate. For the metaplastic model, we used the model with $m=4$ meta-states and set $p_1=0.4$ for the baseline meta-transition probability. We set the transition probability $q_1=0.516$, which yields an initial effective learning rate of 0.3 on the first trial of the block ( Figure 4c). We also tested the choice behavior of a plastic model with differential learning rates for rewarded and unrewarded outcomes (${\alpha }_{+}\ne {\alpha }_{-}$). These learning rates were estimated by fitting the plastic model to the choice behavior of the metaplastic model on each block via maximum-likelihood estimation (optimized with fmincon function in MATLAB).

Similar to the positivity bias simulations, we used a linear decoder to test whether the emergence of positivity bias (${\alpha }_{+}>{\alpha }_{-}$)—driven by metaplasticity over time—can be detected using the information-theoretic metrics. The training procedure involved randomly drawing a set of (${\alpha }_{+},{\alpha }_{-}$) values from the grid of [0.2, 0.4] in a step size of 0.01, under the assumption that the approximate range of $\alpha$ value is known ($\alpha =0.3$ in our case). Each parameter space was labeled as either "optimistic" (${\alpha }_{+}>{\alpha }_{-}$) or "pessimistic" (${\alpha }_{+}<{\alpha }_{-}$). The points with ${\alpha }_{+}={\alpha }_{-}$ were dropped from the training set. We then computed the decoder’s posterior probability on held-out samples, which were generated from the choice behavior of the three models: (1) a plastic model with a single learning rate (${\alpha }_{+}={\alpha }_{-}=\alpha$), (2) the metaplastic model, and (3) a plastic model with separate learning rates (${\alpha }_{+}\ne {\alpha }_{-}$) estimated from the choice behavior of the metaplastic model.

2.3.3. Simulations of Reward Harvest Rate Effects on Behavior

In this set of simulations, we examined whether the influence of the overall reward rate on a long time scale—referred to as reward harvest rate—can be detected using the information-theoretic metrics. We used the same probabilistic reversal learning task as in the positivity bias simulations and tested two RL models for generating choice behavior: (1) the standard RL model with a single learning rate, as described earlier; and (2) an augmented RL model that included an additional adjustment mechanism based on reward harvest rate (RL with RHR model; Figure 5a). The latter model served as the ground truth for a potential pathway through which reward harvest rate could influence choice strategy—specifically, by increasing the tendency to win-stay and lose-switch as the harvest reward rate increases. More specifically, the augmented model tracks a third variable H, representing the overall reward harvest rate independent of any specific choice option, and updates it as follows:

$$H(t+1)=H\left(t\right)+{\alpha }_H(R\left(t\right)-H\left(t\right)),$$

(44)

where ${\alpha }_H$ is the update rate for H. Therefore, H provides an exponentially weighted moving average of harvested rewards. For simplicity, we set ${\alpha }_H=\alpha =0.3$ in the simulations. In practice, although ${\alpha }_H$ can be directly optimized to the data, one can equally justify choosing an arbitrary ${\alpha }_H$ for behavioral analysis to avoid modeling assumptions and obtain model-independent measures. To incorporate a win-stay/lose-switch bias that is modulated by the reward harvest rate H, the choice probability in this model is computed as follows:

$$\begin{array}{c}\hfill P_j\left(t\right)={\displaystyle \frac{1}{1+e^{-\left[\beta (Q_j\left(t\right)-Q_k\left(t\right))+{\beta }_{H}Bia{s}_j\left(t\right)\right]}}},\end{array}$$

(45)

where ${\beta }_H$ is a free parameter controlling the overall influence of the win-stay/lose-switch bias term for option j, denoted as $Bias$. This bias term is calculated as follows:

$$\begin{array}{c}\hfill Bia{s}_j\left(t\right)=C_j(t-1)·\left(R(t-1)-H(t-1)\right),\end{array}$$

(46)

where $C_j$ indicates whether option j was chosen in the previous trial (1 if chosen, -1 otherwise), and the term $R-H$ dynamically adjusts the agent’s tendency to repeat (after reward) or switch (after no reward) based on how much the recent reward outcome differs from reward harvest rate H. In our simulations, we set $\beta =10,{\beta }_H=2$.

To quantify the effect of reward harvest rate on choice strategy, we introduced an additional mutual information metric, referred to as MIRHS, which captures the joint influence of previous reward outcome and the "reward harvest state" (as defined by reward rate) on the agent’s strategy, as follows:

$$\begin{array}{cc}\hfill \mathrm{MIRHS}& =I(R_{t-1},RH{S}_{t-1};St{r}_t)\hfill \\ \hfill & =\sum _{R\in \left\{\mathrm{win},\mathrm{loss}\right\}}\sum _{RHS\in \left\{\mathrm{high},\mathrm{low}\right\}}\sum _{Str\in \left\{\mathrm{stay},\mathrm{switch}\right\}}P(R,RHS,Str){log}_2{\displaystyle \frac{P(RHS,Str)}{P(R,RHS)P\left(Str\right)}},\hfill \end{array}$$

(47)

where RHS denotes the reward harvest state—a binary, discretized variable indicating whether the reward harvest rate H on a given (previous) trial was classified as "high" or "low", based on the median split within the current block (Figure 5b) [35]. We also used a normalized version of this metric, n-MIRHS $=I(R_{t-1},RH{S}_{t-1};St{r}_t)/H\left(St{r}_t\right)$, which is decomposed as follows (dropping subscripts for trials for simplicity):

$$\begin{array}{cc}\hfill \mathrm{n-MIRHS}& ={\mathrm{n-MIRHS}}_{high+}+{\mathrm{n-MIRHS}}_{high-}+{\mathrm{n-MIRHS}}_{low+}+{\mathrm{n-MIRHS}}_{low+},\hfill \\ \hfill & ={\displaystyle \frac{P\left(\mathrm{high},\mathrm{win}\right)·I(R=\mathrm{win},RHS=\mathrm{high};Str)}{H\left(Str\right)}}+{\displaystyle \frac{P\left(\mathrm{high},\mathrm{loss}\right)·I(R=\mathrm{loss},RHS=\mathrm{high};Str)}{H\left(Str\right)}}\hfill \\ \hfill & +{\displaystyle \frac{P\left(\mathrm{low},\mathrm{win}\right)·I(R=\mathrm{win},RHS=\mathrm{low};Str)}{H\left(Str\right)}}+{\displaystyle \frac{P\left(\mathrm{low},\mathrm{loss}\right)·I(R=\mathrm{loss},RHS=\mathrm{low};Str)}{H\left(Str\right)}},\hfill \end{array}$$

(48)

where the subscripts of the metric refer to winning during a high RHS state ($high+$), losing during a high RHS state ($high-$), winning during a low RHS state ($low+$), and losing during a low RHS state ($low-$). These metrics therefore separate the influence of immediate, reward outcome from that of the reward harvest state. To isolate the overall effect of $RHS$, we also computed an alternative decompositions as follows:

$$\begin{array}{c}\hfill {\mathrm{MIRHS}}_{high}={\mathrm{MIRHS}}_{high+}+{\mathrm{MIRHS}}_{high-},\end{array}$$

(49)

$$\begin{array}{c}\hfill {\mathrm{MIRHS}}_{low}={\mathrm{MIRHS}}_{low+}+{\mathrm{MIRHS}}_{low-},\end{array}$$

(50)

2.3.4. Simulations of Learning in Multidimensional Reward Environments

To investigate the presence of alternative learning strategies in the agent’s behavior, we conducted a final set of simulations using a variant of the probabilistic reversal learning task in which different attributes of the choice options predicted reward outcomes at different time points.

In this task, at any given point within a block of trials, only one of the two choice attributes—the shape and the color of the stimuli—is predictive of reward probabilities (80% vs 20%). Initially, the reward schedule for a given block is assigned to the color of the stimuli regardless of their shape, with higher reward probability assigned to either green or orange objects. Therefore, unlike the reversal learning task used in the previous simulations, this task involves two types of reversals: (i) reversal in the reward probability, as previously considered, and (ii) reversal in the feature predictive of reward (Figure 6a). Therefore, in a Color-to-Color (or similarly Shape-to-Shape) block, the reward contingency was reversed between the better and worse values of the same attribute—for example, between two colors (Figure 6c). In contrast, in a Color-to-Shape block, the predictive attribute switched, and reward became associated with the shape of the object (e.g., triangle = 80% and square = 20%), regardless of its color (Figure 6d). The identity of the block type was unknown to the agent, and the two block types were randomly interleaved throughout the session. The Agent therefore had to adapt their learning and choice strategies solely based on the reward feedback they received.

To capture the strategies that more realistic agents might adopt, we used a generalized RL model that dynamically arbitrates between competing learning strategies. To that end, we used a variant of the model introduced by Woo et al. [38], which simultaneously tracks the values of multiple choice attributes (Figure 6b). In this model, the value estimates for each attribute are updated simultaneously as follows:

$$\begin{array}{cc}\hfill Q_{c,i}(t+1)& =Q_{c,i}\left(t\right)+\alpha (R\left(t\right)-Q_{c,i}\left(t\right)),\hfill \end{array}$$

(51)

$$\begin{array}{cc}\hfill Q_{u,i}(t+1)& =(1-\gamma )Q_{u,i}\left(t\right),\hfill \end{array}$$

(52)

where i indexes the choice attribute (Color or Shape), and c indexes the chosen option within each attribute (i.e., Color $\in \{green,orange\}$ and Shape $\in \{triangle,square\}$). The index u refers to the unchosen option within each attribute, and $\gamma$ denotes the decay rate of value for unchosen options. Note that this decay mechanism is biologically motivated and included to account for the increased number of choice features in this model, reflecting the brain’s limited capacity to retain the memory of all value estimates. To generate a choice, the model first computes the overall value, V, of the two choice alternatives through a linear weighted combination of the attributes, as follows:

$$\begin{array}{c}\hfill V_j\left(t\right)=Q_{Color,j}\times \omega \left(t\right)+Q_{Shape,j}\times (1-\omega \left(t\right)),\end{array}$$

(53)

where $\omega \left(t\right)$ is the arbitration weight on trial t specifying the relative contribution of color attribute to overall value, and the index j denotes the choice options to the left and right of the screen (for example, if the leftward option is a green triangle, $V_{left}\left(t\right)=Q_{green}\left(t\right)\omega \left(t\right)+Q_{triangle}\left(t\right)(1-\omega \left(t\right))$). The difference in overall values V is then used to compute choice probability, similar to Equation (37). It is important to note that linear combination of feature values to an overall value is used primarily for simplicity in describing the model and does not necessarily imply that such an integrated value is explicitly constructed. In practice, it is more plausible that the value of each attribute is compared directly—albeit with different weights—when making choices [75].

To dynamically control the arbitration weight between competing strategies, we considered the following update rule:

$$\omega (t+1)=\left\{\begin{array}{cc}\hfill & \omega \left(t\right)+{\alpha }_{\omega }\left|\Delta Rel\left(t\right)\right|(1-\omega \left(t\right)),\mathrm{if}\Delta Rel\left(t\right)>0\hfill \\ \hfill & \omega \left(t\right)+{\alpha }_{\omega }\left|\Delta Rel\left(t\right)\right|(0-\omega \left(t\right)),\mathrm{if}\Delta Rel\left(t\right)<0,\hfill \end{array}\right.$$

(54)

where $\Delta Rel$ specifies the difference in the reliability between color and shape attributes in predicting rewards. Intuitively, the model assigns higher weight to the attribute which is estimated to have higher reliability for the given trial. We defined reliability based on the absolute value of the reward prediction error (RPE) for each attribute. Specifically, the reliability of an attribute is defined as inversely proportional to the magnitude of its RPE—a lower RPE (i.e., less surprise) indicates higher reliability. Based on this definition, the reliability difference $\Delta Rel$ is given by:

$$\begin{array}{cc}\hfill \Delta Rel\left(t\right)& =|RP{E}_{Shape}|-|RP{E}_{Color}|,\hfill \\ \hfill & =|R\left(t\right)-Q_{c,Shape}|-|R\left(t\right)-Q_{c,Color}|,\hfill \end{array}$$

(55)

where $RP{E}_{Color}$ and $RP{E}_{Shape}$ refer to the reward prediction error between actual and predicted reward outcomes based on color or shape attribute, respectively. For example, if the color attribute yields a smaller RPE magnitude than the shape attribute, then $\Delta Rel>0$, and the model increases $\omega$ to bias decision making toward the color attribute on the next trial.

We considered three distinct decision-making strategies, each implemented as a special case of the arbitration model described above. In the first case, the model fixed $\omega =1$ for all trials, representing an agent who relies exclusively on the color attribute. In the second case, $\omega =0.5$ was fixed, modeling an agent who assigns equal weight to both color and shape attributes. In the third case, the full arbitration mechanism was implemented, with the agent dynamically adjusting $\omega$ based on the relative reliability of the two attributes in predicting reward. For this model, $\omega$ was initialized at 0.5 on the first trial. For the full model, we used the following parameter values: $\alpha =0.4,\beta =10,\gamma =0.2,{\alpha }_{\omega }=0.4$.

To quantify the relative dominance of color-based versus shape-based learning strategies, we computed the conditional entropy of reward-dependent strategy (ERDS) separately for each attribute. Specifically, we defined stay/switch strategies separately for color and shape—denoted $St{r}_{Color}$ and $St{r}_{Shape}$, respectively—based on whether the agent repeats or switches their choice option with respect to each attribute after reward feedback. The corresponding ERDS measure for each attribute was then defined as follows:

$$\begin{array}{cc}\hfill {\mathrm{ERDS}}_{Color}& =H\left(St{r}_{Color}\right|R),\hfill \\ \hfill {\mathrm{ERDS}}_{Shape}& =H\left(St{r}_{Shape}\right|R),\hfill \end{array}$$

(56)

To quantify the relative dominance of strategies in response to reward feedback, we computed the difference $\Delta ERDS=ERD{S}_{Shape}-ERD{S}_{Color}$. A higher value of $\Delta ERDS$ indicates greater reliance on a color-based choice strategy.

3. Results

In the following sections, we illustrate the utility of information-theoretic metrics for identifying distinct learning and decision-making mechanisms using simulated data described above. We begin by presenting example behavior of an RL agent with a single learning rate, shown for three values: $\alpha =\{0.2,0.4,0.6\}$ (Figure 2). The time course of the performance—defined as the probability of choosing the more rewarding option $P\left(Better\right)$—shows that, for all three learning rates, performance peaks within approximately 10 to 20 trials but more quickly for larger learning rates. However, the overall performance, calculated across the entire block, does not significantly differ among the three learning rates (rank sum test, $p>.05$; inset in Figure 2b).

Compared to the performance, the information-theoretic metrics exhibited more distinct trends across the different values of $\alpha$ (Figure 2c–f). For example, the entropy of strategy, $H\left(Str\right)$, was highest for the RL agent simulated with $\alpha =0.2$, followed by $\alpha =0.6$ and $\alpha =0.4$, which did not differ significantly from each other (inset in Figure 1c). Moreover, the trajectory of $H\left(Str\right)$, which measures the entropy of stay/switch strategy, decreased over trials within a block and peaked following the reversal—when the better and worse options were swapped. The normalized mutual information between reward and strategy (n-MIRS) revealed that information gained from reward feedback was highest for $\alpha =0.6$, followed by $0.4$, and lowest for $0.2$ (Figure 1d), consistent with the interpreting the learning rate as a measure of how strongly choices are adjusted based on reward feedback. The normalized mutual information between choice option and strategy (n-MIOS) indicated that the agents gradually learned to identify and persist with the better option over trials, as reflected by the increasing $P\left(Better\right)$ and decreasing $H\left(Str\right)$ (Figure 1e). Finally, the normalized mutual information between option-reward combinations and strategy (n-MIROS) revealed that, despite differences in their temporal trajectories, the overall information shared between these variables plateaued at similar levels across all three learning rates (Figure 2f). However, the distinct temporal dynamics led to significant differences among agents as reflected in the average metrics values computed over the entire block (inset in Figure 2f).

Overall, this example illustrates that information-theoretic metrics can detect subtle variations in underlying mechanisms—such as variations in the learning rates—that simpler measures like performance may fail to capture (compare the inset in Figure 2b with those in Figure 2c–f). In the following four sets of simulations, we further show how these information-theoretic metrics can differentiate between the choice behavior of various RL models used as ground truths.

3.1. Revealing Positivity Bias

In the first set of simulations, we examined whether positivity bias—formally defined as a larger learning rate for positive outcomes compared to negative outcome (${\alpha }_{+}>{\alpha }_{-}$)—can be captured using information-theoretic metrics. To that end, we used a standard RL (Figure 3a) to simulate choice behavior in a probabilistic reversal learning task under different reward scheudles: 80/20% and 40/10% reward schedules. We explored different combination of ${\alpha }_{+}$ and ${\alpha }_{-}$ values to examine how asymmetric learning rates influence behavior across these environments.

We found that that higher positivity bias (larger $\Delta \alpha ={\alpha }_{+}-{\alpha }_{-}$) resulted in higher performance across the two reward environments, especially for the 40/10% schedule (Pearson’s correlation between $\Delta \alpha$ and $P\left(Better\right)$; 40/10%: r = .904, p = 4.32×10^-108; 80/20%: r = .702, p = 2.88×10^-44) (Figure 3b). This is consistent with the previous literature [27,44,45,76], demonstrating that positivity bias can improve performance in terms of reward harvest.

Because learning rates control how value estimates are updated in response to reward feedback, we hypothesized that information-theoretic metrics related to reward—specifically ERDS and n-MIRS—would be sufficient to detect positivity bias in the choice behavior. Following this intuition, we first compared the choice behavior of the standard RL agents simulated with selected values of ${\alpha }_{+}$ and ${\alpha }_{-}$ that exemplify "optimistic" (${\alpha }_{+}>{\alpha }_{-}$), "neutral" (${\alpha }_{+}={\alpha }_{-}$), and "pessimistic" (${\alpha }_{+}<{\alpha }_{-}$) tendencies Figure 3c).

We found that the entropy of stay/switch strategy conditioned on reward (ERDS) was highest for the most pessimistic agent (cyan lines in Figure 3d), whereas the normalized mutual information between reward and strategy (n-MIRS) was lowest for the same agent (cyan lines in Figure 3e). However, metrics computed from the neutral agent (black lines) and the two optimistic agents (red and magenta lines) were less distinguished and plateaued at similar values. When simulating the full grid of ${\alpha }_{+}$ and ${\alpha }_{-}$ values, we observed that ERDS was significantly correlated with $\Delta \alpha$, with higher positivity bias associated with lower ERDS (r = -.778, p = 6.25×10^-60; Figure 3f). In contrast, n-MIRS was only weakly correlated with $\Delta \alpha$ (r = -.137, p = .020), and the direction of this relationship was opposite to that observed in the example shown in Figure 3e, suggesting an overall nonlinear relationship. Therefore, while a stronger positivity bias is generally associated with lower ERDS, the absolute values of these metrics alone do not reliably indicate whether a given behavior reflects positivity bias.

To identify candidate metrics that may be more informative of positivity bias, we next examined the decomposition of the above metrics based on reward outcome (reward vs. no reward). Intuitively, there is greater uncertainty about whether the agents will stay/switch following unrewarded trials compared to rewarded ones. Over time, agents learn to maximize reward by adopting a stay-dominated strategy following rewarded trials, as they continue selecting the better option. In contrast, after unrewarded trials, agents are more likely to switch. However, if the chosen option still has a relatively high reward probability (i.e., it is the better option), the likelihood of staying after a no-reward trial can also increase over time.

Consistent with this notion, we found that the trajectory of the decomposition of mutual information metric following no reward (n-MIRS- in Figure 3g) became negative over time, suggesting that negative reward outcomes became less informative about the agents’ subsequent strategy (i.e., $H\left(Str\right)<H\left(Str\right|R=loss)$). However, during the early portion of the block, unrewarded trials were initially informative about choice strategy,as indicated by the positive n-MIRS- values. Optimistic and pessimistic agents exhibited distinct temporal dynamics in how quickly this metric shifted toward negative values (shaded gray area in Figure 3g). When computed over the first 10 trials, the sign of the n-MIRS– metric was approximately predictive of whether an agent exhibited positivity bias or not (Figure 3g inset). Specifically, more optimistic agents exhibited more negative n-MIRS- values, indicating that their tendency to update more strongly after rewarded than unrewarded trials. This makes the no-reward trials less informative of their subsequent strategy. This pattern is consistent with the results shown in Figure 3b, where optimistic agents achieved better performance. To confirm this effect across the full parameter space, we computed n-MIRS- as a function of $\Delta \alpha$ and found a strong linear relationship in the 80/20% environment (r = -.926, p = 1.27×10^-123), as well as in the other two tested reward environments (60/40% and 40/10%; Figure 3h). These results suggest that the decomposed mutual information measures—specifically n-MIRS- computed from the early period of each block—can be used to predict the presence of positivity bias. To test this idea, we conducted a classification analysis in which we randomly sampled groups of optimistic and pessimistic agents and used the information-theoretic metrics to predict whether a given metric profile originated from an optimistic agent using a linear decoder (Figure 3i; see Section 2.3.1 for details).

We found that the n-MIRS- metric alone, when computed from the first 10 trials, yielded high cross-validated decoding accuracy across different environments (Figure 3j), significantly exceeding chance level (signed-rank test against 0.50; $p<.001$ for all environments). Including all available metrics in the decoder led to a small but significant improvement in the decoding accuracy (signed-rank test; $p=.0011$ for 50/10%, $p<.001$ for all other environments). In comparison, a decoder based on logistic regression weights achieved comparable accuracies (white bars in Figure 3j), indicating that the information-theoretic metrics contain comparable information to traditional regression-based measures—while offering more interpretable connections to underlying learning and decision-making processes. Overall, these results illustrate that information-theoretic metrics contain sufficient information to determine the presence of positivity bias when the ground truths are provided.

3.2. Revealing Reward-Dependent Metaplasticity

One potential neural mechanism underlying differential learning rates is reward-dependent metaplasticity, which generates dynamic learning rates by naturally adapting to the recent history of rewards in the environment [46,47,73,77]. To test whether such dynamic changes in the learning rates can be detected using information-theoretic metrics, we implemented a variant of the metaplastic model proposed by Farashahi et al. [46]. In our simulations, we assumed that each of the two choice options is assigned a set of metaplastic synapses which undergo transitions in response to reward feedback (Figure 4b; Section 2.3.2). By computing the "effective" learning rate—defined as the overall rate of value update on each trial—we found that the metaplastic model exhibited diverging learning rates following rewarded and unrewarded trials, with ${\alpha }_{+}>{\alpha }_{-}$ (solid lines in Figure 4c), consistent with previous findings [46]. The difference between the effective ${\alpha }_{+}$ and ${\alpha }_{-}$ increased over time and plateaued just before the reversal (red solid curve in Figure 4c), driven primarily by an increase in ${\alpha }_{+}$ rather than a decrease in ${\alpha }_{-}$. This happens because, as more reward is obtained, the "weak" synapses encoding the better option mostly occupy the most unstable meta-states $w_1$ (Figure 4b), which has the highest transition probability ${\alpha }_1$ toward the "strong" state $s_1$, resulting in a high effective ${\alpha }_{+}$.

Using the choice behavior of this metaplastic agent as ground truth, we fitted a ’plastic’ model (standard RL) with differential learning rates to the simulated choice data from each block to test whether the fitted parameters reflected the pattern ${\alpha }_{+}>{\alpha }_{-}$ (noting that these learning rates are constant by definition in this model). The results confirmed this pattern (red and blue dashed lines in Figure 4c), suggesting that empirically observed trends of ${\alpha }_{+}>{\alpha }_{-}$ in previous studies may, in fact, reflect underlying reward-dependent metaplasticity.

To test whether the distinct learning mechanisms of the plastic and metaplastic models can be identified from choice behavior, we simulated the choice behavior of a metaplastic model and two plastic models, and then computed information-theoretic metrics from the simulated behavior. Specifically, we compared metrics across three models: (1) a plastic model with a single $\alpha =0.3$ (plastic 1-$\alpha$), (2) a metaplastic model initialized with the same effective learning rate of 0.3 on the first trial, and (3) a plastic model with differential learning rates (plastic 2-$\alpha$) estimated from the metaplastic agent.

Comparison of the entropy of choice strategy indicated that the metaplastic agent was more consistent in its strategy compared to both plastic agents (rank sum test on $H\left(Str\right)$: vs. 1-$\alpha$: p = 4.43×10^-28; vs. 2-$\alpha$: p = 2.43×10^-24; Figure 4d). Moreover, the n-MIOS metric revealed that the metaplastic agent had the advantage of increasing the mutual information between the choice of the better option and subsequent strategy (Figure 4e), reflected in its superior performance relative to both plastic models (rank sum test on n-MIOS: vs. 1-$\alpha$: p = 5.58×10^-4; vs. 2-$\alpha$: p = .00280). The two plastic agents (1-$\alpha$ and 2-$\alpha$) were less distinguishable from each other in terms of $H\left(Str\right)$ (p = .0120), n-MIOS (p = .683), and overall performance (p = .848).

These results suggest that, although the parameters of the plastic 2-$\alpha$ model were directly estimated from the metaplastic agent, differences in the underlying learning mechanisms still give rise to distinct patterns in the information-theoretic metrics. This highlights the potential utility of such metrics in model recovery and validation—for example, by assessing whether a given candidate RL model can reproduce the information-theoretic metrics observed in the empirical data [29].

Given the observed differences between plastic and metaplastic models, we next quantified the extent to which the information-theoretic metrics discriminate between different models. We hypothesized that these metrics could be used to decode changes in the learning rates corresponding to positivity bias, using an approach similar to that employed in the previous section. To obtain highest accuracy, we utilized all available information-theoretic metrics and their decompositions to train a linear decoder to discriminate between optimistic and pessimistic agents (see Section 2.3.2 in Materials and Methods for more details). We then applied this decoder to the metrics generated by the three models described above and computed the posterior probability that the decoder would classify the behavior as exhibiting positivity bias.

We found that the posterior probabilities derived from the metaplastic model (magenta line in Figure 4f) exhibited temporal dynamics closely resembling those of effective learning rates (Figure 4c). Specifically, as trials progressed, the decoder increasingly indicated that the behavior originated from an optimistic agents with ${\alpha }_{+}>{\alpha }_{-}$. In contrast, the posterior probabilities for the two plastic agents remained relatively constant across the block. The plastic 1-$\alpha$ agent (black line in Figure 4f) maintained performance at chance level, as it was neither optimistic nor pessimistic by design. On the other hand, choice behavior of the plastic 2-$\alpha$ agent (gray line in Figure 4f) was classified as being optimistic, consistent with its estimated learning rates reflecting ${\alpha }_{+}>{\alpha }_{-}$ (dashed lines in Figure 4c).

3.3. Revealing Behavioral Adjustments due to Reward Harvest Rate

Next, we investigated how information-theoretic metrics can be used to detect the influence cumulative overall reward feedback on all options (i.e., reward harvest rate) on an agent’s choice strategy. To that end, we used two types of RL model to generate choice behavior: (1) the standard RL with a single learning rate $\alpha$, and (2) the augmented RL model incorporating a reward harvest rate component (Figure 5a). Briefly, the augmented model tracks an average reward rate H, which was not tied to any specific choice option, and uses it to modulate the agent’s win-stay and lose-switch tendencies (see Section 2.3.3 in Materials and Methods for more details ). To quantify the effect of reward harvest rate on choice behavior, we defined a discrete variable referred to as the reward harvest state (RHS)—a binary indicator of whether the reward rate on a given trial was above or below the block median [35].

As shown for an example block, high or low reward harvest state (RHS) did not consistently coincide with reward (or no reward) on the previous trial (Figure 5b), given the weak correlation between the reward rate and binary reward feedback (r = .107, p = .343). Exploiting this property, we used RHS as an additional conditioning variable for the strategy $Str$, and measured the joint influence of previous reward and RHS on the agent’s subsequent strategy using the mutual information metric n-MIRHS (see Section 2.3.3 for more details). The trajectory of this metric revealed that overall information was significantly higher for the agent with the additional reward harvest rate component (RL with RHR agent) compared to the standard RL agent (rank sum test, p = 2.56×10^-34), reflecting the added influence of RHS on choice strategy Figure 5c.

To gain insight on how the high and low RHS states influence behavior, we next examined the decomposition of n-MIRHS based on RHS, corresponding to the influence of high (${\mathrm{n-MIRHS}}_{high}$, Figure 5d) and low RHS (${\mathrm{n-MIRHS}}_{low}$, Figure 5e). We found that the shared information between the $high$ state and choice strategy was overall comparable across models but significantly higher for the standard RL agent than for the RL with RHR agent (difference in mean ${\mathrm{n-MIRHS}}_{high}$ = 0.0445; rank sum test, p = 4.67×10^-34). In contrast, ${\mathrm{n-MIRHS}}_{low}$ revealed a more pronounced distinction: it was significantly positive for the RL with RHR agent (signed-rank test, p = 3.90×10^-18), but not significantly different from zero for the standard RL model (signed-rank test, p = .064; mean = 0.0019; inset in Figure 5e).

These results indicate that during the high state—when the reward rate was higher than usual due to successful learning—both models showed positive mutual information between the high RHS and subsequent strategy. In contrast, during the low state—when the agents were receiving rewards at a lower-than-usual rate—RHS was not informative about whether the standard RL agent would stay or switch on the next trial. This was not the case for the RL with RHR agent, for which ${\mathrm{n-MIRHS}}_{low}>0$. Further decompositions based on reward feedback revealed that this effect was driven by a larger magnitude of ${\mathrm{n-MIRHS}}_{low+}$ (following rewarded trials) compared to ${\mathrm{n-MIRHS}}_{low-}$ (following unrewarded trials; compare red and blue bars in the Figure 5e inset for the RL with RHR agent). This pattern reflects an overall positive shift in information content resulting from the architecture of the RL with RHR model, which modulates stay/switch tendency as a function of the reward harvest rate H.

Lastly, to gain further insight into how the n-MIRHS metric rely on the model parameters, we simulated choice behavior of the model with additional modulation by reward harvest rate using the full range of $\alpha$ and ${\beta }_H$ values. Using standardized regression, we then examined the contribution of the ${\beta }_H$ parameter, which determines the influence of reward harvest rate on choice behavior (${\beta }_H=0$ corresponds to the standard RL. We found that, consistent with the example in (Figure 5c), larger values of ${\beta }_H$ predicted overall higher values of n-MIRHS (standardized regression of n-MIRHS on ${\beta }_H$, $\alpha$, and their interaction: $b_1$ = 0.366, p = 3.13×10^-216). Similar regression analyses on each of the decompositions yielded results consistent with Figure 5d,e. More specifically, ${\beta }_H$ did not significantly affect the n-MIRHS decomposition for the high reward state (standardized coefficient predicting ${\mathrm{n-MIRHS}}_{high}$: $b_1$ = 0.0150, p = .329) nor its further reward-based decompositions (${\mathrm{n-MIRHS}}_{high+}$: $b_1$ = 0.024, p = .109; ${\mathrm{n-MIRHS}}_{high-}$: $b_1$ = -0.00665, p = .674). In contrast, higher values of ${\beta }_H$ predicted significantly larger ${\mathrm{n-MIRHS}}_{low}$ ($b_1$ = 0.464, p = 4.37×10^-308). The decompositions based on reward feedback revealed that this effect was strongest for ${\mathrm{n-MIRHS}}_{low-}$, which was significantly higher for larger values of ${\beta }_H$ (standardized coefficient: $b_1$ = 0.523, p = 1.53×10^-253; dashed lines in Figure 5f, right). In comparison, ${\mathrm{n-MIRHS}}_{low+}$ was also significantly predicted by ${\beta }_H$, but to a lesser degree ($b_1$ = 0.121, p = 1.90×10^-19; solid lines in Figure 5f, right). These results suggest that the influence of reward harvest rate on promoting heuristic win-stay/lose-switch is primarily mediated by increased information value of no reward in the low reward state. That is, the larger ${\beta }_H$, corresponding to the effect of reward rate, was associated with less negative ${\mathrm{n-MIRHS}}_{low-}$ toward zero, thus increasing the informativeness of receiving no reward during this state. This can be used to detect modulations of choice behavior by reward harvest rate as these effects can change across blocks of trials.

3.4. Revealing the Presence of Alternative Learning Strategies in Multidimensional Reward Environments

Finally, we examined whether the information-theoretic metrics can be utilized to identify the presence of alternative learning strategies in multidimensional reward environments. In naturalistic settings, reward outcomes often depend on distinct features or attributes of the available choice options. To mimic such scenarios, we simulated a task in which only one of the two choice attributes—either the shape or the color of the stimuli—was associated with reward, thereby introducing uncertainty about which attribute was predictive of reward outcomes (Figure 6a; see Section 2.3.4 for more details). We considered three RL agents, each exhibiting distinct types of choice strategy: (1) an agent that only learns and chooses based on the color attribute (Color-only), (2) an agent that equally weighs the color and shape attributes without arbitration (No Arbitration), and (3) an agent which dynamically arbitrates between two attributes based on the reliability of each attribute in predicting reward (Dynamic Arbitration; Figure 6b).

Examining the arbitration weight $\omega$ across different block types revealed that the Dynamic Arbitration agent successfully adapted its strategy to the environment: it consistently increased $\omega$ during Color-to-Color blocks, indicating a growing reliance on color (magenta curve in Figure 6e, top), and shifted $\omega$ toward a shape-based strategy following reversals in Color-to-Shape blocks (magenta curve in Figure 6f, top). When reversals occurred within the color attribute (Color-to-Color blocks), the performance trajectories ($P\left(Better\right)$) were qualitatively similar across all agents, as expected. In each model, performance dropped immediately after reversals followed by gradual recovery (Figure 6e, bottom). As expected, the Color-only agent reached its peak performance more quickly (Figure 6e, bottom), as it did not consider shape values, which were irrelevant in this condition. During Color-to-Shape blocks, when the color information was no longer predictive of rewards after reversals, the Dynamic Arbitration agent performed the best (Figure 6f, bottom), as this agent appropriately identified the correct reward-predictive attribute by dynamically arbitrating between the two feature dimensions (Figure 6f, bottom).

Although the learning strategy of the Color-only agent can be readily distinguished by its $P\left(Better\right)$ trajectory during the Color-to-Shape blocks, distinguishing among all three agents is more challenging in Color-to-Color blocks based on performance alone. Therefore, we examined whether information-theoretic metrics could be used to distinguish among the three types of learning strategies represented by the RL agents. Although the underlying arbitration weight $\omega$ can, in principle, be estimated through model fitting, behavioral metrics offer a more direct window into learning and decision-making strategies. Moreover, these metrics can be computed over a subset of trials and require far simpler computations compared to model fitting, which depends on continuity in the choice data.

To quantify reward-dependent strategies, we computed the conditional entropy of reward-dependent strategies (ERDS) separately for each attribute—${\mathrm{ERDS}}_{Shape}$ and ${\mathrm{ERDS}}_{Color}$ (Figure 7a,b). We found that during Color-to-Color blocks, the entropy of the shape-based strategy (${\mathrm{ERDS}}_{Shape}$) was close to its maximum value of 1 for all the three agents, reflecting the fact that shape carried no information about reward in this block type (dashed lines in Figure 7a). In contrast, the entropy of the color-based strategy (${\mathrm{ERDS}}_{Color}$) exhibited modulation across trials, with distinct dynamics for each agent (solid lines in Figure 7a). Consistent with the performance trajectories, the Color-only agent showed the lowest entropy, followed by the Dynamic Arbitration agent (rank sum test on ${\mathrm{ERDS}}_{Color}$, p = 7.98×10^-34). To assess the relative dominance of the two learning strategies, we computed the difference between the two entropy values, ${\mathrm{ERDS}}_{Shape}-{\mathrm{ERDS}}_{Color}$. The resulting positive values indicated that all three agents correctly prioritized the color-based learning strategy during this block type, as reflected by lower entropy for the relevant dimension (i.e., color) Figure 7c). These two metrics thus provide insight beyond the trajectories of arbitration weight $\omega$ (Figure 6e top), which does not directly capture changes in the learned values that could drive choice behavior.

During the Color-to-Shape blocks, the entropy values (${\mathrm{ERDS}}_{Shape}$) indicated near-random use of the shape-based strategy for all three agents prior to the reversal (dashed lines in Figure 7b). After the reversal—when shape attribute became informative of reward—the Dynamic Arbitration showed the most pronounced adjustment, characterized by a decreased reliance on the color-based strategy (reflected by as drop in ${\mathrm{ERDS}}_{Color}$; solid magenta line in Figure 7b), and an increased reliance on the shape-based strategy (reflected by a rise in ${\mathrm{ERDS}}_{Shape}$; dashed magenta line). The difference in the two entropy values further revealed that only the agents capable of learning both attributes (No Arbitration and Dynamic Arbitration) were able to shift their strategies after the reversal, as indicated by ${\mathrm{ERDS}}_{Color}>{\mathrm{ERDS}}_{Shape}$ (Figure 7d). In contrast, the Color-only agent exhibited ${\mathrm{ERDS}}_{Color}<{\mathrm{ERDS}}_{Shape}$ throughout the entire block.

To quantify potential interactions between the color-based and shape-based learning strategies, we measured the correlation between ${\mathrm{ERDS}}_{Shape}$ and ${\mathrm{ERDS}}_{Color}$, computed separately from two distinct periods within each block. "Non-stationary" trials were defined as the first 30 trials at the beginning of each block and those immediately following the reversal (highlighted by red bars in Figure 7a,b). In contrast, "steady" trials corresponded to the final 10 trials before the reversal or at end of each block, during which performance had plateaued (blue bars in Figure 7a–d).

This analysis revealed distinct patterns of interaction between the two learning strategies across the three agents (Figure 7e,f). More specifically, the Color-only agent showed no significant correlation between ${\mathrm{ERDS}}_{Shape}$ and ${\mathrm{ERDS}}_{Color}$ during either periods of the Color-to-Shape blocks (non-stationary: r = .137, p = .174; steady: r = .072, p = .477; Figure 7e). This result is expected, as the Color-only agent does not employ any shape-based strategy, which renders response to reward based on shape effectively random. In contrast, the No Arbitration agent exhibited competitive interaction between the two strategies, as indicated by a significant negative correlation between ${\mathrm{ERDS}}_{Shape}$ and ${\mathrm{ERDS}}_{Color}$ (non-stationary: r = -.435, p = 6.08×10^-6; steady: r = -.386, p = 7.43×10^-5; Figure 7e). This pattern reflects the agent’s decision-making strategy , in which color and shape information are always weighted equally, regardless of which attribute is currently predictive of reward (fixed $\omega =0.5$). Interestingly, the Dynamic Arbitration agent exhibited different trend of interaction across the two periods of the block. During the non-stationary phase, this model exhibited a significant negative correlation between ${\mathrm{ERDS}}_{Shape}$ and ${\mathrm{ERDS}}_{Color}$ (r = -.374, p = 1.28×10^-4). However, this correlation disappeared during the steady phase of the block (r = -.0409, p = .686), when the color-based strategy became dominant (i.e., $\omega >0.5$). This shift in correlation pattern can be used to detect dynamic arbitration between alternative learning strategies.

We found overall consistent results during the Color-to-Shape blocks (Figure 7f). The Color-only agent showed no significant interaction between strategies in either period (non-stationary: r = -.0346, p = .733; steady: r = -.055, p = .587). In contrast, the No Arbitration agent exhibited significant negative correlations during both phases of the block (r = -.522, p = 2.59×10^-8; steady: r = -.269, p = .00685). Notably, only the Dynamic Arbitration agent—capable of adapting its behavior based on relative reliability of the two attributes in predicting reward—exhibited a shift in the interaction pattern over time. Specifically, it showed a significant negative correlation during the non-stationary period (r = -.536, p = 9.19×10^-9), which weakened and became non-significant during the steady phase (r = -.151, p = .135), consistent with the pattern observed in the Color-to-Color blocks. Together, these results demonstrate that a negative correlation between ${\mathrm{ERDS}}_{Shape}$ and ${\mathrm{ERDS}}_{Color}$ during the non-stationary phase and the absence of such a correlation during the steady phase is indicative of dynamic arbitration between alternative learning strategies.

4. Discussion

In this study, we demonstrate how behavioral metrics inspired by information theory can be used to identify certain learning and decision-making mechanisms. In particular, we applied metrics based on conditional entropy, normalized mutual information, and their decompositions based on specific outcomes to choice behavior during two variants of the probabilistic reversal learning task—a widely adopted paradigm for studying cognitive flexibility across species. Using these metrics, we investigated whether specific neural or computational mechanisms—specified by the reinforcement learning (RL) models serving as ground truth—could be inferred from choice behavior. To that end, we examined positivity bias, gradual changes in the learning rates due to reward-dependent metaplasticity, the influence of reward harvest rate, and the adoption and arbitration between alternative learning strategies in multidimensional environments.

One of the key strengths of the proposed information-theoretic metrics lies in their versatility and flexibility. In contrast, fitting RL models to choice behavior in order to identify underlying mechanisms requires a continuous stream of choice and reward data, with the precision of the estimated parameters heavily dependent on the amount of available data. As our results demonstrate, information-theoretic metrics circumvent these limitations, as they can be computed from as few as several dozen trials, even when drawn from non-contiguous segments of the data. Directly comparing the information content of our metrics with regression weights for predicting choice (Figure 3j), we found similar decoding performance from both types of measures. However, information-theoretic metrics offer the distinct advantage of being simple and model-free, whereas regression-based features are inherently model-dependent—–requiring decisions about which predictors to include, how many lags to consider, how to evaluate model fit and statistical significance, and how to interpret the resulting regression weights.

Moreover, time courses of information-theoretic metrics can be computed for each trial point by concatenating trial-wise data across different blocks. This approach serves as a useful visualization tool for examining behavioral changes over time. Finally, the analysis in Figure 7 demonstrates how metrics computed from different periods within a block can provide insight into temporal changes in learning and choice strategies. Taken together, the efficiency and flexibility of these metrics make them valuable model-free tools for identifying and testing hypotheses about the mechanisms underlying learning and decision making.

Another important advantage of information-theoretic metrics is their adaptability to target specific variables of interest within a given study. As illustrated in simulations of reward harvest rate effects on behavior (Figure 5), the key variable—reward harvest state—can be incorporated into the mutual information metric n-MIRHS to assess how reward harvest rate influences choice behavior. This measure revealed the role of reward harvest rate in shaping choice strategy under varying reward contingencies by quantifying the information flow from prior reward outcomes and reward states into the agent’s choice strategy. Therefore, these metrics can be flexibly formulated to match the demands of a given research context and provide quantifiable insights into the role of specific variables.

In turn, the insights provided by these metrics can guide the development of improved models that better capture key aspects of behavioral data [29]. As numerous studies have pointed out [78,79,80], the best-fitting model among a set of candidates may still fail to reproduce important features of choice behavior. Therefore, validating candidate models through posterior predictive checks remains a critical step in modeling [26,81]. Information-theoretic metrics offer ideal tools for this process, as they provide summary statistics specific to the variables of interest. For example, different RL models could be validated by comparing information-theoretic metrics computed from simulated data with those derived from empirical data [29,38]. By quantifying the divergence between empirical observations and model-generated data, this comparison both reveals a model’s limitations and pinpoints the mechanisms that must be added for improvement [29]. Such an approach is especially valuable when standard model-comparison metrics are inconclusive—for example, with small or noisy datasets that inherently favor simpler, low-parameter models. In those cases, one can use information-theoretic measures to constrain the candidate models by selecting the one whose simulated statistics most closely match the empirical data. Lastly, although this study focuses on RL models, the same framework can be applied to other model classes—such as those based on Bayesian inference [32,82].

In addition to serving as useful tools for model discovery and validation, information-theoretic metrics may also contribute to data-driven approaches in studies involving choice behavior. Our decoding analyses in Section 3.1 and Section 3.2 serves as a proof-of-concept, showing that the information-theoretic measures contain sufficient information to detect positivity bias when the ground truths are known. This example highlights the broader potential of the metrics in detecting latent cognitive states or mechanisms other than positivity bias (which may lack ground truths in real data). For example, one could train a decoder to predict experimentally defined task states from either behavioral measures (reaction time, reward rate, pupil dilation) or neural recordings (spike trains, EEG, fMRI) by using information-theoretic features extracted from choice data. In this approach, the experimentally defined states serve as the ground-truth labels during training, and the decoder’s ability to recover those labels is then evaluated on held-out behavioral (or neural) data. Future research could apply this model-agnostic yet interpretable framework to infer latent cognitive or neural states.

There are some challenges and considerable potential for extending the information-theoretic framework presented here. For example, while our entropy measures quantify the overall consistency in the stay/switch strategy in response to reward and other relevant variables, they do not directly capture directionality—that is, they do not indicate what causes what. As a result, these metrics should be considered alongside other measures to provide a more complete picture. This limitation makes interpreting mutual information versus conditional entropy particularly challenging. Moreover, while information-theoretic metrics capture sensitivity to reward feedback, they cannot be uniquely mapped onto the learning rates. Finally, although we have focused on conditional entropy and normalized mutual information, other behavioral metrics based on other concepts from information theory can be developed including transfer entropy [83,84], mutual information between discrete and continuous variables [85,86,87], and partial information decomposition [88,89], among others. Incorporating these extensions would allow the behavioral metrics to be generalized to more complex tasks, including those involving more than two alternative options, higher-dimensional feature spaces, and task contexts.