Simulated Dopamine Modulation of a Neurorobotic Model of the Basal Ganglia

1. Introduction

The vertebrate basal ganglia are thought to play an important role in action selection—the resolution of conflicts between alternative motor programs [1,2,3,4,5,6,7]. The effective operation of basal ganglia circuitry, and is regulation of motor behavior, is also known to rely on appropriate levels of dopamine (DA) transmission [3,8,9]. For instance, DA antagonists, or dopamine-depleting brain lesions, have been found to impair a range of instrumental and spontaneous behaviors [10,11,12,13,14,15,16], effect the maintenance of behavior over time [10,17], impair the initiation of movement [18,19], reduce behavior switching [13,20,21,22,23], and can induce bradykinesia (slowed movement) or akinesia (lack of movement) [24,25]. Conversely, DA agonists have been shown to cause increases in behavior switching [21,22,26], or can lead to patterns of repetitive behavior (stereotypy) [27,28,29,30]. Human basal ganglia-related disorders such as Parkinson’s Disease (PD). Schizophrenia, Attention Deficit Hyperactivity Disorder (ADHD), and Tourette’s Syndrome (TS) are also known to involve abnormalities in the dopamine regulation of basal ganglia circuitry [31,32,33,34,35]. Nevertheless, in both humans and animals, there is still much to understand about how variation in tonic dopamine levels can have these different and variable effects on behavior.

In this article we show that when the tonic level of simulated dopamine in a robotic model of the basal ganglia [36] is significantly reduced or increased, relative to a baseline, a variety of behavioral outcomes are observed that provide interesting comparisons with the results of animal studies, and with some of the observed behavioral consequences of dopamine dysregulation in disorders affecting the human basal ganglia. In this way, we hope that this article can contribute to the emerging field of computational psychiatry [37] and the investigation of models of psychopathology via robotics [38].

The structure of the article is organized as follows. Section 2 describes some principles that allow us to measure the effectiveness of action selection, provides a summary of the computational models developed by Gurney, Prescott and Redgrave [39,40] and by Humphries and Gurney [41], that view the basal ganglia as a selection mechanism, and sets out some metrics that allow us to evaluate its performance against our principles. Section 3 describes the embedding of this model in the control architecture of a mobile robot as previously reported by Prescott, Montes Gonzalez et al. [36]. Section 4 describes some experiments with a non-embodied version of the model that provide fresh insight into the effects of tonic dopamine modulation on selection by this model. Section 5 then applies ethological methods to analyse the results of experiments with the robot embedding of the model in which we vary the simulated level of tonic dopamine. Finally, section 6 draws some comparisons with animal and human data, and discusses some of the implications of our study for the use of robotics modelling in neuroscience.

2. The Basal Ganglia Viewed as an Action Selection Mechanism

Requirements for Effective Selection

Given a set of competing and incompatible programs, the requirements for an effective action selection mechanism [3,42] can be summarised as: (i) in selecting a winner, all else being equal, prefer the most strongly supported, or most salient, competitor as indicated by relevant external and internal cues; (ii) allow only one program to be expressed at a given time, this winner should be cleanly selected (i.e. allowed unrestricted access to the motor apparatus) and the losers should be prevented from interfering with its performance, termed lack of distortion; (iii) provide clean switching—a competitor with a slight edge over its rivals should see the competition resolved rapidly and decisively in its favor; and (iv) support action maintenance—a winning competitor may be required to remain active at lower salience levels than are initially required for it to overcome the competition. This latter characteristic, also termed hysteresis [43] or behavioral persistence [44], can prevent unnecessary switching, or ‘dithering’, between closely matched competitors.

Note that this view of action selection treats input salience as the ‘common currency’ according to which diverse behavioral options can be evaluated for purposes of selection—the selector doesn’t need to know what the option is, only how salient it is, with salience being determined by genetics and learning. We note that other ways of selecting between actions are possible that do not rely on salience computation, and that may well exist in the brain, these could operate in a complementary fashion to the centralised action selection mechanisms considered here (see [45] for further discussion).

A Model of Basal Ganglia Intrinsic Circuitry

In a series of computational models, Gurney and co-workers [39,40,41,46] showed that the intrinsic connectivity of the basal ganglia, shown Figure 1 (left) can meet many of these requirements for effective selection via a variety of mechanisms centred on: (i) a set of pathways from the striatum, the basal ganglia’s chief input nucleus, that can generate focused inhibition in basal ganglia output structures—the substantia nigra pars reticulata (SNR) and the internal globus pallidus (GPi) [2] (entopeduncular nucleus in rats); (ii) diffuse excitation of these output structures by the subthalamic nucleus (STN) [47], and (iii) regulation of the contrast between this focused striatal inhibition and diffuse STN excitation by the globus pallidus (GPe) [39,48,49]. The overall mechanism is one that selects by removing tonic inhibition of motor pathways provided by basal ganglia outputs, for selected actions only, whilst maintaining or increasing inhibition on non-selected actions, and is in accord with many theoretical models of basal ganglia viewed as an action selection mechanisms [e.g. 2,3,6]. The novelty of the Gurney et al. model included showing that intrinsic circuitry involving the GP acts to regulate this selection effect, for instance, normalising the level of surround inhibition for different numbers of competitors [40].

The balance between the different intrinsic basal ganglia mechanisms is also thought to depend on the level of tonic dopamine expression which differentially impacts on striatal projection neurons with different receptor types [39,50]. Specifically, striatal projection neurons can be separated into two broad classes. One population contains the neuropeptides substance P and dynorphin, preferentially expresses the D1 subtype of dopamine receptors, and projects primarily to the output nuclei (SNr and GPi). Activity in these ‘D1 neurons’ suppresses the tonic firing in basal ganglia output structures thus acting to select (disinhibit) target structures in the thalamus and brainstem [39,51]. A second population of projection neurons contains enkephalin and preferentially expresses D2 subtype dopamine receptors. The inhibitory projection from these ‘D2 neurons’ constitutes the first leg of an indirect pathway to the output nuclei that has two inhibitory links (Striatum–GPe, GPe–STN), followed by an excitatory one (STN–GPi/SNr). The net effect of D2 activity is therefore to activate output nuclei increasing inhibitory control of the thalamus and brainstem [39,52,53]. Gurney et al. [40] demonstrated that simulation of increasing tonic dopamine in the model basal ganglia has the effect of increasing D1 neuron activity, reducing D2 activity, and consequently reducing activity in GPi/SNr. They concluded that raising tonic dopamine levels makes selection more ‘promiscuous’ increasing the likelihood that target motor pathways will be disinhibited, and potentially leading to ‘soft’ selection—the full or partial disinhibition of multiple channels.

A Model of the Extended Basal Ganglia

Humphries and Gurney [41] extended this intrinsic model, as shown in Figure 1 right, to include extrinsic feedback pathways via the ventral thalamus (VL) and the thalamic reticular nucleus (TRN). This new model provided improved selection, compared to the model of intrinsic circuitry alone, particularly with regard to generating clean selection with absense of distortion (partial expression of losing channels), and the ability to maintain selected actions through positive feedback provided by the basal-ganglia-thalamo-cortical loop.

Humphries and Gurney [41] provide a motivation for, and full description of, the extended basal ganglia model and a detailed description of its neurorobotic implementation is provided by Prescott et al. [36]. Here we note that all of these models are based on standard ‘leaky integrator’ units, where one unit is used to represent activity in a pool of neurons in each of the modelled nuclei illustrated in Figure 1, and for each of the competing basal ganglia ‘channels’. The input to channel i of the model, denoted ${\mathit{s}}_{\mathit{i}}$ indicates the instaneous salience of that channel as computed by structures outside of the basal ganglia or by the striatal projection neurons themselves. The output for channel i, denoted ${\mathit{y}}_{\mathit{i}}^{\mathit{s}\mathit{n}\mathit{r}}$ indicates the instantaneous value of the inhibitory signal from the basal ganglia output nuclei to their targets elsewhere in the brain. Tonic dopamine modulation of the model is provided by a multiplicative factor in the equations specifying afferent input to the striatum, based on a variable parameter λwhere 0.0 ≤ λ ≤ 0.5. In striatal D1 units, where dopamine modulation increases synaptic efficacy, the effective weight is (1+λ); in D2 units, where the effect is to reduce efficacy, the weight is (1-λ). In the investigations described below we vary λwithin the above range to simulate the effects of changing tonic dopamine modulation in the basal ganglia. Note that the intention is to model changes in dopamine that happen over longer time-scales and that we do not attempt, in this study, to model phasic short-latency dopamine responses that may also have an important effect on selection and are considered to be play a critical role in some forms of learning [8,54].

Using Basal Ganglia Outputs as Selection Signals

In order to consider the basal ganglia model as a model of selection we need to place some interpretation on the effects of basal ganglia outputs on targets in the brainstem and thalamus. As noted above, selection corresponds to basal ganglia removing inhibition from the winner(s) and increasing inhibition on the losers. We assume that for any given channel this effect varies between full disinhibition, partial inhibition, and full inhibition, and model this effect via a mechanism termed ‘shunting inhibition’ thought to capture some of the non-linear effects of the Gabaergic outputs from basal ganglia on their targets in vivo (see [36]). Specifically, for the i^th channel, we define the selection, or gating, signal $e_i (0 \le e_i\le 1)$ as

$$e_i=\left\{\begin{array}{c}0:\widehat{e}<0\\ \widehat{e}:0\le \widehat{e}\le 1,\\ 1:\widehat{e}>1\end{array}\right.{\mathrm{w}\mathrm{h}\mathrm{e}\mathrm{r}\mathrm{e} \widehat{e}}_i=1-y_i^{snr}/y_C^{snr}$$

(1)

Here $y_C^{snr}$ is a constant equal to the value of $y_i^{snr}$obtained when the basal ganglia model is run to convergence with zero salience input on all channels (in other words, the tonic output level when there are no active competitors).

Metrics for Measuring Effective Selection

The gating signal, $e_i$, provides a useful normalized measure of selection efficiency that we can use to evaluate any given version of the model against our requirements for effective action selection. It is useful to define some qualitative labels for different values of $e_i$. Allowing a 5% margin from absolute limits, we define the selection state of the ith competitor as fully selected if $0.95\le e_i\le 1.0$, partially selected if $0.05\le e_i<0.95$, and unselected if $e_i<0.05$. It will also be useful to have specific metrics relating to the winning channel, hence, we define $e_w={\mathrm{m}\mathrm{a}\mathrm{x}}_{\forall i}{e}_i$ as the efficiency of the current winner, $1-e_w$as its inefficiency, and

$$d_w=2\left({\sum }_i{e}_i-e_w\right)/{\sum }_i{e}_i$$

as the level of distortion affecting the output of this winner. Note that $d_w$ will equal zero when all other competitors have zero efficiency, will increase with the number of partially disinhibited losers, and will be 1.0 or greater if two or more channels are fully disinhibited (multiple winners). Inspired by ethological research [55], we will also describe an uninterrupted series of time-steps that share the same winner, and for which $e_w\ge 0.05$, as a single bout of behavior.

Finally, we note that the result of the basal ganglia selection competition, as a whole, can be summarised by the vector e. Using the criteria just defined for single competitors we assign the following labels to the possible outcomes of the full competition as defined by the intantaneous value of e:

Clean selection: One competitor fully selected, all others unselected.

No selection: All competitors unselected.

Partial selection: One or more competitors partially selected, no competitor fully selected.

Distorted selection: One competitor only fully selected, at least one other partially selected.

Multiple selection: Two or more competitors fully selected.

3. A Robot Embedding of a Model of Action Selection by the Basal Ganglia

Prescott et al. [36,56] embedded the extended basal ganglia model [41] within the control architecture of a mobile robot in order to demonstrate that signal selection by the embedded model, as described above, could translate into effective action selection for an embodied agent expressing goal-directed behavior. This model was based on consideration of the typical behavior of a hungry rat placed in an open-topped arena with high sides (Figure 2a and Supplementary Video, part 1). In this situation, animals initially show fearful or thigmotaxic behavior—avoiding open areas in the centre of the arena, whilst exploring walls and corners. As animals become more accustomed to the novel environment, they show foraging behavior—collecting food pellets from a dish placed in the centre of the arena and typically consuming them in sheltered areas near the periphery. Salamone [10] showed that effective behavior switching in a similar environment is compromised by the dopamine antagonist Haloperidol and by dopamine-depleting lesions of the striatum, hence the task is an appropriate one for investigation of the effects of variation in simulated dopamine on robot action selection.

In the robot model of this task (Figure 2b and Supplementary Video, part 2), a table-top Khepera mobile robot with a gripper turret is placed in a rectangular arena with illuminated corners, to simulate safe places, and with small foil-covered cylinders to simulate food rewards. Fearful behavior is simulated as staying close to walls and corners; foraging involves searching for, locating, and picking up the cylinders; finally, consummatory behavior is modelled as carrying a cylinder to one of the two illuminated corners and depositing it there. To generate appropriate behavior, robot activity is decomposed into five sub-systems inspired by the ethological classification of behavior. Three of the five action sub-systems—cylinder-seek, wall-seek, and wall-follow—map patterns of input from the robot’s sensors into movements that orient the robot towards or away from specific types of stimuli (e.g. object contours). These behaviors can be viewed as belonging to the ethological category of orienting responses or taxes [see e.g. 57]. The two remaining sub-systems—cylinder-pickup and cylinder-deposit—generate carefully timed movement sequences that achieve specific behavioral outcomes and are modeled on the ethological concept of a fixed action pattern (FAP) [58]. Each action sub-system generates its preferred action at a given moment in the form of a motor vector that specifies target values for the speeds of the two wheels, and for the positions of the gripper arm (raised/lowered) and gripper jaw (open/shut). In the case of the orienting responses, the preferred action is computed using the sensory information available to the robot at that moment, in the case of FAPs, the action specification can also depend on the current value of an internal clock.

In order to make appropriate action selection decisions, the robot needs information about relevant external and internal cues. Signals pertaining to external cues are computed by perceptual sub-systems from the raw sensory data available to the robot via an array of infra-red distance sensor signals, an ambient light sensor, and an optical sensor in the robot gripper. These sensory inputs are used to compute four bipolar signals indicating: the presence (+1) or absence (-1) of a nearby wall, nest area, cylinder, or of an object in the robot gripper. Internal state cues are provided in the form of two real-valued intrinsic drives, loosely analogous to hunger and fear, as calculated by two motivational sub-systems. In the model, ‘fear’ is calculated as a function of exposure to the environment and is reduced with time spent in the environment, whilst ‘hunger’ gradually increases with time and is reduced when cylinders are deposited in the nest corners of the arena.

Figure 3 shows how these different component sub-systems come together and interact with the embedded basal ganglia model. The model is composed of three parts: (i) the robot and its sensory and motor systems; (ii) the embedding architecture, that is, the set of perceptual, motivational, action sub-systems; and its interface to (iii) the extended basal ganglia model. Connections for the first of the five action sub-systems are shown, projections to and from the other action sub-systems are indicated by dotted lines.

As shown in Figure 3, each action sub-system takes inputs from the perceptual and motivational sub-systems, and from an internally-generated busy signal (b) that is only non-zero if the action is currently selected, and that allows that sub-system to selectively boost its own salience. Based on these inputs the action sub-system generates a weighted sum (the weights are hand-tuned) that is an estimate of its own instantaneous salience (s) that is provided as an input to the embedded basal ganglia model. At the same time, the action-generating component of the sub-system calculates its preferred motor vector based on the robot’s sensor input and a feedback signal (f) from the component of basal ganglia model corresponding to the ventrolateral thalamus (VL). This feedback signal is used to update or reset the clock (C) for the action system (in the case of a FAP), and to trigger the busy signal that contributes to its salience calculation.

As noted above, for each action sub-system i, the output of the basal ganglia, ${\mathit{y}}_{\mathit{i}}^{\mathit{s}\mathit{n}\mathit{r}}$, is converted into a gating signal, ${\mathit{e}}_{\mathit{i}}$, via equation 1 which is then used to scale the value of the motor vector for that action. An integrator module then sums all of the motor vectors and passes the aggregate vector through a limiter (L) that constrains all values to lie in the range 0–1, this vector is then converted into the specific motor commands that control the robot. The full robot model operates on a series of discrete time-steps providing sensor updates and modifying its action output at a rate of approximately 7Hz. The embedded basal ganglia model is simulated using the Euler method and run to convergence for each time-step of the robot model.

Full details of the test environment, the robot sensor and motor systems, the embedding architecture components, and the simulation of the extended basal ganglia model, including their motivation in relation to the neuroscientific understanding of relevant brain sub-systems, are provided in [36], which also provides a broader discussion of the use of robotic models in neuroscience.

To illustrate normal functioning of the model, Figure 4 shows a single 240s run with the level of simulated tonic dopamine set at λ = 0.20. The top five lines of the plot show the value of the gating signal, ${\mathit{e}}_{\mathit{i}}$, for each of the five action sub-systems at each time step in the style of a behavioral ethogram. Comparing across the different actions, it is evident that the robot generates extended sequences of behavior with no more than one action sub-system fully selected at any given time. The efficiency of selected actions is at or near 100%, actions are performed over extended bouts (solid blocks of high efficiency) and the inefficiency of the winner (plotted as the sixth line of the plot) is generally near zero. In this run, the robot is initially fearful and seeks the wall (wall-seek), then switches into its wall-follow behavior. This can be viewed as forming a higher-order sequence of avoidance (Av) behavior as labelled in the seventh line of the plot. The final line of the plot shows the activity of the model motivational systems. As the level of simulated fear gradually subsides, simulated hunger is increasing. As a result, at around 50s, the robot rapidly switches into its cylinder-seek behavior. When it subsequently locates a cylinder it switches to cylinder-pickup, then to wall-seek (this time carrying a cylinder), then wall-follow, and, when it finds a lit corner, cylinder-deposit. The higher-order action sequence beginning with cylinder-seek and ending with a successful deposit is labelled as foraging (Fo) in the plot. Releasing the cylinder has the effect of reducing simulated hunger such that the robot is again motivated principally by fear to perform its avoidance-related behaviors (wall-seek and wall-follow). However, the level of simulated hunger gradually rises which leads to two further higher-order foraging sequences interspersed by a period of no behavior. The absence of behavior occurs when neither of the intrinsic motivations is sufficiently strong to trigger any action—the robot sits idle, just as the rat might wait quietly in the corner of the arena.

From the perspective of the observer, the robot’s behavior appears to be integrated and purposeful, individual action bouts are assembled to larger sequences that successfully reduce its drives. In section 5, we will compare this example of effective action selection and integrated behavior with other runs in which the robot demonstrates various forms of behavioral disintegration as the result lowering or raising the level of simulated dopamine in the model basal ganglia.

4. Tonic Dopamine Modulation in the Extended Basal Ganglia Model

Before presenting further results for the robot model, it is useful to investigate the response of a non-embodied version of the extended basal ganglia model to changes in tonic dopamine modulation as this will provide a useful yardstick for evaluating the embodied robotic version, and will help us to better understand the specific consequences that embodiment might entail. This investigation builds on prior studies of simulated tonic dopamine modulation [40,41] by providing a fine-grained analysis across the spectrum of possible λ levels.

To better understand the effect of varying simulated dopamine on the selection properties of the extended basal ganglia model we simulated a five channel model, with two active channels, varying the salience $s_1$ in channel 1 systematically from 0 to 1 in steps of 0.01, then for each value of $s_1$, varying the salience $s_2$of channel 2 from 0 through 1, again in steps of 0.01. For each resulting salience vector $\left(s_1, s_2, 0, 0, 0\right)$ the model was run to convergence and the result classified according to the scheme set out in Section 2. Importantly, selection competitions were run in sequence from low values to high values. The activations levels of all leaky integrators in the model were initialized to zero for each new value of $s_1$ but thereafter, while that salience value was tested, were retained from one competition to the next. In other words, we simulated the situation where channel 1 was initially the only active channel, and gradually increased channel 2 while holding channel 1 constant, the goal being to simulate some aspects of the continuity of experience that we can expect in the robot model in which the recent history of selection competitions may influence the current competition through hysteresis. Previous studies have established that the basal ganglia model, in both its original and extended forms, shows good selection properties, across a wide-range of salience pairings, with the simulated dopamine level set at around λ= 0.20; for this analysis we therefore looked at values of simulated dopamine ranging from 0 through to 0.5 in increments of 0.01.

Figure 5a shows the percentage of action selection competitions, across the 500,000 (50x100x100) runs, falling into each of the selection classes— clean selection, no selection, partial selection, distortion, and multiple selection. Values of λ below 0.01 result in no selection, while in the range 0.04–0.15 partial selection predominates, from 0.15 upwards the majority of competitions end in clean selection with a peak around 0.22, distorted selection begins to appear with values above 0.2, and multiple selection occurs with levels of 0.25 and greater.

Figure 5b shows the average values of efficiency and distortion across all runs at a given level of λ. These graphs indicate that average efficiency increases gradually reaching its maximal value (1.0) at λ = 0.23, distortion increases gradually from zero beginning at around λ = 0.15 and reaching 0.2 by λ = 0.5.

ranging from 0 through to 0.5 in increments of 0.01. Data was obtained through an exhaustive search of a two-dimensional salience space. Partial selection is predominant for low dopamine values, distortion and multiple-selection evident at high dopamine values. B. Average efficiency and distortion across all runs at each level of λ.

Figure 5 shows the average outcome at different levels of λ across all possible $\left(s_1, s_2\right)$ dyads. In order to better understand the interplay between salience, simulated dopamine and selection, in Figure 6 we show the outcome of the simulation for five specific values of simulated dopamine (λ= 0.06, 0.12, 0.22, 0.31, 0.40) but indicating the boundaries of different classes of selection outcomes on the $\left(s_1,s_2\right)$plane. For clean selection only, the plots also distinguish between selection of channel 1 (which is active first) and of channel 2 (which then competes for selection against channel 1).

Several properties of Figure 6 are worth noting. First, at all levels of λ, there is little or no selection at very low salience levels. This is largely as a consequence of the threshold value of the model striatal input neurons which serves to weed-out weakly salient inputs. Second, with low λ (e.g. 0.12), clean selection (C1 or C2 in Figure 6) occurs, if at all, only when there is a high salience input in just one channel, otherwise partial selection is the norm. Third, at all simulated dopamine levels there is no clean selection for strong, evenly matched, salience values (top-right corner of all plots). With low values of λ (0.06, 0.12) the outcome is no selection or partial selection of one or both channels, while with high values (0.31, 0.4) the result is distortion of the selected channel or multiple selection. The dotted line in the central plot (λ= 0.22) is shown to illustrate the extent of hysteresis in the model: channel 1 wins many selection competitions (encroaches across the diagonal) in which channel 2 salience is greater, purely because is was activated first.

To further our understanding of hysteresis in the model, the simulation results described above were reclassified to show the extent to which channel 1, which is always active first, is preferred to channel 2 irrespective of the selection outcome. Thus, the result of each competition was rescored as either a channel 1 win $(e_1>e_2)$, a channel 2 win ${(e}_2>e_1)$, a tie ${(e}_1=e_2\ne 0)$ , or no selection $(e_1=e_2=0)$. Figure 7a shows the results of this reclassification, and reveals that hysteresis is a property of the model for all but the lowest levels of simulated dopamine modulation (λ ≤ 0.06) with channel 1 consistently winning up to 10% more competitions than channel 2.

, required for channel 2 to prevail (i.e. e2>e1) against a channel 1 salience, $s_1$, of 0.3, 0.4, or 0.5, for different values of λ. Data is shown only where there is a clear switch from channel 1 to channel 2 with increasing $s_2$ (i.e. without an intervening interval of no-selection or multiple selection). The degree of hysteresis varies depending on λ and $s_1$, with the value of λ that generates maximum hysteresis decreasing with increasing $s_1$.

However, this is still not the full story. Figure 7b shows a further measure of hysteresis—the level of channel 2 salience required to overcome a given level of channel 1 salience—for three different initial, fixed levels of $s_1$. The plot shows that hysteresis is governed by a complex interaction of λ with salience, specifically, for values of $s_1$ in the range 0.3-0.5 the degree of hysteresis first increases with increasing λ, peaks, and then decreases; at its maximum, channel 2 salience needs to reach 176% of the channel 1 salience in order to win the selection competition. The peak λ value for hysteresis also changes for different values of $s_1$—as the salience of the selected channel increases, the value of λ at which hysteresis is maximal goes lower.

We conclude that the relatively flat level of hysteresis shown across a broad range of λ values in Figure 7a masks a significance dependency on salience. This outcome can be explained by understanding that hysteresis in the model occurs as a consequence of activity in the basal-ganglia-thalamo-cortical feedback loop (via VL and TRN in Figure 1). Activity in this loop increases in proportion to reduced basal ganglia output, in other words, it increases with selection efficiency. With low values of λ, partial selection (low efficiency) predominates for low or intermediate salience values. This outcome results in less positive feedback via the thalamo-cortical pathway than with high salience competitions. Consequently, when λ is low, hysteresis will be maximal with high salience. In contrast, high λ levels result in high efficiency selection with comparatively low-levels of salience input thus generating substantial positive feedback and strong hysteresis. However, high-level salience competitions can result in the partial or full disinhibition of multiple channels (distorted or multiple selection), a consequence of this is increased TRN inhibition of the VL thalamus for the winning channel resulting in a significant reduction in thalamocortical feedback for that channel. This means that with higher levels of λ, the current winner can be more vulnerable to interrupt by its competitors.

In Figure 5, clean selection for the disembodied model was above 75% in the range 0.2≤ λ< 0.3, fell steeply to zero in the lower range 0.0≤ λ< 0.2 and more gradually (to 55%) in the higher range 0.3≤λ≤0.5. Defining these ranges as, respectively, intermediate, low, and high λ, and building on the analysis just described (and in earlier explorations in [36,40,41]), we can make the following hypotheses concerning the effects of varying simulated dopamine in the robotic model:

• Hypothesis 1 (h1). At intermediate levels of λ(0.2≤λ< 0.3) we should expect to see a high proportion of clean selection with selected behaviors fully disinhibited and competing behaviors fully suppressed.
• Hypothesis 2. At low levels of λ (0.0≤λ< 0.2) we should expect a predominance of partial selection or no selection (very low λ) and consequently the slowing or absence of movement.
• Hypothesis 3. For high levels of λ0.3≤λ≤0.5) we should expect to see reduced inhibition of losing channels, leading to distorted or multiple selection, and resulting in motor commands that mix the movement requests of more than one action sub-system.
• Hypothesis 4. At both low and high levels of λ, we should expect to see changes in the hysteresis of selected channels modulated according to the nature of the salience competition (e.g. whether the salience of competing channels is high or low, or evenly matched) as illustrated in Figure 7b. Changes to hysteresis can be expected translate into consequences for action maintainence and for the timing of behavioral switching.

With respect to each of these hypotheses, the observed behavior of the robot may depend on a variety of factors related to its embodiment (discussed further below) and the requirement to generate sequences of integrated behavior. Moreover, whereas the above analysis was based on an exhaustive search of an essentially two-dimensional salience space, the robot model samples behavior-dependant trajectories through a five-dimensional salience space. The actual outcomes with respect to hypotheses 1-4 are therefore only partially predicatable from the disembodied model and to be further determined from observation.

5. Selection in the Neurorobotic Basal Ganglia Model

Based on our analysis of the disembodied model we decided to test the robot for 30 trials each at low, intermediate and high simulated dopamine levels, with five trials, each lasting 120s, at each of 18 different values of λ: low= 0.03, 0.06, 0.09, 0.12, 0.15, 0.18, intermediate= 0.20, 0.21, 0.22, 0.23, 0.25, 0.28, and high= 0.31, 0.34, 0.37, 0.40, 0.43, 0.46. The robot started each trial in the centre of the arena, facing one of the four walls, with four cylinders placed 18cm diagonally in from each corner (Figure 2, right).

In each, trial, which typically consisted of around 800 robot time-steps, the outcome of the basal ganglia selection competition, at each time-step, was classified according to the selection criteria specified above. For each λ value, the percentage of time-steps resulting in each type of selection outcome was then averaged across all five trials regardless of the behavioral outcome on individual trials (which we consider next). The results of this analysis are shown in Figure 8a–e together with a plot of average efficiency and distortion across the different λ levels (8f).

These results shows the expected similarity between the selection profiles for the robotic and non-embodied models, nevertheless there are some important differences. These include, in the robotic model, an increased proportion of partial selection at low λlevels (0.03 ≤ λ≤ 0.12), of clean selection at intermediate and moderately-high levels (0.2≤ λ≤0.4), and of distorted selection at high levels (0.3 ≤ λ≤0.46). There is also an almost complete absence of multiple selection at high λ levels. Whilst average efficiency is similar across the robotic and disembodied models, the robot model overall has less distortion except at the highest λ levels. In the intermediate range of simulated dopamine (λ0.20–0.29) clean selection for the robotic model is in the range 89-95% compared to 73-81% clean selection for the disembodied model.

These results largely reflect the fact the robot model is spending little time sampling the very high salience areas of the state-space, or the very low salience areas, compared to the exhaustive search conducted for the disembodied model. This was confirmed by an analysis of salience values across 15 runs (one at each level of λ) which found that 95% of selection competitions were in the range 0.3–0.75 for the winning channel and 0.2–0.7 for the strongest losing channel (see also [36] for a plot of how the salience space is sampled by the robot model). Note that that there may also be up to five channels with non-zero salience at any time compared to just two in the disembodied model.

Effects of simulated dopamine modulation on behavioral outcome

Our previous robotic study of the basal ganglia [36] showed that an embedded basal ganglia model was able to generate integrate behavior in our biology-inspired foraging task and for a specific intermediate value of simulated dopamine (λ= 0.20). In the current study, we address the question of how varying simulated dopamine impacts behavioral integration, and seek to describe and understand a variety of distinctive patterns of behavioral disintegration that arise when simulated dopamine is reduced or increase relative to this baseline.

To begin this analysis a simple binary classification scheme was developed and applied to the 90 robot trials described above, evaluating each trial according to its success in achieving higher-order behavioral goals. Specifically, we define ‘integrated behavior’ for this task as constituting, at minimum, successful avoidance in the initial ‘high fear/low hunger’ phase, and a successful foraging sequence in the later ‘low fear/high hunger’ phase. Operationally, we define:

(i)

successful avoidance as activity resulting in the discovery of a wall (ignoring any cylinders encountered en route) followed by movement some distance along the wall’s length, and

(ii)

successful foraging as activity resulting in the deposition of a cylinder in a ‘nest’ area.

This classification scheme proved to be sufficiently simple to be applied during live observation of robot behavior, in addition, automatic logs were recorded detailing the robot’s sensory, motivational, and basal ganglia state at each time-step, and the bout structure of its behavioral selections, allowing us to reconstruct and analyse the robot’s behavior post hoc.

The outcome of our initial analysis was as follows. Seven levels of simulated dopamine (0.20–0.28 and 0.37) were scored as generating successful behavior on all five trials, five levels (0.03–0.12 and 0.46) were unsuccessful on all trials, and the remaining six levels (0.15, 0.18, 0.31, 0.34, 0.40, 0.43) generated a mixture of successful and unsuccessful trials.

In order to better understand what was happening at levels of λ that generated mixed results, a quota sampling strategy was implemented in which further trials were conducted until a total of five successful trials, at each of these levels, had been achieved. This required between 1 and 11 trials per level, resulting in an additional 26 trials. Figure 9 shows the total trials (9a) the overall success rate (9b) at different levels of λ, across all 116 trials, assessed against the criteria of success in both avoidance and foraging. Figure 9c shows a more detailed analysis of types of failures under the low and high λregimes that we describe further below.

Figure 9b confirms that the in the range of intermediate λvalues (0.2–0.28), that generates a high proportions of clean selection in Figure 8, the robot also reliably generates integrated sequences of behavior. The absence of any failures in the 30 trials in this range provides a 95% confidence level that the failure rate for this class of models is 10% or less.

In the remainder of this section we consider the nature of the failures in behavioral integration that occur with levels of λ below or above this intermediate range then explore the effects of simulated dopaminemodulation on the timing and frequency of behavior switching. Figure 9c provides an analysis of the types of failure of behavioral integration observed at different level of λ, and as described in Table 1. Figure 10a–e shows some example runs, recorded with low and high λ, that help to illustrate the robot behavior observed at different levels of simulated dopamine.

In the statistical analyses reported below we use an alpha value of 0.05 and report significance values as two-tailed. If Levene’s test is significant then we report “equal variances not assumed” and provide adjusted degrees of freedom and p-values.

Behavioral consequences of low simulated tonic dopamine (λ< 0.2)

Slowed movement and periods of inaction. In section 4 we showed that the model basal ganglia generates partial (low efficiency) selection for low levels of simulated dopamine. Since our robotic model employs the basal ganglia output as a gate on targeted motor systems, the consequence of partial selection in behavioral terms should be that this gate is not fully opened for winning competitors; motor acts should be slowed or even extinguished altogether. This expectation, noted as hypothesis 1 above, was borne out in our study (see Figure 9c) which saw the expected translation of partial/weak selection into slowed movement (sm) for all runs at λ level 0.12 or lower. At λ= 0.06, 0.03 the robot is moving too slowly to complete a successful foraging sequence in the time allowed, thus failing altogether on the criterion for successful avoidance (fa). Periods during which the robot makes no movement (am), despite being otherwise sufficiently motivated, are seen at λ= 0.06 (average of 14s per trial, compared to 2s for intermediate levels of λ) and for longer spells at λ= 0.03 (average of 38s per trial). Note that it is possible to distinguish between the dysfunctional absence of movement due to low λ as seen in Figure 10a, and its appropriate absence during periods of low motivation (as in the period of no selection for λ= 0.20 in Figure 4). The Supplementary Video (part 4) shows an example of slow movement and no movement for an example run with λ= 0.10.

Premature deselection. In the range λ= 0.06–0.15 behavior can break down as the result of the premature deselection of an ongoing behavior, this can be seen as a failure of persistence or action maintenance. At λ= 0.09 or below this typically occured during the initial wall-seek bout leading to an absence of movement and failure to reach the wall as noted above. A further point of vulnerability was seen in the range λ= 0.09–0.15 and occured when the robot attempted to execute the cylinder pickup FAP but either failed to grasped the cylinder (fgc in 9b) or failed to raise the gripper arm at the end of cylinder-pickup bout (fra in 9c). An example of the fgc failure is shown in the Supplementary Video (part 5). Failure to raise the gripper arm occurred in 80% of trials at λ= 0.12 and 50% of trials at λ= 0.15, and also resulted in a behavioral trap, as described in Appendix 1, where the robot detected its lowered arm as an obstacle and engaged in a slow circling behavior until the end of the trial.

Failures more likely at low salience levels. Our experiments show that, at low λ,weakly selected behaviors are typically not executed with sufficient vigour and can be vulnerable to interrupt, further investigation also shows support for hypothesis 4—that the effects of varying simulated dopamine can also depend on salience level. Specifically, comparison across the 10 trials at λ= 0.15 shows that the variability in outcome (successful vs. unsuccessful) resulted from differences in the timing of the initial cylinder-pickup bout across trials—the robot encountered a cylinder, and initiated the cylinder-pickup FAP, significantly later in the successful runs (M= 66.7s, SD= 6.88) compared to the unsuccessful runs (M= 52.0s, SD= 2.23) (independent-samples t-test: t(4.8)= 4.557, p=0.007, equal variances not assumed). Recall that the salience of cylinder-pickup increases with simulated ‘hunger’, which in turn increases gradually with longer search times. In other words, for those runs at λ= 0.15 in which a cylinder is discovered quickly, and in which the robot is therefore more likely fail through premature deselection, the selection of the cylinder-pickup behavior is at a lower salience level than for the successful trials (longer search durations). This can be related to Figure 7b which showed reduced hysteresis, and hence less behavioral persistence, for low values of λ (compared to intermediate values). More generally, in all low λ conditions, robot behaviors are executed more efficiently at higher salience levels, and therefore the symptoms of reduced simulated dopamine such as slowed movement are more pronounced when salience is low.

Behavioral consequences of high simulated tonic dopamine (λ> 0.3)

Distortion of winning channels by active losers. At high levels of λ the non-embodied model predicted reduced inhibition of the motor output from losing channels leading to distortion of the winning action (hypothesis 3). The behavioral consequences of distortion are visible in the robot model with levels of simulated dopamine λ≥ 0.31 and occasionally resulted in behavioral disintegration for λ0.31, 0.34 through failure to complete a foraging bout (ff in Figure 9c). Likelihood of failure increased with very high levels of λ with more than 50% fails at λ0.4, 0.43 and 100% fails at λ0.46. At all of these λlevels, failure to forage was typically due to an inability to grasp a cylinder (fgc), however, other evidence of behavioral disintegration was also evident, particularly, difficulty in tracking walls (lw). Failure to grasp a cylinder oftens results in a second form of behavioral trap where the robot enters repeated cycles of cylinder-seek and (unsuccessful) cylinder-pickup, an example of this is shown Figure 10e (t= 85-120s), an example of this type of failure is shown in the Supplementary Video (part 6).

Failure more likely at high salience levels. That there was a mix of successful and unsuccessful runs, at some high λ levels, indicates that the impact of distortion on behavioral outcome can depend on circumstances. We illustrate this by comparing, in Figure 10d,e, two trials with λ= 0.31 showing that both successful foraging (10d) and disintegrated foraging (10e) are possible at this level. In 10d, the robot quickly locates a cylinder at t= 49s, in 10e, the only unsuccessful run at this λ level, there is a much more protracted cylinder-seek search ending at t= 84s (see Appendix 1 for a detailed commentary and comparison). At higher λ levels (0.40 and 0.43), comparison of successful (M= 37.1s, SD= 6.06) vs. unsuccessful trials (M= 63.3s, SD= 16.4) shows that, on average, in successful runs the robots discovered a cylinder whilst foraging earlier than in unsuccessful trials (independent-samples t-test: t(18)= -4.741, p<0.001). This is the reverse of the situation with low λ—with high simulated dopamine it is the longer search bouts, giving rise to higher salience levels (from increasing ‘hunger’), that tend to result in greater behavioral disintegration. This again matches hypothesis 3—that the effect of varying simulated dopamine on behavior will depend upon salience levels—with contrasting effects seen at low and high λ levels.

From Figure 7b, we can expect reduced hysteresis (behavioral persistence) for higher levels of λ, however, that figure also shows that increasing salience at high λdoes not significantly impact on hysteresis. To understand why the robot performs better at lower levels of salience with high λ we therefore need to look beyond the basal ganglia model itself and to consider the influence of distortion on behavioral persistence via its effect on behavior. This is the topic of our final analysis.

Effects of distortion on behavioral persistence

A key property of the robotic model, that distinguishes it from the non-embodied simulation, is that selection outcomes have behavioral consequences that shape the robot’s subsequent sensory experiencies. More specifically, the robot’s motor output, in part, determines its trajectory through the state-space of perceptual and motivational affordances for future selection competitions. Since varying the level of simulated dopamine can influence motor behavior by slowing movement or by merging partially selection actions with winning ones, it is interesting to establish whether this has any significant consequences for the selection behavior of the embodied model.

Here we explore this issue by examining the some effects of distorted selection on the timing and frequency of behavior switching. To assist this analysis an additional 90 robot trials were performed at all of the λlevels previously tested, but this time with a ‘winner-takes-all’ filter applied to the efficiency values of all sub-systems, such that the winning sub-system was always assigned an efficiency of 1.0, and all losers an efficiency of 0.0. In the following analyses the behavior of this winner-takes-all variant will be contrasted with the ‘soft’ selection generated by the standard model that allows multiple channels to influence motor output.

Timing of behavior switching. Our investigation of the non-embodied model showed significant hysteresis at almost all levels of simulated dopamine in the context of closely-matched salience competitions (Figure 7), this should show up strongly, in the robot model in the initial transition from avoidance to foraging behavior. The key competitors at this point are wall-follow and cylinder-seek and the prime determinant of their relative salience, that eventually allows the latter to prevail, is a gradual, time-determined reduction in ‘fear’ alongside a steady increase in ‘hunger’. The length of the time leading up to this switch from avoidance to foraging therefore provides an measure of the operation of behavioral persistence in the model. Figure 11a plots this ‘time-to-switch’ measure against different levels of λ and shows the different outcomes observed with both the standard model (from the original set of 90 trials) and the new winner-takes-all control. For each dopamine level we plot the average and standard error of the time-to-switch calculated over the five trials.

Comparison with Figure 7b, shows that the graph for the winner-takes-all variant provides a good match to the degree of hysteresis found for a fixed salience (on the initial winning channel) of 0.4. Since the salience of wall-follow preceding the switch is typically in the range 0.3–0.4, this demonstrates that hysteresis in the embodied model basal ganglia generates a corresponding level of behavioral persistence under winner-takes-all conditions. However, the standard model generates an interesting difference from this result. Specifically, two-way ANOVA shows a significant interaction (F(1,16)= 3.641, p<0.001) between model type (standard vs. winner-takes-all) and λ. Posthoc comparisons for low, intermediate and high λ values, show a difference for high values only (λ≥ 0.31) where switching occurs significantly earlier in the standard model (M= 31.7s, sd=6.26) compared to the winner-takes-all variant (M= 45.4s, sd= 5.66) (independent-samples t-test: t(58)= -8.92, p<0.001). We conclude that, with higher λ, the distortion provided by losing channels can significantly reduce behavioral persistence in the robot over and above the reduction resulting from lower hysteresis in the embedded basal ganglia at higher levels of simulated dopamine.

Looking at Figure 10 (panels d and e), which showed behavior for two trials with λ=0.31, we can observe, towards the end of the wall-follow bout (around t=30), a small, but gradually increasing, output on the cylinder-seek channel. It is this ‘leakage’ of motor output from the cylinder-seek sub-system that constitutes the difference between the standard and winner-takes-all versions of the model. A key to understanding the effect of this distortion is to note that the wall-follow behavior is not especially robust, and is sometimes pushed off track by sensor noise or wheel-slip even when driven by a clean motor signal. The effect of the motor noise introduced by partial selection of cylinder-seek is therefore to increase the variability in the robot trajectory making it more difficult to maintain sensor contact with the nearby wall. In this situation, any loss of the wall percept due to distorted movement will lead to a rapid reduction in wall-follow salience and a switch to the cylinder-seek behavior.

Increased switching frequency with high simulated dopamine. If distortion makes some behaviors more vulnerable to interrupt, then we might also expect increased levels of behavior switching. To investigate this possibility, Figure 11b illustrates one specific measure of switch frequency, the total number of bouts occurring during the first avoidance sequence and first foraging sequence of each trial. This measure was preferred to counting bouts (or switches) within a fixed time interval as it allows us to exploit a useful base-line—integrated behavior (by our earlier operational definitions) requires a minimum of seven bouts across these two sequences.

Since this measure can only be applied to trials containing a completed foraging sequence, this analysis only considered λ values in the range 0.15–0.43, and the graph plots the average and standard error of the number of bouts observed for the five successful trials at each simulated dopamine level. These data reveal that the performance of the robot is slightly above the base-line level of seven bouts across most of the range of simulated dopamine values, however, the number of bouts increases substantially for very high λ levels (λ= 0.40, 0.43; M= 21.3 bouts, SD= 4.73). Moreover, as shown in Figure 11b, comparison with winner-takes-all selection at these levels (M= 9.2 bouts, SD= 1.99) shows that the latter requires significantly fewer bouts (independent-samples t-test: t(2.22)= 4.33, p= 0.041, equal variances not assumed). We therefore conclude that the increased switching seen with the standard model is largely due to the distortion of motor output created by losing competitors. Figure 10e shows an example run with λ= 0.40 that illustrates the increased frequency of bout switching (between wall-seek and wall-follow in t= 0–50s) that can occur due to distortion with high simulated dopamine.

These analyses of the effects of increased λ on switch timing and frequency demonstrate that distortion in the robot model does not inevitably lead to a mixed motor output—trying to do two things at once—instead, its effect can be to make certain behavioral states more vulnerable to interrupt which can then lead to an increased frequency of behavior switching.