Not so Basic Instinct: Learning, Evolution, and the Behavioural Hypercycle

1. Introduction

The smartest moves are not always made by the cleverest creatures. Aesop’s fable of the fox and the hedgehog juxtaposes two modes of problem-solving: one imaginative and wide-ranging, the other single-minded and instinctive. As Stephen Jay Gould (2003) put it, these represent alternative tendencies to “diversify and colour” versus “intensify and cover.” In nature, such cognitive styles may be seen as spanning a continuum between the flexible and reflective at one extreme and the fixed and rigid on the other (Dennett, 1984). Both can produce strikingly sophisticated behaviour. Caddisfly larvae, for instance, construct underwater shelters by binding grains of sand, plant fragments, or shell pieces with silk that hardens on contact with water. These larvae sort and arrange materials selectively according to size and weight, using inherited motor routines refined by evolution rather than organized by imaginative planning. Interestingly, as Dawkins (1982) observed, if we saw a more intelligent animal such as a dolphin or chimpanzee performing such underwater engineering, we would marvel at its ingenuity. Yet the caddisfly achieves all this and more with a nervous system whose total weight is a fraction of a milligram. Something similar can be said for orb-weaving spiders, which execute hundreds of geometrically precise movements to spin complex webs, or for mason bees, potter wasps, and termites whose nests similarly embody intricate architectural acumen without insight (Gould & Gould, 2007). These examples show that behavioural complexity need not imply intelligence: remarkable organization can emerge from instinctive programs evolved (somehow) by blind selection acting on genetic variation.

Dawkins went on to suppose that this means we should perhaps be more impressed by the caddisfly than by any dolphin making such moves, because the caddisfly’s performance must be explained by evolution alone. Consequently, there is a greater burden on adaptationist reasoning to explain such instinctive feats than the behaviour of a generally intelligent organism. For the dolphin, nearly any specific act of problem-solving may be explicable under its general cognitive capacities. Once the deferred question as to how the dolphin evolved to be so clever in the first place is answered, that explanation can subsume any specific expressions of this cleverness. But for the caddisfly, no such shortcut exists. The aims of this paper are to consider how complex and adaptive behaviours may have evolved in animals like the caddisfly and what this mechanism suggests about the relationship between instinct, learning, and intelligence across taxa more generally.

1.1. The Baldwin Effect

An idea relevant to how learning can influence the evolution of instinct is the Baldwin effect. To illustrate, imagine some environmental challenge, such as the presence of a nutritious nut or shellfish being abundant but difficult for other animals to handle effectively for consumption (i.e., crack open its hard external shell to expose the edible insides). Some individuals in this environment may learn a “Good Trick”: a behaviour such as by dropping the shell from a great height or placing it on a flat rock and hammering it with a sharp stone (Dennett, 1995). Suppose that consuming the nuts yields a high fitness benefit to any animal learning the behaviour, and that the habitat is sufficiently stable so that it continues to feature as an important food source over many generations. If there is genetic variation in acquiring this technique, selection may then favour genetic variants biasing individuals towards learning it more rapidly, such as by favouring alleles that alter neural architecture to increase the likelihood of manipulating the nut or playing with stones. This makes rediscovery of the Good Trick more probable over successive generations. By favouring such variants as they arise, selection can progressively reduce the environmental input required to acquire the behaviour (genetic assimilation), eventually rendering it instinctive.

An interesting implication of this is that traits may evolve more rapidly than would otherwise occur without learning (Dennett, 1995; Maynard Smith, 1987). This is because the probability of an individual initially being born with precisely the necessary genetic configuration to innately know the Good Trick is vastly different from the likelihood that there may be additive genetic variation influencing how readily the behaviour is learned. Yet the former is what would be required if the capacity for learning did not exist. In the selective landscape metaphor, a genotype completely encoding the Good Trick is like an isolated telegraph pole that selection cannot climb, whereas learning can enable adaptation by exposing to selection infinitesimal approximations of that “ideal” genotype, enabling the adaptation to evolve it more like climbing a gentle hill towards its summit (Dennett, 1995) with modelling confirming that even basic reinforcement learning can greatly accelerate subsequent adaptation (Hinton & Nowlan, 1987).

There is of course an alternative path selection might take to genetic assimilation. It may simply favour enhancing learning capacity itself. That is, rather than encode this specific Good Trick (intensify and cover), the species may evolve a progressively greater capacity for generally learning Good Tricks (diversify and colour). As thus told, it is tempting to suppose that this presents a sort of fork in the road, where some lineages evolve down a path towards robotic repertoires while others towards flexible and greater learning capabilities. Which path is taken has been suggested to hinge on the degree to which characteristics of the environmental challenge remain consistent across generations, with highly stable environments favouring genetic assimilation and fluctuations across generations and throughout a species range promoting plasticity (Stephens, 1991). We demonstrate below that these paths are not as mutually exclusive as they might appear but may in fact become intertwined and converge in high-capacity systems.

1.2. The Structure of Behaviour

To understand this, we must examine the structure of the behaviours in question. Many complex instincts and learned routines share a common motif: a sequence of functionally linked behaviours that unfold in a fixed order once triggered by the detection of a specific stimulus (Thorpe, 1956). A wasp preparing a nest may perform a pattern of digging, provisioning, and sealing. A spider laying out its web follows a highly stereotyped trajectory, modulated only by spatial cues. A greylag goose that begins to retrieve a displaced egg continues the motion to completion even if the egg is removed mid-sequence. Each case shows an apparently hard-wired algorithm for solving a recurring ecological problem. The interesting thing about this structure is that it mirrors patterns long familiar to animal trainers and psychologists, where sequences of several distinct actions can be “chained” together through conditioned reinforcement. Consequences of the completion of each learned response or behavioural step becomes the discriminative cue for the next in the series, with the initial link in the chain being prompted by an environmental stimulus and the terminal behaviour typically being followed by a food reward or some other external source of reinforcement (Skinner, 1938). Textbook examples of chains can include a pigeon pecking, turning, and pulling a string to receive grain, or a rat pressing a lever and then climbing a ladder to reach food, or pulling a chain to then access a lever which can then be pressed to earn a reward.

Denoting each distinct behavioural component in such a sequence as $R_i$, we can compare instinct to a sequential behaviour chain that can be entirely learned, that is, conditioned by “chaining” procedures (Hull, 1943).

$$\begin{gathered} \mathrm{S}\mathrm{i}\mathrm{g}\mathrm{n}\mathrm{S}\mathrm{t}\mathrm{i}\mathrm{m}\mathrm{u}\mathrm{l}\mathrm{u}\mathrm{s}\to R_1\to R_2\to R_3\to R_4\to R_5\to \mathrm{C}\mathrm{o}\mathrm{n}\mathrm{s}\mathrm{u}\mathrm{m}\mathrm{m}\mathrm{a}\mathrm{t}\mathrm{o}\mathrm{r}\mathrm{y}\mathrm{A}\mathrm{c}\mathrm{t}/\mathrm{S}\mathrm{t}\mathrm{i}\mathrm{m}\mathrm{u}\mathrm{l}\mathrm{u}\mathrm{s} \\ {S}_1\to R_1\to S_2\to R_2\to S_3\to R_3\to S_4\to R_4\to S_5\to R_5\to O_5 \end{gathered}$$

where $S_i$ represents a stimulus that $R_i$ is conditioned to be triggered by, which itself is either contingent on execution of $R_{i-1}$ (except $S_1$ which is usually controlled by the environment), and $O_5$ represents the reinforcement delivered upon completion of the chain (usually food). The most commonly used method of chaining is “backward chaining”. This means initially training the animal to perform, say $S_5\to R_5\to O_5$ only (e.g., Jerome et al., 2007). When this is fully learned, descriptively what also occurs is that $S_5$ becomes a conditioned or learned source of reinforcement by association with $O_5$. This means perception of $S_5$ itself becomes capable of reinforcing other associations, at which point the animal is then trained $S_4\to R_4\to S_5\to R_5\to O_5$, which is then repeated (and so on, and so forth).

1.3. Generating and Assimilating Behavioural Chains in Nature

The resemblance raises the question as to whether the two forms share a common underlying logic or evolutionary pathway via the Baldwin effect. If a species has already evolved various, individually simple behaviours, complex adaptive behaviour can conceivably arise from the linking of simpler units into ordered sequences if the combination is successful in unlocking greater or new sources of reinforcement. As with learned chains guided by experimenters in the lab, reinforcement could link behaviours together in nature to form functional sequences. Assuming that reinforcement is sufficiently aligned with positive effects on Darwinian fitness, then, when learning repeatedly produces such a behaviour combination, genetic variants supporting its efficient execution may be favoured, gradually embedding once-plastic routines into inherited architecture. In other words, we may consider if complex instincts, such as any conforming to the “fixed action” motif mentioned above, have evolved via the Baldwin effect. Crucially, this could occur in species that need not be capable of generating similar behaviours based on insight or goal-directed processes. To evolve an instinct via chaining and genetic assimilation requires that the ancestors (1) had basic associative conditioning capacities (e.g., stimulus-response learning); (2) were already capable of individually much simpler behaviours; and (3) would at least occasionally execute such behaviours in novel or challenging situations where serendipitous success would unlock strong sources of reinforcement (these sources presumed here to correspond with positive outcomes on Darwinian fitness).

The reinforcement-based description adopted here should not be taken to imply that behavioural chains in animals are necessarily acquired or deployed through blind trial-and-error mechanisms. In species possessing more elaborate cognitive capacities, behavioural sequences may be assembled and executed through goal-directed systems that represent actions in relation to their consequences, with learned sequences exhibiting hierarchical organization and flexible deployment (Dezfouli & Balleine, 2012, 2013). More generally, learning does not typically consist of rigid stimulus-response mappings, and the same learned relationship may often be expressed through a variety of behavioural forms depending on circumstances (Cahill et al., 2001). That said, we proceed by presupposing deliberately minimal associative conditioning capacities, namely that links of a behavioural sequence can be acquired through reinforcement, secondary conditioning, and the progressive strengthening of associations between its component responses and consequences. This is important when considering animals with much more limited behavioural and cognitive repertoires: spiders, beetles, and other invertebrates may lack the neural wherewithal required for the forms of goal-directed learning found in perhaps most vertebrates, while nevertheless exhibiting reinforcement learning and capacities for associative conditioning (Hollis & Guillette, 2015). In short, the purpose of this description is simply to establish the minimal conditions under which linked behavioural components could be organized by learning into this sort of structure.

Another prerequisite is that organisms are likely to generate novel combinations of simpler behaviours. Although not as extensively documented, evidence for such semi-independent behavioural elements comes from classical phenomena including displacement activities (Tinbergen, 1951), and conflict- (e.g., Van Iersel & Tinbergen, 1948) or frustration-induced behavioural generation (Amsel, 1992; Winterbauer & Bouton, 2010), and in response to novel challenges (e.g., Epstein, 1987; Weir et al., 2002) in which responses normally associated with one motivational or environmental context are expressed in another. Such behavioural spillover can involve the redeployment of previously acquired or evolved routines that become redeployed or exaptated for the new purpose: rodents, for example, can recruit substrate-moving behaviours associated with caching or nest building in defensive burying, while California ground squirrels combine substrate throwing with evasive manoeuvres in responses to rattlesnake predation (De Boer & Koolhaas, 2003; Owings & Coss, 1977). Comparable recombination can be shaped by domestication and training, as illustrated by the reorganization of components of the ancestral predatory sequence into specialized herding behaviour in dogs (McConnell, 1990). Again, these observations do not imply that animals ordinarily construct behavioural sequences through unconstrained recombination; rather, they are mentioned here to help establish that previously acquired or evolved behavioural elements can be redeployed outside their usual contexts, providing plausible raw material from which novel functional combinations may be discovered and subsequently stabilized by learning.

Having established the psychological plausibility of these types of behavioural combinations arising in nature, a final question worth briefly addressing before considering possible evolutionary implications is whether they can be genetically assimilated. It might be objected that genetic assimilation requires that selection be capable of modifying the developmental processes underlying particular behavioural components or transitions between them, and that this seems too detailed compared to genes determining variation in more general attributes. To this it may be noted that much contemporary neurodevelopmental genetics suggests that such specificity is biologically plausible. High-diversity recognition systems, including clustered protocadherins in vertebrates and Dscam-based mechanisms in invertebrates, generate large numbers of distinct neuronal identities and connectivity patterns, providing substantial molecular “addressing power” for the specification of neural circuits (Jin & Lee, 2019; Chen & Maniatis, 2013; Wu & Maniatis, 1999; Yue et al., 2016). Consistent with this, perturbing protocadherin or Dscam diversity disrupts dendritic tiling, axonal targeting, and stereotyped connectivity patterns, demonstrating that such molecular variation can have highly specific consequences for neural architecture and behaviour (Wu & Maniatis, 1999; Dong et al., 2025). It is moreover interesting to note that developmental circuit formation and adult learning engage overlapping molecular pathways, including CREB-dependent transcription and neurotrophin signalling, suggesting that mechanisms capable of modifying neural organization operate both during development and in experience-dependent plasticity (Chowdhury et al., 2023; McDowell et al., 2011; Sakamoto et al., 2011).

In sum, these observations do not demonstrate that learned behavioural sequences are genetically assimilated, but they make it biologically plausible that selection could progressively stabilize particular components or transitions when their functional consequences are sufficiently consistent. Together with evidence for behavioural recombination, they suggest that both the generation of novel behavioural combinations and their subsequent evolutionary stabilization are biologically realistic possibilities rather than assumptions peculiar to the model.

1.4. Aims and Premises

For the sort of behavioural combination so far considered, each component can be seen as operating as if a member of a team. Such “complementarity” between simpler behavioural components has been described as a “behavioural hypercycle”. The term “hypercycle” was originally introduced by Eigen and Schuster (1977; 1979) to describe self-reinforcing catalytic networks among molecular replicators. Roy (2017), extended this logic to behavioural and cultural evolution, arguing that behaviours can similarly form and stabilize as autocatalytic systems. This idea can be illustrated by rewriting the above expressions of behaviour to compare with the classic enzymatic hypercycle as in Figure 1 (we have used the $\cap$ symbol here because it captures the requirement that some aspects of the effects caused by each behaviour in the chain must overlap with the perceptual stimuli necessary to stimulate the next). This conceptually shows how learned behavioural chains can, through iterative feedback between performance, learning, and selection, evolve into instinctive sequences.

What makes something a “hypercycle” is not simply that its components form part of a larger, cooperative assemblage, but that the sequence is functionally closed: successful completion of the cycle must feed-back to increase the persistence and recurrence of the cycle itself. This closure changes the selective problem faced by its constituent quasi-species. When components participate in a closed cycle, they need not each perform all the functions required for successful completion, because the fitness consequences of their activity are mediated through the continued operation of the other components. Specialization and division of labour can therefore emerge from a variety of arrangements, without requiring the components to operate in parallel or series. It may therefore be worth keeping in mind the sequential behavioural chain here is just a useful toy model for then considering what logically follows, and it is possible behavioural components may be in parallel, sequence, or mixed. The crucial characteristic is that the loop is closed in a way that it effectively aligns the evolutionary interest of each quasi-species with the joint product of the whole. A medley relay provides an intuitive analogy: the swimmers can specialize in butterfly, backstroke, breaststroke, and freestyle precisely because each swimmer’s contribution is embedded in a recurrent arrangement in which successful performance of the complete sequence sustains the team, even though the strokes are performed sequentially rather than simultaneously.

In a behavioural hypercycle, closure need not mean that the terminal outcome literally generates the initial stimulus. Rather, the terminal outcome can reinforce the behavioural system that produced it, including through secondary reinforcement of the preceding links in the sequence, thereby increasing the probability that the sequence will subsequently be initiated and completed when the relevant conditions recur. Each link in a learned behavioural hypercycle “earns” its reinforcement by contributing to the sequence under distinct appropriate conditions. Thus, the environment may supply the occasion for the initial response (S₁), but the consequences of completing the sequence (O₅) feed back into the future probability of engaging the sequence. In other words, the closed loop is a property of the functional organization of the behavioural system (not to be confused with a claim that the final outcome must physically produce its initial stimulus, O₅→S₁).

One important property of hypercycles discovered by Eigen and Schuster is that they allow more complex entities to stabilize than would otherwise be possible. In our terms, if (1) behaviours, whether separately evolved or acquired, can be combined by learning into higher order combinations to unlock new benefits, and (2) the challenge is recurring, then evolution may lead to the genetic assimilation of the combination, then such arrangements mean that the total amount of information embodied by the hypercycle and replicated across generations may thereby exceed limits on how complicated a behaviour can otherwise evolve (for further description of this “error catastrophe” problem in the case of behaviour see Appendix A).

Another important property is that successful and stable hypercycles are expected to encourage specialization among their constituent components. For reasons eminently explored by economic theory, greater efficiencies may be discovered through dividing up a task into specialists (E.g., Roy 1951). This means that teaming up in this way is at least theoretically likelier to produce more reward (higher fitness payoffs) than the sum of its parts otherwise deployed. This principle of specialization and division of labour naturally leads to each component further adjusting, whether by practice effects, coevolving, or both, to achieve still higher efficiencies. The same process would be further augmented by duplication and subsequent further specialization of behavioural components (although potentially counteracted by “hold up” or “coordination costs”).

When put in these terms, instinct and learning can be expected to have complementary (mutually reinforcing) roles in the evolution of one another, with learning ability facilitating the evolution of more instincts by discovery and subsequent genetic assimilation, which in turn may maintain selection for learning ability. This departs from the traditional view of alternative paths, such as evolving to become better at learning instead of relying on instinct, which treat learning and instinct as “substitutes”. This means that it is by no means clear that an advanced capacity for flexible behaviour and learning should be negatively correlated with the number of instincts in an animal’s repertoire and raises the question of whether they might even positively covary in important ways. To help clarify the conditions under which instincts emerge and derive implications for understanding learning and behaviour, the model developed in the next section translates this qualitative idea into a set of dynamical relationships between behavioural performance and evolutionary change.

2. The Model

We develop a minimal population-genetic model of a multi-step behavioural hypercycle to show how the Baldwin effect can drive genetic assimilation. We consider behaviour as composed of discrete elements or acts that can be linked into chains. Within an individual’s lifetime, reinforcement strengthens associations between acts that lead to reward, thereby increasing the likelihood that successful sequences will be repeated. Across generations, individuals differ genetically in their predispositions to perform or to learn particular elements of these sequences. When learning yields fitness-enhancing outcomes, selection favours alleles that increase the probability or ease of producing those successful sequences, effectively “hardwiring” portions of what was once learned.

The model thus tracks three coupled processes: (1) behavioural performance, determining the immediate success of sequences; (2) associative learning, which modifies the transition probabilities between behavioural elements; and (3) selection, which alters the population distribution of genotypes that bias learning or performance. Together, these processes form a feedback system akin to the behavioural hypercycle depicted in Figure 1, in which learning catalyses the evolution of instincts and inherited predispositions, in turn, guide the formation of new learned behaviours, as illustrated in Figure 2a.

To keep the model simple, think of the hypercycle as a task made up of an ordered sequence of $N$ steps. Completing all steps successfully yields a fitness bonus $B$ — such as better access to food or mates. For clarity, baseline fitness is normalized to 1 and left implicit, meaning that all fitness expressions are measured relative to this baseline. Each step can be either innately encoded or learned during life. We assume one genetic locus per step, with two possible alleles: (1) an innate allele, which hardwires the step, and (2), a non-innate allele, which leaves the step to be learned during an individual’s lifetime. For tractability, we assume that the probability of learning each step is independent of the others. This is a conservative assumption, since we suspect that relaxing it would strengthen the complementarity between steps in the hypercycle and hence increase assimilation pressure.

In this and subsequent sections, we let $q$ represent learning ability, which we model as the probability of successfully learning any given non-innate component of the hypercycle. We assume $q$ (a polygenic or developmental trait) adapts quasi-continuously between rare fixation events of innate steps. All learned steps share this learning probability. Relying on learning comes with two kinds of expenses. There is the per-step learning cost $\mathcal{l}\left(q\right)$, which is independent of outcome and rises with $q$, reflecting energy, risks, time and other resources spent on each learning attempt. The other type of cost comes from physiological maintenance $M\left(q\right)$, such as building brain tissue, which grows convexly with increases in $q$, representing the expense of maintaining neural or cognitive machinery to support learning. For tractability we assume that learning and encoding costs are additive across steps and that learning success probabilities are independent. This deliberately removes interference among components. Such effects would primarily rescale marginal costs or benefits and therefore shift quantitative thresholds without altering the qualitative regimes identified below. If anything, realistic interference in learning would further favour genetic assimilation, making our results conservative.

Obviously, if $m$ of the total $N$ steps are innate, all the remaining ($N-m$) steps must be learned to complete the chain. Thus, the total learning cost is

$$\mathrm{l}\mathrm{e}\mathrm{a}\mathrm{r}\mathrm{n}\mathrm{i}\mathrm{n}\mathrm{g}\mathrm{c}\mathrm{o}\mathrm{s}\mathrm{t}=\mathcal{l}\left(q\right)\left(N-m\right)+M\left(q\right).$$

(1)

To the extent that behavioural steps do not require learning, they avoid this cost. But we will assume even fully innate steps also incur costs of their own, such as a genetic/developmental cost $g$ per innate allele, such that the total cost of an instinctive behaviour is

$$\mathrm{I}\mathrm{n}\mathrm{s}\mathrm{t}\mathrm{i}\mathrm{n}\mathrm{c}\mathrm{t}\mathrm{c}\mathrm{o}\mathrm{s}\mathrm{t}=gm.$$

(2)

An individual succeeds only if every step is either innately encoded or successfully learned. With $m$ innate steps, the probability of producing the complete hypercycle is

$$P\left(\mathrm{complete}\right)=q^{N-m}$$

(3)

The expected fitness of an individual with learning ability $q$ and $m$ innate steps is the baseline fitness for any ordinary individual in the population plus the benefit of successfully “cracking the nut” (i.e., executing the full chain or Good Trick correctly) minus the three types of costs of knowing the Good Trick (learning, neural, and genetic costs), is measured as

$$W\left(q,m\right)=1+B{q}^{N-m}-\mathcal{l}\left(q\right)\left(N-m\right)-M\left(q\right)-gm.$$

(4)

To see how selection acts on learning ability, we take the partial derivative of fitness with respect to $q$ while holding $m$ constant:

$${\displaystyle \frac{\partial W}{\partial q}}=B\left(N-m\right)q^{N-m-1}-\left(N-m\right){\mathcal{l}}^{\mathrm{\text{'}}}\left(q\right)-M^{\mathrm{\text{'}}}\left(q\right).$$

(5)

This captures the balance between the marginal benefit and marginal costs of improved learning ability on fitness. The benefit term is the marginal improvement in the probability of succeeding in producing the hypercycle, scaled by $B(N-m)q^{N-m-1}$. Costs marginally increase in per-step learning $(N-m){\mathcal{l}}^{\mathrm{\text{'}}}\left(q\right)$ and physiological costs $M^{\mathrm{\text{'}}}\left(q\right)$.

What level of learning ability should then be adaptive? We treat learning ability as adjusting to its local fitness optimum between rare assimilation events (a standard separation-of-timescales assumption). A singular point $q^{*}$ then satisfies

$$B\left(N-m\right){\left(q^{*}\right)}^{N-m-1}=\left(N-m\right){\mathcal{l}}^{\mathrm{\text{'}}}\left(q^{*}\right)+M^{\mathrm{\text{'}}}\left(q^{*}\right).$$

(6)

Within this baseline chain model, holding task structure fixed, increasing the number of innately encoded steps $m$reduces the number of components that remain to be learned. This creates a substitution effect because innate encoding increasingly replaces the role otherwise played by learning. Intuitively, there is less work left for learning to do once a larger fraction of the sequence has been genetically specified. Under the cost structures considered here, this tends to lower the adaptive value of maintaining high learning ability.

Now consider genetic assimilation. Suppose the resident population has $m$ innate steps and learning ability $q$. A rare mutant arises that makes one additional step innate, raising the number of innate steps to $m+1$. Learning ability is assumed not to change initially. The resident fitness is

$$W_{\mathrm{r}\mathrm{e}\mathrm{s}}=1+B{q}^{N-m}-\mathcal{l}\left(q\right)\left(N-m\right)-M\left(q\right)-gm,$$

(7)

while the mutant fitness is:

$$W_{\mathrm{m}\mathrm{u}\mathrm{t}}=1+B{q}^{N-\left(m+1\right)}-\mathcal{l}\left(q\right)\left(N-\left(m+1\right)\right)-M\left(q\right)-g\left(m+1\right).$$

(8)

The selection coefficient is the difference:

$$s=W_{\mathrm{m}\mathrm{u}\mathrm{t}}-W_{\mathrm{r}\mathrm{e}\mathrm{s}}$$

(9)

$$s=B\left(q^{N-\left(m+1\right)}-q^{N-m}\right)+\mathcal{l}\left(q\right)-g$$

(10)

Factor the difference:

$$q^{N-(m+1)}-q^{N-m}=q^{N-m-1}\left(1-q\right)$$

(11)

Thus, the selection coefficient simplifies to

$$s=B{q}^{N-m-1}\left(1-q\right)+\mathcal{l}\left(q\right)-g.$$

(12)

This result isolates three forces: (1) the reliability gain (the benefit from the least reliably learned steps becoming innate); (2) the saved learning cost; and (3) genetic cost. A new innate allele spreads when

$$B{q}^{N-m-1}\left(1-q\right)+\mathcal{l}\left(q\right)>g.$$

(13)

The first term is the marginal benefit of improving the success probability ($B{q}^{N-m-1}(1-q)$). This is larger when $q$ is low (learning is difficult), and when many steps are already innate (so the residual learned chain is shorter and $q^{N-m-1}$ is larger). The remaining terms are marginal increases in per-step and maintenance costs: $\mathcal{l}\left(q\right)$ represents per-step learning cost saved and $g$ is the genetic cost of encoding a step innately that the marginal benefits from genetically assimilating the step must overcome. As $m$ increases, the exponent $N-m-1$ decreases, increasing the marginal gain from assimilating additional steps. Because $0<q<1$, the term $q^{N-m-1}$increases as additional steps become innate. Consequently, the selective advantage of assimilating a new step rises with prior assimilation. This generates a positive feedback process: once the number of innate steps exceeds a critical threshold, selection favours further assimilation, causing assimilation to accelerate rather than stabilize. Figure 2b illustrates this effect as an assimilation tipping point, beyond which genetic assimilation becomes self-reinforcing and may proceed in a cascading fashion

This model shows how learning ability and instinct interact in a feedback loop shaped by costs, benefits, and task complexity. When learning is cheap and effective, evolution favours flexible learners; when learning is costly or unreliable, genetic assimilation dominates. Intermediate cases produce mixed strategies. Crucially, the trajectory is path-dependent: early changes in the balance between innate and learned steps can lock populations into very different evolutionary outcomes. By formalizing these dynamics, we gain a clearer picture of when behavioural hypercycles remain plastic and when they crystallize into instinct.

2.2. Heterogeneous Steps and Greedy Assimilation

The preceding model treats behavioural steps as equivalent, with a common probability of successful learning and a common cost of innate encoding. Real behavioural sequences are unlikely to satisfy either assumption. Different components may vary in their difficulty of acquisition, the resources required to learn them, and the costs of specifying them genetically. We therefore extend the model by allowing these quantities to vary across steps.

Behavioural hypercycles work similarly: they require a sequence of heterogeneous steps completed to earn a fitness benefit. The current set of innate steps has size $I$, with the remaining steps learned anew each lifetime. For each step $i$, learning success probability when non-innate is $q_i$ (equal or variable across steps). The learning cost saved when a step is innate is $c_i$, which may depend on investment in the coevolutionary model but here is treated as given. The genetic cost of an innate allele at step $i$ is $g_i$.

If the set of innate steps is $I$, the probability of completing the entire chain is

$$P_I={\prod }_{i\notin I}{q}_i,$$

(14)

then the expected fitness is

$$W_I=1+B\cdot P_I-{\sum }_{i\notin I}{c}_i-{\sum }_{i\in I}{g}_i.$$

(15)

Consider now a rare mutant that makes an additional step $j$ innate (changing the set to $I\cup \left\{j\right\}$), the change in expected fitness is

$$\mathsf{\Delta }{W}_j=B\cdot P_I\cdot \left({\displaystyle \frac{1}{q_j}}-1\right)+c_j-g_j.$$

(16)

The first term is the marginal increase in completion probability, multiplied by the fitness benefit $B$. This depends on the product of non-innate learning probabilities for all remaining non-innate steps except step $j$. This term is larger when $q_j$ is small (the step $j$ is harder to learn). The second term is simply the saved learning cost. The third is the genetic cost that the mutant must overcome to spread. Importantly, genetic assimilation need not preferentially target the most frequently expressed learned behaviours. Rather, selection favours those steps whose conversion to innate form produces the largest increase in expected fitness, which in the present formulation occurs when learning difficulty and/or learning costs are sufficiently high.

Let us now compare two steps that are candidates for genetic assimilation, $i$ and $j$. Step $i$ is favored to be assimilated before step $j$ if

$$\mathsf{\Delta }{W}_i>\mathsf{\Delta }{W}_j$$

(17)

which rearranges to

$$\left({\displaystyle \frac{1}{q_i}}-1\right)-\left({\displaystyle \frac{1}{q_j}}-1\right)>{\displaystyle \frac{\left(c_j-g_j\right)-\left(c_i-g_i\right)}{B\cdot P_I}}$$

(18)

This condition explicitly shows that assimilation order depends on learning difficulty (left side) and differences in net cost (right side), scaled by the current state given by $P_I$ and $B$. That is, steps that are harder to learn and have higher learning costs offer higher marginal gains when made innate and so become assimilated earlier. High genetic costs oppose assimilation, delaying or preventing fixation of the mutant. But as more steps become innate, these change the landscape for the remaining steps, making the relative importance of cost differences become less important. The process of evolving instincts may then snowball. Because each fixation increases $P_I$, the marginal term for the remaining steps rises, potentially converting previously neutral steps into favourable ones.

If assimilation proceeds iteratively along these lines, it may be well approximated by a greedy algorithm:

• Start with no innate steps ($I=\mathrm{\varnothing }$)
• Compute $\mathsf{\Delta }{W}_j$ for currently available assimilation variants
• If all $\mathsf{\Delta }{W}_j\le 0$, stop (no further assimilation is favoured)
• Otherwise, fix the next step with the greatest positive $\mathsf{\Delta }{W}_j$ (assimilate it), updating $I$ and $P_I$
• Repeat step 2 until no steps remain beneficial or all are innate

This process, as depicted in Figure 3, likely accelerates assimilation because each fixation raises $P_I$, increasing the marginal term for remaining steps. The greedy approach, which picks the step with the largest immediate advantage, is biologically plausible and computationally efficient compared to evaluating all subsets, though global optimality is not guaranteed in cases with strong non-linearities or complementarity effects among steps. We note that the greediness is appropriate here given that Natural Selection is inherently “myopic” in that it climbs the steepest local gradient. By treating assimilation as a “greedy” algorithm, we model evolution as a process that prioritizes fixing the most expensive chokepoints in the hypercycle first. We also note that this process is conditional on candidate mutations being simultaneously available. Under mutation-limited evolution, the realised order will also depend on rates of mutational origin and establishment.

A few things may be noted about how our model would change under different assumptions. Firstly, we note that, for simplicity we treat learning difficulty ($q_i$) and genetic encoding costs ($g_i$) as independent across steps. Correlation between these quantities would primarily rescale marginal gains $\mathsf{\Delta }{W}_k$and thus alter the ordering or speed of assimilation rather than the qualitative dynamics. Positive correlation would dampen assimilation by offsetting the benefits of hard-to-learn steps with higher encoding costs, whereas negative correlation would accelerate the snowballing pattern. The greedy, path-dependent structure of the process remains unchanged.

The above greedy algorithm assumes that learning investment remains fixed while assimilation proceeds. However, if learning ability adapts to the changing structure of the behavioural repertoire, the invasion conditions become dynamic. We can extend the algorithm by allowing learning investment to be re-optimized after every successful assimilation. Let fitness be explicitly dependent on learning investment $q$, such that

$$W_I\left(q\right)=1+B\cdot P_I\left(q\right)-{\sum }_{i\notin I}{c}_i\left(q\right)-{\sum }_{i\in I}{g}_i-M\left(q\right)$$

(19)

where $M\left(q\right)$ is the organism-level cost of learning. For any current innate set $I$, the optimal learning investment is $q_I^{*}={\mathrm{argmax}}_q{W}_I\left(q\right)$. Because the optimal learning investment depends on which components have already become innate, the extended algorithm becomes:

• Find the current optimum learning investment: $q_I^{*}$.
• Calculate candidate invasion advantages $\mathsf{\Delta }{W}_j$ using $q_I^{*}$.
• Assimilate the component with the largest positive $\mathsf{\Delta }{W}_j$.
• Update the innate set: $I\to I\cup \left\{j\right\}$.
• Recalculate optimal learning for the new set: $q_{I\cup \left\{j\right\}}^{*}$.
• Repeat until no remaining steps are beneficial.

This dynamic re-optimization introduces an additional channel of feedback into the path of assimilation. In the static case, the trajectory is driven simply by how current instincts alter the baseline probability of success ($I\to P_I\to \mathsf{\Delta }W$). In the dynamic case, the trajectory depends not only on what has already become innate, but also on how learning adapts in response ($I\to q_I^{*}\to P_I\left(q\right)\to \mathsf{\Delta }W$). Allowing learning investment to re-optimize does not overturn the local substitutive relationship between learning and innate encoding. Instead, it adds another feedback to the greedy assimilation process: as more components become innate, optimal investment in learning tends to decline, altering the selective advantage of assimilating the remaining components. However, this remains a within-repertoire substitution process. To obtain the positive association between learning capacity and innate structure that motivates the broader behavioural-hypercycle argument, something else is required: learning must expand or restructure the repertoire itself. That is precisely what the multi-task reuse model introduces.

2.3. Multi-Task Reuse and the Assimilation of Behavioural Primitives

Thus far, the analysis has considered assimilation within a single behavioural sequence. However, behavioural components need not be unique to a single task. Some components may be repeatedly recruited across different behavioural programs. If so, the fitness consequences of assimilating a component should depend not only on its role within a particular sequence, but also on how broadly it is reused.

To capture this idea, suppose that behaviour consists of a collection of tasks $T_1,\dots ,T_J$, where each task is itself a behavioural hypercycle composed of several constituent components. For example,

$$T_1=\{A, B,C,D\}$$

$$T_2=\{A,E,F,G\}$$

$$T_2=\{A,E,F,G\}$$

In this illustration, behavioural component $A$participates in several tasks, whereas the remaining components are task specific. A primitive such as orienting toward a stimulus, grasping, locomoting, or approaching a target may therefore contribute to multiple behavioural programs. We assume that such components are acquired at the level of the repertoire rather than independently within each task. Consequently, if a component participates in multiple tasks, its acquisition cost is incurred once, while the fitness benefits of successful deployment may accrue across every task in which it is used.

Let the probability of successfully completing task $T_j$ be

$$P\left(T_j\right)={\prod }_{i\in T_j}{q}_i$$

(20)

where $q_i$ denotes the probability that component $i$ is successfully acquired or executed. Total fitness is then

$$W\left(I\right)=1+{\displaystyle {\sum }_{j=1}^J}{B}_{j}P\left(T_j\mid I\right)-{\sum }_{i\notin I}{c}_i-{\sum }_{i\in I}{g}_i$$

(21)

where $B_j$ is the fitness value of successfully completing task $T_j$ provided $c_i$ represents the acquisition cost of component $i$across the repertoire and is not separately incurred once per task.

Suppose now that a mutation genetically assimilates behavioural component $A$. The resulting change in fitness is

$$\mathsf{\Delta }{W}_A={\sum }_{j:A\in T_j}{B}_{j}P\left(T_j\mid I\right)\left({\displaystyle \frac{1}{q_A}}-1\right)+c_A-g_A$$

(22)

where $c_A$ is the learning cost saved by assimilating component $A$, $g_A$ is the genetic cost of encoding it, and the summation extends over every task containing that component. Unlike the single-task case, the benefit of assimilation now accumulates across all tasks in which the component participates. If a primitive is redeployed $r$ times across the behavioural repertoire, then,

$$\mathsf{\Delta }{W}_i\propto r_i$$

(23)

if tasks have equal fitness benefits and equal completion probabilities conditional on component $i$, the marginal benefit of assimilating $\mathrm{i}$increases linearly with the number $r_i$ of tasks in which it occurs. Selection should therefore favour assimilation not only of components that are difficult or costly to learn (§2.2), but also of components that make repeated contributions across many behavioural contexts. Repeated deployment effectively amplifies the fitness return obtained from converting a learned component into an innate one.

This extension generates a simple prediction. The model predicts that, all else equal, behavioural primitives that contribute to many tasks should be more strongly canalized than otherwise equivalent components used in fewer tasks. Locomotor routines, orienting responses, grasping movements, pursuit behaviours, or other widely deployed behavioural building blocks may therefore be favoured targets of genetic assimilation, whereas context-dependent combinations and task-specific contingencies are more likely to remain plastic. More generally, selection should preferentially stabilize the reusable “building blocks” from which behavioural repertoires are assembled.

Multi-task reuse therefore provides a second route by which learning can generate selection for innate behavioural structure. Learning can discover task-specific solutions, while repeated use makes some of their constituent primitives especially valuable candidates for assimilation. This raises a further question. If learning itself is subject to evolutionary change, assimilation need not occur against a fixed learning capacity. Instead, changes in learning capacity and innate encoding may alter each other’s subsequent fitness consequences. We next consider their joint evolution.

2.4. Coevolution of Learning and Innate Encoding

The analyses above have treated learning ability as fixed. This simplification allows the consequences of heterogeneous learning difficulty, genetic assimilation, and behavioural reuse to be considered in isolation, but the central argument of the behavioural hypercycle concerns the coevolution of learning and innate behavioural structure. We therefore now allow learning ability itself to vary.

Let $q$ denote population-level learning ability and let $m$ denote the number of behavioural components that are genetically encoded. Consider a repertoire containing $L$ components per behavioural task. If $m$ components are innate, the remaining $L-m$ components must be learned. As in the preceding models, the probability of successfully completing a task is therefore $P\left(q,m\right)=q^{L-m}$. The important extension is that learning ability may influence not only the probability of successfully acquiring the components of a particular task, but also the size of the behavioural repertoire that can be acquired. Let $J\left(q\right)$ denote the number, or expected number, of behavioural tasks available to a lineage as a function of its learning ability, with $J^{\text{'}}\left(q\right)>0$. This captures the possibility that greater learning ability permits the acquisition of a larger repertoire of behavioural programs. Assuming for simplicity that the fitness contributions of individual tasks are additive, total fitness can then be written as

$$W\left(q,m\right)=BJ\left(q\right)q^{L-m}-\left(L-m\right)c\left(q\right)-mg$$

(24)

where $B$ is the fitness benefit of successfully completing a task and $g$ is the genetic cost of encoding each innate component. Here $c\left(q\right)$ represents the repertoire-level acquisition cost per distinct learned component; because components may be shared across tasks, this cost is not multiplied by the number of tasks. The expression is therefore a simplifying aggregate representation rather than a count of task-specific learning episodes.

This formulation clarifies an important distinction. Within a fixed behavioural repertoire, innate encoding and learning remain alternative means of acquiring the same behavioural components. Increasing $m$ therefore reduces the number of components for which learning is required, producing the substitution between learning and innate encoding described in §2.1. However, this need not exhaust their evolutionary relationship if learning ability also changes the size and structure of the behavioural repertoire.

The reuse model in §2.3 provides the link. Consider a behavioural primitive $A$ that occurs in multiple tasks. Let

$$R_A={\sum }_{j:A\in T_j}{B}_j$$

(25)

denote the fitness-weighted reuse of $A$, where the summation extends across all tasks containing that component. If all tasks have equal fitness value $B$ and $A$ occurs in $r$ tasks, then $R_A=rB$. If $A$ is learned with probability $q$, and each task contains $L$ components, successful completion of a task containing $A$ has probability $q^L$ when $A$ remains learned. Genetically assimilating $A$ increases this probability to $q^{L-1}$. The resulting change in fitness across all tasks that use $A$ is therefore

$$\mathsf{\Delta }{W}_A=R_A\left(1-q\right)q^{L-1}+c_A\left(q\right)-g_A$$

(26)

where $c_A\left(q\right)$ is the learning cost saved by assimilating $A$, and$g_A$ is its genetic encoding cost. This equation makes explicit why reuse favours genetic assimilation. The fitness benefit of assimilating $A$ is multiplied by the fitness-weighted number of behavioural contexts in which $A$ contributes. Thus, all else equal,

$$\mathsf{\Delta }{W}_i\propto R_A$$

(27)

A primitive that is repeatedly recruited across the behavioural repertoire can therefore provide a greater return when genetically encoded than an otherwise equivalent component used in only one task.

The possibility of coevolution arises if the extent of reuse itself depends on learning ability. A lineage with greater learning ability may acquire a larger repertoire of behavioural programs, thereby increasing the number of contexts in which reusable primitives contribute. We can therefore write

$$R_A=R_A\left(q\right), R_A^{\text{'}}>0$$

The marginal advantage of assimilating $A$ then becomes

$$\mathsf{\Delta }{W}_A\left(q\right)=R_A\left(q\right)(1-q)q^{L-1}+c_A\left(q\right)-g_A$$

(28)

The effect of learning ability on the selective advantage of assimilation is given by

$${\displaystyle \frac{d\mathsf{\Delta }{W}_A}{dq}}=q^{L-2}\left[q\left(1-q\right)R_A^{\text{'}}\left(q\right)+R_A\left(q\right)(L-1-Lq)\right]+c_A^{\text{'}}\left(q\right).$$

(29)

This expression separates two opposing effects. The first,

$$q(1-q)R_A^{\text{'}}\left(q\right)$$

(30)

is a repertoire-expansion effect. When greater learning ability increases the fitness-weighted reuse of a primitive, $R_A^{\text{'}}\left(q\right)>0$, this term is positive: greater learning ability increases the number or value of behavioural contexts in which assimilating the primitive is beneficial. The second term,

$$R_A\left(q\right)\left(\left(L-1\right)-Lq\right)$$

(29)

captures the conventional within-task effect. Thus, increasing learning ability does not invariably reduce assimilation advantage even within the fixed task. The reliability benefit of assimilation is low when learning is extremely poor because the rest of the sequence is rarely completed; it can initially rise with $q$, then fall once learning becomes sufficiently reliable. This term therefore tends to oppose assimilation when learning ability is high.

The overall relationship between learning and assimilation is consequently not fixed. When the repertoire-expansion effect is sufficiently large, increasing learning ability increases the selective advantage of assimilating a reusable behavioural primitive ${\displaystyle \frac{d\mathsf{\Delta }{W}_A}{dq}}>0$. Thus, even when increasing learning ability reduces the marginal benefit of innate encoding within an individual task, assimilation can nevertheless become increasingly favoured if greater learning ability sufficiently increases the fitness-weighted reuse of the component across the behavioural repertoire. The critical factor is therefore not learning ability alone, but the extent to which increased learning capacity expands the range of contexts in which reusable behavioural primitives contribute to fitness. For example, if learning ability has little effect on repertoire size, the within-task substitution effect will dominate; if learning ability substantially expands the repertoire and hence the reuse of a primitive, the repertoire-level effect can instead dominate. A locomotor or orienting primitive illustrates the logic: a component that is useful in only one behavioural program has limited scope for reuse, whereas a primitive deployed across foraging, predator avoidance, social interaction, and navigation can contribute to fitness across many contexts. If greater learning ability permits a lineage to acquire a broader repertoire of such programs, the selective value of reliably encoding the shared primitive can increase with learning capacity. Conversely, when repertoire expansion is weak, the within-task substitution effect may dominate, giving ${\displaystyle \frac{d\mathsf{\Delta }{W}_A}{dq}}<0$.

This provides a formal mechanism for the behavioural hypercycle proposed here. Learning can expand the range of behavioural programs that a lineage can acquire, increasing the number of contexts in which reusable behavioural primitives contribute. Repeated use in turn increases the fitness return from genetically assimilating those primitives. Once assimilated, these components reduce the learning burden associated with the expanded repertoire, potentially increasing the returns to maintaining or increasing learning ability. The resulting feedback can therefore be represented schematically as

$$q\to repertoire expansion\to reuse\to k\to reduced learning burden\to greater returns to learning.$$

The model does not imply that learning and innate encoding are necessarily complementary in all circumstances. Rather, it identifies the conditions under which complementarity can emerge from the interaction between learning, repertoire expansion, and reuse. The behavioural hypercycle therefore arises not simply because learned and innate behaviours coexist, but because learning can generate an expanding repertoire in which some behavioural primitives acquire repeated fitness value, creating selection for their assimilation and, in turn, altering the evolutionary value of learning itself. The present extension derives how learning ability changes selection for assimilation. The reciprocal effect of assimilation on selection for learning depends on how released acquisition costs or cognitive resources are reallocated across the repertoire and is therefore treated here as predicted feedback rather than a fully specified dynamic.

3. Discussion

The present model predicts that genetic assimilation of complex, hypercyclic behaviours should operate selectively rather than evenly. On this view, selection does not assimilate entire behavioural repertoires at once, nor does it simply favour the most frequently expressed behaviours. Instead, assimilation pressure is predicated on the fitness consequences of converting particular behavioural components from learned to innate form. Components that are difficult or costly to learn, or that disproportionately limit successful task completion, experience the strongest selection for genetic assimilation. The resulting process resembles a greedy algorithm, with evolution progressively targeting behavioural bottlenecks rather than entire behavioural sequences. We also found that behavioural reuse amplifies this process. Components repeatedly deployed across many contexts generate larger fitness returns when assimilated and thus experience disproportionately strong selection for canalization. In short, reuse concentrates selection on the behavioural components that matter most.

This evolutionary dynamic mirrors a process familiar from individual learning. During skill acquisition, behaviours that are initially effortful and attention-demanding often become increasingly automatic through repeated execution. Motor-sequence learning, habit formation, and the chunking of behavioural routines are all associated with a gradual shift from flexible, deliberative control toward more efficient and stereotyped execution. In vertebrates, such automation has frequently been linked to the progressive concatenation and consolidation of resulting behavioural sequences within corticostriatal circuits (Dezfouli & Balleine, 2013). The present model suggests that an analogous process may occur over evolutionary timescales. The key point is that not all behavioural components are recruited across tasks equally. Components that repeatedly contribute to successful behavioural outcomes experience disproportionate selection for genetic assimilation, progressively shifting control from learning-dependent acquisition toward inherited behavioural organization. In both cases, repeated reuse concentrates selection or reinforcement on the same behavioural components, progressively reducing the need to reconstruct them from scratch.

Importantly, neither automatization nor genetic assimilation is predicted to occur uniformly across all levels of behavioural organization. The reuse model instead predicts differential stabilization of behavioural structure. Behavioural primitives reused across many tasks experience the strongest assimilation pressure, whereas task-specific combinations and context-dependent linkages remain comparatively plastic. This creates a hierarchical organization in which lower-level behavioural building blocks become increasingly canalized while higher-level control remains flexible. Evolution need not replace learning with instinct. Rather, instinct increasingly provides an infrastructure upon which learning operates. In this sense also, the reuse mechanism provides a route by which the specialization and division-of-labour logic motivating the behavioural hypercycle may emerge over evolutionary time.

Extending the model to allow learning investment to adapt also revealed an additional feedback path between assimilation and learning investment. While this alone is insufficient to generate positive complementarity between learning and instinct, the multi-task reuse model identifies repertoire expansion and behavioural reuse as mechanisms capable of transforming a local substitutive relationship into a globally complementary one. This may help clarify a long-standing tension in discussions of the Baldwin effect. Traditional accounts often treat learning and instinct as alternative routes to adaptation (Maynard Smith, 1987; Stephens, 1991). Under this view, the evolution of an innate solution reduces the need for learning, creating a largely zero-sum relationship between the two (Hinton & Nowlan, 1987; Stephens, 1991). The model developed here confirms that this intuition is locally correct. Within a fixed behavioural repertoire, learning and instinct behave primarily as substitutes. As additional components become innate, fewer behaviours remain to be learned and the adaptive value of investing in learning correspondingly declines. However, this substitutive relationship does not necessarily extend to the level of entire behavioural repertoires. Once behavioural learning expands the repertoire in ways that create opportunities for component reuse, the relationship changes qualitatively. Learning discovers novel behavioural arrangements; instinct progressively stabilizes the recurrent building blocks that prove most valuable. Each round of learning therefore creates new opportunities for assimilation, while each round of assimilation creates conditions that support further learning. The behavioural hypercycle evolves from this reciprocal process.

3.1. Testable Predictions

The model generates predictions that distinguish the proposed hypercycle account from the simple view that learning and instinct are competing alternatives. First, genetic assimilation should be selective, targeting behavioural components that disproportionately constrain successful completion. The greedy algorithm predicts that assimilation targets components whose failure most strongly limits successful completion, regardless of whether they are the most frequently expressed behaviours. A rare but critical component, such as an emergency evasion routine, can therefore experience immense assimilation pressure due to its outsized fitness consequences. This could be tested via experimental evolution in rapidly reproducing model organisms (e.g., Drosophila). By constructing tasks with multiple learned components that differ independently in acquisition difficulty, and imposing selection for successful completion over generations, researchers could track whether selection preferentially stabilizes the most limiting elements first.

Second, assimilation should proceed in a path-dependent and cascading manner. Because every successful assimilation event alters the adaptive landscape experienced by remaining behavioural components, early fixation events influence subsequent assimilation opportunities. This implies that lineages may reach different behavioural architectures despite facing similar ecological challenges. Small historical differences in which behavioural components become canalized early may be amplified over evolutionary time, predicting substantial historical contingency in behavioural evolution among closely related species occupying broadly similar ecological niches.

Third, and perhaps most distinctively contrary to simple trade-off models, learning ability and innate behavioural structure are not necessarily negatively correlated. The multi-task extension predicts that functional reuse should amplify the likelihood of assimilation. A behavioural primitive that contributes to several tasks accumulates fitness benefits whenever its reliable performance improves any of those tasks. Reuse and expression frequency need not be the same thing: a component may appear in many different tasks even if no individual task is performed especially often. The prediction is therefore not simply that frequently expressed behaviours become innate, but that components with greater functional reuse should experience stronger selection for assimilation. Empirically, components that occur across many behavioural contexts may consequently be expected to exhibit greater developmental stereotypy and phylogenetic conservation than otherwise comparable components restricted to a small number of ecological situations. More directly, experimental evolution could manipulate the extent to which particular components are reused across tasks and test whether repeated recruitment accelerates their subsequent canalization. The same logic predicts that environmental instability should disproportionately suppress assimilation of task-specific combinations while leaving assimilation of widely reusable primitives comparatively intact.

Finally, the model predicts that learning and innate structure can coexist within a hierarchical behavioural repertoire. More generally, relationships between learning capacity and innate behavioural structure need not be uniformly negative: positive learning-instinct covariance should emerge specifically in systems where learning expands reuse opportunities. This prediction differs from both purely instinctive and purely learning-based accounts. Within a fixed task, learning and innate encoding are substitutes: once a component is reliably encoded, the marginal value of learning that same component declines. Across a repertoire of partially overlapping tasks, however, reusable innate components can complement learning by providing reliable building blocks from which novel combinations can be acquired. The resulting prediction is, then, not a simple positive or negative relationship between learning capacity and innate behaviour, but a structured one: species or populations with greater learning capacity should be capable of supporting larger and more combinatorially diverse behavioural repertoires when reusable components are available, while the components that are repeatedly recruited across those repertoires should show greater canalization. This could be tested by comparing behavioural repertoires across species or populations that differ in learning capacity while separately quantifying the degree of reuse and developmental stereotypy of their constituent components.

Taken together, these predictions suggest that the principal evolutionary consequence of learning is not merely the acquisition of behavioural flexibility, but the generation of future opportunities for genetic assimilation. Learning discovers novel behavioural arrangements, reuse identifies which components are repeatedly valuable, and selection progressively stabilizes those components.

3.2. Limitations

While our model relies on several simplifying assumptions that invite further refinement. Firstly, we assumed independence of components. In §2.1, we modelled the success probability of each step as independent. In nature, the acquisition of behavioural steps may often be synergistic such as when learning one step might make another easier to acquire. Empirical studies of behavioural repertoires seem to show that behaviours covary in modular fashion (Werkhoven et al., 2021), presenting a challenge to evolutionary models that assume independent traits (Wagner & Altenberg, 1996). However, we note that, if such dependencies are positive, incorporating them could amplify the ‘snowball effect’ identified in §2.2, because assimilating one bottleneck could increase the reliability and thus the value of other components.

Secondly, the model currently treats learning difficulty ($q_i$) and genetic encoding costs ($g_i$) as separable dimensions. As noted, it is biologically plausible that behaviours which are highly complex to learn could also be difficult and costly to encode genetically. While our account accommodates a positive covariance between these variables (where assimilation pressure reflects the balance between the difficulty-benefit and the encoding-cost penalty), future quantitative work could explore how severe pleiotropic costs might slow the speed of assimilation cascades. It should nevertheless be noted that, although it may seem intuitive that difficult-to-learn behaviours should also be costly to encode genetically, this relationship need not be universal. For instance, learning difficulty may reflect challenges of environmental discovery, exploration, or coordination that are only weakly related to the complexity of the resulting behavioural solution, which itself might be quite elegantly encoded genetically.

Thirdly, our use of a greedy algorithm to approximate the order of assimilation assumes that candidate mutations for all components are simultaneously available. In natural populations, behavioural evolution is often mutation-limited. The realized evolutionary trajectory will therefore depend not only on the steepest marginal fitness gradients calculated here, but also on mutation rates, standing genetic variation, and population size. Furthermore, while our model tracks selection on learning ability, it does not attempt to derive the specific neural architectures or generative developmental mechanisms that instantiate learning itself. Likewise, although the re-optimizing extension allows learning investment to adapt following assimilation events, the reciprocal evolutionary dynamics by which assimilation reshapes long-term selection on learning remain only partially explored. Future work could model these types of feedback jointly. In other words, the model is best viewed as a generative theoretical framework, rather than a source of precise quantitative predictions.

Finally, an important limitation concerns environmental stability. The present models implicitly assume that selection acts across a sufficiently consistent set of behavioural challenges for genetic assimilation to generate long-term benefits. Rapid or unpredictable environmental change from generation to generation is therefore expected to weaken assimilation pressures by reducing the value of canalizing behaviours that may soon become obsolete. In particular, environmental fluctuation should reduce selection for the assimilation of task-specific linkages and context-dependent behavioural combinations, thereby favouring continued investment in learning ability and behavioural plasticity (Stephens, 1991; West-Eberhard, 2003). In short, environmental instability does not remove the value of assimilation; it changes which components are worth assimilating.

That said, environmental instability should not affect all behavioural components equally. Components that are repeatedly reused across a wide range of tasks have a value that extends beyond any particular environmental context. Locomotor routines, orienting responses, grasping movements, perceptual transformations, and other widely deployed behavioural primitives may consequently remain attractive targets for assimilation even when higher-order behavioural strategies continue to change. Consequently, environmental instability need not eliminate the coexistence of instinct and learning predicted by the model. Indeed, it may strengthen the tendency toward hierarchical organization, with broadly useful behavioural building blocks becoming increasingly canalized while more context-specific combinations remain flexible. This distinction suggests that environmental stability primarily influences which levels of behavioural organization become assimilated rather than determining whether assimilation occurs at all. Stable environments may favour the progressive canalization of increasingly elaborate behavioural sequences, whereas more variable environments constrain assimilation to highly reusable primitives, leaving learning responsible for assembling and adapting those components to local conditions.

3.3. Future Directions

The present analysis considers behavioural hypercycles and associative conditioning within a single organism, but the logic of hierarchical specialization may extend to other forms of behavioural organization. The distributed organization of behaviour in cephalopods provides a useful analogy, where relatively autonomous components of the motor system can contribute to higher-level behavioural control. More generally, a hypercycle may permit behavioural components to specialize in particular functions without requiring each component to remain capable of performing the entire behavioural repertoire. The resulting architecture resembles a division of labour: recurrently useful low-level components can become increasingly specialized and canalized, while higher-level processes retain the flexibility to assemble them into novel combinations. Whether analogous forms of hierarchical specialization arise through genetic assimilation in other animals is an empirical question that could be investigated by examining the relationship between behavioural reuse, developmental stereotypy, and flexibility across levels of behavioural organization.

The core logic of our model may also extend beyond the individual. In social species, the components and transitions of a behavioural hypercycle need not be instantiated entirely within a single body: different individuals may contribute complementary behavioural components to a collectively organized sequence. As shown in Appendix B, such collective dynamics can be formally related to Becker’s (1974) “Rotten Kid Theorem,” in which resource redistribution within a family closes the feedback loop between individual behaviour and collective fitness. This points to a broader possibility in which social interaction provides additional substrates for hypercycle organization, with some components of coordinated behaviour becoming increasingly canalized while others remain responsive to social and environmental variation. Whether genetic assimilation can operate on such multi-agent behavioural systems, and how genetic, learned, and socially transmitted components interact within them, remain open questions.

A further extension concerns mechanisms that preserve learned solutions across generations. The present model assumes that behavioural variants are repeatedly rediscovered through individual learning. Social and cultural transmission provide an alternative possibility: learned solutions may persist beyond the lifetime of the individuals who originally acquire them. In effect, culture can extend the temporal persistence of behavioural innovations, increasing the opportunity for selection to favour genetic variants that reduce their dependence on learning. Rather than repeatedly reconstructing behavioural solutions from scratch, populations inherit partially stabilized behavioural traditions that can persist long enough for genetic assimilation to operate. Importantly, this does not require cultural variants to be perfectly adaptive. Even imperfectly adaptive traditions may persist culturally. For adaptive variants, however, culture acts as an evolutionary bridge — holding behaviour stable long enough for genetic assimilation to catch up.

4. Conclusions

The evolution of complex instinctive behaviour has long been framed as a zero-sum contest between the reliability of genetic inheritance and the flexibility of learning. The analysis developed here demonstrates that this dichotomy is incomplete. By modelling the genetic assimilation of behavioural hypercycles as a dynamic, component-by-component process, we propose that selection does not target entire behavioural sequences uniformly but acts more like a greedy algorithm, preferentially canalizing the most difficult, costly, or constraining behavioural bottlenecks within its orbit. Crucially, when behavioural components are reused across multiple tasks, the relationship between learning and instinct shifts from substitution to complementarity. While they remain local substitutes within any single task, learning and innate encoding become globally complementary across an expanding behavioural repertoire. Learning acts as an engine of discovery, assembling novel behavioural combinations and exposing reusable primitives to selection. In turn, genetic assimilation progressively stabilizes these recurring building blocks, effectively reducing the cognitive burden of the repertoire and generating new opportunities for learning. The resulting behavioural hypercycle is predicted to favour neither wholly learned nor wholly innate behavioural repertoires, but hierarchical behavioural architectures in which stabilized behavioural building blocks create opportunities for the continued discovery of novel combinations. Explaining complex instinct may mean moving beyond simple evolutionary trade-offs, recognizing instead that learning and instinct can act as mutual catalysts in the assembly of behaviour.