Emergence of Memory and Program via Functional Differentiation in Evolutionary Echo State Networks: Toward an Understanding of the Origin of Consciousness by Optimization

1. Introduction

The human brain consists of multiple functional modules, each composed of various types of neurons that carry out a single or multiple functions. The hierarchical architecture of these structures is organized both biological evolution and ontogenetic development, such as those represented in Brodman’s functional map [1]. Thes structures reflect typical processes of functional differentiation. Moreover, recent studies have revealed that subareas within a functional region dynamically cooperate depending on specific tasks―a phenomenon called a functional parcellation [2]. Understanding the neural mechanisms underlying these differentiations is a central challenge in cognitive science and neuroscience, as it may provide insight into the neural basis of consciousness. Nevertheless, research in this direction has not sufficiently progressed. We have previously addressed this issue within the framework of self-organization with constraints, which can be formulated via a variational principle [3]. Using this framework, we have proposed mathematical models that demonstrate the emergence of functional units, such as neurons and neural modules [4,5,6].

In this context, we here propose a new model of functional differentiation that enables the simultaneous emergence of “memory” and “program”, albeit within a specific and limited setting. The model also serves as a mathematical proposal for consciousness grounded in Kolmogorov complexity [7,8]. Kolmogorov complexity measures the minimal number of bits required to describe or compute an object. For example, a period-two periodic sequence consisting of 0 or 1, {010101$\cdots$} of length $2n$ can be represented as “Repeat the sequence ‘01’ $n$ times” with the Kolmogorov complexity on the order of $\log n$. In contrast, a random binary sequence of the same length has the complexity on the order of $n$, since it cannot be compressed. Accordingly, randomness corresponds to maximal complexity, including fully developed chaos. Crutchfield’s $\epsilon$-machine theory formalizes these ideas by classifying temporal patterns in terms of computational structure [9]. We extend this viewpoint by considering observers―biological or computational―as information processing systems. A correct observation requires maximal information gain under minimal energetic cost in generating internal descriptions, where energy consumption is assumed proportional to the Kolmogorov complexity of those descriptions.

Historically, consciousness has often been associated with inference. For instance, Goerge Boole regarded consciousness as arising from inferential sequences represented by binary symbols [10]. Charles Peirce likewise characterized consciousness as an inferential process [11]. These pioneering ideas establish an early link between consciousness, logic, and Kolmogorov complexity.

We assume that memory and program are essential components of conscious activity. The model we propose generates both memory and program structures through an optimization process that naturally defines Kolmogorov complexity. For this reason, the model may illustrate fundamental mechanisms for generating conscious-like processes.

In Section 2, we introduce an adaptable neural network model that extends the echo state network (ESN) [12,13,14] through evolutionary modification of internal connections using a genetic algorithm, yielding what we call the evolutionary ESN (eESN) [4,15,16]. The system is trained on two distinct tasks―memory and program execution―, using a random input time series that represents a highly complex environment. In section 3, we present computation results that illustrate the emergence of the optimized internal structures to yield the memory and program units. Section 4 is devoted to discussion.

2. Materials and Methods

2.1. Model and Network Structure

We consider an eESN consisting of a recurrent network with $\mathit{N}$ nodes. Let $\mathit{x}\left(\mathit{t}\right)=({\mathit{x}}_1\left(\mathit{t}\right),{\mathit{x}}_2\left(\mathit{t}\right),\cdots ,{\mathit{x}}_{\mathit{N}}\left(\mathit{t}\right){)}^{\mathit{T}}\in {\mathbb{R}}^{\mathit{N}}$ denote the state vector of network at discrete time $\mathit{t}\in \mathbb{N}$. Here, a superscript $\mathit{T}$ indicates a transpose. The network receives a scalar random input $\mathit{u}\left(\mathit{t}\right)$, drawn from a uniform distribution over [0, 1], representing a maximal Kolmogorov complexity signal. The network dynamics is specified by

$$x\left(t+1\right)=\left(1-\alpha \right)x\left(t\right)+\alpha \psi (x\left(t\right);u(t\left)\right),$$

(1)

where $\mathit{\psi }(\mathit{x}\left(\mathit{t}\right);\mathit{u}(\mathit{t})$) is a sigmoid function described by

$$\widetilde{\mathrm{t}\mathrm{a}\mathrm{n}\mathrm{h}}\left(\mathrm{Wx}\right(\mathrm{t})+W_{in}u(t),$$

(2)

where $\mathit{W}\in {\mathbb{R}}^{\mathit{N}\times \mathit{N}}$ denotes a $\mathit{N}\times \mathit{N}$ matrix of internal connections in the recurrent neural network and ${\mathit{W}}_{\mathit{i}\mathit{n}}\in {\mathbb{R}}^{\mathit{N}}$ denotes a vector whose elements represent the connections from input to internal neuronal units. The symbol $\widetilde{\mathbf{t}\mathbf{a}\mathbf{n}\mathbf{h}}$ returns the respective value of tanh for each element. Each element of the internal matrix $\mathit{W}$ takes one of the three values $\mathit{\rho },-\mathit{\rho },\mathbf{a}\mathbf{n}\mathbf{d}0.$ This discrete structure allows for faster convergence and clearer interpretation of structural changes during evolution than those of continuous one. The internal network has no self-connections, that is, the diagonal elements of $\mathit{W}$ are zero. We treated the case of leak rate $\mathit{\alpha }=1,$ since this produced an optimal behavior. The output layer consists of two units, defined by

$$y\left(t\right)=W_{out}x\left(t\right)$$

(3)

with ${\mathit{W}}_{\mathit{o}\mathit{u}\mathit{t}}\in {\mathbb{R}}^{2\times \mathit{N}}$ trained by linear regression. The network architecture is shown in Figure 1.

2.2. Two Tasks Required for Consciousness

The eESN is trained on two distinct tasks: program task (logistic map prediction) and memory task (delayed logistic state recall). For the program task, the network learns to reproduce the logistic map for the input $\mathit{u}\left(\mathit{t}\right)$,

$$d_1\left(t\right)\equiv au\left(t\right)(1-u\left(t\right)),$$

(4)

where $\mathit{a}$ is a bifurcation parameter. The memory task requires reconstructing the logistic map state of the $\mathit{\tau }$-steps in the past,

$$d_2\left(t\right)\equiv u\left(t-\tau \right).$$

(5)

Together, these tasks represent fundamental components of algorithmic inference: rule extraction (program) and information retention (memory).

2.3. Learning and Evolution

The training data consist of 1,000-step random input sequence, of which the initial 100 steps are discarded to remove transient. This length order is sufficient to guarantee a stationary random series―the case of $\mathit{a}~4.0$ yields independent-and-identical distributions. In the simulation, we used $\mathit{a}=3.98.$ The target is described by $\mathit{d}\left(\mathit{t}\right)=({\mathit{d}}_1\left(\mathit{t}\right),{\mathit{d}}_2\left(\mathit{t}\right){)}^{\mathit{T}}.$ The network is trained to achieve the least mean square error (MSE):

$$\underset{\left\{W_{\mathit{out}}\right\}}{\min }\sum _t{‖d\left(t\right)-W_{out}x\left(t\right)‖}^2.$$

(6)

The training data set and the target data set are provided by $\mathit{X}=\left(\mathit{x}\left(1\right),\mathit{x}\left(2\right),\cdots ,\mathit{x}\left(\mathit{L}\right)\right)\in {\mathbb{R}}^{\mathit{N}\times \mathit{L}}$ and $\mathit{D}=(\mathit{d}\left(1\right),\mathit{d}\left(2\right),\cdots ,\mathit{d}\left(\mathit{L}\right))\in {\mathbb{R}}^{2\times \mathit{L}},$ respectively. Then, the optimum solution can be obtained by ${\mathit{W}}_{\mathit{o}\mathit{u}\mathit{t}}=\mathit{D}{\mathit{X}}^{\mathit{T}}(\mathit{X}{\mathit{X}}^{\mathit{T}}{)}^{†},$ where $†$ indicates Moore-Penrose inverse matrix. For the memory task, each target series is generated for respective delay-time $\mathit{\tau }=0\sim 20$ and a total MSE was calculated for all target series.

For evolutionary optimization of $\mathit{W}$ and ${\mathit{W}}_{\mathit{i}\mathit{n}}$, we use the CHC genetic algorithm, which combines elitist selection, heterogeneous recombination, and cataclysmic mutation [5,17]. The fitness function is given by

$$-\sum _{d=0}^{20}\left({MSE}_1\left(d\right)+{MSE}_2\left(d\right)\right)-\lambda E_W,$$

(7)

where ${\mathit{M}\mathit{S}\mathit{E}}_{\mathit{i}}(\mathit{i}=\mathrm{1,2})$ are MSEs for the two tasks. ${\mathit{E}}_{\mathit{W}}$ is the number of nonzero internal connections, and $\mathit{\lambda }=$0.001 is a sparsity penalty, whose value was empirically determined.

3. Results

Through supervised learning and evolutionary optimization, the network differentiated into two fundamental subnetworks: one responsible for memory retention and one for implementing the logistic-map rule. This differentiation is accompanied by a phase-transition-like shift from dense random connectivity to sparse and functionally organized structure (Figure 2). Internal structure looks still random, but output units differentiated into memory and program units, respectively.

To highlight the advantage of eESN, we examined conventional ESNs, whose performance depends strongly on the spectral radius $r$. The program task achieved optimal accuracy in $r<0.6$, whereas the memory task requires $r~0.7$, where quasi-chaotic behaviors emerge. As a result, two tasks were incompatible in a single fixed reservoir (Figure 3). The eESN overcomes this limitation by evolving structural specialization.

.

4. Discussion

As a first step toward a mathematical treatment of consciousness grounded in Kolmogorov complexity, we proposed a minimal model capable of generating memory and program units simultaneously via functional differentiation. Using an evolutionary ESN, we demonstrated that these two subsystems cannot be simultaneously optimized in a conventional ESN but can emerge naturally when evolutionary structural adaptation is introduced. The evolutionary process induces a phase transition. Joint minimization of output error, structural redundancy, and connection sparsity―interpreted as minimizing energy consumption―leads to an overall reduction of Kolmogorov complexity of the internal description of memory and program. This suggests that the system reaches an optimal complexity state capable of functioning as an algorithmic descriptor of an unpredictable environment. This stage of organized functionality corresponds to what we call algorithmic consciousness: a primitive form of consciousness characterized by inference, in line with historical accounts identifying consciousness with inferential processes [1,2]. Higher-order consciousness, including self-referential abilities, likely requires additional mechanisms such as social interaction. Such capabilities are not present in the current model. Nevertheless, the results demonstrate how key functional components of consciousness―memory and program―may emerge within a unified computational system governed by optimization and evolutionary constraints.