Decoding Symbols, Computing Signs by Info-Computational Processes in Cognitive Agents

1. Introduction: Semiotics as a Multidisciplinary Field

Semiotics, the study of sign processes and meaning-making, offers profound insights into the communication, culture, and cognition of humans and even other living beings (biosemiotics). (Hoffmeyer 2013) claims that semiosis is concomitant with life, which is tantamount to cognition, as (Stewart 1996) and (Maturana and Varela 1980) show.

Computational approaches explore how information is processed by physical systems, often using computational technology and analogies to machine computation. These approaches have revolutionized our understanding of cognition, artificial intelligence, biology, communication, and more, through pragmatic relationships involving experimentation, implementation, testing, and comparison to the natural world in a two-way learning process, (Rozenberg and Kari 2008).

Historically Semiotics focused on humans and the macroscopic level of description, based on the distinction between humans and nature. This goes back to Plato and Aristotle who explored the relationship between signs and the world, (Chandler, 2022). Even later semiotics work of Augustine of Hippo, John Locke, and Ferdinand de Saussure, focused on humans.

Charles Sanders Peirce defined semiotics as a 'formal doctrine of signs,' which abstracts 'what must be the characters of all signs used by an intelligence capable of learning by experience' (Hartshorne and Weiss 1932: 227). Today, Peirce’s perspective would be interpreted as extending beyond humans to encompass all living organisms, including single living cells. Moreover, Peircean semiotics explores the process of semiosis not only in external communication, as Saussure did, but also in the internal processes of representation mechanisms, with sign systems, reasoning, and the whole meaning construction. That has become the focus of Cognitive Semiotics.

Semiotics is sometimes presented as being in opposition with “objective” “computational”, or “mechanistic” approaches. In this analysis, I will argue for a more inclusive approach where the relationship between semiotics, sciences, and technology largely benefits from mutual collaboration, learning, and exchange. That is the vision of new, broader future knowledge production, more in the direction of synthesis than the opposition of CP Snow’s two cultures.

One early project in that direction was the Cybersemiotics by Søren Brier (Brier 2013; Vidales and Brier 2021; Brier 2008). (Vidales 2022: 53) describes the field of Cybersemiotics as “a transdisciplinary theory of communication, signification, information, and cognition based on the work of Danish scholar Søren Brier. Cybersemiotics is a metatheory that encompasses the research programs of information theory, first and second-order cybernetics, Luhmann’s systems theory, cognitive sciences, Peircean biosemiotics, pragmatic linguistics, and language game theory.“ It is a genuine transdisciplinary theory.

Embracing multiple thought traditions not only enriches our intellectual toolkit bringing pragmatic benefits, but also aligns with values of academic open-mindedness, dialogue, and inclusiveness. Despite the complexity that collaboration between different research fields brings, there are many more benefits. Different traditions provide distinct perspectives for understanding the world. By integrating insights from various paradigms, we achieve a richer, more comprehensive, and nuanced understanding of complex phenomena.

2. Agency and Its Models

In 1986, AI-researcher, and computer- and cognitive scientist Marvin Minsky proposed a theory of natural intelligence titled “The Society of Mind” (Minsky 1986). He described human intelligence as the result of interactions of simple mindless processes called agents. Agency here is defined as the capacity to generate end-directed behaviors. These processes are the basic “thinking entities” which together produce the abilities of minds. Minsky characterized the interactions between the agents as constituting a collective intelligence of a "society of mind", building on the understanding that "minds are what brains do".

The advantage of modeling the mind as a society of agents, rather than deducing its behaviors from fundamental principles or rule-based formal systems, lies in the flexibility of assigning various types of processes to different agents. These agents can have different purposes, diverse ways of representing knowledge, and use different methods of problem-solving.

Over the years, agent-based models have significantly expanded their reach and are now used in a wide range of fields. In a conversation with MIT Technology Review, OpenAI CEO Sam Altman described a future where intelligent agents play a central role in AI, (O’Donnell 2024). The programming techniques and AI technologies required for autonomous agents, such as generative models, are at the forefront of artificial intelligence research, (Yaochu 2023). As (Satyanarayan and Jones 2024) argue, generative artificial intelligence (GenAI) implies more open-ended computer systems, that focus more on meaning than on the syntactic side of information processing.

The question is: are agents not only at the core of human intelligence but also, with different degrees of capabilities, a suitable model for information processing and knowledge generation in general, (Dodig-Crnkovic and Burgin 2024)? A picture from the diverse applications of agent-based approaches emerges, showing them as a universal language for describing interactions in terms of information exchanges in a variety of models, ranging from fundamental physics to societies and ecologies. Research has shown that autonomous agency is a feature of all living systems, from single cells to complex organisms (Dodig-Crnkovic 2016). Biological agency involves the capacity of organisms to sense, process information, and respond to their environment, actively shaping their structures and functions.

Semiotic agency is a specialized aspect of cognitive agency that focuses on the use of signs. Cognitive agency provides a comprehensive framework for understanding how organisms, including humans, perceive, interpret, and interact with their worlds (Umwelts). Semiotic agency exists at different levels of complexity of life. At the basic level, single-celled organisms exhibit semiotic agency through chemotaxis (movement in response to chemical stimuli), 'quorum sensing' (a group decision-making process), and other basal cognition functions (Lyon et al. 2021; Schauder and Bassler 2001; Ben-Jacob 2008; Witzany 2000; 2008). In biosemiotics, simple unicellular organisms are considered semiotic agents as they produce chemical signals and communicate. Through evolution, the agency in living organisms developed increasingly complex features with semiotic abilities, such as autonomy and choice, and the use of signs with contextual and cultural sensitivity. While some use of signs may be automatic or instinctual, more complex semiotic agents can choose how to interpret and respond to signs. This capacity for choice is linked to the agent's cognitive abilities and the richness of the sign system they use. At the most complex level, humans have acquired sophisticated semiotic agency through language and other cultural expressions.

In his project to conceptually derive agency from the physical substrate to the human cognitive level, Terrence Deacon introduced the concept of “incomplete nature” (Deacon 2011) and “absentials”, the missing elements that affect the whole. To fully grasp the nature of life, mind, and intentionality, we must move beyond traditional approaches that focus solely on what is present or tangible. Instead, we should consider the role of absences, constraints, and potentialities. In other words, systems are incomplete in the sense that their behavior and characteristics cannot be fully understood without accounting for what is missing or absent—these absences often define what the system is capable of achieving or becoming.

Deacon’s approach emphasizes the role of absence in creating meaning within a system-level perspective. In contrast, the concept of agency shifts the focus to the capacities of individual agents within those systems. (Sharov and Tønnessen 2021) adopt an agent-based method that is inherently holistic. They integrate insights from biology, semiotics, philosophy, and linguistics to explore how organisms, as individual agents, use signs and symbols to interact with their environment. Their approach contrasts with mechanistic views by recognizing the interconnectedness and purpose that pervade biological systems at all levels.

3. Interaction, Learning, Anticipation: A Nested Hierarchy of Languages of Agency

Looking at levels of organization in nature and means of communication (“languages”) at different levels of granularity, a continuum of languages based on the complexity of agents and their communication can be found. This continuum ranges from the most rudimentary signaling systems to the intricate and abstract languages we associate with advanced beings. So, we distinguish molecular signaling, cellular communication, organismic communication (multicellular organisms), advanced behavioral communication (simple animals), social and mimetic communication (complex social animals), proto-languages (early human ancestors), full-fledged natural languages (modern humans), formal and constructed languages (advanced human societies and their tools).

This continuum represents a progression from deterministic, reactive signaling structures to highly abstract, flexible, and complex communication systems. The complexity of the "language" correlates with the cognitive, social, and technological sophistication of the communicating agents.

“Semiotic agency is not restricted to humans – it is a natural phenomenon that is found in all living organisms, organism parts, multi-organism autonomous units, and even in autonomous human artifacts.”, (Sharov and Tønnessen 2021: 345). Thus “organisms produce internal ‘models’ of relevant parts of their surroundings, or Umwelt (Jakob von Uexküll’s concept)- the wider and more complex for more complex organisms, (Hoffmeyer 2014)(Schank, Lutz, and Wood 2023).

The anticipation and predictive capacity of cognitive agents is closely related to the final cause, or purposeful, goal-directed behavior which was banned from science since Galileo’s time being considered incompatible with scientific thinking. However recent research shows how final causation works on scientific grounds. (Hoffmeyer 2014: 246) argues referring to Deacon that “it must be concluded that life and final causation is – at least potentially – inherent in the fundamental physics”. Intentionality comes from anticipation where “present cues point to future conditions”, (Hoffmeyer, 2013). The activities of living beings are always goal-directed: they all depend on a capacity to anticipate, (Riegler 2001) (Millidge, Seth, and Buckley 2022; Friston and Kiebel 2009)(Clark 2013).

4. Evolution and Agency: Cognition as a Driving Force of Development and Evolution: Computational Approaches

Agency is central to the understanding of the evolution of life. The widely accepted scientific explanation of life over the past fifty years is often referred to as Neo-Darwinism, or "The modern synthesis", that merged the concept of Darwinian evolution with Mendelian genetics. In this framework, the gene is seen primarily as a unit of transmission rather than a unit of construction. Considerable advancement in genetics and developmental biology as well as several related disciplines led (Hoffmeyer 2013: 149) to conclude: “A re-evaluation of core presuppositions of the modern synthesis has become an urgent task.”

“Neo-Darwinism is fruitful, but only up to a certain point - which is that of the semiotic. Beyond that point, the laws of development change: instead of ‘BVSR = Blind Variation + Selective Retention', the logic of the semiotic process, involving mimesis, imagination, conceptualization, and analysis will determine the course of human evolution.'' (Heusden 2009: 130).

As argued by (Stewart 1996) and (Maturana and Varela 1980) life is coincident with cognition, in which information and communication are requisites. The biological development and its evolution are dependent upon communication, (Torday and Rehan 2012) (De Loof, 2015), where communication is the transfer of information between cognitive entities. Thus, life can be described as an information-based phenomenon, (Miller 2018), (Nakajima 2015; Navlakha and Bar-Joseph 2015).

Recent research shows that evolution is a cognitive process based on the goal-directed agency of living cells. No in-principle difference exists between animal (or other non-human living beings) and human semantic strategies. As Miller, Baluska, and Reber have shown, evolution is not based on blind variation (Baluška, Miller, and Reber 2022). Variation is purposeful, goal-oriented, based on cellular (and sub-cellular) agency.

Importantly, (Jablonka and Lamb 2014) and (Sharov and Mikhailovsky 2024) show the multi-dimensionality of evolution and the different organizational (semantic) levels at which development unfolds. What is significant for those approaches is their integrative power. Human is brought back to nature and understood as a part of and product of natural processes. As (Hoffmeyer 2013: 164) explains “The mental sphere does not consist of some mysterious subject matter at all but must be understood as an interface through which our bodies have managed to connect to the world around them, physical as social."

This connection to the world leads to the concept of the extended mind, in which Lorenzo Magnani’s Eco-Cognitive Computationalism proposes a mechanism of cognitive “domestication of ignorant entities” as an explanation of our use of tools from the environment for cognitive purposes. Among those tools are the most recent generative artificial intelligence, (Magnani 2018).

For modeling embodied, embedded, and extended cognition, the agent-based info-computational approach based on natural computing/natural information processing offers a suitable naturalistic framework. This approach bridges the gap between physical, chemical, biological, and cognitive sciences providing a unified model for studying natural and artificial cognitive systems, (Bongard and Levin 2023).

The info-computational approach presents a comprehensive view of nature as a hierarchy of informational structures, where computation and cognition are emergent properties arising from the interactions of agents at various levels of complexity, processes studied by (Manicka and Levin 2019) and (Pfeifer, Iida, and Lungarella 2014). This framework not only enhances our understanding of biological systems but also guides the development of advanced computational technologies and cognitive robotics. The process of learning from nature to computational models and back is excellently described, as already mentioned, by (Rozenberg and Kari 2008).

Agent-based models are appropriate for modeling systems of components with different, even conflicting, goals, (Dodig-Crnkovic 2017) (Dodig-Crnkovic 2022). In Artificial Intelligence, an agent-based system is a distributed AI system used for problem-solving and modeling/management of multi-agent behavior. They are especially suitable approaches to development and evolution and the emergence of intelligent behavior. (Miller 2023).

Agency reflects an entity's capacity to act intentionally and make decisions based on goals or priorities/values. To exercise agency effectively, an entity must engage in continuously adjusting its internal models and taking actions to minimize prediction errors or uncertainty – known as Active Inference. The logic underlying this process is the Free Energy Principle—the idea that agents are fundamentally motivated to reduce free energy by aligning their internal states with the external world.

Free Energy Principle (FEP) and Active Inference theory are foundational computational and conceptual frameworks in neuroscience and cognitive science that describe how living systems maintain their internal states and interact with their environments (Parr, Pezzulo, and Friston 2022). These concepts relate to semiotic and cognitive agency in several significant ways, as they both address how systems process information and generate meaning in response to their surroundings. The Free Energy Principle implies that biological systems, including the brain, strive to minimize free energy, a measure of surprise or uncertainty. Free energy is mathematically defined and relates to the difference between expected and actual sensory inputs. By minimizing free energy, organisms maintain homeostasis and resist disorder. Organisms generate internal models to predict sensory inputs. When there is a discrepancy (prediction error) between expected and actual inputs, the system works to minimize this error. Minimizing free energy helps maintain internal states within viable bounds.

Active Inference means that organisms not only passively receive information but also actively engage with their environment to reduce uncertainty. Organisms take actions that are expected to bring their sensory inputs closer to their predictions, thus minimizing free energy.

5. Cognitive Semiotics

Cognitive semiotics integrates signs, meaning, and cognition (minds), (Zlatev 2018). It is essential to understanding how we come to know the world through signs and languages. Claudio Paolucci, drawing from the work of Umberto Eco and Charles Sanders Peirce, investigates how signs, meanings, and languages influence cognition, (Paolucci 2021).

Terrence Deacon (Deacon 1997) on the other hand presents a comprehensive theory on how the unique features of human communication, particularly symbolic language, emerged and evolved. He integrates insights from neuroscience, anthropology, linguistics, and evolutionary biology to address the complex relationship between language, brain development, and cultural evolution. Deacon elaborates on the co-evolution of brain and language, symbolic reference, the role of social interaction, and human language as a uniquely human adaptation, considering both biological and cultural evolution in explaining its emergence.

Situating the phenomenon of human language in the context of evolution, it may seem that it is orders of magnitude more complex and sophisticated system than “languages” used by other living beings. However, the difference might not be so big seen in the light of evolution and the role of “languages” for different organisms, (Witzany 2000).

If we analyze levels of agency in humans, we will at the bottom find the cellular level at which every single cell in our body possesses basic agency. Organized in tissues, organs, and finally the whole organism, they both influence and are influenced by higher levels of organization. On the level of the human body, there is a considerably more complex agency with new dimensions, communication means, and interpretative mechanisms. Here also extended agency belongs where human agents use social communication and tools to enhance their agency.

6. Semiotics and Info-Computational Models: Information, Computation, Cognition: Agency-Based Hierarchies of Levels

The anticipation and predictive capacity of cognitive agents is closely related to the final cause, or purposeful, goal-directed behavior, which was advocated by Aristotle, but banned from science since Galileo’s time being considered incompatible with scientific thinking. However recent research shows how final causation works on scientific grounds. (Hoffmeyer 2014) argues that “it must be concluded that life and final causation is – at least potentially – inherent in the fundamental physics”. Intentionality comes from anticipation where present cues point to future conditions, (Hoffmeyer, 2013). The activities of living beings are always goal-directed: they all depend on a capacity to anticipate, (Riegler 2001) (Millidge, Seth, and Buckley 2022; Friston and Kiebel 2009)(Clark 2013).

Semiotics, the study of signs and symbols, provides a foundational understanding of how meaning is constructed and interpreted. In the study of sign processes (semiosis), the meaning of a sign is its place in a sign relation. Info-computational models use computational methods to model, simulate, and understand cognitive processes, emphasizing the role of information processing in cognition, so compared to semiotics, they are closer to the physical substrate level of abstraction.

Cognitive agents, including humans, other living organisms, and intelligent machinery, play a central role in interpreting signs and transforming them into symbols. This process involves perception, interpretation, and the active engagement of cognitive systems with their environment. Models of the active relationship between an agent and the environment (including other agents) have been elaborated by (Friston et al. 2012)(Clark 1989; 2013)(Seth 2014; Millidge, Seth, and Buckley 2022).

If we want not only to observe, classify, and systematize but also model, anticipate, simulate, and control semiotic processes, then computational approaches can be very useful epistemic and practical tools. Historical developments of contributing research fields have contributed to the present state of complex relationships when it comes to collaboration and common agendas. Semiotics started focused on humans.

The agency is the basis for linking the interconnected concepts of information, computation, and cognition for both natural and artificial systems (Dodig-Crnkovic 2016). It posits that agents exist at various levels of organization, from elementary particles to complex organisms and ecosystems, each engaging in processes of information exchange and processing/computation. Nature is thus described as a network of informational structures, with agents at different levels of organization. Information is relational and agent-dependent, where agents are entities capable of acting and causing changes. Computation is defined as the process of information exchange between agents, effectively making the dynamics of information the core of natural computation/morphological computation.

The hierarchical organization of information within nature, (Dodig-Crnkovic 2016), from basic physical interactions to complex biological systems, can be explained using Deacon’s levels of natural information, which range from data patterns (syntax) to functional information (pragmatics), reflecting the increasing complexity and functionality at higher levels of organization.

Marcello Barbieri (Barbieri 2024) introduced the idea that living organisms operate through multiple levels of coding and decoding processes, which are crucial for understanding the complexity of life. Barbieri emphasizes that biological systems are not just governed by chemical and physical laws but also by informational laws, and understanding these informational processes is fundamental for a complete picture of life.

Cognition is framed as a property of living agents, arising from cellular and molecular interactions, what Levin termed “Darwin’s agential materials” (Levin 2023). Even simple organisms like bacteria exhibit basic forms of cognition through their ability to process information and adapt to their environment. Under the computational lens, this cellular-level computation forms the basis for more complex cognitive processes in higher organisms.

The concept of autopoiesis is central to understanding cognition as a natural process. Autopoietic systems are self-organizing and self-sustaining, continuously regenerating their components through interactions with their environment. This process is seen as a fundamental form of cognition, applicable to all life forms. Self-organized pattern generation, i.e. morphogenetic self-organization has been studied by (Yaochu 2023).

Morphogenesis is the development of structure in an organism, and it can be described as morphological computation. Natural morphogenetic processes may be used to understand the principles of self-organization and apply them to artificial systems.

7. Conclusions

Connecting info-computational processes in cognitive agents including active inference with coding and decoding symbols and computing signs, we went through questions of agency, basal cognition, and extended evolutionary synthesis, with complexification of biological forms and communication languages. Goal-directedness and bottom-up, top-down (circular) causation appear as a natural consequence of physical substrate-based semiotics and computational models of meaning.

A widespread belief holds that mechanistic explanations and scientific approaches contradict goal-directedness or purposeful agency. Yet, agency, characterized by teleology, autonomy, and semiotic capabilities is embodied in a physical substrate and can be modeled “mechanically”/ formally/ computationally. It is important to note, however, that mechanical models represent only one perspective of an agent. Just as medical diagnostics might use X-rays or chemical analysis to gain a partial understanding of a patient, these models provide specific perspectives that can contribute to a more holistic view when combined with other approaches.

Another important observation is that the mechanism is not what it used to be in the industrial era. Machines are no more indifferent, pre-defined, non-sensible contraptions. Machines are developing into intelligent, sensitive, and sensible systems. We must urgently update our metaphors of mechanism, (Bondgard and Levin 2021) (Levin 2022).

A fresh example of how the contribution to the understanding of the human mind and cognition can come from a “mechanistic”, machine-centered perspective such as AI came recently from Large Language Models demonstrating the ability of a machine to use human language in a natural dialogue without even being conscious, (Dodig-Crnkovic 2023). Our intuitive feeling about language and its connection to consciousness may be supplemented by such experiments in the machine world. With new models of computation, new avenues open for meaning-making (including media production, scientific discovery, text searches and summaries), and a better understanding of semiosis in living systems.

Embracing and integrating diverse thought traditions is crucial for epistemological, ethical, and pragmatic reasons. Every tradition or theory, no matter how robust, has its blind spots or limitations. By engaging with multiple traditions, we can transcend the limitations of a single perspective and achieve a more balanced understanding. Embracing and respecting diverse intellectual traditions fosters the potential for novel syntheses, which in turn drive progress further.