The Measurement Problem and the Search for Fundamental Theory

1. Introduction

Since its inception, quantum theory has faced persistent criticism due to its fundamentally revolutionary concepts, such as wave-particle duality, indeterminism, and non-locality. These principles directly challenge traditional notions of how physical systems are expected to behave, often conflicting with the deterministic and localized frameworks of classical physics. Quantum physics began as a groundbreaking paradigm with Max Planck’s revolutionary proposition in 1900, which introduced the frequency dependence of energy, marking the birth of quantum mechanics. However, this can be considered a culmination of several ideas before Planck, especially the works of Boltzmann involving Blackbody radiation. In 1905, when Einstein explained Hertz’s 1887 discovery of electromagnetic waves, using his theory of the photoelectric effect, the Quantum idea got more validity because it could just easily be verified by means of an experiment. However, it was only when the theory was more structured and organized as a semi-classical description of the physical system that it started getting more attention. Bohr’s model of the atom was an important consideration in this regard.

The idea that a true world exists out there and that a scientist’s work is only to make models of the true world based on her observations is not new. Plato pioneered this idea by distinguishing between the ideal and the real world. He held the Ideal (world of Forms) to be superior, something that can only be apprehended through reason and intellect, and that the Real world of sensory experience as merely a reality of the human condition. Almost two thousand years later, Galileo’s work on the Scientific Method provided a framework for how science should be done. The process was to examine the phenomenon as fully as possible so as to quantify the sensed data into mathematics. To evaluate how successful this process of quantification was, one could just easily devise an experiment aimed at testing the predictions of the model. In this way, the scientific method worked.

The scientific method also incorporates Descartes’ dualistic distinction between mind and body. For a scientific model to approximate objective truth, it has to be unbiased and free of measurement errors. And this is only possible when a physical description can be made without any reference to the mind. Furthermore, Occam’s razor is an essential principle within the scientific method because minimizing variables of observation is sure to make the model closer to truth. Needless to say, the minimization should not be done at the expense of the model not representing the phenomenon.

Before the quantum theory of the form we know today, Maxwell’s Electrodynamics and Newton’s and Galileo’s Classical Mechanics, including Einstein’s SR and GR, were the most dominant physical theories of the world, and all of these were deterministic. So, even for empirical experimentation, one could easily apply the equation of motion, make predictions, testify, and verify the model. The question, "Does the world exist when nobody is observing it?" was not even considered valid within these frameworks. The objective existence of the physical world was not merely assumed but viewed as an essential presupposition for deterministic theories to operate. Thus, the answer to such a question was both implicit and necessary: "Yes, it ought to be."

2. The Role of Philosophy of Science

The philosophy of science plays a crucial role in shaping scientific progress, serving as a guiding framework for scientists grappling with foundational questions and decisions. A notable example of this influence can be observed in the development of Einstein’s Special Relativity. The search for a hypothetical medium called ’aether’ played a crucial role in guiding the foundations of Einstein’s theory of Special Relativity. If not for the null result of the Michelson–Morley experiment, Einstein would have never felt the need to unify Maxwell’s Electrodynamics to the Lorentz transformation equations. Consequently, the first postulate of Special Relativity—the constancy of the speed of light—would not have emerged as a cornerstone of the theory, and the physical theories we know today would have been different.

Another example would be the case of "dark energy." Despite the absence of direct empirical evidence, the concept of dark energy is a central element of the standard model of cosmology, widely accepted and actively utilized in scientific discourse. This highlights the enduring impact of philosophical considerations in shaping scientific paradigms, even in the absence of direct observational evidence. It illustrates how philosophical frameworks can bridge gaps in empirical understanding, driving the evolution of scientific theories.

Ancient Greek philosophers are revered for their original ideas that continue to influence modern science. Among many of their contributions is the concept of composite structures in the universe. Many of these philosophers posited that the cosmos is comprised of fundamental elements like air, water, and fire. Remarkably, our contemporary understanding of the universe parallels this notion, as our physical theories are built upon the idea of a composite structure involving elementary particles such as quarks, leptons, and bosons.

Similarly, the ancient Greek philosophers’ exploration of perfect geometrical structures laid the groundwork for the concept of ’self-evident’ or ’axiomatic’ truths. These truths were considered so intuitively obvious that formal proofs seemed unnecessary. Alternatively, their avoidance of proofs may have stemmed from the inherent complexity of demonstrating the formal proofs of these truths within their logical systems. This could hint towards modern mathematical theories, including Gödel’s Incompleteness Theorem.

Furthermore, Zeno’s paradoxes, which highlight the challenge of distinguishing between the motion and the rest states (at consecutive times) of a moving arrow, offer groundwork for the analysis of rest and motion in Einstein’s Special Relativity.

In light of these reflections, the philosophy of science emerges as a guiding force in our scientific endeavors, showing pathways for exploration and discovery.

The purpose of any ultimate science extends beyond only explaining the process of how. When probability measures are considered, there is mounting evidence to the fact that a universe like ours is very unlikely. Therefore, an ultimate science is also entitled to explain this anomaly. The anthropocentric perspective falls short in providing satisfactory answers. It limits our understanding by placing humans at the center of the universe, potentially obscuring broader truths and possibilities. To address such profound questions, we must either expand the scope of scientific inquiry to include broader philosophical considerations or actively integrate insights from the philosophy of science. In either case, the philosophy of science remains crucial in guiding scientific endeavors. Regrettably, some prominent physicists, such as Hawking and Weinberg, have expressed skepticism regarding the relevance of philosophy to scientific progress, potentially underestimating its role in guiding foundational inquiries.

3. The Measurement Problem

The inherent indeterminism of quantum physics marked a significant departure from classical notions, leading many physicists, including Einstein, to challenge its validity. In quantum mechanics, observation fundamentally alters a system’s outcome. "We can’t observe quantum systems without disturbing them." Bohr, adopting a positivist stance, asserted that "Reality doesn’t exist independent of observation"—a viewpoint that might have challenged the concept of objective reality as traditionally conceived in the scientific method.

By the 1920s, the notion of objective reality had been heavily influenced by Kant’s Transcendentalism [1]. Kant posited that while the true nature of things as they exist independently of perception (the noumenon) remains unknowable, we can nonetheless acquire objective knowledge of phenomena as they appear to us through our cognitive framework.

Bohr seems to have been influenced by this idea of Transcendentalism, as noted by Honner [2] and Faye [3]. However, at times, Bohr has contradicted himself with his Positivist arguments. While Bohr’s ideas on Correspondence and the definitions of Complementarity have significantly changed over time, there was never really a debate about the nature of reality but on how the subjectivity of the measurement process is to be reconciled in the quantum framework. By the 1930s, Bohr was firmly convinced that the observer could never influence the outcome of an experiment. Consequently, he extended his concept of complementarity to incorporate the experimental setup as an integral part of the system.

Critics have accused Bohr of adhering too rigidly to the realism of classical physics while dismissing quantum realism as merely a logical construct. It could be because of this deep conviction in the supremacy of his Correspondence principle that the role of observation in quantum physics was never taken too seriously. Even in the face of new spin statistics (that is strictly based on quantum physics), Bohr stuck with the realism of classical physics. Nevertheless, the correspondence principle undeniably played a crucial role in guiding the early development of quantum theory.

This inability to reconcile the role of observation in experiments with the quantum mechanical framework led to the emergence of what is now known as ’The Measurement Problem.’

If we look at the modern understanding of science, an electron is not a point-like particle in the way Bohr imagined. The best representation of an electron is in terms of probability waves. The electron’s manifestation as a particle or a wave depends on the specific experimental context. This view aligns more closely with the philosophy of conscious realism than with Bohr’s entity realism. The act of measurement, alternately defined by the specification of the experimental setup (as done by Bohr and colleagues), seems to have made a difference in the reality of an electron on whether it appears as a particle or a wave.

This understanding, coupled with several other ideas in physics, hints towards a deeper, generalized understanding of the physical world based on conscious realism, where the act of measurement, i.e., observation itself, is deeply intertwined with the very fabric of reality. Perhaps this could go beyond the conventional spacetime fabric, as neuroscientists like Hoffman [4] suggest. This approach could, in principle, solve the measurement problem and provide a newer paradigm shift in how we look at the universe—more on this in the next chapter.

4. The Copenhagen Interpretation of QM

In 1925, Werner Heisenberg formulated his matrix mechanics and, two years later, introduced the concept of quantum indeterminism through his uncertainty principle, which sharply contrasted the assumptions of classical physics. Bohr, however, viewed this relationship through the lens of his Complementarity, as noted by Faye [5]. Bohr even interpreted the wave-particle duality as another manifestation of this Complementarity.

In 1926, Max Born’s probability interpretation of the square of the wavefunction provided a solid theoretical foundation for the quantum theory. This interpretation allowed the wavefunction to be understood as contributing to representing probabilities.

The challenges posed by the EPR paradox -presented by Einstein, Rosen, and Podolsky in 1935 -prompted Bohr to restrict the scope of Complementarity to the system’s kinematical and dynamical properties. Faced with these challenges and the inability to define the role of quantum indeterminism in experimental outcomes, a need arose for an interpretation of the mathematical formalism of Quantum Mechanics.

Over the course of time from 1925 to 1950, several views emerged describing the mathematical formalism of Quantum mechanics. It was only in 1955 that Heisenberg used the term ’Copenhagen Interpretation (CI)’. According to Heisenberg’s CI, the act of measurement affects the observation. It also asserts that to perform such an act of measurement, we need a measuring apparatus. But, the details of a measuring apparatus aren’t specified, which means a physical system could also be a measuring apparatus. It also means that the measurement process has nothing to do with the individuality of the observer, as asserted by Pauli [6]. This aligns with Bohr’s idea of ’The individuality’ or ’The Unified whole’ of the atomic process, where he viewed the measuring system as a part of the system being observed by means of entanglement.

With this interpretation, the CI attained a means to get away with the wavefunction through a mysterious collapse associated with the observation. Nevertheless, Heisenberg never used the word ’collapse’ but instead viewed it as "reduction of state’ of wavefunction that occurs once a given system is observed by the apparatus. In doing so, the subjectivity associated with the observation process could also be eliminated. Also, according to CI, Heisenberg’s uncertainty relation was not a limitation of knowledge in the classical sense but a feature of the new ontology of Quantum Mechanics.

The idea that the outcomes of the kinematic/dynamical properties of atomic systems can’t be described without reference to the experimental apparatus is also called Bohr’s indefinability thesis. It has a stark resemblance to the ’Incompleteness Theorem.’ One could argue that within the formal system of quantum mechanics, hints emerge of a higher theory—one that encapsulates the observer’s subjectivity as an integral aspect of the reality being observed. Such a theory might involve an axiomatic reconstruction of quantum mechanics, refining the CI’s "collapse of the wavefunction" or Heisenberg’s "reduction of the state." Alternatively, it could reflect a fundamental limitation in the logic of our formal system governed by quantum physics. All these assumptions seem more plausible than the ’absurdity of the Schrodinger’s cat.’

These realizations point towards a theory of consciousness that, in addition to incorporating the subtlety of the observation process, also reconciles the existing theories into a coherent framework of science governed by the scientific method.

5. The Need for a Higher Theory of Consciousness

The idea of generalization and simplification isn’t merely a tool for physicists; it extends to all natural processes, which appear to be governed by fundamental laws. Physicists often discuss the concepts of beauty and symmetry in physical laws, believing that there must exist a higher theory that unifies all fundamental forces. This pursuit of unification can also be seen as an extrapolation of present ideas in modern physics in relation to the four fundamental forces. When it comes to the study of consciousness, it is essential to define it within the framework of these forces that we understand so very well.

Many physicists have noted that successful scientific theories like Quantum Theory and General Relativity are fundamentally incompatible. And among the four fundamental forces, we only have a good idea of unification of three of these forces but gravity.

One approach to unifying these forces is to address the incompatibility within the frameworks of Quantum physics and General relativity. This would require revising one or both theories at a fundamental level. Such revisions would pave the way for a unified theory that also gives the right tools to the theory of consciousness. Maybe this could be done by discovering the physics behind the collapse of the wavefunction. Many of the important scientific theories of the past century were developed in an attempt to resolve the paradoxes within the framework of established theory. In fact, such paradigm shifts define scientific progress, as Kuhn [7] rightly notes in his book The Structures of Scientific Revolutions.

Another possibility of the unification scheme of forces could lie in a new paradigm shift that is beyond resolving this incompatibility. Given the fact that both of these theories have been verified by numerous experiments with a reasonable degree of accuracy, any ordinary revisions will not solve the incompatibility and existing paradoxes. The need of the time is a newer synthesis- a higher theory that not only beautifully unifies these forces and resolves their paradoxes but also extends beyond the framework of these theories. This would make these successful theories special cases of a more fundamental framework.

This is an exciting time in the history of physics. And this hasn’t happened in a long time. In fact, I would argue that a paradigm shift of this magnitude in our understanding of the cosmos has been due for almost a century.

Similar to the unification scheme of fundamental forces, there are two general approaches to a theory of consciousness. On one hand, we can contemplate a theory of the physics of consciousness. Notably, seminal works by Penrose and Hameroff, Integrated Information Theory, Global Workspace theory, and a few others delve into this realm, suggesting the physics behind the emergence of consciousness from physical processes. These theories share a common goal of exploring the underlying mechanisms by which consciousness arises from neural and physical activity. All these theories face challenges posed by the difficulty in reconciling quantum mechanics’ probabilistic nature with the deterministic nature of classical physics, particularly in the context of measurement and observation.

On the other hand, we could also look at consciousness as a fundamental entity and not as an emergent property, as suggested by the works of Hoffman [4]. By embracing this perspective, we have a new paradigm of physical theories grounded in the dynamics of consciousness. All theories suggesting the emergence of consciousness from physical systems are plagued by the Hard problem. Fortunately, this approach doesn’t have to face the Hard problem because we are talking about the emergence of physical processes from consciousness and not the other way around.

6. Framework of Theories of Consciousness

A significant portion of the forthcoming discussion operates at the conceptual level. While some assertions may initially appear speculative, they are grounded in scientific consensus. Indeed, contemporary science often traces its origins to the realm of ideas and conjectures. In this section, I have discussed some of the postulates of the new theory of consciousness.

6.1. Postulate 1:

Subjective experience of consciousness is unique and, therefore, can’t be mimicked by any physical processes

The big question regarding this thesis is whether we can address the ’Hard problem of consciousness,’ which involves resolving the anomaly of how the subjectivity of conscious experience emerges from a system of logic and physical processes. This subjective experience is also termed qualia, and by definition, it seems that no matter how sophisticated a system of proofs is, a super-scientist Mary will never feel the experience of seeing the color red solely with objective descriptions like wavelengths and color. It, therefore, doesn’t make sense to look for the theory of consciousness unless we have either a machinery to resolve this limitation of the Hard problem or a willingness to see past this limitation and accept it as a consequence of the new theory of consciousness. I presume that the hard problem is not a problem at all but a standout feature of any theory of consciousness.

The distinctive nature of consciousness lies in its irreducibility—it cannot be mimicked or reproduced by any physical system or logical framework. Consciousness is inherently special because it operates as the governing force behind this proposed new physics, transcending the boundaries of purely physical or computational models. If the Hard Problem were to be "solved" in a way that reduces consciousness to physical processes, then the uniqueness of conscious experience would be lost.

The only conceivable way to replicate the subjective experience of consciousness is to create another system that itself possesses consciousness. This insight highlights the fundamental distinctiveness of consciousness as not merely an emergent property but as a unique and irreducible phenomenon within the framework of reality.

6.2. Postulate 2:

The laws of physics governing consciousness are Quantum Mechanical and beyond

The quantum theory is considered to describe the underlying system at the most basic level. Therefore, any theory of consciousness has to be consistent with the quantum mechanical description as well. In doing so, non-locality, indeterminism, and duality are also the basic properties of the theory of consciousness. There are many Quantum Mechanical formulations of consciousness that posit consciousness arises due to the complex interactions and dynamics of quantum fields from within the brain. One typical work is discussed by Ricciardi [8]. However, I would presume that rather than describing it as an exclusive process happening in the human brain, we should only look at the human brain as an image of a larger order of the universal fundamental consciousness. In doing so, a conscious initial state goes to a conscious final state under a Hamiltonian governed by the dynamics of conscious experience. Since Quantum Field theory is valid, a simple transition amplitude equation given by the Fermi-Golden rule also works here. This framework can guide the initial developments of the new theory of consciousness.

$${\Gamma }_{i\to f}={\displaystyle \frac{2\pi }{\hslash }}\left|\langle f|H_n|i\rangle \right|\rho \left(E_f\right)$$

(1)

As shown in Figure 1, the probability space of conscious experience is inspired by Hoffman’s network of conscious agents. In fact, this probability space is the fusion of the ideas of Hoffman et al. [4] and those of Ricciardi and Umezawa [8]. A transition occurs due to a perturbing Hamiltonian under a conscious experience. Depending upon whether this perturbing Hamiltonian is weak or strong, the system either makes finer energy corrections in the same initial state or evolves to some final state, given by the laws of quantum mechanics. Any physical experience that can amplify or inhibit this transition has to do so by altering the Hamiltonian. Time, according to uncertainty relation, dictates whether there is coherence or decoherence of conscious experience. Between these initial and final states (represented by nodes) lie the subjective conscious experience.

6.3. Postulate 3:

Intelligence guides the time evolution of a conscious system. The equilibrium of a conscious system doesn’t have any physical significance unless there is intelligence

In the physical universe, equilibrium is a fundamental state achieved through the minimization of energy and the pursuit of stability. This principle often governs the time evolution of physical systems. However, in the context of a conscious system, intelligence introduces a purposeful dynamic that transcends the purely physical imperative for equilibrium. Driven by intent and will, intelligence can enable a conscious system to act contrary to the equilibrium state in pursuit of higher-order objectives. The influence of intelligence is inherently domain-specific and contextual. It could be a need for survival and reproduction, a need for understanding and awareness, or a need for value and meaning. The possibilities are endless.

7. Conclusion and Future Scope

A pivotal result in quantum mechanics, Bell’s theorem, addresses the fundamental incompatibility between the predictions of quantum mechanics and the assumptions of locality and realism that underpin classical physics. Developed by John Bell in 1964, the theorem demonstrates that hidden variable theories cannot replicate the statistical predictions of quantum mechanics without violating the principle of locality. Since the theory of consciousness, as outlined in postulate 2, aligns with quantum mechanics, it inherently incorporates non-locality. This alignment enables the theory of consciousness to provide statistical predictions consistent with quantum mechanics while maintaining coherence with Bell’s inequalities.

The key to this reconciliation lies in the introduction of a novel variable: conscious experience. This variable represents a non-physical, non-local element that interacts with the quantum system during observation or measurement. Unlike classical hidden variables, which are deterministic and local, the conscious variable operates in a non-local framework, allowing it to mediate quantum correlations without violating the constraints of Bell’s theorem. In this sense, the conscious variable is neither a direct replacement for hidden variables nor a purely physical construct but an extension that complements the probabilistic and non-local nature of quantum mechanics.

This perspective does not merely assert the importance of non-locality but reframes the role of observation in quantum mechanics. The conscious variable introduces a layer of interaction that governs the reduction of quantum states, addressing the subjectivity of measurement. Thus, the theory does not contradict Bell’s inequalities; rather, it operates within a paradigm where non-locality and consciousness are integral to the structure of quantum mechanics.

In conclusion, this paper has explored the intricate relationship between quantum mechanics, consciousness, and the philosophy of science, proposing a framework where consciousness is treated as a fundamental variable that bridges the deterministic tendencies of hidden variable theories and the probabilistic, non-local nature of quantum mechanics. By addressing the measurement problem and the apparent incompatibilities between quantum mechanics and classical physics, the theory extends the principles of quantum mechanics to include the observer’s subjective experience as an integral aspect of reality. This perspective aligns with a broader philosophical tradition while suggesting new directions for both theoretical and empirical investigation. Future research must focus on formalizing the mathematical structure of the proposed framework and identifying empirical methods to validate its predictions. If successful, this synthesis could transform our understanding of reality and offer profound insights into the fundamental nature of reality.