Neuromorphic Photonic On-chip Computing

1. Introduction

A determined pursuit for computational systems mirroring the efficiency of the human brain has served as a compelling impetus with significant industrial implications right from the emergence of Artificial Intelligence (AI) dating back to 1950s [1,2]. It has undergone an initial surge of optimism followed by decades of setbacks, primarily attributed to the dearth of computing power during that period. Inspired by human brain's neurosynaptic framework [Box 1], AI endeavors to attain the level of precision equivalent to human performance in tasks that pose difficulties for conventional computing systems yet come naturally to humans. The brain's computational power has an efficiency of around aJ/MAC (multiply-accumulate) operation [3]. In contrast, conventional computers generally bound by von Neumann architecture struggled with the architectural scaling typically consumes 100 pJ/MAC operation [3].

In the present era of prolific data generation and escalating automation, AI has emerged as a transformative force [Box 2][4], reshaping industries and redefining human-machine interactions through Machine Learning (ML)[5] and Deep Learning (DL)[6,7]. These developments are influenced by the advent of computing power reflected in the modern Nvidia graphic processing units (GPUs)[8] and Googles's tensor processing units (TPUs)[9] consuming only around 20 PJ/MAC operation sustaining Moore's Law. Some of the current applications of AI extend across various sectors, encompassing autonomous driving, robotic vision, remote sensing, microscopy, surveillance, deterrence, and the Internet of Things[7,10]. The significant accomplishments of AI algorithms, particularly neural networks (NN) [11,12], have influenced numerous facets of our daily lives, ranging from language translation[9] to cancer diagnosis[13].

1.1 Neuromorphic Electronic Computing: On-Chip AI

The technology requirements for AI have evolved[14,15]. NN, the backbone of neuromorphic computing for AI algorithms, consists of artificial neurons arranged in layers connected to others through synapses (Box 1), allowing these networks to adapt and learn from data and exhibiting inherent parallelism and distributed computing features that confront significant hurdles when implemented in digital electronics[16]. This misalignment with the sequential nature of von Neumann architecture with separate central processing units and dynamic random-access memory in digital computers has led to a shift towards on-chip electronics driven by its potential to enhance computational speed and efficiency by co-locating computing and storage units[17,18,19]. State-of-the-art neuromorphic electronic architectures, which rely on transmission lines and active buffering techniques, face challenges in meeting the demands of massive parallel signal fan-out and fan-in[20]. Architectures resort to digital time-multiplexing, enabling the construction of more extensive NNs at the expense of bandwidth that detours the limitations of electronic writing[16].

Industry and research initiatives emphasize the viability of on-chip electronics. Significant achievements in designing application-specific integrated circuits (ASIC) have been observed in noteworthy projects, including IBM's TrueNorth[21], Neurogrid[19], SpiNNaker[17], Tianjic[22], Microsoft Azure Maia AI Accelerator[23], and Intel's Loihi[18]. Co-locating computing units and storage units within the same chip addresses the inefficiencies of traditional architectures[3]. However, challenges persist; interconnect density[24] and bandwidth limitations[16], scalability concerns[20], and power consumption issues[25] underscore the need for continued innovation. The future envisions more sophisticated applications, ranging from nonlinear programming[26] to real-time learning[27] and intelligent signal processing[28], evidenced by recent applications, including The Human Brain Project[29], ChatGPT[30], and Tesla's Dojo Chip[31]. Applications prioritizing bandwidth and low latency necessitate shifting towards direct, non-digital photonic broadcast interconnects[32].

1.2. Photonics: A Solution Looking for a Problem!

Traditional electronics often overshadow the historical development of photonics due to a lack of advantageous parallels in photonic digital logic[33]. Nevertheless, photonics has achieved remarkable milestones in data transmission[34] and communication[35] and is experiencing a renaissance due to its potential to revolutionize information processing[28]. Photonics, harnessing the power of light for communication and computation, presents a promising alternative to overcome the limitations posed by electronic interconnects[24]. Unlike electrons, photons provide diverse dimensions to control, such as wavelength, polarization, and spatial mode. These advantages of photonics properties allow for faster data processing, reduced power consumption, and inherent parallelism at every level of integrated photonic circuits, positioning it as a transformative solution for contemporary AI[36]. Non-digital computing models, particularly those enabled by photonics, promise solutions to challenges like low latency[3], high bandwidth[20], and low energies[25], giving rise to a field of neuromorphic photonics at the intersection of photonics and the neural engineering framework[37].

1.3. Is Neuromorphic Photonics the Future of AI Technology?

Neuromorphic photonics aims to facilitate matching hardware to the algorithm, supporting many-to-many communication and overcoming the trade-offs inherent in electronic approaches, which are power-hungry[25] and trade-off bandwidth for interconnectivity[16]. With its interconnectivity[32] and matrix multiplication advantages[38], photonic is a promising candidate as waveguides do not suffer from inductance and skin effects. The maturity of enabling technologies in this domain provides a pathway to achieve cascadability and nonlinearity in neuromorphic systems. In particular, demonstrated scalable silicon photonic devices, including waveguide, Mach-Zehnder Interferometer (MZI)[39], micro-ring resonator (MRR)[40], micro comb[41], photonic crystals[42,43], and phase-change material (PCM)[44], contribute and emerged as a promising technology. Leveraging existing Complementary Metal-Oxide-Semiconductor (CMOS) compatibility[45], silicon photonics can be seamlessly integrated with available lightspeed active optoelectronic and CMOS circuits without requiring additional complex processes[46]. This co-design with mature electronics provides a versatile component sensitivity[47] for large-scale photonic fabrication and integration platforms [45], along with calibration and control[48]. In specific applications, such as autonomous driving, the combination of neuromorphic photonics facilitates real-time decision-making[49] with unparalleled efficiency. In healthcare, the ability of photonics to handle vast amounts of medical data accelerates diagnostic processes[13]. Intelligent systems for smart cities leverage photonics' speed and bandwidth advantages for seamless data processing[28].

As neuromorphic photonics continues to push boundaries, research and technology will play pivotal roles. Challenges arise in storing and retrieving neuron weights within on-chip memory[50,51]. Ongoing efforts in optical memories, particularly 'in-memory' computing, have been explored[52]; however, their limitations include difficulties in high-frequency read and write operations[44]. The concept of electronic-photonic co-design emerges as a highly promising approach for implementing neuromorphic photonic systems[36]. This integration should leverage the characteristics of memory types (volatile and non-volatile) in digital or analog domains tailored to ASIC and computational requirements. The envisaged future demands of AI, marked by increased complexity and diverse applications[53], align seamlessly with the strengths of photonics [47]. Research efforts into architectures and algorithms for neuromorphic photonic processors (NPPs) [36,54] signal new directions that hold tremendous promise for the trajectory of AI with a potential of aJ per MAC energy efficiencies and P-MAC/s/mm² processing speed[3]. However, nurturing this synergy will require collaborative efforts to overcome integration, scalability, and compatibility challenges and explore novel applications to propel the field toward new frontiers in intelligent computing on chips. Here we explore the transformative domain of neuromorphic photonics with focus on neuromorphic photonic processing node (NPPN), from signal weighting mechanisms, and photonic neuron (PN) intricacies, to the architecture of neuromorphic photonic networks (NPNs) and training. Additionally, recent advancements in experimentally demonstrated on-chip neuromorphic photonic approaches are discussed, along with potential applications and the hurdles they encounter in scaling up for widespread deployment. There is a strong emphasis on the necessity for groundbreaking on-chip hardware innovations to be compatible with fabrication technology to enhance energy efficiency, performance, and scalability within neuromorphic systems. This extensive exploration aims to provide valuable insights into the present status, challenges, and future possibilities of photonic and neuromorphic computing on-chip.

2. Neuromorphic Photonics Processing Node

The convergence of neuromorphic photonics potentially revolutionizes the computing paradigm[37], where the complexities of brain-inspired processing merge with the capabilities of light-based technologies[55] by harnessing the optical system's inherent direct multiplication[38], multiplexing techniques[56], energy efficiency[25], and speed[3]. At the heart of this endeavor lies the design of a NPPN (Figure 1) inspired by the fundamental components of the brain's neural architecture (Box 1). Designing an NPPN that encapsulates these functionalities necessitates a multidisciplinary approach, integrating insights from neuroscience, materials science, photonics engineering, fabrication technology, system integration, platform cointegration, and packaging[14,57]. Different hardware implementations of neuromorphic photonics have been demonstrated, each designed for specific application classes. These implementations typically involve NPPNs, which consist of nonlinear PN interconnected through configurable photonic synapses (further referred as synapses) for linear weighting. These models differ in PN signal representation (continuous or spiking) and weight configuration (tunable criterion[5] or fixed[58]) to accomplish a particular task per the learning algorithm. In general, continuous-variable nonlinear neurons can be trained using backpropagation[5], while spike-timing-dependent update rules are well-suited for spiking neurons [59,60].

2.1. Neuromorphic Photonic Processing Node Architecture

NPPN architecture is designed to emulate the functionality of traditional neuron while operating in the optical domain (Box 1).

Figure 1. The neuromorphic photonic processing node (NPPN) architecture. It incorporates artificial neural network (ANN) connections and reservoir computing (RC) dynamics. Red, black, and blue colors indicate optical, electrical, and optical/electrical processing or interconnect, respectively. Interconnect lines denote connections in the ANN configuration, while dashed lines indicate the absence of direct connections in the RC configuration.

Figure 1. The neuromorphic photonic processing node (NPPN) architecture. It incorporates artificial neural network (ANN) connections and reservoir computing (RC) dynamics. Red, black, and blue colors indicate optical, electrical, and optical/electrical processing or interconnect, respectively. Interconnect lines denote connections in the ANN configuration, while dashed lines indicate the absence of direct connections in the RC configuration.

At its core, the NPPN comprises two main components: linear operation-weighting and nonlinear activation, supplemented by interconnects, memory, and photonic components. Ideal NPPN should consume minimal energy, possess high endurance, enable easy addressing in large, interconnected networks, provide signal gain and on-chip memory, and offer tunability, active dynamism, reconfigurability, and multi-functionality. Additionally, they should support large fan-in and fan-out, extensive interconnectivity, self-assembly capability, formation of 3D interconnects, and easy manufacturability in large quantities, all at low cost[57].

In the NPPN architecture (Figure 1), input signals are received as optical pulses (Ψ_j) from a multiplexer, representing incoming information. Optical synapses adjust the strength of connections through externally controlled feedback tuning to matrix multiply, simulating synaptic weights that may have on-chip synaptic memory (volatile or non-volatile) for storing. The weighted signals are then linearly summed (W_kj) that control PN dynamics. The PN execute the nonlinear activation, generating optical output signals (Ψ_o) based on modulation and interaction within the components. Moreover, the NPPN architecture exhibits similarities to an artificial neural network (ANN)[27] and reservoir computing (RC)[61] when serving as a reservoir in absence of dashed interconnects, due to the inherent parallelism supported in photonics through multiplexing. This versatility underscores the adaptability and potential of the NPPN architecture to serve as a robust platform for advanced neuromorphic computing tasks. Demonstrated NPNs utilized a variety of PN and synapses. However, one particular interest involves the creation of synapses and PN using identical technologies to facilitate seamless integration within expansive systems and enable the implementation of NN algorithms leveraging the innate physics of optical components.

2.2. Weights (Synapses): Linear Operation

Photonic synapses play a pivotal role in neuromorphic photonic computing, enabling the manipulation and processing of network signals. These synapses govern the strength of connections between neurons by assigning scalar multipliers, known as synaptic weights, to the signals transmitted across them. The weighted signals from upstream neurons are aggregated and modulated by the synaptic weights before being transmitted to downstream neurons, facilitating information and computation integration within the network. Various integrated photonic devices have been developed to implement weighted interconnection (Figure 2). These implementations can be broadly categorized into two groups based on wavelength and optical modes. However, they are not limited to and are extensively explored within the scientific community, particularly in RC. Mode-based approaches leverage interference between different optical paths and coherent input light to implement unitary matrix transforms for weighted interconnection (Figure 2a)[62]. These techniques utilize beam splitters, phase shifters, and MZIs to modulate the power and phase of light signals, enabling control over synaptic weights and interconnection matrices[39,63]. Additionally, advancements in cryogenic architectures and index-tuning mechanisms have expanded the scope of mod-based approaches (Figure 2b)[64], offering enhanced performance and flexibility in weight configuration. On the other hand, in the wavelength-based approach, signals are weighted in parallel using wavelength-division multiplexing (WDM) techniques [56] and tunable filters[48,65,66], such as MRRs[67]. These approaches, exemplified by architectures like broadcast-and-weight (B&W, Figure 2c)[48,68] and photonic cross-connect using indium phosphide (InP) semiconductor optical amplifiers (SOAs, Figure 2d)[69], although they differ in weighting mechanism, offer efficient methods for multiwavelength synapses for NPPN, with different architectures employing variations in weighted addition and attenuation mechanisms. RC uses weighted transient states of a nonlinear element with delayed feedback or time division multiplexing (TDM, Figure 2e)[70]. Non-volatile synapse implementations represent another significant advancement in neuromorphic photonic approaches (Figure 2f, g) [52,71], offering solutions that eliminate the need for electrical inputs for tuning. These approaches depend on optically induced alterations in materials like chalcogenides to regulate light propagation within waveguides. By leveraging the unique properties of non-volatile optical materials, these implementations address challenges related to electrical input/output (I/O) and heat dissipation, paving the way for more efficient and robust NPNs. A Variety of weighting mechanisms are explored[34,65,72,73,74,75,76,77] for developing weighted interconnects in neuromorphic photonics, to realizing high-performance and energy-efficient computing systems [38,56]. By leveraging the capabilities of integrated photonic circuits and novel materials, researchers are unlocking new possibilities for designing and implementing advanced NPNs with unprecedented functionality and adaptability.

2.2. Nonlinear Activation (Photonic Neuron)

PN represents a critical component to emulate the functionality of biological neurons for advanced information-processing tasks[40], which still face the challenge of achieving nonlinear activation while ensuring compatibility with network architectures, including fan-in[20] and cascadability[20]. Researchers have explored the multifaceted devices utilized for implementing PN (Figure 3 and Figure 4). Signifying two categories per the physical representation of signals: all-optical and optical/electrical/optical (O/E/O). In all-optical PN design, the neuron signal is represented solely through material properties or changes in optical susceptibilities.

While all-optical neurons offer inherent speed advantages over O/E/O implementations, they face significant challenges in achieving sufficient output strength to drive subsequent neurons. Solutions to this challenge involve integrating carrier regeneration mechanisms[85], wherein each neuron produces a renewed carrier wave modulated by its output signal, thereby enhancing its strength for downstream transmission. This approach has been demonstrated through various techniques, such as semiconductor carrier populations using cross-gain modulation (Figure 3a)[78], and structural phase transitions (Figure 3b)[44], enabling the realization of all-optical PN with enhanced functionality and cascadability. Despite introducing a new challenge of differentiating controller signals from controlled signals, carrier regeneration enables the amplification of output signals to drive downstream PN efficiently. Another avenue of exploration in PN design involves the O/E/O signal pathway, and optical signals are transduced into electrical currents and back into optical signals within the primary signal pathway (Figure 3c-g).

The nonlinear dynamics of PN using a superconducting electronic signal pathway (Figure 3c)[79,80] and a photodetector-modulator PN for MZI meshes (Figure 3d)[81] were proposed and implemented. Not only that, but this pathway enables the implementation of nonlinearities either in the electronic domain or through optical-electrical conversion stages utilizing modulators (Figure 3e, f)[82,84] or lasers (Figure 3g)[83]. By leveraging electronic components for nonlinear processing, O/E/O neurons can achieve high-bandwidth nonlinear transfer functions unconstrained by the characteristics of input signals to facilitate the generation of output signals more potent than the input essential for neural computation. Furthermore, spiking laser neurons (Figure 4), classified as all-optical or optical/electrical/optical (O/E/O), have showcased robust nonlinearity, carrier regeneration, and neural dynamics within a singular device. This is achieved by harnessing gain, cavity, and saturable processes. Spiking neurons have been demonstrated utilizing various technologies, including mode competition (Figure 4a, b)[88,89], graphene-saturable absorbers (Figure 4c)[90], saturable semiconductor media (Figure 4d, e) [91,92,95], and resonant tunneling diodes (Figure 4f)[93].

When comparing all-optical and O/E/O implementations of PN, the apparent advantage of all-optical approaches lies in their intrinsic speed, often ascribed to the comparatively sluggish carrier drift and current flow phases in O/E/O designs. However, recent developments have challenged this notion, revealing that analog O/E/O devices can exhibit comparable or even superior bandwidth and energy performance[25] compared to their all-optical counterparts. It is worth noting that in PN applying nonlinearities in the digital domain, the maximum system bandwidth is often dictated by the efficiency of the digital subsystem, underscoring the importance of efficient digital processing. Moreover, in PN design, a crucial consideration is the ability to configure and customize nonlinear transfer functions to align with specific NPN tasks. Analog PN offer flexibility in configuring transfer functions through electrical biasing[84]. Conversely, digital counterparts offer the flexibility to implement arbitrary transfer functions. Surprisingly, recent research advancements have demonstrated that programming techniques and neural training can adapt to analog photonic devices' inherent transfer functions, showcasing these devices' potential for efficient and adaptable neural computation[96,97]. Therefore, integrating PN with silicon photonics platforms holds promise for scalable and cost-effective implementation of large-scale on-chip neuromorphic photonic integrated circuits (nPIC) for information processing. Silicon photonics provide a stable ecosystem for research and development, facilitating advancements in technology road mapping, standardized fabrication processes, and broad accessibility to academic research[14]. This integration enables harnessing the scalability and feasibility of photonic integrated circuits (PIC) for realizing advanced NPNs.

3. Neuromorphic Photonic Networks

Data-centric AI-driven applications predict the urgent need for high-efficiency and ultralow power consumption solutions[98]. On-chip neuromorphic photonics has garnered attention as a complementary approach[36,99]. However, synergetic research is needed to identify optimized network topology and algorithms for NPNs[100]. ML and DL breakthroughs have propelled advancements across the hierarchy of networks, driven by various training algorithms that enable networks to adapt to diverse tasks[6,101].

Within this landscape, computational models inspired by the human brain’s structure and function, such as NNs[5,12], have gained prominence, leading to the exploration of various architectures and algorithms to enhance their capabilities. Moreover, integrating photonics into NN design has opened new avenues for achieving high-speed, wide bandwidth, energy-efficient computing supporting massive parallelism[100]. Figure 5 illustrates the evolution from essential ANNs to cutting-edge Spiking Neural Networks (SNNs) and photonic architectures, highlighting the algorithms and architectures that drive their functionality.

At the heart of ML is the concept of ANNs, computational models comprising interconnected nodes, or neurons, organized into layers. ANNs can learn complex patterns and relationships from data through a process known as training. Backpropagation, a fundamental algorithm ANNs use, adjusts synaptic weights based on the error gradient concerning each weight during training [5,102]. This iterative optimization process minimizes the difference between calculated and actual outputs, allowing the network to learn from labeled data. SNNs emulate the event-driven processing of the human brain, utilizing spikes to encode information and achieve efficient computation [44,59,103]. Neurons in SNNs integrate input spikes over time, generating output spikes when the membrane potential exceeds a threshold. This threshold-based firing mechanism, combined with spike-timing-dependent plasticity (STDP)[60], enables SNNs to learn from temporal patterns in data and exhibit robust adaptive behavior. DL represents a paradigm shift in ML, leveraging NN with multiple hidden layers to extract intricate features from raw data. Training through stochastic gradient descent (SGD) and its variants optimize the complex parameters of deep neural networks (DNN) by efficiently navigating the high-dimensional parameter space [6,104]. Convolution neural network (CNNs) are prime examples of DL architectures widely employed in image recognition and computer vision tasks[7,105]. CNNs utilize convolutional layers to detect hierarchical features in input data, then pool layers for dimensionality reduction and fully connected layers for classification. RNNs excel in sequential data by maintaining internal states or memory across time steps, making them suitable for natural language processing[9] and time-series prediction tasks[104]. RC offers a unique approach to neural network training, focusing on adapting only the readout layer while keeping the internal parameters fixed, simplifying the training process, and enhancing scalability, particularly in hardware-consertrained environments[58,70]. RC architectures, characterized by interconnected nonlinear nodes forming a fixed RNN, have shown promise in emulating complex behaviors and achieving efficient information processing.

Supervised[62], unsupervised[44], and reinforced[106] learning represent fundamental techniques in ML, each offering unique approaches to training. These methodologies provide versatile frameworks for optimizing the NN to accomplish specific tasks within various network topologies (Figure 5). Labeled data guides the NN to establish correlations between input data and corresponding output labels in supervised learning, encompassing classification [107], regression, and sequence prediction tasks. Unsupervised learning, conversely, entails training the NN on unlabeled data to unveil latent patterns or structures within the dataset, beneficial for tasks such as clustering, dimensionality reduction, and anomaly detection, enabling the extraction of meaningful representations from the data without predefined labels[44]. Reinforcement learning operates within a framework where an agent learns to navigate an environment by taking actions and receiving feedback through rewards or penalties[106]. This adaptive approach is instrumental in training NN for tasks requiring sequential decision-making, such as game-playing, robotics, and autonomous systems. As we navigate the landscape of network topologies, from ANNs and SNNs to NPNs, we witness the convergence of AI-driven technologies and cutting-edge research[99]. By embracing the complexity of NNs and harnessing the power of photonics, researchers are poised to unlock new frontiers in intelligent computing.

3.1. Neuromorphic Photonic Network: A Proposed Architecture

The NPN represents a pioneering advancement in DL architectures specifically tailored for photonic applications harnessing NPPNs as the fundamental building blocks. This innovative NN architecture integrates principles from CNNs[7,105] and RNNs[102,108], leveraging the unique properties of light for efficient computation. The network consists of two primary components: convolutional layers with activation functions followed by pooling layers and a recurrent feedback loop. The architecture is structured to exploit spatial and temporal correlations in input data, making it well-suited but not limited to tasks such as pattern recognition, classification, and sequential data processing.

A proposed approach to recurrent convolutional neuromorphic photonic networks (RecConv-nPN, Figure 5):

• Instead of traditional convolutional layers, the RecConv-nPN employs NPPNs to perform convolutional operations directly on input photonic signals, enabling them to extract spatial features from the input data and apply nonlinear activation functions simultaneously.
• NPPNs in RC mode, followed by unit weighting within the synapse for pooling operations (Figure 1), eliminating the need for different variety of layers. NPPNs dynamically aggregate information across spatial dimensions of the input data, facilitating down sampling and feature selection while preserving the advantages of photonic processing.
• A defining feature of the RecConv-nPN is its incorporation of a recurrent feedback loop enabled by the inherent memory (synaptic or delayed) properties of NPPNs. The output from the post NPPNs is fed back into the pre NPPNs in the network, allowing for iterative refinement of representations over multiple time steps and capturing temporal dependencies in the input data.
• Following the recurrent processing stage, the output is flattened for processing (say classification). This final layer utilizes standard classification techniques to map the learned features to specific output classes, enabling the network to make accurate predictions based on the input data.

3.1. Neuromorphic Photonic Approaches

State-of-the-art neuromorphic photonic approaches represent AI, leveraging the principles of neuroscience and photonics to develop energy-efficient and high-performance computing systems. Various NPNs proposed and experimentally demonstrated (Figure 6) the potential across different architectures. These approaches encompass different topologies (NPNs), weighting (synapse), and photonic signal representations (PN). Each approach offers unique achievements and challenges, paving the way for innovative solutions in neuromorphic computing (Table 1, Remarks). Solutions to the challenges of these approaches are promising for revolutionizing computing paradigms and enabling advanced AI applications in various domains.

RC leverages the complex dynamics of optical systems to perform computation. By utilizing interconnected delays between the fixed nonlinear nodes with multiple feedback loops, RC architectures demonstrate remarkable capabilities in emulating complex behaviours and processing information efficiently (Figure 6a)[61]. Recent advancements in RC have focused on integrating passive photonic elements, such as waveguides and resonators, to achieve scalable and high-performance computing[109]. Experimental demonstrations have showcased the ability of photonic RC to perform application-specific tasks such as spoken digit recognition, time-series prediction, and signal optimization, making it a promising candidate for various real-world applications[70,110].

Superconducting optoelectronic circuits (SOC) combine the advantages of superconducting electronics and photonics to achieve ultrafast and energy-efficient computing optimized for scalability (Figure 6b)[111]. These circuits exploit the superconducting nanowires, single-photon detectors, and capacitive micro-electromechanical system (MEMS)-based modulators to perform unprecedentedly efficient NN operations on a chip, paving the way for scalable and high-performance neuromorphic computing platforms with an energy cost associated with cooling at cryogenic temperature necessary for operation[64,80,112]. The MZI-based coherent nanophotonic circuits (CNC) leverage coherent light manipulation to perform complex computational tasks (Figure 6c)[62]. Integrating beamsplitters and phase shifters enables these circuits to control optical signals and facilitate high-speed processing, but they are bulky and require high driving voltages[39,63,81]. Experimental demonstrations have shown the potential of CNC for implementing neuromorphic algorithms such as in situ backpropagation[27], self-calibration[96], and asymptotically fault-tolerant programable photonic circuits[97], opening new avenues for ultrafast and energy-efficient computing[113,114,115].

Broadcast and weight networks (B&W) utilize the concept of WDM to perform matrix-vector multiplication efficiently (Figure 6d)[102]. These networks can achieve parallel processing of NN operations by modulating optical signals at specific wavelengths and tuning MRRs[48,66,116]. Experimental implementations have demonstrated its scalability and energy efficiency for solving differential equations, making them promising candidates for nPIC for computing[68,84].

Multiwavelength photonic neurosynaptic networks (MN) integrate PCM-germanium-antimony-tellurium(GST)- based PN (Figure 6e) [44] and synapses[52] on silicon nitride on a silicon dioxide platform with inherent synaptic memory associated with PCM with no energy requirement to maintain its states in case of offline learning. Online learning can be challenging, with individual PCM devices' endurance of 10 switching cycles requiring and requiring nJ of energy per cycle. These networks leverage the optical mode in a controlled manner to perform complex cognitive tasks. Experimental demonstrations have shown the feasibility of implementing MN for tasks such as pattern recognition and associative memory with supervised and unsupervised learning, highlighting their potential for next-generation nPIC [104,105]. On-chip diffractive neural networks (DO) harness the principles of diffractive optics to perform computational operations passively. By integrating diffractive elements in silicon slots filled with silicon dioxide (SSSD), these networks can achieve parallel processing of optical signals with minimal power consumption (Figure 6f)[107]. Experimental implementations have demonstrated the ability of DO to perform image classification[107,115].

3.1. Algorithms and Methods for Training Neuromorphic Photonic Networks

NPNs represent a burgeoning frontier ripe with promise. However, training algorithms face unique challenges due to the nonlinear nature of optical components and the need for efficient optimization techniques for a delicate balance between processing and memory access[99,100]. Unlike conventional AI algorithms deployed in software applications, developing customized training algorithms tailored for photonic hardware implementation could herald a transformative shift in this domain (Table 2). One of the key challenges in training NPNs lies in achieving precision (Table 2, Networks and Training). Traditional backpropagation algorithms, which are highly effective for training deep AI networks, require precise adjustments of weights based on slight variations in error gradients. However, most nanodevices used in NPN are inherently noisy, making it challenging to achieve such fine adjustments. Novel training algorithms that can adapt to the imperfections and variability of physical devices can cash on the inherent properties of photonics. Several strategies are proposed to address the precision challenge in photonic training, including the photonic generative network that harnesses noises of photonic hardware[117], a heuristic photonic recurrent algorithm for the Ising problem[118], and the photonic analog of the backpropagation leveraging adjoint variable methods that can significantly reduce complexity by simplifying the mathematical model associated with training[27] algorithm. Additionally, statistical optimization tools, including genetic algorithms[119], Bayesian optimization[120], nonlinearity inversion[108], and equilibrium propagation[121], have been investigated for optimizing weights in NPNs. These gradient-free algorithms show promise for training NPNs efficiently, particularly for classification tasks with different datasets[122]. Another essential consideration in NPN training is the need for weight-independent variations using external signals (electrical or optical). Furthermore, modifying existing algorithms to work effectively despite device imperfections is crucial, as demonstrated in self-calibrated[96] and asymptotically fault-tolerant[97] programmable NPNs. Unsupervised learning algorithms are particularly intriguing for NPNs and are highly adaptable to device imperfections. STDP is a popular unsupervised learning rule inspired by neuroscience[106]. In STDP, synaptic weights are

modified based on the temporal relationship between pre-synaptic and post-synaptic activities, making it suitable for implementation in NPNs (Table 2, Networks and Training)[44,111]. In addition to precision and unsupervised learning, optimizing nPIC presents challenges. Mitigating the complexity of probing local optical intensities across the circuit is essential for efficient training. Recent advancements propose on-chip reconfigurable[126] and control-free hardware-aware[122] training for NPNs. In contrast, Boolean learning via coordinate descent offers a practical and efficient alternative to error backpropagation[106], enabling high-performance NPNs with programmable connections. These novel training schemes (Table 2), proposed or experimentally validated for NPNs, underscore a dynamic landscape characterized by notable achievements, persistent challenges, and promising prospects also highlighted in state-of-the-art NPN demonstrations[104,123,124,125]. By addressing the challenges and exploring innovative training algorithms, NPNs hold the potential to revolutionize neuromorphic computing paradigms, offering energy-efficient and high-performance solutions for a wide range of applications (Figure 7).

4. Discussion

The demand for AI and its multifaceted applications is accelerating rapidly, exemplified by recent advancements such as DALL-E3 and Sora[53], which require massive computational power for training models with billions of parameters. Such tasks have relied on large-scale clusters of accelerators like GPUs or TPUs[98]. Neuromorphic photonics holds the potential to support these power-hungry processors, particularly for ASIC demands. Photonics leverage its inherent bandwidth, parallelism [20], picosecond latencies[25], and energy efficiency (measured in pJ/FLOP)[3]. Mainstream silicon photonic platforms offer a fundamental device library encompassing modulators, waveguides, and detectors, crucial for constructing signal pathways within diverse neuromorphic architectures. Innovations in conventional manufacturing processes, including the integration of PCMs and superconducting electronics, are being investigated to broaden the range of achievable architectures [45,46,47].

4.1. Exploring the Current State-of-the-Art: Challenges and Solutions

a. 

Synergistic Co-the integration of Photonics with Electronics

Silicon photonics has its advantages but still faces challenges yet to be solved. Coherent on-chip architectures utilizing components like MZIs are susceptible to thermal and fabrication processes, leading to variations in phase, interference, and diffraction. This results in inaccuracies in NPNs, emphasizing the need for strategies to mitigate these effects during training and tuning. Non-coherent architectures face challenges such as heterodyne crosstalk in MRRs and the requirement for numerous modulation resources, limiting scalability and energy efficiency. Efforts to address these challenges include exploring parallel arrangements of MRRs, leveraging microdisks, and facilitating increased integration density on-chip. Variations and reliability issues also present significant hurdles in silicon photonic devices, affecting their performance and stability. Therefore, electronic controllers for feedback and algorithm (calibration), direct current analog signal for biasing, analog to digital, and digital to analog for trans-impedance amplification pose a solution in managing photonic devices and ensuring the stable operation of NPNs. However, high latencies and frequency mismatches between electronic controllers and optical networks pose real-time, high-speed operation challenges.

Thus, integrating on-chip active electronics becomes crucial, and a significant challenge is the need for more complex on-chip electronic circuitry, which demands a higher density of electrical ports than optical ports, with the number of electrical ports scaling quadratically with the optical ports. Despite the challenges, the co-integration of CMOS electronics and photonics holds promise for overcoming many limitations by leveraging the advantages of both technologies while mitigating their drawbacks. Technologies including flip-chip bonding, wirebonding, and monolithic fabrication enable the integration of CMOS electronics optimized for digital control with silicon photonic chips[45,47], offering advantages such as increased interconnection density and reduced parasitic impedance.

b. 

On-chip Light Sources on Silicon Platform

Power efficiency is critical in neuromorphic photonics, with O/E/O conversions consuming considerable power. At the same time, all-optical neurons offer better power efficiency. Additionally, off-chip lasers contribute significantly to power consumption, necessitating research into power-efficient on-chip light sources. Efforts to tackle this challenge involve integrating light sources directly onto the silicon waveguide layer[127], employing methods such as rare-earth-element doping, strain engineering of germanium, and all-silicon emissive defects[80]. This is advantageous as it eliminates the need to send the optical signal off-chip for computation, which is particularly beneficial for neuromorphic photonics systems employing NPNs. The selection of lasers for neuromorphic photonics depends on the type of neuron involved. Multi-die techniques are applicable for systems utilizing modulator-class neurons, where the light source can be positioned outside the chip. However, precise integration of optical gain onto waveguides is essential for laser-class neurons.

4.2. Advancements and Future Directions in Scientific Inquiry

b. 

Fabrication Challenges

An essential aspect of advancing neuromorphic photonics involves enhancing the robustness of systems to environmental fluctuations and overcoming fabrication challenges. Analog circuits, typical in neuromorphic processors, often require trimming to address manufacturing variabilities and environmental sensitivities. In integrated photonics, resonant devices like MRRs pose challenges due to their sensitivity to variations[67]. One approach to mitigate these challenges is resonance trimming, which involves inducing changes in the refractive index of waveguides. Active trimming methods, including heating waveguides to environmental variability, require constant power input and fast response times. Alternatively, passive trimming methods utilize permanent or non-volatile techniques to adjust the refractive index of devices. These methods, including electron-beam-induced compaction[128] and strain of oxide cladding[129], offer solutions for correcting fabrication variations or preprogramming circuits to default states. Moreover, the integration of field-programmable PCMs allows for in-place reconfiguration[51], further enhancing robustness to fabrication discrepancies. In addition to addressing variability, advancements in neuromorphic photonics necessitate the development of analog-aware compilers to map application tasks to photonic hardware effectively[130]. Unlike traditional compilers for electronic systems, photonic compilers must account for idiosyncrasies inherent in representing signals in WDM lightwaves, including nonlinear distortion and limited dynamic range. Collaborative efforts within the academic community are underway to develop these compilers tailored for neuromorphic and programmable photonics[28,37,62,96,131], enabling efficient task mapping and optimization for photonic systems. By addressing these challenges and embracing innovative solutions, the field of neuromorphic photonics can enhance its resilience to environmental fluctuations and fabrication complexities, thereby facilitating the realization of robust and reliable neuromorphic computing systems.

c. 

Integration of Photonic Components

Efficient interfacing of analog processors with photonic hardware requires the development of photonic digital-to-analog converters (DACs) to avoid costly conversions, including digital-to-analog conversion and electro-optic modulation. DACs facilitate high sampling rates, precision, and low distortion while exhibiting reduced susceptibility to electromagnetic noise compared to their electronic counterparts. Photonic DACs can be realized through various methods, including optical intensity weighting of multiwavelength signals modulated with silicon MRRs incorporating depletion-mode PN junctions[132]. Another approach utilizes silicon-on-insulator traveling-wave multi-electrode Mach-Zehnder modulators [133]. Integrating photonic DACs enables the seamless conversion of digital signals to analog photonic signals, facilitating the implementation of advanced functionalities.

Advancements in tailored light sources hold immense potential for pushing the boundaries of neuromorphic photonics. Current efforts focus on developing chip-scale frequency combs utilizing soliton microcombs, a breakthrough achieved through CMOS-compatible PIC. By leveraging resonators designed to compensate for dispersion, these microcombs generate stable and broadband frequency combs with precisely aligned wavelengths, ideal for generating tailored input signals in parallel from a single source[41]. Moreover, the advent of lithium niobate-on-insulator modulators signifies a substantial advancement in photonic chip design, providing compatibility with silicon photonic devices and facilitating the integration of fast electro-optical modulators and efficient nonlinear optical elements on the same chip. These modulators feature high modulation frequencies and low voltage–length products, thereby reducing energy consumption and device footprint in nPIC[134].

c. 

Synaptic Memory

The integration of synaptic memory stands out as a critical area of exploration. Conventional approaches have typically depended on a blend of specialized photonic devices controlled by generalized electronic circuits. However, the absence of standard building blocks like high-level compilers, logic gates, and memory within current photonic platforms necessitates innovative strategies to incorporate memory into neuromorphic processors effectively. For specific ML and neuromorphic applications, such as DL inference, synaptic weights, once trained, may not require frequent updates. In these scenarios, non-volatile analog memory, such as in-memory computing utilizing PCMs, presents a promising solution[50,135]. By interfacing with digital electronic drivers, these memory systems enable real-time NN operation, precomputed weight storage, and direct inference task execution on the hardware[105]. Nevertheless, challenges persist in scenarios where temporary storage of neuron outputs is required, as seen in long short-term memory RNN. In such cases, alternative memory technologies, including digital or short-term analogue electronic memory with electro-optic interfaces to analog photonics, may be better suited. While analog memory may exhibit limitations in precision and noise compared to digital memory, recent studies have shown that even low precision can support effective deep RNN operations. As neuromorphic photonics progress, they are expected to adopt heterogeneous memory technologies, akin to modern computers, comprising various memory types within a single system[131,136]. This evolution may involve integrating electronic memory components alongside novel photonic memory technologies, offering non-volatile and reconfigurable capabilities essential for dynamic neural network operations.

4.2. Envisioning the Future of Neuromorphic Photonics: A Visionary Perspective

Neuromorphic photonics represents a fusion of advanced technologies and innovative architectural designs pivotal for constructing NNs. A prominent concept, the NPP [36], embodies this integration by leveraging state-of-the-art photonic packaging and emerging integrated photonics. The NPP seamlessly merges optical and electronic components within a system-in-package framework, enabling versatile signal processing and control functionalities. Here we deal two fundamental architectures, the NPPN, and the RecConv-nPN, as contributions to the evolving field of neuromorphic photonics, offering blueprints for realizing large-scale NNs. The convergence of photonics and neuromorphic computing signifies a transformative era of technological advancement and scientific exploration. Embracing perspectives and pioneering research endeavors will unlock the vast potential of neuromorphic photonics, reshaping the landscape of on-chip neuromorphic computing for application-specific AI.

5. Conclusion

Neuromorphic photonics basing its foot in photonic Integrated circuits represent a transformative approach to optoelectronic (grabbing the lower-hanging fruit first) or all-optical (too early to demand, requires technological advancement) hardware design, aiming to create systems that mirror the structure and functionality of NNs. This isomorphic relationship between NPNs and their biological counterparts promises remarkable capabilities and has sparked significant technological and societal interest. Over recent years, research in NPNs has witnessed rapid growth, leading to the exploration of diverse architectural concepts, PN models, training techniques, and network topologies. This diversity highlights the dynamic nature of the field, with ongoing efforts aimed at identifying optimal applications where photonics can outperform traditional electronic computing methods. Real-time applications requiring rapid decision-making are particularly promising arenas for deploying neuromorphic photonics. A key focus will be on scaling the integration of PN within single networks. Despite challenges such as the co-packaging of control electronics and light sources, advancements in scalable photonics platforms offer promising avenues for overcoming these obstacles. With modern integrated platforms and innovative ideas and devices for on-chip functionality, neuromorphic photonics is poised to push the boundaries of ML and information processing, unlocking new frontiers in AI and computational capabilities.