The Evolution of Deep Learning: From Perceptrons to Generative Foundation Models

1. Introduction

Deep learning has undergone a remarkable transformation over the past decades, evolving from early neural network models [1] in the 1940s to large language models (LLMs) [2,3] and foundation models in the 2020s. This evolution has enabled a wide range of real-world applications, including healthcare [4], banking and finance [5], surveillance [6], autonomous driving [7], and many others. The rapid progress of AI driven by deep learning has also renewed discussions about the potential emergence of artificial general intelligence (AGI) [8], a possibility that was once considered to be several decades or even centuries away.

Deep learning, a subfield of machine learning, has emerged as the dominant paradigm for learning complex representations from large-scale data. Its resurgence in the early 2010s was driven by the convergence of several factors, including the availability of massive labeled datasets, advances in GPU-based parallel computing, and innovations in neural network architectures and optimization techniques. Early successes in image classification [9,10,11], object detection [12,13], and image segmentation [14,15] demonstrated performance levels that were previously unattainable using traditional machine learning approaches. Unlike conventional methods that relied heavily on manually engineered features, deep learning models learned hierarchical representations directly from data, enabling improved scalability and generalization. Furthermore, the increasing computational capacity and memory of modern hardware made it feasible to train substantially larger and deeper networks, leading to a rapid expansion of deep learning architectures for both general-purpose learning and domain-specific applications.

Over the past decade, deep learning has evolved through a series of architectural breakthroughs, each addressing specific limitations in representation learning, sequential modeling, generative capability, scalability, or computational efficiency. These developments have resulted in several major architectural paradigms, including convolutional neural networks (CNNs) [9,10,16], recurrent neural networks (RNNs) [17,18,19], variational autoencoders (VAEs) [20,21], generative adversarial networks (GANs) [22,23], transformers [24,25], and diffusion models [26,27]. Each paradigm introduced novel mechanisms to overcome the limitations of previous approaches, from learning spatial hierarchies in images and capturing temporal dependencies in sequences to enabling large-scale generative modeling and multimodal learning. Continuous refinements of these architectures over years of research, together with advances in training strategies and scaling, have shaped the modern deep learning landscape.

The rapid growth of deep learning has resulted in numerous survey articles that review specific architectures, learning paradigms, or application domains. For example, [28] provides a comprehensive overview of neural network evolution, covering early shallow networks, multilayer perceptrons, backpropagation, and subsequent developments in convolutional, recurrent, and reinforcement learning architectures. Surveys on convolutional neural networks [29,30] review their fundamental components, including convolution operations, activation functions, pooling mechanisms, and regularization techniques, as well as their applications across different domains. Similarly, surveys on recurrent neural networks [31] summarize the progression from early recurrent architectures to gated variants such as LSTM and GRU. Recent surveys have explored the evolution and taxonomy of transformers [32], categorizing advances from architectural modifications and pre-training strategies to downstream applications. In addition, extensive reviews have examined generative models, including GANs [33] and diffusion models [34]. Complementing these surveys, deep learning textbooks [35,36] provide detailed explanations of fundamental concepts and advanced techniques, serving as valuable resources for researchers and practitioners.

While these surveys and books provide comprehensive coverage of individual architectures or application areas, they generally do not examine deep learning as an evolving field driven by a sequence of challenges, bottlenecks, and breakthroughs. Our work complements existing literature by adopting an evolutionary perspective, focusing on why key innovations emerged, what limitations they addressed, and how they influenced subsequent developments. We highlight major architectural milestones across neural networks, CNNs, RNNs, transformers, GANs, diffusion models, and foundation models, as illustrated in Figure 1. Rather than exhaustively reviewing every variant, we focus on seminal methods that established new research directions and summarize their advantages, motivations, and impact relative to prior approaches. Furthermore, we provide an overview of representative applications where these fundamental architectural advances have been adapted, demonstrating how foundational discoveries in deep learning have translated into practical solutions across diverse domains.

This paper is organized as follows. We begin with the fundamentals of neural networks in Section 2, introducing the basic principles of artificial neurons, network architectures, and learning mechanisms. Section 3 presents the evolution of convolutional neural networks (CNNs), describing their key architectural developments and their impact on visual representation learning. In Section 4, we review recurrent neural networks (RNNs), tracing their progression from early recurrent architectures to advanced variants such as LSTM, GRU, and modern sequence modeling approaches. Section 5 discusses variational autoencoders (VAEs), which established a probabilistic framework for representation learning and have become fundamental components in many generative and multimodal learning systems. Section 6 covers generative adversarial networks (GANs) and their contributions to high-quality data generation and image synthesis. Section 7 focuses on transformers, which have emerged as a dominant architecture underlying many modern foundation models across language, vision, and multimodal applications. Section 8 discusses diffusion models, a powerful generative paradigm that has achieved remarkable success in image, video, 3D, and multimodal generation. Section 9 examines application-specific adaptations of deep learning architectures, highlighting how fundamental discoveries have been tailored to address challenges in diverse domains. Finally, Section 10 concludes the paper by summarizing key evolutionary trends, lessons learned from past breakthroughs, and potential directions for future research.

2. Neural Networks

Neural networks are inspired by the interconnected structure of the human brain, which consists of billions of neurons that communicate through complex networks of connections. Biological neurons serve as fundamental information-processing units, receiving input signals, regulating their strengths and outputting appropriate responses that are propagated to downstream neurons. Billions of interconnected neurons process complex signals to produce intelligent responses [37]. Inspired by this biological mechanism, the first mathematical model of an artificial neuron was introduced in 1943 as a simple computational circuit [1], laying the foundation for the development of artificial neural networks. These early models later evolved into single-layer and multi-layer perceptrons, as illustrated in Figure 2.

1958 - Single Layer Perceptron: In 1958, a single-layer perceptron was proposed [38] that can solve classification tasks using simple thresholding. The model weights can be adjusted using the perceptron learning rule. A single-layer perceptron can be represented as:

$$\widehat{y}=\varphi (\mathbf{w}·\mathbf{x}+b),$$

(1)

where $\varphi$ is a nonlinear activation function, $\mathbf{w}.\mathbf{x}$ is the dot product between weight and input vector, and b is a constant bias. The output is given by

$$\varphi (\mathbf{w}·\mathbf{x}+b)=\left\{\begin{array}{cc}1,\hfill & \mathrm{if}\mathbf{w}·\mathbf{x}+b\ge 0\hfill \\ 0,\hfill & \mathrm{if}\mathbf{w}·\mathbf{x}+b<0.\hfill \end{array}\right.$$

(2)

The perceptron learning rule to update weights follows

$$w=w+\eta (y-\widehat{y})x,$$

(3)

where $\eta$ is the learning rate, y is the label, and $\widehat{y}$ is the prediction.

1969 - Winter in AI: Later, in 1969, it was demonstrated that single-layer perceptrons could only solve linearly separable problems [39] and were incapable of learning more complex functions, such as the XOR operation. Although multilayer perceptrons (MLPs) had the theoretical capability to overcome these limitations and solve nonlinear problems, effective training algorithms for such networks were not available at the time. These limitations contributed to a decline in interest and funding for neural network research, leading to the first AI winter.

1986 - Backpropagation: Training multi-layer perceptrons is not possible with the perceptron learning rule, as it introduces a chain of calculations. Backpropagation [40] updates network weights using the chain rule, allowing errors to be propagated across multiple layers. Weights are updated to minimize the error between predictions and ground truth by computing the gradient of the loss function with respect to the weights. The equation to update weights is given below:

$$\begin{array}{c}\hfill L={\displaystyle \frac{1}{2}}{(y-\widehat{y})}^2\\ \hfill W=W-\eta ·{\displaystyle \frac{\partial L}{\partial W}}\end{array}$$

(4)

where L is the mean squared error, W are the weights in the network, and $\eta$ is the learning rate.

1986 - Multi-Layer Perceptron: With the advent of backpropagation, training deeper neural networks became easier. Now, instead of single-layer networks, multi-layer neural networks [38] can be trained. A two-layer perceptron can be represented as:

$$y={\varphi }_2(\mathbf{w}\left({\varphi }_1(\mathbf{w}·\mathbf{x}+b)\right)+b)$$

(5)

1987 - Second Winter in AI: It became possible to train a multilayer perceptron using backpropagation; however, neural network performance plateaued, training was slow, and datasets were too small. Traditional machine learning models, such as support vector machines and decision trees, outperform neural networks in most tasks. This resulted in another winter in AI from the late 1980s to the early 1990s, characterized by reduced funding.

1990 - 2000 Proposing RNNs and CNNs: The field of neural networks has flourished significantly since the 1990s. Convolutional neural networks (CNNs) [41], recurrent neural networks (RNNs), and long short-term memory networks (LSTMs) have become popular due to the widespread adoption of backpropagation. Later, deep neural networks were termed deep learning, and numerous algorithms and architectural variants for diverse applications were proposed in the 2010s, as discussed below.

3. Convolution Neural Networks

Neural networks are not suitable for visual data. They lose spatial structure, cannot learn hierarchies, are not translationally invariant, and become overly parameterized for image data. Inspired by the visual cortex, convolutional neural networks (CNNs) were proposed in 1979 to efficiently model visual data. The first model to use convolution was the Neocognitron [42]. It introduced convolutional kernels and pooling layers, along with unsupervised learning, for visual recognition.

1989 - 1998 Early CNNs: Before CNNs, hand-crafted features were used to extract key patterns in visual data and later used for classification. This makes performance dependent on the accuracy of manual feature engineering. With the advent of automated learning, LeNet-1 [16] introduced automated feature learning in visual data via convolutions followed by a classification layer. This early end-to-end learning-based CNN architecture laid the foundation for subsequent progress. In 1998, LeNet-5 [41] was proposed as a successor to LeNet-1, featuring a deeper configuration and achieving significant performance improvements.

1998 - 2011 Winter on CNNs: During this era, manual feature extraction methods in computer vision regained prominence. CNNs are computationally expensive, require powerful hardware, are difficult to train, and require large datasets. All these factors halted the development of CNNs and shifted the focus toward using scale-invariant representations [43], including the histogram of oriented gradients [44], and bag-of-words [45].

2012 - Early Large-Scale CNNs: Hand-crafted features failed to perform well beyond a certain threshold on large-scale real-world datasets such as ImageNet. In 2012, AlexNet [9] employed CNNs and once again demonstrated that automated feature learning outperforms hand-crafted features. AlexNet achieved substantial performance improvements and could train on millions of model parameters for a million-scale image dataset, owing to advances in GPU technology. This paved the pathway for future discoveries in CNNs.

2013 - Visualizing and Understanding CNNs: CNNs were once thought to be black boxes, designed purely through trial and error. However, in 2013, ZFNet [46] demonstrated that CNNs possess unique properties that can inform the design of better architectures. It found that CNNs hierarchically learn features, with initial layers responding to edges and corners, middle layers emphasizing textures, and final layers focusing on object parts. ZFNet proposed using a deconvolutional network to visualize these features.

2014 - Deeper CNNs: VGG [10] and GoogleNet [11] were proposed by simply stacking convolutional layers to achieve further performance gains. In this regard, VGG [10] and GoogleNet [11] developed 19- and 22-layer deeper models, respectively. An example of a GoogleNet inception module is depicted in Figure 2.

2015 - Residual Learning: Deeper CNNs perform better up to a certain number of layers. Going beyond that limit reduces performance and can even be worse than the shallower models. The vanishing gradient was found to be the cause, whereby the error does not propagate back to earlier layers and decays across layers. To solve this, ResNet [47] proposed skip or identity connections by adding the previous layer output to the current layer output:

$$x_o=F\left(x\right)+x$$

(6)

where $x_o$ is the output, x is the output from last layer, and $F\left(x\right)$ is the current layer output. With residual connections, ResNet has demonstrated performance gains for model depths up to 152 layers [47]. The basic architectural residual block is shown in Figure 2.

2016 - Wider CNNs: ResNet facilitated training a deeper model; however, each doubling of the number of layers yields only a fractional improvement in accuracy. Additionally, deeper models become expensive to train. WideResNet [48] proposed wider networks by increasing channel count in each layer. They suggested that wider ResNets train faster and achieve similar or better performance than deeper models with thinner (fewer) layers.

2016 - 2018 Efficient CNNs: CNNs achieve better performance on various computer vision tasks, but deploying them on edge devices has been challenging due to the gigantic size of the models. This prompted researchers to develop smaller, more efficient CNNs with accuracy comparable to that of larger models. SqueezeNet [49], the basic building block, is shown in Figure 2 and proposed reducing the number of input channels and replacing a significant portion of the $3\times 3$ convolution kernels in a layer with $1\times 1$ kernels. This reduced computational cost while achieving an error rate equal to that of AlexNet [9]. MobileNet [50] introduced depthwise-separable convolution, a combination of depthwise and pointwise convolutions, to reduce computation and parameter count. ShuffleNet [51] splits channels into groups and applies the same pointwise group convolution filters to each group within a layer; an example is shown in Figure 2.

2017 - 2018 Neural Architecture Search: Designing optimal CNN architectures largely involves trial and error, with hyperparameters such as kernel size, depth, width, resolution, activation function, and others. Neural architecture search [52,53] eliminates this by automatically finding optimal architectures. Given a search space comprising different types of convolutions, activation functions, pooling operations, and other design elements, it identifies a small cell that is repeated multiple times and stacked to construct an architecture. This architecture is evaluated on a smaller dataset prior to large-scale testing. The validation accuracy from the initial test is used as a reward signal for the reinforcement-learning-based controller, which is optimized to propose a better cell structure.

2018 - Squeeze and Excitation: Conventional CNNs treat all channels in a feature map equally. However, not all channels are equally important; some may focus on key regions, whereas others may provide irrelevant information. To highlight essential channels, Squeeze-and-Excitation [54] recalibrates channel-wise feature responses. Channels are squeezed by a global average pooling operation, then excited by transformations and activation functions. The output is multiplied by the input to assign a higher weight to important channels, depicted in Figure 2.

2019 - Compound Model Scaling: Researchers previously considered scaling depth, width, or input resolution independently in CNNs. EfficientNet [55] proposed that these parameters are not independent and instead should be scaled uniformly. With every increase or decrease in input image resolution, the corresponding changes in depth and width should be scaled by the same factor. EfficientNet initially identifies scaling coefficients for these three hyperparameters by running a small grid search on a baseline model, and then uses these coefficients to scale the model according to the specified floating-point operations (FLOPS).

2019 - Convolution Block Attention Module: Squeeze-and-Excitation emphasizes key channels, whereas spatial regions still receive equal importance. However, images contain regions that are unusable for some specific tasks. The convolutional block attention module (CBAM) [56] extends Squeeze-and-Excitation to incorporate spatial attention. It consists of sequential processing through channel- and spatial-attention modules. Channel attention highlights important channels with salient features, whereas spatial attention highlights key spatial regions.

2020 - Non-Local Neural Networks: Transformers have been highly successful in natural language processing due to their attention mechanism. No similar mechanisms were identified in the visual data. As an alternative to transformers, non-local neural networks [57] were proposed in computer vision. Essentially, it is similar to self-attention, which computes similarities at a location with respect to other locations via a weighted sum over all locations.

2021 - CNNs meet Vision Transformers: Vision transformers (ViTs) [25], proposed in 2020, have gained significant attention in the computer vision research community. The building blocks of transformers, such as self-attention, were incorporated into ViTs, introducing the benefits of long-range dependencies and global context in computer vision. However, ViTs lack inductive bias, are data-hungry and computationally expensive. To address this, researchers have proposed architectures that leverage the strengths of both CNNs and vision transformers by combining them. These architectures employ convolution in different ways: within self-attention and token embeddings in [58]; before and after self-attention in [59]; and CNN layers before vision transformer layers in [60].

2022 - CNNs for 2020s: ViTs began to outperform CNNs in the early 2020s, achieving state-of-the-art performance across a range of tasks. Researchers proposed architectures to target both local and global information by combining CNNs and ViTs. However, these architectures lag behind CNNs in training and inference times, requiring larger datasets and greater computational resources. The study ConvNets for the 2020s [61] found that importing design choices from transformers into CNNs can make them perform as well as, or better than, transformers. They used larger kernel sizes, fewer activation and normalization layers, among other modifications, to develop a family of improved ResNet architectures.

4. Recurrent Neural Networks

Feedforward neural networks perform well on classification and regression tasks. However, for pattern completion or sequence modeling, they perform worse. This inspired researchers to modify neural networks into an architectural variant that can process sequential information as humans do. Recurrent neural networks (RNNs) [18,19,62], illustrated in Figure 2, were proposed as a solution to this problem, in which past states are propagated to future timesteps to share historical information.

1982 - Hopfield Networks: Hopfield networks [63] are the first networks to share states across different time steps. It is a single-layer fully connected neural network where each state is connected to all other states except itself. The network behaves like an associative memory to fix noisy inputs. The network is trained using the Hebbian learning rule.

1990 - Backpropagation Through Time: Backpropagation, proposed in 1986, enabled training neural networks. Later in 1990, backpropagation was adopted to train RNNs by unrolling the network into an acyclic graph, in which each time step shares parameters. This training technique for RNNs, known as backpropagation through time (BPTT) [64], has become the foundation for much of the progress in RNNs.

1990 - Early RNNs: The first RNN variant trained using BPTT was the Elman network [62]. It was designed to process sequential data to predict future states, using the hidden state from the previous time step along with the input. Another variant of RNN, called the Jordan network [65], forwards the output of the previous time step to the input.

1991 - Vanishing Gradients in RNNs: Soon enough, researchers realized the worse performance of RNNs on longer sequences. They found that the initial time steps failed to receive sufficient error information due to gradient multiplication across time steps. This chain of multiplications can make the gradient small or even vanish, hindering the learning process [66].

1997 - Long Short Term Memory: As a remedy to the vanishing gradient in RNNs, long short-term memory (LSTM) [18] was proposed in 1997. It retains information from early time steps via a top wire that connects to the entire sequence, thereby serving as long-term memory. This connection allows gradients to propagate across all time steps without obstruction. Additionally, LSTM introduced three gates, forget, input, and output, to restrict and let information flow over short- and long-term dependencies. An example of an LSTM building block is shown in Figure 2

1997 - Bidirectional RNNs: Conventional RNNs process information in a left-to-right direction, limiting their ability to understand sequences. In comparison, bidirectional RNNs [67] process information in both directions with access to past and future data, shown in Figure 2. This improves performance over unidirectional RNNs in applications such as text classification, speech recognition, and named-entity recognition, where the entire sequence of data is available for prediction.

2000 - Peephole LSTM: The long-term memory (cell state) in LSTMs is modified through the forget and input gates. These gates take the input and the previous hidden state to determine whether to forget or retain information in the cell state. In contrast, Peephole LSTM [68] proposed feeding the cell state into all gates. This technique allows LSTM gates to access long-term memory before updating the cell state. The Peephole LSTM achieved better performance than the vanilla LSTM on tasks requiring precise timing.

1998 - 2012 Winter on RNNs: Although numerous architectural variants were proposed between 1998 and 2012, overall progress in RNNs was largely stagnant. This is because they are harder to train, computationally expensive, and slower for prediction. During this era, conventional machine learning flourished, with Hidden Markov models (HMMs) [69] and Conditional Random Fields (CRFs) [70], among others, used for sequential modeling.

2013 - Speech Recognition using Deep RNNs: Before 2013, HMMs were dominant in sequential modeling, achieving higher performance than RNNs. Over time, advances in GPU technology have enabled the training of larger deep learning models, including deep RNNs. In 2013, deep RNNs [17] for speech recognition demonstrated that RNNs can outperform HMMs. This work employed bidirectional RNNs to capture information in both directions, i.e., past and future. It also identified deep RNNs, which stack multiple layers and offer better performance than shallow RNNs. This work revived interest among NLP researchers in RNNs, paving the way for subsequent developments.

2013 - Word Representations in Vector Space: Words were treated as individual entities in NLP. However, each word has synonyms and antonyms that establish relationships in the English dictionary. On this basis, word-to-vector (word2vec) [71] was proposed, which learns to bring similar words closer in vector dimensions or vice versa. Given a large corpus of textual data, the network was trained in an unsupervised manner to predict the center word from its context (i.e., the surrounding words) for a given word. These learned word vector representations are subsequently fed into RNNs as input for training on downstream tasks, such as text classification, named entity recognition, etc.

2014 - Sequence-to-Sequence Learning with RNNs: RNNs have been formalized as excellent sequence context learners in numerous studies. However, their use was limited to discriminative tasks. For tasks such as language translation, text summarization, speech-to-text, and others, a decoder is needed that can generate text based on the given context. Sequence-to-sequence learning with RNNs (seq2seq) [72] in 2014 introduced an end-to-end encoder-decoder architecture, in which both the encoder and decoder were RNNs, shown in Figure 2. The network operates such that the encoder maps the sequence to a contextual vector, which is then forwarded to the decoder for text decoding.

2014 - Gated Recurrent Unit: LSTMs address the vanishing gradient problem, but they are slower, more parameter-intensive, and require large amounts of memory for training. As a solution to this, the gated recurrent unit (GRU) [19] simplified LSTMs by introducing fewer gates, i.e., update and reset gates, which reduced the parameter count by 20-30%, making it faster and lighter. A GRU cell is shown in Figure 2.

2014 - Attention in RNNs: The seq2seq model assigns equal importance to all encoder states and generates a single context vector from them for the entire decoding process. However, input time steps in the sequence are not equally important, and the decoder cannot explicitly look in the past. Attention mechanisms in RNNs [73,74] overcome this by iteratively weighting encoder states for each decoder output. In this way, input words relevant to the current output state are given greater weight than others, thereby improving decoding performance. Additionally, the decoder can now look into the past to identify longer contextual dependencies, a capability that was not previously possible.

2015 - Convolutional LSTM: Individually, CNNs are known to perform well on spatial data, whereas RNNs learn patterns on sequential data. However, certain tasks require both spatial and temporal information processing. For such tasks, the convolutional LSTM (ConvLSTM) [75] processes the input using convolution to encode spatial information. The input, output, and cell state of ConvLSTM have an additional spatial dimension in addition to the temporal dimension.

2015 - Pointer RNNs: Some tasks, such as sorting, Delaunay triangulation, and the traveling salesman problem, do not require textual generation; instead, the output here is some kind of order of the input. For these tasks, pointer networks [76] were proposed as a specialized variant of sequence-to-sequence models. The encoder output is passed through the attention mechanism that generates a distribution over the input sequence. This is forwarded to the decoder, suggesting the possible output element at the current time step from the input.

2015 - Memory Network: Standard RNNs, LSTMs, and GRUs can retain only limited information in their internal states. In contrast, recurrent memory networks [77] introduced an external memory block to store information. These models were proposed for question answering and reasoning tasks. Given a query, the output is generated by reasoning sequentially over external memory, similar to how chatbots work. The model was trained in an end-to-end fashion with memory.

5. Variational Autoencoder

Inspired by the architecture of an auto-encoder and Bayesian inference, the variational auto-encoder (VAE) [20] learns to generate new data. The compressed latent from the encoder in VAEs is transformed into a probability distribution function (PDF) and later used to generate data stochastically. This is contrary to the auto-encoder, where the compressed vector is forwarded to the decoder for image regeneration. The encoder and the decoder of the VAE are two Bayesian networks with PDFs represented as $q(z\mid x)$ and $p(x\mid z)p\left(z\right)$.

2013 - Variational Autoencoder: VAE [20] is a generative model that learns to map the encoder latent representation into a Gaussian distribution. A vector is sampled from the learned Gaussian distribution and forwarded to the decoder for image generation. The training of a VAE optimizes the evidence lower bound (ELBO), which is equivalent to minimizing the difference between the input and the reconstructed image and the Kullback-Leibler divergence between the learned distribution and the Gaussian distribution. It is given as:

$$L\left(x\right)=E_{q(z\mid x)}[logp(x\mid z)]-D_{KL}(q(z\mid x)\parallel p\left(z\right))$$

(7)

where $p(x\mid z)$ is the generated image given the latent z, $\left(q\right(z\mid x)$ is the learned distribution from the compressed input, and $p\left(z\right)$ is the Gaussian distribution.

2015 - Conditional Variational Autoencoder: The generated image content in Vanilla VAEs is uncontrollable. The conditional variational autoencoder (CVAE) [21] encodes the input and decodes it using conditional information to control the output. The condition could be labels or other contextual information to control the image generation process.

2016 - Pixel Variational Autoencoder: The output images generated by vanilla VAEs are low quality and blurry. Pixel VAE [78] overcomes this by introducing an autoregressive decoding procedure. Instead of generating the entire image in a single step, the Pixel VAE generates each pixel based on the previously generated pixels, which are encoded in a compressed latent space.

2017 - Beta Variational Autoencoder: In vanilla VAEs, learned representations are entangled, which means we cannot modify latent representations to generate or change specific features in the output image. To disentangle latent representations, Beta VAE [79] introduced a beta factor that is multiplied by the KL-divergence term in the VAE loss function. For beta values higher than 1, the model is encouraged to encode latent representations closer to the Gaussian distribution, at the cost of lower image reconstruction quality. The latent dimensions of a beta VAE can be used to control individual concepts in the output image.

2017 - Vector Quantized Variational Autoencoder: VAEs suffer from posterior collapse due to the KL divergence term in the loss function. A powerful decoder can ignore the encoder’s latent representation and generate images using only the prior distribution. The vector quantized VAE (VQ-VAE) [80] proposed a discrete, learnable codebook embedding space, which is forwarded to the decoder for image generation. Additionally, once the VQ-VAE is trained, the prior is learned over the codebook embeddings using an autoregressive model. VQ-VAE fixes posterior collapse in VAEs and achieves higher-quality image, video, and audio generation.

2020 - Deep Hierarchical Variational Autoencoder: A single latent space in VAEs has limited capacity to represent complex data distributions. Deep hierarchical VAE (NVAE) [81] proposed a hierarchy of latent variables to encode complex data patterns. NVAE suggested residual modules to improve information flow across the model and depthwise convolution to keep memory usage within limits.

6. Generative Adversarial Network

Earlier image generation frameworks heavily relied on explicit data density estimation [20]. These models are mathematically sound; however, the data generation quality is low, with blurry images. Another image generation method, Generative Adversarial Networks (GANs) [22], was proposed in 2014 and, rather than modeling the data distribution, trains the model in an adversarial manner. This training strategy produces high-resolution and high-fidelity images.

2014 - Generative Adversarial Networks: GANs [22] are generative networks that learn to generate data through an adversarial process. It contains a discriminator and a generator, as shown in Figure 2: the discriminator distinguishes real from fake data, and the generator generates data that the discriminator classifies as real. The complete GAN objective for a minimax game between two players is given in Equation 8.

$$\begin{array}{cc}\hfill \underset{G}{min}\underset{D}{max}\mathcal{L}(D,G)=& {\mathbf{E}}_{\mathbf{x}\sim p_{\mathrm{data}}\left(\mathbf{x}\right)}[logD\left(\mathbf{x}\right)]\hfill \\ \hfill +& {\mathbf{E}}_{\mathbf{z}\sim p_{\mathbf{z}}\left(\mathbf{z}\right)}[log(1-D\left(G\left(\mathbf{z}\right)\right))].\hfill \end{array}$$

(8)

2015 - Conditional GANs: Data from GANs is typically generated by randomly sampling a latent vector from a normal distribution. This does not allow us to control the type of data we are generating. To control the data appearance, GANs are conditioned on auxiliary information, either class labels or other modality data, provided at the input to both the discriminator and the generator [82,83,84], as shown in Figure 2. The modified loss function for conditional GANs [82] is shown in equation 9.

$$\begin{array}{cc}\hfill \underset{G}{min}\underset{D}{max}\mathcal{L}(D,G)=& {\mathbf{E}}_{(\mathbf{x},y)\sim p_{\mathrm{data}}(\mathbf{x},y)}[logD\left(\mathbf{x}\right|\mathbf{y})]\hfill \\ \hfill +& {\mathbf{E}}_{\mathbf{z}\sim p_z\left(\mathbf{z}\right),y\sim p\left(y\right)}[log(1-D\left(G\left(\mathbf{z}\right|\mathbf{y})\right))].\hfill \end{array}$$

(9)

2015 - Deep Convolutional GANs: Deep Convolutional GANs (DCGANs) [23] replace fully connected layers in conventional GANs with convolutional neural networks (CNNs), employ batch normalization in the network, ReLU activation in the generator, and leaky ReLU activation in the discriminator. Through this series of modifications, DCGANs stabilize training and improve the quality of generated data.

2016 - Stacking GANs: Image quality can be enhanced by stacking GANs to progressively generate data. The Laplacian pyramid GAN (LAPGAN) [85] breaks the image generation process into multi-stage refinement steps. It scales up the image in pyramid-style stages, using independent GANs and adversarial training at each stage, as shown in Figure 2. The generator is given the previous-stage image and noise to generate a high-frequency (difference) image, which is then added to the input image to introduce finer details. StackGAN [86] generates a low-resolution image in the first stage, which is then forwarded as input to the second stage for high-resolution image generation. Stacked GANs (SGAN) [87] generate high-quality images by stacking many GANs in a top-down approach, where each GAN generates intermediate representations forwarded to the next stage until the final GAN.

2016 - Image-to-Image Translation: Specialized algorithms are designed to solve various computer vision tasks, such as semantic segmentation, image colorization, and edge-to-photo conversion. The end goal of these algorithms is to convert the input pixels into the desired output of pixels. Pix2pix [88] proposed a single conditional GAN model for this purpose. Here, the model translates the input into the output according to the training data. Pix2pix demonstrates that a single model can be trained for diverse tasks by simply changing the training data. Hence, it eliminates the need for a task-specific architectural design process.

2016 - Disentangled Representation Learning: The noise vector in GANs is highly entangled and does not correspond to individual semantic characteristics in the generated data. To explicitly control the attributes of the generated data, random noise is added to the latent representations of the attributes in GANs. InfoGAN [89] appends the latent codes representing image attributes to the noise. The model is trained to maximize the mutual information between the latent codes and the generator distribution. Disentangled representations are learned using a content and attribute encoder in [90] to transfer target attributes onto the source image content. Disentangled representation learning GAN (DR-GAN) [91] learns to generate new faces with different poses using pose latent, face features, and random noise.

2016 - f-GAN: Conventional GAN minimizes the Jensen-Shannon divergence. f-GAN [92] shows that GANs can be extended to any f-divergence, including the Kullback-Leibler (KL) divergence and Pearson divergences. This nice property of GANs allows adaptation to specific applications and conditions.

2017 - CycleGAN: It is an unpaired image-to-image translation from the source to the target domain. CycleGAN [93] assigns a discriminator and a generator to each of the forward (translating to the target) and backward (translating to the source) mappings, forming a cycle, as illustrated in Figure 2. The objective of CycleGAN [93] contains two adversarial losses and a cycle-consistency loss to avoid contradicting generator mappings, as given in equation 10. In equation 10, G and F are generators, $D_X$ and $D_Y$ are discriminators.

$$\begin{array}{cc}\hfill \mathcal{L}(G,F,D_X,D_Y)=& {\mathcal{L}}_{\mathrm{GAN}}(G,D_Y,X,Y)\hfill \\ \hfill +& {\mathcal{L}}_{\mathrm{GAN}}(F,D_X,Y,X)+\lambda {\mathcal{L}}_{\mathrm{cyc}}(G,F).\hfill \end{array}$$

(10)

2017 - ProgressiveGAN: Training GANs is quite unstable. Additionally, generating high-resolution, high-quality data from GANs is difficult. ProgressiveGANs [94] proposed a training methodology that starts with a 4×4-pixel low-resolution image. As the training progresses, new layers are added to the discriminator and the generator, increasing the image resolution to 1024×1024. The stepwise layer-wise addition enhances training stability and produces high-quality, high-resolution images [94].

2017 - Wasserstein GAN: GANs face many issues, such as mode collapse, vanishing gradients, and unstable training. Wasserstein GAN (WGAN) [95] proposed replacing the Jensen-Shannon divergence with the earth mover’s distance (EMD). WGAN improves training stability by avoiding mode collapse and vanishing gradients.

2017 - Video GANs: GANs have been shown to generate tiny videos by correctly modeling spatiotemporal dynamics through adversarial training. In GANs, the generator maps a noise vector to a video, and the discriminator distinguishes between real and fake videos. A noise vector is a combination of content and motion in MoCoGAN [96], foreground and background in [97], or a video class indicator augmented in [98] to introduce various dynamics.

2018 - BigGANs: Large-scale GAN training is unstable, and models perform worse than their shorter variants. BigGANs [99] showed that scaling up model size is possible if certain design choices are made. As scale increases, GAN training should use larger batch sizes, model capacity should increase in both depth and width, and normalization should be used in both the generator and the discriminator. BigGAN shows that large-scale GANs improve image fidelity and sample diversity.

2018 - StyleGAN: Instead of generating images from the latent code, StyleGAN [100] generates images using an intermediate representation generated by a mapping network containing fully connected layers. The mapping network disentangles the latent space, allowing control over the generated image attributes. Adaptive instance normalization (AdaIN) scales the representations to control the image styles added before each convolution operation. StyleGAN [100] also adds noise to each generator layer and mixing regularization, i.e., mixing two different mappings, which enables better control of generated images. An example of a StyleGAN generator is shown in Figure 2.

2018 - Self-Attention GANs: Convolutional GANs generate images by modeling short-term dependencies in feature maps based on the window size. To benefit from the cues in feature maps from different positions, self-attention GAN (SAGAN) [101] employs a self-attention layer in both the generator and discriminator. Applying spectral normalization to the generator with self-attention significantly improves SAGAN performance.

2022 - Diffusion-GANs: Training GANs is difficult and unstable, and they often suffer from mode collapse, whereas diffusion models achieve stable training with improved image quality. Researchers improve image generation quality by combining the benefits into a single model. Noise from the diffusion process is injected into the generated image by the GAN’s generator [102], thereby stabilizing training. Diffusion2GAN [103] distills a pre-trained diffusion model into a conditional GAN, speeding up the inference.

7. Transformers

Although numerous architectural variants of RNNs and CNNs have been proposed to accelerate training [19] and capture long-term dependencies [18,19], overall progress in sequential modeling has largely plateaued. It was not possible to parallelize the sequence processing with RNNs, which is the major bottleneck behind their slow training and inference. In 2017, Transformers [24] were proposed, enabling parallel processing of input data without recurrent connections, thereby removing dependencies between time steps. Transformers are one of the major breakthroughs in the history of deep learning, leading to remarkable discoveries in NLP and computer vision.

2017 - Transformers: To accelerate training and inference, the input sequence in transformers [24] is processed in parallel, without explicitly maintaining past-future relationships. To maintain a sequential relation, transformers add positional encodings to the input sequence. These embeddings could be cosine, sine, relative, or learned positional encodings. The contextual dependencies between time steps are learned through a self-attention mechanism replicated multiple times, resulting in multi-head self-attention (MHA). The self-attention module computes attention scores at a single time step and compares them with those across the entire sequence. However, this step breaks the causal relationship between the input sequence. To address this, a causal mask is applied to future time steps during processing. All computations within transformers are parallelizable, which makes them particularly advantageous compared with RNNs. The transformer architecture module is shown in Figure 2.

2018 - Generative Pre-trained Transformers: With the advent of transformers, training larger models on large-scale NLP datasets becomes possible. Generative pre-trained transformers (GPT) [104] are transformer models pre-trained in a self-supervised manner with a next-word prediction objective on a large corpus of 7000 books. The purpose is to train a generalizable model that can be fine-tuned for various downstream tasks, rather than training individual models for each task. The model demonstrated improved performance across diverse tasks and laid the foundation for future work on pre-trained transformers.

2018 - Deep Bi-Directional Transformers: Unidirectional transformers, such as GPT, limit contextual understanding because they process data only in a left-to-right direction. In contrast, Bidirectional Encoder Representations from Transformers (BERT) [105] process information in both directions, as shown in Figure 2, outperforming unidirectional models on token-level and sentence-level tasks. BERT pre-trains the model with a masked language modeling objective, in which it predicts masked tokens from unmasked tokens using self-attention, attending to tokens in a non-causal manner. The pre-trained model is later fine-tuned for downstream tasks.

2019 - Long Context Transformers with Recurrence: Transformers have a limited context window that is decided during training. This restricts the input context provided to the model, often splitting the larger input sequence into smaller segments and processing them independently. As a result, the overall contextual meaning is disrupted, potentially leading to incorrect output. To address this, Transformer-XL [106] proposed processing input subsets with shared hidden states from the previous segment. The input is encoded with relative positional encoding, in which each time step is represented by its position within the sequence. This strategy allows processing bigger sequences with recurrence in transformers.

2019 - Unified Text-to-Text Transformer: Building on works like BERT [105] and GPT [104], the unified text-to-text transformer (T5) [107] explored the transfer learning capabilities of large pre-trained models trained on large datasets. T5 unified all NLP tasks in a text-to-text format, in which both the input and output are text. The model is a bidirectional transformer trained with the masked language modeling objective. T5 achieved significant performance gains on numerous downstream tasks.

2020 - GPT-3: In NLP, GPT-3 [2] was a huge success. It is an extension of GPT; however, it has 175 billion parameters and was pre-trained on a gigantic dataset. GPT-3 demonstrated that larger models are effective few-shot and zero-shot learners, encouraging researchers to explore extremely large models.

2020 - Scaling Laws: The literature lacked guidelines on how performance varies with model and dataset size, as well as with model training time. Scaling laws for neural language models [108] identified a power-law relationship among model parameters, training dataset size, and training compute. It has been found that minor architectural tweaks have less effect on performance than scaling the parameter count and dataset size. To achieve higher performance, both the model parameters and dataset size should be increased accordingly.

2020 - Vision Transformers: The great success of transformers in NLP inspired researchers to experiment with them in computer vision. In 2020, vision transformers (ViTs) [25] were proposed, achieving state-of-the-art (SOTA) results in image classification on ImageNet. Similar to transformers in NLP, ViTs split the image into patches, add positional encodings, and process the input through self-attention blocks to learn global dependencies in visual data. An example of a ViT block is shown in Figure 2.

2020 - Data Efficient Image Transformers: ViTs [25] achieved SOTA results only when pre-trained on huge datasets, such as 300 million images, and then fine-tuned on ImageNet. This illustrates that ViTs are data-hungry and cannot outperform CNNs on smaller datasets. Data-efficient image transformers (DeiT) [109] proposed a training strategy that does not require large-scale pre-training; instead, the model is trained directly on ImageNet and achieves SOTA results. DeiT trains the model via knowledge distillation, where the teacher is a large model, typically a CNN, trained on large datasets, and the student is trained on ImageNet using classification and distillation tokens. The student model simultaneously learns two tasks: acquiring knowledge from the teacher and classifying input images.

2021 - Contrastive Language-Image Pre-Training: Large-scale pre-trained models, such as ViTs [25], GPT-3 [2], T5 [107], and others, were trained independently on textual or visual data. These models lack information cues that one modality can provide about the other. As a unification of image and text data, contrastive language-image pre-training (CLIP) [110] pre-trains a model with vision and text transformer encoders on a gigantic dataset (400 million image-text pairs), with the purpose of bringing each modality’s embedding closer to the other for similar concepts. CLIP achieved excellent zero-shot transfer capabilities in various computer vision tasks.

2021 - Swin Transformer: ViTs [25] employ self-attention, which calculates attention scores globally by comparing all patches of the input image. This increases memory usage quadratically, making processing higher-resolution images almost impossible. Building on this, shifted window transformers (Swin) [111] were proposed to restrict self-attention to local windows. Swin Transformers merge window patches from different layers across the network to enable information flow across windows. Additionally, the paper introduces a hierarchical architecture, in contrast to ViTs, but akin to CNNs for multi-scale feature processing. An example of the Swin transformer architecture is provided in Figure 2.

2021 - Switch Transformer: In transformers, all model parameters are active during both training and inference. However, the output of certain parameters is more important than that of others for some samples. In this regard, a mixture-of-experts (MoE) activates only a subset of the network parameters via the routing mechanism, as illustrated in Figure 2. Building on this, Switch transformers [112] proposed a trillion-scale MoE model. It reduced computational and communication costs by using switch routing, in which the input token is sent to only one top expert at a time, rather than 2 as in conventional MoEs [113]. When comparing this trillion-scale model with another large-scale model like T5 [107] while keeping the number of active parameters equivalent in both models, switch transformers outperform T5 across numerous language modeling tasks.

2022 - Flash Attention: Numerous faster and memory-efficient self-attention variants have been proposed in the literature [114,115]. These methods are approximation-based [114], sparse attention-based [115], or reduce the attention window size [111], which results in lower performance than full attention. In contrast, flash attention [116] proposed full self-attention that is faster, uses less memory, and performs equivalently by reducing memory read and write operations between the GPU’s on-chip SRAM and comparatively slower, high-bandwidth memory (HBM). It splits the matrix multiplication into blocks and executes it incrementally, rather than performing it all at once. During this, flash attention records statistics for online softmax, including the normalization factor and running maximum, and then reuses them to approximate the exact softmax output. It avoids the slow read operation from HBM to access the intermediate attention matrix, as in conventional self-attention.

2022 - Reinforcement Learning with Human Feedback: Larger models, such as GPT-3 [2], do not become good at following user intent with large-scale training. They produce harmful, unethical, and unfriendly text. In this regard, InstructGPT [117] fine-tunes GPT-3 on an annotated dataset containing prompts and desired outputs. The fine-tuned model is later used to generate another dataset of model responses, which the annotator then ranks manually for usefulness. This dataset is used to train a reward model to rank responses in a manner similar to a human labeler. In a final step, the model is optimized via reinforcement learning using proximal policy optimization (PPO), in which the policy’s rewards are generated by the reward model. This multi-stage model optimization with human feedback is called reinforcement learning with human feedback (RLHF), which has gained significant popularity in the chatbot and NLP research community and has led to substantial research in this area.

2023 - LLaMA: Closed-source models trained and commercialized from InstructGPT [117] and GPT-3 [2] were trained on closed-source datasets. Reproducing them was difficult, if not impossible, due to the enormous size of these models. LLaMA [3], an open-source model, showed that large language models (LLMs) trained on large public datasets can outperform closed-source models. They trained a series of models, and a 13B-parameter model attains better results than GPT-3’s 175B-parameter model on many benchmarks.

8. Diffusion Models

Inspired by non-equilibrium thermodynamics, diffusion models are image-generation models based on a Markov chain that gradually adds noise to the data at each time step (forward process) until the final image is random noise, and then learns to remove it (reverse process). During inference, the reverse process removes noise at each timestamp from a noisy image, typically sampled from a Gaussian distribution, to generate a content-rich image.

2020 - Denoising Diffusion Probabilistic Models: Denoising diffusion probabilistic models (DDPMs) [27] slowly add Gaussian noise to the data at each time step until the data are completely noisy. The forward process is represented by Equation 11.

$$\begin{array}{cc}\hfill q\left(x_{1:t}\right|x_0)& ={\displaystyle \prod _{t=1}^t}q\left(x_t\right|x_{t-1}),\hfill \\ \hfill q\left(x_t\right|x_{t-1})& =\mathcal{N}(x_t;\sqrt{1-{\beta }_t}{x}_{t-1},{\beta }_{t}I),\hfill \end{array}$$

(11)

where $x_t$ is the image at current time step, ${\beta }_t$ is the variance that increases with the time steps, $x_0$ is the original image. We can get a noisy image at any time step using a closed-form solution in equation 12:

$$\begin{array}{cc}\hfill q\left(x_t\right|x_0)& =\mathcal{N}(x_t;\sqrt{{\overline{\alpha }}_t}{x}_0,(1-{\overline{\alpha }}_t)I),\hfill \\ \hfill {\alpha }_t& =1-{\beta }_t,\hfill \\ \hfill {\overline{\alpha }}_t& ={\displaystyle \prod _{i=1}^t}{\alpha }_i.\hfill \end{array}$$

(12)

The reverse process denoises the image step by step, as shown in equation 13, where ${\mu }_{\theta }$ and ${\Sigma }_{\theta }$ are learnable parameters.

$$\begin{array}{cc}\hfill p_{\theta }\left(x_{t-1}\right|x_t)& =\mathcal{N}(x_{t-1};{\mu }_{\theta }(x_t,t),{\Sigma }_{\theta }(x_t,t)),\hfill \\ \hfill {\mu }_{\theta }(x_t,t)& ={\displaystyle \frac{1}{\sqrt{{\alpha }_t}}}\left(x_t-{\displaystyle \frac{{\beta }_t}{\sqrt{1-{\overline{\alpha }}_t}}}{ϵ}_{\theta }(x_t,t)\right).\hfill \end{array}$$

(13)

The model learns to predict the noise added at each time step in equation 14. The $ϵ$ is the noise and ${ϵ}_{\theta }$ is the trainable network in equation 14.

$$L={\mathbb{E}}_{x_0,ϵ,t}\left[\parallel ϵ-{ϵ}_{\theta }(x_t,t){\parallel }^2\right].$$

(14)

Generating a sample from the network is only a reverse process where random noise is converted into an image containing the content as given in equation 15:

$$x_{t-1}={\displaystyle \frac{1}{\sqrt{{\alpha }_t}}}\left(x_t-{\displaystyle \frac{{\beta }_t}{\sqrt{1-{\overline{\alpha }}_t}}}{ϵ}_{\theta }(x_t,t)\right)+\sqrt{{\beta }_t}z,$$

(15)

where $z\sim \mathcal{N}(0,1)$ is added for stochasticity at each time step. An example of the DDPM forward and reverse processes is shown in Figure 2.

2020 - Denoising Diffusion Implicit Models: Diffusion models generate samples by simulating a Markovian denoising process that spans numerous steps, significantly slowing inference. Denoising Diffusion Implicit Models (DDIMs) [26] proposed a non-Markovian, deterministic diffusion process with fewer steps, enabling sampling 50 times faster than DDPMs [27]. Given a noisy image $x_t$, DDIM predicts the clean image $x_0$ and then uses it to predict $x_{t-1}$. Progressive distillation [158] distills a pre-trained diffusion model into a student diffusion model using half as many sampling steps. In the next stage, the previously trained student becomes the teacher, and the process continues. The final diffusion model has significantly fewer inference sampling steps.

2021 - Score-based Diffusion Models: The diffusion process in score-based diffusion models [159] is a continuous-time stochastic process defined by stochastic differential equations (SDEs), unlike the discrete Markov chain modeling in DDPM. Instead of modeling the data probability distributions $p\left(x\right)$, score-based models learn to match the score of the data distribution, which is the gradient of the log-probability density function ${\Delta }_{x}log\left(p\left(x\right)\right)$. The forward SDE to map input data to noise is written as:

$$dx=f(x,t)dt+g\left(t\right)dW,$$

(16)

where $f(x,t)$ is the drift term, $gt$ is diffusion coefficient, and $dW$ is Weiner process. Sampling and reverse process SDE is defined as:

$$dx=\left[f(x,t)-g^2\left(t\right){\nabla }_{x}log{p}_t\left(x\right)\right]dt+g\left(t\right)d\overline{W},$$

(17)

the training equation becomes:

$$L={\mathbb{E}}_{x_0,x_t,t}{\parallel s_{\theta }(x_t,t)-{\nabla }_{x}logp(x_t\mid x_0)\parallel }^2,$$

(18)

where $s_{\theta }(x_t,t)$ is the score matching network.

2021 - Latent Diffusion Models: Instead of operating in pixel space, latent diffusion models (LDMs) perform diffusion in latent space, as depicted in Figure 2. LDMs comprise a variational autoencoder and a diffusion model, in which the encoder maps the input to a latent space used in both the forward and reverse diffusion processes. The latent from the reverse diffusion process is converted into an image by the decoder. LDMs are computationally efficient due to their lower dimensionality and offer greater expressivity and control through the semantic representations captured in the latent space [160,161].

2021 - Improved Likelihood Diffusion Models: Diffusion models achieve better sample quality; however, their log-likelihood scores underperform those of other generative models. Higher likelihood values tend to capture different modes of the data distribution and to learn better feature representations. Improved DDPM (IDDPM) [162] introduced tweaks, including learning the noise variance, improving the noise schedule, and reducing gradient noise, resulting in improved log-likelihood. The likelihood weighting is suggested in [163] as a weighted combination of score-matching loss that upper-bounds the negative log-likelihood.

2021 - Classifier Guided Diffusion Models: Diffusion models generate high-quality, targeted samples based on the information provided. Classifier gradients are used to guide sampling in unconditional diffusion models [164], as shown in Figure 2. The classifier predicts the class of the noisy image at each time step, and its gradients are mixed with the predicted score to guide the generation process. The $x_{t-1}$ is sampled from $\mathcal{N}(x_t;{\mu }_{\theta }+s{\Sigma }_{\theta }{\nabla }_{x}log{p}_{\varphi }(y\mid x_t),{\Sigma }_{\theta })$, where s is scaling factor and $log{p}_{\varphi }(y\mid x_t)$ is the classifier.

2022 - Text-to-Image Diffusion Models: Text-image synthesis models are trained on text-image data pairs. However, text-image pairs are less abundant than text data. As a result, text-only models, such as LLMs trained on terabytes of text, can encode text more effectively. Based on this, Imagen [165] proposed using a text-only T5-XXL [107] LLM to encode textual prompts for image generation. The diffusion model is conditioned on T5-XXL text embeddings for image generation, followed by super-resolution models to produce high-resolution images. This image-generation pipeline achieved higher image quality than models trained on text-image pairs.

2022 - Classifier-Free Guidance in Diffusion Models: Here, diffusion models are a combination of conditional and unconditional diffusion models. The conditional loss is used for a fraction of the training data. Scores from conditional and unconditional diffusion models are mixed during inference based on the guidance scale [166]. This is illustrated in Figure 2.

2022 - Consistency Models: Conventional diffusion models [26,27] require hundreds of denoising steps before generating an image. This makes diffusion models slow in inference and infeasible for practical applications. In contrast, consistency models [167] have proposed generating images in a single step. Consistency models learn to map any point on the probability flow ordinary differential equation (ODE) trajectory to the input data. The model is trained by ensuring different noisy time steps are denoised to the same output through consistency-preserving loss.

2022 - Video Diffusion Models: Diffusion models were primarily proposed for image generation. Later, they were extended to video generation in [168]. To generate 3D data, the 2D UNet in diffusion models is replaced with a 3D UNet. These networks can process only a limited number of time steps at a time; however, to generate an arbitrary number of frames, it is possible to sample from the model sequentially.

2023 - Conditional Control Text-to-Image Diffusion Models: Text-to-image diffusion models generate images with content consistent with the prompt. However, these models generate content arbitrarily from the prompt, without explicit control over pose, position, color, shape, etc. To provide additional control, ControlNet [169] incorporates both image and text conditions into the model, enabling semantic conditioning via text and structural conditioning via images.

9. Application Specific Architectures

Beyond fundamental research in deep learning, huge progress has been made in application-specific areas. Applications such as object detection, image segmentation, image generation, chatbots, and others have had a massive impact on the real-world use of deep learning. Deep learning architectures developed for these applications incorporate primary concepts and domain-specific knowledge to improve performance. This section provides a summary of highly influential application-related papers proposed in the literature in Table 1. These papers brought great success in their fields and proposed new directions in solving problems. Much of the research in different areas draws inspiration from these architectures, and these are included in the new research as building blocks in one way or another. Table 1 briefly discusses the proposed approach of these architectures. It also provides information on the architectural framework used in these methods, as well as performance, parameters, applications, and other details. Table 1 is organized chronologically to illustrate the evolution and progress over time.

10. Conclusion

We presented an evolutionary timeline of deep learning in this paper, from early neural network developments to transformers and diffusion models. We discussed breakthrough developments and highlighted the importance of the proposition. This paper also provided a summary and other details on highly influential application-specific architectures. We believe this work will help the community quickly review major turning points in the history of deep learning.