Critical Review of Neural Network Generations and Models Design

1. Introduction

Machine learning (ML) is a branch of artificial intelligence that explores computer algorithms capable of automatically improving with experience [1]. It involves building models using training data to make predictions or decisions without explicit programming. ML is utilized in various fields, such as computer vision, speech recognition, biotech, risk management, and cyber security, where traditional algorithms could be more complex and practical to develop. While machine learning is similar to computational statistics, it is not exclusively statistical learning, as mathematical optimization provides methods, theory, and application areas for ML. Data mining emphasizes exploratory data analysis through unsupervised learning [2,3]. Artificial neural network (ANN) is one of the most influential and popular machine learning techniques that simulate the human brain's behavior and structure. Neural networks comprise single/multi-layers of interconnected nodes to process data and transmit generalized information for predicting future conditions or making decisions [4,5]. ANN has shown remarkable success in various applications, including image enhancement, speech recognition, natural language processing (NLP), robotics, and finance. ANNs learn from raw data by adjusting the weights of node connections of complex datasets, resulting in more appropriate figures, which allows them to extract complex patterns and relationships. The field of ANNs has seen several generations of development, each characterized by a distinct set of architectures and algorithms that enable them to perform increasingly complex tasks [6,7]. The first-generation ANNs, which emerged in the 1950s and 1960s, were relatively simple models with only a few layers of neurons, mainly used for pattern classification and regression tasks [8,9]. However, their limited capacity to learn nonlinear relationships and extract meaningful features from complex data made them unsuitable for many real-world problems. In the 1980s and 1990s, second-generation ANNs, such as the multilayer perceptron (MLP) and the backpropagation algorithm, emerged as a more robust approach to machine learning [10,11]. These models used deep architectures with multilayer neurons and the ability to learn linear/nonlinear functions, enabling them to extract more complex patterns and relationships from data. This made them more suitable for many applications [12]. The third generation of ANNs, also known as deep learning, emerged in the 2000s. It is characterized by multilayer deep neural networks, which can learn complex patterns from large datasets [13,14]. This was made possible by the availability of large amounts of labeled data and advances in computing power, which made it feasible to train models with hundreds of layers and billions of parameters. Deep learning applies to many applications, like computer vision, text generation, natural language processing, and speech recognition [15,16,17,18].

Fourth-generation neural networks are a new development focusing on learning from fewer data and being more interpretable. Research areas in this field include neuro-symbolic AI, few-shot learning, and explainable AI [19,20,21,22]. It combines the concepts of ANN with symbolic reasoning techniques to achieve more interpretable results. Few-shot learning aims to develop neural network models that can learn from a small amount of data. Explainable AI focuses on developing neural network models that can explain their predictions. The Structured Probabilistic Model (SPM) and TreeNet are two models used for data processing and modeling. SPM can process structured and unstructured data and is suitable for applications, such as computer vision [23], NLP, and analysis of time series [24]. On the other hand, TreeNet is a robust model used for structured data in regression [25], classification, anomaly detection, and time series analysis applications.

The fifth generation of artificial neural networks (ANNs) can learn from structured and unstructured data. It can incorporate domain knowledge and prior beliefs, resulting in high-quality outputs. However, these models require even more computational resources than fourth-generation models and can be even more challenging to interpret. Spiking Neural Networks (SNNs) are typically considered fifth-generation neural networks, as they can model temporal dynamics and spiking behavior, allowing them to capture the complexity of biological neural networks more accurately [26,27]. Despite the impressive performance of ANNs in many applications, many challenges remain to be addressed, such as their interpretability, robustness, and scalability. Deep neural networks are complex and challenging to be understood by users to make the right decisions. Even slight changes to the input data can result in misclassification or incorrect outputs. This makes it limited to be implemented in some domains [28]. They require extensive data and computing resources, limiting their scalability and applicability in specific settings [29]. There are still various challenges and limitations that these technologies currently need to overcome. These challenges include:

➢

Bias and Fairness: A significant challenge with artificial intelligence and machine learning is ensuring that the algorithms are fair and unbiased. Since machine learning algorithms rely on data, they can inherit biases and prejudices from the data, resulting in unfair or discriminatory outcomes. Ensuring fairness and avoiding bias in machine learning is a complex and ongoing research area.

➢

Interpretability: Another issue with machine learning is the need for more interpretability or transparency of the algorithms. Many machine learning models are complex and nonlinear, making understanding how they arrive at their predictions or decisions challenging. This lack of interpretability can be problematic in domains such as healthcare and finance, where interpretability and explainability are crucial.

➢

Data Quality and Quantity: Machine learning algorithms depend heavily on the quality and quantity of data used to train them. Only complete, accurate, or biased data can positively affect the performance and reliability of the algorithms. Furthermore, in some domains, such as healthcare, acquiring large amounts of high-quality data can be a time-consuming and challenging task.

➢

Privacy and Security: Machine learning often involves using significant amounts of personal and sensitive data, raising concerns about privacy and security. Ensuring the confidentiality and integrity of data remains an ongoing challenge for machine learning practitioners.

➢

Computational Resources: Some machine learning algorithms can be computationally expensive and require significant computational resources, such as powerful GPUs or specialized hardware. This can make it difficult to apply machine learning in resource-constrained environments.

Addressing these challenges and limitations of artificial intelligence and machine learning is a future research and development directions. Researchers are exploring new directions in ANN research to address these challenges, such as incorporating biological inspiration, developing more efficient and energy-saving architectures, and combining ANNs with other AI techniques like evolutionary computation [30,31]. They are also exploring ways to improve the interpretability and robustness of ANNs, such as developing methods for explaining their decisions and designing models less susceptible to adversarial attacks.

Overall, ANN research is an exciting and rapidly-evolving field that generates new insights and solutions to some of society's most pressing challenges. This article offers an overview of the progression of ANN generations and their impact on computing data and improving the simulation of historical data. It also highlights the potential for these advancements to paving the way for developing the next generation of neural networks.

2. Neural Network Components

Artificial neural network (ANN) is one of the most influential and popular machine learning techniques that simulate the human brain's behavior and structure. Neural networks comprise three layers of interconnected nodes: Input, Hidden, and Output. Each neuron has a weight, a numerical value multiplied by the inputs as shown in Figure 1 [32,33].

The neural network consists of the following components:

• The input layers represent the input space vector, which processes data and transmits generalized information.
• The hidden layer is between the input and output nodes with flexible boundaries.
• The output layers are the final nodes of the neural network that consist of resulting information for predicting future conditions or making decisions.
• Weights are computed values based on a machine learning algorithm that adjusts their values depending on the difference between predicted results and the training datasets.
• An activation function that allows a neuron to fire (turn on/off) using a mathematical formula. The node with a value greater or equal to the value of the activation function will fire (give output). Otherwise, it will not fire.

3. Machine-Learning Approaches

There are three broad categories of machine learning approaches based on the type of "signal" or "feedback" available to the system [34,35,36]. Here are some of the main categories:

• Supervised Learning: It trains a model on labeled data that each is associated with known targets. The aim is to understand a function that maps unseen dataset inputs to a generalized outputs and predicting new outputs. Common supervised learning algorithms include linear regression (LR), logistic regression, decision trees (DTs), random forests (RF), and neural networks (NN).
• Unsupervised Learning: It finds patterns/clusters in unlabeled data without known targets using algorithms like k-means clustering, hierarchical clustering, PCA, and autoencoders.
• Semi-Supervised Learning: It is a hybrid approach that combines labeled and unlabeled data to improve accuracy, such as self-training, co-training, and multi-view learning. It uses labeled data to guide learning and unlabeled data to extract generalized features.
• Reinforcement Learning: It involves training a model through the interaction with its environment and rewarding or penalizing a model based on feedback. The objective is to develop a policy that leads to positive outcomes over time. Q-learning, policy gradient methods, and deep reinforcement learning are examples of algorithms used.
• Transfer Learning:
  It is used to leverage knowledge or experience from one task to improve another. It involves transferring relevant features from the source to the target domain and refining the model. Pre-trained neural networks and other algorithms can be used for transfer learning.

Figure 2 shows the different machine learning approaches, their subtypes, and some typical applications.

These machine learning approaches can be further categorized into other subtypes, and the choice of approach depends on different factors, including problem nature, amount of data, and actual outcomes [37,38].

However, neural networks can enhance transfer learning and leverage experience gained in several ways:

• Pre-trained Models: Neural networks can be pre-trained on large, diverse datasets to learn generic features or representations that can be transferred to other tasks or domains. For example, a neural network trained on millions of images can learn to recognize basic visual features such as edges, corners, and textures, which can then be used as the basis for other image recognition tasks. By leveraging the pre-trained features, the model can learn from smaller or more specialized datasets and achieve better performance with fewer data.
• Fine-Tuning: Neural networks can be fine-tuned on new or specific data to adapt to the target task or domain. This involves starting with a pre-trained model and updating the weights or parameters using the target data while keeping the learned features fixed or frozen. Fine-tuning can improve the model's performance on the target task and help it learn more quickly and effectively from limited data.
• Multi-Task Learning: Neural networks can simultaneously be trained on multiple related tasks to share knowledge and improve generalization. This involves designing a single model with multiple outputs or loss functions, each corresponding to a different task or objective. By jointly optimizing the model for all tasks, the network can learn more robust and transferable features and benefit from the experience gained across tasks.
• Domain Adaptation: Neural networks can be adapted to new domains or environments by adjusting their parameters or structure based on the differences between the source and target data. This involves identifying the relevant domain-specific features or biases and modifying the model to account for them. Domain adaptation can improve the model's performance in scenarios where the training and testing data may differ in some way, such as in medical imaging or natural language processing.

Figure 2. Machine learning approaches.

Figure 2. Machine learning approaches.

Neural networks can enhance transfer learning and leverage experience gained by learning generic features, fine-tuning new data, multi-task learning, and domain adaptation [39]. These techniques can improve neural network performance, efficiency, and generalization in various domains and applications.

4. Neural Network Design

Neural Network design refers to the different approaches and strategies to create effective neural network models for solving specific problems. In each of these cases, the neural network model design methodology involves selecting appropriate architectures, hyperparameters, and optimization algorithms to achieve optimal performance [40,41]. Critical analysis points could be considered when evaluating current neural network model design. A comprehensive analysis would need to consider the specific context and objectives of the research. Some possible critical analysis points that could be considered when analyzing current neural network model design methodologies:

• Architecture selection: Different neural network architectures have been developed for different applications. A critical analysis should assess the suitability of each architecture for a given problem, considering factors such as dataset size, task complexity, and the availability of computational resources.
• Hyperparameter tuning: Neural networks have many hyperparameters that must be tuned to achieve optimal performance. A critical analysis should examine the model's performance based on using different hyperparameters, such as the learning rate value, size of batching, activation function, and the number of layers.
• Regularization techniques: Overfit processing is a mutual problem in neural network training. Regularization techniques such as dropout and L1/L2 regularization can help to reduce overfitting. A critical analysis should evaluate the efficiency of such techniques to improve model performance and generalization ability.
• Optimization methods: Gradient descent and its variants commonly optimize neural network models. A critical analysis should assess the limitations of these methods, such as convergence issues and sensitivity to initialization, and explore alternative optimization techniques.
• Interpretability: One major hurdle for neural network models is their need for interpretation. A critical analysis should examine the current state-of-the-art interpretability techniques, such as feature visualization and attribution methods, and assess their effectiveness in providing insights into the model's decision-making process.
• Computational complexity: Training large neural networks can be computationally expensive and time-consuming. A critical analysis should consider the scalability of different neural network architectures and optimization methods and explore ways to reduce the computational complexity of training.
• Robustness: Neural network models are susceptible to adversarial attacks, where small input perturbations can cause significant output changes. A critical analysis should examine the robustness of different neural network architectures and explore ways to improve their resilience to such attacks.

5. Deep Learning Architecture

Deep learning is a class of algorithms and topologies that applies to a wide range of problems. Deep learning involves training neural networks utilizing data samples and rewarding those who perform well [42]. Several architectures and algorithms used in deep learning are wide and varied. We will review the most interesting architectures, which include convolutional neural networks (CNN) [43,44], recurrent neural networks (RNN) [45,46], Autoencoders [47,48], Generative Adversarial Networks (GAN) [49,50], long short-term memory (LSTM) [51,52], and A self-organizing map (SOM) [53,54].

• Convolutional Neural Networks (CNN): These are neural networks that use convolutional layers to extract relevant features from images or other types of data.
• Recurrent Neural Networks (RNN): These are neural networks that use recurrent connections to process sequences of data, such as time series or natural language [55,56].
• Long short-term memory (LSTM): Long short-term memory (LSTM) networks can learn long-term dependencies with sequence prediction problems. It uses memory cells to control information flow and solve the issue of vanishing gradients in RNNs.
• Autoencoders: These are neural networks that use unsupervised learning to learn a compressed representation of input data.
• Generative Adversarial Networks (GANs): These neural networks use a generator and discriminator to generate realistic data from random noise.
• Self-organizing map (SOM): A self-organizing map (SOM) is unsupervised learning aiming to reduce a high dimensionality of input space to construct a low-dimensional output space. It applies competitive learning between adjacent neurons using neighborhood function to maintain the input space topological properties.

Figure 3 shows the main deep learning architectures.

These methodologies can be combined and adapted to create customized hybrid neural network models for specific applications as presented in Table 1.

6. Training Data

It refers to a set of data extracted to train the ANN machine learning model. It is essential to train the ANN model on the input patterns and get the relationships and associations that help make predictions or decisions with new, unseen data. A training dataset has no minimum size but depends on the task's complexity and desired output [57,58]. However, we usually start with a reasonable dataset size and monitor model performance. This process can be repeated with different data sizes until getting high accuracy.

➢

Impacts of Training Data

Training data plays a critical role in determining the efficiency of a neural network. High-quality training data is crucial in examining the accuracy and effectiveness of neural network models. A large and diverse training dataset can lead to more accurate neural network models as it helps cover a broader range of input possibilities. Conversely, sufficient or low-quality training data can result in better performance and accurate predictions [59,60]. It is also essential that the training data is representative of the problem domain and the inputs that the neural network is expected to handle. The training data should reflect the real-world scenarios the neural network encounters. Otherwise, the neural network will not be efficient in simulating and generalizing unseen data.

➢

The quality and quantity of training data

The training data should be preprocessed to eliminate inconsistencies or biases. This includes techniques such as data cleaning, feature scaling, and normalization. Preprocessing the data this way can lead to more efficient neural networks by ensuring that the input data is consistent and that the neural network can better recognize relevant patterns and relationships. In summary, it is important to have a diverse, representative, and preprocessed dataset to ensure optimal performance [60].

Table 2 shows that the quality and quantity of training data can be determined through various methods. In order to evaluate the quality of training data, we need to use normalization and feature scaling for the training data. Data augmentation techniques, including rotation and translation of images. Domain experts can assess the relevance and accuracy of the data and domain knowledge about the problem to be solved. On the other hand, the quantity of training data can be assessed through cross-validation techniques to evaluate the model's performance on new data and learning curves to determine the rate of improvement with additional data. In addition, evaluate model complexity analysis to determine the optimal model size given a fixed amount of data and resource constraints such as available memory and computational power [61,62,63].

7. Neural Network Generations Development

Several generations of neural network developed that include the following.

First-Generation Neural Networks developed in the period between the 1950s and 1960s. The network architecture is simple, comprises only a few layers, and is primarily used in pattern and character recognition applications. One example is Perceptron, which consists of a single-layer feedforward neural network that is mainly used for binary classification tasks. Another example is the Adaline neural network, which consists of a single layer of linear neurons and a continuous activation function [64,65].

Second-generation neural networks were developed in the period between 1980s and 1990s. The neural network architecture is more complicated than first-generation networks and consists of multiple layers of neurons. They are utilized in several applications, such as speech recognition and image processing. One example is Multi-Layer Perceptron (MLP), which is a typical neural network model consisting of multiple perceptron layers and nonlinear activation functions. Another example is the radial Basis Function Network (RBFN), which uses radial basis functions to model complex patterns [66,67,68].

Third-generation neural networks appeared in the 2000s, known as deep learning. Using various activation functions, deep neural network architecture with multilayers can analyze complex patterns from large datasets. Deep learning has been applied in different applications, such as natural language processing, image processing, computer vision, and speech recognition [69,70]. The typical type is a Recurrent Neural Network (RNN) that models sequential features, such as natural language text classification or speech recognition [71,72]. Another type is Convolutional Neural Network (CNN), which is well-suited for image and video processing tasks [73,74].

Fourth-generation neural networks have recently developed that are characterized by their ability to learn faster because they use less data in the training process and are easier to understand. The current research directions of fourth-generation neural networks include neuro-symbolic AI, few-shot learning, and explainable AI. Neuro-symbolic AI combines neural networks and symbolic reasoning approaches to produce the results and make them easy to interpret. Furthermore, Few-Shot neural network models can learn using a small amount of data that mimics humans' abilities to learn new concepts from experience. Also, Explainable AI neural network models help develop more interpretable prototypes, making them more understandable to human users [75,76].

The fifth generation of artificial neural networks (ANNs) can learn from structured and unstructured data. It can incorporate domain knowledge and prior beliefs, resulting in high-quality outputs. However, these models require even more computational resources than fourth-generation models and can be even more challenging to interpret. Spiking Neural Networks (SNNs) are typically considered fifth-generation neural networks, as they can model temporal dynamics and spiking behavior, allowing them to capture the complexity of biological neural networks more accurately [77,78]. Table 3 presents several generations of neural networks.

Neural network generations have no strict definition, and it is possible for them to overlap. Research currently being conducted could lead to new developments and generations of neural network models. Figure 4 illustrates different neural networks architecture

Table 4 presents some examples of neural network models.

8. Transfer Function Impacts

The transfer function is a mathematical function used in neural networks to introduce non-linearity to the output of each neuron in the network. The choice of the transfer function can significantly impact the efficiency and accuracy of the model that is translating each input to a specific output. Sigmoid function is one of the common transfer functions used in neural networks, which maps the input to a value between 0 and 1 [79,80]. The sigmoid function is commonly used in classification tasks, producing outputs that can be interpreted as probabilities. The sigmoid function can cause neural networks to converge slower during training or not at all due to the gradients of the loss function (vanishing gradient) problem. To address the vanishing gradient problem, alternative transfer functions have been developed. For example, the rectified linear unit (ReLU) transfer function has become increasingly popular recently. ReLU maps the input to a value between 0 and infinity and has a much faster convergence rate than sigmoid, making it ideal for deep neural networks. However, the "dying ReLU" problem can affect the functionality of ReLU, where some node in the network can become "dead" and produce zero output. Other transfer functions commonly used include the hyperbolic tangent (Tanh) function, which maps the input to a value between -1 and 1, and the softmax function, used in multi-class classification problems [81,82].

The transfer function's choice can significantly impact a neural network's efficiency and accuracy. Choosing the best transfer function enhances neural network efficiency, which depends on the problem specifications and model characteristics. Therefore, we should involve various transfer functions and evaluate their performance based on the network and problem specifications. Table 5 presents the most used transfer function and their specifications.

9. Hidden layers Impacts

The number of hidden layers in a neural network implementation has the potential to impact its effectiveness and output in various ways. Therefore, more hidden layers in the neural network implementation can improve the accuracy and performance of generalizing the results. It allows us to solve more complex data relationships and attributes. However, On the other hand, adding too many hidden layers can lead to overfitting, and it becomes too specialized for the training data and may perform poorly with unseen data. Additionally, adding more hidden layers can increase the network's computational complexity and training time, which can be a disadvantage in some applications where speed is important. The best number of hidden layers depends on the task specification and dataset and may require some experimentation and tuning [83,84].

However, determining the best hidden layers number is challenging process as it depends on many features, such as the complexity of the task, the dataset's size, and the network's architecture.

It is generally recommended to use more advanced techniques, such as cross-validation or model selection, to empirically determine the optimal number of neurons in a given layer. However, the N / 2 + 3N rule may provide a useful starting point for determining the size of a neural network. It should be relied upon only partially and supplemented with additional analysis and experimentation to ensure that the resulting network is accurate and efficient [85].

However, there are some general guidelines and techniques that can help in this process:

• Start with a small number of hidden layers: It is recommended to start with a simple network architecture that includes one or two hidden layers and then gradually increase the number of hidden layers until the desired performance is achieved.
• Perform model selection: Use techniques such as cross-validation to evaluate the network's performance with different numbers of hidden layers. This can help select the best architecture for the specific task and dataset.
• Consider the dataset size: A simpler network architecture with fewer hidden layers is recommended to avoid overfitting if the dataset is small.
• Consider the complexity of the task: More complex tasks may require a larger number of hidden layers to capture the underlying relationships in the data.
• Regularization techniques: Techniques such as dropout and L1/L2 regularization can help prevent overfitting and reduce the required hidden layers.

10. Neural Network Design

Critical analysis points could be considered when evaluating current neural network model design. A comprehensive analysis would need to consider the specific context and objectives of the research. Some possible critical analysis points that could be considered when analyzing current neural network model design methodologies:

• Architecture selection: Different neural network architectures have been developed for different applications. A critical analysis should assess the suitability of each architecture for a given problem, considering factors such as the size of the dataset, the complexity of the task, and the availability of computational resources.
• Hyperparameter tuning: Neural networks have many hyperparameters that must be tuned to achieve optimal performance. A critical analysis should examine the impact of different hyperparameters on the model's performance, such as learning rate, batch size, activation function, and the number of layers.
• Regularization techniques: Overfitting is a common problem in neural network training. Regularization techniques such as dropout and L1/L2 regularization can reduce overfitting. A critical analysis should evaluate the effectiveness of these techniques in improving model performance and generalization ability.
• Optimization methods: Gradient descent and its variants commonly optimize neural network models. A critical analysis should assess the limitations of these methods, such as convergence issues and sensitivity to initialization, and explore alternative optimization techniques.
• Interpretability: One of the main challenges of neural network models is their need for interpretability. A critical analysis should examine the current state-of-the-art interpretability techniques, such as feature visualization and attribution methods, and assess their effectiveness in providing insights into the model's decision-making process.
• Computational complexity: Training large neural networks can be computationally expensive and time-consuming. A critical analysis should consider the scalability of different neural network architectures and optimization methods and explore ways to reduce the computational complexity of training.
• Robustness: Neural network models are susceptible to adversarial attacks, where small input perturbations can cause significant output changes. A critical analysis should examine the robustness of different neural network architectures and explore ways to improve their resilience to such attacks.

11. Spiking Neurons

Studies of the cortical pyramidal neurons have shown that the timing of individual spikes as a mode of encoding information is crucial in biological neural networks. A presynaptic neuron communicates with a postsynaptic neuron via trains of spikes or action potentials. Biological spikes have a fixed morphology and amplitude. The transmitted information is usually encoded in the frequency of spiking (rate encoding) and/or in the timing of the spikes (pulse encoding) [86]. Pulse encoding is more powerful than rate encoding regarding the wide range of information the same number of neurons may encode. Rate encoding is a special case of pulse encoding. The average firing rate can be computed if the spike timings are known. The early first-generation neurons developed in the 1940s and 1950s did not involve any encoding of the temporal aspect of information processing. These neurons acted as simple integrate-and-fire units that fired if the internal state (defined as the weighted sum of inputs to each neuron) reached a threshold. It did not matter when the threshold was exceeded. Translating this assumption to a biological perspective implied that all inputs to the neuron were synchronous, i.e., contributed to the internal state at precisely the same time and, therefore, could be directly summed. However, unlike biological neurons, the magnitude of the input was allowed to contribute to the internal state. This may have represented a primitive form of rate encoding because a more significant input (representing a higher firing rate of the presynaptic neuron) may cause the postsynaptic neuron to reach the threshold [87].

12. Performance Metrics

Performance metrics are used to evaluate the accuracy and effectiveness of Artificial Neural Networks (ANNs) in solving a given task [88,89]. Here are some commonly used performance metrics for ANNs:

➢

Accuracy measures the proportion of correct predictions the network makes on a test dataset. It is commonly used for classification tasks.

➢

Precision and Recall: They are used for classification tasks with multiple classes. Precision measures the proportion of true positives (correctly classified instances of a given class) to the total number of instances classified as that class. Recall measures the proportion of true positives to the total number of instances of that class in the test dataset.

➢

F1 Score combines precision and recall into a single value. It helps compare the performance of different models.

➢

The correlation coefficient (R) has a value between 0 and 1 determines the relationship between the two variables.

➢

Mean Squared Error (MSE) measures the average of the squared differences between the predicted and actual values.

➢

Root Mean Squared Error (RMSE) is the square root of the MSE and is also commonly used for regression tasks.

➢

Mean Absolute Error (MAE) measures the average absolute differences between the predicted and actual values. It is another commonly used metric for regression tasks.

➢

ROC Curve and Area Under the Curve (AUC) are used for binary classification tasks and measure the tradeoff between true and false positive rates at different thresholds. The ROC curve plots the true positive rate against the false positive rate at different thresholds, while the AUC measures the area under the ROC curve.

➢

Confusion Matrix shows the number of true positives, true negatives, false positives, and false negatives for each class in a classification task.

These are some commonly used performance metrics in ANNs, but other metrics may be more appropriate for specific tasks. Therefore, metrics in ANNs should be selected based on the task and goals of the model [90,91,92,93,94]. Here are some commonly used performance metrics for Artificial Neural Networks (ANNs) along with their formulas:

• Accuracy:

Accuracy = (TP + TN) / (TP + TN + FP + FN)

where TP = true positives, TN = true negatives, FP = false positives, and FN = false negatives.

2.

Precision:

Precision = TP / (TP + FP)

3.

Recall:

Recall = TP / (TP + FN)

4.

F1 Score:

F1 Score = 2 * (Precision * Recall) / (Precision + Recall)

5.

Mean Squared Error (MSE):

MSE = (1 / n) * ∑(i=1 to n) (yi - ŷi)^2

where n = number of instances, yi = actual value, and ŷi = predicted value.

6.

Root Mean Squared Error (RMSE):

RMSE = √(MSE)

7.

Mean Absolute Error (MAE):

MAE = (1 / n) * ∑(i=1 to n) |yi - ŷi|

8.

ROC Curve and Area Under the Curve (AUC):

The ROC curve is generated by plotting the true positive rate (TPR) against the false positive rate (FPR) at different classification thresholds. The AUC is the area under the ROC curve.

TPR = TP / (TP + FN)

FPR = FP / (FP + TN)

9.

Confusion Matrix is an indicator of the evaluation performance that provides a summary of the classification results. It is mainly, a table that shows the number of true positives, true negatives, false positives, and false negatives for each class in a classification task.

These are some commonly used performance metrics for ANNs along with their formulas. It's important to select appropriate metrics based on the task and goals of the model.

13. Conclusion

Neural networks are increasingly used to learn complex patterns and make accurate predictions. Designing an effective neural network model is a challenging task that requires careful consideration of architecture, optimization methods, and regularization techniques. This paper provides a comprehensive overview of the current artificial neural network generation state-of-the-art. It explores various learning approaches, identifies common pitfalls and limitations of existing design methodologies, and proposes future research directions. Despite impressive accomplishments, new developments in the field of neural network computation and architectures. However, Neural networks still suffer from different limitations that need to be addressed, including robustness, understandability, and scalability.

Some pitfalls and limitations of existing neural network design methodologies:

➢

Overfitting is one of the most common problems in neural network design is overfitting, which occurs when the model becomes too complex and learns the training data too well, resulting in poor performance on new, unseen data. This can be mitigated by regularization, early stopping, and data augmentation.

➢

Lack of Interpretability: Understanding how the model arrived at a particular decision or prediction can be challenging. This can be particularly problematic in domains where transparency and accountability are important, such as healthcare or finance. Several techniques have been proposed to address this issue, such as model distillation, attention mechanisms, and gradient-based attribution methods.

➢

Computational Complexity: Neural networks can be computationally expensive to train and evaluate, especially for large datasets and complex architectures. This can limit their practical use in specific domains or applications. Many researchers have explored techniques such as model compression, quantization, and knowledge distillation to address this limitation.

➢

Lack of Generalization: Neural networks can sometimes need help to generalize to new, unseen data that may differ from the training data in some pattern. This is particularly true when training datasets are limited or biased, or the model is too complex and prone to overfitting. Researchers have explored techniques such as transfer learning, domain adaptation, and adversarial training to address this.

➢

Some potential research gaps related to the development of neural network generations:

➢

First Generation Neural Networks:

➢

While the perceptron and Adaline were important early models in neural network history, their limitations in handling non-linearly separable data were quickly realized. However, it is still an open research question how these models can be improved to work on more complex data.

➢

Second Generation Neural Networks:

• MLPs are widely used in many applications. However, there is still a need for research on how to effectively choose the number of hidden layers and neurons and optimize the learning rate and other hyperparameters.
• RBFNs have been shown to work well in specific applications, but their effectiveness can be limited by the choice of radial basis functions and the number of hidden units.

➢

Third Generation Neural Networks:

• CNNs have achieved remarkable success in image and video processing tasks, but there are still challenges in extending their use to other domains, such as natural language processing.
• RNNs have shown promise in modeling sequential data. However, there is still a need for research on overcoming the vanishing gradient problem and handling long-term dependencies effectively.

➢

Fourth Generation Neural Networks:

• Neuro-symbolic AI is an emerging field that combines neural networks with symbolic reasoning techniques, but there are still challenges in effectively integrating these two approaches.
• Few-shot learning is an active area of research, but research is still needed to develop more robust and scalable models that can generalize to new domains.
• Explainable AI is an important area of research. However, there are still challenges in developing accurate and interpretable models and understanding how to communicate the explanations to end users effectively.

Finally, while significant progress has been made in developing neural network models, there are still many open research questions and opportunities for improvement and innovation, such as developing techniques to make ANNs more interpretable and explainable for applications where decisions must be justified or understood. Also, seek to improve transfer Learning to make it more accessible and adaptable to different domains and languages. In addition, researchers need to propose more robust models and increase security-critical defenses against adversarial attacks.