Deep learning-based question answering systems have transformed the discipline of natural language processing (NLP) by automating the extraction of answers from textual data. This survey paper provides a captivating overview of these systems, exploring methodologies, techniques, and architectures such as recurrent neural networks (RNNs), BERT model, and transformer models. Extractive and generative approaches are examined, alongside the challenges of handling complex questions, managing noisy input, and addressing rare or unseen words. This survey serves as a stimulating reference, offering valuable insights to researchers and practitioners, fueling innovation and advancement in question answering systems within NLP.

Chatbots are extensively needed in customer services to handle customer inquiries, such as tracking orders or providing information about products and services. One of the most reliable implementations of chatbots is using the common architectures of LSTM networks named Seq2Seq networks. The networks are using an encoder and a decoder. Seq2Seq chatbot is a type of chat system that is professional enough to pass the Turing test. The Turing test is a way of deciding the accuracy of the machine by examining its response, it should appear like a human response. In this research, we will introduce a novel architecture that can pass the Turing test. The seq2seq Accuracy is improved by making incremental training to the chatbot. The new proposal provides higher accuracy and high similarity to human chat responses.

Imitation Learning is a discipline of Machine Learning primarily concerned with replicating observed behavior of agents known to perform well on a given task, collected in demonstration data sets. In this paper, we set out to introduce a pipeline for collecting demonstrations and training models that can produce motion plans for industrial robots. Object throwing is defined as the motivating use case. Multiple input data modalities are surveyed, and motion capture is selected as the most practicable. Two model architectures operating autoregressively are examined -- feedforward and recurrent neural networks. Trained models execute throws on a real robot successfully, and a battery of quantitative evaluation metrics is proposed, including extrapolated throw accuracy estimates. Recurrent neural networks outperform feedforward ones in most respects, with the best models having an assessed mean throw error on the order of 0.1...0.2 m at distances of 1.5...2.0 m. The data collection, pre-processing, and model training aspects of our proposed approach show promise, but further work is required in developing Cartesian motion planning tools before it is suitable for application in production.

The purpose of this study is to ​​explore the use of a custom Transformer model in brain-computer interfaces (BCIs) that translate the neural activity present when an individual with limited verbal and fine-motor skills attempts to handwrite. As found in previous studies, Transformers have performed better than recurrent neural networks (RNNs) in translation tasks which are similar to its usage here in decoding neural signals into intended handwritten text. Due to the known benefits of a Transformer, the hypothesis was that the Transformer would show promise in the context of a BCI through the recorded metrics. The neural signals of a tetraplegic individual, when they attempted to handwrite, were provided by existing research. Four trials were conducted using data with or without augmentation which the model used when training to separately minimize training loss and validation loss. When comparing the results of a BCI with the implementation of a Transformer model with the original RNN BCI (the original data source), the Transformer model performed less favorably across all four trials. Although the results do not indicate that the Transformer model currently outperforms an RNN BCI, it is important to note that further testing of the model's capabilities (such as training it with a larger and more preferable dataset and/or for longer, comparing training times between the RNN and Transformer, and/or seeing how the Transformer is improved with an offline autocorrect feature) is necessary before determining whether Transformers can enhance communication in this manner.

Neural decoding from spiking activity is an essential tool for understanding the information encoded in population neurons, especially in applications like brain-computer interface (BCI). Various quantitative methods have been proposed and have shown superiorities under different scenarios respectively. From the machine learning perspective, the decoding task is to map the high-dimensional spatial & temporal neuronal activity to the low-dimensional physical quantities (e.g., velocity, position). Because of the complex interactions and the abundant dynamics among neural circuits, good decoding algorithms usually have the capability of capturing flexible spatiotemporal structures embedded in the input feature space. Recently, the Transformer-based models are widely used in processing natural languages and images due to its superior performances in handling long-range and global dependencies. Hence, in this work we examine the potential applications of Transformers in neural decoding and introduce two Transformer-based models. Besides adapting the Transformer to neuronal data, we also propose a data augmentation method for overcoming the data shortage issue. We test our models on three experimental datasets and their performances are comparable to the previous state-of-the-art (SOTA) RNN-based methods. In addition, Transformer-based models show increased decoding performances when the input sequences are longer, while LSTM-based models deteriorate quickly. Our research suggests that Transformer-based models are important additions to the existing neural decoding solutions, especially for large datasets with long temporal dependencies.

Breast Cancer diagnosis is one of the most studied problems in the medical domain. In the medical domain, cancer diagnosis has been studied extensively which instantiates the need of early prediction of cancer disease. For obtaining advance prediction, health records are exploited and given as input to an automated system. This paper focuses on constructing an automated system by employing deep learning based recurrent neural network models. A stacked GRU-LSTM-BRNN is proposed in this paper that accepts health records of a patient for determining possibility of being affected by breast cancer. Proposed model is compared against other baseline classifiers such as stacked Simple-RNN model, stacked LSTM-RNN model, stacked GRU-RNN model. Comparative results obtained in this study indicate that stacked GRU-LSTM-BRNN yield better classification performance for predictions related to breast cancer disease.

Corona Virus Infectious Disease (COVID-19) is newly emerging infectious disease. This disease is known to the globe in early 2019. Poor status of mental health is often caused by unemployment, ongoing socio-economic condition. Poor mental health may even accelerate the process of panic attack. It has been happening rapidly during COVID-19. It has a great effect on human health. This paper utilizes multiple related factors those have impact on causing panic attack. Recurrent Neural Network (RNN) based framework is utilized in this paper that assembles multiple RNN layers along with other parameters into a single platform. This method is implemented by capturing interfering factors and predicts panic attack tendency of people during COVID-19. Early prediction of panic attacks may assist in saving life from unwanted circumstances.

When implementing SVMs, two major problems are encountered: (a) the number of local minima increases exponentially with the number of samples and (b) the quantity of required computer storage, required for a regular quadratic programming solver, increases by an exponential mag-nitude as the problem size expands. The Kernel-Adatron family of algorithms gaining attention lately which has allowed it to handle very large classification and regression problems. Howev-er, these methods treat different types of samples (Noise, border, and core) in the same manner, which causes searches in unpromising areas and increases the number of iterations. In this work, we introduce a hybrid method to overcome these shortcomings, namely Optimal Recurrent Neu-ral Network Density Based Support Vector Machine (Opt-RNN-DBSVM). This method consists of four steps: (a) characterization of different samples, (b) elimination of samples with a low probability of being a support vector, (c) construction of an appropriate recurrent neural network based on an original energy function, and (d) solution of the system of differential equations, managing the dynamics of the RNN, using the Euler-Cauchy method involving an optimal time step. The RNN remembers the regions explored during the search process thanks to its recurrent architecture. We demonstrated that RNN-SVM converges to feasible support vectors and Opt-RNN-DBSVM has a very low time complexity compared to RNN-SVM with constant time step, and KAs-SVM. Several experiments were performed on academic data sets. We used several classification performances measures to compare Opt-RNN-DBSVM to different classification methods and the results obtained show the good performance of the proposed method.

Coastal wave modeling and forecasting are essential in oceanography, sustainable marine energy, and ocean engineering. Precise forecasting of wave speed and direction are crucial for offshore operations, marine energy, risk management, environmental management, coastal and sustainable maritime management. The coastline of Brunei Darussalam can generate between 15 and 126 Giga Watt of wave energy. In this experimental research, we used two numerical approaches, the finite difference, and spectral element methods, to model and simulate wave speed and direction. We calculated the mean error between numerical and analytical solutions. We proposed a novel, promising, univariate time series forecasting model by combining the Long Short-Term (LSTM) with KerasTuner hyperparameters tuning and optimization techniques. This method helps us to improve the accuracy and efficiency of time-series forecasting.. The experimental data was computed from high-precision Acoustic Doppler Current Profiler (ADCP) sensor data. This research is part of the preliminary feasibility analysis of wave energy production in Brunei Darussalam and net zero commitment for a sustainable environment. Seven independent forecast experiments were performed for wave speed and direction in degree and radian units for 1, 3, 6, 8, 10, 12, and 24 hours. Mean squared error (MSE) was adopted as a metric for both training and testing. The experimental results reveal that the wave speed forecast has the lowest MSE compared to direction, regardless of the unit of measure, but has a longer duration. In addition, the direction forecast in the degree unit has the lowest errors compared to the unit of radians; the latter has a longer running time than the former. The model has delivered optimal results throughout the experiments with minor training and test errors. We conducted a thorough evaluations on two benchmark time series datasets, which include the study dataset and air quality index dataset, to validate the performance of the proposed model with other models. The proposed model outperforms cutting-edge forecasting models, such as the conventional LSTM, ARIMA, and Prophet. The model has the slightest forecast error compared to the existing literature’s result.

SQL injection attacks are one of the most common types of attacks on web applications. These attacks exploit vulnerabilities in the application’s database access mechanisms, allowing attackers to execute unauthorized SQL queries. In this study, we propose an architecture for detecting SQL injection attacks using a recurrent neural network (RNN) autoencoder. The proposed architecture was trained on a publicly available dataset of SQL injection attacks. Then compared with several other machine learning models, including ANN, CNN, Decision Tree, Naïve Bayes, SVM, Random Forest, and Logistic Regression. The experimental result showed that the proposed approach achieved an accuracy of 94% and an F1 score of 92%, which demonstrate its effectiveness in detecting QL injection attacks with high accuracy in comparison with other models covered in the study.

This study aimed to learn disturbance information to effectively control multiple small unmanned surface vehicles (USVs) for marine research purposes at sea. The learned disturbance information was used to investigate the control of multiple USV swarms based on the leading–following formation technique. Numerous studies have already been conducted on swarm control using drones in the air; however, there has been a paucity of studies on swarm control in the sea. Contrary to swarm control in aviation, the fluid density of the sea is high, and the environment is constantly changing; thus, a robot system would be significantly affected by disturbances. This ultimately influences the maintenance of swarm formations and the accomplishment of missions [1,2]. Therefore, it is difficult to control numerous moving objects while maintaining a formation at sea by using the existing aerial drone swarm control technology. To solve this problem, we learned a long short-term memory (LSTM) model with time-series marine environmental data to predict maritime disturbances. Thereafter, the model was applied to a mission-based position control algorithm of each USV constituting the swarm to design a marine unmanned floating vessel swarm control system that can be applied to various marine environments. Specifically, to effectively control numerous small USVs for marine research purposes, ocean currents are learned using the LSTM algorithm, and the predicted ocean currents are used to generate a swarm USV control system for USV formations. In this study, a control system model of several small USVs equipped with two rear thrusters and a front lateral thruster was designed. The LSTM algorithm was learned using the previous ocean current data, and the velocity of the following ocean currents was predicted. The predictions were then used as a system disturbance to determine the thrust of the controller. To measure ocean currents in the sea when each USV moves, the velocity, azimuth, and position (latitude, longitude) data from the GPS mounted on the USV were used to measure the velocity and direction of the hull's movement. Further, the flow rate was measured using a flow rate sensor on a small USV. The USV movement and position were controlled using an artificial neural network-PID (ANN-PID) controller. This study entailed a comparative analysis of the designed USV model results and those generated by the simulator, including the behavior control rule of the USV swarm and the path of the actual USV swarm at sea. It was verified that the effectiveness of the USV mathematical model and behavior control rules. By comparing the movement path of the swarm USV with and without the disturbance learning algorithm and ANN-PID control algorithm applied to the designed simulator, the position error and maintenance performance of the swarm formation were analyzed, and the application results were compared.

We propose an advanced filtering scheme based on Recurrent Neural Networks (RNNs) and Deep Learning to enable efficient control strategies for Vertical Farming (VF) applications.  We demonstrate that the best RNN model incorporates five neuron layers, with the first and second containing ninety Long Short-Term Memory neurons.  The third layer implements one Gated Recurrent Units neuron.  The fourth segment incorporates one RNN network, while the output layer is designed by using a single neuron exhibiting a rectified linear activation function.  By utilizing this RNN digital filter, we introduce two variations: (1) A scaled RNN model to tune the filter to the signal of interest, and (2) A moving average filter to eliminate harmonic oscillations of the output waveforms. The RNN models are contrasted with conventional digital Butterworth, Chebyshev I, Chebyshev II, and Elliptic Infinite Impulse Response (IIR) configurations.  The RNN digital filtering schemes avoid introducing unwanted oscillations, which makes them more suitable for VF than their IIR counterparts.  Finally, by utilizing the advanced features of scaling of the RNN model, we demonstrate that the RNN digital filter can be pH selective, as opposed to conventional IIR filters.

To minimise environmental impact, avoid regulatory penalties, and improve competitiveness, energy-intensive manufacturing firms require accurate forecasts of their energy consumption so that precautionary and mitigation measures can be taken. Deep learning is widely touted as a superior analytical technique to traditional artificial neural networks, machine learning, and other classical time series models due to its high dimensionality and problem solving capabilities. Despite this, research on its application in demand-side energy forecasting is limited. We compare two benchmarks (Autoregressive Integrated Moving Average (ARIMA), and an existing manual technique used at the case site) against three deep learning models (simple Recurrent Neural Networks (RNN), Long Short-Term Memory (LSTM), and Gated Recurrent Unit (GRU)) and three machine learning models (Support Vector Regression (SVM), Random Forest, and K-Nearest Neighbors (KNN)) for short term load forecasting (STLF) using data from a Brazilian thermoplastic resin manufacturing plant. We use the grid search method to identify the best configurations for each model, and then use Diebold-Mariano testing to confirm the results. Results suggests that the legacy approach used at the case site is the worst performing, and that the GRU model outperformed all other models tested.

Change Detection (CD) of aerial images refers to identifying and analyzing changes between two or more aerial images of the same location taken at different times. The CD is a highly challenging task due to the need to distinguish relevant changes, such as urban expansion, deforestation, or post-disaster damage assessment, from irrelevant ones, such as light conditions, shadows, and seasonal variations. Many CD papers have recently been published, where most of the papers that proposed a new model contained a comparison between their proposed and state-of-the-art (SOTA) models. While many recent studies propose new deep learning (DL) models for improving CD performance, their comparative analyses are often restricted, lacking comprehensive insights into the proposed models' real-world generalizability, robustness, and performance trade-offs across diverse change characteristics. This paper presents a novel generic framework to systematically benchmark and assess DL-based CD models through three parallel pipelines: 1) cross-testing models on diverse benchmark datasets to evaluate generalization, 2) robustness analysis against different image corruptions, and 3) multi-faceted contour-level analytics evaluating detection sensitivity to change size/complexity. The framework is applied to comparatively evaluate five state-of-the-art DL-based CD models - Changeformer, BIT, Tiny, SNUNet, and CSA-CDGAN. Extensive experiments unveil each model's strengths, limitations and biases, highlighting their relative proficiencies in generalizing across data distributions, resilience to noise corruption, and discriminative capabilities for changes of varying characteristics. The proposed benchmarking framework demonstrates significant potential for guiding the selection of suitable CD models tailored to specific application requirements by comprehensively evaluating their generalizability, robustness, and detection capabilities across diverse real-world scenarios. This systematic evaluation approach can drive future research into developing more robust and versatile CD solutions aligned with practical needs.

Commercial buildings are a significant consumer of energy worldwide. Logistics facilities, and specifically warehouses, are a common building type yet under-researched in the demand-side energy forecasting literature. Warehouses have an idiosyncratic profile when compared to other commercial and industrial buildings with a significant reliance on a small number of energy systems. As such, warehouse owners and operators are increasingly entering in to energy performance contracts with energy service companies (ESCOs) to minimise environmental impact, reduce costs, and improve competitiveness. ESCOs and warehouse owners and operators require accurate forecasts of their energy consumption so that precautionary and mitigation measures can be taken. This paper explores the performance of three machine learning models (Support Vector Regression (SVR), Random Forest, and Extreme Gradient Boosting (XGBoost)), three deep learning models (Recurrent Neural Networks (RNN), Long Short-Term Memory (LSTM), and Gated Recurrent Unit (GRU)), and a classical time series model, Autoregressive Integrated Moving Average (ARIMA) for predicting daily energy consumption. The dataset comprises 8,040 records generated over an 11-month period from January to November 2020 from a non-refrigerated logistics facility located in Ireland. The grid search method was used to identify the best configurations for each model. The proposed XGBoost models outperform other models for both very short load forecasting (VSTLF) and short term load forecasting (STLF); the ARIMA model performed the worst.

This research delves into the realm of predictive modeling within artificial intelligence (AI) and introduces an innovative ensemble approach designed to enhance prognostic precision. The proposed methodology, characterized by its ebullient nature, leverages the collective wisdom of diverse predictive models, synthesizing a comprehensive and robust framework. The pursuit of parsimony are paramount, underscoring the need for efficient model architectures that balance predictive acuracy and computational efficiency. Our study meticulously explores the amalgamation of diverse algorithms, encapsulating the intrinsic complexity of real-world phenomena. We navigate the intricate landscape of feature selection, algorithmic diversity, and model combination, fostering a holistic understanding of the prognostic process. The ebullient ensemble emerges as a dynamic solution, harnessing the synergy of constituent models while mitigating the risk of overfitting. To validate the efficacy of our approach, extensive experiments are conducted on diverse datasets, spanning domains such as healthcare, finance, and natural language processing. Comparative analyses demonstrate the superior predictive performance of our ensemble model in comparison to individual algorithms and traditional ensemble methods. Additionally, we delve into interpretability aspects, elucidating the underlying mechanisms that contribute to the model's prognostic prowess. Our research advances the frontier of predictive modeling in AI, offering a nuanced and sophisticated paradigm for prognostication. The ebullient ensemble approach stands as a testament to the synergy achievable through the amalgamation of diverse predictive models, providing a pragmatic and parsimonious avenue for enhanced prognostic acuracy in artificial intelligence applications.

Deep learning (DL) algorithms have recently emerged from machine learning and soft computing techniques. Since then, several deep learning (DL) algorithms have been recently introduced to scientific communities and are applied in various application domains. Today the usage of DL has become essential due to their intelligence, efficient learning, accuracy and robustness in model building. However, in the scientific literature, a comprehensive list of DL algorithms has not been introduced yet. This paper provides a list of the most popular DL algorithms, along with their applications domains.