ARTICLE | doi:10.20944/preprints202208.0287.v1
Subject: Mathematics & Computer Science, Artificial Intelligence & Robotics Keywords: table detection; document layout analysis; continual learning; incremental learning; experience replay
Online: 16 August 2022 (10:56:59 CEST)
The growing amount of data demands methods that can gradually learn from new samples. However, it is not trivial to continually train a network. Retraining a network with new data usually results in a known phenomenon, called “catastrophic forgetting.” In a nutshell, the performance of the model drops on the previous data by learning from the new instances. This paper explores this issue in the table detection problem. While there are multiple datasets and sophisticated methods for table detection, the utilization of continual learning techniques in this domain was not studied. We employed an effective technique called experience replay and performed extensive experiments on several datasets to investigate the effects of catastrophic forgetting. Results show that our proposed approach mitigates the performance drop by 15 percent. To the best of our knowledge, this is the first time that continual learning techniques are adopted for table detection, and we hope this stands as a baseline for future research.
ARTICLE | doi:10.20944/preprints201808.0194.v1
Subject: Mathematics & Computer Science, Artificial Intelligence & Robotics Keywords: ANN; continual learning; machine intelligence; prediction; vandalism; security; SDR
Online: 9 August 2018 (15:02:06 CEST)
Oil and gas pipeline vandalism is a recurrent problem in oil rich zones of Nigeria and its West African neighbors and remains a challenge for multinationals to set ahead control measures to avert possible damages to operations both in infrastructure and business profit margins. In this paper, an integrative systems model comprising of a machine intelligence technique called Hierarchical Temporal Memory (HTM) and a sequence learning neural network called the Online-Sequential Extreme Learning Machine (OS-ELM) is proposed for monitoring and prediction of pipeline pressure data. The system models the continual prediction of pipeline oil/gas pressure signals useful for secure monitoring and control to avert acts of vandalism in oil and gas installations. The HTM uses a spatial pooler operated in temporal aggregated fashion and is defined as HTM-SP. The OS-ELM technique uses an explicit hierarchical training scheme so that the best cost estimates may be found after a stipulated number of trial runs. We study the performance of three OS-ELM neural activations: the sigmoid (sig), sinusoidal (sin) and radial basis function (rbf) activations. The results indicate improvement factors of 1.297, 1.297 and 1.300 of the HTM-SP over the OS-ELM sigmoid, sinusoidal and radial basis activations respectively.
ARTICLE | doi:10.20944/preprints201809.0104.v1
Subject: Mathematics & Computer Science, Artificial Intelligence & Robotics Keywords: neural networks; statistical physics of learning; on-line learning; concept drift; continual learning; learning vector quantization;
Online: 5 September 2018 (16:27:10 CEST)
We introduce a modelling framework for the investigation of on-line machine learning processes in non-stationary environments. We exemplify the approach in terms of two specific model situations: In the first, we consider the learning of a classification scheme from clustered data by means of prototype-based Learning Vector Quantization (LVQ). In the second, we study the training of layered neural networks with sigmoidal activations for the purpose of regression. In both cases, the target, i.e. the classification or regression scheme, is considered to change continuously while the system is trained from a stream of labeled data. We extend and apply methods borrowed from statistical physics which have been used frequently for the exact description of training dynamics in stationary environments. Extensions of the approach allow for the computation of typical learning curves in the presence of concept drift in a variety of model situations. First results are presented and discussed for stochastic drift processes in classification and regression problems. They indicate that LVQ is capable of tracking a classification scheme under drift to a non-trivial extent. Furthermore, we show that concept drift can cause the persistence of sub-optimal plateau states in gradient based training of layered neural networks for regression.
ARTICLE | doi:10.20944/preprints202208.0117.v1
Subject: Engineering, Electrical & Electronic Engineering Keywords: Continual Learning; Lifelong Learning; Prototypical Networks; Catastrophic Forgetting; Intransigence; Task-free; Incremental Learning; Online Learning; Human Activity Recognition
Online: 5 August 2022 (08:35:15 CEST)
Continual learning (CL), a.k.a lifelong learning, is an emerging research topic that has been attracting increasing interest in the field of machine learning. With human activity recognition (HAR) playing a key role in enabling numerous real-world applications, an essential step towards the long-term deployment of such systems is to extend the activity model to dynamically adapt to changes in people’s everyday behavior. Current research in CL applied to HAR domain is still under-explored with researchers exploring existing methods developed for computer vision in HAR. Moreover, analysis has so far focused on task-incremental or class-incremental learning paradigms where task boundaries are known. This impedes the applicability of such methods for real-world systems. To push this field forward, we build on recent advances in the area of continual learning and design a lifelong adaptive learning framework using Prototypical Networks, LAPNet-HAR, that processes sensor-based data streams in a task-free data-incremental fashion and mitigates catastrophic forgetting using experience replay and continual prototype adaptation. Online learning is further facilitated using contrastive loss to enforce inter-class separation. LAPNet-HAR is evaluated on 5 publicly available activity datasets in terms of its ability to acquire new information while preserving previous knowledge. Our extensive empirical results demonstrate the effectiveness of LAPNet-HAR in task-free CL and uncover useful insights for future challenges.