An Overview of Recent Advances in Natural Language Processing for Information Systems

1. Introduction

Recent years have seen considerable interest in facilitating computational access to information in databases. There is a huge demand for users to be able to access and manipulate the massive amount of data in various servers (e.g., the Cloud and the Internet of Things). Such Information can be in the form of text, images/video, audio (music and speech), and other data (e.g., from sensors). While the recent growth in such accessible data has been massive, ways to reliably and efficiently access this data have not always kept pace. Early question-and-answer systems were very simplistic and limited, with excessive constraints. Recent web search systems and digital assistants (e.g., Siri, Cortana, and Google Now) have become very efficient, but limitations remain.

Most people interact with information systems by queries, either textual (typed) or verbal (speech), using natural language to express their wishes. For efficient data access, such queries must be transformed into suitable representation forms to allow efficient interaction with stored data. Thus, the research area of natural language processing (NLP) is foremost in facilitating such access. NLP converts natural human output (speech or text) into forms that facilitate data access. Typical text includes documents, email, web pages, articles, reports, and blogs. For example, in the domain of answering a question, much useful natural language content in found textbooks, encyclopedias, guidelines, and electronic records.

The NLP field has numerous applications in business Intelligence and knowledge management; specific cases include Named Entity Recognition, Named Entity Linking, Coreference Resolution, Temporal Information Extraction, Relation Extraction, Knowledge Base Construction and reasoning, paraphrase detection, entailment recognition, discourse analysis, grounded language learning and image retrieval, computer-assisted translation, and grammatical error correction. One seeks to efficiently analyze text, speech and other data, to find relevant knowledge and structured information, and to extract salient facts about specific types of events, entities, and relationships.

This paper is an overview of recent advances in NLP to permit better access to stored information. We do not give an extensive survey of methods, as such has been well handled in recent papers [1,2,3,4,5]. Instead, we explain NLP methods without great technical or algorithmic detail, as this allows understanding by non-experts.

2. Applications

There are many ways for access in information systems. In Information Retrieval, also called Information Extraction, a user seeks to obtain desired information from a computer system. One specific version is Question-Answering (QA) . Information systems can provide a list of documents or websites in response to a user query, e.g., what web-searchers Google or Bing do. NLP assists the initial steps via Natural Language Understanding (NLU), to convert inquiries to meta-forms to access the data, and then uses Natural Language Generation (NLG) to format the text or speech output to the user. We describe methods for NLU and NLG below.

Specific forms of NL interaction include: 1) Machine Translation (MT), where the input is speech/text in a user’s (native) language and the output is in a different desired (target) language , 2) Automatic Text Summarization (ATS), which transforms a text into a more concise version , 3) Sentiment Analysis, where the affect or emotion present in some text or speech is estimated automatically , 4) text mining, where useful elements are retrieved from broader text , 5) automatic annotation of web pages .

As an example, NLG could access time-series data in an information system and produce a weather forecast or medical report generation . Summarization of such data could be content selection (what to say) , or surface realization (how to say it, i.e., selecting, ordering, and inflecting words) . The objective of content selection is to choose the correct information to output, for a given semantic input and communicative goal. For summarization, one seeks to reduce a given document in size, while retaining as much important and relevant information as possible, with minimal distortion . Some systems generate computer source code from natural language descriptions: specialized LLMs, such as CodeLLaMa and DeepSeek-Coder , as discussed below.

3. Methods for Question Answering

Given the text of a user question, one NLP procedure is to first attempt to understand the intent of the question. A common way is via forms of semantic parsing (shallow or deep) . A question is converted into a meaning representation, then mapped to database queries. This applies to both word sense discovery (acquisition of vocabulary) and word sense disambiguation (understanding) .

Semantic parsing maps natural language text or utterances into a suitable semantic representation, often based on some formalism or grammar. QA without semantic parsing may use SPARQL (Protocol and Query Language) queries on interlinked data represented by Resource Description Framework triples .

4. Language Models (LMs)

Many applications for information systems involve LMs, which are computational systems that can process natural language. Large LMs (LLMs) are extensive and complex models that learn very detailed statistical properties and semantics from vast amounts of training text. As with all artificial neural networks (ANNs) (section 8), LLMs are based on huge numbers of training examples, which provide the knowledge to understand and transform language.

The task of a LM is estimation of the probability of a specified sequence of text “tokens,” i.e., short sequences of text, usually words, but sometimes shorter (e.g., byte-pairs) and sometimes sets of words. Simple LMs use N-gram statistics, i.e., probabilities of sequences of N successive words in ordinary text. As LMs grew in size and power (with advances in computer memory and speed), the training text used all sorts of available text on the internet and elsewhere (private companies also exploit their access to confidential data).

4.1. Basic N-Gram Models

Unigrams (N = 1) are individual words, and so a unigram probability is simply the general estimated likelihood of any given word among all possible text words (e.g., common “function” words such as a, an, the, with, for, etc. have relatively high probabilities, as they occur more often than “content” words such as nouns, verbs, adjectives, and adverbs). Bigrams are sequences of two successive words in text, trigrams uses 3 words, etc. One can readily see that using large values of N for N-grams uses exponentially large memory. However, it is also the case that natural language often has correlations for large values of N. For example, subject noun phrases and their corresponding verbs usually have high correlation (e.g., animals eat, fish swim, water flows). Many languages allow much correlation across many words in a sequence (e.g., in German, the verb occurs at the end of a sentence in general).

Older NLP system used tokenizers and part-of-speech taggers to identify the syntactic role of each word . More modern systems use neural networks to indirectly do such tasks.

A challenge of basic LMs is to reliably estimate probabilities of successive words that are generally conditioned on long text sequences. Use of N-grams tends to assume the validity of a Markov assumption, i.e., that probabilities can be approximated by limiting the range of conditional probabilities. Simply extending N-grams to large ranges of N not only risks exponential memory growth, but also severe under-training, despite the increasing availability of lots of training text data. There are various approaches to mitigate this problem, such as class LMs that group words into categories based on attributes such as color and size (e.g., rather than have separate individual stored likelihoods for “blue ball” and “red ball,” use merged statistics for COLOR+ball). For more efficient LMs, one can cluster various word classes, or allow word sequences to skip some words .

When combining statistics of N-grams into a single global probability, one can use “back-off” principles, giving more weight to the N-grams of lower N, which are based on larger numbers of training examples (i.e., shorter contexts are observed more often, and suffer less from data sparsity). There are many variations for LMs : Good Turing estimator , deleted interpolation , Katz backoff , and Kneser-Ney smoothing .

Relations among words in a text go beyond simple statistics of word sequences such as N-grams. Relation extraction systems find closed-domain relationships. Open-domain relation extraction systems use specific phrases to define word relationships. To discover such relationships, sentences can be analyzed using techniques such as part-of-speech tagging, dependency parsers, or Named Entity Recognizers .

One method is multi-relational learning, which consists of: (1) statistical relational learning (SRL), such as Markov-logic networks, which directly encode multi-relational graphs using probabilistic models (Getoor et al, 2007); (2) path ranking methods, which explicitly explore the large relational feature space of relations with random walk ; and (3) embedding-based models, which embed multi-relational knowledge into low-dimensional representations of entities and relations via tensor/matrix factorization, Bayesian clustering framework, and neural networks .

4.2. Large Language Models

In recent years, research has developed LMs with much more complex systems called Large Language Models (LLMs) [30,31]. They can provide a very detailed understanding of text, using a limited window of focus. With advanced neural architecture, an LLM processes written instructions or questions from users and generates natural language text as output. After pre-training on large amounts of general information, LLMs have had great recent success in solving a wide range of information tasks by conditioning their models on a very small number of examples (“few-shot”) or even only using instructions describing the task (“zero-shot”). Conditioning an LLM is called “prompting” (section 6), with either manual prompts or automatic ones.

LLMs use autoregressive Transformer neural architectures (section 8.2.3), where word tokens are iteratively predicted. On the other hand, alternative masked diffusion language models (MDLMs) are not constrained to generate data sequentially. In comparison to masked language models, the MDLM objective is a principled variational lower bound, and it supports NLG by ancestral sampling .

Pre-Trained Language Models (PLMs) such as BERT (Devlin et al, 2019) are commonly used in many applications due to their favourable performance. Other notable PLMs are open-source BLOOM , LLaMa-1/2 , OPT , and Falcon ; these have similar performance to closed “product” pre-trained LLMs such as GPT-3 and Chinchilla , ChatGPT, Google’s BARD, PaLM , and Claude . The latter are heavily fine-tuned to align with human preferences, which greatly enhances their usability and safety (reliability), but incurs large cost in computation and human annotation. With (commercial) private training, data and model architecture details are generally not shared with the general public.

LLMs can do a huge amount of unsupervised pre-training on textual corpora, and then use a limited amount of supervised fine-tuning (SFT) on high-quality data from an appropriate domain, which may depend upon the application. This latter stage can be expensive, but is done to align the model with human preferences. Prime methods for this fine tuning are Reinforcement Learning (RL) from Human Feedback (RLHF; [40,41] and Direct Preference Optimization (DPO) . DPO directly optimizes using a loss function derived from the classic BT model . RLHF learns from training environments through interaction and rewards , and has three stages: SFT, reward modeling, and RL. Other methods are: syntax fine-tuning, knowledge preservation fine-tuning, and task-oriented fine-tuning .

As ANNs are almost universally based on huge numbers of examples and their huge networks are impossible to debug, there has been a recent effort toward “explainable AI,” which may eventually allow ANNs that can be interpreted . As of now, however, LLMs sometimes generate incorrect information (though otherwise plausible), known as “hallucinations” .

5. Semantic Role Labeling (SRL)

Semantic hierarchies can offer ways to organize textual (NLP) knowledge.

SRL is used to identify predicates and arguments in text, and to label their semantic roles in sentences. This may involve joint classification of semantic arguments [48,49], feature engineering for SRL [50,51], or WordNet . In the ontology YAGO , categories in Wikipedia are linked onto WordNet, which is limited by the scope of Wikipedia. These operations often involve automatic discovery of hypernym (word with a broad meaning or class) versus hyponym (more specific meaning; subordinate in a class) relations, e.g., “is-a” relations (“X is a dog”).

6. Prompts

NLP often uses fine tuning to raise model performance by use of instructions (“prompts”), without needing to modify existing model parameters . In “few-shot prompting,” one provides a few examples to the model for a specific task. LLMs can be fed with step-by-step reasoning examples, e.g., Chain-of-Thought (CoT) prompting , Automatic Prompt Engineer , and Chain of Code . One can tune such systems by appending learnable tokens to the input of the model. CoT decomposes complex problems into smaller sub-problems, then solves them with natural language reasoning steps. This works best if the knowledge to answer an input question exists in its context or in the model’s parameters (e.g., common sense reasoning, learned from much pre-training). Some NLP approaches combine reasoning and acting with LLMs to handle language reasoning and decision making tasks (e.g., ReAct ).

Prompting is commonly used to improve LLMs for downstream applications. One tries to employ well-designed text prompt sequences in an LLM’s input. Finding an effective prompt from a small training set for a specific task is a discrete optimization problem with a large search space, and classic gradient-based optimization methods are not always useful.

Recent advancements in prompt optimization include: (1) discrete representation-based greedy methods and (2) soft representation-based gradient methods. They leverage gradient information to accelerate prompt optimization. Hard representation methods directly use discrete optimization by using hard tokens as intermediate representations: 1) AutoPrompt uses a token search strategy based on gradients to locate useful prompts, 2) Greedy Coordinate Gradient uses a greedy strategy to advance optimization. Projected gradient descent methods map a soft prompt to the nearest token in the vocabulary at each step of regular gradient descent.

7. Distance Metrics for NLP

There are several ways to measure success in training for information systems. Mean Absolute Error has been used for regression tasks , whereas for classification tasks, one generally uses Precision, Recall and F1-score . For recommendation tasks, Mean Reciprocal Rank is in common usage. NLG tasks find metrics like BLEU .

The reward model (RM) is a metric that estimates the degree to which model outputs align with human preferences. RMs typically follow the classic B-T model , but one also uses regression paradigms and the “LLM as a judge” approach . The Area Under the Receiver Operating Characteristic Curve is a common threshold-independent metric . Human coding metrics include the Jaccard Score .

One can measure the performance of a topic model by the log-likelihood of a model on held-out test documents, i.e., its predictive accuracy. Another measure is perplexity, which is inversely proportional to average log-likelihood. Other common measures are topic coherence and topic diversity . For coherence, point-wise mutual information measures the proximity of words in each topic based on relative co-frequencies with each other. Topic diversity uses the percentage of unique words in a topic set.

8. Neural Networks in Information Systems

As with many computer applications in recent years, artificial neural networks (ANNs) have come to dominate the field of information systems. At the cost of significant computer resources, ANNs provide an automatic processing tool to accomplish a wide range of artificial intelligence tasks, including all those mentioned above for NLP. While termed “neural,” such processing algorithms are only very loosely associated with the natural nervous system found in humans and other living beings. The elemental basis of both natural and artificial neural systems is the neuron (or called a mathematical node in ANNs), which transforms a weighted set of electrical inputs (from other neurons) to a single output in a nonlinear operation. By arranging huge numbers of such neurons/nodes in increasingly complex architectures, arbitrarily complex tasks can be accomplished.

The basic ANN feedforward architecture (“multi-layer perceptron”) arranges neurons in layers, whereby the outputs of one layer feed into a successive (higher) layer, and so forth. The initial lowest layer accepts input data from sensors, and the highest layer can provide a classification of the input into a set of predetermined classes (i.e., do pattern recognition). Alternatively, the output can be a set of data that corresponds to a transformation of the input data (e.g., text-to-speech synthesis, or NLG). The nonlinear processing in ANNs allows analysis and synthesis of complex data patterns, but needs huge network models and much computation. The term deep neural network (DNN) refers to an ANN with several layers of nodes, rather than shallow ANNs such as support vector machines . Intermediate “hidden” layers in an ANN refine features of analysis (e.g., as data progresses, layer-to-layer, in a well-trained ANN, the outputs at each layer gradually move toward more relevant features for a given task). Numbers of nodes in layers vary greatly, and are often chosen empirically.

8.1. Fundamentals of ANNs

For NLP classification such as text interpretation, the training of one node in an ANN can be viewed as optimizing the placement of hyperplane boundaries for class regions in a representation space. For a neuron in biological (natural) neural systems, dendrites route electrical signals to succeeding axons, each of which then yields a binary output consisting of a brief pulse in time when the weighted sum of its inputs exceeds a threshold. The output of an ANN node follows an activation function y = φ(w · x + b), where w is a N-dimensional weight vector, b is a scalar bias, x is a N-dimensional set of inputs for the node, φ(.) is a nonlinearity (e.g., sigmoid, rectifier, or tanh). For each node, its parameters specify the location of a hyperplane in a virtual space, for which the node's binary output chooses either side of the hyperplane. Huge DNNs, with many millions of parameters, are often used to attain enough precision for a desired task.

Training parameters in an ANN starts from initial estimates chosen randomly (or pre-trained, using much unlabeled data). Iterative training alters these parameters in incremental fashion, while minimizing a loss (or cost) function. Direct minimization of accuracy for any specific task is not feasible, because ANN training requires to have a differentiable loss, to allow a product chain of derivatives. This shows the direction and amount to alter parameters at each training iteration, using a typical steepest gradient descent approach. Loss functions approximate a penalty for ANN classification errors. They vary greatly across applications, but a common one is cross-entropy , which uses the log-likelihood of training data, and corresponds to least squares error (L2 norm).

8.2. Common ANN Architectures

In a basic ANN, all nodes in each layer provide their outputs to all nodes in the next layer. Each connection requires a multiply-and-add operation (hence, the popularity of graphical processing units, which specialize in such calculations).

Such a general approach is usually excessively costly (in memory and computation). The useful information in most data (including text) is distributed very non-uniformly in its dimensions (e.g., in time). Thus, most applications use versions of the more efficient ANN components discussed below.

8.2.1. Convolutional Neural Networks (CNNs)

For many tasks, relevant information in an input data sequence is largely localized (i.e., found in limited spans); e.g., in images, objects have useful edges that occupy a small percentage of the full data range. In addition, in localized regions, data may display pseudo-random variations. Such cases suggest use of filters to exploit these factors, to better accomplish practical tasks. It is also common to view data as a tensor (e.g., a matrix, if the data are conveniently viewed in two dimensions). For example, while audio data such as speech is a signal showing air pressure as a function of time, a spectral display notes energy as a function of both time and frequency; smoothing edges of a wide-band spectrogram is useful in ASR. In such cases, 2-D data are multiplied by a square weight matrix over a small range (e.g., 3x3), and results are summed (pooled) . One may also apply 1-dimensional convolution to vectors, in cases such as text, which do not benefit from a 2-D display. CNNs thus usually have alternating layers of “convolution” (weighted multiply-and-add filters), with pooling layers that reduce the size of the vector from the previous layer, and thus do data reduction.

8.2.2. Recurrent Neural Networks (RNNs)

To further take advantage of the uneven distribution of pertinent data across dimensions, nodes in ANNs can be more properly weighted across time. For example, CNNs usually exploit data correlations over small local ranges, but recurrence can be helpful to utilize significant correlations over long ranges. RNNs have neural models where data is fed back from earlier layers, and use distributed hidden states that store information about past input . Specialized network gates (called input, forget, and output) control the flow of data across layers. Typical RNNs are termed long short-term memory and residual network.

8.2.3. Attention

A major recent development in the field of ANNs is called attention - a way to improve performance by emphasizing weights for specific sections of data; this is another method to exploit the non-uniform distribution of information. The focus for attention is determined mathematically as correlation among data using matrix operations (e.g., multiplications and sums) that use queries (inputs), keys (features), and values (desired outputs). The queries come from a decoder layer earlier in an ANN, and keys and values come from encoder outputs. Attention can be readily applied across various dimensions including time, frequency, and ANN layers. While the principle of such correlation-based attention is sound, the simple mechanisms of correlation that are used are often inadequate to exploit well the complex information distribution in data.

Neural models that use attention are often called Transformers . In recurrent networks (RNNs), inputs are processed in sequence. Transformers do not follow sequences, instead using an embedding table with time-positional encodings. These latter use non-linear functions for monotonic temporal mappings.

A major disadvantage of Transformers is their need for more computation than other ANN approaches. Transformers can be quadratic in the length of data sequences. Transformer architecture can learn general structural information from high-dimensional data, and exploit the huge amount of unlabeled data (available generally) through self- and un-supervised learning.

8.3. Neural LLM Architectures

Many information systems that use analysis and synthesis (e.g., of text) apply an approach that involves coding and decoding data. (This is analogous to inverse operations in radio transmission.) In an ANN approach for such an encoder/decoder structure, the initial network layers act as an encoder to automatically learn hidden latent features for data in a compressed representation, and then ensuing layers act as decoder to form a reconstructed data form. In many audio and video coding schemes, the decoder steps often proceed in inverse fashion to the encoder steps. When this encoder-decoder is trained on unlabelled data, it is called an autoencoder.

Many neural systems use deep generative models try to learn complex data distributions through unsupervised training. The so-called GAN (adversarial) architecture uses two networks, generator and discriminator, in a minimum-maximum operation. The generator produces data from a low-dimensional latent representation space, often starting from a simple Gaussian noise vector. The discriminator learns to distinguish between “real” training data and “false” generator outputs. The criterion for generators is to maximize the discriminator’s error rate, while discriminators try to minimize their error rate.

The encoder processes and encodes an input sentence into a “hidden" representational form, which attempts to find relationships between words and the overall textual context. For example, the common module BERT uses only an encoder (no decoder) with bi-directional attention.

In an autoencoder, the decoder utilizes the hidden space of the system to generate a target output text; it converts the abstract representation into specific contextually-relevant expressions; examples are: PLBART , CodeT5 , AlphaCode , and CoTexT .

For many applications, it is not necessary to use both an encoder and a decoder. For example, GPT uses only a decoder; indeed, most current LLMs are decoder-only with causal attention, i.e., earlier tokens in a text do not attend to ensuing tokens. Other decoder-only systems include: Google Gemini and Claude.

9. Sentence Models

Distributional semantic models (DSMs) use vectors that keep track of text contexts (e.g., co-occurring words) in which target terms appear in a large corpus as proxies for meaning representations, and apply geometric techniques to these vectors to measure the similarity in meaning of the corresponding words .

A general class of basic sentence models is that of Neural Bag-of-Words (NBoW) models . These generally consist of a projection layer that maps words, sub-word units or N-grams to high dimensional embeddings; the latter are then combined as components with an operation such as summation. Large-scale word embeddings can provide context for topic models (next section). The Bag-of-Words approach ignores the sequential ordering of words in text.

10. Topic Models

In NLP, a topic model attempts to obtain a general idea of the topics (subjects) of an input text . Topic models use unsupervised machine learning to compress many documents into a short summary that captures the most prevalent subjects in a corpus. When applied to a set of documents, a topic model estimates a set of underlying (latent) topics, each of which describes a semantic concept. Early topic models used Latent Semantic Indexing (LSI) or PLSA [86,87]: using a word-document matrix, a singular value decomposition reduces the dimensionality. Another approach is Latent Dirichlet Allocation (LDA), which uses the Dirichlet distribution, a bag-of-words model, and the same document-term matrix as in LSI . In LDA, a topic is a distribution of words, where each word in a text comes from a mixture of multinomial distributions with Dirichlet as the prior.

Before neural approaches, most topic models were probabilistic graphical models (such as LDA) or non-negative matrix factorization (NNMF). The former model the document generation process with topics as latent variables and model parameters through Variational Inference or Monte Carlo Markov Chain methods like collapsed Gibbs sampling . In probabilistic generative models, model random variables are assumed to be come from certain prior distributions. NNMF directly finds topics via decomposition of a term-document matrix into two low-rank factor matrices.

A simple topic model is the “bag-of-words” model, which represents a document by a vector of word counts, across the vocabulary of all possible words, by the numbers of their occurrences in the text. In graph-based approaches , words are nodes and their co-occurrences use weighted edges. Large-scale word embeddings can provide context for the topic models; they represent words as continuous vectors in a low-dimensional space.

Neural Topic Models (NTMs) use DNNs to model distributions for topic models . Unlike earlier topic models, NTMs can estimate model parameters through automatic gradient back-propagation by adopting deep neural networks to model latent topics, such as Variational Autoencoder . This flexibility enables adjustments to model structures to fit diverse application scenarios. They typically use the re-parameterization trick of Gaussian distributions. In addition, NTMs can handle large-scale datasets via use of parallel computing facilities like GPUs.

One can use LLMs to improve topic modeling. Topic models are largely based on word co-occurrences to infer latent topics, but such information is scarce in short texts such as tweets (i.e., problem of data sparsity). Probabilistic topic models, such as PLSA and LDA, work well on long texts. For short text topic modeling, Biterm Topic Model (BTM) and Dirichlet Multinomial Mixture (DMM) model are common approaches. BTM directly constructs the topic distributions over unordered word-pairs (biterms), while DMM applies auxiliary pre-trained word embeddings to introduce external information from other sources; both use classic Bayesian inference. Contrastive learning is also used for topic models .

Several extensions based on BTM and DMM have also been proposed, such as Generalized Polya Urn-DMM with word embeddings and Multiterm Topic Model . Besides, Semantics-assisted Non-negative Matrix Factorization was lately proposed as an NMF topic model incorporating word-context semantic correlations solved by a block coordinate descent algorithm.

A recent method called Adversarial-neural Topic Model (ATM) uses a generator network to learn a projection function between a document-topic distribution and a document-word distribution. A discriminator network determines if an input document is real or fake and its output signal helps the generator to construct a more realistic document from a random noise drawn from a Dirichlet distribution.

11. Fine Tuning LLMs

The important step of fine-tuning (transfer learning) for LLMs has several versions: prefix, adapter, task-oriented, and parameter-efficient.

In prefix tuning, trainable tokens (soft prompts) are added to both the input and internal layers ; their parameters are shared. In these decoder-only models, instead of a causal mask, a fully visible mask is used for the prefix part of the input sequence, and a causal mask is used for the target sequence.

With adapter tuning, one adds small neural network modules to the original model, then fine-tunes them on specific tasks, thus fine-tuning a few external parameters instead of entire LLMs .

As for task-oriented fine-tuning , Parameter Efficient Fine-Tuning optimizes the subset of parameters fine-tuned, thereby reducing the overall computational complexity. One version of this is Low-Rank Adaptation, which freezes pre-trained model weights and inserts trainable rank decomposition matrices into each layer of the Transformer architecture, which greatly reduces the trainable parameters for downstream tasks .

Most LLMs base their training on fixed sets of data, which can limit performance on specific or dynamic topics. A recent approach called Retrieval Augmented Generation (RAG) integrates real-time knowledge retrieval with LLM generation . RAG uses a neural retriever to find the best text sections by dense similarity, which are concatenated with the query and conditioned into a NLG component. RAG integrates real-time knowledge retrieval with LLM generation, which secures outputs in specific data from current domains.

12. Meaning Representation

A major objective in information systems is to extract meaning from text. One method for this uses distributional semantics. Distributional semantic models (DSMs) employ vectors that follow context where target words in large corpora serve as proxies for meaning representations . They use geometric techniques to measure similarity in the meaning of corresponding words . This follows the Distributional Hypothesis , where words with similar linguistic contexts are likely to have similar meanings. DSMs represent lexical entries using vectors (embeddings) that demonstrate their distribution in text corpora.

Early DSMs established distributional vectors with word co-occurrence frequencies. Then, prediction DSMs learned vectors with shallow ANNs for local surrounding words. Now, contextual DSMs use deep NTMs to generate contextualized vectors for words. Contextual embeddings (e.g., ELMo and BERT) improve on global word representations (e.g., Word2Vec) [1,108].

13. Challenges and Benchmarks

To motivate uniform research in the field, several technical meetings and efforts have been organized. One of the first was the Text REtrieval Conference (TREC), of which there were several in the 1990s . A later one was the Cross Language Evaluation Forum .

The Document Understanding Conference examined query-focused multi-document summarization since 2004; focus on complex queries with specific answers . A preposition disambiguation task was focus in SemEval 2007 .

Benchmarks or datasets to assess language models’ reasoning:

-

CriticBench evaluates a LM’s capability to critique solutions and correct mistakes in reasoning tasks.

-

MathCheck synthesizes solutions containing erroneous steps using the GSM8K dataset .

-

PRM800K builds on the MATH problems .

-

process reward models explicitly assess and guide reasoning at the step or trajectory level.

-

The Stanford Question Answering Dataset and DROP are often used as reading comprehension benchmarks.

-

HellaSwag advances common sense NL inference via use of Adversarial Filtering, which is a data collection paradigm where discriminators iteratively select an adversarial set of machine-generated wrong answers.

-

Flan : a large publicly available set of tasks and methods for instruction tuning.

Older knowledge bases include:

-

DBPedia

-

Freebase

-

Yago2

-

FrameNet

-

Propbank , a large hand-annotated text corpus.

-

Other possibilities include use of expert annotators, or through crowdsourcing (e.g., Amazon’s Mechanical Turk ).

Early software include: MetaMap to identify concepts in text.

SEMREP can detect some relations using hand-crafted rules. As for commercial systems, we have Google Now (with Google’s Knowledge Graph ), and Facebook Graph Search . An earlier system was the Unified Medical Language System , having 2.7 million concepts from over 160 source vocabularies. Distant Supervision has a knowledge base such as Wikipedia or Freebase that is used to automatically tag training examples from the text corpora .

In a related direction, some interactions with databases involve sentence compression, where an input text is transformed into shorter output text, while retaining the general meaning (i.e., omitting lesser detail). Examples are: Ziff-Davis corpus and the Edinburgh compression corpora .

14. Computational Issues

Major factors in all information systems are their computational requirements, i.e., storage space and mathematical operations. In general, the trend has been toward massive systems. For example, LLaMa-2 uses 13 billion parameters (e.g., multiply-and-adds). Fine-tuning a 16-bit LLaMa model with 65 billion parameters requires more than 780 gigabytes of memory . The massive recent progression in model size is readily seen in GPT: GPT-1 had 117 million parameters; GPT-2 1.5 billion, and further to GPT-3 175 billion .

The pre-trained model CodeBERT has a total of 125M parameters, resulting in a model size of 476 MB. Recently proposed models like Codex and CodeGen have over 100 billion parameters and over 100 GB in size.

Training a relatively smaller model like the PolyCoder (2.7 billion paramters) , employing eight NVIDIA RTX 8000 GPUs on a single machine, requires about 6 weeks to train.

15. Conclusions

This overview examined recent developments in information systems, in the context of classical approaches. As with many application areas of artificial intelligence, the field has become dominated by deep learning methods, as they provide rapid and reasonable solutions, at the cost of significant computer memory and computation needs.