How ChatGPT’s Hallucinations (Compared to Gemini’s) Impact Text Summarization with Literary Text

1. Introduction

The paper describes in detail an experiment using ChatGPT.4o [1] in the summarization mode in order to show the positive and negative aspects of the way in which literary text is reduced to fit into a short summary. The choice to use literary rather that journalistic text is done considering that in literary text a greater variety of text typologies can be found thus constituting a more interesting challenge. In particular, literary text may frequently contain both literal and non-literal meaning; it may be organized with sentences with structures in non-canonical order; it may contain new and hard to understand lexical items; the temporal sequence of the storyline may be in some cases reversed introducing flashbacks, and may be interspersed with diegetic or gnomic statements. For this reason, we are using excerpts taken from a novel “The Solid Mandala”[2] written by Noble Prize Australian writer Patrick White which can offer such a variety. We include in Appendix A a brief summary of the storyline of the novel.

As the banner in the starting webpage recites, ChatGPT can make mistakes, it is not infallible and falls into frequent hallucinations. We intend to illustrate extensively a number of recurrent errors that can badly harm the understanding of the content of the novel. GPT’s performance is then compared to Gemini’s in a separate section where we show that mistakes are reduced and overall comments on narrative style are correct. We used a number of different prompts to address storyline and style. The prompts we used are very simple and direct. For this reason, we organized our analysis into three separate sections: the first one is dedicated to summarizing the content by way of what has happened to the protagonists, the story or “fabula”. The second one is dedicated to how the story is being organized, that is the narrative technique, the “sujet” or the plot. The third focuses on the linguistic tools that make the style the way it is, focusing on the verbal complex. It is important to note the peculiarity of White’s style which, according to Gordon Collier ([3], p.130) is a narrative in third-person that uses a technique called “figural consciousness”, that rather than having the “auctorial narrator” has each character – there are three main protagonists - narrate with his/her own personal language and style, and both are very different from one another.

As will appear from the analysis we carried out, mistakes in the output summary produced by ChatGPT can sometimes harm the way in which the content of the story is reported, by assigning actions to the wrong character. These constitute the worst cases, which we assume regard the use of a procedure that we dub “coreference chain” in which all pronouns are bound to a preceding antecedent. We assume that the procedure is based on heuristics rather than on the knowledge acquired by the underlying NN: in some case, the antecedent is wrongly assigned, or not assigned at all confusing the plot. These constitute more difficult cases, in which the antecedent can be assigned only through an inference, or when a cataphora is present. Other recurrent mistakes regard the incorrect usage of semantically ambiguous terms. Typical problems may arise due to oversimplification, i.e., summarization requirement to reduction down a certain threshold. This may cause disappearance of semantically relevant pieces of text, usually restrictive modifiers, that allow the reader to understand the content adequately. As to the choice of the novel, for many critics Patrick White’s novels are difficult and rich of a lot of elaborate suggestions. As said above, White uses a complex mixture of narrative techniques which includes metaphors and other poetic devices. Stream of consciousness is substituted by a technique by which the narrator embodies each character using each time a different language: the result is a figural consciousness speaking not the omniscient narrator. But the main trait of his style is the mixture of “fabula” and “sujet”, i.e., the personality of his characters are strongly interwoven with the storyline and in more abstract terms the form is strongly interwoven with meaning. What is consistently present and acknowledged, is the intent of the author to characterize each protagonist - and in our case the main ones are three - with specific linguistic features realized both by the use of lexical choice and distinct constructions mostly at verbal complex level – but see section 3. In this way each protagonist is identified by specific expressions and significant traits that highlight psychological aspects and the relationships with each other, with the external world and with other characters.

In a previous study [4,5] we followed suggestions by Gordon Colliers [6] which we used to implement a system that tries to identify automatically narremes in the novel, which has been annotated manually. For the annotation we were also inspired by Colliers’ list of features and have been using it with some changes and additions. In particular, we also decided to use Martin & White’s Appraisal Theory Framework [7,8] and introduce features related to Judgement and Affect. The current analysis is thus accompanied and counterbalanced by the deep level of knowledge we acquired in the previous experiments which will be useful to clarify what ChatGPT is consistently unable to understand and why. More on this in section 3 below.

The paper is organized as follows: in section 1.1 below we draw a neat picture of state of the art hallucinations in a number of fields including medicine and healthcare, news, law and story and reference fabrication; in section 2 we focus on summarization of excerpts from the novel, which requires ChatGPT to correctly relate events, actions, and states to the correct protagonist, preserve all needed text to convey complete information and extract appropriate meaning for ambiguous words; in section 3 we concentrate on the style, the plot and the linguistic and rhetoric devices used by the author, and we highlight the level of complexity in the way in which subtle psychological states are presented by manipulating the verbal complex; in the fourth section we make a detailed comparison with Gemini where we highlight the overall better performance; finally in section 5 a discussion is presented to support the previous analysis by the underlying hypothesis and the comparison drawn with Gemini; and then a conclusion follows.

1.1. State of the Art of Hallucinations

To the best of our knowledge, no previous study has appeared on the presence of hallucination in literary books like novels of famous writers. Hallucinations have been the object of extensive research in the last three or four years. We review briefly the most important trends in the literature appeared mainly last year 2024, which regard: fabrication of ungrounded text, omission of relevant information, generating inexistent references for scientific topics, lack of concern for factuality. Topics treated include Medicine, Summarization, Law, and eventually a general abstract topic, Faithfulness.

1.1.1. Hallucinations in Medicine

Hallucinations in medicine in general are regarded very dangerous. This is what is expressed directly in a number of research papers that we review first of all.

In Emsley paper [9], the first reference was to a well-known longitudinal study reporting brain changes over time - but it was inappropriate, as it was not focused on the thalamus. The author was unable to trace three of four references, whether searched by author names, manuscript title or journal details. Similarly, with Google Scholar it was not possible to identify the articles. Entering the Digital Object Identifier (DOI) number in the searches resulted in totally unrelated publications. As the author was becoming increasingly uneasy, he questioned ChatGPT about previous studies whose content he was familiar with, including his own, as a test of accuracy. Some of the answers provided were patently incorrect.

So, the author notes that “The problem therefore goes beyond just creating false references. It includes falsely reporting the content of genuine publications. Thus, while most attention to date has focused on the production of false references as these are the easiest to detect, the veracity of any content inputs provided by ChatGPT cannot be trusted. The cause of these falsifications has been linked to a disturbance in language production, with probabilistic outputs based on estimates of semantic similarity. This allows informed guesses, with bits of false information being mixed with factual information.”

The continuation is even worse: “Alarmed, I assumed I had done something wrong. I instructed ChatGPT to check one of the incorrect references. I received an apology for the mistake and was provided with the “correct” reference. However, this one was also incorrect. And so too with the third attempt.”

In conclusion, prof. Emsley defines ChatGPT’s hallucinations as fabrications, and concludes as follows: “I do not recommend ChatGPT as an aid to scientific writing.”

Pal et al. [10] explain in their paper the importance of being faithful and factual in reporting information related to the medical and in general healthcare domain. But as they say it, LLMs “generate plausible and confident yet incorrect or unverified information. Such hallucinations may be of minimal consequence in casual conversation or other contexts but can pose significant risks when applied to the healthcare sector, where accuracy and reliability are of paramount importance. Misinformation in the medical domain can lead to severe health consequences on patient care and outcomes, the accuracy and reliability of information provided by language models can be a matter of life or death. They pose real-life risks, as they could potentially affect healthcare decisions, diagnosis, and treatment plans.”

To mitigate the consequences of hallucinations they propose “the development of methods to evaluate and mitigate such hallucinations in the Med-HALT framework, which proposes a two-tiered approach to evaluate the presence and impact of hallucinations in generated outputs.”

Their approach is very interesting and still up to date. It includes Reasoning Hallucination Tests (RHTs) which are used to “assess how accurately the language model performs reasoning over the medical input data and whether it generates logically coherent and factually accurate output, without creating fake information.” This test includes: The False Confidence Test (FCT) which “involves presenting a multiple-choice medical question and a randomly suggested correct answer to the language model, tasking it with evaluating the validity of the proposed answer, and providing detailed explanations for its correctness or incorrectness, in addition to explaining why the other options are wrong.” The other test is called None of the Above (NOTA) Test, and here the model “is presented with a multiple-choice medical question where the correct answer is replaced by ’None of the above’, requiring the model to identify this and justify its selection. It tests the model’s ability to distinguish irrelevant or incorrect information. Finally, they conceived of another important test, the Fake Questions Test (FQT), which “involves presenting the model with fake or nonsensical medical questions to examine whether it can correctly identify and handle such queries.” Results are very poor for reasoning tasks but very high when given a binary task like distinguishing between real and fake questions.

Alessia McGowan et al. [11] in the abstract of their paper clarify their work as follows:

“... we aimed to assess whether ChatGPT could reliably produce accurate references to supplement the literature search process. We describe our March 2023 exchange with ChatGPT, which generated thirty-five citations, two of which were real. 12 citations were similar to actual manuscripts (e.g., near-matched title with incorrect author lists, journals, or publication years) and the remaining 21, while plausible, were in fact a pastiche of multiple existent manuscripts...Furthermore, we describe potential explanations for citation fabrication by GPTs, as well as measures being taken to remedy this issue, including reinforcement learning. Our results underscore that output from conversational LLMs should be verified.”

The authors “endeavored to use the most popular GPT large language model, ChatGPT, to conduct a literature search for a machine learning-based manuscript, utilizing NLP on spoken Language to identify linguistic correlates of suicidal behavior in the context of psychosis”. In the paper we find the characterization of error patterns which sounds as follows: “in comparing each of the 12 inaccurate citations with their most closely matched “real” citation, we noted that, in several places, the titles of real articles had been altered such that original terms were replaced with terms from our queries.”

1.1.2. Hallucinations in Summarization

Another frequent topic treated in relation to hallucinations is summarization. Maynez et al. [12] already in 2020 analyzed limitations of large language models for abstractive document summarization and found that these models are highly prone to hallucinate content that is unfaithful to the input document. They conducted a large scale human evaluation of several neural abstractive summarization systems to better understand the types of hallucinations they produce and found substantial amounts of hallucinated content in all model generated summaries.

The problem of document summarization, i.e., “the task of producing a shorter version of a document while preserving its information content requires models to generate text that is not only human-like but also faithful and/or factual given the document.

The authors introduce an interesting classification of hallucination in their analysis which is expressed by the following question: “Do models hallucinate by manipulating the information present in the input document (intrinsic hallucinations) or by adding information not directly inferable from the input document (extrinsic hallucinations)?”, and its follow up represented by the following question: “How much hallucinated content is factual, even when unfaithful?”

As the authors clearly affirm, despite recent improvements in conditional text generation, most summarization systems are trained to maximize the log-likelihood - a strong statistical technique for estimating the parameters of probability distributions based on observed data - of the reference summary at the word-level, which does not necessarily reward models for being faithful. The authors continue by saying that “moreover, models are usually agnostic to the noises or artifacts of the training data, such as reference divergence, making them vulnerable to hallucinations. Thus, models can generate texts that are not consistent with the input, yet would likely have reasonable model log-likelihood.

Hallucinations of intrinsic nature; both use terms or concepts from the document but misrepresent information from the document, making them unfaithful to the document.

Extrinsic hallucinations are model generations that ignore the source material altogether. In extrinsic hallucinations use terms that are not introduced in the document. According to the authors “a model with a poorly-informed decoder and that is agnostic to the divergence issue between the source and target text will function more as an open-ended language model and will be prone to extrinsic hallucinations.

A better definition of hallucinations is contained in the definition that follows:

“A summary S of a document D contains a factual hallucination if it contains information not found in D that is factually correct. Factual hallucinations may be composed of intrinsic hallucinations or extrinsic hallucinations. By definition, abstractive summaries are written to preserve the salient information in the input document, but they are expressed in the words of the summary author as opposed to the input document author.”

In the conclusion of their paper, Kalev et al.[13] express the need to study further the phenomenon of hallucination, and recommends caution in the use of AI for summarization: “No matter the underlying cause, these results suggest significant caution is warranted in automated summarization and that further research is needed into the general prevalence and underlying causes of summarization hallucination.”

Another frequent trend in examining hallucinations, is the one represented by the paper by Catarina G. Belem et al. [14], where Multi Document Summarization (MDS), is considered. In this work, the authors investigate how hallucinations manifest in LLMs when summarizing topic-specific information from multiple documents. Since no benchmarks exist for investigating hallucinations in MDS, they use existing news and conversation datasets, annotated with topic-specific insights, to create two novel multi-document benchmarks. When evaluating 5 LLMs on their benchmarks, they observe that on average, up to 75% of the content in LLM-generated summary is hallucinated, with hallucinations more likely to occur towards the end of the summaries. To understand the characteristics of these hallucinations, they manually evaluate 700+ insights and found that most errors stem from either failing to follow instructions or producing overly generic insights.

In their work, Laban et al. [15] propose to leverage the task of summarization as a testbed for evaluating long-context models and Retrieval Augmented Generation (RAG) systems. They note that summarization requires reasoning over a long context and a careful understanding of the relative importance of content. They use a highly specialized test bed for summarization, the “Summary of a Haystack” (SummHay) task, which requires a system to process the Haystack and generate, given a query, a summary that identifies the relevant insights and precisely cites the source documents. Results are very disappointing.

Huang et al. [16] focus on multi-document news summarization with the task of summarizing diverse information encountered in multiple news articles encompassing the same event. To facilitate this task, they outlined a data collection schema for identifying diverse information and curated a dataset named DiverseSumm. Their analyses suggest that despite the extraordinary capabilities of LLMs in single-document summarization, the proposed task remains a complex challenge for them mainly due to their limited coverage, with GPT-4 only able to cover under 40% of the diverse information on average. Their fine-grained human evaluation results identify that even the most advanced LLM, GPT-4, only covers about 37% of diverse information with optimally designed prompts.

1.1.3. Hallucinations in Law

Another important field of application of AI summarization function is law. Dahl et al. [17] ask the rhetoric question: do large language models (LLMs) know the law? Seen that LLMs are increasingly being used to augment legal practice, education, and research, they note how their revolutionary potential is threatened by the presence of “hallucinations” — i.e., textual output that is not consistent with legal facts. In their paper, they present the first systematic evidence of these hallucinations in public-facing LLMs, documenting trends across jurisdictions, courts, time periods, and cases. To this aim, they have been using OpenAI’s ChatGPT 4 and other public models. As a result, they show that LLMs hallucinate at least 58% of the time, struggle to predict their own hallucinations, and often uncritically accept users’ incorrect legal assumptions. Thus, they conclude by cautioning against the rapid and unsupervised integration of popular LLMs into legal tasks, and to mitigate the effect they develop and make available a typology of legal hallucinations to guide future research in this area.

1.1.4. Hallucinations: Vocabulary Size, Factuality and Faithfulness

One of the underlying causes of hallucination in summarization may be found in the paper by Uluoglakci and Taskaya Témizel [18]. In their paper they note that LLMs encounter challenges when dealing with rare tokens, especially in mixed contexts. No information is made available when un grounded yet plausible text is generated, as the authors comment: “Adversarial effects in real-life scenarios may inadvertently emerge from prompts that combine both common and rare tokens. However, approximately 90% of the time LLMs neglect to indicate their lack of information about a hypothetical phenomenon in similar situations. This characteristic significantly diminishes the reliability of LLMs and impedes their suitability for deployment in critical decision-making systems.” And as to the tendency of LLMs toward hallucination when in presence of nonexistent terms in their training dataset see below.

Another important trend is the one characterized by the evaluation of factuality in generated texts by LLMs. Min et al. [19] in their paper, introduce FACTSCORE, a new evaluation tool that breaks a generation into a series of atomic facts and computes the percentage of atomic facts supported by a reliable knowledge source. As the authors note, “evaluating the factuality of long-form text generated by large language models (LLMs) is non-trivial because (1) generations often contain a mixture of supported and unsupported pieces of information, making binary judgments of quality inadequate, and (2) human evaluation is time-consuming and costly.” Their results “indicate that current LLM training methods are insufficient to prevent hallucinations, emphasizing the need for a fundamental change to ensure the reliability of LLMs.” To test the ability of LLMs to answer questions they used a dataset which comprises one-third of hypothetical questions and two-thirds valid questions. The results determined by the percentage of valid answers to hypothetical questions was 5.72% for GPT-3.5 and 5.64% for Llama2-70B indicating over a 94% error rate. In most cases the models failed to recognize a hypothetical term or refused the existence of a valid term producing hallucinated information. In addition, they omitted the hypothetical term entirely in 5% of their responses.

As Mishra et al. [20] assume, Llama and other similar LMs have demonstrated significant achievements in summarization tasks but struggle with factual inaccuracies, a critical issue in clinical NLP applications where errors could lead to serious consequences. LLMs confront significant challenges, primarily their propensity for generating hallucinations—fabricated information not grounded in source text—and producing factually inconsistent outputs. Such limitations critically undermine the models’ reliability, particularly critical in clinical NLP applications, where inaccuracies could result in serious misdiagnoses.

The authors Kalai & Vempala [21] comment on the tendency of recent language models to generate false but plausible-sounding text with surprising frequency. This fact constitutes a danger seen that such “hallucinations” are an obstacle to the usability of language-based AI systems and can harm people who rely upon their outputs. Their paper shows that there is an inherent statistical lower-bound on the rate that pretrained language models hallucinate certain types of facts, and that this fact has nothing to do with the transformer LM architecture or data quality.

Kin et al. [22] report work done at book level to verify the “faithfulness” of LLMs in reporting the book content in their summaries. To this aim, they decompose each summary into a list of claims (FABLES) which are then individually verified against the input document. Overall, they observed that CLAUDE-3-OPUS is the most faithful book-length summarizer by a significant margin, followed by GPT-4-TURBO. As a result, a qualitative analysis of FABLES reveals that the majority of claims marked as unfaithful are related to events or states of characters and relationships. In their experiment, the authors also found that annotators frequently point out omissions of critical information. Thus, they developed the first taxonomy of omission errors in book-length summarization and observe that key events, details, and themes are frequently omitted by all LLMs. They also observe other content selection errors: for example, even the strongest summarizers, CLAUDE-3-OPUS and GPT-4-TURBO, over-emphasize content towards the end of books to the detriment of the beginning.

As a first conclusion derived from the literature on hallucinations in the use of LLMs we can safely say that the problem can be traced to the intrinsic need to keep the size of the vocabulary in the range of 30K-60K, a decision that is dictated partly by the need to keep the number of parameters to a manageable size and reduce the compute requirements: as Tao et al. put it [23]“A larger vocabulary size improves tokenization fertility, i.e., splitting sentences into fewer tokens, thereby improving the tokenization efficiency. Additionally, a larger vocabulary enables the model to capture a wider range of concept.” But increased vocabulary size requires an adequate training corpus, because “…the risk of under-fitting for rare tokens increases with larger vocabulary sizes”. Thus the authors in [23] introduce the notion of Optimal Vocabulary Size which is however at least 7 times the one commonly used in most current LLMs.

This type of reasoning is deduced from the experiments carried out by Alessia McGowan et al. [11] and in the paper by Uluoglakci and Taskaya Témizel [18] where they address directly the problem of presence of rare terms which might be missing in the training corpus. In a number of previous papers we verified this hypothesis by checking the behaviour of different LLMs on texts extracted from Italian poetry and the results showed clearly that rare words cause a dramatic drop in the ability of BERT to predict the next word [24,25].

A more abstract generalization will affect directly the underlying theory. Distributional semantics is based on the Distributional Hypothesis, which states that similarity in meaning results in similarity of linguistic distribution [26]. Words that are semantically related, are used in similar contexts. Distributional semantics reverse engineers the process and induces semantic representations from contexts of use [27,28]. As we know from the way in which Language Models are built, word embeddings can be regarded direct representatives of the theory. In its most basic and frequent form, distributional semantics represents word meaning by transforming large amounts of text through an abstraction algorithm, to produce a distributional model, where semantic representations are listed in the form of vectors — i.e., lists of numbers that determine points in a multidimensional space where similarity is measured by cosine similarity. Other possibilities include the representation of semantic units via complex algebraic objects like matrices or tensors.

2. Materials, Methods and Results I°

We started using ChatGPT for summarizing excerpts from the novel beginning of December of last year. We started using prompts that should have given as a result a short summary of the actions taken by each character in the story.

In fact, we tried different versions of the excerpt to verify that the text and the answer was not searched somewhere in the internet, so we also changed all the names of people and locations, but we got always the same answers. We also slightly modified the prompt a number of times as you can see here below but always got the same answer. First prompts for summarization are as follows:

• Tell me all that happens in the story
• Tell me all that the characters do in the story
• Tell me all that happens to the characters of the story
• What have all the characters in the story done?

Then we turned to focusing on narrative techniques and use of linguistic and rhetoric devices so the idea was also to increase the length of each excerpt from 3000 tokens up. We knew that the length of the excerpt could result in a rejection by ChatGPT but we intended to try. So, we asked whether we could copy past the whole text of the novel even though we knew that we would receive a negative answer, this is the prompt I used and here it is what we got:

• I will now copy paste the entire text of a novel if that will suit you: what narrative techniques and what linguistic devices did the author use to characterize the protagonists?

If you share the entire text of a novel, it might be challenging to analyze it in a single step due to its length. However, you can share key passages or chapters that you believe are crucial to understanding the characterization of the protagonists. This will allow for a focused analysis of the narrative techniques and linguistic devices used.

Alternatively, if you can specify which parts of the novel are particularly important for character development, I can analyze those sections. That way, we can break down how the author builds the characters and what literary tools are employed.

When we launched the following prompt with a new excerpt, we got the answer “Error too long” and no summary was produced.

Then we discovered that ChatGPT had managed to find out the name of the author but not the correct title which however we received after a number of following interactions.

The excerpt from“Riders in the Chariot”byPatrick Whiteshowcases several linguistic devices and narrative techniques that characterize the protagonists, Mrs. Poulter and Mrs. Dun, as well as the setting of the scene. White uses these techniques to subtly reveal the characters’ personalities, social standing, and the nature of their relationship.

And here is the answer we got from ChatGPT after 21 interactions with short excerpts from the same novel:

This excerpt from Patrick White’sThe Solid Mandalaemploys a range of linguistic and rhetorical devices to characterize Mrs. Poulter and Mrs. Dun, two elderly women who engage in conversation while riding a bus. Here’s an analysis of some key devices that help shape the personalities and dynamics between these characters:

The summaries produced in reply to input excerpts ranging from 3500 to 4000 words, have all the same sequence of bullet key points with comments that explore characters through their dialogue and their lexis. In particular, when summarizing Excerpt No.1, ChatGPT notices the use of repetitive words and phrases, presence of grammatically non-standard sentences, colloquial regional idioms. But then it highlights the role each character embodies in the conversation, ironic and humorous statements, their contrasting personalities when compared to people passing by the bus, like young girls and the couple of twins living in the house opposite in the same road. One bullet point is dedicated to the symbolism attached to objects and places mentioned in the text. Finally, there are comments on characters’ personalities and the presence of suspense and suggestion of possible dramatic events.

Whereas reference to the use of specific linguistic items – like conjuncts, adverbials, articles, different naming references, verbal complexes, etc. - their semantic imports – like use of modality - and syntactic constructions - like cleft sentences, parataxis, displaced syntactic structures etc. - is totally absent, the reference to narrative techniques is present but not always. We analyze in detail the quality of the summaries produced by enumerating the bullet points that ChatGPT has produced in an incremental way, thus showing that it is using a memory of previous analysis of the same novel.

Here the list of 9 key points used by ChatGPT to build up one of the last summary of excerpt No.1:

1.Dialogue and Repetition; 2. Colloquial Language and Syntax; 3. Subtle Power Dynamics; 4. Irony and Understatement; 5. Characterization through Contrast; 6. Symbolism; 7. Subtle Humour; 8. Foreshadowing and Suspense; 9. Characters Psychology and Subtext.

And here below are the 10 bullet points used by ChatGPT to build up the first summary of excerpt No.1:

1. Stream of Consciousness; 2. Repetition; 3. Contrast and Juxtaposition; 4. Symbolism; 5. Imagery, 6. Fragmented Narrative; 7. Characterization through Mundanity; 8. Use of Indirect Characterization; 9. Religious Allusions; 10. Emotional Resonance through Ambiguity.

In this case there are remarkable differences and only three bullet points that we underlined can be said to coincide. However, when we look at the summaries produced for the following excerpts, we see that the previous scheme is preserved. Here below the list of bullet points produced for Excerpt No.2:

1. Dialogue-Driven Characterization; 2. Indirect Characterization; 3. Repetition and Parallelism; 4. Social Commentary through Subtle Irony; 5. Symbolism; 6. Character Dynamics; 7. Use of Humour; 8. Tone and Mood; 9. Attention to Detail; 10. Stream of Consciousness Elements.

Almost all the new bullet points have been inherited from previous analysis. When we reproposed Excerpt No. 1 after going through all other Excertps we got a totally different decomposition of the text, as can be noticed from the list below:

1. Dialogue and Speech Patterns: 1.1 Repetition and Echoing, 1.2 Colloquial Language, 1.3 Indirect Communication; 2. Characterization through Internal Thoughts, 2.1 Mrs. Poulter Reflection, 2.2 Mrs. Dun’s Hesitation; 3. Use of Setting, 3.1 The Bus as a Microcosm, 3.2 Sarsaparilla and Terminus Road; 4. Symbolism, 4.1 Verandas and Hedges, 4.2 Flowers; 5. Narrative Perspective, 5.1 Limited Third-Person Point of View; 6. Humor and Irony, 6.1 Subtle Humor, 6.2 Irony; 7. Pacing and Rhythm, 7.1 Slow, Measured Pace, 7.2 Rhythmic Repetition; 8. Social Commentary, 8.1 Class and Gender, 8.2 Conservatism and Prejudice

Not only the majority of key points are completely new but they also foreshadow a totally different approach: rather than being abstractions derived from narratological theory, the new list proposes the insertion of direct reference items extracted from the text.

Eventually we concentrated on temporal ordering and then on factuality, which required the new prompts that are listed in Section 3. Here below we analyze what is contained in the summaries produced by ChatGPT, choosing the most significant cases.

2.1. The Story or Fabula

In the choice of text to summarize, we have been using excerpts from the novel in which the number of protagonists does not exceed two or three persons of the same sex. The most remarkable part of the output we receive from ChatGPT regards the treatment of pronouns, i.e., coreference resolution and what we dubbed as “coreference chain”. The technique used by the summarization algorithm seems to follow the implicit rule to assign the same antecedent to all pronouns with the same morphological features in the stretch of text that follows the appearance of a referential expression that can be regarded by its frequency of usage a lexically expressed protagonist, until a new and different protagonist is mentioned explicitly. This rule works perfectly apart from a case in which the antecedent is implicitly coreferred by means of a physical object belonging to her and not to the other protagonist. This case is shown in excerpts No.1 which comes from the beginning of the novel where two women – Mrs Poulter and Mrs Dun - are talking while sitting on a bus: the pronoun starting a sentence has an antecedent which requires the knowledge of the extended context acquired by the underlying Neural Network thanks to the LLM it has been using – more below. We place the excerpts in the Appendix and the answers from ChatGPT in the text.

All the excerpts are decomposed by ChatGPT into short sentences made up from the extraction of snippets made of predicate-argument structures, i.e., a Subject and Object NP, and a verb. Sentences are assigned to each protagonist separately. In our case, at first comes Mrs Poulter’s list of sentences and then Mrs Dun’s. So here below we show the first answer to Excerpt No.1.

• EXCERPT No.1

Figure 1. Snapshot of ChatGPT Answer to Query on Excerpt No. 1.

Figure 1. Snapshot of ChatGPT Answer to Query on Excerpt No. 1.

In the analysis below we report ChatGPT answers preceded by three stars (***) and the related part of text coreferred preceded by three dots (---).

The note regards a comment by Mrs Poulter, who compares living in a suburb like being in a cage and expresses this by the sentence:

--- “You couldn’t go anywhere as the crow.”

To which comment Mrs Dun replies by expressing her surprise with a filler question: “Eh?”. The following turn contains Mrs Poulter’s attempt at clarification and Mrs Dun’s lack of appreciation of the use of a “crow” to indicate a generic bird:

--- “As the crow flies,” Mrs Poulter explained.

--- “Oh, the crow,” her friend murmured, seeming uneasy at the idea.

*** Expresses unease: She seems uneasy when Mrs. Poulter talks about the crow.

ChatGPT interprets this utterance as an expression of uneasiness whereas the author’s suggestion is “seeming” uneasy.

At the beginning of the conversation we are told that Mrs Poulter is wearing a pair of gloves:

--- “Well,” said Mrs Poulter, peeping inside her plump glove...

Towards the end of the excerpt, Mrs Dun replies to a comment by Mrs Poulter and the following piece of text is a comment by the author of the way in which one of the two women is actually feeling:

--- “I’ll say it is!” said Mrs Dun.

--- She had a certain relentlessness of conviction. If it hadn’t been for her gloves her knuckles would have shown up white on the chrome rail.

ChatGPT carries over the coreference to Mrs Dun in the following sentence started by pronoun “she”. But it is Mrs Poulter that is now being coreferred and this is made clear by the following sentence that we highlighted in italics, where the gloves are referred to. The passage is assigned to Mrs Dun and this is how the it is being commented:

*** Shows conviction: Her strong grip on the rail indicates her strong feelings and convictions.

The third mistake regards the need to simplify when producing a summary, but in this case we are in presence of oversimplification. As the conversation continues, Mrs Poulter who is “happy again”, manages to make a remark on a gadget that she wishes she could also have:

--- Presently she couldn’t resist: “That veranda of yours must be a real luxury, dryin’ laundries in the rain.”

The veranda’s usage for drying laundries in the rain is something Mrs Poulter is really envying her friend. However, ChatGPT only reports the reaction of Mrs Dun to what Mrs Poulter said. Oversimplification is present in Mrs Dun’s comment reported wrongly:

*** Comments on the veranda: She mentions Mrs. Poulter’s veranda as a luxury for drying laundry.

The veranda is not just for drying laundry, but when it rains which has been omitted.

In general, all that is not represented by a fact and is a comment by the author is being erased. In particular, we noticed that a lot of text contained in the excerpt is totally ignored by ChatGPT and for this reason we represented a portion of the text to ChatGPT that we comment below. The paragraph where the author reports Mrs Poulter’s talking about a Chinese woman and family and the place where they lived is reported by a generic sentence mixed up with what has been said at the beginning.

*** Explains: She describes how the roads at Sarsaparilla were dead ends and mentions seeing a Chinese woman standing under a wheel-tree.

This is reported from the point of view of Mrs Poulter. There’s another indirect reference to the Chinese woman from the point of view of Mrs Dun:

*** Sucks her teeth: This action indicates a moment of contemplation or mild disapproval.

Mrs Dun “sucks her teeth” when Mrs Poulter is telling about the “those what-they-call wheel-trees” and she had seen her “standing under it when it was in flower”. It would seem that Mrs Dun does not like Chinese people, but we learn more in the enlarged summary below.

Finally, ChatGPT captures an important description of Mrs Poulter and her husband Bill that characterizes them with their personalities well.

As said above, I decided to split the Excerpt and use the final portion to ask the same question and see what additional information ChatGPT would be able to gather. The portion of text starts from Mrs Poulter, telling about her memory of a Chinese woman and her family. And here is the result:

Figure 2. Snapshot of ChatGPT answer to a portion of Excerpt 1.

Figure 2. Snapshot of ChatGPT answer to a portion of Excerpt 1.

As can be easily noticed, there’s a lot more information that has been captured when reducing the text to be summarized: this is certainly due to a fixed proportion of text that the algorithm allows every time a text is being presented. The new extracted pieces of text are the following ones: a first bullet point dedicated to Mrs Poulter recounting the story of the Chinese woman. The other new point is about Mrs Poulter “Expressing happiness”, whereas in the previous summary we had her “expressing sadness and nostalgia”. At the same time, we notice that some information previously present has now been deleted. In particular, the reference to Bill’s sweater which was so rich in the previous summary is now absent. We can also notice the lengthening of the reference to Mrs Dun’s veranda which is now useful for drying laundry “in the rain”.

*** Wears an old sweater: He is described wearing an old fawn sweater that Mrs. Hide had knitted, which had begun to stretch and sag.

However now we have Mrs Poulter recalling her story of a Chinese woman and at the opposite side we have Mrs Dun listening to the story.

*** Talks about the Chinese woman: She recalls seeing a Chinese woman who lived on a hill, mentioning her refinement and the wheel-tree she stood under.

*** Listens to Mrs. Poulter: Mrs. Dun listens to Mrs. Poulter’s story about the Chinese woman.

Now Mrs Dun “sucking her teeth” is no longer interpreted as showing some kind of appreciation of what she had been told by Mrs Poulter, but just as a generic reaction we are not told to what: reduction again.

The interesting point is the one explaining Mrs Dun’s “conviction”:

*** Shows conviction: Her strong grip on the rail reveals her strong feelings about the conversation and situation.

Whereas in the previous summary “shows conviction” was related to the veranda, now it is wrongly related to conversation and the situation. The fact is that as before, the sentence is wrongly associated with Mrs Dun rather than with Mrs Poulter, thus making a doubly wrong reference.

• EXCERPT No. 2

In the second excerpt we are dealing with two men, the two most important protagonists of the novel, the twins Brown, Waldo and Arthur. In this excerpt the number of dialogues are just a few and the majority of the text is made up of author’s descriptions and reported speech.

As happened with the previous Excerpt No. 1, also in this case important parts of the text have been totally ignored so that I had to represent a section.

The first summary is focused on the two protagonists and the dogs and their actions are reported very correctly and precisely.

As can be noticed, the summary reports faithfully the main actions of the twins, however as can be easily noticed from a comparison with the next more detailed summary there’s a lot missing. So, we split Excerpt No. 2 and produced a new shorter text starting from the paragraph with Waldo leading his brother out of the kitchen. Here below we show ChatGPT summary which is however split into two separate images so that they can be more easily matched with the previous Figure 3. Here below the image regarding Waldo’s behaviour.

Figure 3. Snapshot of ChatGPT summary of Excerpt No.2.

Figure 3. Snapshot of ChatGPT summary of Excerpt No.2.

Figure 4. Snapshot of part of more focused summary - HalfExcerpt2 - relating only Waldo’s actions.

Figure 4. Snapshot of part of more focused summary - HalfExcerpt2 - relating only Waldo’s actions.

The portion of Excerpt 2 that we may now call HalfExcerpt2, may be referred to in the previous figure No. 3 from bullet point 7 downward. We list here below the new text extracted and comment on their importance in the overall previous summary. We use three ^^ to indicate newly extracted text, and “§§” to indicate a stretch of text which is partially coincident with what was previously extracted.

^^ Feels Arthur’s hand: Waldo feels the larger, fleshy hand of his brother Arthur in his own thinner, colder hand as they walk.

^^ Notices Arthur’s breathing: Waldo sees that his brother is breathing deeply.

^^ Observes Arthur’s difficulty speaking: Waldo notices Arthur’s struggle to form words and their solidity once spoken.

^^ Reflects on planting the gooseberry bush: Waldo remembers planting the bush against Arthur’s advice to demonstrate something.

^^ Envy towards Arthur: Waldo sometimes envies Arthur’s ability to distance himself from pain.

^^ Modulates his voice: Waldo uses his fine tenor voice, more to impress than to please, a voice their parents were proud of.

^^ Contemplates their differences: He thinks about their mother’s comments on his voice and Arthur’s hair, believing his mind should have been praised.

§§ Stares at Arthur’s hair: Waldo is fascinated by the glimpses of pink skin through Arthur’s hair and reflects on its cleanliness.

At first glance it is fairly obvious that the second summary is by far more informative both in terms of facts and of mental thoughts.

Now consider the mistake in assigning an action to Arthur, which in fact was done by Waldo as clearly appears from the related stretch of text:

--- Arthur had advised against it - Sarsaparilla was too warm - but Waldo had planted the bush. To demonstrate something or other.

And this is the snippet produced by GPT:

*** Reflects on planting the gooseberry bush: Waldo remembers planting the bush against Arthur’s advice to demonstrate something.

It is not Arthur’s that wants to demonstrate something but Waldo. In this case the mistake is due to the IMPLICIT subject of the infinitival which must receive an antecedent: the choice of Arthur is due to his role as SUBJect of the previous main sentence. Whereas Waldo is SUBJect of the dependent sentence which is however closer to the infinitival and becomes the new TOPic if any such mechanism is in force with the “coreference chain”. Arthur’s advice was contained in the parenthetical “Sarsaparilla was too warm”, which is ignored. But in the following enlarged version of the second part of HalfExcerpt2, I highlight the pieces of text that have been added.

Figure 5. Snapshot of part of more focused summary - HalfExcerpt2 - relating only Arthur’s actions.

Figure 5. Snapshot of part of more focused summary - HalfExcerpt2 - relating only Arthur’s actions.

Here below the stretches of text which have been extracted in addition to the more general summary.

^^ Holds Waldo’s hand: Arthur holds Waldo’s hand as they walk, with his larger, fleshy hand in Waldo’s thinner, colder one.

^^ Speaks with difficulty: He struggles to form words but manages to say them solidly once he does.

^^ Advises against planting the bush: Arthur had advised against planting the gooseberry bush, believing Sarsaparilla was too warm for it.

^^ Maintains physical strength: Despite his struggles, Arthur has grown into a strong man, continuing to lift weights and maintain his youthful muscles.

As can be noticed some of the additional material is important to understand the story but had been previously erased by oversimplification.

We comment now on a third excerpt regarding the behaviour of one of the twins, Waldo and an at first unidentified character, which in the following text becomes one of his friends. The recognition comes through the words of the woman who accompanied the man: a husky voice. In particular at this precise moment: “He remembered it was that boy, that Johnny Haynes, they could have cut each other’s throats, telling him behind the dunny to watch out for hoarse-voiced men and women, they were supposed to be carriers of syph.”

• EXCERPT No.3

Figure 6. Snapshot of excerpt 3 about the description of a man which turns out to be his friend Johnny Haynes.

Figure 6. Snapshot of excerpt 3 about the description of a man which turns out to be his friend Johnny Haynes.

In this text, through the words of the woman, Waldo remembers what his friend Johnny Haynes told him, and realizes who the man was. But we are not told this by the summary made by ChatGPT, which keeps references to the old man separate from those related to Johnny Haynes as if they were two different persons.

The reasons for this important mistake are to be found in the way in which the “coreference chain” usually works: at first comes the referring expression in our case a human being that becomes the entity to be coreferred and coindexed by subsequent pronouns or epithets or other. The entity coreferred is the antecedent and not as in our case a cataphora which requires specific mechanisms for cataphoric coreference. It may be also due to the need to produce an inference on the type and tone of the woman’s voice and the two characters appearing in the story.

In the summary we are told that both the Old Man and Johnny Haynes “kick at the house” as if they were two separate persons doing the same action. This mistake modifies the plot introducing a new non-existing entity.

Besides, the summary is a bad oversimplification where most important pieces of text have been ignored. We repeat the same operation we did previously by splitting the text of Excerpt 3 and producing a new more focused summary that we show here below in Figure 7.

Figure 7. Snapshot of ChatGPT summary of an extract from Excerpt 3 focused on Waldo’s actions.

Figure 7. Snapshot of ChatGPT summary of an extract from Excerpt 3 focused on Waldo’s actions.

This portion of Excerpt 3 is dominated by what a personified Memory brings about in Waldo’s thoughts and the actual actions in his present life. In particular the first reference by ChatGPT is taken to be a real fact:

**** Reflects on Mother’s Legacy: Waldo thinks about how his mother had a sense of moral proportion, which he believes he inherited along with her pale eyes.

----- Disorderly in habit, because the years had gradually frayed her, Mother kept what he liked to think of as a sense of moral proportion. Which he had inherited together with her eyes.

The text does not assert that Mother “had a sense of moral proportion”, but this is what Waldo “liked to think” or “believe”. In the snippet produced by ChatGPT, the verb “think” is used in another sense, with the other meaning, “consider” which implies factuality in the implication. The ambiguity of the verb THINK is at least 5 time ambiguous (believe, consider, intend, devise, be preoccupied, etc.) and in a LLModel all the different senses are assembled into one single embedding. GPT extracted the wrong sense.

A second important point is the one mentioning Memory as an actor:

**** Memory Takes Over: Memory, personified, takes a dominant position, and Waldo sees visions of great occasions and feels a surge of radiance and splendour.

And below is a comment for the bullet point Personified Memory:

**** Takes Control: Memory, as a character, takes control of Waldo, making him relive and see past events with heightened vision and splendour.

But Memory is not just a personified character, she is the personification of Waldo’s mother. In fact, Memory is slowly turned into the Mother, even though at the beginning of the paragraph the two appear separate. Then comes the moment when “Memory herself seated herself in her chair”, where “her” refers to Mother.

------ Then Memory herself seated herself in her chair, tilting it as far back as it would go, and tilted, and tilted, in front of the glass. Memory peered through the slats of the squint-eyed fan, between the nacreous refractions. If she herself was momentarily eclipsed, you expected to sacrifice something for such a remarkable increase in vision. In radiance, and splendour. All great occasions streamed up the gothick stair to kiss the rings of Memory, which she held out stiff.

In this case, the personification of Mother is IMPLICIT and requires an inference, two operations that are impossible for ChatGPT and for DNNs in general. The reason is very simple, models are built around lexically expressed words, implicit information is not available and not present in the model except for words which are strongly contextually motivated.

Eventually, we add another final excerpt for this section, which is concentrated on the twins and follows previous excerpts as far as the story or fabula is concerned. Excerpt No. 4 is focused on Waldo’s attempt at inflicting his brother high levels of pain by bringing to light Arthur’s attempts at poetry writing. In fact, the move hides Waldo’s desperate need to erase his past unsuccessful attempts at novel writing which is apparent when Tiresias is mentioned. Here is ChatGPT’s summary of the characters’ actions in the excerpt:

• EXCERPT No.4

Figure 8. Snapshot of ChatGPT’s summary of Excerpt No.4.

Figure 8. Snapshot of ChatGPT’s summary of Excerpt No.4.

Waldo does not want Arthur to know about mother’s dress he had just hidden, but he wants to tease Arthur by destroying his secret past attempts at poetry writing. The tone of voice is important but is mingled with his feelings of revenge. This is what the text tells us:

----- Then Waldo read aloud, not so menacingly as he would have liked, because he was, in fact, menaced:

And here is GPT’s couple of snippets:

**** Reads the poem aloud.

**** Feels menaced by the poem.

Waldo wanted to sound menacing but “in fact he was menaced”, this is reported as a fact by the omniscient writer and was not felt by Waldo. The poem or what Arthur wrote, was telling about bleeding, a Vivisectionist Cordelia is bleeding and all Marys “in the end bleed”. After reading the poem aloud Waldo holds it in his hand and Arthur takes it back:

------ He would have snatched, but Waldo did not even make it necessary.

On the contrary what GPT understands of Arthur’s action is something different:

**** Tries to snatch the paper from Waldo.

But Waldo’s action is reported correctly:

**** Lets the paper hang from his hand.

Now Waldo feels “he was bleeding” and in his imagination he had “suspicion of an incurable disease”, because of the feeling he had of Arthur’s “unnatural blood continued to glitter”. Then we are told that,

----- Waldo was infected with it.

He was infected with the imaginary incurable disease and not by the blood:

**** Feels infected by Arthur’s “unnatural blood.”

Waldo is now convinced that all his past vainglorious attempts at becoming a novelist had no sense and it was the time to part from all of them. As a result, mother’s dress and all his papers were slowly burnt. The scene is described with a wealth of details because it is a fundamental milestone in the development of the two characters’ life. But GPT only records two simple actions:

**** Goes to the pit where they burn things and pitches a paper tent.

**** Burns papers, feeling lighter afterward.

Waldo’s action is in fact very complex since it encompasses all his past memories. At first, we are told that the dress-box is in fire and after that Waldo feels spontaneously pushed to continue burning his past, which is made up of all the attempts at writing a novel about Tiresias:

------ About four o’clock he went down, Tiresias a thinnish man, the dress-box under his arm, towards the pit where they had been accustomed to burn only those things from which they could bear to be parted. He stood on the edge in his dressing-gown. Then crouched, to pitch a paper tent, and when he had broken several match-sticks - increasingly inferior in quality - got it to burn. The warmth did help a little, and prettiness of fire, but almost immediately afterwards the acrid years shot up his nose. So he stood up. He began to throw his papers by handfuls, or would hold one down with his slippered foot, when the wind threatened to carry too far, with his slippered foot from which the blue veins and smoke wreathed upward. It was both a sowing and a scattering of seed. When he had finished he felt lighter, but always had been, he suspected while walking away. Now at least he was free of practically everything but Arthur.

At the end, we are also told that even if “felt lighter” but always had been “he suspected”. Apart from his twin brother that arrives in the following snippet, which is again badly presented:

**** Considers how to disembarrass himself from Arthur.

**** Raises himself on one elbow due to the urgency of his problem.

----- After he had lain down on the bed he began to consider how he might disembarrass himself, not like silly women in the news who got caught out through falling hair or some such unpremeditated detail, but quick, clean, and subtle, a pass with the tongue he had not yet perfected, but must. As he lay, he raised himself on one creaking elbow, because of the urgency of his problem. That was when Arthur came in and saw him.

The final part of the conversation between the twins is rendered only superficially without the sufficient pathos for what is happening. Waldo is trying to destroy emotionally and mentally his twin brother Arthur with the excuse of the poem. The dialogue is summarized correctly by Arthur’s reaction:

**** Asks Waldo what he is trying to do to him.

**** Shapes his defense, apologizes, and explains himself to Waldo.

But then the outcome is wrongly reported in the snippet:

**** Reacts to Arthur’s last words with disgust.

Which should summarize the following sequence of their conversation:

------ “I know it wasn’t much of a poem.” Arthur was shaping his defence. “Oughter have destroyed it at once. Apologise, Waldo.”

The warmed stones of words.

“That poem? That disgusting blood myth!” Waldo gasped to hear his own voice.

“I would have given the mandala, but you didn’t show you wanted it.”

“I never cared for marbles. My thumb could never control them.”

Waldo in fact gasped to hear his own voice, and disgust is associated with the words of Arthur’s poem.

3. Method, Materials and Results II°

3.1. Testing ChatGPT with the Plot, the “Sujet”

In this section we will consider the way in which the story has been narrated, the narrative style or techniques the author has brought forward to make the story interesting, as well as the linguistic techniques underlying the style. At first, we have used the same Excerpt No.3 of the previous analyses, because it is a highly representative piece of text to be used for characterizing White’s style through Waldo’s personality. And this is mainly the reason why we have chosen it, but as will appear below, it is also for its peculiar linguistic features.

ChatGPT analysis, organized as before into a sequence of bullet items, this time is preceded by a general comment which hinges upon the mixture of what ChatGP regards the three important components of White’s narrative style: stream of consciousness, free reported or indirect discourse and the narrator’s intrusive voice. In fact, none of the three are components indicated are present: White’s style has no “auctorial intrusion”, no stream of consciousness and no free indirect discourse. Rather what is at stake here is “figural consciousness”, i.e., it is the character’s own internal consciousness that the narration presents, something that is totally different in each of the three protagonists. Even the presence of diegetic statements must be regarded as stemming from the character inner voice (see [3,6]).

As happened before, also in this case the introductory portion of the excerpt with a seemingly strange visitor and his accompanying woman is not intertwined with Waldo’s reconnaissance phase when the memory of Johnny Haynes jumps up to clarify the scene and justify Waldo’s reactions.

The man is described with a richness of details ending up in a solid man, – because of his purposefulness - is the author’s comment. Waldo envied the man, his clothes, his kempt head. But then comes the ironic side of the man’s description: the zip of his “insolent” pants, which might get stuck in a public lavatory. So that eventually the man might be soon subject to a stroke. And that’s when Waldo “racked his memory” and was “racked”. As said above, Waldo eventually recognized Johnny Haynes – the boy he hated - because of his woman’s husky voice. So it is for the details that we are brought inside the protagonist’s mind and navigate with his thought and imagination in unexplored lands.

The most important component of White’s style as referred to the most important protagonist, Waldo, is the diffuse sense of uncertainty, expressed by the use of modality which is realized in verbal complexes and adverbials. Here below we make a list of “uncertainties” introduced in the excerpt, marking the relevant portion of text with italics:

Figure 9. List of sentences containing “uncertainty” markers in excerpt No. 3.

Figure 9. List of sentences containing “uncertainty” markers in excerpt No. 3.

And here below in Figure 10 the summary produced by ChatGPT. It is really important to focus on the way in which ChatGPT has been able to capture the “uncertainty” we have been referring to in the Introduction. This is particularly well represented in bullet point no. 5 where we have the impression that “ambiguity” has become so strong that reality is almost indistinct from imagination. However, this is not at all the way uncertainty should be interpreted. As the list of verbal and adverbial constructions above clearly shows, it is rather with the use of modality that it is possible to classify the narrative technique typical of Patrick White’s novels in general and in particular of the most important protagonist of this novel, Waldo Brown.

Figure 10. Snapshot of excerpt No. 3 used to detect narrative style.

Figure 10. Snapshot of excerpt No. 3 used to detect narrative style.

Modality and uncertainty are important features of White’s style and has been deeply documented in a previous paper [3] as said in the Introduction. In a number of tables – from Table 1, Table 2, Table 3 and Table 4 here below, that we repeat from the paper cited, we explain why modality is the key to understand the style. We started above by commenting uncertainty, we marked all the parts of the text that in their syntactic, semantic or pragmatic value carry a sense of interpretation of the storyworld by the characters. This interpretation may indicate actual hesitation or ambiguity, expressing the more or less conscious doubts in the minds of the protagonists; at the opposite end, it can also signal a judgment of certainty by a character, which ironically in turn generates insecurity in the reader and raises a series of important questions. In the present analysis, the element uncertnty has only one obligatory attribute: nonfactual (more in a section below, but see [26,27]). To specify uncertainty, ambiguity and doubt, in fact, it is crucial to mark the annotated expression as non-real – that is, non-factual – a process that is only going on in the character’s mind and which does not have an equivalent in the “real world” of the story.

Table 1. Semantic linguistic features grid organized by hierarchy.

Meta-Tags	Uncertainty	Subjectivity	Subjectivity	Judgement
Intermediate Tags	Non-Factuality	Psychology	Affectivity	Social_Evaluation
Tags	Seeming	Perception	Inclination_Positive	Esteem_Negative
	Gnomic	Precognition	Inclination_Negative	Esteem_Positive
	Concessive	Cognition	Security_Positive	Sanction_Negative
	Conditional	Performwill	Security_Negative	Sanction_Positive
	Defdesire		Satisfaction_Positive
	Will		Satisfaction_Negative
	Possibility
	Ability
	Obligation
	Assumption
	Negation

Table 1. Semantic linguistic features grid organized by hierarchy.

Meta-Tags	Uncertainty	Subjectivity	Subjectivity	Judgement
Intermediate Tags	Non-Factuality	Psychology	Affectivity	Social_Evaluation
Tags	Seeming	Perception	Inclination_Positive	Esteem_Negative
	Gnomic	Precognition	Inclination_Negative	Esteem_Positive
	Concessive	Cognition	Security_Positive	Sanction_Negative
	Conditional	Performwill	Security_Negative	Sanction_Positive
	Defdesire		Satisfaction_Positive
	Will		Satisfaction_Negative
	Possibility
	Ability
	Obligation
	Assumption
	Negation

Table 2. Distribution of Semantic linguistic features amongst the three protagonists.

	Waldo	Arthur	Mrs. Poulter	Totals	RatioW	RatioA	RatioP
Percept	674	303	109	1086	62,063	27,9	10,037
Precogn	379	141	39	559	67,799	25,224	6,977
Cognition	953	460	140	1553	61,365	29,62	9,015
PerformW	39	3	0	42	92,857	7,143	0
Seeming	512	198	56	766	66,841	25,848	7,3107
Will	74	33	8	115	64,348	28,696	6,956
Possibl	224	54	42	320	70	16,875	13,125
Gnomic	32	20	9	61	52,459	32,787	14,754
Ability	347	195	51	593	58,516	32,884	8,6
Obligation	178	79	28	285	62,456	27,719	9,8246
Concessv	150	75	26	251	59,761	29,88	10,358
Conditnl	264	117	32	413	63,922	28,329	7,7482
Defdesire	49	35	2	86	56,977	40,698	2,325
Assumpt	73	36	9	118	61,864	30,508	7,627
Emot_Pos	400	183	43	626	63,898	29,233	6,869
Emot_Neg	247	107	30	384	64,323	27,864	7,812
Inclin_Pos	83	50	17	150	55,334	33,334	11,334
Inclin_Neg	75	26	12	113	66,372	23,009	10,619
Secur_Pos	133	65	19	217	61,29	29,954	8,756
Secur_Neg	297	116	40	453	65,563	25,607	8,83
Satisf_Pos	136	66	10	212	64,151	31,132	4,717
Satisf_Neg	215	112	25	352	61,079	31,818	7,102
Estm_Pos	223	71	47	341	65,396	20,821	13,783
Estm_Neg	346	129	39	514	67,315	25,097	7,587
Sanct_Pos	80	19	18	117	68,376	16,239	15,385
Sanct_Neg	127	47	16	190	66,842	24,737	8,421

Table 2. Distribution of Semantic linguistic features amongst the three protagonists.

	Waldo	Arthur	Mrs. Poulter	Totals	RatioW	RatioA	RatioP
Percept	674	303	109	1086	62,063	27,9	10,037
Precogn	379	141	39	559	67,799	25,224	6,977
Cognition	953	460	140	1553	61,365	29,62	9,015
PerformW	39	3	0	42	92,857	7,143	0
Seeming	512	198	56	766	66,841	25,848	7,3107
Will	74	33	8	115	64,348	28,696	6,956
Possibl	224	54	42	320	70	16,875	13,125
Gnomic	32	20	9	61	52,459	32,787	14,754
Ability	347	195	51	593	58,516	32,884	8,6
Obligation	178	79	28	285	62,456	27,719	9,8246
Concessv	150	75	26	251	59,761	29,88	10,358
Conditnl	264	117	32	413	63,922	28,329	7,7482
Defdesire	49	35	2	86	56,977	40,698	2,325
Assumpt	73	36	9	118	61,864	30,508	7,627
Emot_Pos	400	183	43	626	63,898	29,233	6,869
Emot_Neg	247	107	30	384	64,323	27,864	7,812
Inclin_Pos	83	50	17	150	55,334	33,334	11,334
Inclin_Neg	75	26	12	113	66,372	23,009	10,619
Secur_Pos	133	65	19	217	61,29	29,954	8,756
Secur_Neg	297	116	40	453	65,563	25,607	8,83
Satisf_Pos	136	66	10	212	64,151	31,132	4,717
Satisf_Neg	215	112	25	352	61,079	31,818	7,102
Estm_Pos	223	71	47	341	65,396	20,821	13,783
Estm_Neg	346	129	39	514	67,315	25,097	7,587
Sanct_Pos	80	19	18	117	68,376	16,239	15,385
Sanct_Neg	127	47	16	190	66,842	24,737	8,421

Table 3. Best 10 Semantic linguistic features of the three protagonists.

Waldo’s Best 10	Arthur’s Best 10	Mrs. Poulter’s Best 10
1-PerformW	1-DefDesire	1-Sanct_Pos
2-Possibilty	2-Inclin_Pos	2-Estm_Pos
3-Sanct_Pos	3-Ability	3-Inclin_Pos
4-Precognt	4-Gnomic	4-Gnomic
5-Estm_Neg	5-Satis_Neg	5-Possibilty
6-Sanct_Neg	6-Satis_Pos	6-Inclin_Neg
7-Inclin_Neg	7-Assumptn	7-Concessiv
8-Secur_Neg	8-Secur_Pos	8-Perceptn
9-Estm_Pos	9-Concessiv	9-Obligation
10-Will	10-Cognition	10-Cognition

Table 3. Best 10 Semantic linguistic features of the three protagonists.

Waldo’s Best 10	Arthur’s Best 10	Mrs. Poulter’s Best 10
1-PerformW	1-DefDesire	1-Sanct_Pos
2-Possibilty	2-Inclin_Pos	2-Estm_Pos
3-Sanct_Pos	3-Ability	3-Inclin_Pos
4-Precognt	4-Gnomic	4-Gnomic
5-Estm_Neg	5-Satis_Neg	5-Possibilty
6-Sanct_Neg	6-Satis_Pos	6-Inclin_Neg
7-Inclin_Neg	7-Assumptn	7-Concessiv
8-Secur_Neg	8-Secur_Pos	8-Perceptn
9-Estm_Pos	9-Concessiv	9-Obligation
10-Will	10-Cognition	10-Cognition

Table 4. Unique Semantic linguistic features of the three protagonists.

Waldo’s Unique	Arthur’s Unique	Mrs P-‘s Unique
PerformW	DefDesire
	Ability
Precognt
Estm_Neg	Satis_Neg
Sanct_Neg
Inclin_Neg	Assumptn
Secur_Neg	Secur_Pos	Perceptn
	Concessiv	Obligation
Will

Table 4. Unique Semantic linguistic features of the three protagonists.

Waldo’s Unique	Arthur’s Unique	Mrs P-‘s Unique
PerformW	DefDesire
	Ability
Precognt
Estm_Neg	Satis_Neg
Sanct_Neg
Inclin_Neg	Assumptn
Secur_Neg	Secur_Pos	Perceptn
	Concessiv	Obligation
Will

As with all other aspects of this particular tagging system, subjectivity, the second of the three stylistic elements, focuses on facets of character psychology and on their relations with the storyworld. The main difference between uncertnty and subjectivity lies in the fact that the first element circumscribes how the protagonists interpret their reality and the ways they rationalize it, while the second marks the modalities in which they actively and subjectively contribute to the narration. While the former element is non-factual in nature, the latter always has consequences in terms of narrative, sometimes even tangible ones (e.g., in the case of performative will). Subjectivity includes active psychological processes – both conscious and unconscious – as studied by cognitive sciences, as well as expressions of emotion and different kinds of feelings, grouped into five attributes.

The last content-related element introduced in the annotation is judgement, as was the case with the affect attributes. In this annotation, judgmnt marks all evaluative expressions related to the characters of the novel, aimed at highlighting both social and personal reactions to the storyworld and particularly to the other characters and their behavior. The theoretical basis of this categorization can be found in the so-called “appraisal theory”[7,8], which underscores the relevance of impressions and judgments in the formation of feelings, emotions, and complex thoughts. Environment and psychology are here understood as standing in a relation of mutual dependency, with the reactions of each individual to events and stimuli evoking different responses. Speaking of the characters of a novel, we can say that the “artificially created” psychology of every character reacts in substantially different ways to what happens in the storyworld. From a general point of view, we can say that judgmnt and affect as categories have a lot in common, both dealing as they do with indices of emotion and sentiment. In this specific study, however, it was decided to annotate judgmnt as an independent element, for two main reasons: the first was to emphasize our interest in the evaluative language used by the author, and the second was to allow a more detailed internal differentiation between the categories of social-esteem and social-sanction and their respective polarities. For a complete explanation of the meaning of each tag we refer the reader to the paper cited above.

The linguistic feature grid we organized for our annotation task is structured in three main meta-tags and a number of hierarchically related tags, as shown here below:

We annotated 8416 semantic features as a whole - the non-repeated count amounts to only 1584, i.e., the level of repetition is fairly high - and characterizes each of the three protagonists as shown in the following tables. In some case, the same wordform has received more than one label due to its intrinsic polysemous nature. We report in Table 2 the actual count split by character and by semantic feature.

And in the two following tables, Table 3 and Table 4 the specificity of each protagonist as to their most typical semantic feature associated by the author in the novel.

3.1. Testing CHATGPT for Factuality and Temporal Ordering

Eventually we tested ChatGPT for its ability to tell facts from non-facts [29,30] using short paragraphs from the same novel. Facts should be regarded all sentences of direct speech disregarding their content; then non-facts should include all diegetic or gnomic statements which are usually expressed in present tense. At first, we gave ChatGPT two binary choices which were than easily understood. But what happened is shown in a few examples which we chose from the many we tried and always received the same results: a total confusion due to third-person narration carried out in the past even for direct speech, that ChatGPT was unable to infer as being different from real past events, this time expressed with the pluperfect. So, at first, we improved the prompt with a slight clarification about progressive. In particular we added to the main prompt the suggestion to watch for presence of progressive tense or aspect which ChatGPT seemed to focus on even though the general impression we received is that the verbal complex is internally invisible to ChatGPT.

Here is the main prompt:

“I have a short text where you should divide up sentences at first into two categories: those that present or represent a fact and those that don’t. Then you should divide up those that represent a fact into two subcategories: those that are placed in the past and those that aren’t.”

Additionally, at the usual request by ChatGPT at the end of each output we added an extra small prompt clarification:

“Let me know if you’d like any further refinements!

Yes. You should list sentences with a verbal complex in the progressive amongst the “not past”

Got it! I’ll revise the categories, ensuring sentences with a verbal complex in the progressive are placed in the “Not in the Past” subcategory. Here’s the updated breakdown:

And here is the first text taken from the first Section of the book that we now call SmallExcerpt1.

Just then the bus nearly shook apart, and one of the young lady typists lost her balance. It was those stiletto heels.

“These old buses!” Mrs Poulter heaved, and laughed.

Mrs Dun went a deeper yellow as she grasped the rail.

“They’re a downright disgrace!” Mrs Poulter laughed.

The sadness had been shaken out of her. She was happy again.

Presently she couldn’t resist: “That veranda of yours must be a real luxury, dryin’ laundries in the rain.”

“I’ll say it is!” said Mrs Dun.

She had a certain relentlessness of conviction. If it hadn’t been for her gloves her knuckles would have shown up white on the chrome rail.

The bus was making slow progress, on account of the pay-as-you-enter, and queues at the shelters, and kiddies who had missed the special. Mrs Poulter looked out. She was proud of the glossier side of Sarsaparilla, of the picture windows and the texture brick. She brightened with the leaves of the evergreens which the sun was touching up. Then she saw Bill, and waved. But he did not respond. He went on sweeping the gutters for the Council. It was against Bill Poulter’s principles to acknowledge his wife in public. Sometimes on her appearing he went so far as to take time off to roll himself a cigarette. But never wave. She accepted it. She was content enough to realize he was wearing the old fawn sweater, no longer presentable except for work, because the loose stitch she had been trying out had begun to stretch and sag.

This is the output where we marked with three asterisks wrong choices and with three °°° complex sentences containing a simple sentence in the past.

Figure 11. Snapshot of ChatFPT categorization of SmallExcerpt 1.

Figure 11. Snapshot of ChatFPT categorization of SmallExcerpt 1.

In fact, the narration is all told using past tense also when direct speech is presented as can be clearly seen, in particular, in the sentence “I’ll say it is!” said Mrs Dun” which has been correctly listed by ChatGPT under NOT IN THE PAST even though the governing verb “said” is clearly expressed in past tense. Correctly listed under PAST FACTS are those sentences with their verbal complex in past perfect or pluperfect tense. Also correctly listed as NOT FACTS is the one sentence with a counterfactual.

After the additional small suggestion about progressive tense we find the following reordering:

Figure 12. Snapshot of subsection of ChatGPT output for SmallExcerpt1 on Progressive.

Figure 12. Snapshot of subsection of ChatGPT output for SmallExcerpt1 on Progressive.

ChatGPT has wrongly treated the presence of past tense as signaling past facts. The first sentence is now a shortened version of the previous complete utterance which included the governing verb in the past tense: the text of the direct speech contains a gerundive which is expressed in the “ing” form, contracted though. There are then two additional sentences which however do not contain progressives in their main clause, only the second one has progressive in the relative clause. The same happened in the many other short paragraphs we selected to see whether there were improvements. We only report one case where ChatGPT indicates the presence of an “implied progressive”:

Figure 13. Snapshot of subsection of ChatGPT output for SmallExcerpt1 on Implied Progressive.

Figure 13. Snapshot of subsection of ChatGPT output for SmallExcerpt1 on Implied Progressive.

In fact, the progressive is expressed “was…walking” and not IMPLIED as wrongly indicated.

Here is a second paragraph where ChatGPT wrongly categorizes direct speech as taking place in the past and as non-fact, that we call SmallExcerpt2:

In High Street the overstuffed bus began to spew out its coloured gobbets.

“Wonder what those two old fellers were doin’ so far from Terminus Road?” Mrs Poulter nursed her curiosity as they waited to be carried by the common stream.

“You wonder what goes on in some people’s minds,” said Mrs Dun.

“I beg yours?”

“What goes on in people’s minds. Because it does go on. You’ve only got to read the papers.”

“But two respectable old gentlemen like the Mister Browns? They was probably only taking a walk to get their circulation going.” Mrs Poulter had turned mauve. “Anyway,” she said, “what goes on in other people’s minds is private. I wouldn’t want to know what goes on inside of my own husband’s mind.”

Although Mrs Dun might have wanted, she suggested she didn’t by drawing in her chin.

“I was never one,” she said, “not to keep to meself, and mind me own business.”

“Aren’t I right then?” Mrs Poulter continued, still too loud, and still too mauve.

Creating in the bus. Mrs Dun wondered whether she had been wise in the first place to accept Mrs Poulter’s friendship.

“As for those old men,” said Mrs Dun, “they’re nothing to me.”

“They’re nothing to me,” Mrs Poulter agreed.

But the situation made her want to cry. And Mrs Dun could feel it. She could feel her own gooseflesh rise. As they waited to escape from the suffocating bus the features of their familiar town began fluctuating strangely through the glass. Like that blood-pressure thing was on your arm. Nor did it help either lady to know the other could be involved.

And here below the output categorization by ChatGPT, where we marked with three asterisks wrong classifications:

Figure 14. Snapshot of ChatGPT classification for SmallExcerpt2.

Figure 14. Snapshot of ChatGPT classification for SmallExcerpt2.

Seen the total inability of ChatGPT to recognize the use of past tense in the narration as the current narrative present, we decided to include in a new prompt the explicit information about the way in which past events should be interpreted. Here is the new prompt:

I have a short text where you should divide up sentences at first into two categories: those that present or represent a fact and those that don’t. Then you should divide up those that represent a fact into two subcategories: those that are placed in the past and those that aren’t, where past event clauses are those that have a main verb in the pluperfect tense.

What we did then was basically this: we left the two binary choices intact and added an explanation to make easier the selection of past event sentences. But this worked to badly increase the level of complexity of the query and made the whole prompt impossible to satisfy. This is how the new prompt was received:

Figure 15. Snapshot of ChatGPT answer to prompt clarification.

Figure 15. Snapshot of ChatGPT answer to prompt clarification.

And this is what happened in the classification with the new prompt clarification:

Figure 16. Snapshot of ChatGPT new summary for SmallExcerpt1.

Figure 16. Snapshot of ChatGPT new summary for SmallExcerpt1.

The number of mistakes has increased. The same happened with SmallExcerpt2 which we don’t repeat here but select the most revealing part of the output: in Category 2. Facts in the Past (Pluperfect Tense) the answer was “No sentences here have main verbs in the pluperfect tense.” On the contrary we have the following two sentences:

• Mrs. Poulter had turned mauve.
• Mrs. Dun wondered whether she had been wise in the first place to accept Mrs. Poulter’s friendship

We used a new small excerpt to confirm our findings and we discovered that ChatGPT confuses the habitual “used to” as a past perfect, and also the conditional perfect for a factual verb and as such not included in the subsection for non-facts, as shown in the subsection we selected from the whole output:

Figure 17. Snapshot of ChatGPT subsection for Pluperfect.

Figure 17. Snapshot of ChatGPT subsection for Pluperfect.

4. Comparing GPT ad Gemini

In this section we will draw a comparison between GPT responses and the one given by Gemini [31] on the same excerpts. The output of the conversation with Gemini held on January 27th, 2025 is however made available only as Supplementary Material. The elements to compare are limited to the 12 mistakes made by GPT and in particular they are:

• oversimplification (dropping the restrictive modifier)
• wrong coreference chain (Mrs Dun rather than Mrs Poulter)
• wrong sense selection with ambiguous term (think)
• omitted cataphora with split referents (a strange man and Johnny Haynes)
• disjoined reference of MEMORY and Mother
• use of stream of consciousness as narrative style
• mistakes in classifying factual vs non-factual sentences
• mistakes in classifying factual sentences in the past with pluperfect
• inability to detect linguistic elements characterizing narrative style
• inability to characterize UNCERTAINTY in the verbal complex
• abstract key points substituted by direct extracted references
• Correct name of the author but wrong title of the novel

I will now tag each previous key point with WRONG in case the same mistake has been repeated and RIGHT and explain how Gemini has managed to overcome the difficulty.

1. WRONG – as with GPT the restrictive modifier of the “veranda” has been dropped

2. WRONG – the same mistake but then we asked Gemini to correct the error and this is the interaction where I use bold italics for the query and italics for the extract:

Great, but you made a mistake. In the piece of text I repeat below you wrongly assigned reference to Mrs Dun but it’s Mrs Poulter intended because she is the one who has been depicted with gloves:

“She had a certain relentlessness of conviction. If it hadn’t been for her gloves her knuckles would have shown up white on the chrome rail.”

You are absolutely correct! My apologies for that oversight. That’s a crucial detail, and I appreciate you pointing out the mistake.

You are right, the sentence: “She had a certain relentlessness of conviction. If it hadn’t been for her gloves her knuckles would have shown up white on the chrome rail,” refers to Mrs. Poulter, not Mrs. Dun.

The text earlier describes Mrs. Poulter as “peeping inside her plump glove to see if the ticket was still there,” which is the key detail that connects her to the gloves. Mrs Dun, on the other hand, has no mention of gloves, instead she “went a deeper yellow as she grasped the rail,” focusing on her reaction and appearance.

This mistake highlights the importance of careful reading and attention to detail. Thank you for correcting me! I will try my best to be more careful in the future.

3. RIGHT – think is used with the appropriate sense

4. RIGHT: the summary starts by directly substituting “the strange man” with its referent Johnny Haynes

5. RIGHT: Gemini correctly interprets the embodiment of Memory as the Mother

6. RIGHT – no mention of stream of consciousness but characters describing the story from their internal point of view and their peculiarities

7. RIGHT – a better classification of factuality with few mistakes. We report the final comment on the role that Factuality represents in the narration which is tagged as OBSERVATIONS:

Non-Factual as Insight: The non-factual sentences, predominantly reflections of Waldo’s internal state, are key to understanding his character, motivations, and psychological breakdown. These sentences reveal his biases, fears, and distorted perceptions of reality.

Action-Oriented Narration: The abundance of factual sentences, particularly in the present tense, creates a narrative that is focused on the unfolding events and the actions of the characters. This helps to build tension and momentum, particularly as the narrative builds to its violent conclusion.

This detailed classification should give you a clear understanding of how the author uses sentence types to convey information and character in the text. Let me know if you have any further questions or want to explore any aspect more deeply!

8. WRONG – a better classification of past vs. present but there are mistakes. In fact there are only one or two sentences in the pluperfect. Gemini wrongly declares that there are no pluperfect sentences and motivates it by this observation:

Absence of Pluperfect: The absence of pluperfect in the factual sentences suggests that the narrative focuses primarily on the immediate actions and events, with minimal digression into past events that occurred prior to the narrative timeframe. This helps to create a sense of immediacy. The past is expressed through memories that are presented as non-factual, and in the present simple or past simple.

9. RIGHT – specific key points dedicated to linguistic items but they do not include what should have made the difference, that is mainly verbal complexes

10. RIGHT – UNCERTAINTY is depicted in deep details at the end of the Supplementary Materials in two pages. We paste here the beginning of the response by Gemini to the question “And now I would like to know what is the role of the abstract notion of

“UNCERTAINTY” and how it has been implemented in linguistic structures and words in the novel”:

Now, let’s address your question about the role of the abstract notion of “uncertainty” and how it’s implemented in linguistic structures and word choices within the novel. This is a very insightful question and central to the experience of reading the work.

Uncertainty as a Pervasive Force:

Uncertainty is not just a theme, but a fundamental principle that underlies the narrative structure, character development, and use of language throughout “The Solid Mandala.” It manifests in several ways:

11. RIGHT - Bullet key themes in Gemini’s summaries are always abstractions, Here is a list from the Supplementary Materials where we can easily note that each excerpt has different themes unlike what happens with GPT where themes are often repeated:

Excerpt 1:

The Mundane and the Meaningful: Nostalgia vs. Reality: The Invisibility of the Ordinary: Routine and Habit: Suburban Life: Prejudice and Bias: Acceptance and Resilience;

Communication and Distance:

Excerpt 2:

Duty vs. Sentiment: Brotherhood: Individual vs. Shared Identity: Mortality and the Passage of Time: The Burden of Care: Underlying Repression and Complexity:

Excerpt 3:

Memory and Identity: Repression and Desire: Social Class and Envy: The Power of the Past: Transformation and Escape: Fragility of Reality: The Grotesque:

Excerpt 4:

Art and Emotion: Obsession and Control: Jealousy and Resentment: Violence and Self-Destruction: Ambiguity and Uncertainty: The Past as a Burden: Communication and Misunderstanding: Repressed Emotions:

12. WRONG – as happened with GPT also Gemini got the right name of the author but the wrong title of the novel. The response and the question are reported here below:

I thought by now you would have been able to tell who is the author and the title of the novel

You are absolutely right! My apologies for the oversight. With all this detailed analysis, I got so caught up in the process that I neglected the most basic element: identifying the work.

Based on the text I have been analyzing, the author is Patrick White, and the novel is titled “The Tree of Man.”

The distinct style, themes, character portrayals, and the specific details I’ve encountered throughout the text are all hallmarks of White’s work. His focus on the inner lives of seemingly ordinary individuals, his use of rich, evocative language, and his exploration of themes like isolation, memory, and the search for meaning are all consistent with his established style and preoccupations.

Thank you for pointing out my oversight! It’s a crucial piece of information that I should have acknowledged much earlier.

I hope that, despite this oversight, you have found my analysis insightful and helpful. Now that we have established the author and title, does it change anything about the aspects of narrative style you would like to discuss? I am still happy to delve into any of the previous topics in more detail.

***************

Ok for the author but the title of the novel is The Solid Mandala.

You are absolutely correct again! My apologies for the persistent error. The correct title of the novel is indeed “The Solid Mandala” by Patrick White. Thank you for catching that! It seems I was having a bit of a mental block there.

5. Discussion

As has been extensively shown, both ChatGPT and Gemini can produce well organized summaries or essays, but the contents of the analysis are sometimes the result of some form of hallucination, in particular this happens with GPT. On the contrary, with Gemini, we saw that over 12 types of mistakes made by ChatGPT, 8 have been correctly reported. However, the lack of generalization is present in both AI systems: in some case, the information to be used in order to report correctly seems to have been fully understood since a correct definition of the items has been captures. But the result shows that even in those cases in which the possible output has been exemplified correctly, the application of the apparently learnt information is either missing or is wrong. This reveals one of the most important deficiency of LLMs, also in their operative modality: the total lack or inability to generalize.

We assume that hallucinations are originated from three important faults in the overall architecture and conception of current AI technology: (1) tokenization that proceeds both at test and training time is wrought on the basis of too small vocabularies and it used subword units to cope with OOVWs; (2) contextual information which is the basis of LLMs knowledge both in syntax and semantics is lost every time an unknown token/wordform is met and sometimes is not put to use even when made available by the model itself; (3) higher abstract information cannot be derived simply by local or chunk-level context, it may require complex inferential operations and some reasoning. When unknown wordforms are met, the model strives to build a new token with subword units, often resulting in illegal or inexistent words (see our papers in [24,25]). As discussed at length in [23] and reported above, smaller vocabularies result in approximate context and wrong embeddings selection in zero shot model performance leading to hallucinations. Similarities between model and input test can in general be misleading, due to the nature itself of cosine measure in vector space, as has been shown in detail in our latest research [24,25]. Finally, as explained in (3) higher abstract information like the one needed to characterize the plot of a novel and the underlying linguistic techniques, require the ability to generalize and then reason on a set of underlying linguistically characterized phenomena in order to choose the correct interpretation which in our case has been mostly missing. The other important linguistic element hard to cope with for LLMs is the well-known fact that human language erases redundant information whenever it is grammatically derivable – and in some cases also in order to produce non-literal language with the aim to generate ambiguity and misunderstanding as happens in satiric contexts or in metaphors. Elided linguistic information – as for instance the subject of an untensed clause like an infinitival or a gerundive - cannot be recovered by context lookup in a model.

Eventually, even though ChatGPT has shown a remarkable ability in organizing summaries their content is deficient, but it may be used as precis by K-9 students and still get a good mark. Different judgement can be given for GEMMA 2 and Gemini which have produced a much richer output with deep and elaborate concepts at all levels of analysis we have tested it. Mistakes appeared but in a much smaller quantity.