How the Connectedness Between Humans and Nature Can Be Increased? Comparing ChatGPT’s Language Use to Scientific Literature

1. Introduction

ChatGPT (Chat Generative Pre-Trained Transformer) is an artificial intelligence (AI) chatbot developed by Open AI based on the large language model GPT-3 [1]. Users can receive personalized answers by entering questions or orders, so-called prompts. Already 2 months after the launch of the chatbot, more than 100 million active users have been registered [2]. One of the main reasons for this great success is probably the huge potential and possibilities of large language models like ChatGPT.

ChatGPT can already write abstracts for scientific texts, which are currently still recognized by AI recognition software, but in some cases fool human reviewers [3]. Without any particular training, ChatGPT was able to pass medical [4] and law exams [5]. ChatGPT can answer questions in a wide variety of disciplines, although care must be taken with the information given out, especially in the medical field [6]. It can be seen as a possible tool to assist, for example, in writing discharge summaries in hospitals, even if these still need to be checked by human doctors [7].

ChatGPT has the ability to write entire texts or articles on its own [8], and scientists are already using it as a tool for writing scientific texts [9] or as a tool to support them [10]. ChatGPT also handles translations at a level that can compete with commercial translation products, at least for languages with sufficient training data [11]. Especially in teaching, ChatGPT can be used as a possible tool with many capabilities and possibilities [12,13], for example, for personalized tutoring, automatic generation of grades, interactive and adaptive learning. However, it can also lead to the decline of human interaction, creativity, or understanding [14]. The majority of people have a very positive attitude towards ChatGPT and only a few people are concerned [15].

Despite all the strengths, there are currently also significant weaknesses and problems, which causes issues and controversies especially in science and research. For complex questions that require a deep understanding of the literature, ChatGPT often produces incorrect or misleading texts [16]. The mathematical abilities of ChatGPT are lower than those of a mathematics graduate student and it often cannot answer questions in this context [17]. Also, language models such as ChatGPT often show undesirable behavior, including making up facts, giving toxic or biased answers, or not following instructions [18]. Leading scholarly publishers agree that ChatGPT cannot be the author of scholarly publications [19,20] and express their concerns about the use of ChatGPT in academic writing [21]. Therefore, it is important that the discussion about the ethical challenges and applicability for scholarly content is held now [22,23].

But to what extent is ChatGPT able to answer questions that are still being studied in the scientific literature? Using an example prompt, this study aims to investigate two questions: First, the question whether ChatGPT uses the same content categories and associated word combinations in its responses that are found in the scientific literature. Second, if ChatGPT is consistent with its answers when repeatedly asked the same question.

2. Materials and Methods

"Answer the question, how can the connectedness between humans and nature be increased?" was chosen as a prompt for this study. This question was chosen because there is no simple or one-dimensional answer to it in the literature. That implies that ChatGPT cannot just give a textbook answer, but must set criteria for which factors are considered important when answering the question. There are different approaches and opinions on which factors have an influence on a person's connectedness to nature. These include, for example, the time a person spends in nature, childhood experiences, environmental education, or the quality of nature experiences [24,25,26].

This question was asked ChatGPT (January 30, 2023 version) 400 times and the answers were recorded. A new chat window was opened for each prompt so that the different answers did not relate to each other, but were answered independently. In general, ChatGPT answered with a numbered list of four to ten items (mean: 5.91 ± 0.93). In rare cases, responses were given with a bulleted list without any numbering. In this case, the bulleted list was numbered in the indicated order. In total, ChatGPT responded to the 400 prompts with a total of 2364 points.

In order to analyze the data, it was necessary to structure the answers provided by ChatGPT and to summarize their content. For this purpose, a categorization system was created in which the answers of ChatGPT can be classified. An inductive approach was adopted, taking into account the responses to the first 50 prompts. This categorization system was created by one of the authors and independently evaluated and improved by two other authors. Assignment criteria and sample responses can be found in Appendix Table A1. In this way, ten categories were obtained. Each of the 2364 responses could only be assigned to one category. To test the reliability of the categorization, three persons independently categorized the answers into the categorization system. The inter-rater reliability between the results of the three persons was calculated using fleiss' kappa [27]. For all categories (with the exception of the rest category) adequate scores could be obtained, which indicates reliability of the categorization system (Table 1).

In order to answer the question whether ChatGPT's uses the same word combinations in its answers as the scientific literature, a literature search was conducted in the scientific database WebOfScience. The aim was to check how often the word combinations used by ChatGPT in the individual categories in relation with nature connectedness were also mentioned in scientific publications. This allows a comparison of how often the categories and phrases mentioned by ChatGPT are found in literature.

Scientific literature search

Because nature connectedness is a frequently used construct that is studied in different disciplines [28] and therefore often named differently, search strings from two literature reviews on the topic were combined to reflect a broad denomination of nature connectedness [25,29].

The search string used in this study for nature connectedness was:

“connection to nature” OR “connectedness with nature” OR “connection with nature” OR “nature relatedness” OR “nature connectedness” OR “environmental identity” OR “Inclusion of nature in the self” OR “Inclusion of nature in self” OR “nature connectivity” OR “connectivity to nature” OR “connectivity with nature” OR “disposition to connect with nature” OR “disposition to connect to nature” OR “emotional affinity to nature” OR “emotional affinity with nature” OR “love and care for nature” OR “Nature connection” OR “nature affinity” OR “kinship with life” OR “biophilia” OR “ecological identity”

To examine how often the categories used by ChatGPT occur in the scientific literature, the 25 most frequently used bigrams and trigrams (word combinations consisting of two or three words) were extracted from the responses of each category. Binding words (it, can, with, or, that, their, and, such, to, as, of, its, in, the, into, on, a, for) were excluded when generation the bi- and trigrams. The proximity operator "NEAR/3" was used between the words of the individual bi- and trigrams. This operator allows a maximum of three other words between the words of the bi- and trigrams. In this way it could be ensured that there is a contextual connection between the words and that they do not occur together by coincidence. The 50 strings within a category were connected with the Boolean operator "or". Only the topic (abstract, keywords, title) were searched in WebOfScience. Since ChatGPT's training data only extends to 2021, the literature searches only included publications up to 2021. A list with the full search strings for each category is in the Appendix (Table A2).

Consistency in responses

To verify how consistent ChatGPT responded, the Normalized Mutual Information (NMI) between all 400 responses was calculated. NMI is defined as:

$$NMI\left(X,Y\right)=\frac{I(X,Y)}{\sqrt{H\left(X\right)H\left(Y\right)}}$$

where I(X, Y) is the mutual information between X (ten-dimensional vector of answer i) and Y (ten-dimensional vector of answer j), and H(X) and H(Y) are the entropies of X and Y, respectively. NMI ranges from 1 to 0, where 1 indicates a perfect match and values close to 0 represent a random distribution.

To perform this analysis, it was checked for each of the 400 answers which of the ten categories were mentioned. A ten-dimensional vector of ones and zeros was created for each answer, indicating which categories where mentioned in each response. Subsequently, using the above formula, the pairwise NMI between all 400 response vectors was calculated (a total of 79,800 pairwise comparisons). From these pairwise NMI values, an overall mean was calculated, providing an estimate of how variably ChatGPT's used the categories in its responses.

3. Results

The results show how often ChatGPT mentioned the different categories in proportion to each other. “Voluntary work”, “activity in nature” and “education” were mentioned particularly often as factors to increase nature connectedness. In addition, it was possible to determine how often ChatGPT used similar word combinations in each category as scientific literature (Table 1). In this process, similarities as well as differences were identified. For example, ChatGPT mentions the categories "time in nature", "ecotourism" and "therapy" significantly less frequently than the literature on connection to nature registered in WebOfScience. On the other hand, “education”, “greenspaces”, “individual environmental behavior” and especially “voluntary work” are mentioned considerably more often by ChatGPT than they appear in the literature. Table 1 and Figure 1 show the similarities and differences between ChatGPT and the WebOfScience search for each category.

The kappa values for inter-rater reliability range from .544 to .982 (Table 1).

4. Discussion

The results of the analysis indicate that the categories used by ChatGPT, using similar phrases and word combinations, are also found in the scientific literature. This means that ChatGPT can answer the question at least to some extent and also makes use of scientifically appropriate expressions. However, the literature and ChatGPT differ in how often each category was mentioned. For example, ChatGPT listed "time in nature" less frequently as a possible reason for increasing nature connectedness than it is considered in the scientific literature. However, time spent in nature is often considered one of the most important reasons for an increased nature connectedness [25,30,31].

Because ChatGPT was trained with a variety of sources up to 2021, which go beyond the scientific literature, it is understandable that the results between ChatGPT and WebOfScience differ. “Voluntary work” in conservation projects is mentioned very frequently in ChatGPTs responses and therefore seems to be much more common in the training data than in the scientific literature.

In addition, the NMI indicates that ChatGPT makes significant differences in the selection of categories among the 400 responses to the same question. An NMI value of 0.5 or higher would indicate more consistent responses [32,33]. In this case, the NMI value is significantly lower with a value of 0.242, which indicates that the given categories vary greatly in the responses. This result provides evidence that ChatGPT does not answer the question systematically or according to self-defined criteria, but according to statistical probabilities that derive from the training data.

Based on the results obtained here, it is once again clear that the responses of ChatGPT and other large language models must be viewed critically. Asking ChatGPT cannot replace a systematic literature search, nor can it be used to obtain answers to scientific questions. Students in particular need to be made aware of these weaknesses, as ChatGPT's quick and well-formulated answers may tempt them to overlook scientific ways of working. In addition, it must be taken into account that ChatGPT, like other language models, is based on probabilities of word sequences and therefore may output incorrect or inaccurate answers [34,35]. These study shows that ChatGPT still has weaknesses in answering questions that do not have a direct clear textbook answer, but require research and background information. Thus, these results are consistent with the assessment of other authors [16].

5. Limitations

Although this study was conducted with great care, there are some limitations that need to be considered. Especially for the large language models, development has been rapid since the release of ChatGPT. Therefore, it cannot be avoided that new versions or new models have already appeared at the time of the publication of a study [36]. This study was conducted with an earlier version of ChatGPT (as of January 30, 2023).

Moreover, this study cannot provide a concrete answer to the question of whether the interventions most frequently cited in scientific sources to increase nature connectedness are also the most important. For example, it is possible that a factor that is still underrepresented in the scientific literature has a very large impact on nature connectedness. What the study was able to determine is whether the categories mentioned by ChatGPT are also represented with similar word combinations in the scientific literature.

Also, despite high kappa values, there could have been minor misclassifications of the three observers when assigning the individual responses from ChatGPT to the categorization system.

6. Conclusions

The results show that ChatGPT can find answers to a complex question and also uses phrases that are found in the scientific literature. However, the large language model is not constant in its answers and there are major differences in the number of factors mentioned compared to the literature. Therefore, ChatGPT cannot currently be used as a tool to replace scientific research or a basic literature search. Especially students and scientists should therefore only use ChatGPT with extreme caution. Currently, the AI cannot yet do the work of a good scientific author.