Prerpints.org logo
Preprint
Review

This version is not peer-reviewed.

Generative Artificial Intelligence, AI for Scientific Writing: A Literature Review

Submitted:

31 May 2024

Posted:

03 June 2024

You are already at the latest version

Abstract
The growing usage of Generative AI tools in scientific writing requires a critical examination of their benefits and challenges. This literature review is aimed at comprehensively analyzing current empirical research articles focused on the application of Generative AI in scientific writing. The Google Scholar database was used to search for the literature. The following keywords were used: "Generative AI" and "academic writing", "LLM" (Large Language Models) and "academic writing", "Generative AI" and "Scientific writing", and "ChatGPT" and "Scientific Writing". The search was restricted to articles published between January 1, 2023, and April 30, 2024. 15 articles were selected as appropriate for the study and analyzed. It was found that, thus far, ChatGPT is the most exploited tool in the studies. AI tools such as Bard (Gemini), Bing, Claude2, and Elicit were also tested. The benefits of Generative AI usage in scientific writing were found to be omnipresent. It can aid in the generation of structured abstracts, titles, introductions, literature reviews, and conclusions of a scientific article. Generative AI also makes writing more efficient and time-saving. Its capabilities in improving language and proofreading are well-established. However, the generation of inaccurate content and references by current commercially available LLMs poses a serious problem. The lack of critical thinking and tendency to produce non-original content are significant drawbacks. Generative AI should be employed with human oversight, serving as an assistant rather than a replacement. Transparency in AI usage in scientific writing is essential, along with the necessity for proper legal regulation.
Keywords: 
Artificial Intelligence
;  
AI
;  
Generative AI
;  
Large Language Models (LLMs)
;  
Scientific Writing
;  
ChatGPT
Subject: 
Social Sciences  -   Education

Introduction

Effective scientific writing is the cornerstone of academic research, allowing researchers to clarify their research objectives and effective presentation of the results [1]. However, the main purpose of the process of scientific writing is to communicate the results to the reading audience [2]. It is a process that requires practice and focus on clarity [3]. The value of scientific writing is in recording data, supporting evidence-based ideas, and building on existing research [4]. All of this makes the process of scientific writing and publishing significantly important for society.
Artificial Intelligence (AI) tools that fall outside the scope of Generative AI are already widely recognized within the scientific writing and publishing process and extensively utilized in academia. They are especially beneficial for non-native English speakers by enhancing clarity, style, and coherence [5]. Moreover, AI is also widely used in detecting plagiarism, generations of citations, and literature management [6]. Generative AI, on the other hand, is a new concept that is transforming society. Its ability to generate sophisticated content quickly has changed traditional academic practices and goes beyond the established AI usage in the field of scientific writing [7]. Currently, AI significantly impacts scientific writing across various fields, offering benefits such as idea organization, suggesting research gaps and questions, fostering review of literature, generating research proposals and methods, as well as aiding in data management and analysis [6,8]. The introduction of ChatGPT in November 2022 has gained huge attention in the area of academic writing and its usage in academia in general. Alongside other Large Language Models (LLMs), which are integral to Generative AI, representing advanced AI tools, their usage in the field of scientific writing is steadily growing [9]. Furthermore, technological revolution and advancement usually precede legal regulations. The absence of regulations and guidance for the ethical usage of Generative AI is a reality. The need for responsible and transparent use opens many dilemmas and vigorous discussion [10,11]. Primarily, discussions and concerns have arisen regarding the potential inclusion of Generative AI (Chat GPT in particular) as an author of a paper. Some ethics organizations and leading publishers oppose such inclusion due to accountability issues. Despite this, ChatGPT was included as a co-author in manuscripts, and these manuscripts have already received citations [12].
On the other hand, along with the growing level of generative AI usage for research and writing, the exploration of potential and capabilities in current commercial Generative AI tools is also expanding. Scientific papers discussing the nuances of existing versions of LLMs and other Generative AI tools for scientific writing and publishing have already been published. However, a comprehensive review of literature summarizing the advantages and challenges of generative AI in scientific writing is lacking.
This article aims to address this gap by not only summarizing research findings but also providing insights beneficial to stakeholders and crucial for shaping future exploration in AI and scientific writing. This literature review will delve into the current state of Generative AI tools used for scientific writing, offering a thorough analysis of studies in this domain and practical applications. It will conclude by evaluating reported functionalities and benefits, while also acknowledging the dilemmas and concerns associated with generative AI usage in scientific writing.

Background

The traditional scientific writing process, which is widely accepted, involves some key steps such as selecting a topic, conducting a review of literature, and writing the article in a structured format [13].
Manuscripts are usually prepared in the IMRAD format: Introduction, Methods, Results, and Discussion [14]. The key components of a research article are underscored, including the title, abstract, introduction, methods, results, discussion, and conclusion [15,16].
Some research focuses on evaluating the soundness of these components. It encompasses a theoretical base, problem statements, variable definition, population description, research design, test instrument, reliability and validity, result consistency, and recommendations for further research [17].
Finally, each manuscript, prior to publication, undergoes formatting and referencing to ensure adherence to specific style guidelines established by the scientific journals [18].
The use of artificial intelligence (AI) is represented in every aspect of modern society. AI usage in scientific writing is a topic that attracts huge attention and opens debate. The potential of AI to facilitate diversity and increase efficiency in scientific communication is highlighted [19,20]. However, its usage raises concerns regarding the impact of AI on research integrity and the role of human researchers in the process.
AI-powered writing tools, such as Grammarly, Zotero, Mendeley, and others, have revolutionized how scientists approach the writing process, and the benefits of their usage are widely accepted and confirmed. These AI writing tools provide valuable assistance in various aspects of scientific writing, including grammar and syntax correction, citation management, and literature review organization [21]. By using AI algorithms, these tools can analyze and understand the context of scientific writing, thereby offering real-time suggestions for improvements. However generative AI is new and its usage and capabilities go beyond these AI tools.
Generative AI, particularly Large Language Models (LLMs), is a relatively new topic that sparks interest and provides new opportunities and challenges simultaneously. After the release of ChatGPT, the usage of LLMs for generating content in scientific papers significantly increased, especially in the field of Computer Science [9].
Research studies have underlined the benefits of LLMs, including accelerating literature review and enhancing the writing process. They also highlight limitations, such as biases and ethical concerns [10,22]. While testing the ability of LLMs in scientific summarization tasks, it was evident that LLMs outperform humans in certain tasks. Nonetheless, they have limitations in generating long summaries and abstractive lay summaries [23].
As generative AI and Large Language Models (LLMs) continue to advance, their usage for scientific writing will increase. The growing ethical dilemmas and concerns will also foster the necessity for regulations and guidelines in this area. Several publishers have already established policies addressing the issue of generative AI usage in manuscripts [24]. Concurrently, the integration of AI usage in writing scientific papers will advance, as well as the utilization of AI in reviewing processes. Editors and reviewers will encounter and increasingly benefit from various opportunities for assessing submitted articles. However, they must remain vigilant about potential threats, such as biases and errors made by LLMs [25].
The issue of the inaccuracy of the information provided and analyzed by Artificial Intelligence (AI) has been present since the beginning of its usage in scientific writing. This applies to commercially available AI tools up to date [26]. Therefore, authors and reviewers should double-check the relevance of the information presented by AI.
The studies in this area raise discussion regarding the ethical issues and challenges emanating from the progress of generative AI and its steady usage in academia. Over the recent period, there has been a significant increase in studies analyzing the pros and cons of AI usage in scientific writing. There is a clear need for a review of the existing results to establish a solid foundation for determining the strengths and weaknesses of current AI tools and their usage for scientific writing. The main aim of this study is to fill this gap and summarize the findings of LLMs usage in scientific writing, distinguishing between the benefits on one hand and the errors and challenges on the other.

AI and its Usage in Medicine and Health Care

Large Language Models (LLMs) are considered the next global breakthrough redefining the healthcare and research industry, thus facilitating various procedures [27]. For example, AI tools in radiation oncology operate significantly faster than human professionals, while also exhibiting a high level of expertise [28]
The analytical capabilities of ChatGPT, Bard (Gemini), and Bing in the detection of emergency cases were tested. Although refinement is required, the three chatbots are innovative and useful tools for emergency care. Their integration could enhance the process of emergency management and improve the output [29].
The usage of ChatGPT in ophthalmology has great potential in clinical decisions, research, and education. However, the risks of inaccuracy and data privacy are omnipresent [30].
ChatGPT, Claude AI, Google Bard, and Perplexity AI were tested for clarity, accuracy, relevance, and completeness for medical suggestions and their usage in making complex medical decisions [31]. It was found that Claude AI championed the completeness and relevance of the responses. While ChatGPT offered consistency, the responses of Google Bard (Gemini) varied and were not consistent.
The usage of Large Language Models (LLMs) was also tested in preventing medication direction errors in online pharmacies. It was found that LLMs can improve accuracy and efficiency in pharmacy operations [32].
On the other hand, the AI-generated information (ChatGPT) is reported to be of poor quality in vascular surgical procedures. Decisions for the patients should be based on human-generated information and by trusted professionals [33]. Language was found to be another limitation of LLMs, particularly ChatGPT. The bot’s training solely on English datasets leads to diminished quality and reliability of responses when applied to other languages. In addition, it shows a lower level of understanding in specific domains, such as Traditional Chinese Medicine [34].
The presence and benefits of AI usage in the field of medicine and health care are evident. However, at this stage of its development, it should be used as an assistant and not a substitution for professionals [35].
In academic medicine, authors should also be aware of the impact their work can have on health care. When using ChatGPT or other language models, it must be done with integrity and uphold the highest standards. The usage of LLMs in academic medicine must be transparent and used for refining and editing the texts only [36].
Among other areas, the advancement and usage of LLMs in the medical field grow alongside various ethical and regulatory challenges. Developers and regulatory bodies will need to collaborate to ensure data protection without compromising innovation. The ultimate goal is to provide responsible integration of LLMs in medicine [37].

Methods

This literature review aims to explore the current state of Generative Artificial Intelligence (AI) and its impact and application on scientific writing, summarizing the results of the existing empirical research. The focus will be on the Large Language Models (LLMs) as a specific type of Generative AI. The first step included a comprehensive search of literature in the Google Scholar database. The search was restricted to articles published between January 1, 2023, and April 30, 2024.
Four sets of keywords were utilized: "Generative AI" AND "academic writing", "LLM" (Large Language Model) AND "academic writing", "Generative AI" AND "Scientific writing", and "ChatGPT" AND "Scientific Writing".
In the second step, the inclusion and exclusion criteria were defined. Studies directly focused on AI and scientific writing was included, while those unrelated to scientific writing and lacking empirical research were excluded. The third step involved extracting information regarding the key focus, main findings, the benefits of AI usage in scientific writing, and the limitations of existing AI tools from the selected studies. The final step included data analysis of the existing capabilities of AI tools in scientific writing, their limitations and challenges, outcomes, and potential areas for future research and development

Results and Analysis

Figure 1 illustrates the selection and inclusion process. Initially, 85 papers were selected through a search of the Google Scholar database. Following the removal of duplicate studies (n=11), 74 papers remained. Subsequently, 40 papers were excluded based on the predetermined criteria for relevance to the topic. The abstracts of the remaining 34 papers were screened. Seven studies were excluded due to lack of open access, two studies lacked peer review, and three studies lacked an academic structure. Ultimately, 15 studies, which provided empirical research on the application of generative AI in scientific writing, were included in this literature review.
To categorize the reviewed articles based on the special focus of each study, as well as their results and conclusions regarding the usage of generative AI in scientific writing, a coding scheme was established. This scheme focuses on both the benefits and limitations of using Generative AI for scientific writing tasks.
Examining the selected studies, ChatGPT emerges as the most commonly tested Large Language Model (LLM) for scientific writing and is frequently featured in research studies. It has undergone rigorous testing and practical analysis within the research domain, providing its efficacy in assisting manuscript preparation. Among the other Generative AI tools examined are Gemini (Google Bard), Claude2, Bing, and Elicit. Currently, ChatGPT-4 represents a pinnacle in LLM development, also serving as the latest commercially available version from OpenAI. Notably, it has surpassed other chatbots in quantitative accuracy [38].
ChatGPT can assist researchers in many aspects, thereby making the writing process faster and time-saving [39,40]. It is capable of creating well-structured abstracts of scientific papers [41]. Furthermore, it is useful for preparing titles, introductions, research questions, and conclusions, along with the first draft of an article[42] [40]. The use of ChatGPT in practical experiments for paper writing shows that the bot has a high potential for becoming a great help to researchers in study design, performing analyses, and drafting the study results into a scientific article [43]. The use of LLMs decreases the time spent to write a review article [39].
"Elicit" serves as a valuable tool for crafting the literature review section of a scientific article. It facilitates the organization of scientific literature, extraction of data, and provision of answers to questions based on research findings [44].
Generative AI avoids human mistakes, making the process of scientific writing more reliable, accurate, and efficient [45,46].
Qualitatively, tested AI tools such as ChatGPT 3.5 and 4, Claude 2, Bing, and Bard (Gemini) exhibited proficiency in merging existing knowledge, but none produced original scientific content [38].
Examination of the references generated by ChatGPT recognized the "AI hallucination problem", where non-existing references are presented, potentially leading to serious legal and ethical issues [47]. In some cases, up to 70% of the references cited were found to be inaccurate. Errors in references were especially prevalent in the DOI numbers [48]. Moreover, ChatGPT fails to identify all relevant literature on the topic and neglects recent literature [39,49].
Generative AI may produce plagiarized content [50], and its level of bias must be quantified. In some cases, poor accuracy of the generated content is an issue as well. Finally, an imbalance in accessibility between high and low-income countries, especially if the software becomes fully paid, is also a reasonable expectation and potential challenge. Therefore, proper regulation of its usage in the field of scientific writing will become a necessity sooner or later [43].
The inability to engage in critical thinking is stated as one of the main limitations of LLMs [51]. Therefore, ensuring the accuracy of results through human supervision and oversight, as well as acknowledging its usage by authors of scientific articles, becomes imperative.

Discussion

Generative AI is a major topic in today’s discourse. Its importance lies in raising public awareness about its usage, implications, and potential outcomes, which could facilitate the enactment of appropriate and timely regulation [52].
Although the number of analyzed studies in this literature review is not extensive, it summarizes the empirical findings of open-access research articles in English, with a special focus on generative AI and scientific writing. Furthermore, it provides additional value by discussing the latest studies, including the most recent versions of commercially available LLM tools and their abilities to assist in scientific writing. Various chatbots and AI tools were also reviewed.
One limitation is the exclusion of valuable studies on topics that are not open-access. In addition, the strict focus on the scientific writing process has resulted in the omission of studies exploring AI and academic writing in broader contexts, as well as AI usage in other academic domains.
As the body of research continues to develop, future studies could address the following areas:
  • Impact of AI on the Quality and Credibility of Scientific Writing: Exploring how AI influences the quality and trustworthiness of scientific publications.
  • Development of Various AI tools Specifically Designed for Scientific Writing: Some Generative AI tools are specifically prepared for scientific writing. Exploring their reliability and benefits could be a valuable thing for the stakeholders.
  • Ethical Guidelines for AI Use in Scientific Writing: Some guidelines for responsible usage of Generative AI in the field of science already exist. National legislations also make efforts to regulate the area. Analysis of the existing status and its impact on the process is also a valuable contribution.

Conclusion

This literature review examined the findings of existing empirical research on artificial intelligence (AI) tools and their application in the field of scientific writing. The studies have shown that ChatGPT is the most investigated tool, and generative AI offers significant advantages for scientific writing. It serves as a helpful tool for preparing important parts of a scientific article, including structured abstracts, titles, introduction, and literature reviews. This technology enhances organization, draft generation, and content accuracy, thereby streamlining the process to make it more efficient and faster. Undoubtedly, generative AI can assist in language editing and proofreading as well.
However, alongside the benefits, the responsible use and disclosure of generative AI by authors are necessary. Errors in references, including non-existing ones, are among the most common errors of commercially available Generative AI tools. Their inability to detect relevant and recent sources has also been confirmed as a barrier. Another challenge is the lack of critical thinking and the inability to create original content.
Despite concerns, the use of generative AI in scientific writing is rapidly growing, with the potential to facilitate the scientific writing process.

Funding Statement

The author did not obtain any funding or support for this study.

Acknowledgments

The content initially included a minor portion that was written with the assistance of Gemini AI. However, the AI-generated content has been meticulously edited and verified by the author.

Conflict of Interest

The author does not report any conflict of interest.

References

  1. Lindsay DM: Scientific Writing = Thinking in Words writing. Europhysics Letters, 143. 2020. [CrossRef]
  2. Gopen GD, Swan JA: The science of scientific writing. American scientist, 78(6), 550-558. 1990, Available at https://www.usenix.org/sites/default/files/gopen_and_swan_science_of_scientific_writing.pdf.
  3. Heard SB: The Scientist's Guide to Writing: How to Write More Easily and Effectively throughout Your Scientific Career, Princeton: Princeton University Press. 2016. [CrossRef]
  4. Okwemba RK: Introduction To Scientific Writing A Review. International Journal of Scientific Research in Science and Technology. 2022. [CrossRef]
  5. Giglio AD, Costa MU: The use of artificial intelligence to improve the scientific writing of non-native English speakers. Revista da Associação Médica Brasileira, 69. 2023. [CrossRef]
  6. Khalifa M, Albadawy M: Using artificial intelligence in academic writing and research: An essential productivity tool. Computer Methods and Programs in Biomedicine Update, 100145. 2024. [CrossRef]
  7. Kaliyadan F, Seetharam KA: ChatGPT-Quo Vadis?. Indian Dermatology Online Journal, 14(4), 457-458. 2023. [CrossRef]
  8. Abd-Elsalam KA, Abdel-Momen SM: Artificial Intelligence's Development and Challenges in Scientific Writing. Egyptian Journal of Agricultural Research. 2023. [CrossRef]
  9. Liang W, Zhang Y, Wu Z, et al.: Mapping the Increasing Use of LLMs in Scientific Papers. 2024, https://arxiv.org/pdf/2404.01268.
  10. Muga G: Editorial —Artificial Intelligence language models in scientific. 2023, https://ui.adsabs.harvard.edu/link_gateway/2023EL....14320000G/. [CrossRef]
  11. Vitente AC, Lazaro RT, Escuadra CJ, et al.: The Use of Artificial Intelligence (AI)-Assisted Technologies in Scientific Discourse. Philippine Journal of Physical Therapy. 2023. [CrossRef]
  12. Nazarovets S, Teixeira da Silva JA: ChatGPT as an “author”: Bibliometric analysis to assess the validity of authorship. Accountability in Research, 1-11. 2024. [CrossRef]
  13. Grimm LJ, Harvey JA: Practical steps to writing a scientific manuscript. Journal of Breast Imaging, 4(6), 640-648. 2022. [CrossRef]
  14. Somashekhar SP: Art of Scientific Writing. Indian Journal of Gynecologic Oncology, 18, 1-3. 2020. [CrossRef]
  15. Lunsford TR, Lunslord BR: How to Critically Read a Journal Research Article. JPO Journal of Prosthetics and Orthotics, 8, 24-31. 1996, https://cdn.ymaws.com/www.oandp.org/resource/resmgr/docs/skc/journal-club/How_to_Critically_Read.pdf.
  16. Fischer BA, Zigmond MJ: Components of a Research Article. 2009, Retrieved from www.survival.pitt.edu.
  17. Nielsen E, Reilly PL: A Guide to Understanding and Evaluating Research Articles. Gifted Child Quarterly, 29, 90 – 92. 1985. [CrossRef]
  18. Watson R: Avoiding Desk Rejection of a Manuscript. Nurse Author & Editor. 2019. [CrossRef]
  19. Carobene A, Padoan A, Cabitza F, et al.: Rising adoption of artificial intelligence in scientific publishing: evaluating the role, risks, and ethical implications in paper drafting and review process. Clinical Chemistry and Laboratory Medicine (CCLM). 2023. [CrossRef]
  20. Khan NA, Osmonaliev K, Sarwar MZ: Pushing the Boundaries of Scientific Research with the use of Artificial Intelligence tools: Navigating Risks and Unleashing Possibilities. Nepal Journal of Epidemiology, 13, 1258 – 1263. 2023. [CrossRef]
  21. Razack HIA, Mathew ST, Saad FFA, et al.: Artificial intelligence-assisted tools for redefining the communication landscape of the scholarly world. Science Editing, 8(2), 134-144. 2021. [CrossRef]
  22. Boyko J, Cohen J, Fox N, et al.: An Interdisciplinary Outlook on Large Language Models for Scientific Research. 2023. [CrossRef]
  23. Fonseca M, Cohen SB: Can Large Language Model Summarizers Adapt to Diverse Scientific Communication Goals? 2024. [CrossRef]
  24. Salimi A, Saheb H: Large language models in ophthalmology scientific writing: ethical considerations blurred lines or not at all? American Journal of Ophthalmology. 2023. [CrossRef] [PubMed]
  25. Gilat R, Cole B J: How will artificial intelligence affect scientific writing, reviewing and editing? The future is here…. Arthroscopy, 39(5), 1119-1120. 2023. [CrossRef]
  26. Dashti M, Londono J, Ghasemi S, et al.: How much can we rely on artificial intelligence chatbots such as the ChatGPT software program to assist with scientific writing?. The Journal of prosthetic dentistry. 2023. [CrossRef]
  27. Vallath AL, Sivasubramanian BP, Chatterjee A, et al.: Ventricular septal rupture and artificial intelligence (AI)-assisted healthcare. Cureus, 15(3). 2023. [CrossRef]
  28. Shanbhag NM, Sumaida AB, Binz T, et al.: Integrating Artificial Intelligence Into Radiation Oncology: Can Humans Spot AI?. Cureus, 15(12). 2023. [CrossRef]
  29. Salazar GZ, Zúñiga D, Vindel CL, et al.: Efficacy of AI Chats to determine an emergency: a comparison between OpenAI’s ChatGPT, Google Bard, and Microsoft Bing AI Chat. Cureus, 15(9). 2023. [CrossRef]
  30. Dossantos J, An J, Javan R: Eyes on AI: ChatGPT's transformative potential impact on ophthalmology. Cureus, 15(6). 2023. [CrossRef]
  31. Uppalapati VK, Nag DS: A Comparative Analysis of AI Models in Complex Medical Decision-Making Scenarios: Evaluating ChatGPT, Claude AI, Bard, and Perplexity. Cureus, 16(1). 2024. [CrossRef]
  32. Pais C, Liu J, Voigt R, et al.: Large language models for preventing medication direction errors in online pharmacies. Nat Med (2024). [CrossRef]
  33. Haidar O, Jaques A, McCaughran PW, et al.: AI-Generated Information for Vascular Patients: Assessing the Standard of Procedure-Specific Information Provided by the ChatGPT AI-Language Model. Cureus, 15(11). 2023. [CrossRef]
  34. Tan Y, Zhang Z, Li M, et al.: MedChatZH: A tuning LLM for traditional Chinese medicine consultations. Computers in Biology and Medicine, 172, 108290. 2024. [CrossRef]
  35. Altamimi I, Altamimi A, Alhumimidi AS, et al.: Snakebite advice and counseling from artificial intelligence: an acute venomous snakebite consultation with ChatGPT. Cureus. 2023, 15:e40351. [CrossRef]
  36. Kim JK, Chua M, Rickard M, et al.: ChatGPT and large language model (LLM) chatbots: The current state of acceptability and a proposal for guidelines on utilization in academic medicine. Journal of Pediatric Urology. 2023. [CrossRef]
  37. Ong JCL, Chang SYH, William W, et al.: Ethical and regulatory challenges of large language models in medicine. The Lancet Digital Health. 2024. [CrossRef]
  38. Lozic E, Stular B: ChatGPT v Bard v Bing v Claude 2 v Aria v human-expert. How good are AI chatbots at scientific writing?. 2023. [CrossRef]
  39. Kacena, M. A., Plotkin, L. I., & Fehrenbacher, J. C. (2024). The use of artificial intelligence in writing scientific review articles. Current Osteoporosis Reports, 1-7. [CrossRef]
  40. Aydin, O., & Karaarslan, E. (2022). OpenAI ChatGPT generated literature review: Digital twin in healthcare. Aydın, Ö., Karaarslan, E.(2022). OpenAI ChatGPT Generated Literature Review: Digital Twin in Healthcare. In Ö. Aydın (Ed.), Emerging Computer Technologies, 2. https://acikerisim.mu.edu.tr/xmlui/bitstream/handle/20.500.12809/10483/Omer.pdf?sequence=1.
  41. Salvagno M, Taccone FS, Gerli AG, et al.: Can artificial intelligence help for scientific writing?. Critical care, 27(1), 75. 2023. [CrossRef]
  42. Hwang T, Aggarwal N, Khan PZ, et al.: Can ChatGPT assist authors with abstract writing in medical journals? Evaluating the quality of scientific abstracts generated by ChatGPT and original abstracts. PLoS ONE 19(2): e0297701. 2024. [CrossRef]
  43. Donlon, E., & Tiernan, P. (2023). Chatbots and Citations: An experiment in academic writing with Generative AI. Irish Journal of Technology Enhanced Learning, 7(2), 75-87. [CrossRef]
  44. Altmäe S, Sola-Leyva A, Salumets A: Artificial intelligence in scientific writing: a friend or a foe?. Reproductive BioMedicine Online, 47(1), 3-9. 2023. [CrossRef]
  45. Kung JY: Elicit. The Journal of the Canadian Health Libraries Association, 44(1), 15. 2023. [CrossRef]
  46. Huang J, Tan M: The role of ChatGPT in scientific communication: writing better scientific review articles. American journal of cancer research, 13(4), 1148. 2023, http://www.ncbi.nlm.nih.gov/pmc/articles/pmc10164801/.
  47. Burger B, Kanbach DK, Kraus S, et al.: On the use of AI-based tools like ChatGPT to support management research. European Journal of Innovation Management, 26(7), 233-241. 2023. [CrossRef]
  48. Athaluri SA, Manthena SV, Kesapragada VKM, et al.: Exploring the boundaries of reality: investigating the phenomenon of artificial intelligence hallucination in scientific writing through ChatGPT references. Cureus, 15(4). 2023. [CrossRef]
  49. Mugaanyi J, Cai L, Cheng S, et al.: Evaluation of Large Language Model Performance and Reliability for Citations and References in Scholarly Writing: Cross-Disciplinary Study. Journal of Medical Internet Research, 26, e52935. 2024. [CrossRef]
  50. Jenko N, Ariyaratne S, Jeys L, et al.: An evaluation of AI generated literature reviews in musculoskeletal radiology. The Surgeon. 2024. [CrossRef]
  51. Hsu, H. P. (2023). Can Generative Artificial Intelligence Write an Academic Journal Article? Opportunities, Challenges, and Implications. Irish Journal of Technology Enhanced Learning, 7(2), 158-171. [CrossRef]
  52. Smith P, Smith L: This season’s artificial intelligence (AI): is today’s AI really that different from the AI of the past? Some reflections and thoughts. AI and Ethics, 1-4. 2023. [CrossRef]
Preprints 108022 g001
Figure 1. PRISMA Flowchart.
Figure 1. PRISMA Flowchart.
Preprints 108022 g001
Table 1. Analysis of the 15 selected studies.
Table 1. Analysis of the 15 selected studies.
Authors and year Title Main Focus Key findings AI Benefits AI limitations
1. Altmae et al. 2023 Artificial intelligence in scientific writing: A friend or a foe Chat GPT for scientific writing ChatGPT has shown a high potential in scientific writing Material organization, draft creation, and proofreading Inaccuracy and non-existing references
2. Athaluri et al. 2023 Exploring the boundaries of reality: Investigating the phenomenon of artificial intelligence hallucination in scientific writing through ChatGPT references Chat GPT-3 hallucination Partly Inaccurate references In general, AI-generated content can be trusted but must be monitored and revised Some non-existing references
3. Kacena et al. 2024 The use of artificial intelligence in writing scientific review articles ChatGPT-4 writing draft manuscript It can reduce the time for writing but the content must be revised by humans AI usage in scientific writing is time-saving Inaccurate references up to 70%, likelihood of plagiarism, and does not include the newest literature
4. Lozic & Stular 2023 ChatGPT v Bard v Bing v Claude 2 v Aria v human-expert. How good are AI chatbots at scientific writing? Comparing different AI chatbots for scholarly writing All chatbots are proficient in merging existing knowledge; however, understanding their limitations is essential ChatGPT-4 outperforms the other chatbots in quantitative accuracy. AI bots in their current versions are not capable of creating original content.
5. Salvagno et al. 2023 Can artificial intelligence help with scientific writing? ChatGPT assistance in scientific writing ChatGPT has a potential for scientific writing accompanied by ethical concerns and challenges ChatGPT can help with the research questions, literature review, formatting, and language review. It can also save time in clinical practice To be used only as an assistant and not replacement for experts. Potential plagiarism and inaccuracies. Paid versions can cause disbalance
6. Aydin & Karaarslan 2022. #break# OpenAI ChatGPT Generated Literature Review: Digital Twin in Healthcare ChatGPT and its assistance in academic writing AI will facilitate the scientific writing process / High level of similarity in the AI-generated content.
7. Babal & Babal 2023. Generative artificial intelligence: Can ChatGPT write a#break#quality abstract? ChatGPT preparation of a conference abstract ChatGPT can become a valuable writing tool.#break# The AI-generated abstract was well-written and without errors One fictitious citation was spotted.
8. Donlon & Tiernan, 2023. Chatbots and Citations: An experiment in academic writing with Generative AI ChatGPT-3.5 (free version) writing an academic paper The bot generated a credible base and was particularly useful in some aspects Useful for generation of title, introduction, and conclusion, including the first draft of a paper. /
9. Hsu 2023 Can Generative Artificial Intelligence Write an Academic Journal Article? Opportunities, Challenges, and Implications ChatGPT-4 tested for preparation of a short academic paper. Although AI is a game-changer for academia the need for balance is evident A valuable tool for idea generation, design, and English writing. Lack of critical thinking is something very important for accuracy. Ethical considerations.
10. Hwang et al. 2024 Can ChatGPT assist authors with abstract writing in medical journals? Evaluating the quality of scientific abstracts generated by ChatGPT and original abstracts ChatGPT 3.5 and 4 versions in preparation of paper abstract following journal templates ChatGPT exhibited proficiency in the content with minimal errors in the abstract ChatGPT generates concise and readable abstracts of scientific articles.#break#Authentic and well-structured abstracts. Minimal errors were spotted.#break#The general quality of the author-generated abstracts was higher.
11. Mugaanyi et al. 2024 Evaluation of Large Language Model Performance and Reliability#break#for Citations and References in Scholarly Writing:#break#Cross-Disciplinary Study ChatGPT 3.5 usage for citations and references in scholarly publishing ChatGPT-generated references and citations vary across various academic disciplines In general, ChatGPT 3.5 generated relevant references and citations. Hallucinations in the identifications of the DOI numbers of the generated references.
12. Kung, J. 2023 Elicit "Elicit" AI as a research assistant "Elicit" is a unique AI tool for summarizing academic literature A useful tool for reviewing literature and answering questions based on scholarly articles. It retrieves articles from one database only
13. Jenko et al. 2024 An evaluation of AI-generated literature reviews in musculoskeletal radiology Assessment of AI-generated literature reviews Although current AI tools are not reliable for routine use, they are capable of producing impressive literature reviews AI saves time. ChatGPT-4 generates good literature reviews. Cannot identify all relevant publications. Requires human oversight.
14. Burger et al. 2023 On the use of AI-based tools like ChatGPT to support management research AI use to support research AI will be integrated into the research to improve some domains Improving the objectivity and the accuracy of the results. Avoid human errors and improve the reproducibility of research. Making the research process faster, more reliable, and more convenient AI does not provide reasoning and causality.
15. Huang &Tan, 2023#break# The role of ChatGPT in scientific communication: Writing better scientific review articles ChatGPT and its application in drafting scientific articles ChatGPT is a powerful tool that facilitates the process of scientific writing, making it more efficient and effective Enhance quality and efficiency in writing literature reviews. It makes the process faster, creates outlines, adds details, and improves the writing style. The generated text must be edited to avoid plagiarism.
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.
Copyright: This open access article is published under a Creative Commons CC BY 4.0 license, which permit the free download, distribution, and reuse, provided that the author and preprint are cited in any reuse.
Prerpints.org logo

Preprints.org is a free preprint server supported by MDPI in Basel, Switzerland.

facebook logotwitter logolinkedin logo
bsky
MDPI Initiatives
Important Links
Subscribe

Disclaimer

Terms of Use

Privacy Policy

Privacy Settings

© 2025 MDPI (Basel, Switzerland) unless otherwise stated