Version 1
: Received: 13 August 2023 / Approved: 14 August 2023 / Online: 14 August 2023 (09:44:19 CEST)
How to cite:
Gonzalez, D.T.; Djulbegovic, M.; Kim, C.; Antonietti, M.; Gameiro, G.R.; Uversky, V.N. A Comparative Study of Large Language Models in Explaining Intrinsically Disordered Proteins. Preprints2023, 2023081014. https://doi.org/10.20944/preprints202308.1014.v1
Gonzalez, D.T.; Djulbegovic, M.; Kim, C.; Antonietti, M.; Gameiro, G.R.; Uversky, V.N. A Comparative Study of Large Language Models in Explaining Intrinsically Disordered Proteins. Preprints 2023, 2023081014. https://doi.org/10.20944/preprints202308.1014.v1
Gonzalez, D.T.; Djulbegovic, M.; Kim, C.; Antonietti, M.; Gameiro, G.R.; Uversky, V.N. A Comparative Study of Large Language Models in Explaining Intrinsically Disordered Proteins. Preprints2023, 2023081014. https://doi.org/10.20944/preprints202308.1014.v1
APA Style
Gonzalez, D.T., Djulbegovic, M., Kim, C., Antonietti, M., Gameiro, G.R., & Uversky, V.N. (2023). A Comparative Study of Large Language Models in Explaining Intrinsically Disordered Proteins. Preprints. https://doi.org/10.20944/preprints202308.1014.v1
Chicago/Turabian Style
Gonzalez, D.T., Gustavo Rosa Gameiro and Vladimir N. Uversky. 2023 "A Comparative Study of Large Language Models in Explaining Intrinsically Disordered Proteins" Preprints. https://doi.org/10.20944/preprints202308.1014.v1
Abstract
(1) Background: Artificial Intelligence (AI) models have shown potential in various educational contexts. However, their utility in explaining complex biological phenomena, such as Intrinsically Disordered Proteins (IDPs), requires further exploration. This study empirically evaluated the performance of various Large Language Models (LLMs) in the educational domain of IDPs. (2) Methods: Four LLMs, GPT-3.5, GPT-4, GPT-4 with Browsing, and Google Bard (PaLM 2), were assessed using a set of IDP-related questions. An expert evaluated their responses across five categories: accuracy, relevance, depth of understanding, clarity, and overall quality. Descriptive statistics, ANOVA, and Tukey's honesty significant difference tests were utilized for analysis. (3) Results: The GPT-4 model consistently outperformed the others across all evaluation categories. Although GPT-4 and GPT-3.5 were not statistically significantly different in performance (p>0.05), GPT-4 was preferred as the best response in 13 out of 15 instances. The AI models with browsing capabilities, GPT-4 with Browsing and Google Bard (PaLM 2) displayed lower performance metrics across the board with statistically significant differences (p<0.0001). (4) Conclusion: Our findings underscore the potential of AI models, particularly LLMs such as GPT-4, in enhancing scientific education, especially in complex domains such as IDPs. Continued innovation and collaboration among AI developers, educators, and researchers are essential to fully harness the potential of AI for enriching scientific education.
Keywords
Artificial Intelligence; Generative Pre-Trained Transformers (GPT); Intrinsically Disordered Proteins (IDPs); Large Language Models (LLMs); Pathways Language Model 2 (PaLM 2)
Subject
Computer Science and Mathematics, Artificial Intelligence and Machine Learning
Copyright:
This is an open access article distributed under the Creative Commons Attribution License which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
,
,
,
*
