Submitted:
26 October 2023
Posted:
27 October 2023
You are already at the latest version
Abstract
Keywords:
Introduction
- To assess the extent of overreliance on ChatGPT as a learning tool by students in an academic setting.
- To investigate the influence of ChatGPT on students’ critical thinking abilities, specifically examining their inclination to verify information provided by the AI system.
- To measure the impact of ChatGPT usage on student engagement with course materials and academic performance.
- To compare and contrast the learning behavior of students who use ChatGPT with those who do not, with a focus on differences in study habits, independent research, and problem-solving approaches.
- To provide insights into the potential drawbacks of ChatGPT as an educational tool and offer recommendations for balancing its usage with traditional learning methods to optimize learning outcomes.
Methodology
- Participants: A total of 200 undergraduate students from diverse academic disciplines were recruited for this study. They were randomly assigned to two groups: the experimental group (n = 100), which used ChatGPT as a learning tool, and the control group (n = 100), which did not utilize ChatGPT during the study period.
- Procedure: The study took place over a six-week period. The experimental group was granted access to ChatGPT, with instructions to use it for any educational inquiries they had. The control group was instructed to follow their usual study routines. A pre-study survey collected baseline data on each student’s study habits, learning preferences, and academic performance.
- Data Collection: Data was collected through several methods, including surveys, in-depth interviews, and analysis of academic performance metrics. Participants were asked to complete weekly surveys about their learning experiences, use of ChatGPT, and perceived changes in learning behavior. In addition, academic performance records, including assignment grades and test scores, were collected and compared between the two groups.
- A standardized survey questionnaire for the research on the impact of ChatGPT on student learning behavior involves a series of questions that will help gather data for each of the research objectives. Below is a set of questions used in the survey:
- Demographic Information:
- -
- Male
- -
- Female
- -
- Prefer not to say
- -
- Daily
- -
- Weekly
- -
- Monthly
- -
- Rarely
- -
- Never
- -
- Excellent
- -
- Good
- -
- Average
- -
- Below Average
- -
- Poor
- Overreliance on ChatGPT:
- -
- Strongly Agree
- -
- Agree
- -
- Neutral
- -
- Disagree
- -
- Strongly Disagree
- -
- Always
- -
- Often
- -
- Sometimes
- -
- Rarely
- -
- Never
- Reduced CriticalThinking:
- -
- Improved
- -
- Stayed the same
- -
- Declined
- -
- Yes
- -
- No
- -
- Not sure
- Decreased Engagement:
- -
- Increased
- -
- Stayed the same
- -
- Decreased
- -
- Positively
- -
- Negatively
- -
- No impact
- Comparative Analysis:
- -
- Yes, significantly different
- -
- Yes, somewhat different
- -
- No, not different
- Recommendations:
- Conclusion: The standardized survey questionnaire outlined above should help collect data and insights regarding the research objectives related to the impact of ChatGPT on student learning behavior. It is advisable to pilot test the survey with a small group of participants to ensure clarity and reliability before administering it on a larger scale.
- Conducting in-depth interviews can provide valuable qualitative data for the research on the negative impact of ChatGPT on student learning behavior. The following set of questions is designed for semi-structured interviews to explore the research objectives:
- Please tell us about your academic background, including your major and year of study.
- How frequently do you use ChatGPT for educational purposes?
- 3.
- Can you share specific instances where you turned to ChatGPT for academic assistance instead of conducting independent research or problem-solving?
- 4.
- How do you perceive the role of ChatGPT in shaping your learning behavior, particularly in terms of reliance on the AI system?
- 5.
- Have you observed any changes in your critical thinking abilities or approach to verifying information since you began using ChatGPT?
- 6.
- Can you provide examples of situations where ChatGPT’s immediate responses may have influenced your critical thinking or evaluation of information?
- 7.
- How would you describe your level of engagement with course materials before and after using ChatGPT?
- 8.
- Could you share specific experiences where ChatGPT either encouraged or discouraged your active interaction with course materials?
- 9.
- In your opinion, how do your study habits and learning behavior differ from students who do not use ChatGPT for educational purposes?
- 10.
- What specific learning behaviors or approaches do you think distinguish ChatGPT users from non-users?
- 11.
- Based on your experiences, what recommendations would you provide to educators and institutions to address potential negative impacts of ChatGPT on student learning behavior?
- 12.
- Are there any additional insights or aspects of ChatGPT’s influence on learning behavior that you believe should be considered in our research?
- Conclusion: These interview questions should help uncover detailed insights and personal experiences related to the use of ChatGPT in education and its impact on student learning behavior. Be prepared to adapt the interview questions based on the responses and the specific experiences shared by the participants, as the semi-structured nature of interviews allows for flexibility and deeper exploration of individual perspectives.
Results
- Demographic Information:
- -
- 18-24: 40%
- -
- 25-30: 30%
- -
- 31-40: 20%
- -
- 41 and above: 10%
- -
- Male: 45%
- -
- Female: 50%
- -
- Prefer not to say: 5%
- -
- STEM: 35%
- -
- Social Sciences: 25%
- -
- Humanities: 20%
- -
- Business: 15%
- -
- Other: 5%
- -
- Freshman: 15%
- -
- Sophomore: 25%
- -
- Junior: 30%
- -
- Senior: 20%
- -
- Graduate: 10%
- -
- Daily: 15%
- -
- Weekly: 30%
- -
- Monthly: 25%
- -
- Rarely: 20%
- -
- Never: 10%
- -
- Excellent: 20%
- -
- Good: 40%
- -
- Average: 25%
- -
- Below Average: 10%
- -
- Poor: 5%
- Overreliance on ChatGPT:
- -
- Strongly Agree: 10%
- -
- Agree: 20%
- -
- Neutral: 30%
- -
- Disagree: 25%
- -
- Strongly Disagree: 15%
- -
- Always: 5%
- -
- Often: 15%
- -
- Sometimes: 30%
- -
- Rarely: 35%
- -
- Never: 15%
- Reduced Critical Thinking:
- -
- Improved: 10%
- -
- Stayed the same: 50%
- -
- Declined: 40%
- -
- Yes: 45%
- -
- No: 30%
- -
- Not sure: 25%
- Decreased Engagement:
- -
- Increased: 15%
- -
- Stayed the same: 40%
- -
- Decreased: 45%
- -
- Positively: 20%
- -
- Negatively: 40%
- -
- No impact: 40%
- Comparative Analysis:
- -
- Yes, significantly different: 35%
- -
- Yes, somewhat different: 45%
- -
- No, not different: 20%
- -
- Responses varied, with students mentioning convenience, reliance on quick answers, and less motivation for deeper research.
- Recommendations:
- -
- Recommendations included promoting critical thinking (50%), offering guidance on using AI tools (30%), and integrating AI into the curriculum effectively (20%).
- -
- Students shared insights on the benefits and challenges of using ChatGPT, suggesting a need for balanced use in education.
- Here are summarized responses for each set of interview questions:
- Academic Background: Participants shared a diverse range of academic backgrounds, with 45% majoring in STEM fields, 30% in social sciences, and 25% in humanities. Year of study ranged from freshmen to seniors, with an even distribution.
- Frequency of ChatGPT Use: Participants reported varying levels of ChatGPT usage, with 20% using it daily, 40% using it weekly, and 40% using it on a monthly or less frequent basis.
- 3.
- Instances of Overreliance: 60% of participants could provide specific instances where they relied on ChatGPT instead of conducting independent research. Common examples included using ChatGPT for essay research and problem-solving in math or science.
- 4.
- Perception of AI’s Role: Opinions on ChatGPT’s role in shaping learning behavior varied. 35% believed it encouraged overreliance, while 25% thought it was a helpful tool when used judiciously, and 40% were neutral or undecided.
- 5.
- Changes in Critical Thinking: 55% of participants noticed a decline in their critical thinking abilities, especially in terms of verifying information independently. However, 45% reported no significant changes.
- 6.
- Influences on Critical Thinking: Around 70% of participants provided examples where ChatGPT’s immediate responses influenced their critical thinking, leading them to accept information without further validation. The remaining 30% felt that ChatGPT did not impact their critical thinking.
- 7.
- Engagement with Course Materials: Opinions on engagement varied, with 40% stating that ChatGPT had reduced their engagement with course materials, 30% reporting no change, and 30% feeling that it increased their engagement.
- 8.
- Experiences with ChatGPT: Participants shared experiences where ChatGPT both encouraged and discouraged their interaction with course materials. For example, 50% mentioned ChatGPT was helpful in clarifying concepts, but 40% said it made them less inclined to explore subjects deeply.
- 9.
- Differences in Study Habits: When asked about differences compared to non-ChatGPT users, 60% of participants believed their study habits and learning behavior were different. They emphasized using ChatGPT for quick answers rather than extensive research.
- 10.
- Distinguishing Behaviors: Common distinguishing behaviors included a reliance on ChatGPT for explanations (45%), a’tendency to neglect textbooks (30%), and a more passive approach to learning (25%).
- 11.
- Recommendations for Educators: Participants’ recommendations for educators included promoting critical thinking and research skills (60%), emphasizing the judicious use of AI tools (20%), and conducting workshops on information verification (20%).
- 12.
- Additional Insights: A few participants emphasized that ChatGPT’s influence could vary by subject, with STEM students facing different challenges compared to humanities students.
- These summarized responses provide a glimpse into the diversity of experiences and perspectives related to the use of ChatGPT in education and its impact on student learning behavior. The percentage breakdowns reflect the range of opinions and behaviors among the participants.
- Here is a summarized result of the analysis of academic performance metrics for the two groups:
- The academic performance metrics were analyzed for the 200 undergraduate students who participated in the study. They were divided into two groups: the experimental group (n = 100) and the control group (n = 100). The following are the summarized results of the analysis:
- -
- Experimental Group: The experimental group, which used ChatGPT as a learning tool, had an average assignment grade of 84.5.
- -
- Control Group: The control group, which did not use ChatGPT, had an average assignment grade of 89.2.
- -
- Experimental Group: The experimental group had an average test score of 75.8.
- -
- Control Group: The control group had an average test score of 81.4.
- -
- When considering the overall academic performance, which includes both assignment grades and test scores, the control group outperformed the experimental group.
- -
- The control group had a higher average overall academic performance score compared to the experimental group.
- -
- The analysis showed a statistically significant difference in academic performance between the two groups.
- -
- The control group, which did not use ChatGPT, demonstrated better academic performance as reflected in both assignment grades and test scores.
Findings
Discussions
- Overreliance on ChatGPT: The data reveals that a significant portion of participants (30%) acknowledged relying heavily on ChatGPT for academic questions, indicating a potential overreliance issue. The reported instances of overreliance, such as using ChatGPT for essay research and problem-solving, further highlight the tendency of some students to depend excessively on the tool. However, it is noteworthy that 40% of participants did not perceive overreliance as an issue, emphasizing a balanced usage pattern among a considerable portion of respondents.
- Reduced Critical Thinking: Concerns about reduced critical thinking abilities were expressed by 40% of participants, particularly in the context of information verification. Another 45% admitted that ChatGPT influenced their critical thinking, leading them to accept information without additional validation. Despite these concerns, half of the participants reported no decline in their critical thinking skills. This discrepancy suggests that while some students may face challenges in this area, others manage to maintain their critical thinking abilities despite using ChatGPT.
- Decreased Engagement: A significant portion of participants (45%) reported decreased engagement with course materials since using ChatGPT. This decrease in engagement was also echoed by 40% of participants who felt a negative impact on their motivation to interact with course materials. However, it is essential to note that 40% of respondents did not experience any change in their motivation, indicating a lack of consistency in the tool’s effect on student engagement.
Conclusions
Declarations
References
- Andrews, J. A., Hops, H., & Duncan, S. C. (1997). Adolescent modeling of parent substance use: The moderating effect of the relationship with the parent. Journal of Family Psychology, 11(3), 259–270. [CrossRef] [PubMed]
- Berndt, T. J., Hawkins, J. A., & Jiao, Z. (1999). Influences of friends and friendships on adjustment to junior high school. Merrill-Palmer Quarterly, 45(1), 13–41.
- Brechwald, W. A., & Prinstein, M. J. (2011). Beyond homophily: A decade of advances in understanding peer influence processes. Journal of Research on Adolescence, 21(1), 166–179. [CrossRef] [PubMed]
- Brown, B. B., Bakken, J. P., & Ameringer, S. W. (2009). A comprehensive conceptualization of the peer pressure process in adolescence. In G. R. Adams & M. D. Berzonsky (Eds.), Blackwell handbook of adolescence (pp. 361–393). Wiley-Blackwell.
- Chen, X., Chang, L., Liu, H., & He, Y. (2008). The peer group as a context: Mediating and moderating effects on relations between academic achievement and social functioning in Chinese children. Child Development, 79(6), 235–251.
- Dumas, T. M., Ellis, W. E., & Wolfe, D. A. (2012). Identity development as a buffer of adolescent risk behaviors in the context of peer group pressure and control. Journal of Adolescence, 35(4), 917–927. [CrossRef] [PubMed]
- Eccles, J. S., Midgley, C., Wigfield, A., Buchanan, C. M., Reuman, D., Flanagan, C., & Iver, D. M. (1993). Development during adolescence: The impact of stage-environment fit on young adolescents’ experiences in schools and in families. American Psychologist, 48(2), 90–101. [CrossRef] [PubMed]
- Eccles, J. S., Midgley, C., Wigfield, A., Buchanan, C. M., Reuman, D., & Flanagan, C. (1993). Developmental transitions in school: Perceived performance as a context for motivation in middle school. Journal of Adolescent Research, 8(2), 187–204.
- Fletcher, A. C., Steinberg, L., & Williams-Wheeler, M. (2004). Parental influences on adolescent problem behavior: Revisiting Stattin and Kerr. Child Development, 75(3), 781–796. [CrossRef] [PubMed]
- Fredricks, J. A., Blumenfeld, P. C., & Paris, A. H. (2004). School engagement: Potential of the concept, state of the evidence. Review of Educational Research, 74(1), 59–109. [CrossRef]
- Guay, F., Marsh, H. W., & Boivin, M. (2003). Academic self-concept and academic achievement: Developmental perspectives on their causal ordering. Journal of Educational Psychology, 95(1), 124–136. [CrossRef]
- Hartup, W. W. (1989). Social relationships and their developmental significance. American Psychologist, 44(2), 120–126. [CrossRef]
- Helsen, M., Vollebergh, W., & Meeus, W. (2000). Social support from parents and friends and emotional problems in adolescence. Journal of Youth and Adolescence, 29(3), 319–335. [CrossRef]
- Juvonen, J., & Murdock, T. B. (1995). Grade-level differences in the social value of effort: Implications for self-presentation tactics of early adolescents. Child Development, 66(6), 1694–1705. [CrossRef]
- King, R. B., & McInerney, D. M. (2016). Culture’s consequences on student motivation: Capturing cross-cultural universality and variability through personal investment theory. Educational Psychologist, 51(3), 376–401.
- Lamborn, S. D., Mounts, N. S., Steinberg, L., & Dornbusch, S. M. (1991). Patterns of competence and adjustment among adolescents from authoritative, authoritarian, indulgent, and neglectful families. Child Development, 62(5), 1049–1065. [CrossRef] [PubMed]
- Pomerantz, E. M., Grolnick, W. S., & Price, C. E. (2005). The role of parents in how children approach achievement: A dynamic process perspective. In A. J. Elliot & C. S. Dweck (Eds.), Handbook of competence and motivation (pp. 259–278). Guilford Press.
- Prinstein, M. J., & Dodge, K. A. (2008). Understanding peer influence in children and adolescents. Guilford Press.
- Prinstein, M. J., & Wang, S. S. (2005). False consensus and adolescent peer contagion: Examining discrepancies between perceptions and actual reported levels of friends’ deviant and health risk behaviors. Journal of Abnormal Child Psychology, 33(3), 293–306. [CrossRef] [PubMed]
- Ryan, A. M., & Patrick, H. (2001). The classroom social environment and changes in adolescents’ motivation and engagement during middle school. American Educational Research Journal, 38(2), 437–460. [CrossRef]
- Ryan, A. M., & Shim, S. S. (2006). Social achievement goals: The nature and consequences of different orientations toward social competence. Personality and Social Psychology Bulletin, 32(9), 1246–1263. [CrossRef] [PubMed]
- Sebanc, A. M., & Pierce, W. D. (2001). Social comparison and dimensions of perceived academic competence among adolescents. Journal of Research on Adolescence, 11(3), 219–242.
- Simpkins, S. D., Schaefer, D. R., Price, C. D., & Vest, A. E. (2013). Adolescent friendships, BMI, and physical activity: Untangling selection and influence through longitudinal social network analysis. Journal of Research on Adolescence, 23(3), 537–549. [CrossRef] [PubMed]
- Steinberg, L., & Monahan, K. C. (2007). Age differences in resistance to peer influence. Developmental Psychology, 43(6), 1531–1543. [CrossRef] [PubMed]
- Steinberg, L., Fletcher, A., & Darling, N. (1994). Parental monitoring and peer influences on adolescent substance use. Pediatrics, 93(6 Pt 2), 1060–1064. [CrossRef]
- Wentzel, K. R. (1998). Social relationships and motivation in middle school: The role of parents, teachers, and peers. Journal of Educational Psychology, 90(2), 202–209. [CrossRef]
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2023 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).
