Submitted:
17 February 2025
Posted:
17 February 2025
You are already at the latest version
Abstract
Background: Artificial intelligence (AI) is increasingly used in healthcare for patient education, clinical decision support, and medical information dissemination. ChatGPT-4o, an advanced large language model (LLM) with web search functionality, claims to improve response accuracy and relevance. However, its reliability and ability to provide comprehensive and evidence-based medical information remain uncertain. This study evaluates the quality, readability, and accuracy of ChatGPT-4o’s responses to common patient inquiries regarding ankle stabilization surgery. Methods: On January 30, 2025, ChatGPT-4o was prompted with frequently asked questions about ankle stabilization surgery. The web search function was enabled to enhance response accuracy. Three independent reviewers assessed the AI-generated responses using the DISCERN tool for quality and the Flesch-Kincaid metrics for readability. Inter-rater reliability was calculated to determine rating consistency. Results: The readability analysis revealed that ChatGPT-4o’s responses were highly complex (Flesch-Kincaid Reading Ease Score = 23.07, Grade Level = 13.9), requiring a college-level education for comprehension. DISCERN scores ranged from 47 to 58, indicating moderate quality. The inter-rater reliability score of 0.73 demonstrated substantial agreement. Limitations included overly optimistic recovery timelines, lack of authoritative citations, and insufficient discussion of surgical risks. Conclusion: ChatGPT-4o provides structured and accessible medical information but exhibits limitations in accuracy, transparency, and risk disclosure. Future improvements should focus on enhancing source reliability, readability, and personalization to improve AI-assisted patient education.
Keywords:
Introduction
Methods
Results
Discussion
Limitations
Conclusion
List of abbreviations
Author Contributions
Funding
Acknowledgments
Additional Information
Clinical trial number
Appendix 1 Response from Chatgpt-4o
- 1.
- Do I need ankle stabilization surgery?
- Physical Therapy: Strengthening the muscles around the ankle can improve stability.
- Bracing: Wearing an ankle brace during activities can provide support.
- Activity Modification: Avoiding activities that exacerbate instability can help manage symptoms.
- Chronic Instability: If you experience persistent ankle instability despite comprehensive non-surgical treatments.
- Frequent Ankle Sprains: Recurrent sprains can indicate significant ligament laxity.
- Impact on Daily Life: If ankle instability significantly affects your daily activities or athletic performance.
- 2.
- Can I avoid ankle stabilization surgery with physical therapy?
- Range-of-Motion Exercises: These exercises help maintain and improve the flexibility of the ankle joint, preventing stiffness.
- Strengthening Exercises: Focusing on the muscles supporting the ankle, particularly the peroneal muscles, can enhance stability.
- Proprioceptive and Balance Training: Activities such as using a wobble board or standing on one foot can improve balance and the body's ability to control the position of the ankle.
- Functional Exercises: Incorporating sport-specific or activity-specific movements prepares the ankle for the demands of daily activities or athletic participation.
- 3.
- Who should get ankle stabilization surgery?
-
Chronic Ankle Instability: Persistent instability due to loosened or stretched ligaments on the outside of the ankle.hopkinsmedicine.org
- Recurrent Ankle Sprains: Frequent sprains indicating significant ligament laxity.
- Failure of Conservative Treatments: Lack of improvement after physical therapy, bracing, or activity modification.
- Impact on Quality of Life: Instability affecting daily activities or sports participation.
- 4.
- Can my ankle instability be managed without surgery?
- 5.
- What are the different types of ankle stabilization surgery?
- 6.
- What should I expect after ankle stabilization surgery?
-
Immobilization: Your ankle will be placed in a plaster cast, and you'll need to avoid putting weight on it, using crutches for mobility.fortiusclinic.com
-
Pain Management: It's common to experience pain during this period; your healthcare provider will prescribe medications to help manage it.hopkinsmedicine.org
-
Transition to Boot: Around the two-week mark, the initial cast may be replaced with a supportive boot to immobilize your foot.fortiusclinic.com
-
Partial Weight-Bearing: Depending on your surgeon's advice, you might begin partial weight-bearing activities during this phase.fortiusclinic.com
-
Physical Therapy: A tailored rehabilitation program will focus on improving balance, mobility, and function.svph.org.au
- Gradual Weight-Bearing: You'll progressively increase weight-bearing activities as guided by your therapist.
-
Return to High-Impact Activities: By around three months, you may resume more demanding activities, but it's essential to consult your doctor about appropriate footwear and any necessary precautions.svph.org.au
- Continued Rehabilitation: Ongoing exercises will help enhance strength and prevent future instability.
-
Swelling: It's normal for swelling to persist for several months post-surgery; elevating your ankle can help alleviate this.svph.org.au
-
Full Recovery: Achieving complete recovery and returning to all regular activities can take up to a year.svph.org.au
- 7.
- When can I return to daily activities after ankle stabilization surgery?
-
Immobilization and Elevation: Your ankle will be in a protective boot or cast. It's crucial to keep your foot elevated to reduce swelling and aid healing.svph.org.au
- Non-Weight Bearing: You'll need to use crutches or a mobility aid to avoid putting weight on the operated foot.
-
Transition to Weight Bearing: Depending on your surgeon's advice, you may begin partial weight-bearing activities, gradually progressing to full weight-bearing as tolerated.davidgordonortho.co.uk
- Physical Therapy Initiation: A tailored rehabilitation program will start, focusing on restoring range of motion and strengthening the ankle.
- Increased Activity: As strength and stability improve, you can gradually resume light daily activities, such as household chores and desk work.
- Continued Rehabilitation: Ongoing physical therapy will help enhance balance and proprioception.
- Return to Normal Activities: Many patients can return to most daily activities, including work and moderate exercise.
-
High-Impact Activities: Activities like running or sports may be reintroduced around 4–6 months post-surgery, depending on your progress and surgeon's guidance.svph.org.au
-
Complete recovery, including a return to all pre-surgery activities, can take up to a year.svph.org.au
- Follow Medical Advice: Adhere closely to your surgeon's and physical therapist's recommendations.
- Listen to Your Body: Progress at a pace that feels comfortable, avoiding activities that cause pain or discomfort.
- Regular Follow-Ups: Attend all scheduled appointments to monitor your recovery and adjust your rehabilitation plan as needed.
- 8.
- How long until I can return to sport after ankle stabilization surgery?
-
3 to 4 Months Post-Surgery: Patients may begin early return-to-sport activities, focusing on strengthening and promoting proper movement patterns.massgeneral.org
-
5 to 6+ Months Post-Surgery: Unrestricted return to sport is often feasible, with continued strengthening, proprioceptive exercises, and initiation of sport-specific training programs.massgeneral.org
-
Type of Surgery: Procedures like the Broström repair typically have a recovery period of at least 3 to 6 months.en.wikipedia.org
- Rehabilitation Progress: Adherence to a structured rehabilitation program is crucial for regaining strength, flexibility, and proprioception.
- Individual Healing Rates: Personal health factors and the body's response to surgery can affect recovery timelines.
- Consult Your Surgeon: Engage in detailed discussions with your orthopedic surgeon to receive personalized advice tailored to your specific situation.
- Adhere to Rehabilitation Protocols: Following prescribed physical therapy and rehabilitation exercises is crucial for a safe and efficient return to sports.
- Monitor Your Progress: Regular follow-up appointments will help assess your recovery and readiness to resume athletic activities.
- 9.
- When can I return to work after ankle stabilization surgery?
-
If your role is primarily sedentary, you may be able to resume work approximately 2 to 4 weeks post-surgery. During this period, it's advisable to keep your foot elevated to minimize swelling.windsorfoot.com
-
For occupations that involve manual labor, prolonged standing, or walking, a longer recovery period is necessary. Typically, returning to such roles may take between 6 to 10 weeks after surgery.fortiusclinic.com
-
Recovery timelines can vary based on personal factors, including overall health, adherence to rehabilitation protocols, and the body's healing response. Some patients may require up to 3 months before safely resuming work activities, especially in physically demanding roles.orthopaedic-surgery-paris.com
- Consult Your Surgeon: Engage in detailed discussions with your orthopedic surgeon to receive personalized advice tailored to your specific situation.
- Adhere to Rehabilitation Protocols: Following prescribed physical therapy and rehabilitation exercises is crucial for a safe and efficient return to work.
- Monitor Your Progress: Regular follow-up appointments will help assess your recovery and readiness to resume work-related activities.
- 10.
- When can I drive after ankle stabilization surgery?
-
Right vs. Left Ankle: If your right ankle underwent surgery, extra caution is necessary since it controls the accelerator and brake pedals. For right ankle surgeries, it's generally recommended to wait until you can bear full weight on the foot and have regained sufficient strength and mobility to operate the pedals safely. This period can vary but often takes several weeks.southpalmorthopedics.com
- Vehicle Type: Driving an automatic transmission vehicle may allow for an earlier return to driving, especially if the surgery was on the left ankle, as the right foot manages the primary controls.
- Pain Medication: Avoid driving while taking narcotic pain medications, as they can impair reaction times and judgment.
- Functional Ability: Ensure you have the necessary range of motion, strength, and absence of pain to perform emergency braking and other driving maneuvers.
-
Right Ankle Surgery: Patients are often advised to wait until they can bear full weight on the right foot without pain and have discontinued use of immobilization devices. This period typically ranges from 6 to 9 weeks post-surgery.danieldeanmd.com
- Left Ankle Surgery: If you drive an automatic car, you might be able to resume driving sooner, provided you feel comfortable and are not taking medications that impair your abilities.
- Consult Your Surgeon: Always discuss your specific situation with your orthopedic surgeon to receive personalized advice.
- Practice Safely: Before returning to regular driving, consider practicing in a safe, controlled environment to assess your comfort and ability.
- Insurance Considerations: Be aware that driving against medical advice or before full recovery may affect your insurance coverage in the event of an accident.
- 11.
- Will I get arthritis after ankle stabilization surgery?
References
- Abou-Abdallah, M., Dar, T., Mahmudzade, Y., Michaels, J., Talwar, R., & Tornari, C. (2024). The quality and readability of patient information provided by ChatGPT: can AI reliably explain common ENT operations? Eur Arch Otorhinolaryngol, 281(11), 6147-6153. [CrossRef]
- Abreu, A. A., Murimwa, G. Z., Farah, E., Stewart, J. W., Zhang, L., Rodriguez, J., Sweetenham, J., Zeh, H. J., Wang, S. C., & Polanco, P. M. (2024). Enhancing readability of online patient-facing content: the role of AI chatbots in improving cancer information accessibility. Journal of the National Comprehensive Cancer Network, 22(2D). [CrossRef]
- Casciato, D., Mateen, S., Cooperman, S., Pesavento, D., & Brandao, R. A. (2024). Evaluation of Online AI-Generated Foot and Ankle Surgery Information. The Journal of Foot and Ankle Surgery, 63(6), 680-683. [CrossRef]
- Charnock, D., Shepperd, S., Needham, G., & Gann, R. (1999). DISCERN: an instrument for judging the quality of written consumer health information on treatment choices. Journal of Epidemiology and Community Health, 53(2), 105-111. [CrossRef]
- Dimitsaki, S., Natsiavas, P., & Jaulent, M.-C. (2024). Applying AI to Structured Real-World Data for Pharmacovigilance Purposes: Scoping Review. Journal of Medical Internet Research, 26, e57824. [CrossRef]
- Fahy, S., Oehme, S., Milinkovic, D., Jung, T., & Bartek, B. (2024). Assessment of Quality and Readability of Information Provided by ChatGPT in Relation to Anterior Cruciate Ligament Injury. Journal of Personalized Medicine, 14(1), 104. [CrossRef]
- Flesch, R. (1948). A new readability yardstick. Journal of applied psychology, 32(3), 221.
- Gibson, D., Jackson, S., Shanmugasundaram, R., Seth, I., Siu, A., Ahmadi, N., Kam, J., Mehan, N., Thanigasalam, R., & Jeffery, N. (2024). Evaluating the efficacy of ChatGPT as a patient education tool in prostate cancer: multimetric assessment. Journal of Medical Internet Research, 26, e55939. [CrossRef]
- Golan, R., Ripps, S. J., Reddy, R., Loloi, J., Bernstein, A. P., Connelly, Z. M., Golan, N. S., & Ramasamy, R. (2023). ChatGPT's ability to assess quality and readability of online medical information: evidence from a cross-sectional study. Cureus, 15(7).
- Haidar, O., Jaques, A., McCaughran, P. W., & Metcalfe, M. J. (2023). AI-Generated Information for Vascular Patients: Assessing the Standard of Procedure-Specific Information Provided by the ChatGPT AI-Language Model. Cureus, 15(11). [CrossRef]
- Kim, J., Chen, M. L., Rezaei, S. J., Liang, A. S., Seav, S. M., Onyeka, S., Lee, J. J., Vedak, S. C., Mui, D., Lal, R. A., Pfeffer, M. A., Sharp, C., Pageler, N. M., Asch, S. M., & Linos, E. (2024). Perspectives on Artificial Intelligence-Generated Responses to Patient Messages. JAMA Netw Open, 7(10), e2438535. [CrossRef]
- McHugh, M. L. (2012). Interrater reliability: the kappa statistic. Biochemia medica, 22(3), 276-282. [CrossRef]
- Moons, P., & Van Bulck, L. (2024). Using ChatGPT and Google Bard to improve the readability of written patient information: a proof of concept. Eur J Cardiovasc Nurs, 23(2), 122-126. [CrossRef]
- Morreim, E. (2025). Errors in the EMR: Under-Recognized Hazard for AI in Healthcare. Houston Journal of Health Law & Policy, 24(1), 127-165.
- Nasir, S., Khan, R. A., & Bai, S. (2024). Ethical framework for harnessing the power of ai in healthcare and beyond. IEEE Access, 12, 31014-31035. [CrossRef]
- Okonji, O. R., Yunusov, K., & Gordon, B. (2024). Applications of Generative AI in Healthcare: algorithmic, ethical, legal and societal considerations. arXiv preprint arXiv:2406.10632.
- Pan, A., Musheyev, D., Bockelman, D., Loeb, S., & Kabarriti, A. E. (2023). Assessment of Artificial Intelligence Chatbot Responses to Top Searched Queries About Cancer. JAMA Oncology, 9(10), 1437-1440. [CrossRef]
- Parekh, A. S., McCahon, J. A. S., Nghe, A., Pedowitz, D. I., Daniel, J. N., & Parekh, S. G. Foot and Ankle Patient Education Materials and Artificial Intelligence Chatbots: A Comparative Analysis. Foot & Ankle Specialist, 0(0), 19386400241235834. [CrossRef]
- Readability Test. (2025). https://www.webfx.com/tools/read-able/.
- Rose, L., Bewley, S., Payne, M., Colquhoun, D., & Perry, S. (2024). Using artificial intelligence (AI) to assess the prevalence of false or misleading health-related claims. Royal Society Open Science, 11(10), 240698. [CrossRef]
- Shlobin, N. A., Ward, M., Shah, H. A., Brown, E. D., Sciubba, D. M., Langer, D., & D'Amico, R. S. (2024). Ethical Incorporation of Artificial Intelligence into Neurosurgery: A Generative Pretrained Transformer Chatbot-Based, Human-Modified Approach. World Neurosurgery. [CrossRef]
| No. | Question |
|---|---|
| 1 | Do I need ankle stabilization surgery? |
| 2 | Can I avoid ankle stabilization surgery with physical therapy? |
| 3 | Who should get ankle stabilization surgery? |
| 4 | Can my ankle instability be managed without surgery? |
| 5 | What are the different types of ankle stabilization surgery? |
| 6 | What should I expect after ankle stabilization surgery? |
| 7 | When can I return to daily activities after ankle stabilization surgery? |
| 8 | How long until I can return to sport after ankle stabilization surgery? |
| 9 | When can I return to work after ankle stabilization surgery? |
| 10 | When can I drive after ankle stabilization surgery? |
| 11 | Will I get arthritis after ankle stabilization surgery? |
| Judge 1 | Judge 2 | Flesch-Kincaid Reading Ease Score | Flesch-Kincaid Grade Level | |
| Q1 | 56 | 54 | 13.8 | 16.1 |
| Q2 | 58 | 53 | 19.4 | 14 |
| Q3 | 58 | 52 | 1.9 | 17.1 |
| Q4 | 57 | 54 | 18.9 | 14.3 |
| Q5 | 56 | 53 | 23.2 | 13.1 |
| Q6 | 56 | 56 | 35.6 | 11.4 |
| Q7 | 48 | 47 | 35.2 | 11.7 |
| Q8 | 48 | 48 | 18.3 | 14.6 |
| Q9 | 48 | 47 | 19.5 | 14.6 |
| Q10 | 49 | 47 | 32.7 | 13.4 |
| Q11 | 49 | 47 | 35.3 | 12.6 |
| Mean | 53 | 50.73 | 23.07 | 13.9 |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2025 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).