Examining the Potential of Generative Language Models for Aviation Safety Analysis: Case Study and Insights using the Aviation Safety Reporting System (ASRS)

Archana Tikayat Ray; Anirudh Prabhakara Bhat; Ryan T White; Van Minh Nguyen; Olivia J Pinon Fischer; Dimitri N Mavris

doi:10.20944/preprints202307.0192.v2

Submit preprint
How it Works
Instructions for Authors
Subject Areas
Advisory Board
screening preprints
About
Awards
Log in/Register

Instructions for Authors
Awards
About
FAQ
Submit Log in/Register

Share this article with

Create alert

Email:

Load new image

preprints.org > computer science and mathematics > artificial intelligence and machine learning > doi: 10.20944/preprints202307.0192.v2

Preprint Article Version 2 Preserved in Portico This version is not peer-reviewed

Examining the Potential of Generative Language Models for Aviation Safety Analysis: Case Study and Insights using the Aviation Safety Reporting System (ASRS)

Archana Tikayat Ray

^* ,

Anirudh Prabhakara Bhat

Ryan T White

Van Minh Nguyen

Olivia J Pinon Fischer

^* ,

Dimitri N Mavris

Version 1 : Received: 4 July 2023 / Approved: 4 July 2023 / Online: 4 July 2023 (10:08:18 CEST)
Version 2 : Received: 4 July 2023 / Approved: 10 July 2023 / Online: 11 July 2023 (07:13:20 CEST)

A peer-reviewed article of this Preprint also exists.

Tikayat Ray, A.; Bhat, A.P.; White, R.T.; Nguyen, V.M.; Pinon Fischer, O.J.; Mavris, D.N. Examining the Potential of Generative Language Models for Aviation Safety Analysis: Case Study and Insights Using the Aviation Safety Reporting System (ASRS). Aerospace 2023, 10, 770. Tikayat Ray, A.; Bhat, A.P.; White, R.T.; Nguyen, V.M.; Pinon Fischer, O.J.; Mavris, D.N. Examining the Potential of Generative Language Models for Aviation Safety Analysis: Case Study and Insights Using the Aviation Safety Reporting System (ASRS). Aerospace 2023, 10, 770.

Abstract

This research investigates the potential application of generative language models, especially ChatGPT, in aviation safety analysis as a means to enhance the efficiency of safety analyses and accelerate the time it takes to process incident reports. In particular, ChatGPT was leveraged to generate incident synopses from narratives, which were subsequently compared with ground truth synopses from the Aviation Safety Reporting System (ASRS) dataset. The comparison was facilitated by using embeddings from Large Language Models (LLMs), with aeroBERT demonstrating the highest similarity due to its aerospace-specific fine-tuning. A positive correlation was observed between synopsis length and their cosine similarity. In a subsequent phase, human factor issues involved in incidents as identified by ChatGPT were compared to human factor issues identified by safety analysts. A concurrence rate of 61% was found, with ChatGPT demonstrating a cautious approach towards attributing human factor issues. Finally, the model was used to attribute incidents to relevant parties. As no dedicated ground truth column existed for this task, a manual evaluation was conducted. ChatGPT attributed the majority of incidents to the Flight Crew, ATC, Ground Personnel, and Maintenance. This study opens new avenues for leveraging AI in aviation safety analysis.

Keywords

Aviation Safety Reporting System; ASRS; Aviation Safety; Human Factors; Large Language Models; LLM; ChatGPT; Generative Language Models; GPT-3.5; aeroBERT; BERT; InstructGPT; Prompt Engineering

Subject

Computer Science and Mathematics, Artificial Intelligence and Machine Learning

Copyright: This is an open access article distributed under the Creative Commons Attribution License which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Download PDF

Comments (1)

Comment 1

Received: 11 July 2023
Commenter: Archana Tikayat Ray
Commenter's Conflict of Interests: Author

Comment: Some typos were corrected.

+ Respond to this comment

Not displayed online.

Importance: How significant is the paper to the field?

Outstanding/highlight paper

Significant contribution

Incremental contribution

No contribution

I am not qualified to judge

Soundness of evidence/arguments presented:

Conclusions well supported

Most conclusions supported (minor revision needed)

Incomplete evidence (major revision needed)

Hypothesis, unsupported conclusions, or proof-of-principle

I am not qualified to judge

Please leave your comment to this article below.

Mathematical equations can be typed in either LaTeX formats \\[ ... \\] or $$ ... $$, or MathML format <math> ... </math>. Try the LaTeX or MathML example.

Type equation:

Preview:

Copy equation into message below

Close menu

Please click a symbol to insert it into the message box below:

Please enter the link here:

Please enter a valid URL

Optionally, you can enter text that should appear as linked text:

Copy link into message below

Close menu

Please enter or paste the URL to the image here (please only use links to jpg/jpeg, png and gif images):

Please enter a valid URL

Copy image into message below

Close menu

Type author name or keywords to filter the list of references in this group (you can add a new citation under Bibliography):

No existing citations in Discussion Group

Wikify editor is a simple editor for wiki-style mark-up. It was written by MDPI for Sciforum in 2014. The rendering of the mark-up is based on Wiky.php with some tweaks. Rendering of mathematical equations is done with MathJax. Please send us a message for support or for reporting bugs.

Close menu

Please declare any conflicts of interest you may have with this paper, financial or otherwise.

I do not have a conflict of interest

I would like to declare a conflict of interest

Display my name on the website Send my name and email address to the authors

Captcha

Renew

I accept the following conditions:

Comments must follow the standards of professional discourse and should focus on the scientific content of the article. Insulting or offensive language, personal attacks and off-topic remarks will not be permitted. Comments must be written in English. Preprints reserves the right to remove comments without notice. Readers who post comments are obliged to declare any competing interests, financial or otherwise.

We encourage comments and feedback from a broad range of readers. See criteria for comments and our Diversity statement.

Leave a public comment
Send a private comment to the author(s)

* All users must log in before leaving a comment

what’s this?

Add a record of this review to Publons to track and showcase your reviewing expertise across the world’s journals.

Downloads 0

Comments 1

Metrics 0

Get PDF Cite Share 0

Bookmark BibSonomy Mendeley Reddit Delicious

Alerts

Notify me about updates to this article or when a peer-reviewed version is published.

Preprints.org is a free preprint server subsidized by MDPI in Basel, Switzerland.

MDPI Initiatives

SciProfiles
Sciforum
Encyclopedia
MDPI Books
Scilit
Proceedings
JAMS

Important links

How it Works
Advisory Board
FAQ
Friendly Journals
Instructions for Authors
About
Statistics

Choose the area that interest you and we will send you notifications of new preprints at your preferred frequency.

Disclaimer

We use cookies on our website to ensure you get the best experience.
Read more about our cookies here.

RSS feed for

Preprints: The Multidisciplinary Preprint Platform

Paste the URL in your RSS reader:

https://www.preprints.org/rss

If you want to subscribe our website in subject-wide, you could find the URL by choosing in the following:

https://www.preprints.org/rss

Please note that by using ScienceDirect RSS feeds, you agree to our Terms of Use and Privacy Policy.