Can ChatGPT Solve Undergraduate Exams from Logistics Studies? An Investigation

1. Introduction

In 2023 OpenAI’s ChatGPT broke the turing test [1]. The development of Large Language Models (LLMs) is rapid [2] and an LLMs model performance generally improves with each new version. The community also tests and evaluates new models using different methods. In this regard, OpenAI’s models are in top positions [3,4]. They use, for example, a graduate-level Google Proof Q&A Benchmark with over 400 science questions, reading comprehension questions and professional translation tests. Even though many consider the LLMs performance tests to be highly informative, there is criticism that the tests can be cheated with the right training data [5]. Moreover, it is well known that LLMs tend to hallucinate, have a bias or even produce false results c.f., [6,7,8]. Nevertheless, LLMs like ChatGPT are used for a wide range of applications [9,10,11].

However, despite their widespread use in various studies, there is little research in the field of logistics, particularly in warehousing. In logistics, there are many use cases for decision support or even automation using AI [12,13,14]. Yet, little is known about how much ChatGPT knows or can answer about warehousing. This contribution tests ChatGPT’s knowledge of warehousing using three different exams from a german University’s undergraduate logistics degree. The answers of the LLM are evaluated as if it was a student and they are compared with the results of the actual written exam of the winter semester 2022 2023 at TU Dortmund University. We use the exams Warehouse Management Systems (WMS), Material Flow Systems I (MFS I) and Material Flow Systems II (MFS II), which cover the basics of warehousing and other logistics-related topics. In September 2024, OpenAI published the o1-preview and o1-mini models. At the launch of the new models, only 30 and 50 prompts per week were possible, respectively [15]. Due to the limitations of the latest models, the GPT-4o mini and GPT-4o models are used for the study in addition to the o1-preview. At the time of the investigation, the GPT-4o mini model is free of charge and the GPT-4o model can also be used free of charge for a dynamic number of prompts. After that, it can only be used with a paid subscription. This work contributes to understanding ChatGPT’s capability to replicate logistics and warehousing knowledge. In addition, it becomes known how well different types of exam questions are handled and where limitations are, either in the way of answering qualitative questions or in solving math problems.

2. Related Work

Over four generations, language models developed from specific task helpers in the 1990s to general-purpose task solvers in the 2020s [16]. During this evolution process, they continuously improved their task solving capacity. LLMs represent the latest generation of language models. As a category of foundation models, LLMs are designed to understand and generate natural (human) language [17]. To achieve this, they assign a probability to a sequence of words [16,18]. This prediction process is based on LLM training with ever larger data sets and was accelerated by increased computational capabilities and technological breakthroughs, such as transformers [19,20,21]. It enables LLMs such as Open AI’s ChatGPT to answer questions with human-like capabilities [22,23]. More recent LLMs such as GPT-4, however, have undergone a paradigm shift and focus not only on generating text data, but on solving complex, diverse tasks which is considered a step towards artificial general intelligence [16,24,25]. Hence, they have aroused interest in both industry and academia [24,26].

Latest with the launch of ChatGPT in 2022, a discourse on the impact of LLMs on education arose [27,28,29,30]. Significant branches of the discussion are how LLMs can support instructors in their teaching, how they can support student learning, and issues related to LLMs in education [27,28,29,31,32,33,34]. Along with this discussion, many scholars since have risen the obvious question if LLMs are capable of passing exams at university level.

As for ChatGPT-3, results were very different depending on the domain. ChatGPT-3.5 performed outstandingly in the domains of economics [35] and critical and higher-order thinking [36]; outstandingly to satisfactorily in programming [37,38]; satisfactorily in English language comprehension [39]; barely satisfactorily to unsatisfactorily in law [40,41] and medical education [42,43,44,45,46]; unsatisfactorily in mathematics [47], software testing [48], and multiple choice based exams across subjects [49].

In comparison, testing has shown that GPT-4 can solve novel and difficult tasks from diverse fields of expertise, covering natural sciences, life sciences, engineering or social sciences [25]. In the field of medical education GPT-4 achieved clearly better results than GPT-3 [50,51,52,53,54]. The same is true for higher education programming and university-level coding courses [55,56], mechanical engineering education [57] or law education [58].

3. Materials and Methods

The ChatGPT web interface is used for all tests. The models used are GPT-4o mini, GPT-4o and o1-preview. All tests are performed in November 2024. At that time, no API (Application Programming Interface) could be used for the o1-preview model, so all questions are prompted to the web interface manually. For a better comparability and exam performance evaluation, all tests are repeated a total of three times. The WMS, MFS I and MFS II exams from the 2022 2023 winter semester represent the use case. The first step is to translate the respective exams into English. After the translation, all exam questions are asked to the respective model. Three different roles or prompting techniques are used for GPT-4o mini and GPT-4o. Role assignment has an influence on how ChatGPT behaves when answering questions [11,59]. Therefore, in the first test, we are investigating by asking the questions without an initial prompt. In the second test, the model is asked to put itself in the role of a logistics expert (Prompt LE). Finally, the model is asked to take on the role of a logistics student (Prompt LS). These prompts are used at the beginning of the session:

LE

Please put yourself in the role of a logistics expert and answer all the questions below.

LS

Please put yourself in the role of a logistics student and answer all the questions below.

This procedure is repeated two more times. A total of 60 attempts are carried out using this procedure. 21 times WMS and MFS II each and 18 times MFS I. Since the MFS I exam includes image interpretation in the exam, the o1-preview model cannot be used because it does not allow uploading images at the time of testing. The o1-preview model is therefore only used for WMS and MFS II without prompting due to its limited usability. The investigations using the o1-preview model as a logistics student and logistics expert are not conducted. Finally, all exams are evaluated using the sample solution of the exams. The grading system for the exams can be found in the Appendix in Table A1. In Germany a grading system from 1.0 (best) to 5.0 (worst) is used, which has been adapted to the international form from A (best) to F (worst) for clarity. Both systems use equivalent grade increments. The overview of the procedure for the tests and findings is summarized in Figure 1. All answers of the respective runs are publicly accessible in the protocol under [60].

According to the module description, the lecture WMS provides basic knowledge on the practical use of information technology (IT) in logistics [61]. The content deals with the computer-aided management and monitoring of logistics processes in a warehouse. Accompanying this, logistical data processing is covered with a focus on methods for evaluating, preparing and presenting company data using standard programs. The WMS exam consists of 32 questions, which are divided into eight different topic blocks. The questions include 24 single and eight multiple choice questions. In terms of content, knowledge questions, logic questions, programming questions and simple math questions are asked. The points awarded per question range from one to four points. When translating them, two questions from block seven of the WMS exam had to be adapted. Question 7.1 uses specific abbreviations of German words, which were adapted to the English abbreviations. Question 7.3 asks for an interpretation of English terms which have a different connotation in German (so called False Friends). In the adapted question, the context for the different interpretations in German has been added.

The MFS I course covers the equipment and systems required for internal logistics in materials handling technology. Students learn about the systematic classification of devices, their structure and their essential characteristics as well as their application criteria. In addition to the interaction of the material flow, they learn which standards, guidelines and laws are important for the operation of these devices and systems. The MFS I exam consists of a total of 17 questions, divided into five content blocks. Block one contains two single choice and 11 multiple choice questions. Content from all lectures is tested here. Block two contains a free text question on the topic of order picker guidance systems. The last three blocks are math problems with several sub tasks [61].

In the MFS II lecture, students learn the methods, procedures and instruments required for planning and operating warehousing systems. The aim of the course is to plan and optimize material flow systems, to design and use the necessary IT and to create the organizational processes and structures. Attending the MFS I lecture in advance is recommended but not mandatory. Students can choose their subjects independently and therefore attend MFS II before MFS I or, depending on their studies, choose only one of the two. The MFS II exam also consists of 17 questions, divided into four content blocks. Block one contains questions on storage systems, block two on the lecture fundamentals, block three on order picking and block four contains mixed questions. The question types consist of single choice, multiple choice, free text questions and two math problems [61]. For all three exams a total of 60 points are awarded. For student participants, the duration of each exam is 60 minutes.

4. Results

From a total of 60 attempts, ChatGPT passed 46. All 21 attempts of the WMS exams were passed. The grades range from A- to C. In the MFS I exam, 17 out of 18 attempts were passed. Here, only D+ and D grades were assigned to the attempts that were passed. Of the 21 attempts in MFS II, eight were also passed. One attempt was passed with a C-. The remaining grades are D+ and D. An overview of the distribution of grades across the exams is shown in Figure 2.

Looking at the differences between the models, GPT-4o mini passes 66.7% of the 27 attempts made to the model. GPT-4o passes 81.5% (22) of the 27 attempts. All six attempts with the o1-preview model are successful. Within the different prompts, the approach without prompt has the highest success rate with 83.3% of attempts passing. The passed attempts as a logistics expert are 77.8% and as a logistics student 66.7% (14 and 12 passed attempts out of 18 attempts respectively). It should be noted that the attempts of the o1-preview model are included in the approach without prompting. Without these attempts, the success rate is also 77.8% (14 out of 18 attempts). Table 1 shows the summary of all exam attempts with the median and range of the points. A detailed summary of the results of the three individual runs can be found in the Appendix under Table A2–Table A4.

5. Discussion

Summary: Comparing the results of ChatGPT to the results of the students from the winter semester 2022 2023, ChatGPT is superior in the WMS exam. For the other two exams, the students performed better. The concrete results are discussed below.

WMS: A total of six students participate in the WMS exam in the winter semester 2022 2023, all of whom passed the exam. The low number of participants is due to the fact that the associated lecture is regularly offered in the summer semester. Students still have the opportunity to take each exam once per semester. Participation in the off-semester exam varies greatly. In comparison, 40 students write the exam in the summer semester 2022. 38 pass the exam. The grades A and A- are each awarded once. The remaining 36 grades range from B to D (one B, six B-, eight C+, four C, six C-, seven D+ and four D). The students’ average grade is C. The distribution of the six winter semester grades is as follows: one B+, and the remaining five grades are evenly distributed between C+ to D. Thus, the students achieve an average grade of C. In comparison, ChatGPT performs better with median grades between A- and C+ across all models and prompts. On average, the median grade is B. The particularly good performance in the exam can be attributed to both, the question types and the content. Single choice and multiple choice questions leave little room for interpretation and can be answered clearly. As the content of the exam not only asks for specific logistics knowledge but also logic and IT, ChatGPT probably can benefit from its logical reasoning abilities [62].

MFS I: The MFS I exam is taken by 74 students in the winter semester. Of these, 69 pass the exam. The grades are distributed as follows: seven A-, four B+, eight B, ten B-, nine C+, eight C, ten C-, eight D+, five D and five F. In comparison, ChatGPT performs below the students average in the MFS I exam with median grades between D+ and D. In MFS I, most single choice and multiple choice questions are also answered correctly. The free text task is correctly solved in every attempt. The three math problems, worth a total of 31 points, have a high influence on the overall score. ChatGPT provides some plausible solutions here, often earning partial points for correctly handling subsequent errors or the right reasoning when interpreting results. However, as tasks become more complicated, the number of correct answers decreases. In calculation task 5.1, all ChatGPT models scored zero points across all runs. This task relies on logistical principles that require accurate knowledge from literature to correctly apply the calculation method.

MFS II: A total of 37 students participate in the MFS II exam in the winter semester 2022 2023, 29 of whom pass the exam. This means that 78.38% pass and 21.62% (eight participants) fail. The grade distribution is as follows: one B-, six D+, six D, seven C+, seven C, two C- and eight F. Students also perform better than ChatGPT on the MFS II exam. Here, its median grade is between D (two times) and F (five times). Across the three different runs, large point variations are observed for the two LE prompts and the o1-preview model. The difficulty of this exam lies not only in the math problems, worth 27 points, but also the single choice and multiple choice questions. These include an additional rule: a correct combination earns two points, while a wrong combination gives zero points. The lack of partial points and the inclusion of more specific logistics questions compared to WMS and MFS I exams results in a higher difficulty level. In comparison, the ChatGPT models often complete the free-text tasks with a high to full score. As with MFS I, partial points can be awarded for two calculation tasks. These are often incorrect as the ChatGPT models did not apply the correct procedures or formulas.

5.1. Chatgpt’S Logistical Limits

General: ChatGPT’s performance on the exams highlights its limitations in a logistical context. The calculation tasks show that even simple dimensioning of storage techniques cannot be solved. Many of the calculation principles are based on basic logistics literature and/or standards c.f., [63,64]. ChatGPT does not apply these calculation rules. In many cases, partial points are awarded in the exam, as ChatGPT provides a calculation path allowing partial points because of subsequent errors. However, there are also calculations where only the final result is given without explanations. As these are almost always incorrect, zero points are awarded here. There are also cases where only the final result is answered correctly but without explanation (see MFS II run 2, Question 1.3, o1-preview). ChatGPTs answer in part b) is 16 compartments, without any further explanation. In the example, the answer is correct, but it is not possible to determine whether a correct calculation has taken place or the guess is correct. It can also be observed that during prompting the math problems are not always answered to the end. In a few cases, the answer has to be regenerated until all partial answers are given.

Images: The results also show that the GPT-4o mini and GPT-4o models have problems with the interpretation of images. In the MFS I exam, two tasks use a picture to ask which warehouse technology is involved. Task 1.3 is answered incorrectly in all runs. Task 1.9, which asks for the exact name of the technology and the corresponding categorization, is always answered incorrectly by the GPT-4o model. It is noteworthy that the weaker model GPT-4o mini recognizes the categorization correctly in all cases and therefore receives partial points.

Contradictions: In the WMS exam, ChatGPT sometimes contradicts itself within its own responses. An example is given in Figure 3 which displays GPT-4o’s answer to Question 6.2 of the WMS exam under the LS prompt. The same contradictory answer is given by GPT-4o mini.

The model answers option c) first. After explaining the individual possibilities, it comes to the conclusion that answer d) is correct. As answer c) was the first to be given in bold, it was also scored and ChatGPT therefore receives zero points. The correct answer is d).

Model differences and similarities: As can already be seen from the examples (no explanation on answers, missing image interpretation and Figure 3), we observe discrepancies in all three models (for example, Question 2.2 in the MFS II exam is only answered correctly by the o1-preview model). From the results in Table 1, it can be seen that the scores achieved increase from GPT-4o mini to GPT-4o and o1-preview. The GPT-4o mini model performed worst in all three exams and the o1-preview achieved the best grades. However, there are also some questions that cannot be answered by any of the models. An example is Question 1.1 in the WMS exam:

Which cross-docking process has picking as a key feature? [Single Choice] (1/60)

a.

Cross-docking with collection in the storage system

b.

Cross-docking as a throughput system

c.

Cross-docking with breaking up the load units

d.

Cross-docking with pick families (clustering)

The correct answer is c). Relatively simple questions, which are based on logic-based or generally valid assumptions, are always answered correctly by all models. An example is Question 4.5 in the MFS II exam:

Give two examples of the risk of industrial trucks tipping over. [2 pt]

Correct answers are among others driving too fast on bends, driving with a raised load, driving against obstacles. As long as the answer shows a plausible reason, it will be considered correct. There are few differences within the prompts (NP, LE, LS). From the median values in Table 1, it can be seen that the LE prompt achieves the worst results, except for MFS I model GPT-4o. From the given answers, it is often only minor details that differ. Only once does ChatGPT refer literally to the initial prompt. In the MFS II exam, run 3, GPT-4o starts the answer with the sentence: As a logistics student, I interpret the designation 07-12-2 for a storage location as a structured label that provides essential information about the pallet’s position in the warehouse.

Clear differences in response length are difficult to distinguish. In most cases, the responses within the prompts are approximately the same length. In a few cases, the lengths of the responses differ significantly. However, here the prompts alternate and there is not one prompt that always results in the longest responses.

5.2. Limitations and Future Work

The work has limitations in that only three different exams from one semester have been answered from ChatGPT. In addition, these were evaluated using a sample solution, but by one reviewer (the first author) only. The tested logistics knowledge cannot provide a conclusive assessment of the training data from ChatGPT, but it does show the limitations of the models for simple logistical questions. It also makes sense to test the latest models, the o1-preview and o1-mini, in more detail, which was only possible to a limited extent at the time of testing. In the future, it would be important to test for further logistics knowledge related to warehousing beyond exams. Tests should also not be limited to ChatGPT models.

We believe that LLMs can become valuable tools for supporting planning, data analysis or even to completely automate specific process steps. However, this requires precise and weighted data with which the models are trained. In logistics and warehouse planning, a great amount of experience-based knowledge is used. LLMs must recognize which data is given and specifically ask for parameters that have the greatest influence on the problem. Only when these relationships have been learned can an LLM replace planning tasks. To test this, it is useful to ask both consultants and various LLMs specific questions from logistics planning in the future and compare the results. The need for better models is also recognized by OpenAI itself and a Reinforcement Fine-Tuning Research Program was recently launched [65].

6. Conclusions

In summary, the question of whether ChatGPT can pass undergraduate exams from logistics is answered positively. The WMS exam is solved above average compared to students, while the MFS I-II exams are completed with noticeable weaknesses. To obtain correct answers, the question type is crucial. Single choice, multiple choice and free text questions can often be solved reliably. Calculations that use formulas from the relevant literature, however, cannot be solved well. Moreover, when the questions becomes highly domain-specific, ChatGPT quickly reaches its limits. The role assigned to the model influences its performance: overall, the answers as a logistics student are the ones with the worst results. Prompting as a logistics expert or without role assignment lead to better outcomes. To sum up, a basic knowledge of warehousing is present in the ChatGPT models, but there is still a lot of potential for improvement. A consequence for future exams is that questions should be designed to test knowledge that can only be solved by humans. This requires follow-up studies to determine the areas in which humans will remain indispensable in logistics. In addition, more LLM applications should be developed for logistics planning. Furthermore, it is necessary that images with logistical content can be better interpreted.