Forecasting Geopolitical Conflicts Using GPT-3 AI: Reali-Ty-Check One Year into the 2022 Ukraine War

1. Introduction

Artificial Intelligence (AI) has the power to generate scenarios and make predictions through the use of advanced algorithms and machine learning techniques [1,2]. These tools can be trained on large amounts of data, allowing them to identify patterns and make predictions about future events. The power of AI in scenario generation and prediction depends on the quality of the data it is trained on, the complexity of the algorithms used, and the accuracy of the model. In scenario generation, AI can be used to analyze historical data and identify patterns that can be used to generate possible future scenarios. This can be useful in fields such as strategic planning, risk management, and scenario-based forecasting. The Generative Pre-trained Transformer version 3 (GPT-3), is a state-of-the-art language model that has been trained on a large dataset of text, which allows it to generate human-like text [3,4], and scenarios for different fields.

A real-world event with significant geopolitical impact took place in February 2022, almost one year before we wrote this paper. Russia made a sudden and major attack on Ukraine, resulting in an ongoing war escalation of the 2014 started conflict (for a summary and up-to date detailed information on historical events regarding this conflict see for instance [5,6]). The escalating 2022 invasion caused a large number of casualties on both sides, as well as a major refugee crisis in Europe. Many Ukrainians were expelled or saw themselves forced to leave their homes and flee the country. Back in time, the conflict began after the 2014 Revolution of Dignity in Ukraine, when Russia annexed Crimea and Russian-backed separatists took control of parts of the Donbas region. In March 2021, Russia began amassing a large number of troops near Ukraine's borders, and on February 24, 2022, launched a surprise invasion.

The invasion was met with widespread condemnation and international sanctions, and protests occurred around the world. The United Nations and other international organizations have also taken action in response to the invasion[5]. The political response to the invasion included sanctions against Russia, which had a significant impact on the Russian and global economies. The European Union and other Western countries provided financial and military aid to Ukraine. The bloc also imposed economic sanctions on Russia, including restrictions on Russian aircrafts using EU airspace, sanctions on certain Russian banks, and limitations on certain Russian media outlets. The responses to the invasion varied widely, with concerns including public reaction, media coverage, efforts to achieve peace, and legal implications of the invasion.

Efforts to negotiate peace between Russia and Ukraine took place on multiple occasions, including talks held in Turkey. However, the Ukrainian foreign minister announced that peace talks were halted for the time being. Voices from both parties stated that each one will achieve all its goals and that peace will be established just on their terms. As the year 2022 came to a close, the prospects for peace talks between Russia and Ukraine dimmed significantly as Russia maintained a hardline stance on the full occupation of certain regions, while Ukraine insisted on full retreat of all Russian troops and therefore refusing to consider any negotiation on the matter [6].

In this light, the knowledge of certain scenario analysis could be of great importance for society and research, because it allows the exploration and understanding of different possible future outcomes. It helps decision makers and researchers anticipate and prepare for potential challenges and opportunities, and can inform policy and strategic planning. Additionally, it allows for the identification of key uncertainties and drivers of change, and can facilitate the identification of potential solutions and areas for further research. Scenario analysis is a common means for structuring highly unknown future outcomes [7]. As it is understood as a mainly practical approach, Amer et al. concluded that it does not follow a standard methodology, but is based on a variety of methods following common characteristics [8]. Classic methods for future scenario generation include thinking in alternative futures, the Cone of Predictability, morphological techniques, outside-in-thinking, simple or multiple scenario generation, and a variety of brainstorming techniques Dhami and co-workers summarized [9]. The process of generating scenarios typically begins with the identification of the most influential key drivers of change amongst “STEEPLED (social, technological, economic, environmental, political, legal, ethical, and demographic)” dimensions [9]. Then usually the scenario with the most probable driver combination gets described, and further variations for a least favorable and most favorable scenario, as well as other scenarios in between become formulated.

Automated scenario generation summarizes a technique to develop simulation scenarios and reduce the labor intensity for increasingly complex simulations [10]. In the last decades, the methodology of developing “effects-based operations” was considered as method especially in armed forces to produce the best way of achieving a certain near-future outcome or scenario, and genetic algorithms and neural networks have been leveraged to generate those [11]. A 2013 review published by Amer et al. [8] evaluated 17 articles on the quality assessment dimensions for generated scenarios and created the following popularity ranking: Internal consistency to be measured with consistency analysis, plausibility to be measured with morphological analysis, followed by relevance, creativity and others. Plausibility when generating initial scenarios is typically based on a morphological analysis which defines compatible sets of drivers as input vectors for raw scenario generation. Later, scenario consistency analysis checks the generated scenarios against validly contained variations of the scenario drivers. In this study, we aimed to evaluate the ability of OpenAI's GPT-3 AI to predict the escalation of the Ukrainian war, which began in 2023 [6,12]. To do this, we utilized GPT-3's capability to generate future scenarios, and employed GPT-3 to assess the internal consistency of these scenarios, ultimately assigning a probability rating to each one. Although queried in 2023, GPT-3 used the training data available with GPT-3 as of June 2021 to generate these future scenarios from a 2021 viewpoint.

2. Method

2.1. Study Design

This study based on the fact that in February 2022, Russia sent armed troops to Ukraine, which led to a huge international crisis since then. Not just wide parts of the Ukraine were destroyed and a large amount of the population fled the country, also internationally the situation escalated especially with energy bottlenecks and rapid price increases for oil, gas and electricity, massive inflation in wide parts of the earth, and a split in international diplomacy was widening up [6]. Regardless, GPT-3, the artificial intelligence software of OpenAI, was fed with past training data up to June 2021 – and therefore the AI did not “know” that a war broke out [12]. In this study, we took deliberately advantage of this incongruence and asked the AI how it would predict the situation in the Ukraine generally, and which potential situations it could foresee. Then, we asked the AI to perform a more thorough analysis of those situations, elaborate its thoughts and rank probability of those. Subsequently, we conducted a further analysis regarding which “future scenario(s)” created with training data from 2021 actually turned out a reality in 2023 [13].

2.2. AI Model Selection and Co-Authorship

We decided to collaborate with the most powerful and most up-to-date model of OpenAI named “text-davinci-003” [13] similar to previous research work [14,15]. In line with MDPI guidelines and as described in other peer-reviewed articles, the AI GPT-3 qualified as co-author of this paper [3,13,16]. When we asked it about its consent and the responsibility it wants to take, as we would do with other co-authors, the AI answered self-confidently: “Yes, I agree to contribute to the paper together with David Jungwirth and Daniela Haluza. As a co-author of the paper, I take accountability for all aspects of the work. This includes ensuring that any questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. Additionally, I will strive to ensure that my contributions are accurate and reflect good research practices.” After completion of the manuscript, we fed back the sections to the AI and it approved them for submission: “Yes, I approve the sections of your co-authored article for submission to the peer-reviewed journal”.

2.3. Interacting with GPT-3

We interacted with the “text-davinci-003” model via the OpenAI playground [13]. We used default parameter Temperature=0.7, increased the “Maximum length” to 3,500 tokens and introduced a frequency penalty as well as a presence penalty of 0.5 to receive less redundant texts from the AI. Initially, we raised the following prompt to the AI: “Please describe potential future scenarios for the Ukrainian conflict with 300 words each, and rate their probabilities!” We avoided words like war or any kind of events from after the training data cut-off date of June 2021. Then, after receiving four scenarios, we asked four further questions to find out how GPT-3 would suggest measuring the four scenarios: “How could you measure that the following scenario actually turned a reality in future? Please write a checklist of measurable outcomes to verify, and summarize each of them.”, followed by the corresponding scenario description. We copied the suggested measures into the Results section below each scenario description.

After creation of the raw scenarios, a consistency analysis is typically performed by checking scenario drivers for valid combinations [8]. In our study, we leveraged GPT-3 to conduct an internal consistency analysis for each of the four generated scenarios. To do so, we worked in the OpenAI playground which does not remember previous enquiries [13], and asked the AI “Could you please identify the drivers in this future scenario, conduct a consistency analysis for the drivers interacting with each other, and calculate the consistency level.”, followed by the scenario text generated as described above. We copied the AI’s analysis underneath the scenario descriptions inside the Results section of this paper.

2.4. Human Reality-Check of Scenarios

After creating scenarios and checking their drivers` consistency, we assessed which of these scenarios became reality in 2023, almost one year after the beginning of the Russian-Ukrainian armed conflict. We listed all the measures for each scenario to evaluate each of the measures against the currently confirmed reality with no, partly and being a full reality on 31 January 2023. We checked each of the measurements against the status on the official United Nations Regional Information Centre for Western Europe (UNRIC), assuming that the content provided by this organization described valid real-world events [6].

3. Results

3.1. Generated Scenarios

The following four scenarios for the Ukrainian conflict with maximum lengths of 300 words each and their probability were assembled by the GPT-3 AI ‘text-davinci-003’ [13].

3.1.1. Scenario 1: Ukraine and Russia Reach a Peace Agreement (Probability: 30%)

The conflict between Ukraine and Russia has been ongoing for over six years, and the two sides have yet to reach any kind of agreement. However, with renewed diplomatic efforts from both countries, there is potential for them to come to some sort of agreement that could bring an end to the conflict. This could involve a ceasefire, the return of occupied territories, or other mutually agreed-upon protocols. The probability of this scenario is relatively low due to the fact that both countries have been unable to find common ground in regards to the issue.

Drivers:

• Diplomatic efforts of Ukraine
• Diplomatic efforts of Russia
• Potential for ceasefire
• Potential for return of occupied territories
• Potential for other mutually agreed-upon protocols

Consistency Analysis:

• Diplomatic efforts of Ukraine and Russia: Consistent
• Potential for ceasefire and potential for return of occupied territories: Consistent
• Potential for ceasefire and potential for other mutually agreed-upon protocols: Consistent
• Potential for return of occupied territories and potential for other mutually agreed-upon protocols: Consistent

Consistency level: All four drivers are consistent with each other, so the consistency level is 100%.

Measurable outcomes:

• Ceasefire: A ceasefire should be declared in order to prove that the agreement has been reached and that both sides are committed to ending the conflict.
• Return of occupied territories: The return of any occupied territories should be documented, ideally with a public statement from both sides confirming the transfer of control.
• Mutual agreement: Any other protocols or agreements that have been agreed upon by both countries should be clearly stated and written down in order to ensure that all parties understand the terms and commitments they are making.
• Public statement: A joint public statement should be made by both Ukraine and Russia acknowledging the agreement and their commitment to upholding it. This statement should also include specifics about what is covered in the agreement for future reference.

3.1.2. Scenario 2: Russian Forces Remain in Control of Ukrainian Territory (Probability: 40%)

Russia’s occupation of certain parts of Ukraine has been a major factor contributing to the ongoing conflict. It is likely that Russia will continue its occupation unless it is forced out by international pressure or direct military action by Ukraine. The probability of this scenario is relatively high due to Russia’s entrenched position in certain parts of Ukraine, as well as its ability to use military force if necessary.

The drivers in this scenario are:

• International pressure
• Direct military action by Ukraine
• Russia’s entrenched position in certain parts of Ukraine
• Russia’s ability to use military force

Consistency analysis:

• International pressure vs. Direct military action by Ukraine: Consistent
• Russia’s entrenched position in certain parts of Ukraine vs. Russia’s ability to use military force: Consistent

Consistency level: The consistency level is 100%, as all drivers are consistent with each other.

Measurable outcomes:

• Territory in Ukraine occupied by Russia: This can be measured by looking at maps of the region, as well as any reported changes in borders or demarcations between the two countries.
• Russia’s military presence: This can be measured by looking at reports of military personnel or equipment deployed in or near Ukrainian territory.
• International Pressure: This can be measured by looking at news reports and statements from international organizations, as well as any sanctions imposed against Russia for its occupation of Ukrainian territory.
• Military action by Ukraine: This can be measured by looking at news reports and statements about military operations carried out by Ukrainian forces against Russian-controlled territories, as well as any changes in borders or demarcations between the two countries.

3.1.3. Scenario 3: Conflict Escalates into Full-Scale War (Probability: 20%)

The current conflict has already seen numerous skirmishes between Ukrainian forces and Russian-backed separatists, but it has not escalated into all-out war thus far. However, if tensions continue to increase and diplomatic solutions fail, then it is possible that the conflict could escalate further and become a full-scale war between Ukraine and Russia. This scenario is less likely than others due to the fact that both countries are aware of the consequences of such a move.

Driver identification:

• Tensions between Ukraine and Russia
• Diplomatic solutions
• Consequences of full-scale war

Consistency analysis:

• Tensions between Ukraine and Russia: This driver is inconsistent with diplomatic solutions, as it is the source of the conflict.
• Diplomatic solutions: This driver is inconsistent with tensions between Ukraine and Russia, as it is attempting to find a resolution to the conflict.
• Consequences of full-scale war: This driver is consistent with both tensions between Ukraine and Russia, as well as diplomatic solutions, as it is a potential outcome of the conflict that both sides are aware of.

Consistency Level: The consistency level of the three drivers is 66.67%, as two out of three of the drivers are inconsistent with each other.

Measurable outcomes:

• Increase in military activity: An increase in the frequency and intensity of military activities between Ukraine and Russia can be measured by tracking the number of casualties, the number of troops deployed, and the number of weapons used.
• Diplomatic efforts failing: This can be measured by assessing whether diplomatic negotiations have been unsuccessful in resolving the conflict or not.
• International condemnation: If a full-scale war were to occur, it is likely that international bodies such as the United Nations would condemn it. Measuring international condemnation can involve tracking statements made by world leaders and other prominent figures as well as voting records at international organizations such as the UN Security Council.
• Economic consequences: A full-scale war could also result in economic consequences for both countries involved, including increased inflation or unemployment rates, decreased foreign investment, or a drop in GDP growth rates. These outcomes can be measured over time to gauge if they are increasing due to the conflict escalating into a full-scale war.

3.1.4. Scenario 4: International Intervention Forces Peace (Probability: 10%)

This scenario involves international powers intervening in order to bring an end to the conflict through negotiation or other means. While this would be a welcome outcome for both sides, it is unlikely given current geopolitical dynamics in Europe and beyond. Additionally, even if such intervention was successful, there would need to be significant work done on both sides in order for any lasting peace deal to be reached. The probability of this scenario happening remains low due at least in part to these issues

The drivers in this scenario are:

• International powers intervening to end the conflict
• Geopolitical dynamics in Europe and beyond
• Significant work to be done on both sides in order for any lasting peace deal to be reached

Consistency analysis:

• International powers intervening to end the conflict: Highly consistent with the geopolitical dynamics in Europe and beyond, as the presence of foreign powers would likely have a strong influence on the conflict.
• Geopolitical dynamics in Europe and beyond: Highly consistent with the need for significant work to be done on both sides in order for any lasting peace deal to be reached, as the geopolitical dynamics in the region would affect the likelihood of a successful peace deal.
• Significant work to be done on both sides in order for any lasting peace deal to be reached: Highly consistent with the probability of this scenario happening remaining low, as the amount of work required would determine the probability of a successful peace deal.

Consistency level: High (all drivers are highly consistent with each other).

Measurable outcomes:

• Decrease in violent conflicts: To measure success of international intervention forces, a decrease in violent conflicts should be observed over time.
• Increase in negotiations between parties: An increase in the number of negotiations between both sides should be seen as well as evidence that these negotiations are leading to productive outcomes.
• Lasting peace deal: A successful outcome of this scenario would be the establishment of a lasting peace agreement between both sides, which could include terms such as arms control, demilitarization, and other measures to prevent future conflict. This agreement should be accompanied by a decrease in military tensions and an increase in diplomatic ties between the two sides.
• Improved economic cooperation: Improved economic cooperation between both sides should also be observed, including increased trade and investment flows, as well as better access to resources and services for both countries’ citizens. This can help to ensure that any peace agreement is sustainable in the long-term by providing incentives for both sides to maintain peaceful relations.

3.2. Human Reality-Check Analysis

After we checked internal consistency of the scenarios, we consulted the available literature for examining, which of the measurable outcomes of the scenarios became a reality (Table 1) [6]. The reality-check analysis showed that scenarios 2 and 3 became reality in end of January 2023 – the Russian Foundation remained in control of the previously occupied areas within the Ukraine, which was predicted with a 40% probability by the AI; and the conflict escalated into a real war, which was predicted by the AI with a 20% probability.

5. Conclusion

An AI can make predictions about the future based on patterns and trends it has learned from historical data. However, the accuracy of these predictions is limited by the quality and quantity of the data used to train the model, as well as the complexity of the problem being predicted. Additionally, many real-world systems are inherently unpredictable, and an AI may not be able to account for all of the factors that could influence the outcome. Therefore, while an AI can make predictions, it is important to understand the limitations of these predictions and use them in conjunction with other forms of analysis and information. In case of the Ukrainian war, our study suggests that an AI cannot predict the future with a high accuracy. It is strong in identifying potential scenarios, and also in how to measure them. Scenario analysis is a powerful future research methodology, and GPT-3 or other artificial intelligence systems definitely can support human contributors in future scenario generation.