AI-Driven Financial Analysis: Exploring ChatGPT's Capabilities and Challenges

1. Introduction

In 1950, Alan Turing published his seminal paper, “Computing Machinery and Intelligence” (Turing, 2012), posing a profound question, “Can machines think?” More than seventy years later, on November 30, 2022, the global community witnessed the launch of ChatGPT (Chat Generative Pre-training Transformer), a revolutionary Artificial Intelligence (AI) language model developed by OpenAI. This innovation, a product of advancements in Large Language Models (LLMs), exhibits capabilities suggestive of thought, comprehension, voice, video, and creativity. Within a remarkably short span, ChatGPT has already precipitated significant changes across various sectors.

Throughout history, transformative technologies like manufacturing automation and the rise of e-commerce have ushered in new epochs. ChatGPT's rapid adoption reflects this historical pattern. For instance, in Europe, the travel company Expedia has harnessed AI chatbots to help users plan cost-effective, eco-friendly trips (Blesiada, 2023). According to Enterprise Apps Today, the technology and education sectors are among the foremost adopters of OpenAI's solutions, with industries such as business services, manufacturing, and finance also integrating AI into their operations (Elad, 2024). A 2023 Goldman Sachs report highlighted AI's potential to displace up to 300 million full-time jobs (Kelly, 2023), sparking debates among financial analysts about the future relevance of their roles in an increasingly automated economy.

ChatGPT, fundamentally a language model, is trained on extensive datasets using advanced Natural Language Processing (NLP) techniques. This training enables the model to perform a wide array of tasks. Its capability is further enhanced by Reinforcement Learning from Human Feedback, which allows ChatGPT to better understand and respond to user expectations. This sets it apart from traditional search engines by providing specific, concise answers instead of a list of relevant links. The addition of features like the Code Interpreter (now known as Advanced Data Analysis) equips ChatGPT Plus users with tools for complex tasks such as financial modeling, forecasting, and risk analysis.

However, the accuracy and quality of ChatGPT's responses can vary based on the question posed, the training data available, the complexity of the topic, and the given instructions or prompts. The evolution of such AI capabilities necessitates a reconsideration of Turing's original question, inviting us to reflect on the nature of intelligence and the intersection of human cognition and artificial computation. In light of ongoing AI-induced transformations, this study aims to uncover the diverse impacts of AI, particularly ChatGPT, on financial analysis and strategy, exploring how the financial sector can adapt to these technological shifts.

The contributions of this study are manifold, enhancing the understanding of academicians, developers, and stakeholders in the integration of ChatGPT into financial practices. This includes providing detailed insights into ChatGPT's financial applications, contrasting its capabilities with those of human analysts, and discussing potential limitations in financial contexts. The paper is organized as follows: Section 2 reviews AI-related financial studies; Section 3 outlines the empirical design and test; Section 4 presents and analyzes findings, and Section 5 concludes the discussion.

2. Artificial Intelligence Techniques and Related Studies in Finance

2.1. AI Applications and Studies

Technological advancements have played a pivotal role in the evolution of the financial services industry. Historically, innovations such as telegrams and Morse code in the late 19th century significantly altered monetary transactions (Saunders, Cornett & Erhemjamts, 2021). The rise of financial technology (FinTech) has further transformed this landscape by influencing various facets such as information management, trading practices, and financial management. AI stands out as a significant and expanding field of interest among scholars and practitioners. Its applications extend across traditional areas like financial markets, trading, banking, investments, optimization, and insurance. Additionally, AI is increasingly pivotal in burgeoning FinTech sectors, including big data analytics, blockchain, and data mining. These applications are crucial for risk management and regulatory compliance (Ahmed, Alshater, El Ammari, & Hammami, 2022; Cao, 2022; Farooq & Chawla, 2021; Lin, 2019).

The momentum of FinTech surged notably in the 20th and 21st centuries, coinciding with the digital transition of banks to online platforms. The 1970s marked a critical turning point with the advent of algorithmic trading in financial institutions, which leveraged computer models to automate trading strategies. This technological evolution enabled the development of advanced trading models that could analyze extensive datasets, identify patterns, and make informed trading decisions. By the 2000s, these algorithmic trading strategies saw a dramatic increase in adoption, driven by technological advancements, an influx of historical and real-time market data, and the growing complexities within financial markets. Studies by Burgess (2021) using frameworks from Oxford University revealed that these AI-driven trading strategies often surpass human traders under various market conditions, including during the COVID-19 pandemic. The integration of machine learning and AI techniques has further refined algorithmic trading, significantly influencing market dynamics, liquidity, and trading strategies, thereby enhancing overall market efficiency (Chaboud, Chiquoine, Hjalmarsson, and Vega, 2014).

Moreover, advanced algorithms and machine learning models have demonstrated their efficacy in analyzing extensive datasets to identify potential risk (Demajo, Vella, & Dingli, 2020; Yu, Härdle, Borke, & Benschop, 2023) and detect patterns of fraudulent activities (Jullum, Løland, Huseby, Ånonsen, & Lorentzen, 2020). Since the 1990s, AI methodologies such as artificial neural networks, support vector machines, ensemble methods, generalized boosting, AdaBoost, and Random Forests have been employed to predict financial distress and failures in banks (Liu, Liu, & Sathye, 2021). Additionally, the application of Explainable AI (XAI) in credit models within the banking sector, such as credit scoring and credit default prediction, has been explored, contributing to the adoption of XAI techniques in the finance industry (Demajo, Vella, and Dingli, 2020; de Lange, Melsom, Vennerød, and Westgaard, 2022.

The use of AI to model behavioral biases has also gained prominence. The integration of Natural Language Processing (NLP) has become increasingly vital in finance studies since the early 21st century, covering areas such as text classification, sentiment analysis, and natural language generation. Research by Tetlock, Saar-Tsechansky, and Macskassy (2008) and Bollen, Mao, and Zeng (2011) has shown the predictive power of sentiment analysis in determining stock market trends, establishing a significant correlation between news sentiment and market behavior. Similarly, Félix, Kräussl, and Stork (2020) have employed machine learning-based models to construct implied volatility sentiment, further highlighting the utility of AI in financial analytics.

2.2. Studies of ChatGPT on Finance

Since its inception in November 2022, ChatGPT has sparked considerable academic interest in its application to finance. Researchers have explored its utility in a variety of financial tasks, including financial document classification, sentiment analysis, named entity recognition in financial texts, and financial data extraction (Zaremba & Demir, 2023). Traditional keyword-based methods in financial sentiment analysis have shown weaknesses, particularly in handling complex texts, as these methods are susceptible to adversarial manipulation (Boukes et al., 2020; Hartmann et al., 2023; Leippold, 2023a).

Further studies have assessed ChatGPT’s ability to interface with explainable artificial intelligence (XAI) models and to demystify complex financial concepts for lay audiences (Wenzlaff & Spaeth, 2022; Yue et al., 2023). Despite its advantages, Leippold (2023b) noted that large language models (LLMs) like GPT-3 may generate unfounded content, as demonstrated in tests involving GPT-3's responses on climate change topics. Additionally, Lopez-Lira and Tang (2023) identified a significant correlation between ChatGPT’s interpretations of corporate news and subsequent stock market reactions, suggesting its accuracy in financial analysis.

In the context of finance research, Dowling and Lucey (2023) highlighted ChatGPT’s contributions across various stages of research, particularly in the study of cryptocurrencies. Hansen and Kazinnik (2023) demonstrated ChatGPT’s effectiveness in analyzing central bank communications, underscoring its value in comparative studies and zero-shot learning capabilities.

Ali and Aysan (2023) investigated ChatGPT’s broader financial industry applications, including content generation, account management, customer support, investment advice, and regulatory compliance. In risk management, Zaremba and Demir (2023) emphasized the ethical and regulatory challenges associated with deploying ChatGPT in financial settings, calling for measures to ensure its responsible use. These inquiries collectively reflect the growing recognition of ChatGPT’s potential to significantly impact financial practices and research.

3. Empirical Design

3.1. AI Chatbots in the Finance Domain

The market for AI in finance is experiencing significant growth, driven by key players who are facilitating this transformation. Services such as KAI, AlphaChat, Growthbotics, and FinChat have been developed to meet the specific requirements of the financial sector. FinChat, in particular, leverages generative AI to provide investment research, offering fundamental investors relevant data through an interactive conversational interface.

A notable advancement in this field is OpenAI’s ChatGPT advanced data analysis. This tool significantly enhances the functionality of AI-driven conversational agents by enabling them to write and execute code. This capability is crucial for performing complex financial analyses, including data visualization, chart creation, file manipulation, and complex mathematical computations. Moreover, ChatGPT can analyze results, engage in logical reasoning, and produce downloadable outputs. These features make it an invaluable resource in financial analysis, where precision and efficiency are paramount.

On May 14, 2024, OpenAI introduced a groundbreaking new voice and video model, further extending the functionalities of AI in financial applications. This development promises to enrich the interactivity and accessibility of financial services, providing a more intuitive and engaging user experience and expanding the reach of AI-enhanced financial solutions.

3.2. Rationale of AI in Financial Analysis

Artificial intelligence (AI) is a branch of computer science dedicated to developing systems and machines capable of performing tasks that typically require human intelligence, such as learning, reasoning, problem-solving, perception, language understanding, and decision-making (Sokolov, 2019).

A central debate in AI research is whether AI competes with human capabilities or enhances human efficiency. A key distinction between traditional tools and AI is the principle of autonomy. Advanced artificial intelligence and sophisticated biological intelligence can exhibit autonomy, learning from data and experiences to make decisions and take actions without direct human intervention (Korteling et al., 2021). In contrast, conventional tools lack autonomy; they operate based on predetermined rules or instructions and require user manipulation to fulfill their intended purposes.

According to a study by Wei et al. (2022), enabling a chain of thought or intermediate reasoning steps enhances the complex reasoning capabilities of LLMs. In financial analysis, reasoning involves using available financial data, information, and relevant factors to make judgments, conclusions, and inferences about companies, businesses, projects, investments, or financial markets. This process demands critical thinking, analysis, and problem-solving skills.

Financial professionals, including analysts, traders, and investors, typically employ reasoning to analyze financial statements, assess performance metrics, forecast future outcomes, and devise strategies for investment, budgeting, and financial planning. These professionals must possess a solid foundation in algebra and mathematics to excel in their roles. They utilize a range of robust tools to support their research, analysis, and investment management efforts. These tools include charting software, technical analysis applications, options and derivatives analyzers, portfolio management solutions, and algorithmic trading platforms. Excel is fundamental for tasks such as ratio analysis, risk management, investment analysis, and asset valuation. When handling vast datasets, a deep understanding of mathematical and statistical techniques is crucial to draw accurate conclusions from financial data.

Furthermore, these professionals are responsible for developing advanced financial models and conducting extensive research. Proficiency in specialized financial software and programming languages like C++, R, SAS, and Python is vital for effectively navigating the financial landscape.

Son et al. (2023) investigated the application of LLMs in financial reasoning, confirming the models' ability to generate coherent investment opinions. Although the study does not provide a detailed description of the reasoning process in financial analysis with LLMs, it underscores the importance of task formulation, synthetic data generation, prompting methods, and evaluation capability in influencing the quality of responses generated by LLMs.

3.3. Financial Reasoning in a Nutshell

Unlike basic mathematical calculations, most financial calculations involve multi-step processes. A prime example is the concept of present value, a fundamental principle in finance widely used to determine the value of shares, bonds, projects, or entire businesses. Calculating present value requires several steps: identifying future cash flows, selecting the appropriate discount rate, determining the number of periods for each cash flow, and computing the present value for each cash flow. When ChatGPT undertakes this task, it establishes a numerical reasoning path to integrate all the information and deduce meaningful answers. However, questions arise regarding the capability of LLMs like ChatGPT to perform more complex tasks within these basic present value calculations, such as determining the appropriate discount rate when it is not provided.

Advanced financial analysis demands accurate reading comprehension, logical interpretation, and the application of financial principles. For instance, evaluating a company's operational status involves interpreting extensive financial statements to extract meaningful insights, identifying data patterns and relationships, and subsequently analyzing and formulating strategies. When making investment decisions, it is essential to consider the cross-temporal and cross-domain characteristics of financial investments, conduct both fundamental and technical analyses, and choose the best investment strategy amidst various uncertainties.

Given these considerations, our empirical examination is structured into three distinct domains within financial analysis: corporate finance, investment, and derivatives. To gain deeper insights into the performance of ChatGPT-4o in managing these tasks, we have categorized them based on the complexity of the reasoning process into Multi-Step Reasoning Tasks and Complex Reasoning Tasks. Additionally, we will assess the effectiveness of traditional tools used by human analysts, such as mathematical equations, Excel, Refinitiv, Stata, and other resources, as benchmarks to evaluate the achievement of our objectives.

3.4. Tasks/Prompt

3.4.1. Multi-Step Reasoning Tasks

The Multi-Step Reasoning task consists of 32 questions covering various topics in corporate finance, investments, and derivatives. These topics include but are not limited to, present value, future value, annuities, payment schedules, investment accumulation, and rate calculations. The task also explores basic futures and options pricing models, value calculation, risk management, and forecasting. Additionally, it includes qualitative assessments leading to decision-making or recommendations regarding future dividend payouts and capital structure. A detailed overview of these tasks can be found in Appendix 1.

These tasks primarily require straightforward calculations or judgments, involving a series of logical or computational steps to reach a specific conclusion. They are usually solvable through explicit logic and analysis without subjective judgments. It is expected that ChatGPT-4o will provide accurate computational formulas and resultant values when addressing such tasks. The aim is to evaluate ChatGPT’s ability to apply logical and analytical reasoning in finance and investment, focusing on precision and objectivity in computations and assessments.

3.4.2. Complex Reasoning Tasks

Complex reasoning tasks require advanced calculations, extensive analysis, and creative thought processes, demanding a higher level of critical thinking compared to Multi-Step reasoning tasks.

To assess these analyses, we have developed six primary tasks. The first task evaluates ChatGPT’s ability to perform technical analysis of randomly selected stocks and provide stock recommendations based on each technical indicator used. The second task aims to determine if ChatGPT can act as a portfolio manager by constructing an investment portfolio that meets the client’s needs, with a focus on the application of Modern Portfolio Theory. The third task centers on corporate finance, emphasizing cash flow analysis and capital budgeting analysis. The fourth and fifth tasks are about financial statement analysis. The sixth task involves a binomial tree analysis. A detailed description of these tasks is available in Appendix 2.

We conduct our evaluations using ChatGPT-4o, the latest and advanced model equipped with a code interpreter, and sophisticated data analysis capabilities. ChatGPT-4o excels in performing complex analyses and computations, allowing seamless interaction with various platforms and applications to ensure the accuracy and reliability of results. This enables comprehensive exploration and execution of tasks in finance and data analysis.

3.4.3. Evaluation Metrics

When financial analysts tackle a financial task, their approach typically begins with reasoning based on previously acquired specialized financial knowledge. They identify relevant concepts, formulas, and solutions applicable to the task at hand. Subsequently, they employ various professional tools to code and execute the task, culminating in the output of results. To scientifically compare the capabilities of large models like ChatGPT with traditional financial professionals, it is essential that these models also adopt a similar workflow. This workflow consists of logical reasoning followed by coding and modeling.

In evaluating the financial mathematics and decision-making performance of ChatGPT-4o, we will assess several metrics that encompass both quantitative and qualitative dimensions. These metrics are derived from generalized university rubrics, specifically tailored for elements of financial mathematics, designed to assess students' proficiency in comprehension, reasoning, modeling, data analysis, and critical thinking capabilities (Selke, 2013). Consequently, we divide the key steps of task processing into two primary modules: reasoning and modeling. The reasoning module includes evaluative dimensions such as task understanding and task deconstruction, while the modeling module encompasses calculation ideas and formulas, as well as accuracy. Additionally, we have incorporated an extra metric for critical thinking to assess ChatGPT-4o's ability in the application of knowledge and the level of critical thinking.

Task Understanding: This dimension gauges the ability to assimilate the prerequisites and objectives of a designated task or problem, evaluating the comprehension of the foundational concepts and principles inherent to the task.

Task Deconstruction: This dimension assesses the capability to fragment a task or problem into manageable and resolvable components or steps, focusing on the identification and isolation of pertinent variables and elements within a task.

Calculation Ideas and Formulas: This dimension scrutinizes the aptness and pertinence of the mathematical concepts, calculations, and formulas employed to decipher tasks, assessing the comprehension and application of mathematical models in problem resolution.

Accuracy: This metric quantifies the correctness and precision of the provided solutions against human analysts.

Critical Thinking: This dimension evaluates the capacity to objectively dissect information and formulate reasoned judgments, applying logical and reflective thinking to draw coherent conclusions and make informed decisions. The depth, quality, and efficacy of critical thinking can be assessed using diverse terminology that delineates the level of critical thinking applied (Stevens & Levi, 2023).

For the criteria of task understanding, task deconstruction, and calculation ideas and formulas, we utilize qualitative scales categorized as basic, intermediate, and advanced to evaluate. The basic level identifies some components or steps of the task but lacks clarity and coherence in breaking it down and struggles to isolate pertinent variables and elements. The intermediate level represents effectively breaking down the task into clear, manageable components, accurately identifying and isolating pertinent variables and elements. The advanced level presents a skillful and coherent deconstruction of the task into detailed, manageable components, demonstrating precise identification and isolation of all pertinent variables and elements.

For the assessment of critical thinking/application of knowledge, we employ descriptors such as practical, applicable, functional, operational, and useful for questions 31 and 32 in the multi-step reasoning tasks. This practical descriptor evaluates if the knowledge applied is realistic and can be implemented in real-world scenarios. The applicable term assesses whether the knowledge is relevant and suitable for the given task. The functional descriptor evaluates if the applied knowledge effectively performs its intended purpose within the task. The operational descriptor checks if the knowledge can be actively used in real-world operations, considering all practical constraints and requirements. The useful descriptor measures the overall utility of the knowledge in achieving the task’s objectives.

Conversely, for complex reasoning tasks such as investment suggestions and corporate strategy, the evaluative process is anchored in varying levels of critical thinking to appraise performance, with terms including advanced, moderate, basic, superficial, and naive. The advanced level signifies a deep and thorough understanding, with the ability to analyze, synthesize, and evaluate information critically. It involves strategic thinking and insightful judgment. The moderate level indicates a reasonable level of critical thinking, where the individual can interpret and analyze information adequately but may not demonstrate the same depth of insight as at the advanced level. The basic level shows a fundamental understanding and ability to apply critical thinking but with limited depth and complexity in reasoning. The superficial level suggests a shallow approach to critical thinking, where the individual’s analysis and evaluation lack depth and are primarily surface-level. The naive level indicates a very simplistic and undeveloped approach to critical thinking, often characterized by a lack of understanding and basic reasoning skills.

This comprehensive evaluative framework ensures a nuanced and multifaceted assessment of both human analysts and ChatGPT in the domains of financial mathematics and decision-making. It allows for a robust comparison and analysis of competencies and proficiencies across diverse tasks and scenarios.

4. Empirical Results and Findings

4.1. Data Collection/Retrieval

First, it is evident that contemporary AI models, including those analogous to ChatGPT, lack the functionalities and capabilities for real-time data retrieval. Consequently, they cannot directly generate the datasets required for specific financial analyses. Instead, these models are primarily limited to guiding users on potential sources from which pertinent data can be acquired, as illustrated in Figure 1.

For academic pursuits, practitioners ranging from students to seasoned professionals such as analysts, traders, and investors might consider platforms like Yahoo Finance, which offers complimentary access to a vast array of financial data. However, for more comprehensive datasets, one may turn to institutional databases. Organizations often provide access to premium platforms like S&P Capital IQ, Bloomberg, and LSEG Refinitiv Workspace, among other specialized software, to facilitate in-depth financial analysis.

Consequently, the data used in our Complex Reasoning Tasks were sourced from S&P Capital IQ and LSEG Workspace for the following stocks listed on the ASX: Chalice Mining (CHN), Vulcan Energy Resources (VUL), Fineos Corporation (FCL), Southern Cross Gold Ltd (SXG), Liontown Resources (LTR), Neuren Pharmaceuticals (NEU), WiseTech Global Ltd (WTC), Aristocrat Leisure Limited (ALL), NextDC Ltd (NXT), and Pro Medicus Limited (PME). Additionally, we retrieved Australian 10-year bond yields from Bloomberg on the 15^th of May and divided it by 252 trading days to get the daily yield.

4.2. Multi-Step Reasoning Tasks Results and Findings

Based on the comprehensive multi-step analytical assessment presented in Table 1, it can be concluded that ChatGPT-4o demonstrates a proficient capability in basic or standard financial analysis reasoning. It follows a step-by-step procedure, working through sequential processes to find solutions, akin to highly capable human analysts. In most cases (27 out of 30), ChatGPT-4o reaches accurate conclusions and exhibits a strong understanding of the task at hand. Due to space constraints, we are only displaying the task results that differ from those of human analysts. Results for other tests can be provided upon request.

Several noteworthy insights emerge from the observations. Firstly, the importance of prompts cannot be overstated. Prompts are instructions or queries entered into the AI’s interface to elicit responses, and they require careful wording and specific instruction. Inadequate instructions or poorly aligned Excel files often result in error messages and failure to achieve meaningful results. During our experiment, we observed that unclear instructions led to such issues.

Secondly, ChatGPT-4o demonstrates the ability to learn from instructions. Of the 30 calculation-focused multi-step reasoning tasks, the answers generated by ChatGPT-4o diverged from those provided by human analysts in only three instances: Tasks 9, 12, and 19. However, with the appropriate instructions or hints, ChatGPT-4o eventually arrives at the correct solutions, similar to those produced by skilled human analysts. For instance, in Task 9, ChatGPT-4o initially struggled with the exponential calculation, repeatedly arriving at an incorrect answer of 22.73%. After a question was asked, it corrected its answer to 19%, aligning with the human analysts' solution. However, on the following day, when the same question was asked again, ChatGPT-4o produced another incorrect answer by using a different approach. Detailed information can be found in Figure 2.

Task 12 involved calculating the internal rate of return (IRR). In the initial attempt, ChatGPT-4o employed a trial-and-error method but persisted in trying with larger rate numbers despite the net present value diminishing. A human analyst had to intervene and provide guidance, after which ChatGPT-4o completed the task. Subsequently, when the same task was entered again, ChatGPT-4o immediately produced the correct answer. However, on another fresh trial the next day, ChatGPT-4o generated an incorrect result by using Python. More detailed information is provided in Figure 3.

The issue with Task 19 pertained to the application of the weighted average cost of capital (WACC) for mergers and acquisitions (M&A). Initially, ChatGPT-4o incorrectly applied the WACC of the acquired firm, resulting in different outcomes compared to those of a human financial analyst. Upon receiving prompts about selecting the appropriate WACC for M&A, ChatGPT-4o correctly identified the use of the acquiring firm’s WACC. Thus, with the proper instructions, it reached the correct conclusion.

Additional observations include instances where ChatGPT-4o does not directly provide final answers. In such cases, it recommends using tools like a financial calculator, Excel, or Python to complete the task.

For conceptual or qualitative tasks, such as Task 31 and Task 32, ChatGPT is capable of producing responses that are logical and adhere to recognized standards. However, these answers tend to be concise and may require further investigation. For example, in Task 31, which involves the understanding and insights into the dividend growth rate, ChatGPT-4o simply applied the average value, overlooking other elements that may affect the growth rate.

Moreover, it is noticeable that responses can vary each time a task is given, even though the main theme is maintained. This variability is a characteristic of artificial intelligence models. Language models, like chatbots, fundamentally operate as probabilistic systems, unlike deterministic systems. This means that posing the same questions can lead to different responses due to the inherent variability in the model's response generation. In these tasks, the wording and structure of the task significantly affect the resulting response generated by the model.

Conversely, for computational or quantitative tasks, the responses, including any incorrect outputs, tend to be consistent across multiple repetitions until intervention occurs. This consistency in computational tasks contrasts with the variability seen in responses to qualitative or conceptual tasks, underscoring the different response mechanisms inherent to artificial intelligence models in different task environments.

Overall, financial analysis is a critical task where even a small error can result in significant financial losses. The ongoing refinement and synergistic collaboration between LLMs and human expertise are crucial to melding analytical precision with human intuition. Therefore, it is recommended to utilize ChatGPT for analysis with great care and caution. It is imperative to always double-check the results to ensure accuracy.

4.3. Complex Reasoning Tasks Results and Findings

Table 2 presents the performance of ChatGPT-4o in executing complex reasoning tasks, highlighting its proficiency in technical analysis, portfolio construction, and corporate finance. For technical analyses and portfolio construction, we asked ChatGPT-4o to select the best 10 ASX-listed stocks based on the performance between January 2024 and the 14^th of May 2024. Prompts are presented in Figure 4. After that, we extracted the daily stock prices from LSEG Workspace.

Within technical analysis, particularly Tasks 1a to 1c in Appendix 2, ChatGPT-4o showed proficiency in performing tasks related to Bollinger Bands, Moving Average Convergence/Divergence (MACD), and Relative Strength Index (RSI). Following this, it offered individual stock recommendations—whether to buy, sell, or hold—based on the latest technical indicators available in our data sample. To validate the outcomes generated by ChatGPT-4o, we utilized LSEG / Refinitiv Workspace to create comparable results, typically formulated by us - human analysts. It was observed that the Bollinger Bands and MACD generated by ChatGPT-4o aligned with those from Workspace. However, discrepancies were identified in the RSI charts between ChatGPT-4o and Workspace.

Further, ChatGPT-4o demonstrated the capability to offer investment recommendations, providing rational justifications to back stock recommendations stemming from each technical indicator. For instance, it could detect a potential bullish crossover in the MACD and subsequently propose a “hold” recommendation. When it detects a bearish crossover in the MACD, it proposes a “sell” recommendation. In addition, when the price goes above the upper Bollinger band, ChatGPT-4o suggested a “sell” as it is in an overbought condition. The results are depicted in Figure 5.

For Complex Reasoning Task Number 2 (i.e. 2a to 2e), ChatGPT-4o demonstrated proficiency in mirroring the responses of human analysts by constructing a global minimum variance portfolio and optimal risky portfolio, determining the weights of each stock in the portfolios, and combining the portfolios. However, there is a discrepancy in the stock weights of the global minimum variance portfolio determined by Excel/Stata and ChatGPT-4o. For an optimal risky portfolio, stock weights provided by ChatGPT-4o are almost the same as the weights computed by Excel and Stata. ChatGPT-4o also successfully constructed an efficient frontier promptly. Both ChatGPT-4o’s results and our analyses are presented in Figure 6.

However, ChatGPT-4o faced challenges in completing Complex reasoning Tasks 3, 4 and 5. Task 3 assessed ChatGPT-4o’s ability in capital budgeting analysis, evaluating its proficiency in interpreting extensive information, distinguishing relevant information, and critical thinking. The results provided by ChatGPT-4o were inconsistent with human analyst calculations, showing errors in analyzing information and recognizing irrelevant costs, and miscomputing depreciated expenses. Moreover, ChatGPT-4o failed to create a detailed capital budgeting template, outlining each cash inflow and outflow item annually.

In Tasks 4 and 5, we asked ChatGPT-4o conduct financial statement analyses (task 4 is a basic financial statement analysis, task 5 is a complex financial statement analysis). These tasks were sourced from the CFA problems test bank in the book of Essentials of Investments (Bodie et al. 2022). However, the final results varied significantly when compared to those of a human analyst. For example, without explicit instruction, ChatGPT-4o would apply the three-component DuPont formula analysis for Task 4 instead of the commonly used five-component method. Additionally, it appears that ChatGPT-4o struggles to accurately retrieve data tables formatted as images. As shown in Figure 9, the Excel template created by ChatGPT-4o displays different values. Any issues encountered in the initial step led to markedly different results or interpretations in subsequent steps. Results are presented in Figure 7, Figure 8, and Figure 9, respectively.

Task 6 assessed whether ChatGPT-4o could calculate the American call option price using binomial tree approach. As shown in Figure 10, tree diagram generated by DerivaGem indicates that early exercise is optimal at certain node (indicated by red value). However, ChatGPT-4o concluded that early exercise is not optimal at any step. In addition, it wasn’t able to display the binomial tree diagram, even after instructing it to follow DerivaGem diagram. Lastly, we attempted to use the new voice interaction feature in ChatGPT-4o. GPT-4o could provide a better tree diagram, but the option prices and early exercise decisions remained incorrect.

4.4. Other Observations

During our reasoning evaluation, several issues related to ChatGPT-4o were identified.

First, even when the same methods have been applied, a discrepancy exists between the charts produced by ChatGPT-4o and Workspace. Since both ChatGPT-4o and Workspace are tools or software used by human analysts to draw conclusions, it is plausible that the charts are slightly different from one another. Despite the existing discrepancies in both charts, the stock recommendations using RSI from both ChatGPT-4o and Workspace are consistent (RSI value lies between lower and upper bands).

Second, ChatGPT-4o relies mainly on Python programming. According to Dilmegani (2024), “the code interpreter only supports Python as a language”. Difference in programming methods may cause differences such as the stock weights in the construction of a global minimum variance portfolio. In addition, this requires Python experts or analysts with proficient Python skills to detect any discrepancies in the calculation method.

Third, the capital budgeting and financial statement analyses exposed a shortfall in ChatGPT-4o’s capability to replicate human analytical processes, particularly in offering sequential calculations and in creating Excel-like templates outlining each cash flow item. This indicates that ChatGPT-4o generates responses based on patterns learned during training and doesn’t understand context or infer meanings in the way humans do. This highlights a limitation in ChatGPT-4o’s ability to accurately process comprehensive information, suggesting a potential obstacle in its capability to assimilate and analyze complex data sets accurately.

Lastly, ChatGPT-4o may not provide accurate results for specialized finance areas such as derivative securities. Although GPT-4o was able to perform the step-by-step calculations like a human analyst, the results were not correct. Furthermore, it has to rely on Python programming to display the tree diagram, but the structure is quite different from a normal binomial tree diagram.

5. Conclusion

The advancement of artificial intelligence (AI) and large language models (LLMs) such as ChatGPT has showcased remarkable capabilities, particularly in financial analysis. This study has examined the reasoning abilities of ChatGPT-4o through various financial tasks, providing significant insights into the strengths and limitations of LLMs in mimicking human analytical processes.

ChatGPT-4o has demonstrated considerable skill in performing standard financial reasoning tasks, closely aligning its analytical approach with that of human analysts. It excels in logical reasoning, task decomposition, and generating solutions, which are essential for tasks like financial modelling and forecasting. However, the study also highlights several challenges and limitations.

The variability in ChatGPT-4o's responses, especially for qualitative tasks, underscores the importance of explicit instructions and careful task formulation. The discrepancies observed in some tasks between ChatGPT-4o and human analysts emphasize the need for robust evaluation metrics to ensure consistent and reliable outputs. Additionally, ChatGPT-4o encountered difficulties with complex tasks requiring a higher level of analytical depth and comprehensive understanding, indicating its limitations in replicating intricate human analytical methods.

Despite these challenges, the prospective integration of ChatGPT-4o with specialized financial data providers and tools, such as Bloomberg, S&P Capital IQ, and Excel etc., represents a transformative shift in the financial sector. This integration is poised to significantly enhance human analytical processes, enabling financial professionals to concentrate more on critical decision-making elements. The recent introduction of advanced features, such as the code interpreter and voice and video models, has further expanded the functionalities of AI in financial applications, thereby solidifying its status as an invaluable resource for complex financial analyses.

In conclusion, while ChatGPT-4o exhibits strong analytical reasoning capabilities, its integration with human insight is essential to fully harness cognitive capabilities in financial analysis. The ongoing refinement and collaboration between LLMs and human expertise are crucial to achieving analytical precision and leveraging the full potential of AI in finance. It is recommended to use ChatGPT-4o with caution, always verifying results to ensure accuracy and reliability in financial decision-making.