The Use of AI in Software Engineering: Synthetic Knowledge Synthesis of Recent Research Literature

1. Introduction

In the last couple of years, artificial intelligence (AI) has witnessed exponential growth in development, the rise of AI use, and increased public interest on different levels, from individual to organizational [1,2,3]. Modern AI utilizes machine learning and other advanced techniques to generate new knowledge, content, hypotheses, and even innovative ideas by identifying patterns and information usually found in big data-size databases. This has catalyzed the use of AI in a broad spectrum of applications, including software design and development [4]. Software companies shifted their focus to deploying AI paradigms to their existing development processes. The academic community started to research and inject new AI-based approaches to provide solutions to traditional software engineering problems [5] and critical activities [6]. Examples include software testing [7], maintenance [8], requirements extraction [9], ambiguity resolution [10], software vulnerability detection [11], and software engineering education [12]. Despite the increasing prevalence of AI use in software engineering, a comprehensive and holistic understanding of the current status, possible target applications, practical software engineering usage scenarios, and unavoidable limitations, ethical concerns, and challenges remain unclear [6].

To close the above gap, this paper presents a comprehensive research landscape of the current research literature on the use of AI in software engineering. The landscapes aim to serve as a framework for informing and solving theoretical and practical challenges in software development and design related to AI SU. The research community and practicing software engineers can use it to improve their understanding of this fast-growing and highly innovative area. It can also inform novice researchers, grant administrators, software managers, and interested readers lacking specific domain knowledge to develop a perspective on essential research dimensions. Finally, the landscape can guide and inform further research and serve as a starting point for more formal knowledge and evidence synthesis approaches.

2. Materials and Methods

The research landscape representing AI use in software engineering was induced by Synthetic Knowledge Synthesis (SKS) [13]. SKS integrates quantitative and qualitative synthesis by triangulating descriptive bibliometrics, bibliometric mapping, and content analysis, thus reducing the weaknesses of traditional knowledge synthesis approaches [14]. A research landscape is a map/network of the relationships and associations between bibliometric units. In our present study, those units represent author keywords. Links on the map are current relations, proximity similarity, and node size popularity. Landscape areas (colored clusters) in our study represent strongly associated authors' keywords, either thematically or timewise. The third component of SKS is a content analysis [7], a resourceful approach which, in our case, was used for a qualitative analysis of phenomena contained in research publications to obtain their objective and holistic descriptions in the forms of categories and themes. Scopus (Elsevier, The Netherlands) was used as the source bibliographic database because it is deemed the largest abstract and citation database of the reviewed research literature. In addition to advanced analytics services, it enables 20,000 records to be exported simultaneously.

The search query shown below was constructed using the recommendation provided by Farooq et al. [15].

TITLE-ABS-KEY(("artificial intelligence" OR "machine learning" OR "deep learning" OR "intelligent system" OR "support vector machine" OR ("decision tree" AND (induction OR heuristic)) OR "random forest" OR "Markov decision process" OR "hidden Markov model" OR "fuzzy logic" OR "k-nearest neighbor" OR "naive Bayes" OR "Bayesian learning" OR "artificial neural network" OR "convolutional neural network" OR "recurrent neural network" OR "generative adversarial network" OR "deep belief network" OR "perceptron" OR {natural language processing} OR {natural language understanding} OR {general language model}) and ({software engineering} OR {software design} or {software development})) AND PUBYEAR > 2018 AND PUBYEAR < 2025

The search was performed on February 14th, 2024. The resulting corpus was analyzed using SKS, focusing on bibliometric mapping and content analysis. Finally, using the identified themes and categories as a basis, we performed a literature synthesis and review.

3. Results and Discussion

The search resulted in 9080 publications. Among them, there were 5187 conference papers, 3097 articles, 354 conference reviews, 185 book chapters,179 review papers, 15 editorials, ten retracted papers, nine errata, five short surveys, four notes, and 1 data paper. The recall value of the search was 0.95. The paper type distribution shows that most publications were conference-related papers, revealing that research is still in a maturation phase and that core knowledge is still forming. This finding is also confirmed by the fact that the three most prolific titles are conference proceedings, namely Advances In Intelligent Systems And Computing (n=456), ACM International Conference Proceeding Series (n=341), and Lecture Notes In Computer Science Including Subseries Lecture Notes In Artificial Intelligence And Lecture Notes In Bioinformatics (n=249). The first journal source title is Information Sciences (n=245), followed by another conference proceedings, Ceur Workshop Proceedings (n=190), indicating that the list of core journals has not yet been established. The H-index of the above source titles lies between 58 and 446, meaning that their quality is averagely high and that most publications are not yet published in top-tier journals. The H-index of the whole sub-field of AI use in software engineering has been 87 for the last five years.

The research productivity trend shown in Figure 1 is surprising since the productivity peak of the total number of publications was already reached in 2020. However, the number of publications stabilized in 2022. The decreasing number of publications is mainly due to the decreasing number of conference papers, while the number of articles increased in 2022. Both above facts might reveal the start of a positive trend toward reaching the research maturity.

The far most productive countries were China (n=2042), followed by the United States (n=1193), India (n=934), Germany (n=445), and Canada (n=381). That is in line with Scimago Country Rankings (Elsevier, Amsterdam, Netherlands), where the United States is first ranked in Software and second in Artificial Intelligence, China is first In Artificial Intelligence and second in Software, and other top countries are among the ten most productive in both categories. All top productive countries also belong to G20 [16]. China also prevails among the most productive institutions; among the first five, four are from China, namely the Ministry of Education of the People's Republic of China (n=90), Chinese Academy of Sciences (n=89), Nanjing University (n=72), and Peking University (n=71). The only non-Cina institution among the top five in third place is Monash University, Australia (n=73). The most productive USA institution is Chalmers University (n=46), which is in 16th place, and the most productive European one is the Chalmers University of Technology. Sweden (n=54) in 9th place.

Another important indicator of the research state of a scientific field/sub-field is research funding [17]. Our analysis showed that 41.1% of papers are funded, which is notably more than in many other disciplines [18], however, less than in a comparable subfield, namely the use of AI in pediatrics, where 47.4% of papers were funded [2]. The most prolific funding sponsor is the National Natural Science Foundation of China (n=987), the National Science Foundation, USA (n=284), the National Key Research and Development Program of China (n=226), the Horizon 2020 Framework Programme (n=154), and the European Regional Development Fund (n=105).

3.1. Identification of Main Research Themes

Content analysis of the research landscape consisted of 146 author keywords (Figure 2), revealing 15 categories and five themes, which are summarized in Table 1.

3.1.1. Literature Review of Research Categories and Themes

Use of artificial intelligence in management of software development life cycle

Ethical use of AI-based software engineering

Vakkuri et al. [19] noted that ethical considerations are mostly ignored while developing AI-based software systems. Consequently, general and high-level guidelines for managing ethics issues have been proposed [20,21,22]-

Use of fuzzy logic in software development and testing.

Fuzzy logic techniques have been used in selecting software requirements from elicited software requirements or ordering them by preferences [23], cloud-based testing adaption [24], and software effort estimation [25]

Automation of software testing in an agile environment. Artificial intelligence has been used to generate test cases for automatic testing in agile environments [26,27,28] and to automate other phases of the software development lifecycle [29].

Project management of software life cycle using fuzzy logic.

Fuzzy logic techniques have been used to support project management activities like cost and effort estimation [30,31], imputation of missing values in empirical software project management [32], management of outsourcing [33], and risk assessment in the agile environment [34] and software product promotion [35].

Data science and big data in software development.

Data science and the availability of extensive software development databases enabled the rise of computer, and AI-aided software engineering [36], estimation of story points in agile environments [37], and support of empirical software engineering in general [38].

Natural language processing (NLP) in software engineering

Natural language processing in software development

NLP technology can drastically improve software development tasks [39]. It can support bug categorization [40], development of more secure software [41], program decomposition [42], classifying commitments [43], programming and coding [44], writing coherent and factually correct readmes [45], model-driven engineering [46], deployment of design patterns [47] and traceability management [48]-

Natural language processing in software requirements engineering

NLP can support human-performed linguistic analysis in requirements engineering [49,50], such as identifying domain concepts [51,52,53,54], establishing traceability links [55], requirement classification [56,57], handling ambiguity [58,59], preference extraction form scenarios [60], classification of non-functional requirements [61], standardization of requirements in agile approaches [62] and requirement elicitation [63].

User stories understanding with natural language processing

NLP has also been used to extract feature, goal, and domain models from user stories [51,54,64], improve the completeness of acceptance criteria [65], or build software structures from user stories [66].

Machine learning in fault/defect prediction and effort estimation

Software development effort estimation

Software development effort estimation is one of the most popular AI techniques used by Intelligent software engineering [67], and cost estimation is one of the most crucial software engineering tasks [68]. Different machine learning algorithms like random forests [69], differential evolution [70], or extreme learning [71]. AI-based effort and cost estimation are used in traditional [72] and agile environments [73].

Data mining in software fault/defect prediction

Data mining and machine learning are used to classify software faults [74] for their detection [75] and prediction [76,77].

Machine learning and software metrics

Machine learning is used to detect code smells [78,79], support software size metric estimation [80], asses software component reusability [81], or predict test flakiness [82].

Deep learning in empirical software engineering focusing on code management

Deep learning in program comprehension and vulnerability detection

In various ways, deep learning is used in software and program code comprehension [83]. For example, Hybrid-DeepCom and DeepComenter tools automatically generate code comments for Java functional units [84,85]. Similarly, deep learning is also used to generate pseudo-code from program code [86], classify code according to readability [87], or summarise code [88]. In the same context, deep learning is also used to detect code vulnerability [89,90]-

Technical depth and code smell detection

Machine learning has recently been often used to detect self-admitted technical depth [91,92,93] or technical debt in general [94,95]. Technical debt is closely linked to code smells and anti-patterns, which are spotted [96,97], classified [98,99], or identified [100]with artificial intelligence techniques.

COVID-19 influence on software engineering

During the COVID-19 pandemic, highly collaborative software development teams were bound to work online in a distributed manner, and many software companies transferred to hybrid models after the pandemic. In such environments, AI can be used to enhance the well-being of developers [101,102].

Mining software repositories to improve software quality

Mining software repositories to enhance the quality of software and software maintenance

Software fault triaging has become a significant activity in software maintenance. Guo et al. [103] used convolutional neural networks to learn about developers' fault reports and then automatically perform software fault triaging. Triaging has also been performed using the K Nearest Neighbour approach on stack traces and categorical features [104]. A long short-term memory algorithm has been used to estimate the fault-fixing times [105] and dependency graphs for semantic versioning of third-party library components [106]. A fine-tuned Transformer has been employed to predict both the objective behind opening an issue and its priority level in GitHub repositories [107].

GitHub and Open source software as sources for mining software development data

Data mining on GitHub or Open source repositories has been used to create evolving project data sets for intelligent/empirical software engineering [108,109], for example, for Eclipse Modelling Framework metamodels formation [110,111] anomaly detection [112] or identification of migration reasons [113].

3.2. Timeline of the Recent Research and Hot Topics

Figure 3 reveals that according to the average age of author keywords, the period 2020-2024 started with the research on the use of data and text mining in combination with deep learning in search-based software engineering, focusing on program comprehension, software metrics, and effort estimation in agile environments. In the middle of the period, the research was mainly linked to machine learning in software testing, mining software repositories for model-driven engineering, code smells, and software development effort estimation. Hot topics seem to be the research on the use of large language models and explainable machine learning (i.e., decision trees) for fault prediction, code understanding, ethics, and vulnerability detection.

3.2. Research Gaps and Challenges

Our analysis also revealed the same research gaps and challenges that should be dealt with and require future research. At the time being, AI is not yet reliable enough, and software engineers and developers must thoroughly check its algorithms' output. Additionally, the increased use of AI in software engineering might result in more and more code, which will be less and less understood of how it works, establishing a positive feedback loop, which will lead to more and more checking, which might result in that, that more time will be spent on checking then developing.

AI needs vast amounts of data to create its models. That’s not a problem if software engineers can train AI algorithms using the data from public and open repositories – but if they work in unique domains, appropriate training sets might not be available.

As with AI use in general, much more research is needed to resolve ethical concerns, even more so because software engineers are generally not trained and educated to deal with societal impact and ethical issues.

Extensive use of AI in software engineering will require new specialists’ skills, which should be incorporated into software engineering curricula. Advanced tools, especially large language models, require sizeable computational power, storage space, and energy supply, which might increase costs. AI-based tools may require extra licensing fees.

3.4. Possible Future Research Trends

Based on the above analysis, several feature directions come to mind:

• Development of transparent, fair, ethical, responsible, and sustainable intelligent software development processes.
• Self-adapting software that adapts to evolving user requirements.
• Self-healing and self-reflecting software returns to a more functional condition after faults or performance and cybersecurity issues.
• Collaborative software development eco-systems where AI partners with human developers take team dynamics and self-organization into account.
• New software engineering curricula.
• Adaptive continuous learning platforms for software developers and engineers.

4. Conclusions

In conclusion, our analysis revealed that research on AI has significantly impacted software development in recent years. From natural language processing in software engineering, the use of artificial intelligence in the management of software development life cycle, the use of machine learning in fault/defect prediction and effort estimation, employment of deep learning in intelligent software engineering and code management to mining software repositories to improve software quality, AI has changed the way developers and engineers build software. The convergence of AI and software engineering has the potential to significantly reduce needed resources, improve quality, and increase user experience with more intelligent user-centric applications. On the other hand, it may enhance the well-being of software developers and engineers with automation of repetitive tasks, reduced workload, improved and more reliable/accurate predictive analysis, and speeding up the development cycle. Finally, it can also help software managers monitor overall team status, state, and performance, providing them with notifications if a team member has been over-utilized, over-extended, or is heading to burnout.