A Survey on Artificial Intelligence based Internet of Vehicles utilizing Unmanned Aerial Vehicles

1. Introduction

The Internet of Vehicles (IoV) is one of the subsystems of Internet of Things (IoT) that consists of mobile vehicles installed with sensors, processing units and software to communicate with one another over the internet or other communication networks [1,2]. IoV is a decentralised system which promises the security and privacy of vehicular and users’ data. It integrates different technologies by providing reliable communication tools [3,4].

In an IoV, vehicle-to-everything (V2X) communication combines the data sharing from vehicles to infrastructure (V2I), between vehicles (V2V), pedestrian (V2P), roadside units (V2R) and unmanned aerial vehicles (V2U). This would play a key role in intelligently managing vehicular traffic as well as network data traffic, providing safer roads and increasing vehicular energy efficiency. However, V2X communication has it’s own limitations. In an IoV, the vehicles are highly mobile with varying densities and need permanent communication links for reliable data exchanges. In most cases, the fixed infrastructures such as fixed RSUs and BSs are unable to provide satisfactory communication and computational services to the vehicles and as a result of this, the QoS deteriorates.

Incorporation of unmanned aerial vehicle (UAV) with IoV can be very advantageous to guarantee a reliable communication infrastructure as it provides better line-of-sight connectivity that would greatly help in load balancing, mobility management, routing solutions and cost effective communications. An UAV can operate independently and has the ability to move and carry data from one place to another. Installed with sensors, computer units, cameras, GPS, wireless transceivers to communicate, an UAV can have a predetermined flight path, interact with the environment around it and if needed dynamically adapt their paths during the flight. Therefore, they are great assets to IoV systems [5]. In particular, UAVs can overcome the limitations of fixed RSUs because they can dynamically change their speed and position, and they can gather and transfer data from one area to another area [6].

Table 1. List of Abbreviations and Notations.

Abbreviation	Description	Abbreviation	Description
3DQN	Double Dueling Deep Q Network	LTE	Long Term Evolution
3GPP	3rd Generation Partnership Program	MACEL	Multi-Agent Collaborative Environment Learning
5G	5th Generation	MAC	Medium Access Control
A3C	Asynchronous Actor Citric	MARL	Multi-Agent Reinforcement Learning
AEC	Average Energy Constraint	MDP	Markov Decision Process
ASR-CUMS	Automated Slice Resource Control and Update Management System	MJDDPG	Multi-objective Joint Optimization-Oriented DDPG Algorithm
ARdeep	Adoptive Reliable Deep	MLP	Multi-Layer Perceptron
BT-MP-DQN	Beam-forming Control and Trajectory-Multi-Pass Deep Q Network	mmWAVE	Millimeter Wave
CA-MOEA	Clustering based Adoptive Multi Objective Evolutionary Algorithm	MOEA/D	A Multi-objective Evolutionary Algorithm Based on Decomposition
CCSRL	Cluster-enabled Cooperative Scheduling based on Reinforcement Learning	MSA-LS	Mobile Service Amount based Link scheduling
CEPF	Context Aware Packet Forwarding	NGSIM	Next generation Simulation
CKF	Constant Kalman Filter	QAGR	Geographic Routing with Q-Learning
CSMA	Carrier Sense Multiple Access	QFHR	Q-learning and Fuzzy-based Hierarchical Routing Solution
DDPG	Deep Deterministic Policy Gradient	QLBR	Q-Learning based Load Balancing Routing
DFS	Depth First Search	QLFMOR	Q-Learning based Fuzzy Logic for Multi Objective Routing Algorithm
DGCIM	Dual Graph Coloring based Interference Management	QTAR	Q-Learning based Traffic Aware Routing
DQL	Deep Q-Learning	RRB	Radio Resource Block
DQN	Deep Q-Network	RRPV	Reinforcement Learning Routing Protocol for Vehicles
DRL	Deep Reinforcement Learning	SGD	Stochastic Gradient Descent
DRQN	Deep Recurrent Object Networks	SUMO	Simulation of Urban Mobility
EED	End-to-End Delay	SWIPT	Simultaneous Wireless and power Transfer
FANET	Flying Ad-hoc Network	UAS	Unmanned Aerial System
FLRLR	Fuzzy Logic Reinforcement Learning based Routing	UE	User Equipment
GCS	Ground Control Station	UCPA	UAV based Clustering and Positioning Protocol
GMM	Gaussian Mixture Model	URLLC	Ultra-Reliable Low-Latency Communications
GPGC-RLF	Grouping Graph Coloring with Recursive Largest First	V2N	Vehicle-to-Network
GYGC	Greedy Graph Coloring	V2P	Vehicle-to-Pedestrian
IMU	Inertial Measurement Unit	V2R	Vehicle-to-Roadside Infrastructure
iProPHET	Improved Probability Routing Protocol using History of Encounters and Transitivity	VANET	Vehicular Ad-Hoc Network
ITS	Intelligent Transport Systems	VEC	Vehicular Edge Computing
JTSM	Joint Time Series Modeling	VUE	Vehicle User Equipment
LBTO	Load Balancing and Task Offloading	WPT	Wireless Power Transfer
LIDAR	Light Detection and Ranging	WMMSE	Weighted Minimum Mean Square Error
LPA	Long Prediction Algorithm	WSN	Wireless Sensor Network

Table 1. List of Abbreviations and Notations.

Abbreviation	Description	Abbreviation	Description
3DQN	Double Dueling Deep Q Network	LTE	Long Term Evolution
3GPP	3rd Generation Partnership Program	MACEL	Multi-Agent Collaborative Environment Learning
5G	5th Generation	MAC	Medium Access Control
A3C	Asynchronous Actor Citric	MARL	Multi-Agent Reinforcement Learning
AEC	Average Energy Constraint	MDP	Markov Decision Process
ASR-CUMS	Automated Slice Resource Control and Update Management System	MJDDPG	Multi-objective Joint Optimization-Oriented DDPG Algorithm
ARdeep	Adoptive Reliable Deep	MLP	Multi-Layer Perceptron
BT-MP-DQN	Beam-forming Control and Trajectory-Multi-Pass Deep Q Network	mmWAVE	Millimeter Wave
CA-MOEA	Clustering based Adoptive Multi Objective Evolutionary Algorithm	MOEA/D	A Multi-objective Evolutionary Algorithm Based on Decomposition
CCSRL	Cluster-enabled Cooperative Scheduling based on Reinforcement Learning	MSA-LS	Mobile Service Amount based Link scheduling
CEPF	Context Aware Packet Forwarding	NGSIM	Next generation Simulation
CKF	Constant Kalman Filter	QAGR	Geographic Routing with Q-Learning
CSMA	Carrier Sense Multiple Access	QFHR	Q-learning and Fuzzy-based Hierarchical Routing Solution
DDPG	Deep Deterministic Policy Gradient	QLBR	Q-Learning based Load Balancing Routing
DFS	Depth First Search	QLFMOR	Q-Learning based Fuzzy Logic for Multi Objective Routing Algorithm
DGCIM	Dual Graph Coloring based Interference Management	QTAR	Q-Learning based Traffic Aware Routing
DQL	Deep Q-Learning	RRB	Radio Resource Block
DQN	Deep Q-Network	RRPV	Reinforcement Learning Routing Protocol for Vehicles
DRL	Deep Reinforcement Learning	SGD	Stochastic Gradient Descent
DRQN	Deep Recurrent Object Networks	SUMO	Simulation of Urban Mobility
EED	End-to-End Delay	SWIPT	Simultaneous Wireless and power Transfer
FANET	Flying Ad-hoc Network	UAS	Unmanned Aerial System
FLRLR	Fuzzy Logic Reinforcement Learning based Routing	UE	User Equipment
GCS	Ground Control Station	UCPA	UAV based Clustering and Positioning Protocol
GMM	Gaussian Mixture Model	URLLC	Ultra-Reliable Low-Latency Communications
GPGC-RLF	Grouping Graph Coloring with Recursive Largest First	V2N	Vehicle-to-Network
GYGC	Greedy Graph Coloring	V2P	Vehicle-to-Pedestrian
IMU	Inertial Measurement Unit	V2R	Vehicle-to-Roadside Infrastructure
iProPHET	Improved Probability Routing Protocol using History of Encounters and Transitivity	VANET	Vehicular Ad-Hoc Network
ITS	Intelligent Transport Systems	VEC	Vehicular Edge Computing
JTSM	Joint Time Series Modeling	VUE	Vehicle User Equipment
LBTO	Load Balancing and Task Offloading	WPT	Wireless Power Transfer
LIDAR	Light Detection and Ranging	WMMSE	Weighted Minimum Mean Square Error
LPA	Long Prediction Algorithm	WSN	Wireless Sensor Network

2. Related Work and Survey Contribution

Since last decade, AI has been integrated in IoV as a viable solution to assist in many communication and traffic application related problems. AI combined with V2X can support complex vehicular applications such as traffic control, self governing vehicular mobility, and data storage. ML and Deep Learning (DL), one of the frequently used sub-classes of AI approaches, are thoroughly implemented in IoV to solve complex problems with the help of extensive amounts of available data [7].

Most DL based models require large datasets with various traffic features to train and test the models [8]. However, in vehicular communication, real-time historical data related to routing, channel condition, vehicle’s mobility and resources is not readily available. In such a scenario, it is not possible to use supervised DL techniques. In the absence of historical data, reinforcement learning (RL) has appeared as a very powerful solution in IoV which allows a vehicle to take the decisions by itself for various networking tasks [9,10,11]. The agents which are mostly the vehicles, gather the information about the dynamic environment and then takes an appropriate action to achieve a certain purpose such as resource and mobility management, routing decision and predicting network traffic.

The applications of AI/ML are extensively studied in wireless networks and VANET communications in the existing literature. In [12], mobility and traffic patterns are discussed using the AI/ML techniques in a dynamic vehicular network. The authors presented ML techniques to optimize network performance related to security, handover, resource management and congestion control. The survey in [13] classifies the vehicular research in transportation and network related areas. A detailed AI/ML based vehicular network scenario such as data offloading, mobile edge computing (MEC), network security and transportation cases, related to platooning, autonomous navigation and safety are discussed in detail [13]. The authors list all the software used to implement AI and also reviewed the relevant research problems in this field. In [14], a detailed review of ML techniques is presented in vehicular network related to resource and network traffic management, and reliability. The authors expanded their work to include cognitive radio (CR), beamforming, routing, orthogonal frequency-division multiple access (OFDMA) and Non-orthogonal multiple access tasks in [15].

An overview of ML, CR, VANET, and CR-VANET architectures including functions, open issues and future challenges is presented in [16]. Moreover, the applications of AI/ML in CR-VANET in autonomous vehicular network and their union is also reviewed in this paper. In [17], the authors first discussed the Federated Learning (FL) and its use in wireless IoT. Then, this survey paper points out and discusses the technical challenges for FL based vehicular IoT with future research directions. The survey paper [18], critically review the ML and Deep Reinforcement Learning (DRL) models for MEC decision based offloading in IoV. Main focus of the paper is on buffer and energy aware ML enabled Quality of Experience (QoE) optimization, and it summarizes the recent related research and methods and presents their comparison. In [19], the authors surveyed and analyzed the resource allocation scenarios. In addition to this, the design challenges for resource management in VANETs using ML are presented as well. In [20], a detailed overview of the RL and DRL techniques in IoV network such as joint user association and beam forming, caching, data offloading decisions, energy efficient management of resources and vehicular infrastructure management is presented. Then, future trends, challenges and open issue in 6G based IoV are discussed.

In [21], primary ML concepts based on ML for Wireless Sensor Networks (WSN) and VANETs are summarized briefly. Moreover, ML based WSN and VANETs applications, open issues and challenges are discussed. In [22] a comprehensive survey of AI/ML techniques is presented and then the strengths and weaknesses of these AI models for the VANET environment including safety, traffic, infotainment applications, security, routing, resource and mobility management are provided. In [23] authors surveyed resource allocation techniques on DSRC, Cellular-V2X (C-V2X) and heterogeneous VANET. The AI/ML techniques are reviewed with respect to their integration in VANETs and utilization in designing the several resource allocation tasks related to user association, handover and virtual resource management for V2V and V2I communications. However, AI/ML on V2X is not the main focus of the paper.

In [24], the RL-based routing schemes are classified depending upon the centralized and distributed learning process. Moreover, they surveyed position-based, cluster-based and topology-based routing protocol. The survey in [25] summarizes the vehicular network and Smart Transport Infrastructure (STI) in detail. The paper deals with FL and it’s application in vehicular networks. It elaborates vehicular-IoT (VIoTs), blockchain, FL, and intelligent transportation infrastructure. Then, the FL and blockchain based security and privacy applications in VANET environment are discussed in detail. The challenges arise from integration of FL and blockchain are pointed out in the survey with indication of future research direction. In [26], the survey presents a compilation of network-controlled functions that have been optimized through data-driven approaches in vehicular environments. The research related to integration of AI/ML and V2X communications in the areas such as handover and resource management or user association, caching, routing, beam-forming optimization and QoS prediction are extensively reviewed. This survey classifies the training architecture into centralized, distributed or federated model for each ML technique. The time complexity of supervised, unsupervised and RL models used in the literature are discussed.

In [27], the authors cover the applications of UAVs based IoV network. This work does not include the detailed implementation of AI/ML in UAV based IoV and only mentions a few papers related to software defined network (SDN) based fog computing and AI/ML networks. However, it covers the areas such as privacy, security, congestion and network delays, and communication protocols. In [28], the authors review the Internet of Drones (IoD) and classify the IoD-UAV according to its applications in the areas of resource allocation, aerial surveillance and security and mobility in all the possible IoT based fields. This survey concludes that the most used AI technique in IoD is Convolution Neural Networks (CNN) and the most common areas of research are resource and mobility management. However, this survey completely ignores the IoD based IoV networks. The summary of the survey papers with AI/ML application in IoV networks is provided in Table 2

In our study of the previous literature on UAV based IoV, it is noticed that the application of ML in resource management and routing for UAV or IoD based IoV are not explored yet. At the time of writing of this survey paper, there is no comprehensive survey conducted on this topic. We extensively reviewed the inclusion of autonomous vehicles (AVs) with UAVs and the integration of AI/ML applications in UAV based IoV communications for physical and computation resource allocation and routing algorithms. The paper presents the existing AI/ML based solutions in UAV-IoV, lists the challenges and issues in UAV based IoV with future directions.

The rest of the paper is organized as follows: A basic discussion on the nuts and bolts of IoV and UAV is presented in Section III. In Section IV, AI/ML based resource management in UAV-IoV, V2X and UAV networks is covered. Section V covers AI/ML application in UAV-IoV and V2X and UAV based routing protocols and Section VI presents the challenges and future work and finally we conclude the paper in Section VII.

3. Overview of IoV and UAV Networks

A UAV-based IoV network provides users such as vehicles, passengers and surrounding infrastructure fast data transmission services. It is the result of amalgamation of heterogeneous networks at different computing and communication layers [49,50]. To overcome the limitations of the IoV network related to routing, mobility management, resource allocation, network monitoring and traffic applications, UAVs are proving to be a viable technology. UAV-based IoV networks use cellular network and communication protocols to ensure uninterrupted network connectivity to meet QoS requirements [14]. A UAV assisted IoV network architecture is shown in the Figure where UAVs are flying in the urban area, facilitating the underneath vehicles and infrastructure by providing communication facilities. The UAVs also do communicate with one another.

This section provides a detailed overview of IoV and VANET communication technologies and UAV components and network architecture. The taxonomies and advantages of these technologies are also provided.

3.1. IoV Communication Technologies

IoV is the VANET with internet which provides real-time communication among the vehicles and infrastructure (V2I), vehicles (V2V), pedestrians (V2P), roadside units (V2R) and UAVs (V2U). VANET does not have the access to internet and vehicles communicate with infrastructure using Dedicated Short Range Communication (DSRC) [29]. DSRC is built on IEEE 802.11p [30] and in 2012 became a standard with allocation of 75 MHz bandwidth within the frequency range of 5.85-5.925 GHz by the Federal Communications Commission (FCC). It can achieve the latency of as low as 100 ms. However, DSRC is not good enough to support latency requirements of the future communication based applications as DSRC is based on Carrier-Sense Multiple Access with Collision Avoidance (CSMA/CA) Medium Access Control (MAC) protocol which causes unbounded latency and reliability issues in vehicular communications [31].

Visible Light Communications (VLC) have the potential to solve the spectrum scarcity issue. Its spectrum ranges from 430 to 790 THz [32]. As compared to DSRC, VLC is not impacted by electromagnetic interference. It has low latency, and is less vulnerable to security related attacks than radio base systems [33].

However, the aforementioned standards are unable to assure steady network connectivity to highly mobile vehicles. The long term evolution (LTE) or device to device (D2D) and DSRC can only provide up to 100 Mbps or 3-27 Mbps respectively [34]. The milimeter wave (mmWave) can provide more than 1 Gbps for V2V communication [35,36] and it appeared to be very promising. Recently, mmWave based Giga-V2V (GiV2V) has got a lot of attention in VANET communication. The mmWave is a good fit for applications with rich data and high definition requirements such as camera and LiDAR sensors. But mmWave has its own drawbacks. For instance, its range is limited, it has high penetration loss, LoS requirements, poor diffraction characteristics and mainly it is more costly.

In 2017, Cellular-V2X (C-V2X) was introduced by the 3rd Generation Partnership Project (3GPP) which relies on the capabilities of 4G, 5G and future 6G cellular networks [37]. C-V2X provides high level of safety by ensuring the elevated system performance, communication range and security. In C-V2X, as soon as the wireless link breaks between the cellular BS and vehicle, PC5 [38] interface creates a direct communication channel to provide uninterrupted communication link between two vehicles. The Uu [39] is another logical interface used in C-V2X. IoV-6G could prove a ground breaking technology as it overcomes the shortcomings of 5G by fulfilling the strict key performance indicators (KPIs) [40]

3.2. UAV

Unmanned Aerial System (UAS) is consisted of an UAV and a remote control system to operate the UAV [41]. The basic components and the communication architecture of UAV are discussed in this section.

3.2.1. Components of UAV

A UAV used in IoV environment is often made up of single board computer with CPU, memory and sensors. The sensors play pivotal role in sensing the environment including the global positioning system (GPS), accelerometers, camera and gyroscope for navigation [42], a battery to provide power, a transceiver to exchange data between UAVs and Ground Control Station (GCS) [43], UAV flight status indicator devices [44], flight controller to control UAV take off and landing etc. [45], and finally an inertial measurement unit (IMU) to manage UAV altitude [46].

In UAV based IoV, an UAV and a vehicle can communicate in real-time using a data link, namely Line-of-Sight (LoS), Visual Line of Sight (VLoS) and Beyond Line-of-Sight (BLoS) link. This enables the UAVs to work as mobile base stations (mBSs), to transmit and receive data to and from vehicles, fixed infrastructure, and other UAVs [6,47].

3.2.2. UAV Communication Architecture

UAV communication is divided into two categories namely, centralised communication and decentralised communication.

In centralised communication as shown in Figure 2, UAV communicates with a central controller. There are three types of centralised communications. First is UAV-GCS in which UAV fetches data from GCS using a communication link. This setup does not perform well in adverse weather conditions. Second is UAV-satellite or UAV-High Altitude Platform (HAP) communication which is preferred when there is large distance between UAV and GCS. Third one is cellular technology based UAV-cellular communication. It uses BSs to facilitate the routing technology between nodes [48,49].

In decentralised communication architecture as shown in Figure 3, UAVs build a direct or indirect communication link with the GCS. The gateway UAVs can relay data between GCS and other UAVs in the network. The UAVs, in ad-hoc networks, communicate with each other via wireless devices without relying on GCS [50,51]. There are three types of decentralized UAV communication networks namely simple UAV, multi-group and multi-layer ad-hoc network. In UAV ad-hoc network, the backbone UAV communicates with GCS using high power over long ranges. Then, the backbone UAV becomes a gateway and communicates with other UAVs at short ranges using low power. In multi-group UAV ad-hoc network, each UAV group behaves as FANET with one UAV selected as a backbone UAV to communicate with the GCS. Lastly, in multi-layer UAV Ad-hoc network, the lower layer is used for intra-group communication and upper layer is made up of backbone UAVs to ensure inter UAV and GCS communication. In UAV based communication, multiple links formation help covering the large communication area [48].

4. AI based Resource Allocation in UAV and IoV Networks

In this section, we cover the research contribution in the area of resource allocation for IoV, UAV as well as UAV based IoV networks using AI.

4.1. AI for Resource Allocation in IoV Network

Different objectives have been considered by the authors for AI based resource allocation ranging from load balancing, better QoS or QoE to energy and latency minimization. In this section we focus on reviewing the contributions and research progress in AI based resource allocation for IoV networks.

In [52], the authors combine the network function virtualization (NFV) and QoS guaranteed network slicing (NS) with RL in beyond 5G (B5G) architecture. The slice acceptance advantage of 5% is achieved as compared to the best existing work. One impediment of the research is that the authors assumed the RRB resource allocation in the RAN over direct data rate in RAN. Secondly, there is room for improving the slice allocation capability of the proposed algorithm.

In [53], the authors devised a method for multiple V2V links to reuse the V2I spectrum by using the multi-agent RL. In the training stage proposed model is centralized and in implementation stage it is decentralized. The MARL and SARL algorithms are used for the comparison purpose. The proposed model considerably improves the overall system level performance.

In [54], the authors proposed an AI based task offloading and resource management model. There are two offloading layers proposed. First layer selects between cloud computing (CC) and MEC server using random forest (RF) for task offloading and computing platform is selected in second layer by using DDPG algorithm. The proposed model is compared with KNN, MLP and SVM. T-Drive trajectory dataset, from Microsoft website, is used for the model training and testing. This dataset includes the GPS trajectories of 10,357 taxis during one week in Beijing. The RF model achieves the highest classification accuracy score of 99.83% for task offloading decision as compared to the KNN, MLP and SVM achieved 98%, 94.81% and 90.94%, respectively. Moreover, it is reported that the DDPG significantly minimized the latency cost by 85%.

In [55], the authors proposed the algorithm to build an energy efficient communication system using capabilities of AI and intelligent reflecting surface (IRS). The IRS reflecting signal propagation is used to produce efficient beam-forming for directional transmission in vehicles. Then, DRL model is used for network resource control and allocation. An energy efficiency-maximizing model is formulated and joint optimization of the settings of all vehicles is done for effective and latency-efficient communication. The proposed model is compared with model without IRS and random resource allocation model and the proposed model outperforms both the models.

In [56], a model named ARTNet is proposed to make an AI enabled V2X framework for dividing resources effectively and improvised communication. To achieve their target the authors implemented the proposed algorithm in the software defined vehicular based FC (SDV-F) architecture. In ARTNet, the controllers are responsible to achieve optimal resource utilization at the fog layer, and maximum reduction in the average delay of time critical IoV applications. The proposed model is compared with NRPO and AEC in terms of energy consumption, average latency and energy shortfall.

In [57], an integrated fuzzy based approach is taken to coordinate and manage the abundant resources available in VANETs. In this approach, the system decides the resources that vehicles should use when set under different circumstances including the condition of the network created among vehicles, which is represented by the QoS in the network, its time duration, its size, and the available resource, together with the application requirements. The proposed method achieves higher QoS.

In [58], to tackle the problem of scarcity of computational resources, a selection criteria is proposed to select volunteers’ vehicles capable of executing the computationally intensive task. For the volunteer vehicle identification, the authors used various machine learning based regression techniques including LR, SVR, KNN, DT, RF, GB, XGBoosting, AdaBoost, and ridge regression. For the training and testing of the models vehicular onboard unit computing capability dataset is collected. It contains three different datasets. All three datasets have seven features but different number of samples or sizes. The dataset1 has 17800 samples, dataset2 is made up of 4600 samples and finally dataset3 has only 250 samples. Among all the models implemented, SVR and ridge regression have lowest error value and maximum accuracy.

In [59], the authors predicted leading vehicle trajectory with the speed and acceleration information of the vehicle at front using the proposed method based on joint time-series modeling approach. Moreover, before making any prediction the gaussian mixture model (GMM) is applied on the leading vehicle to predict the driving styles. After style prediction, a LSTM and RNN based proposed joint time series modeling (JTSM) method is implemented to predict the front vehicle trajectories. The Next Generation Simulation (NGSIM) dataset on the US101, and I-80 is used to train and test the proposed model. The proposed model is compared with constant kalman filter (CKF), LSTM, multiple LSTM (MLLSTM). The proposed model shows significant improvement in terms of RMSE.

In [60], authors predict vehicles’ trajectory by using LSTM algorithm. Then, the predicted value is provided to the QL algorithm to figure out the optimal resource allocation policy for the nodes. The real world vehicle trajectory data is used in this research provided by Didi Chuxing, a ride sharing company. The dataset contains 600,000 traffic trajectory samples. The ultimate goal is to enhance the QoS for non-safety related services in MEC-based vehicular networks and the proposed model outperforms the other models.

In [61], the authors implemented RNN to predict resource availability pattern based on the position of vehicle and RSU at the time of prediction. The RSU burden is shared by the parked and slow vehicles. The performance of the proposed method is better than the heuristic and other ML based resource allocation algorithms.

In [62], for improved communication and network security, the authors proposed a cluster-enabled cooperative scheduling based on RL (CCSRL). The experimental results show that the CCSRL provide better PDR and PLR as compared to the Wu’s scheme and mobile service amount based link scheduling (MSA-LS). MATLAB is used as simulation software, Veins software is used to generate the vehicle distribution, and SUMO generates the road topology.

In [63], the authors divide the coverage area into clusters and then federated DRL obtains the global model for each cluster. The global model is implemented every time there is a new vehicle added in the cluster. The performance of the proposed algorithm is better than the other decentralized learning scheme.

In [64], the researchers introduced a wireless full-duplex (FD) technique to enhance spectrum efficiency and optimize system throughput. This approach employs V2V links within an FD cellular network. Managing resource allocation and power control becomes challenging due to the interference between self-interference (SI) at FD base stations (BS) and V2I links. To address this issue, they proposed a Dual Graph Coloring-based Interference Management (DGCIM) scheme. The two graph coloring schemes employed are the Grouping Graph Coloring with Recursive Largest First (GPGC-RLF) and Greedy Graph Coloring (GYGC) algorithms, respectively. Simulation results demonstrate that the proposed model satisfies high reliability and low latency constraints for V2V links.

In [65] a VNF-RACAG scheme is proposed to derive the optimal number of clusters. It reduced and minimizes the end-to-end delay in edge networks as well as allocated physical resources to users and minimize the number of transfers between clusters. The VNF-RACAG scheme uses a NFV stochastic model as well as geographic contexts and transfer history of the users in the chaining time optimization process. The proposed scheme is compared with the WiNE and PSwH schemes. The simulation are done in MATLAB and the results showed that end-to-end delay is minimized by using the proposed VNF-RACAG algorithm. The VNF-RACAG scheme implemented gradient descent to calculate the number of clusters to minimize the end-to-end delay. After this, a graph partitioning algorithm is performed to minimize the movement between the clusters.

In [66], the authors performed the resource assignment and video selection for vehicular devices by using the stochastic optimization. The aim is to achieve maximum QoE and secure a threshold duration of video at each receiver. A highway environment is simulated with different vehicular densities. The proposed method achieves better results with compared with two different baseline methods.

In [67], Two-Arch2 algorithm is adopted by introducing a hierarchical clustering idea to optimize 5G based IoV architecture. The proposed algorithm is compared with NSGA-III, Two-Arch2, CA-MOEA and MOEA/D. The experimental results show that the improved algorithm can obtain the better resource allocation scheme than the other algorithms in terms of stability, energy consumption and load balancing.

In [68], a multi-platform intelligent offloading and resource allocation algorithm is proposed to dynamically organize the computing resources. KNN algorithm selects the best option available out of cloud computing, mobile edge computing, or local computing platforms. In addition, when the task is offloaded to a desired server, RL is implemented to solve the resource allocation strategy. The state is defined as the MEC computing capacity, actions are offloading decision and computation resource allocation and the reward is the minimum total cost. The proposed joint optimisation is compared with full MEC and Full local techniques and it is concluded that the proposed scheme reduced the total system cost and optimize the overall system performance.

In [69], a centralized multi-agent learning is adopted for joint communication, computing and caching resource allocation problem. The RSUs are controlled by central the controller to assign resources in their coverage areas. The soft actor critic (SAC) algorithm updates policy in the central controller and share with edge agents to regulate their resource allocation strategies. The proposed algorithm increases the system’s performance as compared to other schemes.

In [70], the authors optimized the resource allocation and computation offloading by using MDP. The aim is to maximize the network operators’ revenue and for this DDPG algorithm is used to optimize the the wireless and computing resources dynamically. Compared with average and random algorithms methods, the suggested method is more effectual with little resources. This paper ignores the dynamics of task offloading as optimization.

In [71], for V2V communication, the authors proposed a RL based decentralized resource allocation mechanism. It is applied to both unicast and broadcast scenarios. V2V links are agents and based on the minimum interference select their spectrum and transmitted power for V2I and V2V links. The transmitted power is divided into three levels and the agents select them based on their state information.

In [72], the authors perform queue-length resource allocation. At controller level, the network safety flows are managed. The safety flows get more priority ratio based on the criticality, and the non-safety flows get less priority. The bandwidth allocation is the main fairness allocation criterion to obtain maximum rate for different application. The simulation environment use mininet-wifi for multiple RSUs and vehicles to communicate in V2v and V2I scenarios. The authors implemented LSTM, CNN, and DNN and compare their results. LSTM outperforms all models in terms of accuracy.

In [73], the authors used RNN to automate the resource management for the IoV network. The dataset used is called GWAT-13 Materna dataset with 12 attributes. available on Materna 13, an open source directory. It has three traces expanded over three months period with each trace having 850 VMs data on average. The prediction results of ML models are used by ASR-CUMS to decide the resource requirements and update the physical resources.

In [74], the authors proposed a DL based CSI estimation technique is proposed. It is assumed that the environment remains the same so the DNN model can learn the non-linear CSI relations with insignificant overhead. Moreover, a resource allocation based on dynamic eMMB and URLLC network slicing is implemented for vehicles. The proposed scheme attains 50% reduction in overhead and 12% higher threshold violations compared to an ideal case with perfect CSI knowledge.

In [75] the authors maximize the overall system throughput by implementing the weighted minimum mean square error (WMMSE) algorithm for power allocation. Simulation results show that the DNN algorithm improves the approximation of the iterative WMMSE algorithm achieves the computational overhead reduction.

ML techniques have proven to be very effective in VANET environment yet face some issues. For instance, the SVM faces issues such as the choice of the optimized kernel and models complexity, for KNN, the optimum value of k changes from one dataset to another which makes it a time-consuming and complex method to find the optimum value of k, RF focuses on the construction of several DTs which is time consuming and in vehicular network for time-sensitive applications this method is not recommended. The RL is the most commonly applied technique in IoV. However, it is noted that in vehicular networks it suffers from the issues of dimensionality makes it harder to implement RL on physical systems.

4.2. AI for Resource Allocation in UAV Network

The mobility as well as LOS links provided by UAVs make them a promising alternate to the fixed BS for the wireless communication networks. Similarly, AI has gained a lot of interest in wireless communication as it has the ability to learn from data and also learn from the environment and take decision herself. That is why, the research community is working on integrating intelligence in UAVs networks using different AI algorithms. In this section, we cover the potential applications of AI in UAV based wireless networks. These networks when extended, can work as a basic but fundamental ground for UAV based vehicular networks.

In [76], the authors handle the radio resource management in cellular based UAVs by combining resource block (RB) allocation and beam-forming design. To realize this, at terrestrial BSs an outer MDP characterizes the dynamic RB possession environment while the inner MDP is used to track the time-varying feature of B2D small-scale fading. A DRL-aided hybrid D3QN-TD3 algorithm is proposed to maximize the reduction of the UAV user equipment’s (DUE’s) ergodic outage duration(EOD) for outer and inner MDPs. In discrete domain, RB coordination is achieved using the deep double duelling Q network (D3QN). The beam forming in continuous time is achieved using twin delayed deep deterministic policy gradient (TD3).

In [77], the authors combined the features of FL and MARL, for location deployment and resource allocation and proposed a multi-agent collaborative environment learning (MACEL) algorithm to optimize the network utility by ensuring the user coverage. The UAVs act as agents which are trained using the Deep-Q-Network (DQN) algorithm. The FL model does the aggregation of local gradients and weights at cloud server to make a global model. The final results demonstrate the superiority of the proposed model in adapting to the ever changing environment.

In [78], to deploy UAVs-BS, the authors proposed regression neural networks (GRNN) and RF based models. For this purpose, the system implements the load prediction algorithm (LPA) to predict maco-cell congestion based on load history generated by the mobile network. Then, to calculate the accurate number of UAV-BS required, the UAV-BSs clustering and positioning algorithm (UCPA), is implemented. The results showed that the RF model achieved the accuracy of over 85%.

In [79] the authors integrate DNN in UAV at MEC for communication resource allocation, model optimization and UAV trajectory control to ensure the service latency minimization while ensuring the learning accuracy and energy consumption requirements.

In [80], the authors proposed the optimal UAV-RIS deployment mechanism based on dueling DQN to facilitate multiple DL users with the help of the MC-PD-NOMA scheme. The total transmit power is minimized. Moreover, UAVs’s trajectories are optimized jointly along with sub-carrier allocations, phase shifts at each RIS. The results show that more total transmit power is achieved using the proposed model when compared with other baseline methods.

In [81], the authors maximize the sum rate of the UAV-enabled multi-cast network by jointly designing the UAV movement, re-configurable intelligent surface (RIS) reflection matrix, and beam-forming design from the UAV to users based on a multi-pass deep Q Network (BT-MP-DQN). In the proposed model, the UAV is the agent and beam-forming control and trajectory design are considered system actions. The movement of the UAV is discrete action whereas the beam-forming design is continuous action. It is shown that the achievable rate is maximized as well as the minimum rate of multi-cast group users is achieved.

In [82], the multi agent DRL model is applied for resource management on cellular and IoT networks by integrating UAV in the system. First, the resource management related challenges in UAVs-BS based IoT networks are pointed out. The K-means algorithm and round robin scheduling algorithms are used for clustering and service request queue respectively. The accuracy, RMSE and testing time(s), are used as metrics to compare the proposed method with previous works. The proposed system achieves 94% resource management accuracy, RMSE score of 2.40% and testing times(s) levels of 1.05s. In terms of precision and recall estimation (as 88.05%, 93.04% and 92%, 90.01% for two of the clusters), the proposed model performs better than the Q-learning approach.

In [83], the paper deploys mobile UAVs as BSs. The aim of this research is to achieve data collection and energy transmission maximization as well as minimum UAV energy consumption. This could be achieved by UAV transmission power, hovering time slots, and flight decisions optimization. This multi-objective optimization problem is solved by implementing the MJDDPG dynamic resource allocation algorithm. In this study, the experimental results of the proposed model are not compared with any other models to proof the authenticity of the model.

In [84],the authors proposed a dynamic information exchange management schemes in a UAV network. First scheme shares the channel exchange information. Then, the dynamic time slot model is implemented in second scheme where priority based time slot sharing is performed. For both the schemes, to solve the problem of dynamic management, a DQN network coupled with LSTM is proposed. The LSTM network ensure the fast convergence of the DQN model.

In [85], the authors proposed the MDP and then an Actor-Critic based RL technique (A2C) solution in UAVs to offload the computational tasks of the ground users and achieve the minimum mission time. This scheme achieves the terrestrial users offloading through UAVs and in turn the offloading of UAVs through MEC server.

In [86], the authors probe the offloading of the task in UAV via MEC servers to minimize latency and the energy of the UAVs. Each UAV is associated with its corresponding task by keeping check of the available energy along with the optimal MEC server selection two Q-learning models are proposed. In [87], the Multi Agent RL (MARL) model is proposed for resource allocation for multiple UAVs network. The authors perform the sub-channel and user selection and allocate power to each user by considering parameters such as the signal-to-noise ratio, LOS and Non-LOS conditions. The proposed MARL algorithm achieves a good trade-off between the information exchange overhead and the system performance.

In [88], the authors consider a swarm of HAPS for communication and aim at comparing the RL and SI algorithms. The SI algorithm performs better as compared to RL as it converges quickly by maintaining the stable user coverage profile. On the other hand, the RL algorithm shows some convergence dips. However, the overall coverage rates are higher for RL.

In [89], the authors proposed FL based resource allocation method for joint optimization of the UAV position and local accuracy of FL model and user computation and communication resource. After formulating the joint problem, these three problems are solved by divided them into separate sub problems. The proposed algorithm is compared with the fixed altitude UAV-assisted FL ratio and performs better learning and reduces the system overall energy consumption.

In [90], authors perform joint power allocation and scheduling for a UAV swarm network. In the network one drone is selected as a leader and all other drones are make a group of drones following the leader. Every group transmits the update of its local FL model to the leader drone so it could combine all the local parameters for global parameter update to the global model. While the drones exchange updates, the wireless transmissions gets affected by many internal and external losses and interference. The impact of these factors on FL model is also considered. Moreover, experimental results approve the FL convergence analysis and the combined design strategy minimize the convergence time by 35% in comparison to the other state of the art designs.

In [91], the authors perform task offloading to manage the resources by ensuring the energy and latency minimization for high-altitude balloon networks. The user association is decided using SVM-based FL model. The FL provided the privacy as the data is not being shared across the network.

To summarize, the ML methods have been recently (the last two years) widely exploited in dynamic resource allocation in UAV and fixed BS communication. The supervised learning, RL and FL are studied for fast making decision and response, resource allocation, task offloading, joint power allocation in high-dynamic UAV networks. However, the implementation of AI models is limited to the basic and much simpler state of the art models. The researchers ignored the limitation of these models while implementing them to solve the problem in hand.

4.3. AI for Resource Allocation in UAV-IoV Network

The heterogeneity of vehicular networks, their high dynamic nature with fast moving have made the vehicular network more complex and demand new requirements for networking algorithms that could meet the stringent network control and resource allocation demands such as efficient spectrum sharing, transmission power maximization and computational resource management to minimize the energy requirements. UAV-IoV networks are three dimensional and contrast to terrestrial networks, the (UAV) base station itself is moving. Therefore,traditional optimization techniques are unable to capture complex patterns. In this section we review AI/ML based resource allocation in UAV-IoV networks. A brief summary of these papers is provided in Table 3.

Resource management in UAV-IoV is divided into radio resource allocation and computation resources management. The radio resource allocation is further divided into spectrum and channel access optimization. The main goal of radio resource management is to limit channel interference, power usage and network congestion. The computation resource management includes service, task and traffic offloading in MEC where the edge cloud nodes located in BSs and/or UAVs. This decentralization of the system generates faster response times as compared to the central deployments.

In [92], the authors focus on bandwidth allocation, location control deployment and trajectory of UAVs for communication capacity maximization to enable the UAV to process more data at edge computing. They propose an actor-critic mixing network (AC-Mix) and multi-attentive DDPG (MA2DDPG) network. The AC-Mix is the combination of Qmix (relies on Q function and don not deal with continuous values) mixed with the actor–critic framework. The critic in AC-Mix has a individual value network and a mixing network maintained by a virtual agent. Each agent inputs a state–action pair to GRU network which in turn outputs the global action value. TensorFlow 1.15.0 is used to design all algorithms. For comparison, user fairness, and load balance metrics are presented. MA2DDPG convergence velocity is 30.0% and 63.3%, better than the MADDPG algorithm.

In [93], the authors proposed a mechanism for energy harvesting by UAV from BS and vehicles using wireless power transfer (WPT) and simultaneous wireless information and power transfer (SWIPT) techniques, respectively. Maximum data offloading to the UAV is the main goal of this research. For optimized resource management and UAV velocity a DRL-based resource allocation and speed optimization (DRL-RASO) model is adopted. The proposed model achieves much higher offloading rates data offloaded to the UAV than the DRL-FTPA and Dueling-DQN. The authors named a simple DRL based DDPG algorithm DRL-RASO and no modification is suggested in the basic algorithm. The reported difference between DRL-RASO and DRL-FTPA is 5.79%.

In [94], an UAV based vehicular network is built to deal with caching and computing problem in addition to BS. The energy minimization is achieved by combining the cache refreshing optimization, computation unloading and status age updates. The online decision making is performed using DDPG. The BS decides if the cache needs to be refreshed, task has to be executed and what should be the bandwidth distribution. The total energy consumption is the reward function. The learning performance of proposed model is compared with traditional DDPG algorithm in terms of convergence rate. Then, the energy consumption for four bench marks namely random refreshing, random offloading, popular refreshing and equal bandwidth is calculated. It is claimed that proposed model outperforms DDPG in terms of system energy consumption and computational capabilities of UAV MEC server. But the authors do not report any results obtained using DDPG model.

In [95], authors formulated a combined auction-coalition building method to assign UAV coalitions to various IoV groups. The coalition formation game maximizes the combined profits of all UAVs. The auction-coalition creation method is suggested to achieve the UAV coalition stability. The reported simulation results show that the FL communication latency is decreased.

In [96], the authors proposed a FL based approach for the development of IoV based applications. The authors used the Gale-Shapley algorithm to match the lowest cost UAV to each sub-region. The simulation results show that the lowest marginal cost of node coverage for a UAV is assigned to each sub-region for task completion. The UAV energy harvesting is suggested as a future work for UAVs to continue their flight without any need of going back to their charging stations.

The authors in [97], proposed a secure bandwidth allocation scheme based on the game theory on the IoV assisted by UAVs. Also, for delay reduction and increased data privacy the proposed blockchain-based system introduces an emerging consensus mechanism. Furthermore, to allocate the limited safe bandwidth, based on the real-time feedback of each UAV an optimal decision search algorithm based on gradient descent to achieve Stackelberg equilibrium is proposed. The performance of the proposed scheme is compared with the many-to-one scheme, the maximum signal-strength-indicator (max-RSSI) scheme, the maximum signal-to-interference-plus-noise-ratio (max-SINR) scheme, the Auction-based UAV Swarm Many-to-Many scheme (AMMA) and UE-Optimal Many-to-One Matching scheme (UMOA). The proposed scheme achieves better throughput of about 95% as compared to other models but the authors do not provide any data to strengthen their claim about privacy and secured bandwidth allocation.

In [98], the vehicular task offloading optimization problem is dealt by jointly considering the resource allocation, and the security assurance. Then, this problem is divided into two separate problems and finally iterative algorithm called LBTO is proposed. LBTO decides if a certain MEC is selected depending on the load of the MECs and uses the Lagrangian dual decomposition for optimized offloading ratio and the computation resource. Task to be processed is selected based on the size of the task, computing resources required to execute a task, task’s allowed latency and ratio of the offloaded task at UAV MEC or locally to total task. For security assurance wiretap coding is used to protect the vehicle’s MEC. The proposed algorithm provides better task offloading ratio and delay than the other algorithms. However, this research considers that the UAVs are fixed and that is why completely ignores their energy consumption during mobility in the objective function.

In [99], the authors proposed a system wise computation capacity maximization of UAV based MEC for a group of vehicles equipped with WPT. They proposed a second-order convex approximation based method to solve a sequence of sub-problems. The platooning vehicles and UAVs are coupled.The drawback of this technique is that it does not use the AI/ML to solve the non-convex problem to give a comparative analysis.

In [100] the authors proposed a MDP based model for UAVs to take trajectory decisions to maximize the communication coverage area. The Actor-Critic algorithm learns the environment. The problem under consideration is mixed integer non-linear and non-convex problem. The DRL model learns the underlying non-linearity and non-convexity optimally. The model inputs are residual energy of each UAV, number and position of vehicles, positions of UAVs with respect to ground level etc. The UAV travelling distance is taken as the action. The penalty on the network incurred if UAV does not provide coverage to a vehicle, the deployment of a new UAV, the remaining energy of each UAV and the UAV goes outside the designated path. The proposed model is claimed to be compared with random dispatch, fixed dispatch and hovering models for metrics such as energy consumption, time versus maximum performance and vehicular density versus average convergence. But, there is no comparison of the proposed model provided with other state of the art models. Overall 40% of improvement is reported in converge area of the UAVs using the proposed model.

In [101], the authors combined auction-integration (AI) formations to integrate UAV into the groups of IoV elements with target of achieving the total revenue maximization of a single UAV. The proposed method is compared with the merge-and-split algorithm to find the UAV coalitions formation to the IoV groups, and random UAV coalition partitioning with second-price auction.

In [102], the authors proposed a FL-based technique to make sure security for IoV applications. The accurate reporting of UAV types is achieved by applying MD contracts with its identity qualities and using many sources of variability. The least costly UAV for each sub-region is found using the Gale-Shapley method. The results show the success of the proposed matching scheme.

In [103], the authors proposed a model free Q network to select the best UAV advice with least stalling time. The results show that the proposed system provide a high-quality user experience. This paper is limited in its scope as the limitation of UAV server capacity, speed and vehicle speed are not considered.

The research in the area of UAV based IoV networks is in its infancy. Many issues in the research are not addressed and ignored. The main focus of research is on system energy minimization, energy harvesting, caching and bandwidth allocation. Although in achieving these goals the researches ignore the latency requirements, changes in UAV height, the variations in vehicular node density. To manage resources, reinforcement learning is the most rampantly used AI technique. However, the studies used the state of the art RL models to minimize the system energy and bandwidth allocation etc. The lack of dataset availability in UAV based IoV networks makes it very hard to use ML and DL models for resource management and to explore the proven potential of these AI models. Furthermore, DL techniques have not been investigated because of the restricted power and processing resources available in UAV. In addition to this, the most important issue of security and data privacy are not considered in the research. The unencrypted and unauthenticated channels are used for UAV based communication which make them vulnerable to cyber attacks. Federated Learning can play a pivotal role in providing the security and privacy by training the ML model on data without transferring it to cloud server.

5. AI based Routing in UAV and IoV Networks

In this section, we cover the research contribution in the area of routing protocols proposed and designed for IoV, UAV as well as UAV based IoV networks using AI.

5.1. AI for Routing in IoV Network

In opportunistic networks, node selection poses a crucial challenge due to nodes lacking information about the state of other nodes. Furthermore, in IoV, traditional routing protocols fall short of achieving optimal performance. To address these challenges, the authors introduced a machine learning-based multi-copy routing algorithm called iPRoPHET (Improved PRoPHET) in their work [104]. iPRoPHET leverages dynamically changing contextual information of nodes and the delivery probability of PRoPHET for effective message transfer. Employing a random forest, iPRoPHET classifies nodes as reliable or non-reliable forwarders based on contextual information provided during each routing decision. The training data is derived from simulations. The proposed model undergoes evaluation using metrics such as delivery probability, hop count, overhead ratio, and latency, demonstrating performance on par with similar multi-copy routing algorithms.

In [105], the authors proposed a stochastic chaos based adaptive routing with prediction (SCARP) to predict traffic flow using DL networks to suggest a node discovery routing principle. The connectivity loss and delay are minimized, and guarantee a secure data transmission between vehicles. In this research, region of Puducherry U.T, India is selected for traffic data collection. The simulation software namely SUMO and OMNET++ are used to create traffic and network scenarios respectively. The metrics namely accuracy, PDR, delay and sensitivity are used to compare the proposed method with existing state of the art routing algorithms.

In [106] a Q- learning based geographical routing scheme with intersection-based V2X routing (IV2XQ) is introduced. First the best road segment at the intersections for routing is selected using Q- learning at intersections. Then best relay node selection is performed using greedy routing strategy. The central server is the agent which uses the historical traffic data to select the optimal path. The environment which is our entire network, rewards the agent if it takes the right action and choose the correct road segment to forward a data packet. It is reported that the proposed scheme increased the PDR and considerably controlled the network congestion.

In [107], a RL based routing (best two hops) and context aware edge node selection scheme to forward packets scheme (CEPF) is proposed. Both unicast and broadcast communications are supported by CEPF. This routing protocol reduces the forward nodes and increases the resource efficiency. The decentralized fuzzy logic is implemented to select the edge nodes based on vehicular velocity, mobile nodes traveling in same direction, and communication link condition. The edge node is the vehicle with highest node score. For rout discovery operation RL is used in which each packet is the agent and the action agent takes is the selection of the next-hop node.

In [108], authors integrate the RL and fuzzy logic and proposed a reinforcement routing protocol named RRPV. A DynaQ technique is implemented on the fuzzy logic to build the model. The link stability and connection quality are two inputs for fuzzy logic-based system. The fuzzy system determines the link quality and this result is fed as the state transition probability in MDP. In the RL process, vehicles are agents and each agent has two states, namely F to send a packet and D to deliver packets to adjacent vehicle. Hello message delivery to the neighboring vehicle is the action of the vehicles. Moreover, the link condition and the euclidean distance of two neighboring vehicles define reward function. The routing table of each vehicle consists of Q-values and V-values.

In [109], the authors introduced traffic-aware routing protocol based on Q-learning (QTAR). It contains two routing algorithms based on Q learning to send data packets between vehicles (V2V Q learning) and between RSUs (R2R Q learning). Vehicles broadcast HelloV2V messages containing their velocity and location related information and the RSUs exchange HelloR2R messages with each other.

In [110] a routing protocol named RLRC for clustered network based on the K-harmonic means (KHM) clustering to assign vehicles to different cluster and RL to exchange data between two cluster heads (CHs) is proposed. In RLRC, hello message is used to share the vehicle velocity and position with neighbouring vehicles. In this process, each node behaves as an agent, state set is defined as neighboring CHs, and a next-hop CH selection is the action taken by the agents. The reward function is based on the link quality parameter.

In [111], the authors proposed a Q-learning based routing scheme called reliable self-adaptive routing scheme (RSAR). The vehicles are agents. The action is a beacon message including vehicle speed, location and Q value, sent to the next vehicle. Moreover, the decentralized learning process is adopted with the number of hops, bandwidth, and link reliability as learning parameters. The RSAR finds the most fit relay vehicular node and solve network segmentation problem.

In [112], the authors proposed a routing technique which allows the central server and vehicles and RSUs network nodes to access the updated traffic information based on intersection- based Q-learning (IRQ). The global traffic view is obtained by IRQ as well as by the central server to form a routing solution. Here, the central server behaves as an agent. The central server is also responsible for network congestion control in the routes. The IRQ uses a greedy routing approach in V2V and V2R routing decision where for V2V routing, the vehicle closest to the target is chosen to forward the data packet and in V2I scenario, the RSU located at intersection delivers data packets to the corresponding road section. If there is no vehicle available, RSU waits and holds the packet until it finds a vehicle to rely information.

In [113], a routing protocol based on Q-learning and fuzzy-based hierarchical (QFHR) is proposed. The routing algorithm is capable of doing traffic pattern recognition, routing between intersections, and at road sections. The RSUs are equipped with Q learning to find multiple routing paths. Moreover, the vehicles use the greedy technique to find the most fitted path in each road section. A fuzzy solution works as the alternate for route recovery in case the main algorithm fails and selects the next node.

To summarize, The integration of routing protocols with AI is mostly based on RL models. Although the research results show that RL is quite successful in taking the routing decisions yet it comes with certain limitations. The large state and action sets effect the convergence rate and increase delay in the routing process. Therefore, in future, the researchers must focus filtering state and action spaces based on the required criteria. Most of the research is using Q table and as state and action set increase, the dimension of the Q table increases which requires more memory to store it and in turn increase the latency demand of the system. So, the management of the Q-table should be considered in future research.

5.2. AI for Routing in UAV Network

In [114], the management of multiple cooperative UAVs is addressed. The routing problem in this system is divided into two stages: initial planning and routing solution. In the initial planning stage, regions to be visited are grouped into clusters based on the distance criterion (FCM algorithm), with each cluster assigned to a UAV. The route solving stage determines the best route for each agent, considering the clusters from the initial planning stage and a variant of the Orienteering Problem. The transformer deep learning architecture is employed to solve the Orienteering Problem with shared regions, coupled with a DRL framework. The proposed model is evaluated using multiple OP-MP-TN datasets under various environmental conditions, demonstrating its superiority over state-of-the-art models in cooperative and non-cooperative scenarios.

In the work presented by [115], UAV location optimization and relay path planning are jointly achieved using a graph neural network based on Reinforcement Learning (RGNN) algorithm. The proposed model exhibits significantly lower time complexity compared to traditional optimization methods. The Location GNN (LGNN) optimizes UAV locations, and the RGNN selects the optimal relay path based on information provided by the LGNN. The method outperforms brute-force approaches in terms of time complexity.

The study conducted by [116] introduces a novel routing protocol based on ant behavior routing, enhancing end-to-end security through data encryption using the Pheromone update process. Experiments conducted in Network Simulator-2 show that AntHocNet performs well in terms of packet drop rate, throughput, and bandwidth utilization, achieving significant optimizations compared to other routing techniques.

In [117], collision-free routing policies for UAVs are designed using multi-agent Reinforcement Learning (RL). The authors propose a multi-resolution, multi-agent, mean-field RL algorithm named 3M-RL for UAV flight planning. Each UAV makes decisions based on local observations without direct communication with other UAVs. The routing policy is trained using an Actor-Critic neural network with multi-resolution observations, demonstrating effectiveness in various complex scenarios in both 2D and 3D space.

The predictive ad-hoc routing combined with Reinforcement Learning and trajectory knowledge protocol (PARRoT) is introduced by [118]. This protocol aims to achieve lower latency and high robustness by predicting future node positions and sharing information with adjacent nodes. PARRoT separates networking from path planning, enhancing overall system efficiency.

In [119], fuzzy logic is employed to identify adjacent nodes in real-time, while Reinforcement Learning is used to reduce the number of hops in a routing algorithm named Fuzzy Logic Reinforcement Learning-based Routing Algorithm (FLRLR). FLRLR reduces the average number of simulation hops and ensures higher link connectivity, showing comparative advantages over the ant colony optimization (ACO) algorithm.

The adaptive and reliable routing protocol called ARdeep is proposed in [120]. This deep learning-based protocol autonomously distinguishes network variations using a Markov Decision Process (MDP) model. Factors such as Packet Error Rate (PER), link status, connection time, and nodes’ remaining energy influence routing decisions, and ARdeep outperforms QGeo in experimental comparisons.

The study by [121] introduces the Q-Learning-based Fuzzy Logic for Multi-Objective Routing Algorithm in Flying Ad-Hoc Networks (QLFLMOR). QLFLMOR uses Q-learning and fuzzy logic in UAVs to select the optimal routing path based on link and path-level parameters. Experimental results demonstrate that QLFLMOR achieves lower hop count and energy consumption compared to other routing algorithms.

To summarize, most routing protocols assumed that the UAV networks are fully connected but it is not the case in reality and the broken links cause failure of routing protocols. In the conventional routing protocols, the node mobility is designed for 2D spaces whereas the UAV moves in 3D space and in most of the researches UAV mobility is converted to 2D scenarios. In future, RL-based method should be modified and implemented to accommodate nonlinear UAV movement and implement light-weight routing protocols for small sized UAVs that are mostly in use in real world.

5.3. AI for Routing in UAV-IoV Network

Efficient data dissemination among vehicles and optimization of multi-hop path and relay selection is a complex task in IoV. The network latency and reliability considering the increasing vehicle density in future networks is very crucial in making routing decisions. In this regard, the UAV based routing in IoV is relatively new area and not properly explored yet. The summary of AI/ML UAV-IoV routing protocols is provided in Table 4.

In [122], traffic congestion problem is dealt using a Q-learning based load balancing routing (Q-LBR). It estimated the network load using low-overhead technique, Q-learning based load balancing and finally implements a reward control function for Q-learning convergence. The simulation results show that Q-LBR achieves better PDR, network utilization and latency with 8%, 28% and 30% improvement, respectively.

In [123], VRU algorithm is proposed to route data for vehicles using UAVs and intra-UAV data routing. The NS-2.35 simulator under Linux Ubuntu 12.04 is used to generate urban scenario to evaluate the VRU routing performance. VanetMobiSim and MobiSim generates mobility of vehicles and UAVs, respectively. results show that VRU improved the PDR by 16%, detection ratio by 7% decreases end-to-end delay by an average of 13% and overhead by 40% compared to other reviewed routing protocol.

In [124], an adaptive UAV assisted geographic routing with Q-Learning (QAGR) is proposed. Routing is done using two different components namely aerial and base components. UAVs uses the combination of fuzzy-logic and depth-first-search (DFS) algorithm to find the global routing path. This routing path information is transferred to the requesting vehicle at ground. A fix-sized Q-table is maintained at vehicle which is updated with the global routing path.

In [125], the relay selection problem is addressed for UAV based VANET. The authors calculate the communication interruption probabilities, energy consumption and delay consumption. Q-learning is being used for relay selection. The simulation results exhibit the performance improvement in terms of delivery ratio, delivery latency, and signaling overhead as compared to five other schemes, namely, STPTC-VANETs, STP-A2G VANETs, TC-A2G VANETs, CORF-A2G VANETs, and UMTAP-A2G VANETs.

In [126], a routing solution employing a flooding technique is adopted to enhance data delivery reliability and ensure robust paths for vehicles through ad hoc collaboration with UAVs. The evaluation of the proposed model is conducted using the NS-2 simulation tool, with vehicle mobility simulated using the Simulation of Urban Mobility (SUMO) software. For the UAVs, a random walk mobility pattern is applied to 10 UAVs, with a maximum altitude of 300 meters during flight. Notably, this paper does not incorporate machine learning techniques and relies solely on conventional routing methods. Evaluation metrics encompass PDR, end-to-end delay (EED), overhead, and average number of hops. It’s worth mentioning that the authors do not compare their proposed technique with other routing methods. Nevertheless, the suggested protocol demonstrates a PDR that is 90% superior to RBVT-R and OLSR.

In the research in UAV based IoV routing protocols the control messages are exchanged periodically and the flooding routing messages consume excessive bandwidth resulting in high overhead. To address this issue, broadcast control mechanisms offer the flexibility to dynamically adjust the broadcast time interval of control messages. Additionally, by selectively filtering nodes based on factors such as link quality and movement information, it becomes possible to regulate the flooding process of routing messages. In addition to this, most of the routing protocols designed using RL model are considering QoS requirements. The researcher should focus on including other objectives such as link quality and delay and include them in reward function.

6. Challenges and Open Issues with Future Directions

This section discusses limitations of ML/AI algorithms and the simulation software used to train ML algorithms.

6.1. Major Limitations and Challenges of AI/ML

Numerous AI/ML based IoV resource management and routing algorithms are proposed in the last five years. Although AI/ML approaches are data driven and can produce fairly accurate solutions in most scenarios, there a number of limitations with these approaches as well. Here are some of the key limitations associated with ML and DL:

• Application Specific ML based models are application specific. It means if we train a DL model on certain vehicular application such as network data congestion prediction or classification, the model would be able to provide high quality results in same area, but it would not predict or classify the vehicular traffic congestion in a different area.
• Noisy and Incomplete Data: ML agents may have to deal with noisy and incomplete data, which can impact their learning and decision-making processes.

Furthermore, the ML and DL models are data dependent and their success lies in the availability of data. Especially, most DL algorithms need adequate data. However, in UAV based vehicular communication the historical data for many time sensitive tasks such as resource management, mobility prediction, routing decisions is not readily available. Therefore, there is a need to pay more attention to provide open source and highly reliable data related to UAV vehicles. It means we need to have a mechanism to produce and estimate the accurate data-set size to train and test the ML and DL algorithms.

This is the reason that most of the research used RL for offloading the task and routing. RL and it’s variants are proved to be very effective in handling the non-convex problems such as task management, energy efficiency and routing. However, in RL, the agent’s action depends on the reward or penalty it gets which in turn could come handy in training the model. Specifically, in routing related problems, as the state and action sets become large, it affects the convergence and increase the latency and increase the dimentionality of the Q tables which results in more memory consumption.

Recently, FL has emerged as a reliable solution a because it provides data privacy and reduces the time complexity. Unfortunately, the FL could be subject to backdoor attacks where the integrity of the model can be threaten by injecting the poisoning data/model. In addition to this, convergence of the FL model is another issue as convergence is problem specific such as loss function convexity and the number of times a model has to be updated. So, in case there is no sufficient data, the model will fail to give accurate results. Moreover, the UAV-IoV network is heterogeneous consists of drones with different sizes and specifications, and the vehicles with dynamic computational and processing capabilities and varying GPUs. When FL is implemented in such a heterogeneous network, all drones and vehicles will have different response times. In the FL, the model updates at each communication round and the delays introduced at this time causes the slow convergence of the model.

To conclude the discussion, although AI/ML has some drawbacks, yet the ML, RL and FL based solutions for UAV based IoV networks have produced better results for resource management and routing when compared with other techniques to handle the non-convex vehicular and UAV based network problems.

6.2. Testbed and Datasets

During this survey paper, it is noted that most of the research is done using network simulator version (NS2), NS3, MATLAB, SUMO and OMNET++ Vissim, AVISPA and RaspberryPi to create vehicular traffic scenarios and network analysis. The accuracy of these tools is questionable and the performance evaluation of these methods is not comparable with the real world scenarios. The real strengths and weaknesses of these algorithms can be estimated when they are implemented in real world scenarios. As mentioned in the section above, the research community should focus on providing the reliable wireless and cellular data to train the AI models on real world data and scenarios.

7. Conclusion

UAV based aerial networks add a third spacial dimension to wireless networks. UAVs are also special because they can serve as mobile base stations. Their mobility will add more versatility as well as more complexity to vehicular networks. Therefore, AI/ML will play a much important role in UAV based IoV.

In this paper, we present an extensive comparative study on the application of AI algorithms in the UAV based IoV paradigms. We have reviewed resource management and routing techniques in UAV based IoV using different AI models.

AI-based algorithms for UAV-IoV applications have enhanced the system performance as compared to the traditional algorithms. When multiple AI algorithms are combined together to utilised their best features, it brought out an almost optimum solution for resource management and routing in UAV-IoV networks. Also, the overall system throughput improved with energy minimization and higher latency. However, the high computational resource utilization by AI algorithms in a dynamic vehicular and UAV environment is the major issue. For this reason, the emerging MEC and VEC architectures often attempt to offload the computing tasks to a RSUs or to Fog nodes made it possible to have these resources out of the vehicles and UAVs to share computation burden of AI with edge computation servers. IoV has already shown a great success in optimizing the network operations. nevertheless, the inclusion of UAVs has made the network more dynamic and mobile. AI/ML has the potential to handle such a dynamic and mobile network.

Based on the papers we surveyed, it can be incurred that in future UAV based IoV, empowered by AI/ML can autonomously operate without any dependence on third party to perform configuration and optimization.