Optimization Method of Energy Saving Strategy for Networked Driving in Road Sections with Frequent Traffic Flow Changes

1. Introduction

Under the background of global energy crisis and increasingly severe environmental problems, energy conservation and emission reduction in the field of transportation has become a key link to achieve sustainable development. Especially in the sections with frequent traffic flow [1], such as the intersections of urban roads, suburban junctions and other areas, the driving state of vehicles changes frequently, and the energy consumption rises sharply. It is urgent to explore the energy-saving driving strategy in this scenario.

The optimization method of energy-saving driving of connected vehicles is a hot research content in recent years. Literature [2] designed a vehicle flow dynamic allocation system based on vehicle road collaboration based on new technologies such as c-v2x and edge computing, which reduced the operation delay in the weaving bottleneck area and fuel consumption. Wang[3] described a collaborative ecological driving (CED) system for the signal corridor, developed a longitudinal control model based on its role and distance from the intersection for artificial vehicles and different Cavs, and achieved the maximization of vehicle energy efficiency. Xu[4] for the ecological driving scenario of connected vehicles at urban signalized intersections, Legendre pseudospectral method is used to solve the optimal speed curve, and the system output is close to the global optimal solution, highlighting the excellent energy-saving potential of vehicles. Sun Xiaochong [5] proposed an ecological driving strategy for intelligent Internet connected vehicles considering secondary queuing. By improving IDM car following model and combining Internet connected information, an ecological driving strategy was constructed. Litiezhu [6] proposed a rule-based two-stage energy-saving driving optimization method. Dong[7] proposed an enhanced ecological control method (EEAC) strategy, designed the EEAC strategy using a hierarchical framework, and verified the energy efficiency improvement of the strategy. Literature [8,9] established an energy-saving control algorithm based on CACC. At this stage, the energy-saving optimization methods mainly focus on connected driving vehicles, and the research on mixed traffic flow is less.

In recent years, reinforcement learning algorithm to solve the problem of path optimization and energy-saving driving strategy in complex traffic environment has gradually increased. Chen[10] proposed a deep reinforcement learning method for end-to-end automatic driving, which achieved good results in complex urban driving scenes by introducing the sequential potential environment model. Saxena[11] successfully introduced a driving benchmark in dense traffic by modeling with deep reinforcement learning and implementing a continuous control strategy for the action space of autonomous vehicles. Feng Yao [12] proposed a lane changing trajectory planning method for intelligent connected vehicles based on deep reinforcement learning, which improved the safety and efficiency of intelligent connected vehicles and reduced fuel consumption. Jiang Han [13] established the CAV driving energy consumption model by constructing the state space of the reinforcement learning framework, trained the algorithm model with the deep certainty strategy, and verified the feasibility of the algorithm. Guo[14] designed a deep deterministic strategy gradient (DDPG) algorithm, which enables controlled vehicles to learn a perfect longitudinal fuel saving strategy and perform appropriate lane changing operations at an appropriate time to avoid Lane congestion. Zeng Xiaoqing [15] optimized the energy-saving driving model through reinforcement learning algorithm Q-learning. Some reinforcement learning algorithms have high complexity, large amount of calculation, and the Q value is overestimated.

To sum up, based on the disadvantages of the traditional energy-saving strategy and the existing reinforcement learning algorithm does not fit, the application framework of energy-saving driving strategy on the road with frequent traffic flow changes is constructed for the mixed traffic flow. A hybrid traffic flow driving model for the Internet of vehicles is established, and the d3qn deep reinforcement learning algorithm is used to optimize the energy-saving driving model. Finally, the effectiveness of the energy-saving driving strategy is verified by simulation, and the fluctuation of traffic flow in frequency varying sections is greatly reduced.

2. Analysis of Mixed Traffic Flow Driving Scenarios

2.1. Analysis of Energy Saving Driving Strategies

In the upstream functional area of a signal controlled intersection, vehicles need to make lane selection in advance to match downstream turning requirements. In this dynamic process, the lane change decision of manually driven vehicles highly depends on the individual cognitive ability, risk preference, and environmental perception level of the driver. This highly subjective and heterogeneous decision-making mechanism is prone to frequent and non collaborative lane changing behaviors, significantly exacerbating traffic flow disturbances and potentially leading to a decrease in local traffic efficiency or even potential conflict risks. In contrast, networked autonomous vehicles operate in a vehicle road collaborative environment. It relies on the vehicle networking platform to transmit high-precision, low latency real-time traffic information, including but not limited to signal phase and timing, road geometry topology, surrounding vehicle status, and global traffic situation. Based on these multi-source heterogeneous data, the collaborative decision-making module of CAVs can generate globally or locally optimal lane changing strategies, thereby achieving collaborative optimization of path planning and lane keeping/changing.

To optimize driving strategies for this mixed traffic flow, an energy-saving driving model for mixed traffic flow needs to be constructed. Firstly, the lane following characteristics of the mixed traffic fleet need to be identified from the perspective of traffic flow. This is because the lane following behavior between different vehicle models in mixed traffic flow can affect the overall operation status of the fleet. Therefore, accurately characterizing the lane following behavior within mixed traffic flow is crucial. Secondly, it is necessary to control the energy consumption of each vehicle model in the mixed traffic flow. In order to achieve unified measurement of energy consumption evaluation, it is necessary to establish a parameter unified mixed traffic flow energy consumption model based on the powertrain characteristics and driving features of different vehicle models. This model will provide a consistent benchmark for the quantitative evaluation of the effectiveness of energy-saving driving strategies. Finally, simulate and verify the energy-saving driving strategy. Figure 1 is a decision-making flowchart for networked autonomous vehicles.

2.2. Energy Saving Framework for Connected Driving

The vehicle networking energy-saving driving application framework constructed in this article includes four parts: roadside detection unit, roadside indication unit, vehicle networking information platform, hybrid fleet, and lane. The road test detection unit consists of a video detector, a laser radar, and a magnetic frequency detector. The lanes are divided into sections with stable traffic flow, sections guided by CAV vehicles, and sections with variable traffic flow frequency. The framework diagram of energy-saving driving application for connected vehicles is shown in Figure 2.

The roadside detection unit detects road conditions and reports the number, location, and other information of the detected vehicles to the vehicle networking platform through the information transmission module. After receiving vehicle information on the road, the vehicle networking platform publishes the information to the connected vehicles in the road section. The connected vehicles control the vehicles to guide the fleet on the CAV vehicle guidance road section according to the integrated decision-making and control method of (HRL), and control the deceleration, acceleration, and lane changing behavior of the fleet. The energy-saving strategy control flowchart is shown in Figure 3.

3. A Driving Model for Networked Mixed Traffic Flow

3.1. Vehicle Following and Lane Changing Model

In the Internet of Vehicles environment, autonomous vehicle, equipped with laser radar, infrared video ranging and other on-board sensing equipment, can achieve braking control with shorter response time by virtue of the information interaction and collaboration capabilities of the Internet of Vehicles, so as to obtain higher precision and better numerical safety distance threshold. For manually driven vehicles, based on the differences in driver driving styles, they can be divided into three categories: aggressive, conservative, and cautious. Therefore, the driver adventure coefficient a is introduced. The acceleration of the preceding vehicle can reflect its motion state trend. When the current vehicle is decelerating, the following vehicle needs to reserve a larger safety distance. Based on this, the acceleration influence factor b of the preceding vehicle is introduced to improve and optimize the Gipps safety distance following model. The optimized model is shown in equation (1). $a_{n+1}(t)$

$$D_{\mathrm{safe},n}=v_n(t){\tau }_n+{\displaystyle \frac{v_n{(t)}^2}{2b_n}}-\beta {\displaystyle \frac{v_{n+1}{(t)}^2}{2b_{n+1}}}+\gamma \cdot a_{n+1}(t)\cdot {\tau }_n^2$$

(1)

In the formula: $D_{\mathrm{safe},n}$ represents the minimum safe following distance of vehicle n; $v_n(t)$ represents the speed of vehicle n at time t; ${\tau }_n$ represents the reaction time required for vehicle n to apply emergency braking; $b_n$ represents the maximum deceleration of vehicle n, and $a_{n+1}(t)$ represents the acceleration of the preceding vehicle；For autonomous vehicles, the risk factor can be set as a fixed value $\beta =1$; The risk factor of manually driven vehicles $\beta \in (0,1)$; Sensitivity coefficient $\gamma \in (0.5,1)$, when $a_{n+1}(t)<0$, this term was positive, increasing the safety distance; when $a_{n+1}(t)>0$, this item was negative and the distance could be appropriately reduced.

After considering the acceleration of the preceding vehicle, the velocity of the preceding vehicle is corrected to the predicted value, as shown in equation (2):

$$v_{n+1}^{\mathrm{pred}}(t+{\tau }_n)=v_{n+1}(t)+a_{n+1}(t)\cdot {\tau }_n$$

(2)

In the formula: $v_{n+1}^{\mathrm{pred}}(t+{\tau }_n)$ indicate the predicted speed of the preceding vehicle.

Enter the formula for safe speed, as shown in equation (3):

$$v_{\mathrm{safe},n}(t)=b_n{\tau }_n+\sqrt{\beta {\left[v_{n+1}^{\mathrm{pred}}(t+{\tau }_n)\right]}^2+b_n^2{\tau }_n^2+2b_n{d}_n(t)}$$

(3)

In the formula: $v_{\mathrm{safe},n}(t)$ represents the safe speed at which vehicle n does not collide with the preceding vehicle at time t; $d_n(t)$ Indicate the distance between vehicle n and the preceding vehicle at time t.

The above artificial driving model belongs to the ideal physical model. In actual driving scenarios, when in a high-density traffic environment, the psychological pressure of drivers tends to increase, which may lead to unnecessary deceleration behaviors without the interference of the preceding vehicle (such as unconsciously releasing the accelerator pedal). To accurately characterize the probability characteristics of the random slowing behavior, this paper introduces the Richards growth curve model from the field of biology and constructs a functional mapping relationship between road traffic density and driver random slowing probability. Through this model, it is possible to simulate the dynamic process where the psychological burden on drivers gradually increases as road traffic density continues to increase, ultimately leading to a corresponding increase in the probability of implementing deceleration operations during the driving process. This is more in line with the complex characteristics of actual human driving behavior. Set the initial value (basic slowing tendency under low density) to 0.2, the maximum slowing probability A under high density to 0.4, the growth efficiency (sensitivity of density to pressure) K to 0.05, the metabolic rate (improvement of psychological tolerance after high pressure adaptation) m to 0.95, and the random slowing probability curve equation to equation (4):

$$P_{\mathrm{slow}}=0.2+0.2{\left(1-B{e}^{-0.05\rho }\right)}^{{\displaystyle \frac{1}{-0.95}}}$$

(4)

B is the theoretical capacity of a road, which is the maximum number of vehicles that a unit length (1 km) of road can carry under ideal conditions; $\rho$ represents the density of road traffic flow, which characterizes the distribution of vehicles within a unit length of the road; N is the actual total number of vehicles on the road; L is the length of the road in kilometers. In actual traffic scenarios, as the traffic density$\rho$ continues to increase, the psychological burden on drivers exhibits dynamic changes: in the initial stage, the increase in traffic density increases the complexity of the driving environment, and the psychological burden on drivers accelerates and accumulates with the increase in density, directly manifested as the growth rate of the probability of slowing down with the machine gradually increasing (slowing down behaviors such as unnecessary deceleration, delayed operation response, etc.); When the traffic density exceeds a certain critical threshold (i.e. the inflection point of the curve), due to the high stress psychological state of the driver, the marginal stimulus effect of increasing traffic density on their psychology gradually decreases. At this time, the growth rate of the random slowing probability changes from increasing to decreasing, and finally approaches the theoretical probability upper limit of 0.4 asymptotically.

The lane change behavior for mixed traffic flow can only be triggered when the lane change conditions are met. Consider free lane changing behavior. When the speed on this lane is insufficient and the safety distance between the front and rear is within the constraint conditions, lane changing can be carried out.

The triggering condition is shown in equation (5):

$$v_{\mathrm{lead}}<\eta \cdot v_{\mathrm{ego}}(\eta =0.85)$$

(5)

In the formula: $v_{\mathrm{lead}}$ indicate the speed of the preceding vehicle in this lane, $v_{\mathrm{ego}}$ indicate the current speed of the vehicle, $\eta$ representing the speed ratio threshold (triggering lane change demand), the value is 0.85.

The forward safety distance constraint is shown in equation (6):

$$d_{\mathrm{lead}}>{\tau }_{\mathrm{lead}}\cdot \max (v_{\mathrm{ego}},v_{\mathrm{lead}})+L_{\mathrm{veh}}$$

(6)

In the formula: $d_{\mathrm{lead}}$ indicate the distance to the vehicle in front of this lane; ${\tau }_{\mathrm{lead}}$ indicating forward safe time interval, 1.8 (aggressive)~2.2 (conservative); $L_{\mathrm{veh}}$ indicates standard vehicle length, ranging from 4.5 (sedan) to 5 (SUV).

The backward safety distance constraint is shown in equation (7):

$$d_{\mathrm{lag}}>{\tau }_{\mathrm{lag}}\cdot (v_{\mathrm{lag}}-v_{\mathrm{ego}})+\gamma \cdot {\displaystyle \frac{{(v_{\mathrm{lag}}-v_{\mathrm{ego}})}^2}{2a_{\mathrm{lag},\max }}}$$

(7)

In the formula: $d_{\mathrm{lag}}$ represents the distance to the vehicle behind the target lane; ${\tau }_{\mathrm{lag}}$ indicate the safe time interval for the rear vehicle, ranging from 1.2 (aggressive) to 1.8 (conservative); $\gamma$ representing the driver's risk preference factor, ranging from 0.8 (aggressive) to 1.2 (conservative); $a_{\mathrm{lag},\max }$ indicate the maximum braking deceleration of the rear vehicle; $v_{\mathrm{lag}}$ indicates the speed of the vehicle behind the target lane.

The comprehensive decision function for lane changing is shown in equation (8):

$$\begin{array}{l}\\ \mathrm{lane} \mathrm{change}=\left\{\begin{array}{ll}1,\hfill & \begin{array}{l}\mathrm{if} (v_{\mathrm{lead}}<\eta v_{\mathrm{ego}})\wedge (d_{\mathrm{lead}}>{\tau }_{\mathrm{lead}}{v}_{\mathrm{ref}}\\ +L_{\mathrm{veh}})\wedge (d_{\mathrm{lag}}>{\tau }_{\mathrm{lag}}\Delta v+\gamma {\displaystyle \frac{\Delta v^2}{2a_{\mathrm{lag},\max }}})\end{array}\hfill \\ 0,\hfill & \mathrm{otherwise}\hfill \end{array}\right.\end{array}$$

(8)

Where: $v_{\mathrm{ref}}=\max (v_{\mathrm{ego}},v_{\mathrm{lead}})$，$\Delta v=v_{\mathrm{lag}}-v_{\mathrm{ego}}$。

3.2. Hybrid Fleet Energy Consumption Model

Regarding the energy consumption of hybrid fleets, considering that the existing vehicle types mainly include pure fuel vehicles, pure electric vehicles, and hybrid vehicles, and hybrid vehicles also include plug-in hybrid and extended range hybrid, it is possible to consider setting a power type coefficient a to divide hybrid vehicles into a combination of pure fuel vehicles and pure electric vehicles. The energy consumption of electric vehicles is mainly related to the power of the motor, while that of fuel vehicles is affected by the thermal efficiency of the engine. In order to facilitate the analysis of the energy consumption of the fleet from the same dimension, this article represents the energy consumption of both electric vehicles and fuel vehicles as kj. Specifically, in the energy consumption calculation of electric vehicles, the energy unit of the battery is converted from kw·h to kj units of energy value; In the energy consumption calculation of fuel vehicles, the calorific value of the consumed fuel is converted into an energy value in units of kj, assuming 1kw=3600kj.

Existing research usually establishes energy consumption models for AV and HDV separately, which leads to the need to calculate and stack different vehicle models for hybrid fleets, making the calculations complex and difficult to optimize collaborative strategies. For the convenience of calculation, a unified energy consumption model is established by separately modeling the acceleration/uniform speed (a ≥ 0) and deceleration (a ＜ 0) operating conditions.

Step 1: In the acceleration/uniform speed stage, the theoretical basis adopted by the electric part is: instantaneous power of the electric vehicle=base load + driving resistance power + acceleration power,

Driving resistance power$\propto v+v^3$ (rolling resistance + air resistance), $v+v^2$ is simplified to reduce the risk of overfitting. The acceleration power $\propto a^2$ reflects the nonlinear variation of motor efficiency with load (experimental calibration shows that the square term is better than the linear term). The specific expression is equation (9):

$$P_{\mathrm{AV}}=k_0+k_{1}v+k_2{v}^2+k_3{a}^2$$

(9)

In the formula: $k_0$ represents the basic power consumption of vehicle electronic devices, air conditioning, etc., set to 0.05kw; $k_{1}v$ is the linear term of speed, represents the energy consumption to overcome rolling resistance (tire deformation, transmission friction), $k_1$ set to 0.003; $k_2{v}^2$ is the square term of speed, representing the energy consumption of customer service air resistance, $k_2$set to 1.8e^-4; $k_3{a}^2$ is the square term of acceleration, representing the energy cost of kinetic energy changes (additional load on the motor/engine), $k_3$ set to 0.0012.

The basic theory used in the fuel section is that the fuel consumption rate is related to the effective fuel consumption rate (BSFC) of the engine, which is exponentially related to the speed (v) and load (a). The specific expression is equation (10):

$$P_{\mathrm{HDV}}=\exp (m_0+m_{1}v+m_{2}a)$$

(10)

In the formula: $m_0$represents the basic fuel consumption of fuel vehicles at idle speed, $m_0$set to -1.85; $m_{1}v$representing the relationship between combustion efficiency and load, affected by speed, $m_1$set to 0.04; $m_{2}a$represents the non-linear relationship between throttle opening and fuel injection quantity, influenced by acceleration,$m_2$is set to 0.08;

Step 2: During the deceleration phase, the basic theory used for the electric part is: The regenerative braking power is brake torque × speed. The experiment shows that torque is correlated with deceleration to the power of 0.8 (nonlinear energy recovery efficiency). The specific expression is equation (11):

$$\begin{array}{l}{P}_{\mathrm{AV}\_\mathrm{brake}}=k_0+\beta |a{|}^{0.8}\\ \beta =\left\{\begin{array}{ll}0.65\hfill & v>5\mathrm{m}/\mathrm{s}\hfill \\ 0.35\hfill & v\le 5\mathrm{m}/\mathrm{s}\hfill \end{array}\right.\end{array}$$

(11)

In the formula: $\beta |a{|}^{0.8}$represents the regenerative braking rate of electric vehicles, $\beta$represents the proportion of recovered kinetic energy, and the higher the speed, the greater the motor reaction force, and the higher the proportion of kinetic energy recovered.

The basic theory used in the fuel section is that the fuel injection amount during deceleration of a fuel vehicle is approximately equal to the idle fuel consumption (only maintaining engine operation), which is weakly correlated with speed (intake compensation). The specific expression is equation (12):

$$P_{\mathrm{HDV}\_\mathrm{brake}}=\exp (m_0+m_{1}v)$$

(12)

Step 3: Substitute the power type coefficient a and obtain the unified form of the energy consumption model for the hybrid fleet as equation (13):

$$P(t)=\left\{\begin{array}{ll}\lambda \cdot [k_0+k_{1}v(t)+k_{2}v{(t)}^2+k_{3}a{(t)}^2]+(1-\lambda )\cdot \exp [m_0+m_{1}v(t)+m_{2}a(t)]\hfill & a\ge 0\hfill \\ \lambda \cdot [k_0+\beta \cdot (|a{|}^{0.8})]+(1-\lambda )\cdot \exp [m_0+m_{1}v(t)]\hfill & a<0\hfill \end{array}\right.$$

(13)

Establishing a unified energy consumption model for hybrid fleets avoids the need for separately calculating energy consumption by vehicle type and solves the problem of complex calculations and difficult collaborative optimization strategies.

3.3. Energy Saving Driving Model for Hybrid Fleet

To find the optimal energy-saving driving strategy for road sections with variable traffic flow frequency, the energy-saving driving strategy can be divided into two types of implementation paths based on the traffic status (distribution of vehicles in the lane) of the nearby intersection obtained in the vehicle networking environment. When the traffic scene features perceive that the traffic flow ahead is in a steady state and the intersection signal light is green, guide the vehicle to follow the vehicle ahead in a stable mode within the vehicle guidance area; When a waiting queue vehicle is detected ahead and the adjacent lane belongs to the target available lane, guide the vehicle to perform a lane change operation within the vehicle guidance area. The above two scenarios were both carried out under the condition of satisfying the car following and lane changing model, ultimately resulting in a driving strategy that guides the safe and energy-saving operation of the vehicle.

Based on dynamic programming theory, the road driving space is subjected to spatially uniform discretization, and then deconstructed into a continuous multi-stage decision space. At the same time, the driving state of the vehicle is divided into three dimensions: distance, speed, and relative state. Based on the characteristics of actual driving scenarios, the safety distance model is extended to adapt to the constraints of vehicle driving behavior. Among them, the definition of moderate following distance is that the current following distance can meet the safety requirements for the following vehicle to follow at a constant speed. Therefore, the extended formula for calculating the relative following distance is equations (14) – (15):

$$D_{\mathrm{mid},n}=v_n(t)\cdot {\displaystyle \frac{v_{n+1}(t)}{2b_{n+1}}}+{\displaystyle \frac{v_n{(t)}^2}{2b_n}}-{\displaystyle \frac{v_{n+1}{(t)}^2}{2b_{n+1}}}$$

(14)

$$D_{\mathrm{far},n}=\left[v_n(t)+a_n\right]\cdot {\displaystyle \frac{v_{n+1}(t)}{b_{n+1}}}+{\displaystyle \frac{v_n{(t)}^2}{2b_n}}-{\displaystyle \frac{v_{n+1}{(t)}^2}{2b_{n+1}}}$$

(15)

In the formula: $D_{\mathrm{mid},n}$is the moderate following distance of vehicle n; $D_{\mathrm{far},n}$is the longer following distance of vehicle n; $v_n(t)$ is the following speed of vehicle n at time t; $a_n$is the maximum acceleration of vehicle n; $b_n$is the maximum deceleration of vehicle n.

Segmented solution is applied to the driving space ahead. Assuming the straight space length of the driving space ahead is L, the driving space is divided into n sub intervals, each with a length of $l_{gap}$. Generally, the value of $l_{gap}$ is small, and the running speed within the interval can be simplified to take the average of the end of stage speeds. For vehicles that maintain a following state, the calculation formula for the running time of the subinterval can be obtained as shown in equation (16):

$$T_k={\displaystyle \frac{2l_{\mathrm{gap}}}{v_k+v_{k+1}}}$$

(16)

For vehicles engaged in lane changing behavior, let the lane width be $l_{\mathrm{width}}$,the approximate length traveled by the vehicle is $\sqrt{{l_{\mathrm{gap}}}^2+{l_{\mathrm{width}}}^2}$,At this time, the running time of the subinterval is shown in equation (17):

$$T_k={\displaystyle \frac{2\sqrt{{l_{\mathrm{gap}}}^2+{l_{\mathrm{width}}}^2}}{v_k+v_{k+1}}}$$

(17)

Assuming constant acceleration within the subinterval, combined with the energy consumption model calculation formula, the energy consumption calculation formula for each stage of state transition is shown in equation (18):

$$E_k={\displaystyle {\int }_{T_k}^{T_{k+1}}{P}_{(t)}}dt$$

(18)

In the formula: $E_k$is the total energy consumption of stage k; $P_{(t)}$is the instantaneous energy consumption of vehicles; $T_k$is the running time for phase k.

4. Energy Saving Driving Strategy Based on Deep Reinforcement Learning

Adopting deep reinforcement learning algorithms to solve the optimal strategy for achieving energy-efficient driving. The flowchart of the deep reinforcement learning algorithm is shown in Figure 4. Traditional reinforcement learning algorithms are mostly suitable for discrete and low dimensional state spaces, which store each state and action in a table (such as Q-table). When the state space is large or continuous (such as continuous parameters such as speed and distance in autonomous driving), the table cannot be stored due to the explosion of dimensions, resulting in algorithm failure. The D3QN deep reinforcement learning algorithm introduces deep neural networks (CNN or fully connected networks) to process high-dimensional continuous state spaces, which can directly extract features from raw data (such as sensor data, image pixels) without the need for manual design of state features. For example, in energy-saving road scenarios, multi-dimensional continuous data such as vehicle speed, relative distance, and position of vehicles ahead can be directly input, and the state representation can be automatically learned through the network.

4.1. D3QN Algorithm

The D3QN algorithm model is shown in Figure 5. In the initial state, the agent inputs the environmental state matrix S_t into the prediction network, and outputs the Q values of each action after calculation by the prediction network. Then, using strategy $\epsilon -greedy$, randomly select an action a_t to execute and interact with the environment to receive a reward $r_t$.$\epsilon -greedy$ strategy refers to randomly selecting an action with a probability of $\epsilon$ and selecting the action corresponding to the maximum Q value calculated by the current prediction network with a probability of $1-\epsilon$, as shown in equation (19):

$$a_t=\left\{\begin{array}{cc}{a}_i,& \epsilon \\ \arg \max Q\left(s_t,a_t;\theta \right),& 1-\epsilon \end{array}\right.$$

(19)

The interaction between intelligent agents and the environment will generate a series of experience sequences $<s_t,r_t,a_t,s_{t+1}>$,Store it in the experience replay pool as training samples, and uniformly sample a certain batch size of sequences from it during each training session. As the algorithm continues to run, the number of samples in the experience replay pool increases, and the sampling of the prediction network becomes more diverse. The experience replay mechanism breaks the strong correlation between consecutive experiences, making each training independent and the results more reliable.

In deep reinforcement learning, the D3QN algorithm integrates two key technologies, Double and Dueling, to enhance the stability and performance of the model.

（1）Double Q Network

Two networks are used in the model, one is the current network with an initial parameterization of $Q$, whose value changes with the operation of the algorithm, and the other is the target network with an initialization parameter of $Q^{-}$. The structures of the two networks are the same to facilitate parameter assignment.

Current network $Q_{\mathrm{online}}(s,a;\theta )$：

$$\begin{array}{l}{h}_1=\mathrm{ReLU}(W_1{s}_t+b_1) h_2=\mathrm{ReLU}(W_2{h}_1+b_2)\\ V(s;{\theta }_v)=W_v{h}_2+b_v\\ A(s,a;{\theta }_a)=W_a{h}_2+b_a\\ Q_{online}(s,a;\theta )=V(s;{\theta }_v)+\left(A(s,a;{\theta }_a)-{\displaystyle \frac{1}{\left|\mathcal{A}\right|}}{\displaystyle \sum _{a^{\prime }}A}(s,a^{\prime };{\theta }_a)\right)\end{array}$$

(20)

In the formula: $Q_{\mathrm{online}}$represents the final value predicted by the online network; $s_t$state vector representing time step t; $\theta$representing online network parameters; h representing the output of the hidden layer; $W_1,b_1$representing the weights and biases of the first layer; $W_2,b_2$representing the weights and biases of the second layer; $V(s;{\theta }_v)$representing state value scalar; $A(s,a;{\theta }_a)$representing the action advantage vector.

Target network $Q_{\mathrm{target}}(s,a;{\theta }^{-})$：

$$Q_{target}(s,a;{\theta }^{-})=V(s;{\theta }_v^{-})+\left(\begin{array}{l}A(s,a;{\theta }_a^{-})\\ -{\displaystyle \frac{1}{\left|\mathcal{A}\right|}}{\displaystyle \sum _{a^{\prime }}A}(s,a^{\prime };{\theta }_a^{-})\end{array}\right)$$

(21)

The target network structure is the same as the online network, parameter ${\theta }^{-}$Regular replication $\theta$.

The prediction network is used to select actions, and the parameter Q is constantly updated; The target network is used to calculate the temporal difference value y, with the parameter $Q^{-}$fixed and replaced with the latest prediction network Q at regular intervals. The calculation formula for the target value Q is shown in equation (22):

$$y_t=r_t+\gamma \cdot Q_{\mathrm{target}}\left(s_{t+1},\underset{a^{\prime }}{\arg \max }{Q}_{\mathrm{online}}\left(s_{t+1},a^{\prime };\theta \right);{\theta }^{-}\right)$$

(22)

（2）Dueling Q Network

In the Dueling network architecture design, the Q network is divided into two parts. The first part is only related to state s and is independent of the specific action a used, denoted as the state value function$v(s)$. The second part is related to both state s and action a, denoted as the advantage function $A(s,a)$.

$$Q\left(s, a\right)=V\left(s\right)+A\left(s,a\right)$$

(23)

The D3QN algorithm uses a network based on Convolutional Neural Network (CNN), and its specific neural network structure is shown in Figure 6:

4.2. Loss Function

The loss function of D3QN is based on the parameter $\theta$of the prediction network when selecting the target Q-value action, that is, selecting the action corresponding to the maximum Q-value in the prediction network in the current state, and then calculating the Q-value in the target network. This reduces the correlation between action selection and target Q-value calculation, effectively avoiding the problem of overestimation. In D3QN (Dueling Double DQN), Mean Squared Error (MSE) is used as the loss function by default, as shown in equation (24):

$$L_{\mathrm{D}3\mathrm{QN}}=E\left[{\left(r_t+\gamma Q^{-}\left(\begin{array}{l}{s}_{t+1},\arg {\max }_a\\ \left(Q(s_{t+1},a;\theta )\right);{\theta }^{-}\end{array}\right)-Q(s_t,a_t;\theta )\right)}^2\right]$$

(24)

In the formula: E represents the expected operation, the expected distribution of training data (experience replay pool samples); The instant reward obtained by agent $r_t$ after performing an action at time t; $Q^{-}$ represents the target Q-value calculation term, reflecting the core logic of the “Double Q-Network”; $Q(s_t,a_t;\theta )$ represents the Q value prediction of the current main Q network (parameter $\theta$ ) for executing action $a_t$ in state $s_t$ at time t; T represents the actual action performed by the agent at time t, based on sample data from the experience replay pool.

4.3. Reward Function

The reward function needs to evaluate the fuel consumption of sub intervals and the entire driving process, taking into account the fuel consumption optimality during each state transition and overall strategy learning. Therefore, the relative distance to the preceding vehicle in this lane is introduced as a feedback value in the reward function of the algorithm. Reduce the reward values in order of relative distance to indicate the promotion of following vehicles at a longer distance. The specific function is shown in equation (25):

$$R(s_t,a_t)=\alpha \cdot \left[R-P(v_t,a_t)\cdot T(s_t,a_t)\right]+\beta \cdot h(s_t,a_t)+\lambda {\coprod }_{\mathrm{safe}}$$

(25)

In the formula: $R(s_t,a_t)$represents the reward value obtained by the agent when executing action $a_t$ in state $s_t$; $\alpha$, $\beta$ and $\lambda$ are both weight coefficients used to balance the impact of different components in the reward function on the total reward; R represents the benchmark reward under ideal conditions; $P(v_t,a_t)$ represents a function related to the feasibility of the action; $T(s_t,a_t)$represents the state transition constraint loss; $h(s_t,a_t)$ represents the penalty function, used to characterize the special correlation benefits between state $s_t$ and action $a_t$; ${\coprod }_{\mathrm{safe}}$ represents a security indicator function used to reinforce rewards for security behaviors or punishments for violations.

5. Simulation Experiment and Result Analysis

5.1. Simulation Experiment Design

The simulation experiment uses the open-source micro traffic simulation software SUMO (Simulation of Urban Mobility), which interacts with Python through an external interface (TraCI) to obtain real-time simulation data, control simulation elements, and achieve closed-loop simulation.

The simulation section of energy-saving driving strategy in this article is selected from the intersection of Qingdao Haigang Road and Binhai Avenue to the intersection of Dawan Port Road and Binhai Avenue, with a total length of 1610m. The distance between the upstream and downstream of this section of intersection is relatively far, and the traffic flow is moderate. The simulation is studied in three scenarios, scenario one is a traffic scenario in a natural traffic state without setting any energy-saving strategies; Scenario 2 is a scenario where energy-saving driving strategies are set up, and the energy-saving driving strategy method is Q-Learning reinforcement learning; Scenario three is a scenario layout with energy-saving driving strategies set, and the energy-saving driving strategy method is the D3QN deep reinforcement learning method. According to the current Technical Standards for Highway Engineering, four lane first-class highway can adapt to an annual average daily traffic volume of 15000 to 30000 vehicles, so the upper limit of the simulation of vehicle flow density in this paper is set as 30 pcu·km^-1. Considering that the proportion of autonomous connected vehicles in traffic flow varies, it will have a significant impact on the results of simulation experiments. In order to comprehensively verify the applicability of the proposed energy-saving driving strategy, simulation experiments will be conducted for various scenarios with different proportions of autonomous connected vehicles. By conducting simulation tests in mixed traffic flow environments with different proportions (gradient settings from low to high proportions), the performance of the strategy in various scenarios is analyzed in depth, and its adaptability to mixed traffic flow environments of different degrees is clarified to ensure that the strategy can cope with complex and changing traffic conditions in practical applications. The following is the reference data used in this simulation, as shown in Table 1:

5.2. Solution of Energy Saving Driving Strategy

The solution process uses the d3qn deep reinforcement learning algorithm constructed in Section 3 to iteratively learn the energy-saving driving strategy. The initial parameter settings of the d3qn energy-saving algorithm are shown in Table 2 below:

Firstly, the convergence of the algorithm in the iterative learning process is verified. Because the environmental state quantity discussed in this paper is large, and the scale of Q table and R table is large, through the analysis of convergence, it can be clear whether the obtained Q table is globally optimal, so as to avoid falling into the local optimal situation. In the simulation process, the convergence of the algorithm in the generation selection learning process is calculated, and the results are shown in Figure 6. It can be seen from the convergence curve in Figure 7 that after 2000 iterations of the cycle, the calculated vehicle fuel consumption becomes relatively fixed and has the characteristics of convergence.

5.3. Simulation Result

In the simulation process, for mixed traffic flow, the vehicle flow density takes 0 as the initial value and gradually increases according to the gradient of 5pcu・km^-1 until it reaches 30pcu・km^-1. For the proportion of automatic connected vehicles, it is initially set to 0 (that is, the simulation of the conventional vehicle scene with all manual driving in the road section), and then increases in order with a gradient of 20% to 100% (the scene of fully realizing automatic connected driving in the corresponding road section). Through iterative learning of the algorithm, the fuel consumption results of the controlled vehicle under each environment are finally obtained. The specific data are shown in Table 3 and Figure 8.

Under the same CAV proportion, with the increase of traffic density on the road, the driving space of controlled vehicles will be more restrained, resulting in the fluctuation of driving strategies. Therefore, the fuel consumption of different driving strategies is increasing to varying degrees, and the rise of scenario 2 and scenario 3 using energy-saving driving strategy is more gentle, indicating that the energy-saving driving strategy proposed in this paper has better energy-saving effect when the road is more crowded.

The energy-saving effects of scenario 2 and scenario 3 are significantly higher than those of the conventional driving strategy in scenario 1. Table 3 further compares the energy-saving effects of d3qn energy-saving driving strategy and Q-learning energy-saving driving strategy, specifically the fuel consumption of the energy-saving strategy and the energy-saving effect of the conventional strategy under different CAV proportions and traffic densities.

It can be seen from the data results in Table 3 that the energy-saving effect of d3qn energy-saving strategy is always better than that of Q-learning energy-saving strategy. Through calculation, the fuel consumption of d3qn energy-saving strategy is 8.41%~6.67% lower than that of conventional strategy, and the fuel consumption of d3qn energy-saving strategy is 1.94%~1.5% lower than that of Q-learning energy-saving strategy.

Analyze the fluctuation of vehicle speed on the road with frequent traffic flow, count the overall vehicle speed under the three scenarios, and quantify the fluctuation amplitude by using the standard deviation. The comparison diagram of vehicle speed time series is shown in Figure 9, and the distribution diagram of vehicle speed probability density is shown in Figure 10.

It can be seen from Figure 9 that the fluctuation of the conventional strategy is obvious in a short period of time in the sections with frequent traffic flow, the fluctuation of the Q-learning strategy is significantly reduced, and the d3qn strategy is the lowest, indicating that the energy-saving driving strategy can reduce the fluctuation of traffic flow and improve the stability of traffic flow, and the d3qn strategy is superior to Q-learning.

According to the probability density distribution diagram of vehicle speed in Figure 10, the vehicle speed of conventional driving strategy is relatively dispersed, the energy-saving driving strategy is relatively centralized, and q3dn strategy is more centralized than Q-learning strategy. D3qn strategy can control the vehicle speed between 30~50km/h, which greatly ensures the stability of traffic flow.

6. Conclusion

(1) Based on the in-depth analysis of traffic flow frequency changing sections, combined with the Internet of vehicles technology, the application framework of energy-saving driving strategy for traffic flow frequency changing sections is constructed. Its core is that the network connected autonomous vehicles operate in the vehicle road collaborative environment. Relying on the high-precision and low delay real-time traffic information transmitted by the vehicle network platform, the collaborative decision-making module of Cavs can generate the global or local optimal lane change strategy, so as to realize the collaborative optimization of path planning and lane maintenance/transformation.

(2) A car following and lane changing model of mixed traffic flow is constructed, and the behavior difference between manual driving and intelligent driving vehicles is accurately described by introducing the driver risk coefficient, the influence factor of the acceleration of the vehicle in front and the random slowing probability model; The unified energy consumption model integrates the energy consumption calculation of fuel vehicles, electric vehicles and hybrid vehicles through the power type coefficient, which solves the complexity of superposition calculation by vehicle type, and provides a unified benchmark for collaborative optimization. Combined with the real-time information interaction (roadside detection, platform decision-making, vehicle collaboration) in the Internet of vehicles environment, the model can dynamically adjust the driving strategy (such as stable car following or safe lane changing) according to the traffic state, taking into account the safety and energy conservation.

(3) D3qn deep reinforcement learning algorithm is introduced as the learning engine of energy-saving driving strategy, which not only solves the limitations of traditional reinforcement learning, but also optimizes the balance between energy saving and safety. By separating the state value function and the action advantage function, the algorithm gives priority to long-term energy-saving actions, and introduces the safety distance penalty term and random moderation probability constraint, so that CAV can take into account the minimum fuel consumption and collision risk avoidance when guiding the hybrid fleet.

(4) The simulation scenario is from the intersection of Haigang road and Binhai Avenue in Qingdao to the intersection of dawangang road and Binhai Avenue. According to the simulation results, compared with the conventional strategy, d3qn strategy can save 8.41%~6.67% of fuel consumption under different proportions of intelligent connected vehicles (CAV); Compared with Q-learning strategy, it can still save 1.94%~1.5% of fuel consumption. And with the increase of traffic density, this energy-saving advantage is more obvious. When the traffic flow density increases from 0 to 30 pcu・km^-1, the increase of fuel consumption with d3qn strategy is more gentle, indicating that it has stronger energy-saving adaptability in congestion scenarios. In terms of traffic flow stability, d3qn strategy can effectively reduce the speed fluctuation. Through the quantitative analysis of the standard deviation, the vehicle speed of the conventional strategy fluctuates violently in a short time. Although the Q-learning strategy has improved, the speed fluctuation of the d3qn strategy is the lowest. From the probability density distribution of vehicle speed, the d3qn strategy can centrally control the vehicle speed in the range of 30~50 km/h, significantly improve the stability of traffic flow, and reduce the energy consumption waste and traffic efficiency loss caused by frequent acceleration and deceleration.