The IEEE 802.11ah standard is introduced to address the growing scale of Internet of Things (IoT) applications. To reduce contention and enhance energy efficiency within the system, the restricted access window (RAW) mechanism is introduced at the medium access control (MAC) layer to manage the significant number of stations accessing the network. However, the RAW parameters need to be appropriately determined to achieve optimal network performance. In this paper, we optimize the configuration of RAW parameters in the IEEE 802.11ah-based IoT network. We analyze and propose an RAW parameters optimization problem with the objective of improving network throughput and formulate it as a Markov decision process. To cope with the complex and dynamic network conditions, we propose a proximal policy optimization (PPO)-based deep reinforcement learning (DRL) algorithm to find the optimal RAW parameters. We construct network environments with periodic and random traffic in the NS-3 simulator to validate the performance of the proposed PPO-based RAW parameters optimization algorithm. Simulation results reveal that using the PPO-based DRL algorithm, optimal RAW parameters can be obtained under different network conditions, thereby significantly improving network throughput.

This research delves into the realm of predictive modeling within artificial intelligence (AI) and introduces an innovative ensemble approach designed to enhance prognostic precision. The proposed methodology, characterized by its ebullient nature, leverages the collective wisdom of diverse predictive models, synthesizing a comprehensive and robust framework. The pursuit of parsimony are paramount, underscoring the need for efficient model architectures that balance predictive acuracy and computational efficiency. Our study meticulously explores the amalgamation of diverse algorithms, encapsulating the intrinsic complexity of real-world phenomena. We navigate the intricate landscape of feature selection, algorithmic diversity, and model combination, fostering a holistic understanding of the prognostic process. The ebullient ensemble emerges as a dynamic solution, harnessing the synergy of constituent models while mitigating the risk of overfitting. To validate the efficacy of our approach, extensive experiments are conducted on diverse datasets, spanning domains such as healthcare, finance, and natural language processing. Comparative analyses demonstrate the superior predictive performance of our ensemble model in comparison to individual algorithms and traditional ensemble methods. Additionally, we delve into interpretability aspects, elucidating the underlying mechanisms that contribute to the model's prognostic prowess. Our research advances the frontier of predictive modeling in AI, offering a nuanced and sophisticated paradigm for prognostication. The ebullient ensemble approach stands as a testament to the synergy achievable through the amalgamation of diverse predictive models, providing a pragmatic and parsimonious avenue for enhanced prognostic acuracy in artificial intelligence applications.

It is necessary to address the scarcity of crushed stones for pavement structural layers. So fly ash can be proved to be promising solution as more than 270 tonnes of fly ash is generated in India. Though, numerous research has been conducted for the use of fly ash intreated and untreated form, high volume of fine particles and brittleness of the stabilized fly ash pose a great problem for its use in subbase and base. Moreover, stiffness or modulus of stabilized fly ash is vital elastic parameter which is used for mechanistic pavement design. Hence, in this study an extensive experimental investigation is carried out to study its strength and stiffness properties such as compressive strength, indirect tensile strength and flexural strength, cyclic indirect tensile modulus and flexural modulus of fiber reinforced cement stabilized fly ash, stone dust, aggregate mixtures. The stone dust and aggregates have been added to enhance the gradation of the composite’s mixture. The study presents the effect of fiber on strength and stiffness properties. The experimental result reveals that addition of polypropylene (PP) fibers up to 0.25 wt.% enhances the compressive strength and any further addition of fiber results in decrease of the strength. However, indirect tensile strength and flexural strength increases with increase in fiber percentage up to 0.5 wt.%. Cement content is observed to be the dominant parameter for stabilized materials. Suitable relationships have been developed between strength and modulus parameters for stabilized mixtures. Based on the strength and stiffness study, 70% fly ash and 30% stone dust-aggregate and 60% fly ash and 40% stone dust-aggregate with 6% cement can be considered for the base layer. Based on the indirect tensile strength and flexural strength behavior, 0.35% is considered as the optimum fiber percentage.

Stagnation at local optima represents a significant challenge in bio–inspired optimization algorithms, often leading to suboptimal solutions. This paper addresses this issue by proposing a hybrid model that combines the Orca Predator Algorithm with Deep Q–Learning. Orca Predator Algorithm is an optimization technique that mimics the hunting behavior of orcas. It solves complex optimization problems by exploring and exploiting search spaces efficiently. Deep Q–Learning is a reinforcement learning technique that combines Q–Learning with deep neural networks. This integration aims to turn the stagnation problem into an opportunity for more focused and effective exploitation, enhancing the optimization technique’s performance and accuracy. The proposed hybrid model leverages the biomimetic strengths of Orca Predator Algorithm to identify promising regions nearby in the search space, complemented by the fine–tuning capabilities of Deep Q–Learning to navigate these areas precisely. The practical application of this approach is evaluated using the high–dimensional Heartbeat Categorization Dataset, focusing on the feature selection problem. This dataset, comprising complex electrocardiogram signals, provided a robust platform for testing the feature selection capabilities of our hybrid model. Our experimental results are encouraging, showcasing the hybrid strategy capability to identify relevant features without significantly compromising the performance metrics of machine learning models. This analysis was performed by comparing the improved method of Orca Predator Algorithm against its native version and a set of state–of–the–art algorithms.

Abstract:The objects of fine-grained image categories(e.g., bird species) are various subclass under different categories. Because the differences between subclass are very subtle and most of them are concentrated in multiple local areas, the task of fine-grained image recognition is very challenging. At the same time, some fine-grained networks tend to focus on a certain region when judging the target category, resulting in the lack of other auxiliary regional features. To this end, Inception V3 is used as the backbone network, and an enhanced and complementary fine-grained image classification network is designed. While adopting the method of reinforcement learning to obtain more detailed fine grain image features, the complementary network can obtain the complementary discriminant area of the target through the method of attention erasure to increase the network's perception of the overall target. Finally, experiments are conducted on CUB-200-2011, FGVC Aircraft and Stanford dogs three open datasets. The experimental results show that the proposed model has better performance.

Digital Microfluidic Biochips (DMFBs), used in various kinds of fields like DNA analysis, clinical diagnosis, and PCR testing, have made biochemical experiments more compact, efficient, and user-friendly than previous ways. However, their reliability is often compromised by their inability to adapt to all kinds of errors. All errors in biochips can be categorized into two types: known errors and unknown errors. Known errors are detectable before the start of the routing process through sensors or cameras. Unknown errors, in contrast, become apparent only during the routing process and remain undetected by sensors or cameras, which is the biggest issue to unexpectedly stop the routing process and diminishes the reliability of biochips. This paper introduces a deep reinforcement learning-based routing algorithm designed to manage not only known errors but also unknown errors. Our experiments demonstrate that our algorithm outperforms previous ones in terms of the success rate of the routing in the scenario including both known errors and unknown errors. Additionally, our algorithm contributes to detecting unknown errors during the routing process and identifying the most efficient routing path with high probability.

There is only a very short reaction time for people to find the best way out of a building in a fire outbreak. Software applications can be used to assist the rapid evacuation of victims; however, this is an arduous task, which requires an understanding of advanced technologies. Since well-known pathway algorithms (such as, Dijkstra, Bellman-Ford and A*) can lead to serious problems over performance, we decided to make use of deep reinforcement learning techniques. A wide range of strategies including a random initialization of replay buffer and transfer learning were assessed in three projects involving schools of different sizes. The results showed the proposal was viable and that in most cases the performance of transfer learning was superior. In addition, the study raised challenges that had to be faced in the future.

The study examines the behavior of the Mechanically Stabilized Earth (MSE) wall using available reinforcement materials in Danang, Vietnam. The MSE was reinforced by the self-fabricated galvanized steel grids using CB300V steel with 3 cm ribs. The backfill soil is a sandy clay soil from the local area with a low cohesion. A full-scale model with full instrumentation was installed to investigate the distribution of tensile forces along the reinforcement layers. The highest load that caused the wall to collapse due to internal instability (reinforcement rupture) was 302 kN/m2, which is 15 times the design load of 20 kN/m2. The failure surface within the reinforced soil had a parabolic sliding shape which is similar to the theoretical studies. At the failure load level, the maximum lateral displacement at the top of the wall facing was small (3.9 mm) which is significantly lower than the allowable displacements for the retaining wall. Furthermore, a numerical model using FLAC software was applied to simulate the performance of the MSE wall. The modeling results are in good agreement with the physical model. Thus, the self-fabricated galvanized steel grids could be confidentially used in combination with the local backfill soil for the MSE walls.

This paper is primarily focused on the robust control of an inverted pendulum system based on the policy iteration in reinforcement learning. First, a mathematical model of the single inverted pendulum system is established through a force analysis of the pendulum and trolley. Second, based on the theory of robust optimal control, the robust control of the uncertain linear inverted pendulum system is transformed into an optimal control problem with an appropriate performance index. Moreover, for the uncertain linear and nonlinear systems, two reinforcement-learning control algorithms are proposed using the policy iteration method. Finally, two numerical examples are provided to validate the reinforcement learning algorithms for the robust control of the inverted pendulum systems.

In the dynamic and complex environment of industrial control rooms, operators are often inundated with a multitude of tasks and alerts, which can lead to an overwhelming situation known as task overload. This state can precipitate decision fatigue and a heavier reliance on cognitive biases, potentially compromising the decision-making process. To mitigate such risks, the implementation of decision support systems becomes crucial. These systems are designed to assist operators in making swift and well-informed decisions, particularly when they sense their own judgment may be faltering. Our research introduces an AI-based framework that leverages dynamic influence diagrams and reinforcement learning to construct an effective decision support system. The cornerstone of this framework is the creation of a robust, interpretable, and efficient tool that supports control room operators during critical process disturbances. By integrating expert knowledge, the dynamic influence diagram forms a comprehensive model that captures the uncertainties inherent in complex industrial processes. It is adept at anomaly detection and recommending optimal actions. Moreover, this model is enhanced through a strategic partnership with reinforcement learning algorithms, which fine-tune the recommendations to be more context-specific and precise. The ultimate goal of our framework is to provide operators with a live, reliable decision support system that can significantly improve their response during process upsets. This paper outlines the development of our AI framework and its application within a simulated control room environment. Our findings indicate that the use of decision support systems can lead to improved operator performance and reduced cognitive workload. However, it also reveals a trade-off with situation awareness, which tends to diminish as operators may become overly reliant on the system’s guidance. Trust emerges as a critical factor in the adoption and effectiveness of decision support systems. Our research underscores the importance of balancing the benefits of decision support with the need for maintaining operator engagement and comprehension during process operations.

Reinforcement Learning (RL) has demonstrated considerable potential in solving challenges across various domains, notably in autonomous driving. Nevertheless, implementing RL in autonomous driving comes with its own set of difficulties, such as the overestimation phenomenon, extensive learning time, and sparse reward problems. Although solutions like Hindsight Experience Replay (HER) have been proposed to alleviate these issues, the direct utilization of RL in autonomous vehicles remains constrained due to the intricate fusion of information and the possibility of system failures during the learning process. In this paper, we present a novel RL-based autonomous driving system technology that combines Obstacle Dependent Gaussian (ODG) RL, Soft Actor-Critic (SAC), and meta-learning algorithms. Our approach addresses key issues in RL, including the overestimation phenomenon and sparse reward problems, by incorporating prior knowledge derived from the ODG algorithm. We evaluated our proposed algorithm on official F1 circuits, using high-fidelity racing simulations with complex dynamics. The results demonstrate exceptional performance, with our method achieving up to 89% faster learning speed compared to existing algorithms in these environments.

Long-span steel structure trusses are widely used in factory buildings, but with the growth of service time and the increase of dynamic load fatigue, a considerable part of the long-span truss with dynamic load appears serious transverse cracks at the bottom of the middle span and oblique deformation of the abdomen in the operation process. The U-shaped cracks at the bottom and belly, as well as the mid-span down deflection of the main truss, reduced the functional function of the factory building truss structure, and had to limit the original crane load, which affected the normal safety and durability of the structure. Therefore, the application principle of the variable axial force cable system in the long-span factory building truss structure and 3D3S software modeling [1] were used. Analyzing and studying the reinforcement method of large-span powerhouse truss can provide practical experience for subsequent similar projects. In view of the above phenomenon, the large-span powerhouse trusses of Hongcheng Powerhouse 1 and No.2 located in Tonglu, Zhejiang Province are used as the research object, and the variable axial force cable method is proposed to strengthen and lift the load. Considering the span of large-span powerhouse truss, the cable system with 22m controlling force of 400kN is proposed to be selected for powerhouse 1, and the cable system with variable axial force of 24m is proposed to be selected for Powerhouse 2. The force model of large-span truss is established by using the finite element method commonly used to analyze the force of truss. Under two working conditions, The influence of reinforcement effect is analyzed and compared from three aspects: stiffness, bearing capacity and stability. And the phenomenon of uneven stress distribution is analyzed. The stress distribution characteristics of each node are understood by simulating the most disadvantageous node plates with the greatest internal force before and after reinforcement.

The author previously reported an experimental result of evolutionary reinforcement learning of neural network controllers for the pendulum task. In the previous work, a conventional multilayer perceptron was employed in which connection weights were real numbers. In this study, the author experimentally applies an evolutionary algorithm to the reinforcement training of binary neural networks. In both studies, the same task and the same evolutionary algorithm are utilized, i.e. the pendulum control problem and Evolution Strategy respectively. The only differences lie in the memory size per connection weight and the model size of the neural network. The findings from this study are (1) the performance of the binary MLP with 32 hidden units was inferior to that of the real-valued MLP with 16 hidden units; however, this difference was not statistically significant (p >.05); (2) the trained binary MLP successfully swung the pendulum swiftly into an inverted position and maintained its stability after inversion, as the real-valued MLP had done; and (3) the memory size required to record the binary MLP with 32 hidden units is 3.1% or 6.2% of the memory size required to record the real-valued MLP with 16 hidden units.

The opened beams always confused the designers due to the guidelines missing. In this research, six hybrid reinforced beams reinforced with mixed steel and basalt fiber reinforced polymer (BFRP) bars had constant cross-sections of 150mm x 300mm and a clear span of 1800mm. Generally, five beams have symmetrical rectangular openings with dimensions of 150mm x 250mm located at a distance of 250mm (equivalent to the beam effective depth) from the beam support in addition to a solid beam that is served as a control reference. The studied parameters included the effect of using internal reinforcement (steel or BFRP bars) provided along the opening or by incorporating an external BFRP sheet around the opening corners. Also, the conduction of double enhancement with internal steel reinforcement bars further external strengthening BFRP sheet was investigated. The relevant results showed that the opened beam without enhancement lost almost 74.66% of the maximum load compared with the solid beam. Placing internal steel or BFRP bars around the openings increased the maximum load by 62.07% and 59.68%, respectively in comparison to the non-enhanced opened beams. Using an external BFRP sheet to strengthen the opening corners of the beam enhanced the maximum load by 76.39% compared with the non-enhanced opened beam. Therefore, if the beam double enhancement with an external BFRP sheet and internal steel reinforcement around the openings, the maximum load increased by 137.40% compared with the non-enhanced opened beam. Ultimately to further analyze the experimental results and confirm their findings, the study was extended to include the numerical analysis using three dimensional finite element modeling and the results correlated very well with the experimental ones.

This report gives a comprehensive summary of two problems about graph convolutional networks (GCNs): over-smoothing and heterophily challenges, and outlines future directions to explore.

Aiming at the problem of hypersonic morphing vehicle avoiding no-fly zones and reaching the target, an improved predictor-corrector guidance method is proposed. Firstly, the aircraft motion model and the constraint model are established. Then, the basic algorithm is given, the Q-learning method is used to design the attack angle and sweep angle scheme to ensure that the aircraft can fly over the low-altitude zones. The B-spline curve is used to design the location of flight path points and the bank angle scheme is designed according to the predictor-corrector method, so that the aircraft can fly around to avoid high-altitude zones. Next, Monte Carlo reinforcement learning(MCRL) method is used to improve predictor-corrector method and Deep Neural Network(DNN) is used to fit reward function. The improved method can generate trajectory with better performance. Simulation results verify the effectiveness of the proposed algorithm.

The shield tunnels inevitably endure various damage with the service time increasing. Meanwhile, steel corrugated plate has been used extensively in multiple conditions and proved effective to strengthen the segmental joints by full-scale tests. A numerical model is proposed to probe the feasibility of using new Stainless Steel Corrugated Plate (SSCP) to reinforce the shield tunnel segments. A new method, called virtual tracking element technology, is employed to achieve the simulation of realistic stress state of the segmental joint. Moreover, the segmental joint component analysis and parametric study are conducted based on the numerical model. The results demonstrate that: (1) the virtual tracking element technology proves valid and efficient to simulate the secondary stress state of segmental joint; (2) SSCP reinforcement is not fully utilized when the grade of segmental concrete is C50 and it has abundant safety margin for potential overload; (3) SSCP reinforcement is well-performed regardless of the burial depth and reinforcement in advance is recommended.

Evolutionary algorithms find applicability in reinforcement learning of neural networks due to their independence from gradient-based methods. To achieve successful training of neural networks using evolutionary algorithms, careful considerations must be made to select appropriate algorithms due to the availability of various algorithmic variations. The author previously reported experimental evaluations on Evolution Strategy for reinforcement learning of neural networks, utilizing the Acrobot control task. In this study, Genetic Algorithm is adopted as another instance of major evolutionary algorithms. Experimental results demonstrate that there was no statistically significant difference between the experimental performances of GA and ES, but the priority of generations and offsprings was different; GA performed better with a greater number of generations while ES performed better with a greater number of offsprings. Eight hidden units were the best among four variations (4, 8, 16 or 32 units), which aligns with previous study using ES.

The reliability of concrete cable-stayed bridge structure is time variant. This is due to strength deterioration and time-variant mechanical and environmental loadings. Demands for keeping concrete cable-stayed bridge structure in functional and reliable states require maintenance interventions. Limited financial resources require optimal maintenance strategies. These strategies have to consider the effects of uncertainties not only on the structural reliability deterioration but also on the maintenance interventions. By analyzing variation of concrete cable-stayed concrete cable-stayed bridge liabilities and benefit expectation through different maintenance programs, on the condition of satisfying the concrete cable-stayed bridge performance, with the maximum expected benefit as the target to discuss repair and reinforcement strategies for the concrete cable-stayed bridges in service. The method takes into account the present performance and future expected benefit, optimizing the performance of the concrete cable-stayed bridge and the expected benefit through determination policies of concrete cable-stayed concrete cable-stayed bridge maintenance program. The proposed method also includes an advanced cost model which takes into account the interaction between costs of maintenance interventions and their effect on reliability. Examples are presented demonstrating the application of the proposed method to a stock of existing deteriorating concrete cable-stayed bridges under various maintenance scenarios.

The increasing popularity of carbon-reinforced concrete (CRC) is attributed to its exceptional tensile properties, low density, no corrosion phenomenon, and remarkable flexibility, allowing it to be easily shaped into various forms. This research investigates the feasibility of using a special 2D Netzgitterträger (NetzGT) reinforcement system, featuring a net-shaped fabricated textile made of multiple diagonally offset rovings with overlapping edge strands, as a viable alternative to traditional steel reinforcement in concrete beams. This reinforcement is manufactured from carbon rovings with three different diagonal angles of 50⁰, 60⁰, and 70⁰ respectively. Laboratory experiments were conducted to assess the mechanical behavior of beams reinforced with the 2D NetzGT reinforcement. Bending and shear tests were performed on beams with varying numbers of overlapped edge roving and roving angles to evaluate the tensile capacity and failure characteristics of beams. The increase in the number of overlapped edge rovings led to a noticeable increase in the maximum tensile force. Tensile tests on strands were also performed with the increasing number of overlapped rovings to analyze their tensile strength. Additionally, single yarn pull-out tests were also conducted to examine the influence of the roving angle on the bond strength between the carbon textile roving and the concrete matrix

Thin people who undergo implant insertion into the chest wall face a high risk of implant exposure to the external environment. Five months after an 8-year-old boy underwent implantable cardioverter-defibrillator (ICD) implantation in a subcutaneous pocket in the left anterolateral chest wall to manage long QT syndrome, ICD replacement became necessary owing to exposure risk from distal and lateral thinning of the ICD pocket. Pocket rupture and exposure would increase the risk of infection; therefore, we performed ICD removal and primary pocket closure. Two weeks later, a new suprafascial pocket was created, acellular dermal matrix (ADM) was attached to the inner wall to prevent ICD protrusion, and a new ICD was inserted. One year postoperatively, the ADM was engrafted, and no complications were observed. A thin subcutaneous layer increases the risk of ICD implantation complications. Inner wall strengthening with ADM can help prevent pocket rupture.

This paper presents an innovative multi-agent, data-driven reinforcement learning (RL) approach to develop and utilize the thermal equivalent network model that represents the motor's thermal dynamics. A multi-agent reinforcement learning is designed and trained to adjust the model parameters using data from several motor driving cycles. To ensure the incoming driving cycle matches the historical data before employing the pre-trained RL agents, offline statistical analysis and clustering techniques are developed and used. Numerical simulations highlighted the RL agent's ability to develop strategies that effectively address the variability of driving cycles. The proposed RL framework showed its capability to accurately reflect the motor's thermal behavior under various driving conditions.

The conducted investigation encompassed the comprehensive integration of mechanical, envi-ronmental, chemical, and microstructural evaluations of a composite amalgamating sandy soil and a synthetic polymer at two distinct concentrations (2.5% and %) across multiple curing in-tervals (0, 1, 2, 4, 7, 15, 30, and 45 days). The studied soil originates from an environmentally sig-nificant protected area in Brazil. The implementation of mechanisms for soil improvement in the region must adhere to technical criteria without causing environmental harm. Direct shear testing was conducted, permeability was assessed, and microstructure analysis and XRD and XRF/EDX studies of both the soil and composites were conducted. It was observed that longer curing times yielded improved results in shear stress, friction angle, and cohesive intercept, with SP_5% exhibiting the highest values compared to the soil and SP_2.5%. Adding the polymeric so-lution to the soil contributed to cementation and cohesion gains in the substrate. Through mi-crostructural characterization, the polymer's role as a cementing agent for the grains is evident, forming a film on the grains and binding them together. Based on the analyses and studies con-ducted in the research, it can be concluded that there is technical feasibility for applying the pol-ymeric solution at both dosages in geotechnical projects.

Modern power system integrates more and more new energy and use a large number of power electronic equipment. This makes it face more challenges in online optimization and real-time control. Deep reinforcement learning(DRL) has the ability of processing big data and high-dimensional features, as well as the ability of independently learning and optimizing decision-making in complex environments. In this paper, we explore DRL based online combination optimization method of grid section for large complex power system. In our method, to improve the convergence speed of the model, we propose to discretize the output action of the unit and simplify the action space. We also design a reinforcement learning loss function with strong constraints to further improve the convergence speed of the model and facilitate the algorithm to obtain the stable solution. Moreover, to avoid the local optimal solution problem caused by the discretization of the output action, we propose to use the annealing optimization algorithm to make the granularity of the unit output finer. We verify our method on IEEE 118-bus system. The experimental results show that our model has fast convergence speed and better performance, and can obtain stable solutions.

The transportation industry is rapidly transitioning from Internal Combustion Engine (ICE) based vehicles to Electric Vehicles (EVs) to promote clean energy. However, large-scale adoption of EVs can compromise the reliability of the power grids by introducing large uncertainty in the demand. Demand response with a controlled charge scheduling strategy for EVs can mitigate such issues. In this paper, a deep reinforcement learning- based charge scheduling strategy is developed for individual EVs by considering user’s dynamic driving behavior and charging preferences. The temporal dynamics of user’s anxiety about charging the EV battery is rigorously addressed. A dynamic weight allocation technique is applied to continuously tune user’s priority for charging and cost-saving with respect to charging duration. The sequential charging control problem is formulated as a Markov decision process, and an episodic approach to the deep deterministic policy gradient (DDPG) algorithm with target policy smoothing and delayed policy update techniques is applied to develop the optimal charge scheduling strategy. A real-world dataset that captures user’s driving behavior, such as arrival time, departure time, and charging duration, is utilized in this study. The extensive simulation results reveal the effectiveness of the proposed algorithm in minimizing energy cost while satisfying user’s charging requirements.

For Synthetic Aperture Radar (SAR) image registration, successive processes following feature extraction are required by both the traditional feature-based method and the deep learning method. Among these processes, the feature matching process—whose time and space complexity are related to the number of feature points extracted from sensed and reference images, as well as the dimension of feature descriptors—proves to be particularly time-consuming. Additionally, the successive processes introduce data sharing and memory occupancy issues, requiring an elaborate design to prevent memory leaks. To address these challenges, this paper introduces the OptionEM-based reinforcement learning framework to achieve end-to-end SAR image registration. This framework outputs registered images directly without requiring feature matching and calculation of the transformation matrix, leading to significant processing time savings. The Transformer architecture is employed to learn image features, while a correlation network is introduced to learn the correlation and transformation matrix between image pairs. Reinforcement learning, as a decision process, can dynamically correct errors, making it more efficient and robust compared to supervised learning mechanisms like deep learning. We present a hierarchical reinforcement learning framework combined with episodic memory to mitigate the inherent problem of invalid exploration in generalized reinforcement learning algorithms. This approach effectively combines coarse and fine registration, further enhancing training efficiency. Experiments conducted on three sets of SAR images, acquired by TerraSAR-X and Sentinel-1A, demonstrate that the proposed method’s average runtime is sub-second, achieving subpixel registration accuracy.

In this work, we propose an approach for the autonomous navigation of mobile robots using fusion of sensor data by a Double Deep Q-Network with collision avoidance by detecting moving people via computer vision techniques. We evaluate two data fusion methods for the proposed autonomous navigation approach: Interactive and Late fusion strategy. Both are used to integrate mobile robot sensors through the following sensors: GPS, IMU and, an RGB-D camera. The proposed collision avoidance module is implemented along with the sensor fusion architecture in order to prevent the autonomous mobile robot from colliding with moving people. The simulation results indicate a significant impact on the success of completing the proposed mission by the mobile robot with the fusion of sensors, indicating a performance increase (success rate) of 27% in relation to navigation without sensor fusion. With the addition of moving people in the environment, deploying the people detection and collision avoidance security module has improved about 14% the success rate when compared to that of the autonomous navigation approach without the security module. Video was developed with robot navigation using the DDQN-Late Fusion https://www.loom.com/share/684afa6a5b0148afadc9a200ab9f3483.

In this study, reinforcement learning (RL) was used in factory simulation to optimize storage devices for use in Industry 4.0 and digital twins. First, we defined an RL environment, modeled it, and validated its ability to simulate a real physical system. Subsequently, we introduced a method to calculate reward signals and apply them to the environment to ensure the alignment of the behavior of the RL agent with the task objective. The stocker simulation model was used to validate the effectiveness of RL. The model is a storage device that simulates logistics in a manufacturing production area. The results revealed that RL is a useful tool for automating and optimizing complex logistics systems and increase the applicability of RL in logistics. We proposed a novel method for creating an agent through learning using the proximal policy optimization algorithm, and the agent was optimized by configuring various learning options. The application of reinforcement learning resulted in an effectiveness of 30% to 100%, and methods can be expanded to other fields.

Abstract—Online learning platforms provide diverse course resources, but this often result in the issue of information overload. Learners always want to learn courses that are appropriate for their knowledge level and preferences quickly and accurately. Effective course recommendation plays a key role in helping learners select appropriate courses and improving the efficiency of online learning. However, when a user is enrolled in multiple courses, Existing course recommendation methods face the challenge in accurately recommending the target course that is most relevant to the user, because of the noise courses. In this paper, we propose a novel reinforcement learning model named Actor-Critic Hierarchical Reinforcement Learning (ACHRL). The model incorporates the Actor-Critic method to construct the profile reviser. This can remove noise courses and make personalized course recommendation effectively. Furthermore, we propose a policy gradient based on temporal difference error to reduce the variance in the training process, to speed up the convergence of the model, and improves the accuracy of the recommendation. We evaluate the proposed model on two real datasets, and the experimental results show that the proposed model is significantly outperforms the existing recommendation models (improving 3.77% to 13.66% in terms of HR@5).

Evolutionary algorithms find applicability in reinforcement learning of neural networks due to their independence from gradient-based methods. To achieve successful training of neural networks using evolutionary algorithms, careful considerations must be made to select appropriate algorithms due to the availability of various algorithmic variations. In Part1 and Part2, the author previously reported experimental evaluations on Genetic Algorithm and Evolution Strategy for reinforcement learning of neural networks, utilizing the Acrobot control task. This article constitutes Part3 of the series of comparative research. In this study, Differential Evolution is adopted as the third instance of major evolutionary algorithms. The experimental results show a statistically significant superiority of DE over both GA and ES (p < .01). In addition, DE exhibits its robustness to variations in hyperparameter configurations (the number of offsprings and generations). In the previous experiments, both ES and GA showed significant performance differences depending on the configurations, whereas in the experiment reported in this article, such differences are not detected for DE.

In August 2020, the World Health Assembly launched a global initiative to eliminate cervical cancer by 2030, setting three primary targets. One key goal is to achieve a 70% screening coverage rate for cervical cancer, primarily relying on the precise analysis of Papanicolaou (Pap) or digital Pap smears. However, the responsibility of reviewing Pap smear samples to identify potentially cancerous cells primarily falls on pathologists, a task known to be exceptionally challenging and time-consuming. This paper proposes a solution to address the shortage of pathologists for cervical cancer screening. It leverages the OpenAI-GYM API to create a deep reinforcement learning environment utilizing liquid-based Pap smear images. By employing the Proximal Policy Optimization algorithm, autonomous agents navigate Pap smear images, identifying cells with the aid of rewards, penalties, and accumulated experiences. Furthermore, the use of a pre-trained convolutional neuronal network like Res-Net54 enhances the classification of detected cells based on their potential for malignancy. The ultimate goal of this study is to develop a highly efficient, automated Papanicolaou analysis system, ultimately reducing the need for human intervention in regions with limited pathologists.

Recently, smart mobility intelligent traffic services have become a critical task in Intelligent Transportation Systems (ITS). This involves not only the use of advanced sensors and controllers but also the ability to respond to real-time traffic situations at intersections, alleviate congestion, and generate policies to prevent traffic jams. DRL (Deep Reinforcement Learning) provides a natural framework for processing tasks. In DRL, each intersection can control itself and coordinate with neighbors to achieve optimal network-wide policies. However, comparing approaches remains a challenging task due to the existence of numerous possible configurations. This research performs a critical comparison of various traffic controllers found in the literature. It demonstrates that using a nonlinear approximator for coordination mechanisms and enhancing observability at each intersection are key performance drivers.

Machine learning offers advanced tools for efficient management of radio resources in modern wireless networks. In this study, we leverage a multi-agent deep reinforcement learning (DRL) approach, specifically the Parameterized Deep Q-Network (DQN), to address the challenging problem of power allocation and user association in massive multiple-input multiple-output (M-MIMO) communication networks. Our approach tackles a multi-objective optimization problem aiming to maximize network utility while meeting stringent quality of service requirements in M-MIMO networks. To address the non-convex and nonlinear nature of this problem, we introduce a novel multi-agent DQN framework. This framework defines a large action space, state space, and reward functions, enabling us to learn a near-optimal policy. Simulation results demonstrate the superiority of our Parameterized Deep DQN (PD-DQN) approach when compared to traditional DQN and RL methods. Specifically, we show that our approach outperforms traditional DQN methods in terms of convergence speed and final performance. Additionally, our approach shows 72.2 % and 108.5 % improvement over DQN methods and RL method respectively in handling large-scale multi-agent problems in M-MIMO networks.

The problem addressed in this article consists of the motion control of a quadrotor, comprising model disturbances and uncertainties. In order to tackle model uncertainty, adaptive control based on reinforcement learning is used. The distinctive feature of this article, in comparison with other works on quadrotor control using reinforcement learning is the exploration of the underlying optimal control problem in which a quadratic cost and a linear dynamics allow an algorithm that runs in real time. Instead of identifying a plant model, adaptation is obtained by approximating the performance index given by the Q-function using directional forgetting recursive least squares that rely on a linear regressor build from quadratic functions of input/output data. The adaptive algorithm proposed is tested in simulation in a cascade control structure that drives a quadrotor. Simulations show the improvement in performance that results when the proposed algorithm is turned on.

Deep reinforcement learning is one of the research hotspots in artificial intelligence and has been successfully applied in many research areas, however, the low training efficiency and high demand for samples are problems that limit the application To address these problems, a hierarchical episodic control model extending episodic memory to the domain of hierarchical reinforcement learning is proposed in this paper. The model is theoretically justified and employs a hierarchical implicit memory planning approach for counterfactual trajectory value estimation. Starting from the final step and recursively moving back along the trajectory, a hidden plan is formed within the episodic memory. Experience is aggregated both along trajectories and across trajectories, and the model is updated using a multi-headed backpropagation similar to bootstrapped neural networks. This model extends the parameterized episodic memory framework to the realm of hierarchical reinforcement learning and is theoretically analyzed to demonstrate its convergence and effectiveness. Experiments conducted in Four Room, Mujoco, and UE4-based active tracking , highlight that the hierarchical episodic control model effectively enhances training efficiency. It demonstrates notable improvements in both low-dimensional and high-dimensional environments, even in cases of sparse rewards.

Evolutionary algorithms and swarm intelligence algorithms find applicability in reinforcement learning of neural networks due to their independence from gradient-based methods. To achieve successful training of neural networks using these algorithms, careful considerations must be made to select appropriate algorithms due to the availability of various algorithmic variations. In Part1, 2 and 3, the author previously reported experimental evaluations on Evolution Strategy, Genetic Algorithm, and Differential Evolution for reinforcement learning of neural networks, utilizing the Acrobot control task. This article constitutes Part4 of the series of comparative research. In this study, Particle Swarm Optimization is adopted as an instance of major swarm intelligence algorithms. The experimental result shows that PSO performed worse than all of DE, GA and ES. The difference between PSO and DE was statistically significant (p<.01). In addition, PSO exhibited lower capability in exploring solutions in high-dimensional search spaces than DE, GA, and ES did. A larger swarm size compensated for the weakness of PSO in global exploration, thus making itself more beneficial than a larger number of swarm search iterations.

The application of space robotic manipulators and heightened autonomy for ios represents a paramount pursuit for leading space agencies, given the substantial threat posed by space debris to operational satellites and forthcoming space endeavors. This work presents a guidance algorithm based on drl to solve for the space manipulator path-planning during the motion synchronization phase with the mission target. The goal is the trajectory generation and control of a spacecraft equipped with a 7-dof robotic manipulator, such that its end effector remains stationary with respect to the target point of capture. The ppo drl algorithm is used to optimize the manipulator’s guidance law, and the autonomous agent generates the desired joints rates of the robotic arm, which are then integrated and passed to a model-based feedback linearization controller. The agent is first trained to optimize its guidance policy and then tested extensively to validate the results against a simulated environment representing the motion synchronization scenario of an ios mission.

With the proliferating of video surveillance system deployment and related applications, real-time video analysis is very critical to achieve intelligent monitoring, autonomous driving, etc. It is non-trivial to achieve high accuracy and low latency video stream analysis through the traditional cloud computing. In this paper, we propose a non-orthogonal multiple access (NOMA) based edge real-time video analysis framework with one edge server (ES) and multiple user equipments (UEs). A cost minimization problem composed of delay, energy and accuracy is formulated to improve the QoE of UEs. In order to efficiently solve this problem, we propose the joint video frame resolution scaling, task offloading, and resource allocation algorithm based on the Deep Q-Learning Network (JVFRS-TO-RA-DQN), which effectively overcomes the sparsity of the single-layer reward function and accelerates the training convergence speed. JVFRS-TO-RA-DQN consists of two DQN networks to reduce the curse of dimension, which respectively select the offloading and resource allocation action, the resolution scaling action. Experimental results show that JVFRS-TO-RA-DQN can effectively reduce the cost of transmission and computation, and have better performance in convergence compared to other baseline.

As a new plant slope protection technology, living stumps can effectively improve slope stability. The current study attemptes to investigate the mechanical characteristics of the root system of living stumps and their reinforcement mechanism on slopes by carrying out a three-dimensional living stump slope modeling test. Using ABS materials, a 3D model of a living elm stump was created by 3D printing and a slope model test was carried out. The reinforcement mechanism of the living stump was studied through a combination of model testing and numerical simulation. The results indicate that, due to the presence of living stumps in the lower and middle parts of the slope, the maximum shear stress area of the soil moved to a deeper direction. The stress state of the soil around the living stump was effectively improved due to the existence of the lateral root system.Living stumps in the lower part of the slope cross the potential sliding surface and transferred the soil shear stress gradually to the root system through root-soil interaction. Moreover, the tap roots and lateral roots of living stumps form a robust spatial structure that bears the shear stress of soil together, thus increasing the stability of the slope.

The convergence of artificial intelligence and the Internet of Things (IoT) has made remarkable strides in the realm of industry. The IoT devices in this computing model gather data from various sources and send it to edge servers for real-time processing, which presents a challenge to the existing message queue systems in Artificial Intelligence of Things (AIoT) Edge computing. These systems lack the adaptability to respond to the current state of the system, such as changes in the number of devices, message size, and frequency, and optimize the message transmission mechanism accordingly. Hence, it is critical to devise an approach that can effectively decouple message processing and mitigate workload fluctuations in the AIoT computing environment. To this end, this study introduces a distributed message queue system that is specifically tailored to the AIoT edge environment, utilizing a reinforcement learning approach to optimize message queue performance. Empirical findings reveal that this pioneering method significantly improves system throughput while handling varying message scenarios.

Recent years have witnessed significant advances in reinforcement learning (RL), which has registered remarkable success in solving various sequential decision-making problems in machine learning. Most of the successful RL applications, e.g., the games of Go and Poker, robotics, and autonomous driving, involve the participation of more than one single agent, which naturally fall into the realm of multi-agent RL (MARL), a domain with a relatively long history, and has recently re-emerged due to advances in single-agent RL techniques. Although, typically, the communication protocol between agents is manually specified and not altered during training, recently, some papers have shown signs of trying to emerge a communication between agents on the one hand and, on the other hand, to understand what is exchanged between agents. So, there is a growing body of literature on this topic which includes qualitative and quantitative studies and the ones that apply mixed methods. This study presents the scoping review of the methodological strategies undertaken in a total of 16 research articles. The results present the critical appraisal of quantitative methods in terms of validity and reliability and for qualitative methods considering four trustworthiness factors. In the end, relevant insights are further explored with implications and reflections on how they can benefit one's research in the field.

In the history of high-performance polymer composite parts with continuous in-plane fibre reinforcement, the comparatively moderate structural-mechanical properties in their out-of- plane-z-direction and the limited load carrying capability between individual reinforcement layers within the laminated structures have always been one of the greatest challenges. Plenty of research work has been carried out to improve load transfer capability as well as fracture toughness for crack opening events between the reinforcement layers by means of out-of-plane fibre reinforcement implementation, for example by sewing or z-pinning technologies. However, the results have not become established across a wide range of applications so far, because the technologies used have led to considerable losses of in-plane strength properties. In this paper first results are presented for a new method of a low fraction z-fibre reinforcement by means of very thin diameter laser drilled holes filled with short carbon fibres, demonstrating potential to improve Mode II fracture toughness.

The working environment of mining electric locomotives is wet and muddy coal mine 1 roadway. There may be idling or slipping between the wheels and rails of mining electric locomotives 2 due to low friction between the wheel and rail and insufficient utilization of creep rate. Therefore, it 3 is necessary to control the creep rate within a reasonable range. In this paper, the autonomous control 4 algorithm for mining electric locomotives based on improved ε-greedy is theoretically proven to be 5 convergent and effective firstly. Secondly, after analyzing the contact state between the wheel and rail 6 under wet and slippery road conditions, it was concluded that the value of creep rate is an important 7 factor affecting the autonomous driving of mining electric locomotives. Therefore, the autonomous 8 control method for mining electric locomotives based on creep control is proposed in this paper. 9 Finally, the effectiveness of the proposed method was verified through simulation. The problem of 10 wheel slipping and idling caused by insufficient friction of mining electric locomotives in coal mining 11 environments is effectively suppressed. Autonomous operation of vehicles with optimal driving 12 efficiency can be achieved through quantitative control and utilization of the creep rate between 13 wheels and rails

In this paper we present the exploration of indoor positioning technologies for UAVs, and navigation techniques for path planning and obstacle avoidance. The objective is to perform warehouse inventory tasks using a drone to search for barcodes or markers to identify the objects. For the indoor positioning techniques, we employ Visual-Inertial Odometry (VIO), Ultra-Wideband (UWB), AprilTag fiducial markers, and Simultaneous Localization and Mapping (SLAM). These algorithms encompass global positioning, local positioning, and pre-mapping positioning, comparing the merits and drawbacks of various techniques and trajectories. In UAV navigation, we combine SLAM-based RTAB-Map indoor mapping and navigation path planning of the ROS for indoor environments. This system enables precise drone positioning indoors and utilizes global and local path planners to generate flight paths that avoid dynamic, static, unknown, and known obstacles, demonstrating high practicality and feasibility. To achieve the warehouse inventory inspection, a reinforcement learning approach is proposed to recognize the markers by adjusting the UAV's viewpoint. We address several main problems in inventory management, including efficiently planning paths while ensuring a certain level of detection rate. Two reinforcement learning techniques, AC (Actor-Critic) and PPO (Proximal Policy Optimization), are implemented based on AprilTag identification. Testing is performed in both simulated and real-world environments, and the effectiveness of the proposed method is validated.Code is available at https://github.com/kellen080/Navigation and https://github.com/kellen080/Indoor\_Positioning.