Aiming at the problem of model error and tracking dependence in the process of intelligent vehicle motion planning, an intelligent vehicle model transfer trajectory planning method based on deep reinforcement learning is proposed, which obtain an effective control action sequence directly. Firstly, an abstract model of the real environment is extracted. On this basis, Deep Deterministic Policy Gradient (DDPG) and vehicle dynamic model are adopted to jointly train a reinforcement learning model, and to decide the optimal intelligent driving maneuver. Secondly, the actual scene is transferred to equivalent virtual abstract scene by transfer model, furthermore, the control action and trajectory sequences are calculated according to trained deep reinforcement learning model. Thirdly, the optimal trajectory sequence is selected according to evaluation function in the real environment. Finally, the results demonstrate that the proposed method can deal with the problem of intelligent vehicle trajectory planning for continuous input and continuous output. The model transfer method improves the model generalization performance. Compared with the traditional trajectory planning, the proposed method output continuous rotation angle control sequence, meanwhile, the lateral control error is also reduced.

In the history of high-performance polymer composite parts with continuous in-plane fibre reinforcement, the comparatively moderate structural-mechanical properties in their out-of- plane-z-direction and the limited load carrying capability between individual reinforcement layers within the laminated structures have always been one of the greatest challenges. Plenty of research work has been carried out to improve load transfer capability as well as fracture toughness for crack opening events between the reinforcement layers by means of out-of-plane fibre reinforcement implementation, for example by sewing or z-pinning technologies. However, the results have not become established across a wide range of applications so far, because the technologies used have led to considerable losses of in-plane strength properties. In this paper first results are presented for a new method of a low fraction z-fibre reinforcement by means of very thin diameter laser drilled holes filled with short carbon fibres, demonstrating potential to improve Mode II fracture toughness.

Plasma technology shows tremendous potential for revolutionizing oncology research and treatment. Reactive oxygen and nitrogen species, electromagnetic emissions generated through gas plasma jets, have attracted significant attention due to their selective cytotoxicity towards cancer cells. To leverage the full potential of plasma medicine, researchers have explored the use of mathematical models and various subsets of machine learning, such as reinforcement learning, and deep learning. This review emphasizes the significant application of AI algorithms in the adaptive plasma system, paving the way for precision and dynamic cancer treatment. Realizing the full potential of AI in plasma medicine, requires research efforts, data sharing and interdisciplinary collaborations. Unravelling the complex mechanisms, developing real-time diagnostics, and optimizing AI models will be crucial to harness the true power of plasma technology in oncology. The integration of personalized and dynamic plasma therapies, alongside AI and diagnostic sensors, presents a transformative approach to cancer treatment with the potential to improve outcomes globally.

Abstract:The objects of fine-grained image categories(e.g., bird species) are various subclass under different categories. Because the differences between subclass are very subtle and most of them are concentrated in multiple local areas, the task of fine-grained image recognition is very challenging. At the same time, some fine-grained networks tend to focus on a certain region when judging the target category, resulting in the lack of other auxiliary regional features. To this end, Inception V3 is used as the backbone network, and an enhanced and complementary fine-grained image classification network is designed. While adopting the method of reinforcement learning to obtain more detailed fine grain image features, the complementary network can obtain the complementary discriminant area of the target through the method of attention erasure to increase the network's perception of the overall target. Finally, experiments are conducted on CUB-200-2011, FGVC Aircraft and Stanford dogs three open datasets. The experimental results show that the proposed model has better performance.

The opened beams always confused the designers due to the guidelines missing. In this research, six hybrid reinforced beams reinforced with mixed steel and basalt fiber reinforced polymer (BFRP) bars had constant cross-sections of 150mm x 300mm and a clear span of 1800mm. Generally, five beams have symmetrical rectangular openings with dimensions of 150mm x 250mm located at a distance of 250mm (equivalent to the beam effective depth) from the beam support in addition to a solid beam that is served as a control reference. The studied parameters included the effect of using internal reinforcement (steel or BFRP bars) provided along the opening or by incorporating an external BFRP sheet around the opening corners. Also, the conduction of double enhancement with internal steel reinforcement bars further external strengthening BFRP sheet was investigated. The relevant results showed that the opened beam without enhancement lost almost 74.66% of the maximum load compared with the solid beam. Placing internal steel or BFRP bars around the openings increased the maximum load by 62.07% and 59.68%, respectively in comparison to the non-enhanced opened beams. Using an external BFRP sheet to strengthen the opening corners of the beam enhanced the maximum load by 76.39% compared with the non-enhanced opened beam. Therefore, if the beam double enhancement with an external BFRP sheet and internal steel reinforcement around the openings, the maximum load increased by 137.40% compared with the non-enhanced opened beam. Ultimately to further analyze the experimental results and confirm their findings, the study was extended to include the numerical analysis using three dimensional finite element modeling and the results correlated very well with the experimental ones.

In the past decades, the world is actively working towards the global goal of net-zero emission. To decrease emissions, there is a notable trend of electrification in transportation, which is a transition from traditional fuel-based systems to electrical power systems onboard different transportation platforms. For this transition, it is important to study the electrical structure, specifically the onboard microgrid, powered by various energy sources. In this paper, traditional energy management strategies for onboard microgrid systems are discussed, which usually require complicated optimization algorithms and high computation capabilities. Driven by the recent advancements in information technologies, artificial intelligence and digital twin have gained much interest within the transportation sector. These technologies can effectively utilize data to achieve intelligent decision-making, optimize resource utilization, and save energy consumption. This paper presents an overview of the usage of these emerging information technologies in energy management strategies, providing an overall summary and classification of the practical applications. In addition, after examining the potential challenges associated with artificial intelligence and digital twin, this paper also discusses future trends in this field.

Attitude control of a novel regional truss-braced wing aircraft with low stability characteristics is addressed in this paper using Reinforcement Learning (RL). In recent years, RL has been increasingly employed in challenging applications, particularly, autonomous flight control. However, a significant predicament confronting discrete RL algorithms is the dimension limitation of the state-action table and difficulties in defining the elements of the RL environment. To address these issues, in this paper, a detailed mathematical model of the mentioned aircraft is first developed to shape an RL environment. Subsequently, Q-learning, the most prevalent discrete RL algorithm will be implemented in both the Markov Decision Process (MDP), and Partially Observable Markov Decision Process (POMDP) frameworks to control the longitudinal mode of the air vehicle. In order to eliminate residual fluctuations that are a consequence of discrete action selection, and simultaneously track variable pitch angles, a Fuzzy Action Assignment (FAA) method is proposed to generate continuous control commands using the trained Q-table. Accordingly, it will be proved that by defining an accurate reward function, along with observing all crucial states (which is equivalent to satisfying the Markov Property), the performance of the introduced control system surpasses a well-tuned Proportional–Integral–Derivative (PID) controller.

The use of digital twins for the development of Autonomous Maritime Surface Vessels (AMSVs) has enormous potential to resolve the increasing need for water-based navigation and safety at the seas. Aiming at the problem of lack of broad and integrated digital twin implementations with live data along with the absence of a digital twin-driven framework for AMSV design and development, an application framework for the development of a fully autonomous vessel using an integrated digital twin in a 3D simulation environment has been presented. Our framework has four layers to ensure that the simulation and the real-world boat as well as the environment are as close as possible. Åboat, an experimental research platform for maritime automation and autonomous surface ship applications, equipped with two trolling electric motors, cameras, LiDARs, IMU and GPS has been used as the case study to provide a proof of concept. Åboat and its sensors, alongwith the environment have been replicated in a 3D simulation environment. Using the proposed application framework, we develop obstacle detection and path planning systems based on machine learning which leverage live data from a 3D simulation environment to mirror the complex dynamics of the real world.

Online reinforcement learning (RL) algorithms are increasingly used to personalize digital interventions in the fields of mobile health and online education. Common challenges in designing and testing an RL algorithm in these settings include ensuring the RL algorithm can learn and run stably under real-time constraints, and accounting for the complexity of the environment, e.g., a lack of accurate mechanistic models for the user dynamics. To guide how one can tackle these challenges, we extend the PCS (Predictability, Computability, Stability) framework, a data science framework that incorporates best practices from machine learning and statistics in supervised learning (Yu and Kumbier, 2020), to the design of RL algorithms for the digital interventions setting. Further, we provide guidelines on how to design simulation environments, a crucial tool for evaluating RL candidate algorithms using the PCS framework. We illustrate the use of the PCS framework for designing an RL algorithm for Oralytics, a mobile health study aiming to improve users' tooth-brushing behaviors through the personalized delivery of intervention messages. Oralytics will go into the field in late 2022.

To improve the interfacial bonding of sisal fiber reinforced polylactide biocomposites, polylactide (PLA) and sisal fibers (SF) were melt-blended to fabricate bio-based composites via in situ reactive interfacial compatibilization with the addition of an epoxy-functionalized oligomer (ADR). The FTIR analysis and SEM characterization demonstrated that PLA molecular chain was bonded to the fiber surface and epoxy-functionalized oligomer played a hinge-like role between sisal fibers and PLA matrix, which resulted in improved interfacial adhesion between fibers and PLA matrix. The interfacial reaction and microstructures of composites were further investigated by thermal and rheological analyses, which indicated that the mobility of the PLA molecular chain in composites was restricted because of the introduction of ADR oligomer, which in turn reflected the improved interfacial interaction between SF and PLA matrix. These conclusions were further investigated by the calculated activation energies of glass transition relaxation (△Ea) of composites via dynamic mechanical analysis. The mechanical properties of PLA/SF composites were simultaneously reinforced and toughened via addition of ADR oligomer. The interfacial interaction and structure-properties relationship of composites are key points of this study.

The transportation industry is rapidly transitioning from Internal Combustion Engine (ICE) based vehicles to Electric Vehicles (EVs) to promote clean energy. However, large-scale adoption of EVs can compromise the reliability of the power grids by introducing large uncertainty in the demand. Demand response with a controlled charge scheduling strategy for EVs can mitigate such issues. In this paper, a deep reinforcement learning- based charge scheduling strategy is developed for individual EVs by considering user’s dynamic driving behavior and charging preferences. The temporal dynamics of user’s anxiety about charging the EV battery is rigorously addressed. A dynamic weight allocation technique is applied to continuously tune user’s priority for charging and cost-saving with respect to charging duration. The sequential charging control problem is formulated as a Markov decision process, and an episodic approach to the deep deterministic policy gradient (DDPG) algorithm with target policy smoothing and delayed policy update techniques is applied to develop the optimal charge scheduling strategy. A real-world dataset that captures user’s driving behavior, such as arrival time, departure time, and charging duration, is utilized in this study. The extensive simulation results reveal the effectiveness of the proposed algorithm in minimizing energy cost while satisfying user’s charging requirements.

Aiming at the problem of hypersonic morphing vehicle avoiding no-fly zones and reaching the target, an improved predictor-corrector guidance method is proposed. Firstly, the aircraft motion model and the constraint model are established. Then, the basic algorithm is given, the Q-learning method is used to design the attack angle and sweep angle scheme to ensure that the aircraft can fly over the low-altitude zones. The B-spline curve is used to design the location of flight path points and the bank angle scheme is designed according to the predictor-corrector method, so that the aircraft can fly around to avoid high-altitude zones. Next, Monte Carlo reinforcement learning(MCRL) method is used to improve predictor-corrector method and Deep Neural Network(DNN) is used to fit reward function. The improved method can generate trajectory with better performance. Simulation results verify the effectiveness of the proposed algorithm.

In tunnel excavation with boring machines, the tunnel face is supported to avoid collapse and minimise settlement. This article proposes the use of reinforcement learning, specifically the Deep Q-Network algorithm, to predict the face support pressure. The approach is tested both analytically and numerically. By using the soil properties ahead of the tunnel face and the overburden depth as the input, the algorithm is capable of predicting the optimal tunnel face support pressure, adapting to changes in geological and geometrical conditions.

The development of synthetic biology has enabled us to make massive progress on biotechnology and to approach research questions from a brand new perspective. In particular, the design and study of gene regulatory networks in vitro, in vivo and in silico, have played an increasingly indispensable role in understanding and controlling biological phenomena. Among them, it is of great interest to understand how associative learning is formed at the molecular circuit level. Noticeably, mathematical models have been increasingly used to predict the behaviors of molecular circuits. The Fernando’s model, which is thought to be one of the first works in this line of research using the Hill equation, attempted to design a synthetic circuit that mimics Hebbian learning in the neural network architecture. In this article, we carry out in-depth computational analysis of the model and demonstrate that the reinforcement effect can be achieved by choosing the proper parameter values. We also construct a novel circuit that can demonstrate forced dissociation, which was not observed in the Fernando’s model. Our work can be readily used as reference for synthetic biologists who consider implementing the circuits of this kind in biological systems.

Serious damage caused by floor heave in the coal given chamber of a vertical coal bunker is one of the challenges faced in underground coal mines. Engineering practice shows that it is more difficult to maintain the coal given chamber (CGC) than a roadway. More importantly, repairing the CGC during mining practice will pose major safety risks and reduce production. Based on the case of the serious collapse that occurred in the bearing structure of the CGC at the lower part of the 214# coal bunker in Xiashijie mine, China, this work analysed (i) the main factors influencing floor heave and (ii) the failure mechanism of the load-bearing structure in the CGC using FLAC^2D numerical models and expansion experiment. The analysis results indicate that: the floor heave, caused mainly by mine water, is the basic reason leading to the instability and repeated failure of the CGC in the 214# coal bunker. Then a new coal bunker, without building the CGC, is proposed and put into practice to replace the 214# coal bunker. The FLAC^3D software program is adopted to establish the numerical model of the wall-mounted coal bunker (WMCB), and the stability of the rock surrounding the WMCB is simulated and analysed. The results show that: (1) the rock surrounding the sandstone segment is basically stable. (2) The surrounding rock in the coal seam segment, which moves into the inside of the bunker, is the main zone of deformation for the entire rock mass surrounding the bunker. Then the surrounding rock is controlled effectively by means of high-strength bolt–cable combined supporting technology. According to the geological conditions of the WMCB, the self-bearing system, which includes (i) H-steel beams, (ii) H-steel brackets, and (iii) self-locking anchor cables, is established and serves as a substitute for the CGC to transfer the whole weight of the bunker to stable surrounding rock. The stability of the new coal bunker has been verified by field testing, and the coal mine has gained economic benefit to a value of 158.026174 million RMB over three years. The new WMCB thus made production more effective and can provide helpful references for construction of vertical bunkers under similar geological conditions.

This systematic literature review explores the burgeoning use of Artificial Intelligence (AI) in manufacturing systems, in line with the principles of Industry 4.0 and the growth of smart factories. In this landscape, AI is crucial in addressing the complexity and dynamism of contemporary manufacturing processes, including machine breakdowns, fluctuating orders and unpredictable job arrivals. This systematic literature review, conducted using the Scopus database and bibliometric tools, pursues two primary objectives. First, it identifies the prevailing trends in solving scheduling problems with AI and identifies the most commonly used AI techniques in the literature. Secondly, it analyses how authors have successfully employed AI to address production scheduling challenges in real-world industrial settings and assesses the benefits obtained by companies. The dynamic nature of manufacturing systems requires adaptive scheduling paradigms. AI, including Particle Swarm Optimization, Neural Networks, and Reinforcement Learning, is applied to optimize production processes, predict machine failures, and achieve substantial benefits. In real-world applications, these AI-driven solutions have led to reduced production costs, enhanced energy efficiency, and more efficient scheduling processes. AI is increasingly recognized as an essential tool in addressing the evolving challenges of modern manufacturing environments.

Electronic devices are sensitive to electromagnetic (EM) emissions requiring electromagnetic shielding protection to assure good operation, preventing noise, malfunctioning, or even burning. To assure protection, it is important to develop suitable material and design solutions for electronic enclosures. Most common enclosures are made with metal alloys by traditional manufacturing methods. However, resourcing to thermoplastic composites combined with additive manufac-turing (AM) technologies emerges as an alternative that enables to fabricate complex parts that are lightweight, consolidated and oxidation and corrosion resistant. In this research, an AM technique based in material extrusion was used to print 2 mm thick specimens with a multi-material made of micro-carbon fiber (CF) filled polyamide, reinforced, at specific layers, by continuous carbon fibers stacked with a 90° rotation to each other. Specimens electromagnetic shielding effectiveness (EMSE) was evaluated in the frequency band of 0.03 – 3 GHz by coaxial transmission line method. De-pending on the number of CF layers, the EM shielding obtained can up to 70 dB, with a specific shielding up to 60 dB.cm3/g, predominantly by absorption mechanism, being 22 times higher that without the CF layers. These findings fundament this innovative approach for lightweight cus-tomizable solutions for EM shielding applications.

With the proliferating of video surveillance system deployment and related applications, real-time video analysis is very critical to achieve intelligent monitoring, autonomous driving, etc. It is non-trivial to achieve high accuracy and low latency video stream analysis through the traditional cloud computing. In this paper, we propose a non-orthogonal multiple access (NOMA) based edge real-time video analysis framework with one edge server (ES) and multiple user equipments (UEs). A cost minimization problem composed of delay, energy and accuracy is formulated to improve the QoE of UEs. In order to efficiently solve this problem, we propose the joint video frame resolution scaling, task offloading, and resource allocation algorithm based on the Deep Q-Learning Network (JVFRS-TO-RA-DQN), which effectively overcomes the sparsity of the single-layer reward function and accelerates the training convergence speed. JVFRS-TO-RA-DQN consists of two DQN networks to reduce the curse of dimension, which respectively select the offloading and resource allocation action, the resolution scaling action. Experimental results show that JVFRS-TO-RA-DQN can effectively reduce the cost of transmission and computation, and have better performance in convergence compared to other baseline.

Ever-increasing need for improving the livability of a city and improve outcomes for its residents, over the last decade, the adoption of technology to develop urbanised societies around the world has given rise to the need for developing smart cities. The speed at which the world population is growing, the use of Internet of Things in smart cities have really advanced the quality of life. One significant area of concern within the smart city framework is waste management. If the waste within a city is not adequately managed, then it leads to issues in the health of the citizens. Additionally, the waste management has such a high impact on the environmental footprint, hence the need to have a smart way of managing waste is of critical importance. Through our research, we analyse the challenges of waste management within a city to understand the impact of the problem on to the citizens and overall city operations. We then investigate ways in which we can solve these problems using the emerging technologies, such as the Internet of Things, to collect valuable data of large volumes arriving at an astronomical rate, then apply multi-agent deep reinforcement learning algorithms to harness the power of big data to extract meaningful information and actionable insights. We ingest data generated by our Internet of Things into our algorithm for three main purposes including providing the notifications to an external system, for example, a map navigation engine out of the scope for this project but a future extension for route optimisation and waste vehicle tracking; extracting and reporting the actionable insights from the underlying data; and consuming the extracted data for predictive forecasting to draw out the unknown patterns of waste fill levels within various geographical locations and again send out triggers and notification to external systems for example a waste collection authority who can efficiently schedule the waste collection vehicles and optimise the route. To achieve the above mentioned outcomes, we propose a framework that is agnostic of the hardware that it connects to and can effectively interface with a wide variety of hardware keeping a level of abstraction in the architecture.

The collision avoidance mechanism adopted by the IEEE 802.11 standard is not optimal. The mechanism employs a binary exponential backoff (BEB) algorithm in the medium access control (MAC) layer. Such an algorithm increases the backoff interval whenever a collision is detected to minimize the probability of subsequent collisions. However, the expansion of the backoff interval causes degradation of the radio spectrum utilization (i.e., bandwidth wastage). That problem worsens when the network has to manage the channel access to a dense number of stations, leading to a dramatic decrease in network performance. Furthermore, a wrong backoff setting increases the probability of collisions such that the stations experience numerous collisions before achieving the optimal backoff value. Therefore, to mitigate bandwidth wastage and, consequently, maximize the network performance, this work proposes using reinforcement learning (RL) algorithms, namely Deep Q Learning (DQN) and Deep Deterministic Policy Gradient (DDPG), to tackle such an optimization problem. As for the simulations, the NS-3 network simulator is used along with a toolkit known as NS3-gym, which integrates a reinforcement-learning (RL) framework into NS-3. The results demonstrate that DQN and DDPG have much better performance than BEB for static and dynamic scenarios, regardless of the number of stations. Moreover, the performance difference is amplified as the number of stations increases, with DQN and DDPG showing a 27% increase in throughput with 50 stations compared to BEB. Furthermore, DQN and DDPG presented similar performances.

The steep rise in reinforcement learning (RL) in various applications in energy as well as the penetration of home automation in recent years are the motivation for this article. It surveys the use of RL in various home energy management system (HEMS) applications. There is a focus on deep neural network (DNN) models in RL. The article provides an overview of reinforcement learning. This is followed with discussions on state-of-the-art methods for value, policy, and actor–critic methods in deep reinforcement learning (DRL). In order to make the published literature in reinforcement learning more accessible to the HEMS community, verbal descriptions are accompanied with explanatory figures as well as mathematical expressions using standard machine learning terminology. Next, a detailed survey of how reinforcement learning is used in different HEMS domains is described. The survey also considers what kind of reinforcement learning algorithms are used in each HEMS application. It suggests that research in this direction is still in its infancy. Lastly, the article proposes four performance metrics to evaluate RL methods.

Deep reinforcement learning (DRL) has achieved significant results in many Machine Learning (ML) benchmarks. In this short survey we provide an overview of DRL applied to trading on financial markets, including a short meta-analysis using Google Scholar, with an emphasis on using hierarchy for dividing the problem space as well as using model-based RL to learn a world model of the trading environment which can be used for prediction. In addition, multiple risk measures are defined and discussed, which not only provide a way of quantifying the performance of various algorithms, but they can also act as (dense) reward-shaping mechanisms for the agent. We discuss in detail the various state representations used for financial markets, which we consider critical for the success and efficiency of such DRL agents. The market in focus for this survey is the cryptocurrency market.

In recent years, advancements in brain science and neuroscience have significantly influenced the field of computer science, particularly in the domain of reinforcement learning (RL). Drawing insights from neurobiology and neuropsychology, researchers have leveraged these findings to develop novel mechanisms for understanding intelligent decision-making processes in the brain. Concurrently, the emergence of quantum computing has opened new frontiers in artificial intelligence, leading to the development of quantum machine learning (QML). This study introduces a novel model that integrates Quantum Spiking Neural Networks (QSNN) and Quantum Long Short-Term Memory (QLSTM) architectures, inspired by the complex workings of the human brain. Specifically designed for reinforcement learning tasks in energy-efficient environments, our approach progresses through two distinct stages mirroring sensory and memory systems. In the initial stage, analogous to the brain’s hypothalamus, low-level information is extracted to emulate sensory data processing patterns. Subsequently, resembling the hippocampus, this information is processed at a higher level, capturing and memorizing correlated patterns. We conduct a comparative analysis of our model against existing quantum models such as Quantum Neural Networks and their classical counterparts, elucidating its unique contributions. Through empirical results, we aim to underscore the effectiveness of our approach in optimizing energy consumption in practical scenarios.

Continuous greenhouse gas emissions are causing global warming and impacting the habitats of many animals. Researchers in the field of electric power are making efforts to mitigate this situation. Operating and maintaining the power grid in an economic, low-carbon, and stable is challenging. To address the issue, we propose a grid dispatching technique that combines prediction technology, reinforcement learning, and optimization technology. Prediction technology can forecast future power demand and solar power generation, while reinforcement learning and optimization technology can make charging and discharging decisions for energy storage devices based on current and future grid conditions. In the power system, the aggregation of distributed energy resources increases uncertainty, particularly due to the fluctuating generation of renewable energy. This requires the use of advanced predictive control techniques to ensure long-term economic and decarbonization goals. In this paper, we present a real-time dispatching framework that integrates deep learning-based prediction, reinforcement learning-based decision-making, and stochastic optimization techniques. The framework can rapidly adapt to target uncertainty caused by various factors in real-time data distribution and control processes. The proposed framework achieved global Champion in the NeurIPS Challenge 2022 competition and demonstrated its effectiveness in practical scenarios of intelligent building energy management.

It is necessary to address the scarcity of crushed stones for pavement structural layers. So fly ash can be proved to be promising solution as more than 270 tonnes of fly ash is generated in India. Though, numerous research has been conducted for the use of fly ash intreated and untreated form, high volume of fine particles and brittleness of the stabilized fly ash pose a great problem for its use in subbase and base. Moreover, stiffness or modulus of stabilized fly ash is vital elastic parameter which is used for mechanistic pavement design. Hence, in this study an extensive experimental investigation is carried out to study its strength and stiffness properties such as compressive strength, indirect tensile strength and flexural strength, cyclic indirect tensile modulus and flexural modulus of fiber reinforced cement stabilized fly ash, stone dust, aggregate mixtures. The stone dust and aggregates have been added to enhance the gradation of the composite’s mixture. The study presents the effect of fiber on strength and stiffness properties. The experimental result reveals that addition of polypropylene (PP) fibers up to 0.25 wt.% enhances the compressive strength and any further addition of fiber results in decrease of the strength. However, indirect tensile strength and flexural strength increases with increase in fiber percentage up to 0.5 wt.%. Cement content is observed to be the dominant parameter for stabilized materials. Suitable relationships have been developed between strength and modulus parameters for stabilized mixtures. Based on the strength and stiffness study, 70% fly ash and 30% stone dust-aggregate and 60% fly ash and 40% stone dust-aggregate with 6% cement can be considered for the base layer. Based on the indirect tensile strength and flexural strength behavior, 0.35% is considered as the optimum fiber percentage.

In this paper, we consider a reconfigurable intelligent surface (RIS)-assisted integrated satellite-high altitude platform-terrestrial networks (IS-HAP-TNs) that can improve network performance by exploiting HAP's stability and RIS's reflection. Specifically, the reflector RIS is installed on the side of HAP to reflect signals from the multiple ground user equipments (UEs) to the satellite. To aim at maximising system sum rate, we jointly optimize the transmit beamforming matrix at the ground UEs and RIS phase shift matrix. Due to the limitation of the unit modulus of the RIS reflective elements constraint, the combinatorial optimization problem is difficult to tackle it effectively by traditional solving methods. Based on this, this paper studies deep reinforcement learning (DRL) algorithm to achieve online decision making for this joint optimization problem. In addition, it is verified through simulation experiments that the proposed DRL algorithm outperforms the standard scheme in terms of system performance and execution time, and higher computing speed, making real-time decision making truly feasible。

This work describes a new transducer prototype for continuous monitoring both in the structural and geotechnical fields. The transducer is synthetically constituted by a wire of optical fiber embedded between two fiber tapes (fiberglass or carbon fiber) and glued by a matrix of polyester resin. The fiber optical wire ends have been connected to a control unit whose detection system is based on Brillouin optical time-domain frequency analysis. Three laboratory tests were carried out to evaluate the sensor's reliability and accuracy. In each experiment, the transducer was applied to a sample of inclinometer casing sets in different configurations and with different constraint conditions. The experimental data collected were compared with theoretical models and with data obtained from the use of different measuring instruments to perform validation and calibration of the transducer at the same time. Several diagrams allow comparing the transducer and highlighting its suitability for monitoring and maintenance of structures. The characteristic of the transducer suggests its use as a mixed system for reinforcing and monitoring, especially in lifetime maintenance of critical infrastructures such as transportation and service networks, and historical heritage.

Among all the active confinement techniques, the use of pre-tensioned stainless steel straps has recently gained much attention. The flexibility of the stainless steel straps allows us to bend and pass them through the thickness of the masonry, thus creating a three-dimensional strengthening system between the two opposite facings. The use of the same perforation for the passage of several straps closed in a loop generates a continuous strengthening system that prevents parts of the structure from falling and injuring the occupants during seismic events. However, the perforations can nullify the in-plane strengthening, as they act as cylindrical hinges and make the reinforcement system labile for certain strap arrangements. Diagonal compression tests on square masonry panels performed in the present study show that the straps improve neither strength nor ductility when running along the mortar head and bed joints, arranged in square meshes. Conversely, they improve both strength and ductility when the straps make angles of 45° with the mortar joints. Furthermore, the experimental results show that the straps exert an anisotropic effect that decreases the apparent in-plane Poisson ratio. They also provide new insights into the diagonal compression test and allow formulating a new proposal for the pseudo-ductility factor.

Recently, smart mobility intelligent traffic services have become a critical task in Intelligent Transportation Systems (ITS). This involves not only the use of advanced sensors and controllers but also the ability to respond to real-time traffic situations at intersections, alleviate congestion, and generate policies to prevent traffic jams. DRL (Deep Reinforcement Learning) provides a natural framework for processing tasks. In DRL, each intersection can control itself and coordinate with neighbors to achieve optimal network-wide policies. However, comparing approaches remains a challenging task due to the existence of numerous possible configurations. This research performs a critical comparison of various traffic controllers found in the literature. It demonstrates that using a nonlinear approximator for coordination mechanisms and enhancing observability at each intersection are key performance drivers.

Manipulating cloth-like deformable objects (CDOs) is a long-standing problem in the robotics community. CDOs are flexible (non-rigid) objects that do not show a detectable level of com- pression strength while two points on the article are pushed towards each otherand include objects such as ropes (1D), fabrics (2D) and bags (3D). In general, CDOs’ many degrees of freedom (DoF) introduce severe self-occlusion and complex state-action dynamics as significant obstacles for perception and manipulation systems. These challenges exacerbate existing issues of modern robotic control methods such as imitation learning (IL) and reinforcement learning (RL). This review focuses on the application details of data-driven control methods on four major task families in this domain: cloth-shaping, rope manipulation, dressing and bag manip- ulation. Furthermore, we identify specific inductive biases in these four domains that present challenges for more general IL and RL algorithms, and summarise the future direction for the development of the field.

Cladding is typically used to protect components from wear and corrosion while also improving the aesthetic value and reliability of the substrate. The cladding process induces significant residual stresses due to the temperature difference between the substrate and the clad layer. However, these residual stresses could be effectively utilized by modifying processes and geometrical parameters. This paper introduces a novel methodology for using the weld-cladding process as a cost-effective alternative to various existing reinforcement techniques. The numerical analyses are performed to maximize the reinforcement of a cylindrical tool. The investigation of how the weld cladding develops compressive stresses on the specimen in response to a change in the weld beads and the welding sequence is presented. For the benchmark shape, experimental verification of the numerical model is performed. The impact of the distance between the weld beads and the effect of the tool diameter is numerically investigated. Furthermore, the variation in compressive stresses due to temperature fluctuations during the extrusion process has been evaluated. The results showed that adequate compressive stresses are generated on the welded parts through the cladding process after cooling. Hence, the targeted reinforcement of the substrate can be achieved by optimizing the welding sequence and process parameters.

Rapid search and localization for nuclear sources can be an important aspect in preventing human harm from illicit material in dirty bombs or from contamination. In the case of a single mobile radiation detector, there are numerous challenges to overcome such as weak source intensity, multiple sources, background radiation, and the presence of obstructions, i.e., a non-convex environment. In this work, we investigate the sequential decision making capability of deep reinforcement learning in the nuclear source search context. A novel neural network architecture (RAD-A2C) based on the actor critic (A2C) framework and a particle filter gated recurrent unit for localization is proposed. Performance is studied in a randomized 20 x 20 m convex and non-convex environment across a range of signal-to-noise ratio (SNR)s for a single detector and single source. RAD-A2C performance is compared to both an information-driven controller that uses a bootstrap particle filter and to a gradient search (GS) algorithm. We find that the RAD-A2C has comparable performance to the information-driven controller across SNR in a convex environment and at lower computational complexity per action. The RAD-A2C far outperforms the GS algorithm in the non-convex environment with greater than 95% median completion rate for up to seven obstructions.

Currently, the most important challenge in any assessment of state-of-the-art sensor technology and its reliability is to achieve road traffic safety targets. The research reported in this paper is focused on the design of a procedure for evaluating the reliability of Internet-of-Things (IoT) sensors and the use of a Cyber-Physical System (CPS) for the implementation of that evaluation procedure to gauge reliability. An important requirement for the generation of real critical situations under safety conditions is the capability of managing a co-simulation environment, in which both real and virtual data sensory information can be processed. An IoT case study that consists of a LiDAR-based collaborative map is then proposed, in which both real and virtual computing nodes with their corresponding sensors exchange information. Specifically, the sensor chosen for this study is a Ibeo Lux 4-layer LiDAR sensor with IoT added capabilities. Implementation is through an artificial-intelligence-based modeling library for sensor data-prediction error, at a local level, and a self-learning-based decision-making model supported on a Q-learning method, at a global level. Its aim is to determine the best model behavior and to trigger the updating procedure, if required. Finally, an experimental evaluation of this framework is also performed using simulated and real data

The reliability of concrete cable-stayed bridge structure is time variant. This is due to strength deterioration and time-variant mechanical and environmental loadings. Demands for keeping concrete cable-stayed bridge structure in functional and reliable states require maintenance interventions. Limited financial resources require optimal maintenance strategies. These strategies have to consider the effects of uncertainties not only on the structural reliability deterioration but also on the maintenance interventions. By analyzing variation of concrete cable-stayed concrete cable-stayed bridge liabilities and benefit expectation through different maintenance programs, on the condition of satisfying the concrete cable-stayed bridge performance, with the maximum expected benefit as the target to discuss repair and reinforcement strategies for the concrete cable-stayed bridges in service. The method takes into account the present performance and future expected benefit, optimizing the performance of the concrete cable-stayed bridge and the expected benefit through determination policies of concrete cable-stayed concrete cable-stayed bridge maintenance program. The proposed method also includes an advanced cost model which takes into account the interaction between costs of maintenance interventions and their effect on reliability. Examples are presented demonstrating the application of the proposed method to a stock of existing deteriorating concrete cable-stayed bridges under various maintenance scenarios.

Detecting, tracking, and classifying unmanned aerial vehicles (UAVs) in a swarm presents significant challenges due to their small and diverse radar cross-sections, multiple flight altitudes, velocities, and close trajectories. To overcome these challenges, adjustments of the radar parameters and/or position of the radar (for airborne platforms) are often required during runtime. The runtime adjustments help to overcome the anomalies in the detection, tracking, and classification of UAVs. The runtime adjustments are performed either manually or through fixed algorithms, each of which can have its limitations for complex and dynamic scenarios. In this work, we propose the use of multi-agent reinforcement learning (RL) to carry out the runtime adjustment of the radar parameters and position of the radar platform. The radar used in our work is a multibeam multifunction phased array radar (MMPAR) placed onboard UAVs. The simulations show the cognitive adjustment of the MMPAR parameters and position of the airborne platform using RL helps to overcome anomalies in the detection, tracking, and classification of UAVs in a swarm. A comparison with other artificial intelligence (AI) algorithms shows that RL performs better due to runtime learning of the environment through rewards.

The study examines the behavior of the Mechanically Stabilized Earth (MSE) wall using available reinforcement materials in Danang, Vietnam. The MSE was reinforced by the self-fabricated galvanized steel grids using CB300V steel with 3 cm ribs. The backfill soil is a sandy clay soil from the local area with a low cohesion. A full-scale model with full instrumentation was installed to investigate the distribution of tensile forces along the reinforcement layers. The highest load that caused the wall to collapse due to internal instability (reinforcement rupture) was 302 kN/m2, which is 15 times the design load of 20 kN/m2. The failure surface within the reinforced soil had a parabolic sliding shape which is similar to the theoretical studies. At the failure load level, the maximum lateral displacement at the top of the wall facing was small (3.9 mm) which is significantly lower than the allowable displacements for the retaining wall. Furthermore, a numerical model using FLAC software was applied to simulate the performance of the MSE wall. The modeling results are in good agreement with the physical model. Thus, the self-fabricated galvanized steel grids could be confidentially used in combination with the local backfill soil for the MSE walls.

In the dynamic and complex environment of industrial control rooms, operators are often inundated with a multitude of tasks and alerts, which can lead to an overwhelming situation known as task overload. This state can precipitate decision fatigue and a heavier reliance on cognitive biases, potentially compromising the decision-making process. To mitigate such risks, the implementation of decision support systems becomes crucial. These systems are designed to assist operators in making swift and well-informed decisions, particularly when they sense their own judgment may be faltering. Our research introduces an AI-based framework that leverages dynamic influence diagrams and reinforcement learning to construct an effective decision support system. The cornerstone of this framework is the creation of a robust, interpretable, and efficient tool that supports control room operators during critical process disturbances. By integrating expert knowledge, the dynamic influence diagram forms a comprehensive model that captures the uncertainties inherent in complex industrial processes. It is adept at anomaly detection and recommending optimal actions. Moreover, this model is enhanced through a strategic partnership with reinforcement learning algorithms, which fine-tune the recommendations to be more context-specific and precise. The ultimate goal of our framework is to provide operators with a live, reliable decision support system that can significantly improve their response during process upsets. This paper outlines the development of our AI framework and its application within a simulated control room environment. Our findings indicate that the use of decision support systems can lead to improved operator performance and reduced cognitive workload. However, it also reveals a trade-off with situation awareness, which tends to diminish as operators may become overly reliant on the system’s guidance. Trust emerges as a critical factor in the adoption and effectiveness of decision support systems. Our research underscores the importance of balancing the benefits of decision support with the need for maintaining operator engagement and comprehension during process operations.

The paper provides a comprehensive review of the recent advancements and methodologies in Human-Robot Collaboration (HRC) applied to the manufacturing assembly process. In modern manufacturing, the assembly process involves intricate and time-consuming operations, often necessitating flexible manual interventions. However, the cost and stability issues associated with manual labor highlight the need for collaborative solutions integrating humans and robots. HRC, as a viable solution, involves the joint effort of humans and robots in manufacturing tasks, presenting advantages in terms of precision, reproducibility, and cycle time. This review categorizes and discusses methodologies such as task allocation, reinforcement learning, and Cyber-Physical Systems (CPS)-based planning approaches that facilitate HRC in the assembly process. It also explores experiments and future trends to address challenges and enhance efficiency in manufacturing assembly through intelligent collaboration between humans and robots. The objective of this research is to provide insights and directions for further research in HRC to optimize manufacturing processes. By analyzing the existing state-of-the-art and presenting future prospects, this paper aims to guide researchers and practitioners toward more effective implementations of HRC in manufacturing assembly, ultimately leading to improved operational efficiency and productivity.

This article focuses on developing a continuous authentication system using behavioral biometrics to recognize users accessing computing devices. The user’s distinct behavioral biometric is captured through keystroke dynamics, and reward-based reinforcement learning (RL) ideas are applied to recognize them throughout the session. The suggested system adds an extra layer of security to traditional authentication methods, forming a robust continuous authentication system that can be added to static authentication systems. The methodology involves training a RL model to detect unusual user typing patterns and flag suspicious activity. Each user has an agent trained on their historical data, which is preprocessed and used to create episodes for the agent to learn from. The environment involves fetching observations and randomly corrupting them to learn out-of-order behavior. The observation vector includes both running features and summary features. The re-ward function is binary and minimalistic. The Principal Component Analysis (PCA) model is used to encode the running features, and the Double Deep Q-Network (DDQN) algorithm with a fully connected neural network is used as the policy net. The evaluation achieved an average training accuracy and EER (equal error rate) of 94.7% and 0.0126 and test accuracy and ERR of 81.06% and 0.0323 for all users when the number of encoder features was increased. Therefore, it is concluded that by continuously learning and adapting to changing behavior patterns, this approach can provide more secure and personalized authentication, lowering the possibility of unauthorized access and cyberattacks. Overall, the use of reinforcement learning and behavioral biometrics for continuous authentication has the potential to significantly enhance security in the digital age and are effective in identifying each user.

In the evolving landscape of Portfolio Management (PM), the fusion of advanced machine 1 learning techniques with traditional financial methodologies has opened new avenues for innovation. 2 Our study introduces a cutting-edge model combining Deep Reinforcement Learning (DRL) with 3 a Non-stationary Transformer architecture. This model is specifically designed to decode complex 4 patterns in financial time series data, enhancing portfolio management strategies with deeper insights 5 and robustness. It effectively tackles the challenges of data heterogeneity and market uncertainty, 6 key obstacles in PM. Our approach integrates key macroeconomic indicators and targeted news 7 sentiment analysis into its framework, capturing a comprehensive picture of market dynamics. This 8 amalgamation of varied data types addresses the multifaceted nature of financial markets, enhancing 9 the model’s ability to navigate through the complexities of asset management. Rigorous testing 10 demonstrates the model’s efficacy, highlighting the benefits of blending diverse data sources and 11 sophisticated algorithmic approaches in mastering the nuances of PM.12

Long-span steel structure trusses are widely used in factory buildings, but with the growth of service time and the increase of dynamic load fatigue, a considerable part of the long-span truss with dynamic load appears serious transverse cracks at the bottom of the middle span and oblique deformation of the abdomen in the operation process. The U-shaped cracks at the bottom and belly, as well as the mid-span down deflection of the main truss, reduced the functional function of the factory building truss structure, and had to limit the original crane load, which affected the normal safety and durability of the structure. Therefore, the application principle of the variable axial force cable system in the long-span factory building truss structure and 3D3S software modeling [1] were used. Analyzing and studying the reinforcement method of large-span powerhouse truss can provide practical experience for subsequent similar projects. In view of the above phenomenon, the large-span powerhouse trusses of Hongcheng Powerhouse 1 and No.2 located in Tonglu, Zhejiang Province are used as the research object, and the variable axial force cable method is proposed to strengthen and lift the load. Considering the span of large-span powerhouse truss, the cable system with 22m controlling force of 400kN is proposed to be selected for powerhouse 1, and the cable system with variable axial force of 24m is proposed to be selected for Powerhouse 2. The force model of large-span truss is established by using the finite element method commonly used to analyze the force of truss. Under two working conditions, The influence of reinforcement effect is analyzed and compared from three aspects: stiffness, bearing capacity and stability. And the phenomenon of uneven stress distribution is analyzed. The stress distribution characteristics of each node are understood by simulating the most disadvantageous node plates with the greatest internal force before and after reinforcement.

In this paper, we study the performance of various deep reinforcement learning algorithms to enhance the energy management system of a microgrid. We propose a novel microgrid model that consists of a wind turbine generator, an energy storage system, a set of thermostatically controlled loads, a set of price-responsive loads, and a connection to the main grid. The proposed energy management system is designed to coordinate among the different flexible sources by defining the priority resources, direct demand control signals, and electricity prices. Seven deep reinforcement learning algorithms were implemented and are empirically compared in this paper. The numerical results show that the deep reinforcement learning algorithms differ widely in their ability to converge to optimal policies. By adding an experience replay and a semi-deterministic training phase to the well-known asynchronous advantage actor-critic algorithm, we achieved the highest model performance as well as convergence to near-optimal policies.

The area of Central and Eastern Europe, and thus Poland, is not exposed to effects of seismic actions. Any possible tremors can be caused by coal or copper mining. Wind, rheological effects, the impact of other objects or a non-uniform substrate are the predominant types of loading included in calculations for stiffening walls. The majority of buildings in Poland, as in most other European countries, are low, medium-high brick buildings. Some traditional materials, like solid brick (>10% of construction materials market) are still used. But autoclaved aerated concrete (AAC) and cement-sand calcium-silicate (Ca-Si) elements with thin joints are prevailing (>70% of the market) on the Polish market. Adding reinforcement only to bed joints in a wall is a satisfactory solution (in addition to confining) for seismic actions occurring in Poland that improves ULS and SLS. This paper presents results from our own tests on testing horizontal shear walls without reinforcement and with different types of reinforcement. This discussion includes 51 walls made of solid brick (CB) reinforced with steel bars and steel trusses, results from tests on 15 walls made of calcium-silicate (Ca-Si) and AAC masonry units reinforced with steel trusses and plastic meshes. Taking into account our own tests and those conducted by other authors, empirical relationships were determined on the basis of more than 90 walls. They are applicable to design and construction phase to determine the likely effect of reinforcement on cracking stress that damage shear deformation and wall stiffness.

This research delves into the realm of predictive modeling within artificial intelligence (AI) and introduces an innovative ensemble approach designed to enhance prognostic precision. The proposed methodology, characterized by its ebullient nature, leverages the collective wisdom of diverse predictive models, synthesizing a comprehensive and robust framework. The pursuit of parsimony are paramount, underscoring the need for efficient model architectures that balance predictive acuracy and computational efficiency. Our study meticulously explores the amalgamation of diverse algorithms, encapsulating the intrinsic complexity of real-world phenomena. We navigate the intricate landscape of feature selection, algorithmic diversity, and model combination, fostering a holistic understanding of the prognostic process. The ebullient ensemble emerges as a dynamic solution, harnessing the synergy of constituent models while mitigating the risk of overfitting. To validate the efficacy of our approach, extensive experiments are conducted on diverse datasets, spanning domains such as healthcare, finance, and natural language processing. Comparative analyses demonstrate the superior predictive performance of our ensemble model in comparison to individual algorithms and traditional ensemble methods. Additionally, we delve into interpretability aspects, elucidating the underlying mechanisms that contribute to the model's prognostic prowess. Our research advances the frontier of predictive modeling in AI, offering a nuanced and sophisticated paradigm for prognostication. The ebullient ensemble approach stands as a testament to the synergy achievable through the amalgamation of diverse predictive models, providing a pragmatic and parsimonious avenue for enhanced prognostic acuracy in artificial intelligence applications.