Electrification of society, increasing renewable energy sources and mobility charging lead to new loading patterns for power transformers. Dynamic load conditions induce enhanced mechanical stress on the transformers’ windings, potentially causing degradation of the solid insulation over time and compromising the transformer's short-circuit withstand capability. Thermal expansion of the windings, caused by losses in the copper conductors, occurring as the transformer is load-ed, increases the stress. Conversely, magnetic losses in the core and tie plate expansion contribute to a reduction in stress. This paper presents the effect of step changes in core losses and copper losses by on-line measurements of the clamping pressure, to better understand the mechanical stresses acting upon the solid insulation cellulose materials.

Recent research underscores the potential hazards that Backdoor Attacks pose to natural language processing (NLP) models. A thorough exploration of these attack methodologies is critical for comprehending the susceptibility of such models. Under normal circumstances, a model compromised by a backdoor attack will produce standard outputs; however, the presence of a specific trigger within the input leads to erroneous results. This paper focuses on the vulnerability of BERT, a widely recognized model in numerous NLP applications, by introducing a novel backdoor attack strategy that effectively compromises it. We manipulate the attention heads in BERT to enhance the backdoor attack. The efficacy of this method is demonstrated through experiments conducted on clean-label attack and a Sentiment Analysis task.

Language models like BERT dominate current NLP research due to their robust performance, but they are vulnerable to backdoor attacks. Such attacks cause the model to consistently generate incorrect predictions when specific triggers are present in the input, while maintaining normal behavior on clean inputs. In this paper, we propose a straightforward data poisoning method targeting the BERT architecture. Our approach does not involve complex modifications to the model or its training process; instead, it relies solely on altering a small portion of the training data. By introducing simple perturbations into just 10% of the training dataset, we demonstrate the feasibility of injecting a backdoor into the model. Our experimental results show a high attack success rate, indicating that the model trained on the poisoned data can reliably associate the trigger with the attacker's desired outputs while its performance on clean data remains unaffected. This highlights the stealth and effectiveness of our method, emphasizing the need for improved defensive strategies to protect against such threats. Our study underscores the critical importance of ongoing research and development to safeguard AI systems from malicious exploitation, ensuring the security and reliability of NLP applications.

Ultrasound (US) has become a widely used imaging modality in clinical practice, characterized by its rapidly evolving technology, advantages, and unique challenges such as low imaging quality and high variability. There is a critical need to develop advanced automatic US image analysis methods to enhance diagnostic accuracy and objectivity. Vision transformer, a recent innovation in machine learning, has demonstrated significant potential in various research fields, including general image analysis and computer vision, due to its capacity to process large datasets and learn complex patterns. Its suitability for automatic US image analysis tasks, such as classification, detection, and segmentation, has been recognized. This review provides an introduction to vision transformer and discusses its applications in specific US image analysis tasks, while also addressing the open challenges and potential future trends in its application in medical US image analysis. Vision transformer has shown promise in enhancing the accuracy and efficiency of ultrasound image analysis and is expected to play an increasingly important role in the diagnosis and treatment of medical conditions using ultrasound imaging as technology progresses.

In many of the current software applications, numerous vulnerabilities may be present.1 Attackers attempt to exploit existing vulnerabilities that lead to security breaches, unauthorized entry,2 data theft, or incapacitation of a computer system. Rather than addressing software or hardware3 vulnerabilities at a later stage, it is better to address them immediately. DevSecOps, when utilized4 in application development, tackles these vulnerabilities at an early stage. AIBughunter is a tool5 that addresses this problem in software and was developed by the ASWM research group to predict,6 classify, and repair software vulnerabilities. AIBughunter integrates LineVul to find vulnerable7 code lines and returns information about the type of vulnerability and its severity to developers.8 It also includes a tool, VulRepair, which detects and repairs vulnerabilities. VulRepair currently9 predicts patches for vulnerable functions at 44%. In order to become truly effective, this number 10 needs to be increased. This study examines VulRepair to see whether the 44% Perfect Prediction 11 can be increased. VulRepair is a T5 based model that uses Natural Language and Programming 12 Languages for pre-training along with Byte Pair Encoding. It outperforms other existing models, 13 such as VRepair and CodeBERT. However, the hyperparameters may not be optimized due to the 14 development of new optimizers. We review a Deep Neural Network (DNN) optimizer developed by 15 Google in 2023. This optimizer called Evolved Sign Momentum (LION) is available in PyTorch. We 16 applied this optimizer to VulRepair and tested its influence on hyperparameters. After adjusting the 17 hyperparameters, we obtained a 56% Perfect Prediction, which exceeds the value of the VulRepair 18 report of 44%. This means that VulRepair can repair more vulnerabilities and avoid more attacks. As 19 far as we know, our approach of utilizing an alternative to AdamW, the standard optimizer, has not 20 been previously applied to enhance VulRepair and similar models. 21

In this paper, we introduce Syntax-Enhanced Transformer Model (SET), a groundbreaking approach in the realm of Transformer-based language modeling that seeks to redefine the boundaries of linguistic analysis and comprehension. SET innovatively combine (i) the well-established high-level performance, scalability, and adaptability of traditional Transformers with (ii) a sophisticated analysis of syntactic structures. This synergy is enabled by a novel attention mechanism tailored to parse syntactic nuances and a deterministic process adept at transforming linearized parse trees into meaningful linguistic representations. Our comprehensive experiments reveal that SET significantly advance the field by surpassing existing benchmarks in sentence-level language modeling perplexity. They exhibit exceptional proficiency in tasks that require an acute awareness of syntax, setting new standards for language models in understanding complex linguistic structures. Furthermore, SET demonstrate an enhanced capability to grasp nuanced linguistic patterns that have traditionally been challenging for standard Transformer models. However, our studies also uncover a unique aspect of SET: while they excel in sentence-level tasks, their representation of sentences as singular vectors—owing to the syntactic composition constraints intrinsic to their design—introduces certain limitations in document-level language modeling. This observation points to an intriguing area for future exploration; it suggests the potential need for an alternative or complementary memory mechanism within Transformer models, one that functions independently from, yet in harmony with, syntactic structures. Such a mechanism could be pivotal in enhancing the model's ability to comprehend and process long-form texts effectively. In conclusion, SET mark a significant stride in the journey towards more sophisticated, syntax-aware language models. They offer promising insights into the integration of deep linguistic knowledge with cutting-edge machine learning techniques, potentially opening doors to a new era of natural language understanding and processing.

The evolution of sentence representation learning, especially with parse tree encoders, has shown remarkable progress. Traditional approaches predominantly rely on recursive encoding of tree structures, which impedes parallel processing capabilities. Additionally, these methods often overlook the significance of dependency tree arc labels. To overcome these limitations, we introduce the Syntax-Enhanced Transformer (SET), incorporating a novel dual-attention mechanism that integrates relation-focused attention alongside traditional self-attention. This design effectively encodes both dependency and spatial positional relationships within sentence dependency trees. Our approach innovatively incorporates syntactic information into the Transformer framework without compromising its inherent parallelizability. The SET demonstrates superior or comparable performance to contemporary methods across various sentence representation tasks, significantly enhancing computational efficiency.

We introduce a novel hybrid Transformer-CNN architecture for robotic grasp detection, designed to enhance the accuracy of grasping unknown objects. Our proposed architecture has two key designs. Firstly, we develop a hierarchical transformer as the encoder, incorporating the external attention to effectively capture the correlation features across the data. Secondly, the decoder is constructed with cross-layer connections to efficiently fuse multi-scale features. Channel attention is introduced in the decoder to model the correlation between channels and to adaptively recalibrate the channel correlation feature response, thereby increasing the weight of the effective channels. Our method is evaluated on the Cornell and Jacquard public datasets, achieving an image-wise detection accuracy of 98.3% and 95.8% on each dataset, respectively. Additionally, we achieve object-wise detection accuracy of 96.9% and 92.4% on the same datasets. A physical experiment is also performed using the Elite 6Dof robot, with a grasping accuracy rate of 93.3%, demonstrating the proposed method's ability to grasp unknown objects in real-world scenarios. The results of this study show that our proposed method outperforms other state-of-the-art methods.

Breast cancer is the most prevalent type of disease among women. It has become one of the foremost causes of death among women globally. Early detection plays a significant role in administering personalized treatment and improving patient outcomes. Mammography procedures are often used to detect early-stage cancer cells. This traditional method of mammography while valuable has limitations of potential for false positives and negatives, patient discomfort, and radiation exposure. Therefore, there is a probe for more accurate techniques required in detecting breast cancer, leading to exploring the potential of machine learning in the classification of diagnostic images due to its efficiency and accuracy. This study conducted a comparative analysis of pre-trained CNNs (ResNet50 and VGG16) and Vision Transformers (ViT-Base and SWIN Transformer) with the inclusion of ViT-Base trained from scratch model architectures to effectively classify breast cancer mammographic images into benign and malignant cases. The Swin transformer exhibits superior performance with 99.8% accuracy and a precision of 99.8%. These findings demonstrate the efficiency of deep learning to accurately classify breast cancer mammographic images for the diagnosis of breast cancer, leading to improvement in patient outcomes.

This paper detects bidding behaviors in the auction with the random cutoff, where a winning bid needs to be above and the closest to the random cutoff. In this auction, each bidder is eager to predict the random cutoff in order to use the predicted random cutoff as bid. Furthermore, one of deep learning models, the transformer model is employed as one empirical method to anticipate the random cutoff. The result shows that the predicted random cutoff turns out to be a winner with probability of about 20 percent. This paper contributes to economic literature in terms of investigating the connection between a deep learning model and an auction based on the economic theory.

In the dynamic landscape of E-commerce, the proliferation of user-generated queries regarding products highlights the critical need for advanced automatic question answering (AQA) systems. These systems are instrumental in harnessing the vast array of information available online to deliver immediate and informative responses to potential buyers. Recognizing the complexity of such queries, which often require synthesis of reviews, product specifications, and responses to similar questions, we introduce a sophisticated model, the Comprehensive E-commerce Answer Generation System (CEAGS). This model adeptly navigates the challenges posed by irrelevant data and sentiment ambiguity in user-generated content. Our approach leverages a dual-phase process of relevance determination and sentiment clarity, setting the stage for a transformative response generation mechanism. Empirical analyses reveal the superiority of CEAGS, with our relevance assessment framework outstripping existing models by a notable margin in precision metrics, and our answer generation module achieving unprecedented gains in content preservation and coherence. Notably, CEAGS marks a pioneering contribution to the E-commerce domain by integrating disparate information sources to formulate responses that are not only accurate but also contextually rich and user-centric.

Cloud detection in remote sensing images is a crucial preprocessing step that efficiently identifies and extracts cloud-covered areas within the images, ensuring the precision and reliability of subsequent analyses and applications. Given the diversity of clouds and the intricacies of the surface, distinguishing the boundaries between thin clouds and the underlying surface is a major challenge in cloud detection. To address these challenges, an advanced cloud detection method, CloudformerV3, is presented in this paper. The proposed method employs a multi-scale adapter to incorporate dark and bright channel prior information into the model's backbone, enhancing the model's ability to capture prior information and multi-scale details from remote sensing images. Additionally, multi-level large window attention is utilized, enabling high-resolution feature maps and low-resolution feature maps to mutually focus and subsequently merge during the resolution recovery phase. This facilitates the establishment of connections between different levels of feature maps and offers comprehensive contextual information for the model's decoder. Experimental results on the GF1_WHU dataset demonstrate that the method introduced in this paper exhibits superior detection accuracy when compared to state-of-the-art cloud detection models. Furthermore, enhanced detection performance is achieved along cloud edges and with respect to thin clouds, showcasing the efficacy of the proposed method.

Previous camera self-calibration methods have exhibited certain notable shortcomings. On one hand, they either exclusively emphasized scene cues or solely focused on vehicle-related cues, resulting in a lack of adaptability to diverse scenarios and a limited number of effective features. Furthermore, these methods either solely utilized geometric features within traffic scenes or exclusively extracted semantic information, failing to comprehensively consider both aspects. This limited the comprehensive feature extraction from scenes, ultimately leading to a decrease in calibration accuracy. Additionally, conventional vanishing point-based self-calibration methods often required the design of additional edge-background models and manual parameter tuning, thereby increasing operational complexity and the potential for errors. Given these observed limitations, and in order to address these challenges, we propose an innovative roadside camera self-calibration model based on the Transformer architecture. This model possesses a unique capability to simultaneously learn scene features and vehicle features within traffic scenarios while considering both geometric and semantic information. Through this approach, our model can overcome the constraints of prior methods, enhancing calibration accuracy and robustness while reducing operational complexity and the potential for errors. Our method outperforms existing approaches on both real-world dataset scenarios and publicly available datasets, demonstrating the effectiveness of our approach.

The operation of transformers suffers from natural or man-made disasters, subjecting the equipment to prolonged exposure to temperature, lightning strikes, transient over-voltages, and over-current, gradually producing insulation aging phenomenon, which results in partial discharges within the transformer. Without dealing with it beforehand properly, it may lead to insulation breakdown. Nowadays, with the mature development of microchip control technology and wireless communication technology, which is helpful for realizing the research for online real-time and long-term monitoring capabilities for power equipment. A transformer operating condition monitoring and diagnostic system that can rapidly and effectively prevent the breakdown of transformer insulation, which can impact the overall power supply system is developed in this paper. The system is divided into two parts based on its function. The first part is the operating condition monitoring and real-time alert unit for the transformer, primarily composed of an embedded computing module, a sensor module, and a transmission module. Its work content is to capture real-time online partial discharge ultrasonic signals and high frequency current comparator pulse signals, and analyze and determine the condition of the internal insulation and operational status whether it is fault or not. The second part is the central control unit, consisting mainly of an industrial computer and a transmission module. Its work content is to store the ultrasonic and high frequency current comparator pulse signals from the transformer operation monitoring and real-time alert unit. It uses the software to eliminate signal noise and interference, and then utilizes artificial neural networks to identify the type of fault. In this paper, the research for the phenomenon of insulation degradation in transformer equipment aims to establish aging trend profiles through the dual malfunctioning alert and aging process analysis, as well as the long-term measurement of ultrasonic signals from partial discharges and high-frequency current comparator pulse signals. This approach provides crucial reference data and a foundation for disposable on-site measurement, assessing the phenomenon of insulation aging stages, defects and damage of the transformers effectively. It allows for timely replacement or maintenance in advance, achieving the prevention of malfunctioning and predictive diagnosis of aging.

Recent advancements in neural language models, particularly attention-based architectures like the Transformer, have substantially surpassed traditional methods in various natural language processing tasks. These models adeptly generate nuanced token representations by considering contextual relations within a sequence. However, augmenting these models with explicit syntactic knowledge, such as part of speech tags, has been found to remarkably bolster their effectiveness, especially under constrained data scenarios. This study introduces the Linguistic Enhanced Transformer (LET), which integrates multiple syntactic features, showing a notable increase in translation accuracy, evidenced by an improvement of up to 1.99 BLEU points on subsets of the WMT '14 English-German dataset. Furthermore, this paper demonstrates that enriching BERT models with syntax-aware embeddings enhances their performance on several GLUE benchmark tasks.

Although wireless capsule endoscopy (WCE) detects small bowel diseases effectively, it has some limitations. For example, the reading process can be time-consuming due to the numerous images generated per case, and lesion detection accuracy may rely on the operators' skills and experiences. Hence, many researchers have recently developed deep learning-based methods to address these limitations. However, they tend to select only a portion of the images from a given WCE video and analyze each image individually. In this study, we note that more information can be extracted from the unused frames and temporal relations of sequential frames. Specifically, to increase the accuracy of lesion detection without depending on experts' frame selection skills, we suggest using whole video frames as the input to the deep-learning system. Thus, we propose a new Transformer-based neural encoder that takes the entire video as the input, exploiting the power of the Transformer to extract long-term global correlation within and between the input frames. Subsequently, we can capture the temporal context of the input frames and the attentional features within a frame. Tests on benchmark datasets of four WCE videos showed 95.1% sensitivity and 83.4% sensitivity. These results may significantly advance automated lesion detection techniques for WCE images. Our code is available at https://github.com/syupoh/VWCE-Net.git.

Detection of partial discharge (PD) activities in high-voltage equipment can be done according to the several mechanisms of signal detection, which are electromagnetic wave signal detection, acoustic signal detection, chemical reactions, electrical signal detection, or optical emission detection. Recently, multiple methods of detection and localization of partial discharge activities occurred in power transformers and gas-insulated switchgear (GIS) have been proposed to monitor the health condition of high-voltage equipment, especially when the awareness regarding preventive maintenance has been emphasized at the industrial level and electrical providers. In aligning the needs of industrial and the improvement of PD detection methods, this manuscript focused on reviewing the current practice methods for the detection and localization of PD signals in high-voltage equipment and comparing their efficacy, in summarizing the future direction of research work-related methods of PD detection. The comparative reviews are discussed in terms of the mechanism of PD signal detection, indication parameters, calibration technique, advantages, and limitations of each method of PD measurement in detail.

In practical applications, the detection of objects with various sizes is a common requirement for most detectors. The feature pyramid network (FPN) is widely adopted as a framework to address this challenge. The field is witnessing an increasing number of transformer-based target detectors due to the widespread adoption of transformer technology. This paper initially examines the design flaws in FPN and transformer-based target detectors, followed by the introduction of a new transformer-based approach called Texturized Instance Guidance (TIG-DETR) to address these issues. Specifically, TIG-DETR comprises a backbone network, a new pyramidal structure known as Texture-Enhanced FPN (TE-FPN), and an enhanced DETR detector.The TE-FPN is composed of three components: a bottom-up pathway for enhancing texture information in the feature map, a lightweight attention module to address confounding effects resulting from cross-scale fusion, and a standard attention module to enhance the final output features.The improved DETR detector utilizes Shifted Window based Self-Attention to replace the multi-headed self-attention module in DETR, thereby accelerating model convergence. Moreover, it incorporates an Instance Based Advanced Guidance Module to enhance instance perception in the image by employing a pre-local self-attentive mechanism for recognizing larger instances. By employing TE-FPN instead of FPN in Faster RCNN with Resnet-50 as the backbone network, we achieve a 1.9% improvement in average accuracy. TIG-DETR achieves an average accuracy of 44.1 with Resnet-50 as the backbone network.

Existing methods for 3D human pose estimation mainly divide the task into two stages. The first stage identifies the 2D coordinates of the human joints in the input image, namely the 2D human joint coordinates. The second stage uses the results from the first stage as input to recover the depth information of human joints from the 2D human joint coordinates to achieve 3D human pose estimation. However, the recognition accuracy of the two-stage method relies heavily on the results of the first stage and includes too many redundant processing steps, which reduces the inference efficiency of the network. To address these issues, we propose the EDD, a fully End-to-end 3D human pose estimation method based on transformer architecture with Dual Decoders. By learning multiple human poses, the model can directly infer all 3D human poses in the image using a pose decoder, and then further optimize the recognition result using a joint decoder based on the kinematic relations between joints. With the attention mechanism, this method can adaptively focus on the most relevant features to the target joint, effectively overcoming the feature misalignment problem in the human pose estimation task and greatly improving the model performance. Any complex post-processing step, such as non-maximum suppression, is eliminated, further improving the efficiency of the model. The results show that the method achieves an accuracy of 87.4% on the MuPoTS-3D dataset, significantly improving the accuracy of end-to-end 3D human pose estimation methods.

In power systems there are complex transformer structures, whose accurate analysis is not possible using the techniques available today. This paper presents a systematic data driven analysis method for coupled inductors of arbitrary complexity. The method first establishes a winding matrix N mapping the windings to the limbs of the transformer. A permeance matrix P is created from the reluctance network of the magnetic core. A generalized inductance matrix L mapping currents in the transformer windings to the induced voltages is generated based on the winding (N) and permeance (P) matrices. The inductance matrix representation of a coupled inductor is then transformed to an admittance matrix, which can be integrated to the nodal analysis of the electrical circuit surrounding the coupled inductor. The method presented is validated by simulations with real transformer structures using electromagnetic transient program (EMTP/ATP).

Large power transformers are generally associated with a maximum capacity rating of 100 MVA or higher. These large liquid dielectric power transformers are a custom-built piece of equipment, thus very expensive, and a backbone element of the power grid. In extreme cases as, for example, severe geomagnetic disturbances, permanently monitoring their condition will enhance their electrical reliability and resilience to guarantee efficient management of its life cycle. However, some traditional monitoring/diagnosis techniques have singular features when applied to large power transformers and their interlinked subsystems. In this context, and since that information is hardly put in evidence and compiled in the literature, this paper reviews the particularities of monitoring and diagnosing those assets.

Aiming to address the issue that the existing multi-view stereo reconstruction methods have insufficient adaptability to the repetitive patterns and weak textures in the multi-view images, this paper proposes a three-dimensional (3D) reconstruction algorithm based on feature enhancement and weight optimization MVSNet (Abbreviated as FEWO-MVSNet). To obtain accurate and detailed global and local features, we first develop an adaptive feature enhancement approach to obtain multi-scale information from the images. Second, we introduce an attention mechanism and a spatial feature capture module to enable high-sensitivity detection for weak texture features. Third, based on the 3D convolutional neural network, the fine depth map for multi-view images can be predicted and the complete 3D model is subsequently reconstructed. Last, we evaluated the proposed FEWO-MVSNet through training and testing on the DTU, BlendedMVS, and Tanks&Temples datasets. The results demonstrate significant superiorities of our method for 3D reconstruction from multi-view images, with our method ranking first in accuracy and second in completeness when compared to the existing representative methods.

Against the background of the sky, imaging and perception of aircraft are cru- cial. Among these, the location of aircrafts is not a very difficult problem in general, because that deep learning algorithms and object detection models based on convolutional neural network(CNN) are ever-evolving. In order to have a better identification of various aircrafts, or to accurately locate different types of flying targets in the sky, it becomes significant to distinguish and recognize different types of aircrafts. One question is that, distinguishing and recognizing between sub-categories of aircrafts pose great challenges. Although fine-grained recognition focuses on exploring and studying such problems, aircrafts under different sub-categories and granularities lead us to rethink the application of features. We noticed that features in swin-transformer demonstrates the under- standing and processing of images, fully showcasing the encoding and indexing of information. Through this research and proposed approach, we discovered a better understanding of features encoding and use, which are inspired by the feature encoding and computation in swin-transformer. In our paper, our ap- proach has achieved effects on features encoded graphically, manifested from the architecture design and convolutional neural network computation, and outper- forms other famous fine-graiend classification models on this issue. Not only approach we proposed has demonstrated superior performance in fine-grained aicraft classification, but also the mechanisms of the feature encoding under different sample space partitions is revealed for this issue, which provides a re- search for the representation of aircraft fineness characteristics. The relationship between representation orientation of aircraft feature information under various grained divisions, shows that the recognition of different categories according to man-made, such as aircraft, can be achieved by means of the feature represen- tations studied in this paper, for more specific defined classification tasks and various man-made targets partition criteria, which may influences the princi- ple of design for calculation and feature extraction in fine-grained classification models.

With the development of human-computer interaction, the importance of emotion recognition is increasing. Emotion recognition technology provides practical benefits in various industries such as improving user experience, education, and organizational productivity. In education, emotion recognition can enable real-time understanding of students' emotional states and provide tailored feedback, and in the workplace, by monitoring employees' emotional states, it can improve work performance and job satisfaction. Therefore, multimodal-based research that combines text, speech, and video data for emotion recognition is being conducted in various industries. In this study, we propose an emotion recognition method that combines text and speech data, reflecting the characteristics of the Korean language. For text, KoELECTRA is used to perform embedding, and for speech, MFCC and pitch analysis are used to extract features. Finally, a multimodal transformer model is proposed that combines these two data types to perform emotion recognition. The multimodal transformer model processes text and speech data separately, and then learns the interaction between the two modalities through a Cross-Modal Attention mechanism. Through this, the complementary information from text and speech is effectively combined, improving emotion recognition performance. Experimental results show that the proposed model outperforms single-modality models, achieving a high accuracy of 73.13% and an F1-Score of 0.7344 in emotion classification. This study contributes to the advancement of emotion recognition technology by combining various language and modality data, and higher performance can be expected through the integration of additional modalities in the future.

Diabetic retinopathy, a common complication of diabetes, is further exacerbated by factors such as hypertension and obesity. This study introduces the Diabetic Retinopathy Convolutional Transformer (DRCT) model, which combines convolutional and transformer techniques to enhance the classification of retinal images. The DRCT model achieved an impressive average F1 score of 0.97, reflecting its high accuracy in detecting true positives while minimizing false positives. Throughout 100 training epochs, the model exhibited strong generalization capabilities, achieving superior validation accuracy with minimal overfitting. On a newly evaluated dataset, the model attained precision and recall scores of 96.93% and 98.89%, respectively, indicating a well-balanced handling of false positives and false negatives. The model's ability to classify retinal images into five distinct diabetic retinopathy categories demonstrates its potential to significantly improve automated diagnosis and aid in clinical decision-making.

This study introduces a sophisticated neural network structure for segmenting breast tumors. It achieves this by combining a pretrained Vision Transformer (ViT) model with a U-Net framework. The U-Net architecture, commonly employed for biomedical image segmentation, is further enhanced with Depthwise Separable Convolutional Blocks to decrease computational complexity and parameter count, resulting in better efficiency and less overfitting. The Vision Transformer, renowned for its robust feature extraction capabilities utilizing self-attention processes, efficiently captures the overall context within images, surpassing the performance of conventional convolutional networks. By using a pretrained ViT as the encoder in our U-Net model, we take advantage of its extensive feature representations acquired from extensive datasets, resulting in a major enhancement in the model’s ability to generalize and train efficiently. The suggested model has exceptional performance in segmenting breast cancers from medical images, highlighting the advantages of integrating transformer-based encoders with efficient U-Net topologies. This hybrid methodology emphasizes the capabilities of transformers in the field of medical image processing and establishes a new standard for accuracy and efficiency in activities related to tumor segmentation.

Natural language processing model (NLP) are used in chatbots to understand user input, interpret its meaning, and generate conversational responses to provide immediate and consistent assistance. This reduces problem-solving time and staff workload and increases user satisfaction. There are both rule-based chatbots, which use decision trees and are programmed to answer specific questions, and self-learning chatbots, which can handle more complex conversations through continuous learning about data and user interactions. However, only a few chatbots have been developed specifically for the Italian language. This work proposes an NLP model to develop a powerful and efficient Italian QA (Question Answering) chatbot that is easy to use for Italian Public Administration (PA). The proposed model is based on BERT (Bidirectional Encoder Representations from Transformer) architecture, where an Encoder/Decoder module and a Highway Network module have been added to perform more efficient filtering of input text. The Italian version of the Stanford Question Answering Dataset (SQuAD-IT) is used to test the proposed model. The proposed model is one of the first models developed for Italian language-specific chatbots.

Visual object tracking is an important technology in camera based sensor networks, which has a wide range of practicability in auto drive system. A transformer is a deep learning model that adopts the mechanism of self-attention, and it differentially weights the significance of each part of the input data. It has been widely applied in the field of visual tracking. Unfortunately, the security of the transformer model is unclear. It makes such transformer-based applications be exposed to security threats. In this work, the security of the transformer model is investigated with the important component of autonomous driving, visual tracking. Such deep-learning-based visual tracking is vulnerable to adversarial attacks, so adversarial attacks are implemented as the security threats to conduct the investigation. First, adversarial examples are generated on top of video sequences to degrade tracking performance, and the frame-by-frame temporal motion is taken into consideration when generating perturbations over the predicted tracking results. Then, the influence of perturbations on performance is sequentially investigated and analyzed. Finally, numerous experiments on OTB100, VOT2018, and GOT-10k data sets demonstrate that the executed adversarial examples are effective on the performance drops of the transformer-based visual tracking.

The rapid growth in scientific article publications allows us to access articles as soon as possible. Therefore, automatic summarization systems (ATSs) are widely preferred. In most studies, the en-tire source document is expected to be summarized, just as it would be summarized by a human. Summarizing long articles, such as scientific articles, is quite difficult due to token restraint and extraction of scientific words. To address this problem, a novel Graph-Based Abstractive Summa-rization (GBAS) model is proposed, which is a novel scientific text summarization model based on SciBERT and the graph transformer network (GTN). The document's integrity is maintained since the SciIE system uses the graph structure to create a terminology-based document structure. Therefore, long documents are also summarized. The proposed model is compared with baseline models and human evaluation. Human evaluation results show that the results of the proposed model are informative, fluent, and consistent with the ground-truth summary. The experimental results indicate that the proposed model outperforms baseline models with a 37.10 and 34.96 ROUGE-L score.

In the era of Artificial Intelligence (AI), comprehending and responding to non-verbal communication is increasingly vital. This research extends AI's reach in bridging communication gaps, notably benefiting American Sign Language (ASL) and Taiwan Sign Language (TSL) communities. It focuses on employing various AI models, especially the Hierarchical Vision Transformer with Shifted Windows (Swin), for recognizing diverse sign language datasets. The study assesses Swin architecture's adaptability to different sign languages, aiming to create a universal platform for Unvoiced communities. Utilizing deep learning and transformer technologies, hybrid application prototypes have been developed for ASL-to-English translations and vice versa, with plans to expand this to multiple sign languages. The Swin models, trained on varied dataset sizes, show considerable accuracy, indicating their flexibility and effectiveness. This research underscores major advancements in sign language recognition and underlines a commitment to inclusive communication in the digital era. Future work will focus on enhancing these models and broadening their scope to include more sign languages, integrating multimodality and Large Language Models (LLMs), thereby fostering global inclusivity.

The Chat Generative Pre-training Transformer (GPT), also known as ChatGPT, is a powerful generative AI model that can simulate human-like dialogues across a variety of domains. However, this popularity has attracted the attention of malicious actors who exploit ChatGPT to launch cyberattacks. This paper examines the tactics that adversaries use to leverage ChatGPT in a variety of cyberattacks. Attackers pose as regular users and manipulate ChatGPT’s vulnerability to malicious interactions, particularly in the context of cyber assault. The paper presents illustrative examples of cyberattacks that are possible with ChatGPT and discusses the realm of ChatGPT-fueled cybersecurity threats. The paper also investigates the extent of user awareness of the relationship between ChatGPT and cyberattacks. A survey of 253 participants was conducted, and their responses were measured on a three-point Likert scale. The results provide a comprehensive understanding of how ChatGPT can be used to improve business processes and identify areas for improvement. Over 80% of the participants agreed that cyber criminals use ChatGPT for malicious purposes. This finding underscores the importance of improving the security of this novel model. Organizations must take steps to protect their computational infrastructure. This analysis also highlights opportunities for streamlining processes, improving service quality, and increasing efficiency. Finally, the paper provides recommendations for using ChatGPT in a secure manner, outlining ways to mitigate potential cyberattacks and strengthen defenses against adversaries.

Structural magnetic resonance imaging (sMRI) is widely used in the clinical diagnosis of diseases due to its advantages: high definition and noninvasive. Therefore, computer-aided diagnosis based on sMRI images is broadly applied in classifying Alzheimer’s disease (AD). Due to the excellent performance of Transformer in computer vision, Vision Transformer (ViT) has been employed for AD classification in recent years. ViT relies on access to large datasets, while the sample size of brain imaging datasets is relatively insufficient. Moreover, the pre-processing procedures of brain sMRI images are complex and labor-intensive. To overcome the limitations mentioned above, we propose Resizer Swin Transformer (RST), a deep learning model that can extract information from brain sMRI images that are only briefly processed to achieve multi-scale and cross-channel features. In addition, we pre-trained our RST on a natural image dataset and obtained better performance. The experimental results of ADNI and AIBL datasets prove that RST can achieve better classification performance in AD prediction compared with CNN-based and Transformer models.

Buildings are responsible for approximately 40% of the world’s energy consumption and 36% of the total carbon dioxide emissions. Building occupancy is essential, enabling Occupant-Centric Control for zero emissions and decarbonization. Although existing machine learning and deep learning methods for building occupancy prediction have achieved remarked progress, their analyses remain limited when applied to complex real-world scenarios. Besides, there is a high expectation for Transformer algorithms to predict building occupancy accurately. Therefore, this paper presents an Occupancy Prediction Transformer network (OPTnet). We fuse and feed multi-sensor data (building occupancy, indoor environmental conditions, HVAC operations) into a Transformer model to forecast the future occupancy presence in multiple zones. We perform experimental analysis and compare it to different occupancy prediction methods (e.g., Decision Tree, Long Short-Term Memory networks, Multi-layer perceptron) and diverse time horizons (1,2,3,5,10,20,30mins). The performance metrics (e.g., Accuracy and Mean Squared Error) are employed to evaluate the effectiveness of prediction algorithms. Our OPTnet method achieves superior performance on our experimental two-week data compared to existing methods. The improved performance signifies its potential to enhance HVAC control systems and energy optimization strategies. We will make the code publicly available at https://github.com/kailaisun/occupancy-prediction-binary to promote transparency and reproducibility.

Communication is a key method of expressing one's thoughts and opinions. Amongst many modalities, speech and writing are the most powerful and common forms of human communication. Analysing what and how people think has inherently been an interesting and progressive research domain. This includes bimodal sentiment analysis which is an emerging area in natural language processing (NLP) and has received a great deal of attention in recent years in a variety of areas including social opinion mining, health care, banking, and so on. At present, there are limited studies on bimodal conversational sentiment analysis as it proves to be a challenging area given the complex nature of the way humans express sentiment cues across various modalities. To address this gap, a comparison of the performance of multiple data modality models has been conducted on the MELD dataset, a widely-used dataset for benchmarking sentiment analysis within the research community. Our work then demonstrates the results of combining acoustic and linguistic representations. Lastly, our proposed neural network-based ensemble learning technique is employed over six transformer and deep learning-based models, achieving a State-Of-The-Art (SOTA) accuracy.

Deep learning technologies for skin cancer detection have been dramatically advanced based on high resolution dermoscopic images. The low-cost approach based on clinical skin images of low resolution is promising but remains technically challenging due to their undermined image quality. In this paper, we propose a coarse-to-fine efficient super resolution transformer (CF-ESRT) network to reconstruct the dermoscopy-level high resolution skin image from a low resolution clinical image. By connecting the refinement network to the original super resolution transformers and applying perceptual and gradient losses, our framework noticeably improves the finer texture details of skin lesions in the super resolution (SR) images, and is effective to elevate the perceptual quality of the SR images. Quantitative and qualitative evaluations show that our method outperforms ESRT the basis model as well as the other state-of-the-art SR models.

The prediction of drug-target interactions plays a fundamental role in facilitating drug discovery, where the goal is to find prospective drug candidates. With the increase in the number of drug-protein interactions, machine learning techniques, especially deep learning methods, have become applicable for drug-target interaction discovery because they significantly reduce the required experimental workload. In this paper, we present a spontaneous formulation of the drug-target interaction prediction problem as an instance of multi-instance learning. We address the problem in three stages, first organizing given drug and target sequences into instances via a private-public mechanism, then identifying the predicted scores of all instances in the same bag, and finally combining all the predicted scores as the output prediction. A comprehensive evaluation demonstrates that the proposed method outperforms other state-of-the-art methods on three benchmark datasets.

Robot grasp modeling and implementation is one of essential abilities for a robot with embodied artificial intelligence. However, most existing deep learning-based grasp methods suffer a large number of parameters and heavy computation overheads. To address this issue, by fully exploiting the complementary capabilities of both CNNs and Transformers, we propose a smart grasp deep network with one-way fusion strategy via context path and spatial path (SGNet), which enjoys a lightweight structure, fast inference speed and easy deployment on devices with limited computation resources. Specifically, the context path employs lightweight depthwise separable convolution to achieve fast down-sampling while a novel DSFormer module mainly by integrating Transformer is to extract global and context-rich features. The spatial path efficiently fuses feature information from the context path in one-way manner and generate high-resolution feature maps via point-by-point convolution operations. Experimental results show the proposed model with only 1M parameters has a significantly overall performance, achieving 99.4% accuracy on Cornell dataset and 93.4% accuracy on Jacquard dataset, as well as within 12.5ms inference time.

Traditional chemical research often relies on trial-and-error synthesis and characterization methods. Now, modern machine learning (ML) offers data-driven approaches for predicting properties, like water solubility, directly from chemical structure. But with various data representation schemes for molecular structure and model approaches to select from, it can be difficult for non-experts to determine best practices for utilizing ML. To clarify this landscape of choices, this study uses the ESOL molecular solubility dataset to compare the performance of a selection of different models on different data representations. First, we compare three classical regression methods (linear, ridge, LASSO) on three common data representations (RDKit fingerprint, Morgan fingerprint, intuition-selected molecular features). Then, we demonstrate how two distinct deep learning approaches (multilayer perceptron, graph convolution) can achieve accurate predictions even when prior intuition about feature-property correlations are absent. Finally, we outline a modern attention-based approach, inspired by successes in language modeling and fine-tuned by Bayesian optimization, to achieve a prediction methodology that is more general and performant than the previous approaches. This attention-based approach operates directly on the common SMILES string molecular representation, without requiring as many model parameters as other deep learning approaches or preprocessing into a representative fingerprint or vector of intuitively selected features. In short, when selected molecular features are known to likely correlate with a feature of interest, it may be possible to achieve good predictive modeling without turning to massive deep learning approaches. When particular features are not known a priori, graph approaches are a common solution, and we further demonstrate how a modern hyperparameter optimized attention approach can perform even better.

The fusion of global contextual information with local cropped block details is crucial for segmenting ultra-high resolution images. In this study, we introduce a novel fusion mechanism termed Global-Local Deep Fusion (GL-Deep Fusion) based on an enhanced transformer architecture, which efficiently integrates global contextual information and local details. Specifically, we propose the Global-Local Synthesis Networks (GLSNet), a dual-branch network where one branch processes the entire original image, while the other branch handles cropped local patches as input. The feature fusion of different branches in GLSNet is achieved through GL-Deep Fusion, significantly enhancing the accuracy of ultra-high resolution image segmentation. Particularly effective is GLSNet in identifying tiny overlapping items. To optimize GPU memory utilization, we meticulously design a dual-branch architecture that proficiently leverages the features it extracts, seamlessly integrating them into the enhanced transformer framework of GL-Deep Fusion. Extensive experiments conducted on challenging benchmarks, including DeepGlobe and Vaihingen datasets, demonstrate that GLSNet achieves a new state-of-the-art performance in terms of GPU memory utilization and segmentation accuracy tradeoff.

Time series forecasting has been a challenging area in the field of Artificial Intel-ligence. Various approaches such as linear Neural Networks, Recurrent Linear Neural Networks, Convolutional Neural Networks and recently Transformers have been attempted for the time series forecasting domain. Although, Trans-former-based architectures have been outstanding in the Natural Language Pro-cessing domain, especially in autoregressive language modeling, the initial at-tempts to use transformers in the time series arena have met with mixed success. One of the recent papers demonstrated that modeling the multi-variate time se-ries problem as independent channels on a transformer architecture produced lower mean square and mean absolute errors on benchmarks, however, another recent paper claimed that a simple linear neural network out-performs previous approaches including transformer-based designs. We investigate this paradox in detail comparing the linear Neural Network and Transformer based designs, provide insights to why a certain approach may be better for a particular type of problem. We also improve upon the recently proposed simple linear Neural Network based architecture by using dual pipelines with batch normalization and reversible instance normalization. Our enhanced architecture outperforms all existing architectures for time series forecasting on majority of the popular benchmarks.

Underwater visual detection technology plays a pivotal role in fields such as marine exploration. With the increasing demand for underwater monitoring, the quest for efficient and reliable methods for underwater target recognition has become particularly significant. To address this requirement, this study developed an innovative underwater object detection architecture based on YOLOv8, named YOLOv8-MU, aimed at significantly enhancing detection accuracy.By integrating the LarK module proposed in UniRepLKNet to optimize the backbone network, YOLOv8-MU aims to achieve a larger receptive field without increasing the model’s depth. Further, this research introduces C2fSTR, an innovative method that combines Swin Transformer with the C2f module. Additionally, we have incorporated the SPPFCSPC_EMA module, which combines Cross-Stage Partial Fast Spatial Pyramid Pooling (SPPFCSPC) with attention mechanisms, significantly improving the detection accuracy and robustness of various biological targets. Moreover, by introducing a fusion block based on DAMO-YOLO into the neck of the model, we further enhanced the model’s capability in multi-scale feature extraction. Finally, the adoption of the MPDIoU loss function, designed around vertex distance, effectively tackles the challenges of localization accuracy and boundary clarity in underwater organism detection. Experimental results on the URPC2019 dataset demonstrate that the YOLOv8-MU model achieved an mAP@0.5 of 78.4%, marking improvements of 5.6%, 1.1%, and 4.0% over YOLOv5s, YOLOv7, and YOLOv8n respectively, indicating the leading performance (SOTA) of this method. On the other hand, further evaluation on the URPC2020 dataset confirmed the generalization capability of the YOLOv8-MU architecture, with its mAP@0.5 reaching 80.4%, surpassing various models including YOLOv5x and YOLOv8n, showcasing the wide applicability of our proposed improved model architecture.

In the application domain, accurate word sense identification is crucial for improving the performance of machine translation, information retrieval and end-to-end communication tasks. However, word polysemy is a major obstacle to accurate semantic identification. Therefore, word semantic disambiguation has always been one of the key challenges in natural language processing and has attracted the attention of a large number of researchers. This research proposes an innovative disambiguation algorithm based on the large-scale Bert model and the Polly encoder framework, and introduces WordNet as a benchmark for word semantic. By exploiting the ability of the pre-trained model to extract and learn semantic information, and using a specially designed forward propagation algorithm and loss function to fine-tune the large-scale Bert model, the model has high Accuracy and robustness. In this research, several experiments were conducted on the Semcor 3.0 semantic dataset. The experimental results show that the model proposed in this research shows excellent performance on the Semcor test set, with an Accuracy of 86.1% and an F1 score of 0.847, which is a significant improvement over the traditional model.

Multispectral image (MSI) and hyperspectral image (HSI) fusion (MHIF) aims to address the challenge of acquiring high-resolution (HR) HSI images. This field combines a low-resolution (LR) HSI with an HR-MSI to reconstruct HR-HSI. Existing methods directly utilize transformers to perform feature extraction and fusion. Despite the demonstrated success, there exist two limitations: 1) Employing the entire transformer model for feature extraction and fusion fails to fully harness the transformer’s potential in integrating the spectral of the HSI and spatial information of the MSI. 2) HSI has a strong spectral correlation and exhibits sparsity in the spatial domain. Existing transformer-based models do not optimize this physical property, which makes their methods prone to spectral distortion. To accomplish these issues, this paper introduces a novel framework for MHIF called Sparse Mix-Attention Transformer (SAMformer). Specifically, to fully harness the advantages of the Transformer architecture, we propose a Spectral Mix Attention Block (SMAB), which concatenates the keys and values extracted from LR-HSI and HR-MSI to create a new multi-head attention module. This design facilitates the extraction of detailed long-range information across spatial and spectral dimensions. Besides, to address the spatial sparsity inherent in HSI, we incorporated a sparse mechanism within the core of SMAB called Sparse Spectral Mix Attention Block (SSMAB). In the SSMAB, we compute attention maps from queries and keys and select the K highly correlated values as the sparse attention map. This approach enables us to achieve a sparse representation of spatial information while eliminating spatially disruptive noise. Extensive experiments conducted on three benchmark datasets, namely Cave, Harvard, and Pavia Center, demonstrate the SMAformer method outperforms state-of-the-art methods.

Heat transfer fluids are used in various industrial systems to maintain them in perfect operating conditions. Extensive research efforts have been dedicated to enhance the thermal properties of these heat transfer fluids to improve their efficiency. Developing nanofluids is a potential candidate for such enhancements. This study investigates the impact of incorporating TiO2 nanoparticles into two types of oils: transformer oil (NYTRO LIBRA) and virgin coconut oil (manufactured by Govi Aruna Pvt. Ltd.) at different temperatures and with varying volume fractions. The nanofluids were prepared using a two-step method by adding CTAB (Cetyltrimethylammonium bromide) surfactant. To minimize nanoparticle agglomeration, this study employed relatively low-volume fractions. Thermal properties by means of thermal conductivity, thermal diffusivity, and volumetric heat capacity were measured in accordance with ASTM (American Society for Testing and Materials) standard methods, using a multifunctional thermal conductivity meter (LAMBDA thermal conductivity meter). The measured thermal conductivity values were compared with theoretical models and previous research findings. It was confirmed that the modification of thermal properties was enhanced by doping TiO2 nanoparticles with different volume fraction.

The research results presented in the article were carried out during the realization of the project, the aim of which is to develop a method of drying cellulose insulation in power transformers with the use of synthetic ester. This method uses a very high water absorption of the ester. During the drying of transformers, the ester is systematically contaminated with mineral oil, which gradually loses its ability to absorb water. Information on the oil concentration in the mixture is needed in two cases: at the stage of making a decision on the treatment of the mixture and during its treatment. The article presents the results of investigations of two methods: 1) based on the measurement of the mixture density, and 2) based on the measurement of the capacitance of the capacitor immersed in the mixture. The conducted research shows that the method of measuring the density of the mixture gives an uncertainty of 2.6 p. %, while the method of measuring the capacitance of a capacitor gives an uncertainty of 2.2 p. %. A significant advantage of the method of measuring the capacitance is the possibility of using it online to control the ester treatment process.

In recent years, transformer-based models have gained prominence in multivariate long-term time series forecasting (LTSF), demonstrating significant advancements despite facing challenges such as high computational demands, difficulty in capturing temporal dynamics, and managing long-term dependencies. The emergence of LTSF-Linear, with its straightforward linear architecture, has notably outperformed transformer-based counterparts, prompting a reevaluation of the transformer's utility in time series forecasting. In response, this paper presents an adaptation of a recent architecture termed extended LSTM (xLSTM) for LTSF. xLSTM incorporates exponential gating and a revised memory structure with higher capacity that has good potential for LTSF. Our adopted architecture for LTSF termed as xLSTMTime surpasses current approaches. We compare xLSTMTime's performance against various state-of-the-art models across multiple real-world datasets, demonstrating superior forecasting capabilities. Our findings suggest that refined recurrent architectures can offer competitive alternatives to transformer-based models in LTSF tasks, potentially redefining the landscape of time series forecasting.

As split learning (SL) becomes popular for addressing data privacy and resource limitation issues in edge model training process, it is crucial to pre-validate whether the extensive use of wireless and computing resources in SL can achieve the expected training latency and convergence. While digital twin network (DTN) can potentially estimate network strategy performance in pre-validation environments, they are still in their infancy for SL tasks, facing challenges like unknown non-i.i.d. data distributions, inaccurate channel states, and misreport resource availability across devices. To address these challenges, this paper proposes a TransNeural algorithm for DTN pre-validation environment to estimate SL latency and convergence. First, the TransNeural algorithm integrates transformers to efficiently model data similarities between different devices, considering different data distributions and device participate sequence greatly influence SL training convergence. Second, it leverages neural network to automatically establish the complex relationships between SL latency and convergence with data distributions, wireless and computing resources, dataset sizes, and training iterations. Deviations in user reports are also accounted for in the estimation process. Simulations show that the TransNeural algorithm improves latency estimation accuracy by 13.44% and convergence estimation accuracy by 116.74% compared to traditional equation-based methods.

Pulmonary sarcoidosis is a multisystem granulomatous interstitial lung disease (ILD) with a variable presentation and prognosis. Early accurate detection of pulmonary sarcoidosis may prevent progression to pulmonary fibrosis, a serious and potentially life-threatening form of the disease. However, the lack of a gold-standard diagnostic test and specific radiographic findings pose challenges in diagnosing pulmonary sarcoidosis. Chest computed tomography (CT) imaging is commonly used but requires expert, chest-trained radiologists to differentiate pulmonary sarcoidosis from lung malignancies, infections, and other ILDs. In this work, we developed a multichannel, CT and radiomics-guided ensemble network (RadCT-CNNViT) with visual explainability for pulmonary sarcoidosis vs. lung cancer (LCa) classification using chest CT images. We leverage CT and hand-crafted radiomics features as input channels, and a 3D convolutional neural network (CNN) and vision transformer (ViT) ensemble network for feature extraction and fusion before a classification head. The 3D CNN sub-network captures localized spatial information of lesions, while the ViT sub-network captures long-range, global dependencies between features. Through multichannel input and feature fusion, our model achieved the highest performance with accuracy, sensitivity, specificity, and combined AUC of 0.93±0.04, 0.94±0.04, 0.93±0.08 and 0.97 respectively in a 5-fold cross-validation study with pulmonary sarcoidosis (n=126) and LCa (n=93) cases. The model offers promising potential for improving the diagnosis of pulmonary sarcoidosis. A detailed ablation study showing the impact of CNN+ViT compared to CNN or ViT alone, and CT+radiomics input, compared to CT or radiomics alone, is also presented in this work.

Color image binarization plays a pivotal role in image preprocessing work, significantly impacting subsequent tasks, particularly in text recognition. This paper concentrates on Document Image Binarization (DIB), aiming to separate a image into foreground (text) and background (non-text content). Through a thorough analysis of conventional and deep learning-based approaches, we conclude that prevailing DIB methods leverage deep learning technology. Furthermore, we explore the receptive fields pre- and post-network training to underscore the Transformer model's advantages. Subsequently, we introduce a lightweight model based on the U-Net structure, enhanced with the Mobile ViT module to better capture global information features in document images. Given its adeptness at learning both local and global features, our proposed model exhibits superior performance on two standard datasets (DIBCO2012 and DIBCO2017) compared to state-of-the-art methods. Notably, our proposed DIB method presents a straightforward end-to-end model devoid of additional image preprocessing or post-processing. Moreover, its parameter count is less than a quarter of the HIP'23 model, which achives best results on three datasets(DIBCO2012, DIBCO2017 and DIBCO2018). Finally, two sets of ablation experiments were conducted to verify the effectiveness of the proposed binarization model.

The purpose of this study is to ​​explore the use of a custom Transformer model in brain-computer interfaces (BCIs) that translate the neural activity present when an individual with limited verbal and fine-motor skills attempts to handwrite. As found in previous studies, Transformers have performed better than recurrent neural networks (RNNs) in translation tasks which are similar to its usage here in decoding neural signals into intended handwritten text. Due to the known benefits of a Transformer, the hypothesis was that the Transformer would show promise in the context of a BCI through the recorded metrics. The neural signals of a tetraplegic individual, when they attempted to handwrite, were provided by existing research. Four trials were conducted using data with or without augmentation which the model used when training to separately minimize training loss and validation loss. When comparing the results of a BCI with the implementation of a Transformer model with the original RNN BCI (the original data source), the Transformer model performed less favorably across all four trials. Although the results do not indicate that the Transformer model currently outperforms an RNN BCI, it is important to note that further testing of the model's capabilities (such as training it with a larger and more preferable dataset and/or for longer, comparing training times between the RNN and Transformer, and/or seeing how the Transformer is improved with an offline autocorrect feature) is necessary before determining whether Transformers can enhance communication in this manner.

A digital twin is a virtual representation that serves as the real-time digital counterpart of a physical object or system. Recent advancements have extended the concept of digital twins to humans, incorporating complex biological data such as DNA (Predictiv Care, Inc.) and immune system profiles. These sophisticated models go beyond mere pictorial representations, offering a more holistic digital reflection. However, a significant gap remains. The current human digital twin models are not capable of learning one’s preferences. In this pilot study, we introduce Diginality, a chatbot powered by HyperCLOVA X, a Large Language Model (LLM) developed by NAVER, Inc. Diginality, learns one’s preferences with custom training from data collected by interview-style questions on the user’s topic of interest. Our findings demonstrate that Diginality successfully answers one’s preferences, thereby adding a new dimension to the concept of human digital twins. This work represents a pioneering step towards creating a more comprehensive and psychologically nuanced human digital twin.

Plasma generation by means of electrical discharge requires specialized power supply systems. The applicability of plasma for various plasma processes depends on its parameters, and these in turn depend on the parameters of the power supply systems. Arc plasma can be unstable, generating a lot of electromagnetic interference and overvoltage and overcurrent. The power system of a plasma reactor must guarantee good plasma control characteristics, be immune to disturbances and ensure good cooperation with the power grid. The article analyzes the cooperation of a three-phase plasma reactor, with gliding arc discharge, with a power supply system of a new type. This system integrates an AC/DC/AC converter with a five-column transformer of special design in a single device. Using the properties of magnetic circuits, it was possible to integrate the functions of ignition and sustaining the burning of the discharge in the reactor in a single transformer. Proper design of the transformer is crucial to achieve good cooperation of the AC/DC/AC converter with both the plasma reactor and the power supply network. The presented power supply design shows a number of positive features predisposing it to powering arc plasma reactors.

Identification of plants is a challenging task which aims to identify the family, genus, and species level according to morphological features. Automated deep learning-based computer vision algorithms are widely used for identifying plants and can help users to narrow down the possibilities. However, numerous morphological similarities between and within species make the classification difficult. In this paper, we tested a custom convolution neural network (CNN) and vision transformer (ViT) based models using the PyTorch framework to classify plants. We used a large dataset of 88K and 16K images for classifying plants at genus and species levels respectively. Our results show that for classifying plants at the genus level, ViT models perform better compared to CNN-based models ResNet50 and ResNet-RS-420, and other state-of-the-art CNN-based models suggested in previous studies on a similar dataset. The ViT model achieved top accuracy of 83.3% for classifying plants at the genus level. ViT models also perform better for classifying plants at the species level compared to CNN-based models ResNet50 and ResNet-RS-420, with a top accuracy of 92.5%. We show that the correct set of augmentation techniques plays an important role in classification success.

Due to the influence of background conditions, lighting conditions, occlusion issues and the image resolution, how to extract robust person features is one of the difficulties in ReID research. Vision in Transformers (ViT) has achieved significant results in the field of computer vision. However, the existing problems still limit its application in ReID due to slow extraction of person features and difficulty in utilizing local features of people. To solve the mentioned problems, we utilize Pyramid Vision Transformer (PVT) as the backbone of feature extraction and propose a PVT-based ReID method in conjunction with other studies. Firstly, some improvements suitable for ReID are used on the PVT backbone, and we establish a basic model by using powerful methods verified on CNN-based ReID. Secondly, in an effort to further promote the robustness of the person features extracted by the PVT backbone, two new modules are designed. (1) The local feature clustering (LFC) is recommend to enhance the robustness of person features by calculating the distance between local features and global feature to select the most discrete local features and clustering them. (2) The side information embeddings (SIE) are used to encode non-visual information and send it into the network for training to reduce its impact on person features. Finally, the experiments show that PVTReID has achieved excellent results in ReID datasets and are 20% faster on average than CNN-based ReID methods.

Fine-grained image classification remains an ongoing challenge in the computer vision field, which is particularly intended to identify objects within sub-categories. It is a difficult task since there is a minimal and substantial intra-class variance. The current methods address the issue by first locating selective regions with Region Proposal Networks (RPN), object localization, or part localization, followed by implementing a CNN Network or SVM classifier to those selective regions. This approach, however, makes the process simple by implementing a single-stage end-to-end feature encoding with a localization method, which leads to improved feature representations of individual tokens/regions by integrating the transformer encoder blocks into the Yolov5 backbone structure. These Transformer Encoder Blocks, with their self-attention mechanism, effectively captured the global dependencies and enabled the model to learn relationships between distant regions. This improved the model ability to understand context and captured long-range spatial relationships in the image. We also replaced the Yolov5 detection heads with three transformer heads at the output for object recognition using the discriminative and informative features maps from transformer encoder blocks. We established the potential of the single stage detector for the fine-grained image recognition task, by achieving state of the art 93.4% accuracy, as well as outperforming the existing Yolov5 model. The effectiveness of our approach is assessed using the Stanford car dataset, which includes 16,185 images of 196 different classes of vehicles with significantly identical visual appearances.

Safe, high-quality and economical electric energy transportation is the basic requirements of modern power system operation. Transformer, as one of the core equipment of power system and national grid system, has the characteristics of diverse types, variable models and wide deployment. It is the basic equipment for power system to realize voltage change and electric energy distribution. Because the power system transformer needs to operate with load for a long time, the probability of failure is usually higher than that of other power equipment in general. This paper starts from the transformer fault and fault diagnosis. Firstly, the transformer fault types, traditional transformer fault diagnosis techniques and the advantages of machine learning in transformer fault diagnosis are reviewed. Secondly, the application of information system theory and information entropy in transformer fault diagnosis is introduced. Then it introduces machine learning technology, feature selection and extraction technology in transformer fault diagnosis, and machine learning algorithms such as support vector machine and extreme learning machine for transformer fault diagnosis. Finally, the potential development trend of information system theory, information entropy and machine learning in transformer fault diagnosis is forecasted. The combination of transformer fault prediction and machine learning algorithm is helpful for power system maintenance personnel to accurately predict the running state of power equipment, and also provides a new method and technology for the safe and reliable operation and regular maintenance of power system.

This paper describes the successful application of the Transformer model used in the natural language processing and vision tasks as a means of processing the time series of signals from gyroscope and accelerometer sensors for the classification of human activities. The Transformer model is based on deep neural networks with many layers which can generalize well on signals. All measured signals come from a smartphone placed in a waist bag. Activity prediction is sequence-to-sequence, each time step of the signal is assigned a designation of the performed activity. Emphasis is placed on attention mechanisms, which express individual dependencies between signal values within a time series. In comparison with another recent result, the recognition precision was improved from 89.67 percent to 99.2 percent. The transformer model should in the future be included among the top options in machine learning methods for human activity recognition.

Low-frequency audible noise generated by the magnetostriction effect inherent to the operation of power transformers has become a major drawback, especially in cases where the electrical substation is located in urban areas subject to strict environmental regulations that imposes sound pressure limits, differing for day and night periods. Such regulations apply a +5 dB penalty if a tonal component of noise is present, which is clearly the case of magnetostriction noise, typically concentrated at twice the industrial frequency (50 Hz or 60 Hz, depending on the country). The strategy used to eliminate the tonal characteristics, therefore contributing to establish compliance with the applicable regulation and to alleviate the discomfort it causes to the human ear, consisted in superimposing to the substation noise a masking sound synthesized from “sounds of nature” with suitable intensities, to flatten the noise spectrum while enhancing the soundscape. The masking system (heavy-duty speakers powered by a microprocessor platform) was validated at an already judicialized urban scenario. Measurement results confirmed that the masking solution was capable of flattening the tonal frequencies, whose beneficial effect yielded the cancellation of the public civil action filed by the neighbors. The proposed solution is ready to be replicated to other scenarios.

The measurement range of a conventional current sensor is narrow because it is used with signals relative to the rated values of the measurement range from a voltage-type device. Consequently, multiple current sensors must be used in accordance with the measurement range. To address this problem, this paper proposes a new current sensor with a clamp-shaped part for low current measurement and a simple straight structure for high current measurement. The output signals of the proposed current sensor are amplified with a Hall element using the magnetic force of a rectangular air gap inside the clamp. To verify the characteristics of the proposed current sensor, a current was applied to an external load, and the value determined by the current sensor noted. Then, electromagnetic field analysis was performed through current sensor modeling and the results obtained compared to the actual sensor results. The proposed sensor had a 1% linearity in the output signals and exhibited dynamic characteristics over a wide current range.

Haze always shrouds satellite images, obscuring valuable geographic information for military surveillance, natural calamity surveillance and mineral resource exploration. Satellite image dehazing (SID) provides the possibility for better applications of satellite images. Most of the existing dehazing methods are tailored for natural images and are not very effective for satellite images with non-homogeneous haze since the semantic structure information and inconsistent attenuation are not fully considered. To tackle this problem, this study proposes a hierarchical semantic-guided contextual structure-aware network (SCSNet) for satellite image dehazing. Specifically, a hybrid CNN-transformer architecture integrated with a hierarchical semantic guidance (HSG) module is presented to learn semantic structure information by synergetically complementing local representation from non-local features. Furthermore, a cross-layer fusion (CLF) module is specially designed to replace the traditional skip connection during the feature decoding stage, so as to reinforce the attention to the spatial regions and feature channels with more serious attenuation. The results on the SateHaze1k, RS-Haze, and RSID datasets demonstrate that the proposed SCSNet can achieve effective dehazing and outperforms existing state-of-the-art methods.

In the realm of the five-category classification endeavor, there has been limited exploration into applied techniques for classifying Arabic text. These methods have primarily leaned on singletask learning, incorporating manually crafted features that lack robust sentence representations. Recently, the Transformer paradigm has emerged as a highly promising alternative. However, when these models are trained using single-task learning, they often face challenges in achieving outstanding performance and generating robust latent feature representations, especially when dealing with small datasets. This issue is particularly pronounced in the context of the Arabic dialect, which has a scarcity of available resources. Given these constraints, this study introduces an innovative approach to dissecting sentiment in Arabic text. This approach combines Inductive Transfer (INT) with the Transformer paradigm to augment the adaptability of the model and refine the representation of sentences. By employing self-attention SE-A and feed-forward sub-layers as a shared Transformer encoder for both the five-category and three-category Arabic text classification tasks, this proposed model adeptly discerns sentiment in Arabic dialect sentences. The empirical findings underscore the commendable performance of the proposed model, as demonstrated in assessments of the Hotel Arabic-Reviews Dataset, the Book Reviews Arabic Dataset, and the LARB dataset.

Agriculture is pivotal in national economies, with pest detection significantly influencing food quality and quantity. Pest classification remains challenging in automated agriculture monitoring systems, exacerbated by the non-uniform pest scales and the scarcity of high-quality datasets. In this study, we constructed a pest dataset by acquiring domain-agnostic images from the Internet and resizing them to a standardized 299x299 pixel format. Additionally, we employed diffusion models to generate supplementary data. While Convolutional Neural Networks (CNNs) are prevalent for prediction and classification, they often lack effective global information integration and discriminative feature representation. To address these limitations, we propose the RS Transformer, an innovative model that combines elements like the Region Proposal Network, Swin Transformer, and ROI Align. Additionally, we introduce the Randomly Generated Stable Diffusion Dataset (RGSDD) to augment the availability of high-quality pest datasets. Extensive experimental evaluations demonstrate the superiority of our approach compared to both two-stage models (SSD and Faster R-CNN) and one-stage models (YOLOv3, YOLOv4, YOLOv5m, YOLOv8, and DETR). We rigorously assess performance using metrics such as mean Average Precision (mAP), F1Score, Recall, and mean Detection Time (mDT). Our research contributes to advancing pest detection methodologies in automated agriculture systems, promising improved food production and quality.

Artificial intelligence (AI)-based models hold the potential to transform higher education if adopted properly and ethically. A prime example is ChatGPT with earlier studies indicating its widespread adoption in academia and by university students. The current study aimed to identify the factors influencing the attitude towards ChatGPT and its usage among university students in Arab countries. A previous survey instrument termed Technology Acceptance Model Edited to Assess ChatGPT Adoption (TAME-ChatGPT) was administered online using a convenience-based approach among the contacts of the authors. Confirmatory factor analysis (CFA) for the survey constructs was done using root mean square error of approximation (RMSEA), standardized root mean square residual (SRMR), comparative fit index (CFI), and Tucker-Lewis index (TLI). The final study sample comprised a total of 2240 participant divided as follows: Iraq (n=736, 32.9%), Kuwait (n=582, 26.0%), Egypt (n=417, 18.6%), Lebanon (n=263, 11.7%), and Jordan (n=242, 10.8%). A total of 1048 respondents heard of ChatGPT before the study (46.8%), of which 551 used ChatGPT (52.6%). The mean scores of TAME-ChatGPT constructs showed that the ease of ChatGPT use, positive attitude towards technology and social influence, higher perceived usefulness, the influence of behavioral/cognitive factors, low perceived risks and low anxiety were the determinants of positive attitude to ChatGPT and its use. For both the attitude and usage scales of TAME-ChatGPT, CFA collectively yielded satisfactory fit indices as indicated by low RMSEA and SRMR together with high CFI and TLI. Multivariate analysis showed that attitude to ChatGPT use was significantly influenced by country of residence, age, university type, and the latest grade point average of the students. The current study confirmed the validity of TAME-ChatGPT as a survey instrument to assess the possible determinants of ChatGPT use among university students in Arab countries. The study findings highlighted that successful adoption of ChatGPT in higher education could be dependent on perceived ease of use, usefulness, positive attitudes to technology, social influence, behavioral/cognitive factors, lower anxiety, and minimal perceived risks. The utility of ChatGPT in higher education requires policies that should be tailored for various settings, considering the differences observed in attitude towards ChatGPT among participating students in this study.

Infrared and visible image fusion technologies are used to characterize the same scene by diverse modalities. However, most existing deep learning-based fusion methods are designed as symmetric networks, which ignore the differences between modal images and lead to the source image information loss during feature extraction. In this paper, we propose a new fusion framework for the different characteristics of infrared and visible images. Specifically, we design a dual-stream asymmetric network with two different feature extraction networks to extract infrared and visible feature maps respectively. The transformer architecture is introduced in the infrared feature extraction branch, which can force the network to focus on the local features of infrared images while still obtaining their contextual information. And the visible feature extraction branch uses residual dense blocks to fully extract the rich background and texture detail information of visible images. In this way, it can provide better infrared targets and visible details for the fused image. Experimental results on multiple datasets indicate that DSA-Net outperforms state-of-the-art methods in both qualitative and quantitative evaluations. In addition, we also apply the fusion results to the target detection task, which indirectly demonstrates the fusion performances of our method.

Many engineering systems are described by using differential models whose solutions, generally obtained after discretization, can exhibit a noticeable deviation with respect to the response of the physical systems that those models are expected to represent. In those circumstances one possibility consists of enriching the model in order to reproduce the physical system behavior. The present paper considers a dynamical system, and proposes enriching the model solution by learning the dynamical model of the gap between the system response and the model-based prediction, while ensuring that the time integration of the learnt model remains stable. The proposed methodology is applied on the simulation of the top-oil temperature evolution of a power transformer, whose data is provided by RTE, the French Electricity Transmission System Operator.

The battle of currents between AC and DC reignited as a result of the development in the field of power electronics. The efficiency of DC distribution systems is highly dependent on the efficiency of distribution converter, which calls for optimized schemes for efficiency enhancement of distribution converters. Modular solid-state transformers play a vital role in DC Distribution Networks and Renewable Energy systems (RES).This paper deals with efficiency-based load distribution for Solid State Transformers (SSTs) in DC distribution networks. Aim is to achieve a set of minimum inputs that are consistent with output while considering constraints and efficiency. As the main feature of modularity is associated with a three-stage structure of SSTs. This modular structure has been optimized using Ant Lion Optimizer (ALO) and validated by applying it EIA (Energy Information Agency) DC Distribution Network which contains SSTs. In the DC distribution grid, modular SSTs provide promising conversion of DC power from medium voltage to lower DC range (400V). The proposed algorithm is simulated in MATLAB and also compared with two other metaheuristic algorithms. The obtained results prove that the proposed method can significantly reduce input requirements for producing the same output while satisfying the specified constraints.

3D point cloud classification tasks have been a hot topic in recent years. Most existing point cloud processing frameworks lack context-aware features due to the deficiency of sufficient local feature extraction information. Therefore, we design an augmented sampling and grouping (ASG) module to efficiently obtain fine-grained features from the original point cloud. In particular, this method strengthens the domain near each centroid and makes reasonable use of the local mean and global standard deviation to mine point cloud’s local and global features. In addition to this, inspired by the transformer structure UFO-ViT in 2D vision tasks, we first try to use a linearly-normalized attention mechanism in point cloud processing tasks, investigating a novel transformer-based point cloud classification architecture UFO-Net. An effective local feature learning module is adopted as a bridging technique to connect different feature extraction modules. Importantly, UFO-Net employs multiple stacked blocks to better capture feature representation of the point cloud. Extensive ablation experiments on public datasets show that our method outperforms other state-of-the-art methods. For instance, our network performed with 93.7% overall accuracy on the ModelNet40 dataset, which was 0.5% higher than PCT. Our network also archived 83.8% overall accuracy on the ScanObjectNN dataset, which is 3.8% better than PCT.

Indoor localization is an active area of research dominated by traditional machine-learning techniques. Deep learning-based systems have shown unprecedented improvements and have accomplished exceptional results over the past decade, especially the Transformer network within natural language processing (NLP) and computer vision domains. We propose the hyper-class Transformer (HyTra), an encoder-only Transformer with multiple classification heads (one per class) and learnable embeddings, to investigate the effectiveness of Transformer-based models for received signal strength (RSS) based WiFi fingerprinting. HyTra leverages learnable embeddings and the self-attention mechanism to determine the relative position of the wireless access points (WAPs) within the high-dimensional embedding space, improving the prediction of user location. From an NLP perspective, we consider a fixed order sequence of all observed WAPs as a sentence and the captured RSS value(s) for every given WAP at a given reference point from a given user as words. We test our proposed network on public and private datasets of different sizes, proving that the quality of the learned embeddings and overall accuracy improves with increments in samples.

Predicting Length of Stay (LoS) and understanding its underlying factors is essential to minimize the risk of hospital-acquired conditions, improve financial, operational, and clinical outcomes, and to better manage future pandemics. The purpose of this study is to forecast patients’ LoS using a deep learning model and analyze cohorts of risk factors minimizing or maximizing LoS. We employed various pre-processing techniques, SMOTE-N to balance data, and Tab-Transformer model to forecast LoS. Finally, Apriori algorithm was applied to analyze cohorts of risk factors influencing LoS at hospital. The Tab-Transformer outperformed the base Machine Learning models with an F1-score (.92), precision (.83), recall (.93), and accuracy (.73) for discharge dataset, and F1-score (.84), precision (.75), recall (.98), and accuracy (.77) for deceased dataset. The association mining algorithm was able to identify significant risk factors/indicators belonging to lab, X-Ray, and clinical data such as elevated LDH, and D-Dimer, lymphocytes count, and comorbidities such as hypertension and diabetes responsible for extending patients LoS. It also reveals what treatments has reduced the symptoms of COVID-19 patients leading to reduction in LoS particularly when no vaccines or medication such as Paxlovid were available.

A low-cost medium power carbon dioxide (CO₂) laser system is designed, constructed, and characterized to produce coherent, monochromatic laser radiation in the Infrared region. The laser cavity is simulated and designed by using ZEMAX optic studio. A switch-mode high tension pump source is designed and constructed using a flyback transformer and simulated using NI Multisim to study the voltage behavior at different node points. A prototype cooling system/chiller is designed and built using the Thermo-Electric Coolers (TEC) to remove the excess heat produced during laser action. Various parameters, like pumping mechanism, chiller stability, efficiency, output power, and current at different applied voltages, are studied. The chiller efficiency at different output powers of the laser is analyzed, which clearly shows that the chiller's cooling rate is good enough to compensate for the heat generated by the laser system. The center wavelength of the carbon dioxide laser is 10.6μm with FWHM of 1.2nm simulated in the ZEMAX optic studio. The output beam penetration through salt rock (NaCl), wood, and acrylic sheet at various output powers is analyzed to measure the penetration depth rate of the CO₂ laser.

Neural decoding from spiking activity is an essential tool for understanding the information encoded in population neurons, especially in applications like brain-computer interface (BCI). Various quantitative methods have been proposed and have shown superiorities under different scenarios respectively. From the machine learning perspective, the decoding task is to map the high-dimensional spatial & temporal neuronal activity to the low-dimensional physical quantities (e.g., velocity, position). Because of the complex interactions and the abundant dynamics among neural circuits, good decoding algorithms usually have the capability of capturing flexible spatiotemporal structures embedded in the input feature space. Recently, the Transformer-based models are widely used in processing natural languages and images due to its superior performances in handling long-range and global dependencies. Hence, in this work we examine the potential applications of Transformers in neural decoding and introduce two Transformer-based models. Besides adapting the Transformer to neuronal data, we also propose a data augmentation method for overcoming the data shortage issue. We test our models on three experimental datasets and their performances are comparable to the previous state-of-the-art (SOTA) RNN-based methods. In addition, Transformer-based models show increased decoding performances when the input sequences are longer, while LSTM-based models deteriorate quickly. Our research suggests that Transformer-based models are important additions to the existing neural decoding solutions, especially for large datasets with long temporal dependencies.

Studies on integrating Spiking Neural Networks (SNNs) with the Transformer architecture holds promise for enabling models to achieve ultra-low energy consumption while possessing the performance of the Transformer architecture. Currently, studies on ANN-to-SNN conversion of integrating Spiking Neural Networks (SNNs) with the Transformer architecture mainly focuses on simple activation functions in MLPs, and has not yet addressed the mismatch between the Softmax activation function in the self-attention mechanism and the computation rules of SNNs. Consequently, the ANN-to-SNN conversion efforts have consistently failed to make the Transformer architecture directly applicable to SNNs. To address this challenge, we propose the Spiking-Softmax method, which integrates Spiking Exponential Neuron (SI-exp) and Spiking Collaboration Normalized Neuron (SI-norm). The Spiking-Softmax method accurately simulates the Softmax activation function with only 12 time steps. Building upon this, we propose the Spike Integrated Transformer conversion (SIT-conversion) method, which enables the conversion of the Transformer architecture to SNNs. The SNNs generated by the SIT-conversion of Transformer models of various sizes achieve accuracy nearly identical to their ANNs counterparts, achieving nearly lossless and ultra-low-latency ANN-to-SNN conversion. This work represents the first implementation of simulating the Softmax activation function and fully converting the Transformer architecture into SNNs through spike firing.

Functional graded materials (FGMs) have been successfully applied in improving the insulation performance of insulators, yet there is currently no relevant literature on their application in dry-type transformers. Therefore, it is meaningful to study on the feasibility of using FGMs to improve the insulation performance of dry-type transformers through research on electric field distribution. This paper sets FGMs in the dry-type transformer spacer only or both spacer and winding end to change the relative permittivity values in the corresponding areas. By comparing the electric field distribution characteristics on the inner and outer surfaces of the high-voltage coil and the lower surface of the spacer of the dry-type transformer under each control method with the case without setting FGMs, the study found that when FGMs were only set on the spacer, the control methods set in this paper could not improve the electric field distribution of the transformer. When FGMs are employed at the spacers and end windings, two modulation strategies can mitigate the electric field intensity on the outer surface of high-voltage windings. Specifically, a continuous increase in relative permittivity can reduce the electric field intensity by 34.02%, while an initial decrease followed by an increase can result in a reduction of 13.58%. This paper provides theoretical guidance for insulation design of transformers with high voltage levels of 66 kV and above, especially as transformers with high voltage levels face greater challenges in regulating surface electric fields.

Recommendation systems are crucial in navigating the vast digital market. However, user data’s dynamic and non-stationary nature often hinders their efficacy. Traditional models struggle to adapt to the evolving preferences and behaviours inherent in user interaction data, posing a significant challenge for accurate prediction and personalisation. Addressing this, we propose a novel theoretical framework, the Non-stationary Transformer, designed to capture and leverage the temporal dynamics within data effectively. This approach enhances the traditional transformer architecture by introducing mechanisms accounting for non-stationary elements, offering a robust and adaptable solution for recommendation systems. Our experimental analysis, encompassing deep learning and reinforcement learning paradigms, demonstrates the framework’s superiority over benchmark models. The empirical results confirm the efficacy of our proposed framework, which not only provides significant performance enhancements, approximately 8% in Logloss reduction and up to 2% increase in F1 score but also underscores its potential applicability across accumulative reward scenarios. These findings advocate adopting Non-stationary Transformer models to tackle the complexities of today’s recommendation tasks.

Few-shot object detection (FSOD) aims to address the challenge of requiring a substantial amount of annotations for training in conventional object detection, which is very labor-intensive. However, existing few-shot methods achieve high precision with the sacrifice of time-consuming for exhaustive fine-tuning, or take poor performance in novel-class adaptation. We presume the major reason is that the valuable correlation feature among different categories is insufficiently exploited, hindering the generalization of knowledge from base to novel categories for object detection. In this paper, we propose Few-Shot object detection via Correlation-RPN and Transformer Encoder-Decoder (CRTED), a novel training network to learn object-relevant features of inter-class correlation and intra-class compactness while suppressing object-agnostic features in the background with limited annotated samples. And we also introduce a 4-way tuple-contrast training strategy to positively activate the training progress of our object detector. Experiments over two few-shot benchmarks (Pascal VOC, MS-COCO) demonstrate that, our proposed CRTED without further fine-tuning can achieve comparable performance with current state-of-the-art fine-tuned works. The codes and pre-trained models will be released.

Code summarization plays a vital role in aiding developers with program comprehension by generating corresponding textual descriptions for code snippets. While recent approaches have concentrated on encoding the textual and structural characteristics of source code, they often neglect the global hierarchical features causing limited code representation. Addressing this gap, our paper introduces the Statement-grained Hierarchy enhanced Transformer model (SHT), a novel framework that integrates global hierarchy, syntax, and token sequences to automatically generate summaries for code snippets. SHT is distinctively designed with two encoders to learn both hierarchical and sequential features of code. One relational attention encoder processes the statement-grained hierarchical graph, producing hierarchical embeddings. Subsequently, another sequence encoder integrates these hierarchical structures with token sequences. The resulting enriched representation is then fed into a vanilla Transformer decoder, which effectively generates concise and informative summarizations. Our extensive experiments demonstrate that SHT significantly outperforms state-of-the-art approaches on two widely used Java benchmarks. This underscores the effectiveness of incorporating global hierarchical information in enhancing the quality of code summarizations.

Accurate estimation of Remaining Useful Life (RUL) for aircraft engines is essential for ensuring safety and uninterrupted operations in the aviation industry. Numerous investigations have leveraged the success of attention-based Transformer architecture in sequence modeling tasks, particularly in its application to RUL prediction. These studies primarily focus on utilizing onboard sensor readings as input predictors. While various Transformer-based approaches have demonstrated improvement in RUL predictions, their exclusive focus on temporal attention within multivariate time series sensor readings, without considering sensor-wise attention, raises concerns about potential inaccuracies in RUL predictions. To address this concern, our paper proposes a novel solution in the form of a two-stage attention based hierarchical transformer (STAR) frame-work. This approach incorporates a two-stage attention mechanism, systematically addressing both temporal and sensor-wise attentions. Furthermore, we enhance the STAR RUL prediction framework by integrate hierarchical encoder-decoder structures to capture valuable information across different time scales. By conducting extensive numerical experiments with the CMAPSS datasets, we demonstrate that our proposed STAR framework significantly outperforms current state-of-the-art models for RUL prediction.

Solid-state transformer (SST) is a promising technology for smart grids due to its flexible power control (better reliability) and high efficacy (by decreasing losses) compared with traditional transformers. The SST is designed in tandem with three stages, i.e., the input, isolation, and output stages. As three converter stages are used to design the SST to make a more reliable device in practice, it is highly prone to instabilities. Moreover, the key objective of SST design is to implement a modern power distribution system to make it more intelligent and reliable. The stability issue can be even more complicated by the presence of distributed energy resources in the distribution system. Thus, the stability of SST must be measured prior to /during the design, and to determine, a state-space model is used, which is developed in this paper. Each stage of SST is modeled with its controller, and the system's stability is measured through the controllability and observability test. Further, the conversion from state-space to transfer function model is done, and the stability of SST is shown using frequency and time-domain diagrams. With these, the different plots, namely, the Bode plot, Nyquist plot, Nichols chart, Root locus, pole-zero plot, and Eigen plot, are used to ascertain the relative and absolute stability. Finally, the SST Simulink model is tested and validated through hardwate-in-the-loop simulation, i.e., OPALRT, to show its effectiveness and applicability.

In the realm of power delivery to end-users, transformers are indispensable, with their malfunctions leading to substantial economic, safety, and environmental repercussions. The need for persistent surveillance is accentuated for in-situ oil-filled transformers, given the potential degradation of oil and the emergence of related ageing by-products. As the focus tilts towards online detection methodologies for transformer oil ageing, bypassing challenges associated with traditional offline methods such as sample contamination and misinterpretation, fiber optic sensors are gaining trac-tion due to their compact nature, cost-effectiveness, and resilience to electromagnetic disturbances typical in high-voltage environments. This work delves into the sensitivity analysis of intensi-ty-modulated plastic optical fiber sensors. The investigation encompasses key determinants such as the influence of optical source wavelengths, noise response dynamics, ramifications of varying sensing lengths, and repeatability assessments. Findings underscore that elongating the sensing length detrimentally affects both the linearity response and repeatability, largely attributed to the diminished resistance to external noise. Additionally, the choice of the optical source wavelength proved to be a critical variable in assessing sensor sensitivity.

Social media platforms are increasingly enabling the propagation of content from groups related to drug use, thus posing risks for the wider population and, in particular, individuals who are amenable to drug use and drug addiction. The detection of drug use content on social media platforms is a priority for governments, technology companies, and drug law enforcement organizations. To counter this issue, various techniques have been developed to identify and promptly remove drug use content, while also blocking its creators from network access. In this paper, we introduce a manually annotated Twitter dataset, comprising 156,521 tweets published between 2008 and 2022, specifically compiled for the purpose of drug use detection. The dataset underwent annotation by several group of expert annotators who classified the tweets as either drug use or non-drug use. Exploratory data analysis was conducted to comprehend the dataset's characteristics. Various classification algorithms, including SVM, XGBoost, RF, NB, LSTM, and BERT were employed using the dataset. Among the traditional machine learning models, SVM utilizing term frequency-inverse document frequency features achieved the highest F1-Score (0.9017). However, BERT with textual features concatenated with numerical and categorical features in ensemble method surpassed the performance of traditional models, attaining F1-Score of 0.9112. To facilitate future research and enhance English online drug use classification accuracy, the dataset will be made publicly available.

The continuous advancements in healthcare technology have empowered the discovery, diagnosis, and prediction of diseases, revolutionizing the field. Artificial intelligence (AI) is expected to play a pivotal role in achieving the goals of precision medicine, particularly in disease prevention, detection, and personalized treatment. This study aims to determine the optimal combination of the mother wavelet and AI model for the analysis of pediatric electroretinogram (ERG) signals. The dataset, consisting of signals and corresponding diagnoses, undergoes Continuous Wavelet Transform (CWT) using commonly used wavelets to obtain a time-frequency representation. Wavelet images were used for the training of five widely used deep learning models: VGG-11, ResNet-50, DensNet-121, ResNext-50, and Vision Transformer, to evaluate their accuracy in classifying healthy and unhealthy patients. The findings demonstrate that the combination of Ricker Wavelet and Vision Transformer consistently yields the highest median accuracy values for ERG analysis, as evidenced by the upper and lower quartile values. The median balanced accuracy of the obtained combination of the three considered types of ERG signals in the article are 0.83, 0.85, and 0.88. However, other wavelet types also achieved high accuracy levels, indicating the importance of carefully selecting the mother wavelet for accurate classification. The study provides valuable insights into the effectiveness of different combinations of wavelets and models in classifying ERG wavelet scalograms.

This primer provides an overview of the rapidly evolving field of generative artificial intelligence, specifically focusing on large language models like ChatGPT (OpenAI) and Bard (Google). Large language models have demonstrated unprecedented capabilities in responding to natural language prompts. The aim of this primer is to demystify the underlying theory and architecture of large language models, providing intuitive explanations for a broader audience. Learners seeking to gain insight into the technical underpinnings of large language models must sift through rapidly growing and fragmented literature on the topic. This primer brings all the main concepts into a single digestible document. Topics covered include text tokenization, vocabulary construction, token embedding, context embedding with attention mechanisms, artificial neural networks, and objective functions in model training. The primer also explores state-of-the-art methods in training large language models to generalize on specific applications and to align with human intentions. Finally, an introduction to the concept of prompt engineering highlights the importance of effective human-machine interaction through natural language in harnessing the full potential of artificial intelligence chatbots. This comprehensive yet accessible primer will benefit students and researchers seeking foundational knowledge and a deeper understanding of the inner workings of existing and emerging artificial intelligence models. The author hopes that the primer will encourage further responsible innovation and informed discussions about these increasingly powerful tools.

Large Language Models (LLMs) have revolutionized the field of Artificial Intelligence with their ability to understand and generate natural language discourse. This has led to the use of LLMs in various creative fields such as music, art, and storytelling, which has sparked a debate on the impact of LLMs on human creativity. This article explores the cyclic effect of LLMs on human creativity, where LLMs can generate an ever-evolving cycle of creative output from both humans and machines. The article also highlights the importance of responsible and ethical use of LLMs, including privacy and copyright issues with the training data used by LLMs. The article emphasizes the need for transparency, data security measures, and AI governance policies to protect user data and ensure the safe use of LLMs. It also calls for the development of educational materials and resources to increase public understanding of LLMs and their ethical implications. The article concludes by highlighting the potential of LLMs as powerful tools for communication and education, but also emphasizes the need for ethical considerations to ensure the responsible and beneficial use of LLMs in society.

Paraphrase Generation is one of the most important and challenging tasks in the field of Natural Language Generation. The paraphrasing techniques help to identify or to extract/generate phrases/sentences conveying the similar meaning. The paraphrasing task can be bifurcated into two sub-tasks namely, Paraphrase Identification (PI) and Paraphrase Generation (PG). Most of the existing proposed state-of-the-art systems have the potential to solve only one problem at a time. This paper proposes a light-weight unified model that can simultaneously classify whether given pair of sentences are paraphrases of each other and the model can also generate multiple paraphrases given an input sentence. Paraphrase Generation module aims to generate fluent and semantically similar paraphrases and the Paraphrase Identification systemaims to classify whether sentences pair are paraphrases of each other or not. The proposed approach uses an amalgamation of data sampling or data variety with a granular fine-tuned Text-To-Text Transfer Transformer (T5) model. This paper proposes a unified approach which aims to solve the problems of Paraphrase Identification and generation by using carefully selected data-points and a fine-tuned T5 model. The highlight of this study is that the same light-weight model trained by keeping the objective of Paraphrase Generation can also be used for solving the Paraphrase Identification task. Hence, the proposed system is light-weight in terms of the model’s size along with the data used to train the model which facilitates the quick learning of the model without having to compromise with the results. The proposed system is then evaluated against the popular evaluation metrics like BLEU (BiLingual Evaluation Understudy):, ROUGE (Recall-Oriented Understudy for Gisting Evaluation), METEOR, WER (Word Error Rate), and GLEU (Google-BLEU) for Paraphrase Generation and classification metrics like accuracy, precision, recall and F1-score for Paraphrase Identification system. The proposed model achieves state-of-the-art results on both the tasks of Paraphrase Identification and paraphrase Generation.

Since the beginning of the internet era, there has been an explosion of growth in structured data (such as numbers, symbols, and labels) as well as unstructured data (including images, videos, and text). Efficient and accurate mixed query of these two types of data is a key technology to achieve high-quality information retrieval, and it is also a major challenge that needs to be solved urgently in the industry. In this study, we employ an advanced Transformer model that combines strategies and fine-tuning techniques for multi-task learning. Specifically, the model is first pre-trained on a large-scale, general-purpose dataset to learn different types of data representations and basic language comprehension skills. After that, we fine-tuned the parameters of the model to better suit these specific data processing tasks for specific application scenarios, such as image annotation, video content analysis, and structured data query. At the heart of the model is the self-attention mechanism, which allows the model to automatically emphasize the important parts and ignore irrelevant information when processing the input data. In addition, we have introduced task-specific adaptation layers that are designed to add additional processing power to the original Transformer architecture, such as a semantic analysis layer for unstructured text data and a relational extraction layer for structured data. This combination of general pre-training and task-specific fine-tuning allows the model to flexibly process and integrate information from different data sources, improving processing efficiency and accuracy. Experimental results show that the model performs well in a variety of data processing tasks, significantly improves the accuracy and efficiency of information retrieval, and verifies the strong potential and adaptability of large language models in processing mixed data types.

To enhance the accuracy and efficiency of address parsing in traffic accident handling, this study designs a deep address parsing model based on BERT, named ASSPM. This model effectively addresses the limitations of existing deep learning models in Chinese address parsing tasks by integrating address spatial semantics. First, a lexicon-enhancement method based on hybrid representation is employed, transforming the spatial semantics extracted by the Address Spatial Semantics Extractor (ASSE) into a symbolic semantic space that can be integrated with the feature vector space of the deep model. This achieves effective fusion of spatial semantics and the deep model at the model level. Second, a lexicon adapter enhanced with spatial semantics is utilized to integrate spatial semantics into the Transformer layers of the BERT model, achieving semantic fusion at the BERT level. Finally, comparative experiments on a custom address dataset and a public Chinese address dataset validate the proposed method’s superior performance over other baseline methods. This research provides robust support for the application of deep address parsing models integrating address spatial semantics in traffic accident handling, promising to improve the efficiency and accuracy of traffic accident management.

Electric Transformers make up some of the most critical components of a power system. Their reliable operation is critical for an efficient power grid as well as economic viability for the transformer operators and owners. Power transformers, like other High Voltage (HV) electrical equipment experience aging and insulation degradation due to chemical, mechanical and electrical forces during their operation. Partial Dischargers (PD) are localized electrical discharges that develop within insulation systems of HV electrical equipment such as transformers. In transformers, PD occurs in different forms and various locations and insulation both internal and external of the transformer. PD originates as small pulses but tends to increase in size and intensity which can result in complete insulation failure. Monitoring partial discharges has proven to provide valuable information on the state of the insulation systems of the power transformer, allowing transformer operators to make calculated decisions for the maintenance and life of plant plans. This systematic literature review aims to systematically examine the use of machine learning techniques in classifying PD in transformers to present a complete indicator to allow for future research in the field. The systematic review surveyed a total of 81 research literature published from 2010 to 2023 that fulfilled a specific methodology which was developed as part of this study. The results revealed that supervised learning has been the most widely used Artificial Intelligence (AI) algorithm utilized, primarily in the form of Support Vector Machine (SVM). Regarding PD, the survey revealed that most researchers tend to use numerous types of PD and compare them to one another. Furthermore, the use of artificial PD defect models to simulate the occurrence of PD is widely used versus the use of actual power transformers. Most of the literature tends to not specify the physical characteristics of PD, such as the magnitude of PD, PD inception voltage and PD extinction voltage.

Surface water plays a crucial role in climate change, human production, and life, making accurate monitoring and observation of surface water particularly important. However, due to the significant diversity and complexity of water distribution in surface space, accurate mapping of surface water faces considerable challenges. When extracting water bodies from medium-resolution satellite remote sensing images, CNN methods may suffer from limitations in receptive fields and insufficient context modeling capabilities, leading to the loss of water body boundary details and poor fusion of multiscale features. Currently, there is relatively little research on this issue; therefore, it is necessary to explore new combinations of deep learning networks to address these challenges. The purpose of this study is to address the above issues. We propose a new combination of deep learning networks that fully utilize multiscale information to enhance water features. Specifically, we first combine deformable convolutions with the Swin Transformer to increase effective receptive fields while better integrating global semantic information. This combination can capture features of water bodies at different scales, improve the accuracy and integrity of water extraction, and provide reliable technical support for detailed water body extraction. We tested the newly constructed model using Sentinel-2 satellite images. Our model achieved results of over 90%, with an average accuracy of 97.89%, average precision of 94.98%, average recall of 90.05%, and an average F1 score of 92.33%. In addition, our model achieved an accuracy of 98.03% in mountainous areas. Our experiments and results validate the potential of combining the Swin Transformer and deformable convolutions in detailed water body extraction.

Estimating the Remaining Useful Life (RUL) of aircraft engines holds a pivotal role in enhancing safety, optimizing operations, and promoting sustainability, thus being a crucial component of modern aviation management. Precise RUL predictions offer valuable insights into an engine’s condition, enabling informed decisions regarding maintenance and crew scheduling. In this context, we propose a novel RUL prediction approach in this paper, harnessing the power of Bi-directional LSTM and Transformer architectures, known for their success in sequence modeling, such as natural languages. We adopt the encoder part of the full Transformer as the backbone of our framework, integrating it with a self-supervised denoising autoencoder that utilizes Bidirectional LSTM for improved feature extraction. Within our framework, a sequence of multivariate time series sensor measurements serves as the input, initially processed by the Bidirectional LSTM autoencoder to extract essential features. Subsequently, these feature values are fed into our Transformer encoder backbone for RUL prediction. Notably, our approach simultaneously trains the autoencoder and Transformer encoder, different from the naive sequential training method. Through a series of numerical experiments carried out on the C-MAPSS datasets, we demonstrate that the efficacy of our proposed models either surpasses or stands on par with that of other existing methods.

Human pose estimation is a complex detection task in which the network needs to capture the rich information contained in the images. In this paper, we propose MSTPose (Multi-Scale Transformer for human Pose estimation). Specifically, MSTPose leverages a high-resolution Convolution neural network (CNN) to extract texture information from images. For the feature maps from three different scales produced by the backbone network, each branch performs the coordinate attention operations. The feature maps are then spatially and channel-wise flattened, combined with keypoint tokens generated through random initialization, and fed into a parallel Transformer structure to learn spatial dependencies between features. As the Transformer outputs one-dimensional sequential features, the mainstream two-dimensional heatmap method is abandoned in favor of one-dimensional coordinate vector regression. The experiments show that MSTPose outperforms other CNN-based pose estimation models and demonstrates clear advantages over CNN + Transformer networks of similar types.

The computer network world is changing and the NetDevOps approach has brought the dynamics of applications and systems into the field of communication infrastructure. Businesses are changing and businesses are faced with difficulties related to the diversity of hardware and software that make up those infrastructures. The "Intent-Based Networking - Concepts and Definitions" document describes the different parts of the ecosystem that could be involved in NetDevOps. The recognize, generate intent, translate and refine features need a new way to implement algorithms. This is where artificial intelligence comes in.

Sentiment analysis is a pivotal tool for gauging the public’s perception and understanding of human communication across digital social media platforms. However, due to linguistic complexities and limited resources, sentiment analysis is not well-represented in many African languages. While benchmark Africa-Centric Pre-trained Language Models (PLMs) have been developed for various Natural Language Processing (NLP) tasks, their applications in eXplainable Artificial Intelligence (XAI) remain unexplored. In this study, we introduce a novel approach that combines Africa-centric PLMs with XAI techniques for sentiment analysis. We demonstrate that applying attention mechanisms and visualisation techniques improves the transformer-based model’s transparency, trustworthiness, and decision-making abilities when making sentiment predictions. We then employ the SAfriSenti—a multilingual sentiment corpus for South African under-resourced languages. We use the corpus to perform various sentiment analysis experiments and also enable comprehensive evaluations, comparing the performance of Africa-centric models against mainstream PLMs. The Afro-XLMR model outperformed all models and achieved an average F1-score performance of 71.04% across the five tested languages and the lowest error rate among the evaluated models. Additionally, we incorporated techniques like Local Interpretive Model-Agnostic Interpretation (LIME) and Shapley Additive Interpretation (SHAP) in the sentiment classifier’s output to enhance the Afro-XLMR model’s interpretability and explainability. As a result, the use of XAI strategies ensures that sentiment predictions are not only accurate and interpretable but also understandable, fostering trust and reliability in the decision-making of AI-driven NLP technologies, particularly in the context of African languages.

In Brain-Computer Interface Motor Imagery (BCI-MI) systems, Convolutional Neural Networks (CNNs) have traditionally dominated as the deep learning method of choice, demonstrating significant advancements in state-of-the-art studies. Recently, Transformer models with attention mechanisms have emerged as a sophisticated technique, enhancing the capture of long-term dependencies and intricate feature relationships in BCI-MI. This research investigates the performance of EEG-TCNet and EEG-Conformer models, which are trained and validated using various hyperparameters and bandpass filters during preprocessing to assess improvements in model accuracy. Additionally, this study introduces the EEG-TCNTransformer, a novel model that integrates the convolutional architecture of EEG-TCNet with a series of self-attention blocks employing a multi-head structure. The EEG-TCNTransformer achieves an accuracy of 82.97% without the application of bandpass filtering. The source code of EEG-TCNTransformer is available on GitHub.

The development of new turbomachinery designs requires numerous time-consuming and computationally intensive computational fluid dynamics (CFD) calculations. However, most of the generated high spatial resolution data remains unused at later development steps. That is also the case with automated optimization processes that use only a few integral values to determine objectives and constraints. Therefore the development of a data-driven AI model was initiated to ensure the potential of further use of the CFD data which is also used to train the AI model. The presented method subsequently provides a fast approximation of the 3D flow for new designs. In this paper, the structure of the developed AI model is presented and the approximation quality is analysed using a complex, state-of-the-art compressor test case. It is shown that the AI model can reproduce many characteristics of the 3D flow of new designs, and performance measures such as efficiency can be derived from these flow predictions. In addition, the complex test case revealed that greater design variation reduces the AI approximation quality which can lead to undesirable exploratory behaviour in an optimisation setup. Overall, the test case has shown promising results and has provided hints for further improvements of the AI model.

To enhance the overall reliability of the power system, engineers have redirected their focus toward health monitoring and early detection of faults in transformers. Among these faults, transformer core defects demand particular attention. While fault simulation using software has traditionally been the preferred approach, these methods suffer from data inaccuracies in real-world conditions. Consequently, conducting actual experimental setups with induced faults is imperative to investigate core issues. This study uses Hilbert Transform (HT) as a signal processing technique to extract crucial data characteristics, thereby enhancing the performance of the classifier model. The research involves analyzing electric current signals from a single-phase 1kVA transformer. A comparative assessment of our proposed model was conducted using raw data and Fast Fourier Transform (FFT), evaluating accuracy, precision, recall, F1-score, and computational time. The results demonstrate a significant improvement in all metrics for the classifier models, particularly the k-nearest neighbor (KNN) algorithm, which exhibited values of 83.89%, 84.39%, 83.89%, 83.79%, and 0.0156 seconds, respectively. Future work aims to extend this analysis under different loading conditions.

The formation of the required AC voltage level is currently one of the most important problems. An unstable voltage level can lead to the failure of domestic and industrial equipment, also to the appearance of a lighting equipment flicker effect, which can adversely affect the operating mode. In this regard, the development of AC voltage regulators that perform the function of stabilizers has become relevant. In turn, new topologies should be energy efficient and reliable, this can be achieved by reducing the number of semiconductor elements, which in turn will reduce losses and increase efficiency. In addition to voltage stabilization, AC voltage regulators have found application as soft-start devices for asynchronous motors, and have become competitors to frequency converters. The power level can vary from units to tens of kilowatts where such devices can be used. This paper provides a review of AC voltage regulators, which can be used both as soft starters and AC voltage stabilizers. The analysis of the circuits was carried out and the efficiency was evaluated through their total harmonic distortions (THD), power factor and efficiency coefficient. Experiments were also performed for a number of circuits, where their operating modes were evaluated. Eventually, all circuits were divided into full number of switches with two or more AC switches per phase, with a reduced number of switches, where one AC switch per phase and with low number of switches – with one complementary pair of AC switches for all three phases.

"Finding fresh water in the ocean of data." is a challenge that all deep learning domains struggle with, especially in the area of hyperspectral image analysis. As hyperspectral remote sensing technology advances by leaps and bounds, there are increasing amounts of hyperspectral images(HSIs) can be available. Whereas in fact, these unlabeled HSIs are powerless to be used as material to driven a supervised learning task due to the extremely expensive labeling costs and some unknown regions. Although learning-based methods have achieved remarkable performance due to their superior ability to represent features, at the cost, these methods are complex, inflexible and tough to carry out transfer learning. In this paper, we propose the "Instructional Mask AutoEncoder"(IMAE), which is a simple and powerful self-supervised learner for HSI classification that uses a transformer-based mask autoencoder to extract the general features of HSIs through a self-reconstructing agent task. Moreover, we utilize the metric learning to perform an instructor which can direct the model focus on the human interested region of the input so that we can alleviate the defects of transformer-based model such as local attention distraction, lack of inductive bias and tremendous training data requirement. In downstream forward propagation, instead of global average pooling, we employ a learnable aggregation to put the tokens into fullplay. The obtained results illustrate that our method effectively accelerates the convergence rate and promotes the performance in downstream task.

Interactions between proteins and ions are essential for various biological functions like structural stability, metabolism, and signal transport. Given that more than half of all proteins bind to ions, it becomes crucial to identify ion-binding sites. Accurate identification of protein-ion binding sites helps us to understand proteins’ biological functions and plays a significant role in drug discovery. While several computational approaches have been proposed, this remains a challenging problem due to the small size and high versatility of metals and acid radicals. In this study, we propose IonPred, a sequence-based approach that employs ELECTRA (Efficiently Learning an Encoder that Classifies Token Replacements Accurately) to predict ion binding sites using only raw protein sequences. We successfully fine-tuned our pretrained model to predict the binding sites for nine metal ions (Zn2+, Cu2+, Fe2+, Fe3+, Ca2+, Mg2+, Mn2+, Na+, and K+) and four acid radical ion ligands (CO32−, SO42−, PO43−, NO2−). IonPred surpassed six current state-of-the-art tools by over 44.65% and 28.46% respectively in F1 score and MCC when compared on an independent test dataset. Our method is more computationally efficient than existing tools producing prediction results for a hundred sequences for a specific ion in under ten minutes.

Accurate prediction of crop production is essential in effectively managing agricultural countries' food security and economic resilience. This study evaluates the performance of statistical and machine learning-based methods for large-scale crop production forecasting. We predict the quarterly production of 325 crops (including fruits, vegetables, cereals, non-food, and industrial crops) across 83 provinces in the Philippines. Using a comprehensive dataset of 10,949 time series over 13 years, we demonstrate that a global forecasting approach using a state-of-the-art deep learning architecture, the transformer, significantly outperforms traditional local forecasting approaches built on statistical and baseline methods. By leveraging cross-series information, our proposed way is scalable and works well even with time series that are short, sparse, intermittent, or exhibit structural breaks/regime shifts. The results of this study further advance the field of applied forecasting in agricultural production and provide a practical and effective decision-support tool for policymakers that oversee the farm sector on a national scale.

Deep convolutional neural networks (CNNs) have been widely used in various medical imaging tasks. However, due to the intrinsic locality of convolution operation, CNNs generally cannot model long-range dependencies well, which are important for accurately identifying breast cancer from unregistered multi-view mammograms. This motivates us to leverage the architecture of Vision Transformers to capture long-range relationships of multiple mammograms from the same patient. For this purpose, we employ local Transformer blocks to separately learn patch relationships within a specific view (CC/MLO) of one side (right/left) of mammogram. The outputs from different views and sides are concatenated and fed into global Transformer blocks, to jointly learn patch relationships between two different views of the left and right breasts. To evaluate the proposed model, we retrospectively assembled a dataset involving 949 sets of CC and MLO view mammograms, which include 470 malignant cases and 479 normal or benign cases. We trained and evaluated the model using a five-fold cross-validation method. Without any arduous preprocessing steps (e.g., optimal window cropping, chest wall or pectoral muscle removal, two-view image registration, etc.), our two-view Transformer-based model achieves lesion classification accuracy of 77.0% and area under ROC curve (AUC = 0.814), outperforming state-of-the-art multi-view CNNs by 3.1% and 3%, respectively. Meanwhile, the new two-view model improves mammographic case classification accuracies of two single-view models by 7.4% (CC) and 4.5% (MLO), respectively. The promising results unveil the great potential of using Transformers to develop high-performing computer-aided diagnosis (CADx) schemes of mammography.