In this paper, we present a multimodal dataset for the analysis of classroom atmosphere, based on the behavior and voice of teachers in teaching scenarios. we propose four visual models, three audio models, and one visual-audio dual-modality model to be tested on our dataset. The results indicate that the CH-CC dataset is feasible and reliable and that the visual modality plays a major role in the analysis of this dataset.

Autonomous driving vehicles rely on sensors for the robust perception of surroundings. Such vehicles are equipped with multiple perceptive sensors with a high level of redundancy to ensure safety and reliability in any driving condition. However, multi-sensor, such as camera, LiDAR and radar, systems bring up the requirements related to sensor calibration and synchronization, which are the fundamental blocks of any autonomous system. On the other hand, sensor fusion and integration have become important aspects of autonomous driving research and directly determine the efficiency and accuracy of advanced functions such as object detection and path planning. Classical model-based estimation and data-driven models are two mainstream approaches to achieving such integration. Most recent research is shifting to the latter, showing high robustness in real-world applications but requiring large quantities of data to be collected, synchronized, and properly categorized. To generalize the implementation of the multi-sensor perceptive system, we introduce an end-to-end generic sensor dataset collection framework that includes both hardware deploying solutions and sensor fusion algorithms. The framework prototype integrates a diverse set of sensors, such as cameras, LiDAR, and radar. Furthermore, we present a universal toolbox to calibrate and synchronize three types of sensors based on their characteristics. The framework also includes the fusion algorithms, which utilize the merits of three sensors , namely, camera, LiDAR and radar, and fuse their sensory information in a manner that is helpful for object detection and tracking research. The generality of this framework makes it applicable in any robotic or autonomous applications, also suitable for quick and large-scale practical deployment.

Improper sitting positions are known as the primary reason for back pain and the emergence of musculoskeletal disorders (MSD) among individuals who spend prolonged time working with computer screens, keyboards, and mice. At the same time, it is well understood that automated technological tools can play an important role in the process of unhealthy habit alteration, so plenty of research efforts are focused on research and technology development (RTD) activities that aim to provide support to the prevention of back pain or the development of MSD. Here, we report on creating a new resource in support of RTD activities aiming at the automated detection of improper sitting positions. It consists of multimodal multisensory recordings of 100 persons made with a video recorder, camera, and wrist-attached sensors that capture physiological signals (PPG, EDA, Skin temperature), as well as motion sensors (three-axis accelerometer). Our multimodal multisensory dataset (MMD-MSD) opens new opportunities for modeling the body stance (sitting posture and movements), physiological state (stress level, attention, emotional arousal, valence), and performance (success rate on the Stroop test) of people working with a computer. Finally, we demonstrate two use cases: improper neck posture detection from pictures and cognitive load detection from physiological signals.

The advent of the era of big data will bring more convenience to people and greater development to society. But at the same time, it will also bring people the problem of 'information overload', i.e., when people are faced with huge information data, there are many redundant and worthless data. The redundant and worthless data information seriously interferes with the accurate selection of information data. Even though people can use Internet search engines to access information data, they cannot meet the individual needs of specific users in specific contexts. The personalized needs of a particular user in a particular context. Therefore, how to find useful and valuable information quickly has become one of the key issues in the development of big data. With the advent of the era of big data, recommendation systems, as an important technology to alleviate information overload, have been widely used in the field of e-commerce. Recommender systems suffer from a key problem: data sparsity. The sparsity of user history rating data causes insufficient training of collaborative filtering recommendation models, which leads to a significant decrease in the accuracy of recommendations. In fact, traditional recommendation systems tend to focus on scoring information and ignore the contextual context in which users interact. There are various contextual modal information in people's real life, which also plays an important role in the recommendation process. In this paper we achieve data degradation and feature extraction, solving the problem of sparse data in the recommendation process. An interaction context-aware sub-model is constructed based on a tensor decomposition model with interaction context information to model the specific influence of interaction context in the recommendation process. Then an attribute context-aware sub-model is constructed based on the matrix decomposition model and using attribute context information to model the influence of user attribute contexts and item attribute contexts on recommendations. In the process of building the model, the method not only utilizes the explicit feedback rating information of users in the original dataset, but also utilizes the interaction context and attribute context information of the implicit feedback and the unlabeled rating data. We evaluate our model by extensive experiments. The results illustrate the effectiveness of our recommender model.

Multimodal emotion classification (MEC) involves analyzing and identifying human emotions by integrating data from multiple sources, such as audio, video, and text. This approach leverages the complementary strengths of each modality to enhance the accuracy and robustness of emotion recognition systems. However, one significant challenge is effectively integrating these diverse data sources, each with unique characteristics and levels of noise. Additionally, the scarcity of large, annotated multimodal datasets in Bangla limits the training and evaluation of models. In this work, we unveiled a pioneering multimodal Bangla dataset, MAViT-Bangla (Multimodal Audio Video Text Bangla dataset). This dataset, comprising 1002 samples across audio, video, and text modalities, is a unique resource for emotion recognition studies in the Bangla language. It features emotional categories such as anger, fear, joy, and sadness, providing a comprehensive platform for research. Additionally, we developed a framework for emotion recognition that employs a cross-modal attention mechanism among unimodal features. This mechanism fosters the interaction and fusion of features from different modalities, enhancing the model’s ability to capture nuanced emotional cues. The effectiveness of this approach was demonstrated by achieving an F1-score of 0.64, a significant improvement over unimodal methods.

Traumatic brain injury (TBI) is a leading cause of death and disability worldwide. Despite extensive research efforts, the majority of trialed monotherapies to date have failed to demonstrate significant benefit. It has been suggested that this is due to the complex pathophysiology of TBI, which may possibly be addressed by a combination of therapeutic interventions. In this article, we have reviewed combinations of different pharmacologic treatments, combinations of non-pharmacologic interventions, and combined pharmacologic and non-pharmacologic interventions for TBI. Both preclinical and clinical studies have been included. While promising results have been found in animal models, clinical trials of combination therapies have not yet shown clear benefit. This may possibly be due to their application without consideration of the evolving pathophysiology of TBI. Improvements of this paradigm may come from novel interventions guided by multimodal neuromonitoring and multimodal imaging techniques, as well as the application of multi-targeted non-pharmacologic and endogenous therapies. There also needs to be a greater representation of female subjects in preclinical and clinical studies.

Many engineered approaches have been proposed over the years for solving the hard problem of performing indoor localisation using smartphone sensors. However, specialising these solutions for difficult edge cases remains challenging. Here we propose an end-to-end hybrid multimodal deep neural network localisation system, MM-Loc, relying on zero hand-engineered features, learning them automatically from data instead. This is achieved by using modality-specific neural networks to extract preliminary features from each sensing modality, which are then combined by cross-modality neural structures. We show that our choice of modality-specific neural architectures is capable of estimating the location with good accuracy independently. But for better accuracy, a multimodal neural network fusing the features of early modality-specific representations is a better proposition. Our proposed MM-Loc solution is tested on cross-modality samples characterised by different sampling rates and data representation (inertial sensors, magnetic and WiFi signals), outperforming traditional approaches for location estimation. MM-Loc elegantly trains directly from data unlike conventional indoor positioning systems, which rely on human intuition.

Generic text embeddings have demonstrated considerable success across a multitude of applications. However, these embeddings are typically derived by modeling the co-occurrence patterns within text-only corpora, which can limit their ability to generalize effectively across diverse contexts. In this study, we investigate methodologies that incorporate visual information into textual representations to overcome these limitations. Through extensive ablation studies, we introduce a novel and straightforward architecture named VisualText Fusion Network (VTFN). This architecture not only surpasses existing multimodal approaches on a range of well-established benchmark datasets but also achieves state-of-the-art performance on image-related textual datasets while utilizing significantly less training data. Our findings underscore the potential of integrating visual modalities to substantially enhance the robustness and applicability of text embeddings, paving the way for more nuanced and contextually rich semantic representations.

Multimodal knowledge graph completion necessitates the integration of information from multiple modalities (such as images and text) into the structural representation of entities to improve link prediction. However, most existing studies have overlooked the interaction between different modalities. To address this issue, this paper proposed a Transformer-based knowledge graph link prediction model (MM-Transformer) that fuses multimodal features. Different modal encoders are employed to extract structural, visual and textual features, and hybrid key-value calculations are performed on features from different modalities based on the Transformer architecture. The similarities of textual tags to structural tags and visual tags are calculated and aggregated respectively, and multimodal entity representations are modeled and optimized to reduce the heterogeneity of the representations. Experimental results demonstrate that, compared to the current multimodal state-of-the-art methods, the proposed method achieves significant performance improvements in knowledge graph link prediction tasks. This proves that the proposed method effectively addresses the problem of multimodal feature fusion in knowledge graph link prediction tasks.

The analysis of social media content plays a crucial role in uncovering intricate user behaviors and trends across diverse digital platforms. Social media is inherently multimodal, incorporating texts, images, audios, and videos, each offering unique insights into user engagement and preferences. Traditional classification methods often focus narrowly on the most prominent modality, typically neglecting the synergistic potential of integrating multiple data types. To address this gap, we introduce the Unified Multimodal Classifier (UMC), a suite of streamlined models that adeptly harness and integrate these varied modalities. UMC leverages a novel architecture that combines a pooling layer with auxiliary learning tasks, facilitating the formation of a robust and shared feature space. This integration allows UMC to not only accommodate but also capitalize on the inherent diversity of social media data. The models are designed to be inherently flexible, adjusting to the availability of data modalities and maintaining high classification accuracy under varied conditions. In emotion classification scenarios, UMC has shown exceptional performance, significantly outperforming traditional methods by effectively synthesizing information across modalities. Its robustness is highlighted in its ability to deliver consistent results, even in the absence of one or more modalities. The simplicity and efficiency of UMC make it a potent tool for social media analytics, enabling it to achieve higher accuracies and maintain effectiveness even with limited data inputs. This adaptive capability ensures that UMC can be deployed in real-world applications where data incompleteness is common, thus broadening the scope of its applicability in various analytical contexts.

In this work, we introduce the TriFusion Network, an innovative deep learning framework designed for the simultaneous analysis of auditory, visual, and textual data to accurately assess emotional states. The architecture of the TriFusion Network is uniquely structured, featuring both independent processing pathways for each modality and integrated layers that harness the combined strengths of these modalities to enhance emotion recognition capabilities. Our approach addresses the complexities inherent in multimodal data integration, with a focus on optimizing the interplay between modality-specific features and their joint representation. Extensive experimental evaluations on the challenging AVEC Sentiment Analysis in the Wild dataset highlight the TriFusion Network's robust performance. It significantly outperforms traditional models that rely on simple feature-level concatenation or complex score-level fusion techniques. Notably, the TriFusion Network achieves Concordance Correlation Coefficients (CCC) of 0.606, 0.534, and 0.170 for the arousal, valence, and liking dimensions respectively, demonstrating substantial improvements over existing methods. These results not only confirm the effectiveness of the TriFusion Network in capturing and interpreting complex emotional cues but also underscore its potential as a versatile tool in real-world applications where accurate emotion recognition is critical.

Abstract: In this comprehensive review, we delve into the significance of the mul-timodal imaging in diagnosing and managing of complications of congenital optic nerve anomalies.. While the Fundus examination is the gold standard tool in diagnosisc of these pathologies, SD OCT and OCTA could shed a light in the pathogenesis, and treatment. Moreover , this review seeks to offer a comprehensive insight into a multimodal approach of these rare congenital pathologies. In conclusion, congenital anomalies of the optic nerve represent a major challenge for ophthalmologists Further research could be clarify the pathophysiology of these diseases and define a correct and specific treatment approach. Keywords: keyword 1; ODC, ODP, MGS, OCT; OCTA, Multimodal imaging.

The study combines two critical aspects of climate research in Nigeria. The first segment focuses on improving predictive performance of global circulation models (GCMs) employing ensemble techniques. ANN ensemble displayed significant improvement in the accuracy of climate projections, particularly in high-altitude geographic areas where traditional models exhibit limitations. Performance evaluation metrics such as Mean Absolute Error (MAE), Root Mean Squared Error (RMSE), and Nash–Sutcliffe Efficiency (NSE) demonstrate the effectiveness of the ANN multimodal ensemble. Building on the improved climate modelling, the second segment delves into a spatial and temporal analysis of rainfall and mean temperature patterns under climate change in Nigeria. By considering two Representative Concentration Pathways (RCPs), namely RCP4.5 and RCP8.5, the study utilizes dynamically downscaled GCMs obtained from the Coupled Model Intercomparison Project phase 5 (CMIP5). The Mann-Kendall test reveals significant spatiotemporal variations in climate trends across the country. The warming trend is particularly pronounced in the northern regions, with RCP8.5 showing more severe impacts compared to RCP4.5. This integrated approach not only highlights the importance of multimodal ensembles in climate modelling but also provides valuable insights into the spatial and temporal patterns of climate change to inform effective adaptation and mitigation strategies for Nigeria.

In today’s Web 2.0 era, online social media has become an integral part of our lives. In the course of the information revolution, the form of information has undergone a radical change, from simple text information to today’s integrated video, image, text and audio, and there has also been a great change in the way of dissemination and access, as people nowadays do not just rely on traditional media to passively receive information, but more actively and selectively obtain information from social media. Therefore, it has become a great challenge for us to effectively utilize these massive and integrated multi-modal media information to form an effective system of retrieval, browsing, analysis and usage. Unlike movies and traditional long-form video content, micro-videos are usually short in length, between a few seconds and tens of seconds, which allows users to quickly browse different contents and make full use of the fragmented time in their lives, while users can also share their micro-videos to their friends or the public, forming a unique social way. Video contains rich multimodal information, and fusing information from multiple modalities in a video recommendation task can improve the accuracy of the video recommendation task.According to the micro-video recommendation task, a new combinatorial network model is proposed to combine the discrete features of each modality into the overall features of various modalities through the network, and then fuse the various modal features to obtain the overall video features, which will be used for recommendation. In order to verify the effectiveness of the algorithm proposed in this paper, experiments are conducted in the public dataset, and it is shown the effectiveness of our model.

Affordances are an important concept in cognition, which can be applied to robots in order to perform a successful human-robot interaction (HRI). In this paper we explore and discuss the idea of emotional affordances and propose a viable model for implementation into HRI. We consider “2-ways” affordances: perceived object triggering an emotion, and perceived human emotion expression triggering an action. In order to make the implementation generic, the proposed model includes a library that can be customised depending on the specific robot and application’s scenario. We present the AAA (Affordance-Appraisal-Arousal) model, which incorporates Plutchik’s Wheel of Emotions, and show some examples of simulation and possible scenarios.

Generic text embeddings have demonstrated considerable success across a multitude of applications. However, these embeddings are typically derived by modeling the co-occurrence patterns within text-only corpora, which can limit their ability to generalize effectively across diverse contexts. In this study, we investigate methodologies that incorporate visual information into textual representations to overcome these limitations. Through extensive ablation studies, we introduce a novel and straightforward architecture named VisualText Fusion Network (VTFN). This architecture not only surpasses existing multimodal approaches on a range of well-established benchmark datasets but also achieves state-of-the-art performance on image-related textual datasets while utilizing significantly less training data. Our findings underscore the potential of integrating visual modalities to substantially enhance the robustness and applicability of text embeddings, paving the way for more nuanced and contextually rich semantic representations.

With the development of human-computer interaction, the importance of emotion recognition is increasing. Emotion recognition technology provides practical benefits in various industries such as improving user experience, education, and organizational productivity. In education, emotion recognition can enable real-time understanding of students' emotional states and provide tailored feedback, and in the workplace, by monitoring employees' emotional states, it can improve work performance and job satisfaction. Therefore, multimodal-based research that combines text, speech, and video data for emotion recognition is being conducted in various industries. In this study, we propose an emotion recognition method that combines text and speech data, reflecting the characteristics of the Korean language. For text, KoELECTRA is used to perform embedding, and for speech, MFCC and pitch analysis are used to extract features. Finally, a multimodal transformer model is proposed that combines these two data types to perform emotion recognition. The multimodal transformer model processes text and speech data separately, and then learns the interaction between the two modalities through a Cross-Modal Attention mechanism. Through this, the complementary information from text and speech is effectively combined, improving emotion recognition performance. Experimental results show that the proposed model outperforms single-modality models, achieving a high accuracy of 73.13% and an F1-Score of 0.7344 in emotion classification. This study contributes to the advancement of emotion recognition technology by combining various language and modality data, and higher performance can be expected through the integration of additional modalities in the future.

(1) Background: With the global population aging, the demand for effective rehabilitation products for the elderly is increasing. Traditional products often fail to meet the comprehensive needs of elderly users, who require solutions addressing physical, psychological, and social aspects; (2) Methods: Using the Analytic Hierarchy Process, Quality Function Deployment, Axiomatic Design, and the System Usability Scale to identify and prioritize user needs, market research and user interviews were conducted to understand the preferences of the elderly and obtain design specifications. The focus was on multimodal sensory feedback, which was validated through the study; (3) Results: Elderly users have a strong preference for rehabilitation products that are durable, easy to use and have comprehensive sensory feedback. Intelligent rehabilitation bed as the carrier, strengthened structure, high strength material, ergonomic adjustment, integrated health monitoring system. Using digital technology for real-time health monitoring and personalized feedback to provide a comprehensive recovery experience, high user satisfaction was reported in the evaluation; (4) Conclusion: It is important to emphasize user-centered design and integrate multi-modal sensory feedback to meet the comprehensive rehabilitation needs of elderly users. Future studies should expand user samples and expert opinions to refine and validate the design and improve its effectiveness and usability.

Federated learning (FL) has become a burgeoning and attractive research area, which provides a collaborative training scheme for distributed data sources with privacy concerns. Most existing FL studies focus on taking unimodal data, such as images and text, as the model input and resolving the heterogeneity challenge, i.e., the non-identically distributed (non-IID) challenge caused by data distribution imbalance related to data labels and data amount. In real-world applications, data are usually described by multiple modalities. However, to the best of our knowledge, only a handful of studies have been proposed to improve the system performance by utilizing multimodal data. In this survey paper, we identify the significance of this emerging research topic – multimodal federated learning (MFL) and perform a literature review on the state-of-art MFL methods. Furthermore, we categorize multi-modal federated learning into congruent and incongruent multimodal federated learning based on whether all clients possess the same modal combinations. We investigate the feasible application tasks and related benchmarks for MFL. Lastly, we summarize the promising directions and fundamental challenges in this field for future research.

Effective healthcare relies on accurate and timely diagnosis; however, obtaining large amounts of training data while maintaining patient privacy remains challenging. This study introduces a novel approach utilizing federated learning (FL) and a cross-device multi-modal model for clin-ical event classification using vital signs data. Our architecture leverages FL to train machine learning models, including Random Forest, AdaBoost, and SGD ensemble model, on vital signs data from a diverse clientele at a Boston hospital (MIMIC-IV dataset). The FL structure preserves patient privacy by training directly on each client's device without transferring sensitive data. The study demonstrates the potential of FL in privacy-preserving clinical event classification, achieving an impressive accuracy of 98.9%. These findings underscore the significance of FL and cross-device ensemble technology in healthcare applications, enabling the analysis of large amounts of sensitive patient data while safeguarding privacy.

Accurate evaluation of patients with disorders of consciousness (DoC) is crucial for personalized treatment. However, misdiagnosis remains a serious issue. Neuroimaging methods could observe the conscious activity in patients who have no evidence of consciousness in behavior, and provide objective and quantitative indexes to assist doctors in their diagnosis. In the review, we discussed the current research based on the evaluation of consciousness rehabilitation after DoC using EEG, fMRI, PET, and fNIRS, as well as the advantages and limitations of each method. Nowadays single-modal neuroimaging can no longer meet the researchers` demand. Considering both spatial and temporal resolution, recent studies have attempted to focus on the multi-modal method which can enhance the capability of neuroimaging methods in the evaluation of DoC. As neuroimaging devices become wireless, integrated, and portable, multi-modal neuroimaging methods will drive new advancements in brain science research.

Hateful and abusive speech presents a major challenge for all online social media platforms. Recent advances in Natural Language Processing and Natural Language Understanding allow more accurate detection of hate speech in textual streams. This study presents a multimodal approach to hate speech detection by combining Computer Vision and Natural Language processing models for abusive context detection. Our study focuses on Twitter messages and, more specifically, on hateful, xenophobic and racist speech in Greek aimed at refugees and migrants. In our approach we combine transfer learning and fine-tuning of Bidirectional Encoder Representations from Transformers (BERT) and Residual Neural Networks (Resnet). Our contribution includes the development of a new dataset for hate speech classification, consisting of tweet ids, along with the code to obtain their visual appearance, as they would have been rendered in a web browser. We have also released a pre-trained Language Model trained on Greek tweets, which has been used in our experiments. We report a consistently high level of accuracy (accuracy score=0.970, f1-score=0.947 in our best model) in racist and xenophobic speech detection.

Brain-Computer Interfaces (BCIs) are essential in advancing medical diagnosis and treatment by providing non-invasive tools to assess neurological states. Among these, Motor Imagery (MI), where patients mentally simulate motor tasks without physical movement, has proven to be an effective paradigm for diagnosing and monitoring neurological conditions. Electroencephalography (EEG) is widely used for MI data collection due to its high temporal resolution, cost-effectiveness, and portability. However, EEG signals can be noisy from a number of sources, including physiological artifacts and electromagnetic interference. They can also vary from person to person, which makes it harder to extract features and understand the signals. Additionally, this variability, influenced by genetic and cognitive factors, presents challenges for developing subject-independent solutions. To address these limitations, this paper presents a Multimodal and Explainable Deep Learning (MEDL) approach for MI-EEG classification and physiological interpretability. Our approach involves: i) evaluating different deep learning (DL) models for subject-dependent MI-EEG discrimination; ii) employing Class Activation Mapping (CAM) to visualize relevant MI-EEG features; and iii) utilizing a Questionnaire-MI Performance Canonical Correlation Analysis (QMIP-CCA) to provide multidomain interpretability. On the GIGAScience MI dataset, experiments show that shallow neural networks are good at classifying MI-EEG data, while the CAM-based method finds spatio-frequency patterns. Moreover, the QMIP-CCA framework successfully correlates physiological data with MI-EEG performance, offering an enhanced, interpretable solution for BCIs.

Introduction Sepsis-associated encephalopathy (SAE) is common in sepsis and is linked to high mortality and impaired neurologic outcome. Brain ultrasonography (US), including transcranial Doppler (TCD) and transcranial color-coded duplex sonography (TCCS), is a non-invasive tool for cerebral monitoring and might be useful in sepsis. The aims of this review are: 1) to explore whether brain perfusion is altered in sepsis, 2) to determine the role of brain US in guiding resuscitation, and 3) its ability to predict neurological outcomes in septic patients. Materials and methods This scoping review adhered to the PRISMA-ScR protocol and JBI methodology. A comprehensive literature search was conducted across MEDLINE, Scopus, and Web of Science. The review included studies on adults with sepsis or septic shock in the ICU. Randomised controlled trials, clinical trials, observational studies and systematic reviews were included. All the data related to the review's key questions were extracted and charted independently by two authors. Results 625 articles were screened, and 34 were ultimately included after application of exclusion criteria. 85% of the studies were observational 15% were systematic reviews with or without meta-analysis. The studies primarily focused on cerebral blood flow (CBF) alterations in sepsis, reporting variable results. Findings showed variable cerebral autoregulation (CAR) impairment, with higher incidence in the early stages of sepsis and associations with poor neurological outcomes. However, while alterations in CAR and CBF are common in septic patients, their impact on neurological outcomes and mortality is not clear. Regarding resuscitation, only a paucity of studies were found and no conclusions can be drawn. Conclusions The current body of evidence has a poor grade, but it is possible to say that brain US can identify cerebral perfusions alterations. Its usage during resuscitation is feasible and warrants further investigations. Use of brain US during resuscitation and for prediction of outcome is not currently supported by evidence. Further studies on brain perfusion, acquisition of 2D images of parenchyma (TCCS) and application of a multimodal brain monitoring might be useful in these contexts.

The rapidly evolving domain of multimodal sentiment analysis is crucial for unraveling the intricate layers of emotional expression in social media content, customer service interactions, and personal vlogs. This research introduces a cutting-edge Advanced Cross-Modal Gating (ACMG) framework that significantly enhances the precision of sentiment analysis by refining the interplay among textual, auditory, and visual modalities. Our approach addresses three foundational aspects of sentiment analysis: (1) Advanced learning of cross-modal interactions, which focuses on extracting and synthesizing sentiment from varied modal inputs, thus providing a holistic view of expressed emotions; (2) Mastery over the temporal dynamics of multimodal data, enabling the model to maintain context and sentiment continuity over extended interactions; and (3) Deployment of a novel fusion strategy that not only integrates unimodal and cross-modal cues but also dynamically adjusts the influence of each modality based on its contextual relevance to the sentiment being expressed. The exploration of these dimensions reveals that the nuanced modeling of cross-modal interactions is crucial for enhancing model responsiveness and accuracy. By applying the ACMG model to two highly regarded datasets—CMU Multimodal Opinion Level Sentiment Intensity (CMU-MOSI) and CMU Multimodal Opinion Sentiment and Emotion Intensity (CMU-MOSEI)—we achieve groundbreaking accuracies of 83.9\% and 81.1\%, respectively. These results represent significant improvements of 1.6\% and 1.34\% over the current state-of-the-art, showcasing the superior performance and potential of our approach in navigating the complexities of multimodal sentiment analysis.

AI advances integrate generative design tools in architecture, providing architects with sophisticated design options. It enables the creation of intricate, high-performing projects by exploring diverse design possibilities with AI and algorithms. Generative AI and generative design empower architects to create better-performing, sustainable, and efficient design solutions and explore diverse design possibilities. This paper leverages multimodal generative AI to enhance design creativity by combining textual and visual inputs. Blockchain technology converts design metadata into NFTs, ensuring secure, authentic, and traceable data storage. The framework addresses data ownership, legal adherence, and client-architect collaboration and is entirely scalable for digital design authentication. This research exemplifies the pragmatic fusion of Generative AI and blockchain technology applied in architectural design for more transparent, secure, and effective results. This study provides a strategy that uses generative AI technologies to achieve an efficient and creative workflow in the early stages of architectural design.

In response to the issues of sensor noise and accumulated motion errors that lead to low positioning accuracy and insufficient robustness during large maneuvers or continuous flight of small unmanned aerial vehicles (UAVs) in complex environments, this paper proposes a multi-source and multimodal data fusion method. Initially, it employs a multimodal data fusion of various sensors including GPS (Global Positioning System), IMU (Inertial Measurement Unit), and visual sensors to complement the strengths and weaknesses of each hardware component, thereby eliminating motion errors to enhance accuracy. To mitigate the impact of sudden changes in sensor data, the method introduces a digital twin UAV data source. By utilizing the Extended Kalman Filter algorithm, it fuses data from both the real UAV and its digital twin, and the filtered positional information is fed back into the control system of the real UAV. This enables real-time correction of UAV positional deviations caused by sensor noise and environmental disturbances. The multi-source multimodal fusion Kalman filter method proposed in this paper significantly improves the positioning accuracy of UAVs in complex scenarios and the overall stability of the system. This method holds significant value in maintaining high-precision positioning in variable environments and has important practical implications for enhancing UAV navigation and application efficiency.

This study aims to enhance multimodal transportation systems by mitigating CO2 emissions and enhancing operational efficiency. It introduces a novel centralized cargo concentration approach tailored for regions facing geographical challenges. The method proposes for direct transportation of cargo from its loading origin to export ports, bypassing intermediate hubs. The mathematical model determines the most efficient means of transport for each route, factoring in variables like distance, volume, and cargo type. Research results indicate that, in scenarios with a high concentration of cargo, multiple hubs may not be necessary, which could streamline transportation and logistics operations. Modal preferences vary based on regional dynamics and cargo characteristics, with rail and sea transport emerging as preferable options under specific circumstances, surpassing the efficacy of road transport. The proposed model showcases reductions in both logistics costs and CO2 emissions compared to road-centric scenarios. It underscores the strategic integration of diverse transportation modes as pivotal for enhancing efficiency and sustainability. This study furnishes a framework adaptable to optimizing multimodal transportation systems in regions sharing similar geographical and logistical attributes.

Microbial foodborne pathogens can cause major risks to public health and the food industry. Rapid and accurate detection of these pathogens is crucial to prevent outbreaks and ensure food safety. Traditional single-mode biosensing approaches have their limitations in terms of specificity, sensitivity, and speed. Over the years, there has been an increasing research interest in developing multimodal biosensing strategies that integrate multiple sensing techniques to achieve enhanced accuracy, efficacy, and precision in foodborne pathogen detection. This article review paper explores the current state of multimodal biosensing technologies and their promising applications in the food industry. We discuss various integrated biosensing platforms, including optical, electrochemical, and nanomaterial-based systems, among others. The advantages, challenges, and future prospects of multimodal biosensing for foodborne pathogens are thoroughly examined, highlighting the potential impact of these advanced technologies on food safety and public health.

This paper proposes a multimodal fusion 3D target detection algorithm based on the attention mechanism to improve the performance of 3D target detection. The algorithm utilizes point cloud data and information from camera. For image feature extraction, the ResNet50+FPN architecture extracts features at four levels. Point cloud feature extraction employs the voxel method and FCN to extract point and voxel features. The fusion of image and point cloud features is achieved through regional point fusion and regional voxel fusion methods. After information fusion, the Coordinate attention mechanism and SimAM attention mechanism extract fusion features at a deep level. The algorithm's performance is evaluated using the DAIR-V2X dataset. The results show that compared to the Part-A2 algorithm, the proposed algorithm improves the mAP value by 7.9% in BEV view and 7.8% in 3D view at IOU=0.5 (cars) and IOU=0.25 (pedestrians and cyclist). At IOU=0.7 (cars) and IOU=0.5 (pedestrians and cyclist), the mAP value of the SECOND algorithm is improved by 5.4% in the BEV view and 4.3% in the 3D view, compared to other comparison algorithms.

Multimodal traveling is expected to enhance mobility for users, reduce inequalities of car ownership and reduce emissions. In the same context, ride-sharing aims to minimize negative impacts related to emissions, reduce travelling costs and congestion, and increase passenger vehicle occupancy, and public transit ridership when planned for first/last mile trips. The goal of this study is to present the planning of a multimodal pilot demonstration and the challenges that emerged during and after its implementation in Athens, Greece. The demo aims to enhance the connection of low-density Attica Region areas to public transport (PT) modes, and specifically to the metro, through the provision of demand responsive ride-sharing services. During the demo period two different applications were utilized: the “Travel Companion” app and the “Driver Companion” app, which refer to passengers and drivers of the ride-sharing service. Identification of demo participants is performed through a Stated Preference (SP) experiment. Results of the demo implementation, as well as challenges that were faced show that although participants are willing to try new mobility solutions, the readiness and reliability of the new service are essential attributes towards maintaining existing users and engaging new ones.

With the prevalence and evolution of social media platforms, social media data have emerged as a crucial source for obtaining spatio-temporal information about various urban events. Providing accurate spatio-temporal information of these events enhances the capacities of urban management and emergency response. However, existing research mostly focuses on the textual content while mining this spatio-temporal information, often neglecting data from other modalities like images and videos.To address this, our study introduces a novel method for extracting spatio-temporal information from multi-modal social media data (MIST-SMMD), serving as a valuable supplement to current urban event monitoring methods. Leveraging deep learning and Geographic Information System (GIS) technologies, we extract spatio-temporal information from large-scale, multi-modal Weibo data about urban waterlogging events at both coarse and fine granularities.Through an in-depth experimental evaluation of the “July 20 Zhengzhou extreme rainstorm” event, the results show that in coarse-grained spatial information extraction solely using textual data, our method achieves a Spatial Precision of 87.54% within a 60m range and 100% Spatial Precision within a 201m range. In the fine-grained spatial information extraction, by incorporating other modalities such as images and videos, the Spatial Error saw a significant improvement, with MAESE increasing by 95.53% and RMSESE by 93.62%.These outcomes illustrate the capability of the MIST-SMMD method in extracting spatio-temporal information of urban events at both coarse and fine granularities. They also confirm the notable advantage of multi-modal data in enhancing the accuracy of spatial information extraction.

With the rise of piano teaching in recent years, many people have joined the team of piano learners. However, the expensive cost of manual instruction and the unique one-on-one teaching model have made piano learning an extravagant event. Most existing approaches based on the audio modality aim to evaluate piano players' skills. Unfortunately, these methods ignored the information contained in the video, which led to a one-sided and simplistic evaluation of the piano player's skills. More recently, multimodal-based methods are proposed to assess the skill level of piano players using both video and audio information. However, existing multimodal approaches use shallow networks to extract video and audio features, which are deficient in extracting complex spatio-temporal and time-frequency features from piano performance. Furthermore, the fingering and the pitch-rhythm information of the piano performance is contained in the spatio-temporal and time-frequency features, respectively. In the paper, we propose a ResNet-based audio-visual fusion model that combines video and audio features to assess the skill level of piano players. Firstly, ResNet18-3D is used as the backbone network for our visual branches, which can extract feature information from the video data. Then, we consider ResNet18-2D as the backbone network of the aural branch and extract the feature information from the audio data. The extracted video features are fused with the audio features to generate multimodal features for the final piano skill evaluation. The experimental results on the PISA dataset show that our proposed audio-visual fusion model, with a validation accuracy of 70.80%, outperforms the state-of-the-art methods in both performance and efficiency. Then, we also explore the impact of different layers of ResNet on model performance, and the experimental results show that the audio-visual fusion model dealing with the piano skill assessment problem can make full use of both feature information when the number of video features is close to the number of audio features.

Digital storytelling is a powerful tool for language learning that has gained popularity in recent years. By incorporating different modes of communication such as text, images, audio, and video, digital storytelling provides learners with engaging and interactive experiences that promote language acquisition and cultural awareness. Digital storytelling can also promote interaction among learners, which is crucial for language learning, and provide learners with opportunities to receive feedback and support from their peers. Additionally, digital storytelling can be used to promote cultural sensitivity and intercultural dialogue among language learners. Through storytelling, learners can gain insights into the customs, values, and beliefs of different communities, and create stories that reflect their own cultural backgrounds and experiences. This paper provides an overview of the benefits of digital storytelling for language learning and highlights the various ways in which it can be used to enhance language acquisition and cultural awareness. It concludes with recommendations on how to effectively incorporate digital storytelling into language teaching practices.

The growing use of sensor tools and the Internet of Things requires sensors to understand the applications. There are major difficulties in realistic situations, though, that can impact the efficiency of the recognition system. Recently, as the utility of deep learning in many fields has been shown, various deep approaches were researched to tackle the challenges of detection and recognition. We present in this review a sample of specialized deep learning approaches for the identification of sensor-based human behaviour. Next, we present the multi-modal sensory data and include information for the public databases which can be used in different challenge tasks for study. A new taxonomy is then suggested, to organize deep approaches according to challenges. Deep problems and approaches connected to problems are summarized and evaluated to provide an analysis of the ongoing advancement in science. By the conclusion of this research, we are answering unanswered issues and providing perspectives into the future.

Predicting patient readmission is an important task for healthcare risk management, as it can help prevent adverse events, reduce costs, and improve patient outcomes. In this paper, we compare various conventional machine learning models on a multimodal dataset of electronic discharge records from an Irish acute hospital. We \khoi{evaluate the effectiveness of several widely-used Machine Learning models} that leverage patient demographics, historical hospitalization records, and clinical diagnoses codes, to forecast future clinical risks. \khoi{Our work focuses on addressing two key challenges in the medical fields: data imbalance and the variety of data types in order to boost the performance of Machine Learning algorithms. Through extensive benchmarking and the application of a variety of feature engineering techniques, we successfully improved the Area Under the Curve (AUROC) score from 0.628 to 0.7 across our models on the test dataset}. Furthermore, we also employ \khoiz{Shapley Additive Explanations (SHAP)} value visualization to\reviewertwo{v In the abstract line 6 SHAP is mentioned in initials should be written in details: SHAP (SHapley Additive exPlanations).} interpret the model predictions and identify both the key data features and disease codes associated to readmission risks, identifying a specific set of diagnoses codes that are significant predictors of readmission within 30 days. Our study demonstrates how we effectively utilize the routinely collected hospital data to forecast patient readmission through the use of conventional machine learning while applying explainable AI techniques to explore the correlation between data features and patient readmission rate.

A novel approach to enhance Additive Friction Stir Deposition (AFSD) is presented through the integration of Multimodal Retrieval Augmented Generation (RAG). AFSD which is a solid-state metal additive manufacturing process characterized by its complexity, necessitating advanced analytical tools. A multimodal RAG system is implemented, integrating textual, and visual data. The methodology involves text extraction using LangChain's PyPDFLoader, followed by chunking and embedding generation via Google's Generative AI model. ChromaDB is utilized for vector storage and efficient information retrieval. The system, powered by Google's Gemini large language model, demonstrates proficiency in generating detailed explanations of friction-based deposition processes, identifying manufacturing techniques from visual data, and evaluating material microstructures. Information from multiple sources is synthesized to produce context-aware responses. Process control, quality assurance, and decision-making in AFSD are significantly enhanced by this approach. The system's capabilities are illustrated through the analysis of various friction-based deposition techniques and microstructure evaluation.

In this study, we introduce an innovative framework for multimodal learning that leverages enhanced fusion gate units within gated neural network architectures. The proposed Fusion Gate Unit (FGU) serves as a pivotal component in neural network designs, aiming to derive a comprehensive intermediate representation by amalgamating data from diverse modalities. The FGU is adept at determining the extent to which each modality influences the unit's activation through the utilization of multiplicative gating mechanisms. We conducted evaluations on a multilabel genre classification task for movies, utilizing both plot summaries and poster images as input modalities. The results demonstrate that the FGU significantly elevates the macro F-score compared to single-modality approaches and surpasses existing fusion techniques, including mixture of experts models. Additionally, we present the MM-IMDb dataset alongside this publication, which, to our knowledge, represents the most extensive publicly accessible multimodal dataset for movie genre prediction to date. This dataset is expected to facilitate further research and development in the field of multimodal information processing.

Background. Epithelial ovarian cancer (EOC) is the deadliest gynecological malignancy worldwide. Brain metastasis (BM) is quite an uncommon presentation. However, the likelihood of central nervous system (CNS) metastasization should be considered in the context of disseminated disease. The therapeutic management of BMs is an unmet clinical need, to date. Case presentation. We identified across different Cancer Centers six cases of both BRCA wild-type and BRCA-mutated EOC spreading to the CNS. They presented either with a single brain lesion or with multiple lesions and most of them had intracranial-only disease. All cases received Poly-ADP ribose polymerase inhibitor (PARPi) maintenance, as by clinical practice, for a long time within a multimodal treatment approach. We also provide an insight into the available body of work regarding the management of this intriguing disease setting, with a glimpse of future therapeutic challenges. Conclusion. Despite the lack of unanimous guidelines, multimodal care pathways should be encouraged for optimal disease control of this unfortunate patient’s subset. Albeit not being directly investigated in BM patients, the PARPi maintenance is deemed to have a valuable role in this setting. Prospective research, aimed to implement worthwhile strategies in the multimodal patient journey of BMs from EOC, is eagerly awaited.

Multimodal remote sensing data classification can enhance the model’s ability to distinguish land features through multimodal data fusion. In this context, how to help models understand the relationship between multimodal data and target tasks is the focus of researchers. Inspired by human feedback learning mechanisms, causal reasoning mechanisms, and knowledge induction mechanisms, this paper integrates causal learning, reinforcement learning, and meta learning into a unified remote sensing data classification framework and proposed the causal meta-reinforcement learning (CMRL). First, based on feedback learning mechanisms, we have overcome the limitations of traditional implicit optimization of fusion features and customized a reinforcement learning environment for multimodal remote sensing data classification tasks. Through feedback interactive learning between agents and the environment, we help them understand the complex relationships between multimodal data and labels, thereby achieving full mining of multimodal complementary information. Second, based on the causal inference mechanism we designed causal distribution prediction action, classification reward, and causal intervention reward, capturing pure causal factors in multimodal data and cutting off false statistical associations between non-causal factors and class labels. Finally, based on the knowledge induction mechanism, we designed a bi-layer optimization mechanism based on meta-learning. By constructing a meta training task and meta validation task simulation model in the generalization scenario of unseen data, we helped the model induce cross-task shared knowledge, thereby improving its generalization ability for unseen multimodal data. The experimental results on multiple sets of multimodal datasets show that the proposed method achieved state-of-the-art performance in multimodal remote sensing data classification tasks.

Background: The use of wearable devices has burgeoned over the past decade, including wearables for infants and young children. Typically, such devices assess a single modality, and few have undergone scientific validation. To address this gap, we developed an infant wearable platform, LittleBeats™, that integrates electrocardiogram (ECG), motion, and audio sensors on a single printed circuit board to permit daylong remote assessments of infants and caregivers in home environments. Objective: Complementing our prior reports focused on LittleBeats™ data collected among infants and children in the home context, our main objective here is to present a technical validation of each sensor modality against established laboratory protocols and gold-standard equipment. Materials and Methods: We conducted five studies, including assessments of (a) interbeat interval (IBI) data obtained from the LittleBeats™ ECG sensor versus the gold-standard BIOPAC among adults (Study 1, N=16) and infants (Study 2, N=5), (b) performance of automated activity recognition (upright vs. walk vs. glide vs. squat ) among adults using accelerometer data obtained from LittleBeats™ versus Google Pixel 1 smartphone (Study 3, N=12), and (c) performance of speech emotion recognition (SER; Study 4, N=8) and automatic speech recognition (ASR; Study 5, N =12) algorithms among adult samples using audio data from LittleBeats™ versus smartphone. Results: Results for IBI data obtained from the LittleBeats™ ECG sensor indicate acceptable mean absolute percent error (MAPE) rates for both the adults (MAPE = 5.29% to 5.97%; Study 1) and infants (MAPE =0.96% to 1.66%; Study 2) across low- and high-challenge sessions, as well as expected patterns of change in respiratory sinus arrythmia (RSA) across sessions. For activity recognition (Study 3), LittleBeats™ data showed good to excellent performance (Accuracy = 89%, F1-score=88%; Cohen’s kappa =.79), although the smartphone outperformed LittleBeats™ by less than 4%. Finally, performance on the SER applied to LittleBeats™ versus smartphone audio data (Study 4) indicated comparable performance, with a matched-pairs test indicating no significant difference in error rates (p=.26). On the ASR task (Study 5), the best performing algorithm yielded relatively low word error rates, although LittleBeats™ (4.16%) versus smartphone (2.73%) error rates are somewhat higher. Discussion: Taken together, results from these controlled laboratory studies conducted predominantly with adults indicate that the LittleBeats™ sensors yield data quality that is largely comparable to those obtained from gold-standard devices and protocols that have been used extensively in prior research. The advantages of the LittleBeats™ platform is the (a) integration of multiple sensors into one platform that was (b) designed specifically for use with infants and young children. Leveraging data from the multiple modalities and fine-tuning postprocessing steps to further increase data quality, combined with assessing data quality among infants and young children in their natural environments, will be key next steps in this research program.

Multimodal user interfaces promise a natural and intuitive human machine interactions. But is the extra effort for the development of a complex multi-sensor system justified, or can users also be satisfied with one input modality already? This study investigates interactions in an industrial weld inspection workstation. Three unimodal interfaces, including spatial interaction with buttons augmented on a workpiece or a worktable, and speech commands, were tested individually and in a multimodal combination. Within the unimodal conditions, users preferred the augmented worktable, but overall, the interindividual usage of all input technologies in the multimodal condition was ranked best. Our findings indicate that the implementation and use of multiple input modalities is valuable, and that it is difficult to predict the usability of individual input modalities for complex systems.

Improved oil recovery from tight carbonate formations may provide the world with a major source of lower-rate power over several decades. Here we provide an overview of the Arab D formation in the largest oil field on earth, the Ghawar. We investigate the occurrence of microporosity of different origins and sizes using scanning electron microscopy (SEM) and pore casting techniques. Then, we present a robust calculation of the probability of invasion and oil saturation distribution in the nested micropores using mercury injection capillary pressure data available in the literature. We show that large portions of the micropores in Arab D formation would have been bypassed during primary drainage unless the invading crude oil ganglia were sufficiently long. Considering the asphaltenic nature of oil in the Ghawar, we expect the invaded portions of the pores to turn mixed-wet, thus becoming inaccessible to waterflooding until further measures are taken to modify the system’s chemistry.

Effective pain management in laboratory animals is crucial for both animal welfare and the reliability of scientific research. We retrospectively examined the effects of carprofen as post-operative analgesia in Sprague Dawley rats following stereotactic surgery. Our data indicate that administering carprofen twice daily (5 mg/kg), as currently recommended by GV-SOLAS, led to adverse effects such as reduced food and water intake, disrupted fecal excretion, and abdominal bloating consistent with peritonitis. Continued administration exacerbated these symptoms, with post-mortem findings of intestinal obstructions and ulcers. However, when the frequency was reduced to once daily, such adverse symptoms were not observed. These results are based on incidental data collected from various neuroscientific experiments, resulting in small and uneven sample groups across various experimental cohorts. The inherent imbalances in these groups present challenges for statistical interpretation. While the findings suggest that less frequent carprofen use may reduce adverse effects, the surgical interventions and concurrent use of other drugs in these experiments likely exacerbated these outcomes. Further investigation into the interactions between carprofen, surgical stress, and other perioperative factors is needed to refine analgesia protocols in laboratory animals. Despite these limitations, these observations contribute to understanding analgesia protocols and may assist in improving animal welfare practices.

The collaboration of multimodal transport documents is the foundation for conducting multimodal transport operations. Blockchain technology can effectively address trust issues and information sharing difficulties in current multimodal transport document collaboration. However, in current blockchain-based electronic document research, the bottleneck problem lies in the collaboration of multiparty in multimodal transport under the "one-bill coverage system". We design a blockchain platform architecture for multimodal transport, reshaping collaboration processes for different modes of transport based on the blockchain platform. It proposes a collaboration model for multimodal transport under the "one-bill coverage system" and performs model solving and case analysis. The obtained collaboration strategies are embedded into the platform's application layer. Finally, using Solidity language, it writes smart contracts related to multimodal transport under the "one-bill coverage system " and deploys and executes them on the Remix platform.

A multimodal detection system with complementary capabilities for efficient detection was developed for impurity detection. The system consisted of a visible light camera, a multispectral camera, image correction and registration algorithms. It can obtain spectral feature and color feature at the same time, and has higher spatial resolution than a single spectral camera. This system was applied to detect impurities in Pu ’er tea to verify its high efficiency. The spectral and color features of each pixel in the images of Pu ’er tea were obtained by this system and used for pixel classification. The experimental results show that the accuracy of Support Vector Machine (SVM) model based on combined features is 93%, which is 7% higher than that based on only spectral features. By applying median filtering algorithm and contour detection algorithm to the label matrix extracted from pixel-classified images, 8 impurities except hair were detected successfully. Moreover, taking advantage of the high resolution of visible light camera, small impurities can be clearly imaged. By comparing the segmented color image with the pixel-classified image, small impurities such as hair could be detected successfully. Finally, it is proved that the system can obtain multiple images to allow a more detailed and comprehensive understanding of the detected items, and has excellent ability to detect small impurities.

Molecular analysis of growing teratoma syndrome has not been extensively studied. Here, we report a 14-year-old boy with a growing mass during treatment for a mixed germ cell tumor of the pineal region. Tumor markers were negative; thus, growing teratoma syndrome was suspected. A radical resection via the occipital transtentorial approach was performed, and histopathological examination revealed a teratoma with malignant features. Methylation classifier analysis confirmed the diagnosis of teratoma, and DMRT1 loss and 12p gain were identified by copy number variation analysis, potentially elucidating the cause of growth and malignant transformation of the teratoma. The patient remains in remission after intense chemoradiation treatment as a high-risk germ cell tumor.

The combination of non-steroidal anti-inflammatory drugs (NSAIDs) with non-opioid analgesics is common in clinical practice for the treatment of acute pain conditions like post-operative and post-traumatic pain. Besides that, and despite the very satisfactory results achieved by oral analgesics, parenteral analgesia is a key tool in the treatment of painful conditions when the enteral routes are not convenient. The parenteral ready-to-use fixed-dose combinations of NSAIDs with non-opioid analgesics like paracetamol or metamizole could play a central role in the treatment of painful conditions since they are able to gather into only one formulation the advantages of multimodal and parenteral analgesia. Surprisingly, only very recently a parenteral ready-to-use fixed-dose combination of Ibuprofen/Paracetamol was launched to the market. This review aims to review the current availability of NSAID-based Parenteral Fixed-dose combinations with paracetamol and metamizole in the European and American markets, and how the combination of such drugs could play a central role in a multimodal analgesia strategy. Also, it is explored how the parenteral formulations of NSAIDs, paracetamol, and metamizole could serve as starting elements for the development of new parenteral ready-to-use fixed-dose combinations. With this review, we concluded that, despite the well-recognized utility of combining NSAIDs with non-opioid analgesics, there are in the literature several randomized clinical trial studies unable to demonstrate clear advantages concerning their efficacy and safety. In the future, it is deemed necessary to perform clinical trials specifically designed to assess the efficacy and safety of these fixed-dose formulations to generate solid evidence of their clinical advantages.

Worldwide, one of the main causes of death among young adults is multiple trauma. In these pa-tients hemorrhagic shock represents the leading cause for worsening of the clinical status and for increased morbidity and mortality. This is due to a multifactorial complex involving cellular, bi-ological, and biophysical mechanisms. The most important mechanisms affecting clinical out-come are oxidative stress, the augmentation of pro-inflammatory status, immune deficiency, dis-ruptions in the coagulation cascade, imbalances in electrolyte and acid-base homeostasis. Poly-trauma patients in hemorrhagic shock need adequate fluid management to ensure hemodynamic stability that must consider not only the maintenance of adequate blood pressure, but also the ad-equate oxygenation of tissues for optimal cellular function. In the current clinical practice, fluid resuscitation in polytrauma patients uses a variety of widely studied pharmacological products, such as crystalloids, colloids, blood transfusions, and the infusion of other blood products. Alt-hough these products exist, an agreement was not reached on a standard administration protocol that could be generally applied for all patients. Moreover, numerous studies have reported a se-ries of adverse events related to fluid resuscitation and to the inadequate use of these products. This review aims at describing the impact the administration of all the solutions used in fluid re-suscitation might have on the cellular and pathophysiological mechanisms in the case of poly-trauma patients suffering from hemorrhagic shock.

: Multimodal Large Language Models (MLLMs) harness comprehensive knowledge spanning text, images, and audio to adeptly tackle complex problems, including zero-shot in-context learning scenarios. This study explores the ability of MLLMs in visually solving the Traveling Salesman Problem (TSP) and Multiple Traveling Salesman Problem (mTSP) using images that portray point distributions on a two-dimensional plane. We introduce a novel approach employing multiple specialized agents within the MLLM framework, each dedicated to optimizing solutions for these combinatorial challenges. Our experimental investigation includes rigorous evaluations across zero-shot settings and introduces innovative multi-agent zero-shot in-context scenarios. The re-sults demonstrated that both multi-agent models—Multi-Agent 1, which includes the Initializer, Critic, and Scorer agents, and Multi-Agent 2, which comprises only the Initializer and Critic agents—significantly improved solution quality for TSP and mTSP problems. Multi-Agent 1 ex-celled in environments requiring detailed route refinement and evaluation, providing a robust framework for sophisticated optimizations. In contrast, Multi-Agent 2, focusing on iterative re-finements by the Initializer and Critic, proved effective for rapid decision-making scenarios. These experiments yield promising outcomes, showcasing the robust visual reasoning capabilities of MLLMs in addressing diverse combinatorial problems. The findings underscore the potential of MLLMs as powerful tools in computational optimization, offering insights that could inspire further advancements in this promising field.

Gold nanoclusters (AuNCs) with fluorescence in the Near Infrared (NIR) by both one- and two-photon electronic excitation were incorporated in mesoporous silica nanoparticles (MSNs) using a novel one-pot synthesis procedure where the condensation polymerization of alkoxysilane monomers in the presence of the AuNCs and a surfactant produce hybrid MSNs of 49 nm diameter. This method was further developed to prepare 30 nm diameter nanocomposite particles with simultaneous NIR fluorescence and superparamagnetic properties, with a core composed of superparamagnetic manganese ferrite nanoparticles (MnFe2O4) coated with a thin silica layer, and a shell of mesoporous silica decorated with AuNCs. The nanocomposite particles feature NIR-photoluminescence with 0.6% quantum yield and large Stokes shift (290 nm), and superparamagnetic response at 300 K, with a saturation magnetization of 13.4 emu g-1. The conjugation of NIR photoluminescence and superparamagnetic properties in the biocompatible nanocomposite has high potential for application in multimodal bioimaging.

Previous studies to recognize negative emotions (e.g. disgust, fear, sadness) for mental health care have used heavy equipment directly attaching electroencephalogram (EEG) electrodes to the head, making it difficult to use in daily life, and they have proposed binary classification methods to determine whether negative emotion or not. To tackle this problem, we propose a negative emotion recognition system to collect multimodal biosignal data such as five EEG signals in an EEG headset and heart rate, galvanic skin response, and skin temperature in a smart band for classifying multiple negative emotions. It consists of android Internet of Things (IoT) application, an oneM2M-compliant IoT server, and a machine learning server. The android IoT application upload the biosignal data to the IoT server. By using the biosignal data stored in the IoT server, the machine learning server recognizes the negative emotions of disgust, fear, and sadness using a multi-class support vector machine (SVM) model with a radial basis function kernel (RBF). The experimental results showed that the multi-class SVM model achieved 93% accuracy when considering all the multimodal biosignal data. Moreover, when considering only data in the smart band, it could achieve 98% accuracy by optimizing the hyper-parameter of the RBF kernel.

This study was performed to synthesize multimodal radiopharmaceutical designed for the diagnosis and treatment of prostate cancer. To achieve this goal, Super Paramagnetic Iron Oxide (SPIO) nanoparticles were used as a platform for targeting molecule (PSMA-617) and for complexation of two scandium radionuclides, 44Sc for PET imaging and 47Sc for radionuclide therapy. TEM and XPS images showed that the Fe3O4 NPs have a uniform cubic shape and a diameter from 38 to 50 nm. The Fe3O4 core are surrounded by SiO2 and an organic layer. The saturation magnetization of the SPION core was 60 emu/g. However, coating the SPIONs with silica and polyglycerol reduces the magnetization significantly. The obtained bioconjugates were labeled with 44Sc and 47Sc, with a yield higher than 97%. The radiobioconjugate exhibited high affinity and cytotoxicity towards the human prostate cancer LNCaP (PSMA+) cell line, much higher than for PC-3 (PSMA-) cells. High cytotoxicity of the radiobioconjugate was confirmed by radiotoxicity studies on LNCaP 3D-spheroids. In addition, the magnetic properties of the radiobioconjugate should allow for its use in guide drug delivery driven by magnetic field gradient.

Pipeline integrity is an important area of concern for the oil and gas, refining, chemical, hydrogen, carbon sequestration, and electric-power industries, due to the safety risks associated with pipeline failures. Regular monitoring, inspection, and maintenance of these facilities is therefore required for safe operation. Large standoff magnetometry (LSM) is a non-intrusive, passive magnetometer-based mea- surement technology that has shown promise in detecting defects (anomalies) in regions of elevated mechanical stresses. However, analyzing the noisy multi-sensor LSM data to clearly identify regions of anomalies is a significant challenge. This is mainly due to the high frequency of the data collection, mis-alignment between consecutive inspections and sensors, as well as the number of sensor measurements recorded. In this paper we present LSM defect identification approach based on ma- chine learning (ML). We show that this ML approach is able to successfully detect anomalous readings using a series of methods with increasing model complexity and capacity. The methods start from unsupervised learning with "point" methods and eventually increase complexity to supervised learning with sequence methods and multi-output predictions. We observe data leakage issues for some methods with randomized train/test splitting and resolve them by specific non-randomized splitting of training and validation data. We also achieve a 200x acceleration of support-vector classifier (SVC) method by porting computations from CPU to GPU leveraging the cuML RAPIDS AI library. For sequence methods, we develop a customized Convolutional Neural Network (CNN) architecture based on 1D convolutional filters to identify and characterize multiple properties of these defects. In the end, we report the scalability of the best-performing methods and compare them, for viability in field trials.

Medical imaging is an active field of research that fosters the necessity for novel multimodal imaging probes. In this line, nanoparticle-based contrast agents are of special interest, since those can host functional entities either within their interior, reducing potential toxic effects of the imaging tracers, and on their surface, providing high payloads of probes, due to their large surface-to-volume ratio. The long-term stability of the particles in solution is an aspect usually under-tackled during probe design in research laboratories, since their performance is generally tested briefly after synthesis. This may jeopardize a later translation into practical medical devices, due to stability reasons. To dig into the effects of nanoparticle aging in solution, respect to their behavior in vivo, iron oxide stealth nanoparticles were used at two stages (3 weeks vs. 9 months in solution), analyzing their biodistribution in mice. Both sets of nanoprobes showed similar sizes, zeta potentials and morphology, as observed by DLS and TEM but, fresh nanoparticles accumulated in the kidneys after systemic administration, while aged ones accumulated in liver and spleen, confirming an enormous effect of particle aging on their in vivo behavior, despite barely noticeable changes perceived on a simple inspection of their structural integrity.

Preclinical cell tracking is enhanced with a multimodal imaging approach. Bioluminescence imaging (BLI) is a highly sensitive optical modality which relies on engineering cells to constitutively express a luciferase gene. Magnetic particle imaging (MPI) is a newer imaging modality which directly detects superparamagnetic iron oxide (SPIO) particles used to label cells. Here, we compare BLI and MPI for imaging cells in vitro and in vivo. Mouse 4T1 breast carcinoma cells were transduced to express firefly luciferase, labeled with SPIO (ProMag) and imaged as cell samples and after subcutaneous injection into mice. For cell samples, the BLI and MPI signals were strongly correlated with cell number. Both modalities presented limitations for imaging cells in vivo. For BLI, weak signal penetration, signal attenuation and scattering prevented detection of cells for mice with hair and for cells far from the tissue surface. For MPI, background signals obscured the detection of low cell numbers due to the limited dynamic range and cell number could not be accurately quantified from in vivo images. It is important to understand the shortcomings of these imaging modalities to develop strategies to improve cellular detection sensitivity.

Humans’ performance varies due to the mental resources that are available to successfully pursue a task. To monitor users’ current cognitive resources in naturalistic scenarios, it is essential to not only measure demands induced by the task itself but also consider situational and environmental influences. We conducted a multimodal study with 18 participants (nine female, M = 25.9 with SD = 3.8 years). In this study, we recorded, respiratory, ocular, cardiac, and brain activity using functional near-infrared spectroscopy (fNIRS) while participants performed an adapted version of the warship commander task with concurrent emotional speech distraction. We tested the feasibility of decoding the experienced mental effort with a multimodal machine learning architecture. The architecture comprised feature engineering, model optimisation, and model selection to combine multimodal measurements in a cross-subject classification. Our approach reduces possible overfitting and reliably distinguishes two different levels of mental effort. These findings contribute to the prediction of different states of mental effort and pave the way toward generalised state monitoring across individuals in realistic applications.

Currently, skin cancer is the most commonly diagnosed form of cancer in humans and is one of the leading causes of death in patients with cancer. Biopsy methods are an invasive research method and are not always available for primary diagnosis. Imaging methods have low accuracy and depend on the experience of the dermatologist. Artificial intelligence technologies can match and surpass visual analysis methods in accuracy, but they have the risk of a false negative response when a malignant pigmented lesion can be recognized as benign. One possible way to improve accuracy and reduce the risk of false negatives is to analyze heterogeneous data, combine different preprocessing methods, and use modified loss functions to eliminate the negative impact of unbalanced dermatological data. The paper proposes a multimodal neural network system with a modified cross-entropy loss function that is sensitive to unbalanced heterogeneous dermatological data. The accuracy of recognition in 10 diagnostically significant categories for the proposed system was 85.19%. The novelty of the proposed system lies in the use of cross-entropy loss when training the modified function with the help of weight coefficients. The introduction of weighting factors has reduced the number of false negative forecasts, as well as improved accuracy by 1.02-4.03 percentage points compared to the original multimodal systems. The introduction of the proposed multimodal system as an auxiliary diagnostic tool can reduce the consumption of financial and labor resources involved in the medical industry, as well as increase the chance of early detection of skin cancer.

As the use of AI in medical imaging has increased, so has the need to explain a model’s results. Segmentation models are one technique used to produce explainable results. Due to larger size and sophistication, segmentation models which operate on 2D data can often produce better results than models operating on 3D data. In the real world, imaging is combined with clinical factors for diagnosis. To replicate this, multimodal models are used which combine image and text modalities. I propose a multimodal arrangement that converts 3D scans to 2D and uses a 2D segmentation model (DeepLabV3) to analyze the images. This is combined with clinical biomarkers to achieve a complete confidence score. Using the Medical Segmentation Decathlon Lung (MSDL) dataset and the LUng CAncer Screening dataset (LUCAS), I achieved a testing Dice coefficient of 0.91 on segmentation with a receiver operating characteristic (ROC), average precision (AP), and F-Score of 0.89, 0.91, and 0.85 respectively on the final multimodal results. This approach raises the F-score from the previously achieved value of 0.508 to 0.85, creating a new baseline of what is achievable with multimodal lung cancer diagnostics and possible methods to achieve these accuracies.

Selective mutism (SM) is an “Anxiety Disorder” characterized by a child’s “persistent inability” to communicate verbally in some (or all) contexts of social life that is associated with other cognitive-affective disorders. Cognitive-behavioural assessment and psychological intervention may be complex to be performed due to both the difficulty to administered standardized neuropsychological batteries (e.g., language-mediated tests) and the difficulty to involve family and teachers in the intervention program, respectively. Here, a single-case study is described in which a female Filipina child with SM underwent a comprehensive neuropsychological assessment and multimodal therapeutical intervention between the ages of 7 to 11 years. More specifically, the psychological intervention included cognitive-behavioral psychotherapy to both improve social-cognitive skills and learning abilities and to reduce anxiety, and speech therapy. Moreover, the parents and teachers were actively involved in the therapeutic process and they were also administered a psycho-education program. After this specific treatment, at the age of 11 years the girl began to verbalize in both therapy and school contexts, although she still was used to adopt non-verbal strategies. A parallel and gradual improvement in communicative-linguistic skills and school learning was also observed. In conclusion, the present report highlights the importance to apply an "integrated and multimodal intervention" to treat SM in children that includes also psycho-education intervention for both parents and teachers.

Locoregional anaesthetic techniques are invaluable for providing multimodal analgesia for painful surgical procedures. This prospective, randomised study describes a nerve stimulator-guided bra-chial plexus blockade (BPB) in rabbits undergoing orthopaedic surgery in comparison to systemic lidocaine. Premedication was provided with intramuscular (IM) medetomidine, fentanyl and midazolam. Anaesthesia was induced (propofol IV) and maintained with isoflurane. Nine rabbits received a lidocaine BPB (2%; 0.3 ml kg-1), and eight received lidocaine constant rate infusion (CRI) (2 mg kg-1 IV, followed by 100 µg kg-1 minute-1). Rescue analgesia was provided with Fentanyl IV. Carprofen was administered at the end of surgery. Post-operative pain was determined using the Rabbit Grimace Scale (RGS) and a composite pain scale. Buprenorphine was administered accord-ing to pain score for two hours after extubation. Rabbits were filmed during the first two hours to measure distance travelled and behaviours. Food intake and faeces output were compared. Every rabbit in CRI required intraoperative rescue analgesia compared to no rabbits in BPB. How-ever, rabbits in both groups had similar pain scores and there was no difference in administration of postoperative analgesia. There were no significant differences in food intake or faeces produc-tion over 18 hours, and no significant differences in distance travelled, or behaviours examined during the first two hours. BPB seems superior for intraoperative analgesia. Postoperatively both groups were comparable.

Estimating Visual Quality of Street Space (VQoSS) is pivotal for urban design, environmental sustainability, civic engagement, etc. Recent advancements, notably in deep learning, have enabled large-scale analysis. However, traditional deep learning approaches are hampered by extensive data annotation requirements and limited adaptability across diverse VQoSS tasks. Multimodal Large Language Models (MLLMs) have recently demonstrated proficiency in various computer vision tasks, positioning them as promising tools for automated VQoSS assessment. In this paper, we pioneer the application of MLLMs to VQoSS change estimation, with our empirical findings affirming their effectiveness. In addition, we introduce Street Quality GPT (SQ-GPT), a model that distills knowledge from the current most powerful but inaccessible (not free) GPT-4, requiring no human efforts. SQ-GPT approaches GPT-4’s performance and is viable for large-scale VQoSS change estimation. In a case study of Nanjin, we showcase the practicality of SQ-GPT and knowledge distillation pipeline. Our work promises to be a valuable asset for future urban studies research.

Semi-autonomous vehicles (AVs) enable drivers to engage in non-driving tasks but require them to be ready to take control during critical situations. This "out-of-the-loop" problem demands a quick transition to active information processing, raising safety concerns and anxiety. Multimodal signals in AVs aim to deliver take-over requests and facilitate driver-vehicle cooperation. However, the effectiveness of auditory, visual, or combined signals in improving situational awareness and reaction time for safe maneuvering remains unclear. This study investigates how signal modalities affect driver’s behavior using virtual reality (VR). We measured drivers’ reaction times from signal onset to take-over response and gaze dwell time for situational awareness across twelve critical events. Furthermore, we assessed self-reported anxiety and trust levels using the Autonomous Vehicle Acceptance Model questionnaire. Results showed visual signals significantly reduced reaction times, whereas auditory signals did not. Additionally, any warning signal, together with seeing driving hazards, increased successful maneuvering. Analysis of gaze dwell time on driving hazards revealed that audio and visual signals improved situational awareness. Lastly, warning signals reduced anxiety and increased trust. These results highlight the distinct effectiveness of signal modalities in improving driver reaction times, situational awareness, and perceived safety, mitigating the “out-of-the-loop” problem and fostering human-vehicle cooperation.

With the development of general anesthesia techniques and anesthetic substances, brought new horizons for the expansion and improvement of surgical techniques. Nevertheless, more complex surgical procedures brought a higher complexity and longer duration for general anesthesia that led to a series of adverse events such as hemodynamic instability, under- or overdosage of anesthetic drugs, as well as an increased number of post-anesthetic events. In order to adapt the anesthesia according to the particularities of each patient, the multimodal monitoring of these patients is highly recommended. Classically, general anesthesia monitoring consists of the analysis of vital functions and gas exchange. Multimodal monitoring refers to the concomitant monitoring of the degree of hypnosis and the nociceptive-antinociceptive balance. By titrating anesthetic drugs according to these parameters, clinical benefits can be obtained, such as hemodynamic stabilization, reduction of awakening times, and the reduction of post-operative complications. Another important aspect is the impact on the status of inflammation and the redox balance. By minimizing inflammatory and oxidative impact one can achieve a faster recovery that will lead to both increased patient satisfaction and an increase in patient safety. The purpose of this literature review is to present the most modern multimodal monitoring techniques, respectively to discuss the particularities of each technique.

Timely and accurate information on tree species is crucial for the sustainable management of natural resources, forest inventory, biodiversity detection, and carbon stock calculation. The advancement of remote sensing technology and artificial intelligence has facilitated the acquisition and analysis of remote sensing data, resulting in more precise and effective classification of tree species. Multimodal remote sensing data and deep learning seem to be the current tree species classification research mainstream, whether or not. The current review on the remote sensing data and deep learning tree species classification methods perspectives to analyze the unimodal and multimodal remote sensing data and classification methods in this realm is missing. To bridge the gap, we search for major trends in the remote sensing data and tree species classification methods, provide a detailed overview of classic deep learning-based methods for tree species classification, and discuss the limitations.

Physicians often label anatomical structure sets in Digital Imaging and Communications in Medicine (DICOM) images with nonstandard names. As these names vary widely, the standardization of the nonstandard names in the Organs at Risk (OARs), Planning Target Volumes (PTVs), and 'Other' organs inside the area of interest is a vital problem. Prior works considered traditional machine learning approaches on structure sets with moderate success. This paper presents integrated deep learning methods applied to structure sets by integrating the multimodal data compiled from the radiotherapy centers administered by the US Veterans Health Administration (VHA) and the Department of Radiation Oncology at Virginia Commonwealth University (VCU). The de-identified radiation oncology data collected from VHA and VCU radiotherapy centers have 16,290 prostate structures. Our method integrates the heterogeneous (textual and imaging) multimodal data with Convolutional Neural Network (CNN)-based deep learning approaches like CNN, Visual Geometry Group (VGG) network, and Residual Network (ResNet). Our model presents improved results in prostate (RT) structure name standardization. Evaluation of our methods with macro-averaged F1 Score shows that our deep learning model with single-modal textual data usually performs better than the previous studies. We also experimented with various combinations of multimodal data (masked images, masked dose) besides textual data. The models perform well on the textual data alone, while the addition of imaging data shows that deep neural networks can achieve improved performance using information present in the other modalities. Additionally, using masked images and masked doses along with text leads to an overall performance improvement with the various CNN-based architectures than using all the modalities together. Undersampling the majority class leads to further performance enhancement. The VGG network on the masked image-dose data combined with CNNs on the text data performs the best and establishes the state-of-the-art in this domain.

Increased mitotic activity is associated with the genesis and aggressiveness of many cancers. To assess the clinical value of mitotic activity as prognostic biomarker, we performed a pan-cancer study on the mitotic network activity index (MNAI) constructed based on 54-gene mitotic apparatus network. Our pan-cancer assessment on TCGA (33 tumor types, 10,061 patients) and validation on other publicly available cohorts (23 tumor types, 9,209 patients) confirmed the significant association of MNAI with overall survival, progression-free survival and other prognostic endpoints in multiple cancer types, including Lower-Grade Gliomas (LGG), Breast Invasive Carcinoma (BRCA) and many others. We also showed significant association of MNAI with genetic instability, which provides a biological explanation of its prognostic impact at pan-cancer landscape. Futhermore, we found that patients with high MNAI benefit more from anti-PD-1 and Anti-CTLA-4 treatment. In addition, we demonstrated on LGG and BRCA that the multimodal integration of MNAI and the AI-empowerd Cellular Morphometric Subtypes (CMS) significantly improved the predictive power of prognosis compared to MNAI and CMS alone. Our results suggest that MNAI can be used as a potential prognostic biomarker for different tumor types toward different clinical endpoints, and multimodal integration of MNAI and CMS exceeds individual biomarker for precision prognosis.

Acute low back pain (LBP) stands as a leading cause of activity limitation and work absenteeism, and its associated healthcare expenditures are expected to become substantial when acute LBP develops into a chronic and even refractory condition. Therefore, early intervention is crucial to prevent progression to chronic pain whose management is particularly challenging and for which the most effective pharmacological therapy is still controversial. Current guideline treatment recommendations vary and are mostly driven by expertise with opinion differing across different interventions. Thus, it is difficult to formulate evidence-based guidance when relatively few randomized clinical trials did explore the diagnosis and management of LBP while employing different selection criteria, statistical analyses, and outcome measurements. This narrative review aims to provide a critical appraisal of current acute LBP management by discussing the unmet needs and areas of improvement from bench-to-bedside and proposes multimodal analgesia as the way forward to attain an effective and prolonged pain relief and functional recovery in patients with acute LBP.

In cardiac surgery, post-sternotomy pain is a significant issue, peaking within 48 hours and requiring proper analgesia for both acute relief and to avoid chronicization. Opioids are commonly used for pain management post-surgery but pose risks like adverse effects and dependency. Post-cardiac surgery pain can stem from various sources - somatic, visceral, and neuropathic, making opioid reliance a concern. Multimodal analgesia, combining different medications and regional anesthesia techniques, is increasingly recommended to decrease opioid use and its related problems. Strategies include acetaminophen, gabapentinoids, NMDA antagonists, alpha-2 agonists, intravenous lidocaine, anti-inflammatory drugs, and regional anesthesia. Implementing these approaches can enhance pain control, lessen opioid reliance, and improve cardiac surgery outcomes. The ERAS® Cardiac Society strongly advocates for an opioid-sparing multimodal approach to benefit patient recovery by reducing complications and enhancing satisfaction. This review aims to consolidate current evidence to assist healthcare providers in customizing pain management for patients post-cardiac surgery, emphasizing reduced opioid use and optimizing the recovery process.

Androgenetic alopecia, the most common cause of hair loss affecting both men and women, is typically treated using pharmaceutical options such as minoxidil and finasteride. While these medications work for many individuals, they are not suitable options for all. To date, the only non-pharmaceutical option that the United States Food and Drug Administration has cleared as a treatment for androgenetic alopecia is low-level laser therapy (LLLT). Various clinical trials utilizing LLLT devices of various types are available. However, a myriad of other physical treatments for this form of hair loss are reported in the literature. This review evaluated the effectiveness of microneedling, pulsed-electromagnetic field (PEMF) therapy, low-level laser therapy (LLLT), fractional laser therapy, and non-ablative laser therapy for the treatment of androgenetic alopecia (AGA). It also explores the potential of multimodal treatments combining these physical therapies. LLLT has the majority of evidence in the literature as a physical therapy for androgenetic alopecia. However, other physical treatments, such as non-ablative laser treatments, and multimodal approaches, such as PEMF-LLLT, seem to have the potential to be equally or more promising and merit further exploration.

The aim of the study is to evaluate the predictive factors of response to induction chemotherapy in patients with resectable NSCLC, treated at the unit of Thoracic Surgery of Siena University Hospital with radical surgery. From January 1, 2013 to December 31, 2020, 78 patients were recruited. We analyzed the outcomes in terms of 5-years OS based on the Estimated Regression Rate, N2 downstaging and age; two patients’ subgroups were created by age (Group A: age <66 years-old; Group B: age >66 years-old). No 5-year OS difference was observed based on age, while it was observed in patients with N2 downstaging (p=0.031). Bewtween patients with N2 downstaging, only patients in Group A had a significantly increased 5-year OS (p=0.019), while this was not observed in Group B (p=0-321); the same result was observed with the Estimated Regression Rate > 50% (Group A p=0.005; Group B p=0.391). The percentage of disease regression and the N2 down-staging after induction chemotherapy have great value on the survival, although this advantage seems to be observed mostly in younger patients. A multidisciplinary oncologic discussion of clinical cases could provide support in the careful selection of the ideal patients to undergo neoadjuvant treatment before radical surgery.

Document classification is one of the most critical steps in the document analysis pipeline. There are two types of approaches for document classification, known as image-based and multimodal approaches. The image-based document classification approaches are solely based on the inherent visual cues of the document images. In contrast, the multimodal approach co-learns the visual and textual features, and it has proved to be more effective. Nonetheless, these approaches require a huge amount of data. This paper presents a novel approach for document classification that works with a small amount of data and outperforms other approaches. The proposed approach incorporates a hierarchical attention network(HAN) for the textual stream and the EfficientNet-B0 for the image stream. The hierarchical attention network in the textual stream uses the dynamic word embedding through fine-tuned BERT. HAN incorporates both the word level and sentence level features. While the earlier approaches rely on training on a large corpus (RVL-CDIP), we show that our approach works with a small amount of data (Tobacco-3482). To this end, we trained the neural network at Tobacco-3428 from scratch. Thereby, we outperform state-of-the-art by obtaining an accuracy of 90.3%. This results in a relative error reduction rate of 7.9%.

There have been multiple technological advancements that promise to gradually enable devices to measure and record signals with high resolution and accuracy in the domain of Brain Computer Interfaces (BCI). Multi-modal BCIs have been able to gain significant traction given the potential to enhance signal processing by integrating different recording modalities. In this review, we explore the integration of multiple neuroimaging and neurophysiological modalities including Electroencephalography (EEG), Magnetoencephalography (MEG), Functional Magnetic Resonance Imaging (fMRI), Electrocorticography (ECoG), and Single-Unit Activity (SUA). This multimodal approach leverages the high temporal resolution of EEG and MEG with the spatial precision of fMRI, the invasive yet precise nature of ECoG, and the single-neuron specificity provided by SUA. The paper highlights the advantages of integrating multiple modalities, such as increased accuracy and reliability, and discusses the challenges and limitations of multimodal integration. Furthermore, we explain the data acquisition approaches for each of these modalities. We also demonstrate various software programs that help in extracting, cleaning, and refining the data. We conclude this paper with discussion on the available literature highlighting recent advances, challenges, and future directions for each of these modalities.

Perioperative acute pain management in pediatric patients is essential to reduce complications. Adenoidectomy-Tonsillectomy are surgical procedures requiring pain control, and risk minimization for postoperative bleeding, nausea, and vomiting. Despite their known secondary effects, the use of opioid analgesics remains preponderant in pediatric perioperative management. We present an updated review on perioperative pain management in children and describe the development and implementation of a multimodal analgesia protocol aimed to improve patients’ pain management while consistently reducing opioids use. Information from relevant articles was summarized, then compared to our clinical needs. Learnings were used to create and implement a multimodal analgesia protocol to be used in patients 3-9 years-old undergoing adenoidectomy/tonsillectomy. Analgesic strategies have emerged to reduce or avoid the use of opioids. Among these strategies, combining different non-opioid analgesics (Ibuprofen, Paracetamol, Metamizole) has been shown to be an effective and safe pharmacological strategy when implemented as part of postoperative multimodal analgesia protocols. Significant evidence associating the use of NSAIDs with a bigger risk of postoperative bleeding does not exist. Multimodal and preventive analgesia has shown to provide significantly more effective analgesia than some opioid regimens. Ibuprofen offers highly effective analgesia for postoperative pain, particularly when combined with acetaminophen.

Inflammation plays a key role in the induction of choroidal neovascularization (CNV). Inflammatory choroidal neovascularization (iCNV) is a severe but uncommon complication of both infectious and non-infectious uveitides. It is hypothesized that its pathogenesis is similar to that of wet age-related macular degeneration (AMD), and involves hypoxia as well as the release of vascular endothelial growth factor, stromal cell-derived factor 1-alpha, and other mediators. Inflammatory CNV develops when inflammation or infection directly involves the retinal pigment epithelium (RPE)–Bruch’s membrane complex. Inflammation itself can compromise perfusion, generating a gradient of retinal-choroidal hypoxia that additionally promotes the formation of choroidal neovascularization in the course of uveitis. The development of choroidal neovascularization is extremely common in conditions such as punctate inner choroidopathy, multifocal choroiditis, serpiginous choroiditis, and presumed ocular histoplasmosis syndrome. Although the majority of iCNV cases are well defined and appear as the “classic” type on fluorescein angiography, the diagnosis of iCNV is challenging due to difficulties in differentiating between inflammatory choroiditis lesions and choroidal neovascularization. Modern multimodal imaging, particularly the recently introduced technology of optical coherence tomography (OCT) and OCT angiography (noninvasive and rapid imaging modalities) can reveal additional features that aid the diagnosis of iCNV. However, more studies are needed to establish their role in the diagnosis and evaluation of iCNV activity. Keywords: uveitis; inflammatory choroidal neovascularization; multimodal imaging; fluorescein angiography; indocyanine green angiography; optical coherence tomography; optical coherence tomography angiography; near-infrared autofluorescence

Advanced MRI methods and PET using radiolabeled amino acids provide valuable information in addition to conventional MR imaging for brain tumor diagnostics. These methods are particularly helpful in challenging situations such as the differentiation of malignant processes from benign lesions, the identification of non-enhancing glioma subregions, the differentiation of tumor progression from treatment-related changes, and the early assessment of response to anticancer therapy. The debate over which of the methods is preferable in which situation is ongoing and has been addressed in numerous studies. Currently, most radiology and nuclear medicine departments perform these examinations independently of each other leading to multiple examinations for the patient. The advent of hybrid PET/MRI allowed a convergence of the methods but to date simultaneous imaging has reached little relevance in clinical neuro-oncology. This is partly due to the limited availability of hybrid PET/MRI scanners, but is also due to the fact that PET is a second-line examination in brain tumors. PET is only required in equivocal situations, and spatial co-registration of PET examinations of the brain to previous MRI is possible without disadvantage. A key factor for the benefit of PET/MRI in neuro-oncology is a multimodal approach that provides decisive improvements in the diagnostics of brain tumors compared with a single modality. This systematic review focuses on studies that were able to demonstrate the additive value of amino acid PET and ‘advanced’ MRI in the diagnosis of brain tumors. Available studies suggest that the combination of amino acid PET and advanced MRI improves grading and the histomolecular characterization of newly diagnosed tumors. However, data concerning the delineation of tumor extent and biopsy guidance are of limited value. A clear additive diagnostic value of amino acid PET and advanced MRI can be achieved regarding the differentiation of tumor recurrence from treatment-related changes. Here, PET-guided evaluation of advanced MR methods seems to be helpful. In summary, there is growing evidence that a multimodal approach can achieve decisive improvements in the diagnostics of brain tumors, for which hybrid PET/MRI offers optimal conditions.

Preliminary clinical data suggest pain reduction through fasting in different diagnoses. This uncontrolled observational clinical study examined the effects of prolonged modified fasting on pain and functional parameters in hip and knee osteoarthritis. Patients admitted to the inpatient department of Internal Medicine and Nature-based Therapies of the Immanuel Hospital Berlin between February 2018 and March 2020, answered questionnaires at the beginning and end of inpatient treatment, as well as 3, 6 and 12 months after discharge. Additionally, selected blood and anthropometric parameters were routinely assessed during the inpatient stay. Fasting was performed as part of a multimodal integrative treatment program, with daily caloric intake of <600 kcal for 7.7 ± 1.7 days. N=125 consecutive patients were included. Results revealed an amelioration of overall symptomatology (WOMAC Index score: -14.8±13.31; p<0.001; d=0.78), and pain alleviation (NRS Pain: -2.7±1.98, p<0.001, d=1.48). Pain medication was reduced, stopped, or replaced by herbal remedies in 36% of patients. Improvements were also observed in secondary outcome parameters, including increased quality of life (WHO-5: +4.5±4.94, p<0.001, d=0.94), reduced anxiety (HADS-A: -2.1±2.91, p<0001, d=0.55) and depression (HADS-D: -2.3±3.01, p<0.001, d=0.65), decreases in body weight (-3.6 kg ± 1.65, p< 0.001, d=0.21), and blood pressure (systolic: -6.2±15.93, p<0.001, d= 0.43; diastolic: -3.7±10.55, p<0.001, d=0.43). Results suggest that patients with osteoarthritis of the lower extremities may profit from a prolonged fast embedded in a multimodal integrative treatment regarding quality of life, pain, and disease-specific functional parameters. Confirmatory RCTs are warranted to further investigate these hypotheses.

In the evolving landscape of Portfolio Management (PM), the fusion of advanced machine 1 learning techniques with traditional financial methodologies has opened new avenues for innovation. 2 Our study introduces a cutting-edge model combining Deep Reinforcement Learning (DRL) with 3 a Non-stationary Transformer architecture. This model is specifically designed to decode complex 4 patterns in financial time series data, enhancing portfolio management strategies with deeper insights 5 and robustness. It effectively tackles the challenges of data heterogeneity and market uncertainty, 6 key obstacles in PM. Our approach integrates key macroeconomic indicators and targeted news 7 sentiment analysis into its framework, capturing a comprehensive picture of market dynamics. This 8 amalgamation of varied data types addresses the multifaceted nature of financial markets, enhancing 9 the model’s ability to navigate through the complexities of asset management. Rigorous testing 10 demonstrates the model’s efficacy, highlighting the benefits of blending diverse data sources and 11 sophisticated algorithmic approaches in mastering the nuances of PM.12

Integrating voice assistant technology can potentially enhance accessibility and user interaction in web-based platforms for geospatial data visualisation and processing. This paper presents an open-source development incorporating a voice virtual assistant into the BStreams web application platform, designed specifically for visualising geospatial data. The study aimed to mitigate the observed limitations of virtual assistants in understanding the semantics and context of user commands in the geospatial domain. A discourse framework specialised for geospatial data was developed using a structured prototype methodology. In addition, a survey assisted in identifying commonly used English terminologies for geospatial data interaction. The questionnaire results were subsequently assessed using the ChatGPT natural language processing model. The application used open-source frameworks, including Web Speech API, Leaflet, Mapbox geocoding, and JavaScript. This investigation indicates that voice virtual assistants can potentially enhance human-computer interaction for geospatial data visualisation and processing. However, challenges related to natural language understanding, accents, and domain knowledge were raised. The paper advocates for further research to refine the integration of voice technology in this domain.

The human mind is multimodal. Yet most behavioral studies rely on century-old measures of behavior—task accuracy and latency (response time). Multimodal and multisensory analysis of human behavior creates a better understanding of how the mind works. The problem is that designing and implementing these experiments is technically complex and costly. This paper introduces versatile and economical means of developing multimodal-multisensory human experiments. We provide an experimental design framework that automatically integrates and synchronizes measures including electroencephalogram (EEG), galvanic skin response (GSR), eye-tracking, virtual reality (VR), body movement, mouse/cursor motion and response time. Unlike proprietary systems (e.g., iMotions), our system is free and open-source; it integrates PsychoPy, Unity and Lab Streaming Layer (LSL). The system embeds LSL inside PsychoPy/Unity for the synchronization of multiple sensory signals—gaze motion, electroencephalogram (EEG), galvanic skin response (GSR), mouse/cursor movement, and body motion—with low-cost consumer-grade devices in a simple behavioral task designed by PsychoPy and a virtual reality environment designed by Unity. This tutorial shows a step-by-step process by which a complex multimodal-multisensory experiment can be designed and implemented in a few hours. When conducting the experiment, all of the data synchronization and recoding of the data to disk will be done automatically.

Integrating advanced air mobility (AAM) into the logistics of high-value electronic commodities can enhance efficiency and promote sustainability. The objective of this study is to optimize the logistics network for high-value electronics by integrating AAM solutions, specifically using heavy-lift cargo drones for middle-mile transport and using the mostly rural and small urban U.S. state of North Dakota as a case study. The analysis utilized geographic information system (GIS) and spatial optimization models to strategically assign underutilized airports as multimodal freight hubs to facilitate the shift from long-haul trucks to middle-mile air transport. Key findings demonstrate that electronics, because of their high value-to-weight ratio, are ideally suited for air transport. Comparative analysis shows that transport by drones can reduce the average cost per ton by up to 60% compared to traditional trucking. Optimization results indicate that a small number of strategically placed logistical hubs can reduce average travel distances by more than 13% for last-mile deliveries. Cost analyses demonstrate the viability of drones for middle-mile transport, especially on lower-volume rural routes, highlighting their efficiency and flexibility. The study emphasizes the importance of utilizing existing infrastructure to optimize the logistics network. By replacing truck traffic with drones, AAM can mitigate road congestion, reduce emissions, and extend infrastructure lifespan. These insights have critical implications for supply chain managers, shippers, urban planners, and policymakers, providing a decision support system and a roadmap for integrating AAM into logistics strategies.

Smart, reliable, and connected multi-modal mobility has been a long-standing goal of transit services. This paper focuses on the smart, seamless, and multi-modal mobility service in the context of “Mobility as a Service” (MaaS). Intelligent mobility is the smarter, greener, and more efficient movement of passengers around the world. Increasingly, mobility is approached as a service. This study first conducts an extensive literature review on mobility behavior and demand pattern of MaaS end-users. It then extends the mechanism of supply chain, MaaS, synergy (i.e., vertical cooperation synergy, horizontal competition synergy), and coopetition to develop the multi-tier closed-loop intelligent mobility service supply chain network. This paper explains the intelligent mobility service supply chain network from following perspectives: (i) mobility service taxonomy of MaaS; (ii) aims of intelligent mobility service supply chain network; (iii) urban rail transit (URT)-centered alternatives for integrated multimodal journey planning, i.e. access + URT + egress, and both access and egress can be served by Mobility-On-Demand (MOD) transport; (iv) node member imperatives. From a synthesis of insights from the ‘during’ journey, this study puts forward the synergetic design of intelligent mobility service supply chain network, including: (i) multi-tier closed-loop structure; (ii) key nodes identification for the physical multimodal transport network in the supply chain; (iii) hybrid synergy mechanisms among the partners, i.e., synergy principle, temporal splitting approach for coopetition synergy; (iv) index systems and evaluation methods for synergy measurement. This study also contributes to the integrated multimodal journey planning. In concluding, the paper highlights the important implications of the proposed intelligent mobility service supply chain network for MaaS bundle design and adverse effects reduction, resulting from 1 + 1 > 2 synergy effects.

Background: This study addresses the burgeoning impact of Large Language Models (LLMs) such as ChatGPT on the integrity of online examinations and seeks to demonstrate latent and advanced LLM reasoning capabilities. The research devises an iterative self-reflective strategy for invoking critical thinking and higher-order reasoning in LLMs based on multimodal exam questions using visuals and text. Methods: The proposed strategy was demonstrated and evaluated on real exam questions by subject experts and the performance of ChatGPT (GPT-4) with vision was estimated on an additional dataset of 600 text descriptions of multimodal exam questions, focusing on its response to complex, visual, and textual prompts. Results: The results indicate that latent multi-hop reasoning capabilities can be invoked using the proposed self-reflective strategy, which can effectively steer ChatGPT towards correct answers by integrating critical thinking from each modality into the final response. Meanwhile, ChatGPT demonstrated considerable proficiency in being able to answer multimodal exam questions across 12 subjects. Conclusions: This work challenges findings from recent studies which claim that existing LLMs are unable to reason on complex multimodal tasks, and therefore this study underscores the need for enhanced online exam security measures like proctoring software, that can mitigate the potential misuse of AI technologies in educational settings.

This case report demonstrates the rare but potentially serious complication of pulmonary vein stenosis (PVS) diagnosed several years after radiofrequency ablation for atrial fibrillation. Pulmonary vein stenosis can remain asymptomatic until significant occlusion occurs, after which it can present with a variety of symptoms, mimicking numerous cardiovascular and pulmonary pathologies. Early diagnosis and treatment rely on consistent follow-up using appropriate diagnostic imaging modalities, which are paramount in preventing severe complications.