With the development of general anesthesia techniques and anesthetic substances, brought new horizons for the expansion and improvement of surgical techniques. Nevertheless, more complex surgical procedures brought a higher complexity and longer duration for general anesthesia that led to a series of adverse events such as hemodynamic instability, under- or overdosage of anesthetic drugs, as well as an increased number of post-anesthetic events. In order to adapt the anesthesia according to the particularities of each patient, the multimodal monitoring of these patients is highly recommended. Classically, general anesthesia monitoring consists of the analysis of vital functions and gas exchange. Multimodal monitoring refers to the concomitant monitoring of the degree of hypnosis and the nociceptive-antinociceptive balance. By titrating anesthetic drugs according to these parameters, clinical benefits can be obtained, such as hemodynamic stabilization, reduction of awakening times, and the reduction of post-operative complications. Another important aspect is the impact on the status of inflammation and the redox balance. By minimizing inflammatory and oxidative impact one can achieve a faster recovery that will lead to both increased patient satisfaction and an increase in patient safety. The purpose of this literature review is to present the most modern multimodal monitoring techniques, respectively to discuss the particularities of each technique.

The advent of the era of big data will bring more convenience to people and greater development to society. But at the same time, it will also bring people the problem of 'information overload', i.e., when people are faced with huge information data, there are many redundant and worthless data. The redundant and worthless data information seriously interferes with the accurate selection of information data. Even though people can use Internet search engines to access information data, they cannot meet the individual needs of specific users in specific contexts. The personalized needs of a particular user in a particular context. Therefore, how to find useful and valuable information quickly has become one of the key issues in the development of big data. With the advent of the era of big data, recommendation systems, as an important technology to alleviate information overload, have been widely used in the field of e-commerce. Recommender systems suffer from a key problem: data sparsity. The sparsity of user history rating data causes insufficient training of collaborative filtering recommendation models, which leads to a significant decrease in the accuracy of recommendations. In fact, traditional recommendation systems tend to focus on scoring information and ignore the contextual context in which users interact. There are various contextual modal information in people's real life, which also plays an important role in the recommendation process. In this paper we achieve data degradation and feature extraction, solving the problem of sparse data in the recommendation process. An interaction context-aware sub-model is constructed based on a tensor decomposition model with interaction context information to model the specific influence of interaction context in the recommendation process. Then an attribute context-aware sub-model is constructed based on the matrix decomposition model and using attribute context information to model the influence of user attribute contexts and item attribute contexts on recommendations. In the process of building the model, the method not only utilizes the explicit feedback rating information of users in the original dataset, but also utilizes the interaction context and attribute context information of the implicit feedback and the unlabeled rating data. We evaluate our model by extensive experiments. The results illustrate the effectiveness of our recommender model.

Traumatic brain injury (TBI) is a leading cause of death and disability worldwide. Despite extensive research efforts, the majority of trialed monotherapies to date have failed to demonstrate significant benefit. It has been suggested that this is due to the complex pathophysiology of TBI, which may possibly be addressed by a combination of therapeutic interventions. In this article, we have reviewed combinations of different pharmacologic treatments, combinations of non-pharmacologic interventions, and combined pharmacologic and non-pharmacologic interventions for TBI. Both preclinical and clinical studies have been included. While promising results have been found in animal models, clinical trials of combination therapies have not yet shown clear benefit. This may possibly be due to their application without consideration of the evolving pathophysiology of TBI. Improvements of this paradigm may come from novel interventions guided by multimodal neuromonitoring and multimodal imaging techniques, as well as the application of multi-targeted non-pharmacologic and endogenous therapies. There also needs to be a greater representation of female subjects in preclinical and clinical studies.

Many engineered approaches have been proposed over the years for solving the hard problem of performing indoor localisation using smartphone sensors. However, specialising these solutions for difficult edge cases remains challenging. Here we propose an end-to-end hybrid multimodal deep neural network localisation system, MM-Loc, relying on zero hand-engineered features, learning them automatically from data instead. This is achieved by using modality-specific neural networks to extract preliminary features from each sensing modality, which are then combined by cross-modality neural structures. We show that our choice of modality-specific neural architectures is capable of estimating the location with good accuracy independently. But for better accuracy, a multimodal neural network fusing the features of early modality-specific representations is a better proposition. Our proposed MM-Loc solution is tested on cross-modality samples characterised by different sampling rates and data representation (inertial sensors, magnetic and WiFi signals), outperforming traditional approaches for location estimation. MM-Loc elegantly trains directly from data unlike conventional indoor positioning systems, which rely on human intuition.

In today’s Web 2.0 era, online social media has become an integral part of our lives. In the course of the information revolution, the form of information has undergone a radical change, from simple text information to today’s integrated video, image, text and audio, and there has also been a great change in the way of dissemination and access, as people nowadays do not just rely on traditional media to passively receive information, but more actively and selectively obtain information from social media. Therefore, it has become a great challenge for us to effectively utilize these massive and integrated multi-modal media information to form an effective system of retrieval, browsing, analysis and usage. Unlike movies and traditional long-form video content, micro-videos are usually short in length, between a few seconds and tens of seconds, which allows users to quickly browse different contents and make full use of the fragmented time in their lives, while users can also share their micro-videos to their friends or the public, forming a unique social way. Video contains rich multimodal information, and fusing information from multiple modalities in a video recommendation task can improve the accuracy of the video recommendation task.According to the micro-video recommendation task, a new combinatorial network model is proposed to combine the discrete features of each modality into the overall features of various modalities through the network, and then fuse the various modal features to obtain the overall video features, which will be used for recommendation. In order to verify the effectiveness of the algorithm proposed in this paper, experiments are conducted in the public dataset, and it is shown the effectiveness of our model.

Affordances are an important concept in cognition, which can be applied to robots in order to perform a successful human-robot interaction (HRI). In this paper we explore and discuss the idea of emotional affordances and propose a viable model for implementation into HRI. We consider “2-ways” affordances: perceived object triggering an emotion, and perceived human emotion expression triggering an action. In order to make the implementation generic, the proposed model includes a library that can be customised depending on the specific robot and application’s scenario. We present the AAA (Affordance-Appraisal-Arousal) model, which incorporates Plutchik’s Wheel of Emotions, and show some examples of simulation and possible scenarios.

Federated learning (FL) has become a burgeoning and attractive research area, which provides a collaborative training scheme for distributed data sources with privacy concerns. Most existing FL studies focus on taking unimodal data, such as images and text, as the model input and resolving the heterogeneity challenge, i.e., the non-identically distributed (non-IID) challenge caused by data distribution imbalance related to data labels and data amount. In real-world applications, data are usually described by multiple modalities. However, to the best of our knowledge, only a handful of studies have been proposed to improve the system performance by utilizing multimodal data. In this survey paper, we identify the significance of this emerging research topic – multimodal federated learning (MFL) and perform a literature review on the state-of-art MFL methods. Furthermore, we categorize multi-modal federated learning into congruent and incongruent multimodal federated learning based on whether all clients possess the same modal combinations. We investigate the feasible application tasks and related benchmarks for MFL. Lastly, we summarize the promising directions and fundamental challenges in this field for future research.

Effective healthcare relies on accurate and timely diagnosis; however, obtaining large amounts of training data while maintaining patient privacy remains challenging. This study introduces a novel approach utilizing federated learning (FL) and a cross-device multi-modal model for clin-ical event classification using vital signs data. Our architecture leverages FL to train machine learning models, including Random Forest, AdaBoost, and SGD ensemble model, on vital signs data from a diverse clientele at a Boston hospital (MIMIC-IV dataset). The FL structure preserves patient privacy by training directly on each client's device without transferring sensitive data. The study demonstrates the potential of FL in privacy-preserving clinical event classification, achieving an impressive accuracy of 98.9%. These findings underscore the significance of FL and cross-device ensemble technology in healthcare applications, enabling the analysis of large amounts of sensitive patient data while safeguarding privacy.

Accurate evaluation of patients with disorders of consciousness (DoC) is crucial for personalized treatment. However, misdiagnosis remains a serious issue. Neuroimaging methods could observe the conscious activity in patients who have no evidence of consciousness in behavior, and provide objective and quantitative indexes to assist doctors in their diagnosis. In the review, we discussed the current research based on the evaluation of consciousness rehabilitation after DoC using EEG, fMRI, PET, and fNIRS, as well as the advantages and limitations of each method. Nowadays single-modal neuroimaging can no longer meet the researchers` demand. Considering both spatial and temporal resolution, recent studies have attempted to focus on the multi-modal method which can enhance the capability of neuroimaging methods in the evaluation of DoC. As neuroimaging devices become wireless, integrated, and portable, multi-modal neuroimaging methods will drive new advancements in brain science research.

In this paper, we present a multimodal dataset for the analysis of classroom atmosphere, based on the behavior and voice of teachers in teaching scenarios. we propose four visual models, three audio models, and one visual-audio dual-modality model to be tested on our dataset. The results indicate that the CH-CC dataset is feasible and reliable and that the visual modality plays a major role in the analysis of this dataset.

Hateful and abusive speech presents a major challenge for all online social media platforms. Recent advances in Natural Language Processing and Natural Language Understanding allow more accurate detection of hate speech in textual streams. This study presents a multimodal approach to hate speech detection by combining Computer Vision and Natural Language processing models for abusive context detection. Our study focuses on Twitter messages and, more specifically, on hateful, xenophobic and racist speech in Greek aimed at refugees and migrants. In our approach we combine transfer learning and fine-tuning of Bidirectional Encoder Representations from Transformers (BERT) and Residual Neural Networks (Resnet). Our contribution includes the development of a new dataset for hate speech classification, consisting of tweet ids, along with the code to obtain their visual appearance, as they would have been rendered in a web browser. We have also released a pre-trained Language Model trained on Greek tweets, which has been used in our experiments. We report a consistently high level of accuracy (accuracy score=0.970, f1-score=0.947 in our best model) in racist and xenophobic speech detection.

Microbial foodborne pathogens can cause major risks to public health and the food industry. Rapid and accurate detection of these pathogens is crucial to prevent outbreaks and ensure food safety. Traditional single-mode biosensing approaches have their limitations in terms of specificity, sensitivity, and speed. Over the years, there has been an increasing research interest in developing multimodal biosensing strategies that integrate multiple sensing techniques to achieve enhanced accuracy, efficacy, and precision in foodborne pathogen detection. This article review paper explores the current state of multimodal biosensing technologies and their promising applications in the food industry. We discuss various integrated biosensing platforms, including optical, electrochemical, and nanomaterial-based systems, among others. The advantages, challenges, and future prospects of multimodal biosensing for foodborne pathogens are thoroughly examined, highlighting the potential impact of these advanced technologies on food safety and public health.

This paper proposes a multimodal fusion 3D target detection algorithm based on the attention mechanism to improve the performance of 3D target detection. The algorithm utilizes point cloud data and information from camera. For image feature extraction, the ResNet50+FPN architecture extracts features at four levels. Point cloud feature extraction employs the voxel method and FCN to extract point and voxel features. The fusion of image and point cloud features is achieved through regional point fusion and regional voxel fusion methods. After information fusion, the Coordinate attention mechanism and SimAM attention mechanism extract fusion features at a deep level. The algorithm's performance is evaluated using the DAIR-V2X dataset. The results show that compared to the Part-A2 algorithm, the proposed algorithm improves the mAP value by 7.9% in BEV view and 7.8% in 3D view at IOU=0.5 (cars) and IOU=0.25 (pedestrians and cyclist). At IOU=0.7 (cars) and IOU=0.5 (pedestrians and cyclist), the mAP value of the SECOND algorithm is improved by 5.4% in the BEV view and 4.3% in the 3D view, compared to other comparison algorithms.

Multimodal traveling is expected to enhance mobility for users, reduce inequalities of car ownership and reduce emissions. In the same context, ride-sharing aims to minimize negative impacts related to emissions, reduce travelling costs and congestion, and increase passenger vehicle occupancy, and public transit ridership when planned for first/last mile trips. The goal of this study is to present the planning of a multimodal pilot demonstration and the challenges that emerged during and after its implementation in Athens, Greece. The demo aims to enhance the connection of low-density Attica Region areas to public transport (PT) modes, and specifically to the metro, through the provision of demand responsive ride-sharing services. During the demo period two different applications were utilized: the “Travel Companion” app and the “Driver Companion” app, which refer to passengers and drivers of the ride-sharing service. Identification of demo participants is performed through a Stated Preference (SP) experiment. Results of the demo implementation, as well as challenges that were faced show that although participants are willing to try new mobility solutions, the readiness and reliability of the new service are essential attributes towards maintaining existing users and engaging new ones.

With the prevalence and evolution of social media platforms, social media data have emerged as a crucial source for obtaining spatio-temporal information about various urban events. Providing accurate spatio-temporal information of these events enhances the capacities of urban management and emergency response. However, existing research mostly focuses on the textual content while mining this spatio-temporal information, often neglecting data from other modalities like images and videos.To address this, our study introduces a novel method for extracting spatio-temporal information from multi-modal social media data (MIST-SMMD), serving as a valuable supplement to current urban event monitoring methods. Leveraging deep learning and Geographic Information System (GIS) technologies, we extract spatio-temporal information from large-scale, multi-modal Weibo data about urban waterlogging events at both coarse and fine granularities.Through an in-depth experimental evaluation of the “July 20 Zhengzhou extreme rainstorm” event, the results show that in coarse-grained spatial information extraction solely using textual data, our method achieves a Spatial Precision of 87.54% within a 60m range and 100% Spatial Precision within a 201m range. In the fine-grained spatial information extraction, by incorporating other modalities such as images and videos, the Spatial Error saw a significant improvement, with MAESE increasing by 95.53% and RMSESE by 93.62%.These outcomes illustrate the capability of the MIST-SMMD method in extracting spatio-temporal information of urban events at both coarse and fine granularities. They also confirm the notable advantage of multi-modal data in enhancing the accuracy of spatial information extraction.

With the rise of piano teaching in recent years, many people have joined the team of piano learners. However, the expensive cost of manual instruction and the unique one-on-one teaching model have made piano learning an extravagant event. Most existing approaches based on the audio modality aim to evaluate piano players' skills. Unfortunately, these methods ignored the information contained in the video, which led to a one-sided and simplistic evaluation of the piano player's skills. More recently, multimodal-based methods are proposed to assess the skill level of piano players using both video and audio information. However, existing multimodal approaches use shallow networks to extract video and audio features, which are deficient in extracting complex spatio-temporal and time-frequency features from piano performance. Furthermore, the fingering and the pitch-rhythm information of the piano performance is contained in the spatio-temporal and time-frequency features, respectively. In the paper, we propose a ResNet-based audio-visual fusion model that combines video and audio features to assess the skill level of piano players. Firstly, ResNet18-3D is used as the backbone network for our visual branches, which can extract feature information from the video data. Then, we consider ResNet18-2D as the backbone network of the aural branch and extract the feature information from the audio data. The extracted video features are fused with the audio features to generate multimodal features for the final piano skill evaluation. The experimental results on the PISA dataset show that our proposed audio-visual fusion model, with a validation accuracy of 70.80%, outperforms the state-of-the-art methods in both performance and efficiency. Then, we also explore the impact of different layers of ResNet on model performance, and the experimental results show that the audio-visual fusion model dealing with the piano skill assessment problem can make full use of both feature information when the number of video features is close to the number of audio features.

Digital storytelling is a powerful tool for language learning that has gained popularity in recent years. By incorporating different modes of communication such as text, images, audio, and video, digital storytelling provides learners with engaging and interactive experiences that promote language acquisition and cultural awareness. Digital storytelling can also promote interaction among learners, which is crucial for language learning, and provide learners with opportunities to receive feedback and support from their peers. Additionally, digital storytelling can be used to promote cultural sensitivity and intercultural dialogue among language learners. Through storytelling, learners can gain insights into the customs, values, and beliefs of different communities, and create stories that reflect their own cultural backgrounds and experiences. This paper provides an overview of the benefits of digital storytelling for language learning and highlights the various ways in which it can be used to enhance language acquisition and cultural awareness. It concludes with recommendations on how to effectively incorporate digital storytelling into language teaching practices.

The growing use of sensor tools and the Internet of Things requires sensors to understand the applications. There are major difficulties in realistic situations, though, that can impact the efficiency of the recognition system. Recently, as the utility of deep learning in many fields has been shown, various deep approaches were researched to tackle the challenges of detection and recognition. We present in this review a sample of specialized deep learning approaches for the identification of sensor-based human behaviour. Next, we present the multi-modal sensory data and include information for the public databases which can be used in different challenge tasks for study. A new taxonomy is then suggested, to organize deep approaches according to challenges. Deep problems and approaches connected to problems are summarized and evaluated to provide an analysis of the ongoing advancement in science. By the conclusion of this research, we are answering unanswered issues and providing perspectives into the future.

Multimodal user interfaces promise a natural and intuitive human machine interactions. But is the extra effort for the development of a complex multi-sensor system justified, or can users also be satisfied with one input modality already? This study investigates interactions in an industrial weld inspection workstation. Three unimodal interfaces, including spatial interaction with buttons augmented on a workpiece or a worktable, and speech commands, were tested individually and in a multimodal combination. Within the unimodal conditions, users preferred the augmented worktable, but overall, the interindividual usage of all input technologies in the multimodal condition was ranked best. Our findings indicate that the implementation and use of multiple input modalities is valuable, and that it is difficult to predict the usability of individual input modalities for complex systems.

Improved oil recovery from tight carbonate formations may provide the world with a major source of lower-rate power over several decades. Here we provide an overview of the Arab D formation in the largest oil field on earth, the Ghawar. We investigate the occurrence of microporosity of different origins and sizes using scanning electron microscopy (SEM) and pore casting techniques. Then, we present a robust calculation of the probability of invasion and oil saturation distribution in the nested micropores using mercury injection capillary pressure data available in the literature. We show that large portions of the micropores in Arab D formation would have been bypassed during primary drainage unless the invading crude oil ganglia were sufficiently long. Considering the asphaltenic nature of oil in the Ghawar, we expect the invaded portions of the pores to turn mixed-wet, thus becoming inaccessible to waterflooding until further measures are taken to modify the system’s chemistry.

The combination of non-steroidal anti-inflammatory drugs (NSAIDs) with non-opioid analgesics is common in clinical practice for the treatment of acute pain conditions like post-operative and post-traumatic pain. Besides that, and despite the very satisfactory results achieved by oral analgesics, parenteral analgesia is a key tool in the treatment of painful conditions when the enteral routes are not convenient. The parenteral ready-to-use fixed-dose combinations of NSAIDs with non-opioid analgesics like paracetamol or metamizole could play a central role in the treatment of painful conditions since they are able to gather into only one formulation the advantages of multimodal and parenteral analgesia. Surprisingly, only very recently a parenteral ready-to-use fixed-dose combination of Ibuprofen/Paracetamol was launched to the market. This review aims to review the current availability of NSAID-based Parenteral Fixed-dose combinations with paracetamol and metamizole in the European and American markets, and how the combination of such drugs could play a central role in a multimodal analgesia strategy. Also, it is explored how the parenteral formulations of NSAIDs, paracetamol, and metamizole could serve as starting elements for the development of new parenteral ready-to-use fixed-dose combinations. With this review, we concluded that, despite the well-recognized utility of combining NSAIDs with non-opioid analgesics, there are in the literature several randomized clinical trial studies unable to demonstrate clear advantages concerning their efficacy and safety. In the future, it is deemed necessary to perform clinical trials specifically designed to assess the efficacy and safety of these fixed-dose formulations to generate solid evidence of their clinical advantages.

Worldwide, one of the main causes of death among young adults is multiple trauma. In these pa-tients hemorrhagic shock represents the leading cause for worsening of the clinical status and for increased morbidity and mortality. This is due to a multifactorial complex involving cellular, bi-ological, and biophysical mechanisms. The most important mechanisms affecting clinical out-come are oxidative stress, the augmentation of pro-inflammatory status, immune deficiency, dis-ruptions in the coagulation cascade, imbalances in electrolyte and acid-base homeostasis. Poly-trauma patients in hemorrhagic shock need adequate fluid management to ensure hemodynamic stability that must consider not only the maintenance of adequate blood pressure, but also the ad-equate oxygenation of tissues for optimal cellular function. In the current clinical practice, fluid resuscitation in polytrauma patients uses a variety of widely studied pharmacological products, such as crystalloids, colloids, blood transfusions, and the infusion of other blood products. Alt-hough these products exist, an agreement was not reached on a standard administration protocol that could be generally applied for all patients. Moreover, numerous studies have reported a se-ries of adverse events related to fluid resuscitation and to the inadequate use of these products. This review aims at describing the impact the administration of all the solutions used in fluid re-suscitation might have on the cellular and pathophysiological mechanisms in the case of poly-trauma patients suffering from hemorrhagic shock.

Gold nanoclusters (AuNCs) with fluorescence in the Near Infrared (NIR) by both one- and two-photon electronic excitation were incorporated in mesoporous silica nanoparticles (MSNs) using a novel one-pot synthesis procedure where the condensation polymerization of alkoxysilane monomers in the presence of the AuNCs and a surfactant produce hybrid MSNs of 49 nm diameter. This method was further developed to prepare 30 nm diameter nanocomposite particles with simultaneous NIR fluorescence and superparamagnetic properties, with a core composed of superparamagnetic manganese ferrite nanoparticles (MnFe2O4) coated with a thin silica layer, and a shell of mesoporous silica decorated with AuNCs. The nanocomposite particles feature NIR-photoluminescence with 0.6% quantum yield and large Stokes shift (290 nm), and superparamagnetic response at 300 K, with a saturation magnetization of 13.4 emu g-1. The conjugation of NIR photoluminescence and superparamagnetic properties in the biocompatible nanocomposite has high potential for application in multimodal bioimaging.

Previous studies to recognize negative emotions (e.g. disgust, fear, sadness) for mental health care have used heavy equipment directly attaching electroencephalogram (EEG) electrodes to the head, making it difficult to use in daily life, and they have proposed binary classification methods to determine whether negative emotion or not. To tackle this problem, we propose a negative emotion recognition system to collect multimodal biosignal data such as five EEG signals in an EEG headset and heart rate, galvanic skin response, and skin temperature in a smart band for classifying multiple negative emotions. It consists of android Internet of Things (IoT) application, an oneM2M-compliant IoT server, and a machine learning server. The android IoT application upload the biosignal data to the IoT server. By using the biosignal data stored in the IoT server, the machine learning server recognizes the negative emotions of disgust, fear, and sadness using a multi-class support vector machine (SVM) model with a radial basis function kernel (RBF). The experimental results showed that the multi-class SVM model achieved 93% accuracy when considering all the multimodal biosignal data. Moreover, when considering only data in the smart band, it could achieve 98% accuracy by optimizing the hyper-parameter of the RBF kernel.

This study was performed to synthesize multimodal radiopharmaceutical designed for the diagnosis and treatment of prostate cancer. To achieve this goal, Super Paramagnetic Iron Oxide (SPIO) nanoparticles were used as a platform for targeting molecule (PSMA-617) and for complexation of two scandium radionuclides, 44Sc for PET imaging and 47Sc for radionuclide therapy. TEM and XPS images showed that the Fe3O4 NPs have a uniform cubic shape and a diameter from 38 to 50 nm. The Fe3O4 core are surrounded by SiO2 and an organic layer. The saturation magnetization of the SPION core was 60 emu/g. However, coating the SPIONs with silica and polyglycerol reduces the magnetization significantly. The obtained bioconjugates were labeled with 44Sc and 47Sc, with a yield higher than 97%. The radiobioconjugate exhibited high affinity and cytotoxicity towards the human prostate cancer LNCaP (PSMA+) cell line, much higher than for PC-3 (PSMA-) cells. High cytotoxicity of the radiobioconjugate was confirmed by radiotoxicity studies on LNCaP 3D-spheroids. In addition, the magnetic properties of the radiobioconjugate should allow for its use in guide drug delivery driven by magnetic field gradient.

Pipeline integrity is an important area of concern for the oil and gas, refining, chemical, hydrogen, carbon sequestration, and electric-power industries, due to the safety risks associated with pipeline failures. Regular monitoring, inspection, and maintenance of these facilities is therefore required for safe operation. Large standoff magnetometry (LSM) is a non-intrusive, passive magnetometer-based mea- surement technology that has shown promise in detecting defects (anomalies) in regions of elevated mechanical stresses. However, analyzing the noisy multi-sensor LSM data to clearly identify regions of anomalies is a significant challenge. This is mainly due to the high frequency of the data collection, mis-alignment between consecutive inspections and sensors, as well as the number of sensor measurements recorded. In this paper we present LSM defect identification approach based on ma- chine learning (ML). We show that this ML approach is able to successfully detect anomalous readings using a series of methods with increasing model complexity and capacity. The methods start from unsupervised learning with "point" methods and eventually increase complexity to supervised learning with sequence methods and multi-output predictions. We observe data leakage issues for some methods with randomized train/test splitting and resolve them by specific non-randomized splitting of training and validation data. We also achieve a 200x acceleration of support-vector classifier (SVC) method by porting computations from CPU to GPU leveraging the cuML RAPIDS AI library. For sequence methods, we develop a customized Convolutional Neural Network (CNN) architecture based on 1D convolutional filters to identify and characterize multiple properties of these defects. In the end, we report the scalability of the best-performing methods and compare them, for viability in field trials.

Medical imaging is an active field of research that fosters the necessity for novel multimodal imaging probes. In this line, nanoparticle-based contrast agents are of special interest, since those can host functional entities either within their interior, reducing potential toxic effects of the imaging tracers, and on their surface, providing high payloads of probes, due to their large surface-to-volume ratio. The long-term stability of the particles in solution is an aspect usually under-tackled during probe design in research laboratories, since their performance is generally tested briefly after synthesis. This may jeopardize a later translation into practical medical devices, due to stability reasons. To dig into the effects of nanoparticle aging in solution, respect to their behavior in vivo, iron oxide stealth nanoparticles were used at two stages (3 weeks vs. 9 months in solution), analyzing their biodistribution in mice. Both sets of nanoprobes showed similar sizes, zeta potentials and morphology, as observed by DLS and TEM but, fresh nanoparticles accumulated in the kidneys after systemic administration, while aged ones accumulated in liver and spleen, confirming an enormous effect of particle aging on their in vivo behavior, despite barely noticeable changes perceived on a simple inspection of their structural integrity.

Autonomous driving vehicles rely on sensors for the robust perception of surroundings. Such vehicles are equipped with multiple perceptive sensors with a high level of redundancy to ensure safety and reliability in any driving condition. However, multi-sensor, such as camera, LiDAR and radar, systems bring up the requirements related to sensor calibration and synchronization, which are the fundamental blocks of any autonomous system. On the other hand, sensor fusion and integration have become important aspects of autonomous driving research and directly determine the efficiency and accuracy of advanced functions such as object detection and path planning. Classical model-based estimation and data-driven models are two mainstream approaches to achieving such integration. Most recent research is shifting to the latter, showing high robustness in real-world applications but requiring large quantities of data to be collected, synchronized, and properly categorized. To generalize the implementation of the multi-sensor perceptive system, we introduce an end-to-end generic sensor dataset collection framework that includes both hardware deploying solutions and sensor fusion algorithms. The framework prototype integrates a diverse set of sensors, such as cameras, LiDAR, and radar. Furthermore, we present a universal toolbox to calibrate and synchronize three types of sensors based on their characteristics. The framework also includes the fusion algorithms, which utilize the merits of three sensors , namely, camera, LiDAR and radar, and fuse their sensory information in a manner that is helpful for object detection and tracking research. The generality of this framework makes it applicable in any robotic or autonomous applications, also suitable for quick and large-scale practical deployment.

Humans’ performance varies due to the mental resources that are available to successfully pursue a task. To monitor users’ current cognitive resources in naturalistic scenarios, it is essential to not only measure demands induced by the task itself but also consider situational and environmental influences. We conducted a multimodal study with 18 participants (nine female, M = 25.9 with SD = 3.8 years). In this study, we recorded, respiratory, ocular, cardiac, and brain activity using functional near-infrared spectroscopy (fNIRS) while participants performed an adapted version of the warship commander task with concurrent emotional speech distraction. We tested the feasibility of decoding the experienced mental effort with a multimodal machine learning architecture. The architecture comprised feature engineering, model optimisation, and model selection to combine multimodal measurements in a cross-subject classification. Our approach reduces possible overfitting and reliably distinguishes two different levels of mental effort. These findings contribute to the prediction of different states of mental effort and pave the way toward generalised state monitoring across individuals in realistic applications.

Currently, skin cancer is the most commonly diagnosed form of cancer in humans and is one of the leading causes of death in patients with cancer. Biopsy methods are an invasive research method and are not always available for primary diagnosis. Imaging methods have low accuracy and depend on the experience of the dermatologist. Artificial intelligence technologies can match and surpass visual analysis methods in accuracy, but they have the risk of a false negative response when a malignant pigmented lesion can be recognized as benign. One possible way to improve accuracy and reduce the risk of false negatives is to analyze heterogeneous data, combine different preprocessing methods, and use modified loss functions to eliminate the negative impact of unbalanced dermatological data. The paper proposes a multimodal neural network system with a modified cross-entropy loss function that is sensitive to unbalanced heterogeneous dermatological data. The accuracy of recognition in 10 diagnostically significant categories for the proposed system was 85.19%. The novelty of the proposed system lies in the use of cross-entropy loss when training the modified function with the help of weight coefficients. The introduction of weighting factors has reduced the number of false negative forecasts, as well as improved accuracy by 1.02-4.03 percentage points compared to the original multimodal systems. The introduction of the proposed multimodal system as an auxiliary diagnostic tool can reduce the consumption of financial and labor resources involved in the medical industry, as well as increase the chance of early detection of skin cancer.

Estimating Visual Quality of Street Space (VQoSS) is pivotal for urban design, environmental sustainability, civic engagement, etc. Recent advancements, notably in deep learning, have enabled large-scale analysis. However, traditional deep learning approaches are hampered by extensive data annotation requirements and limited adaptability across diverse VQoSS tasks. Multimodal Large Language Models (MLLMs) have recently demonstrated proficiency in various computer vision tasks, positioning them as promising tools for automated VQoSS assessment. In this paper, we pioneer the application of MLLMs to VQoSS change estimation, with our empirical findings affirming their effectiveness. In addition, we introduce Street Quality GPT (SQ-GPT), a model that distills knowledge from the current most powerful but inaccessible (not free) GPT-4, requiring no human efforts. SQ-GPT approaches GPT-4’s performance and is viable for large-scale VQoSS change estimation. In a case study of Nanjin, we showcase the practicality of SQ-GPT and knowledge distillation pipeline. Our work promises to be a valuable asset for future urban studies research.

Physicians often label anatomical structure sets in Digital Imaging and Communications in Medicine (DICOM) images with nonstandard names. As these names vary widely, the standardization of the nonstandard names in the Organs at Risk (OARs), Planning Target Volumes (PTVs), and 'Other' organs inside the area of interest is a vital problem. Prior works considered traditional machine learning approaches on structure sets with moderate success. This paper presents integrated deep learning methods applied to structure sets by integrating the multimodal data compiled from the radiotherapy centers administered by the US Veterans Health Administration (VHA) and the Department of Radiation Oncology at Virginia Commonwealth University (VCU). The de-identified radiation oncology data collected from VHA and VCU radiotherapy centers have 16,290 prostate structures. Our method integrates the heterogeneous (textual and imaging) multimodal data with Convolutional Neural Network (CNN)-based deep learning approaches like CNN, Visual Geometry Group (VGG) network, and Residual Network (ResNet). Our model presents improved results in prostate (RT) structure name standardization. Evaluation of our methods with macro-averaged F1 Score shows that our deep learning model with single-modal textual data usually performs better than the previous studies. We also experimented with various combinations of multimodal data (masked images, masked dose) besides textual data. The models perform well on the textual data alone, while the addition of imaging data shows that deep neural networks can achieve improved performance using information present in the other modalities. Additionally, using masked images and masked doses along with text leads to an overall performance improvement with the various CNN-based architectures than using all the modalities together. Undersampling the majority class leads to further performance enhancement. The VGG network on the masked image-dose data combined with CNNs on the text data performs the best and establishes the state-of-the-art in this domain.

Increased mitotic activity is associated with the genesis and aggressiveness of many cancers. To assess the clinical value of mitotic activity as prognostic biomarker, we performed a pan-cancer study on the mitotic network activity index (MNAI) constructed based on 54-gene mitotic apparatus network. Our pan-cancer assessment on TCGA (33 tumor types, 10,061 patients) and validation on other publicly available cohorts (23 tumor types, 9,209 patients) confirmed the significant association of MNAI with overall survival, progression-free survival and other prognostic endpoints in multiple cancer types, including Lower-Grade Gliomas (LGG), Breast Invasive Carcinoma (BRCA) and many others. We also showed significant association of MNAI with genetic instability, which provides a biological explanation of its prognostic impact at pan-cancer landscape. Futhermore, we found that patients with high MNAI benefit more from anti-PD-1 and Anti-CTLA-4 treatment. In addition, we demonstrated on LGG and BRCA that the multimodal integration of MNAI and the AI-empowerd Cellular Morphometric Subtypes (CMS) significantly improved the predictive power of prognosis compared to MNAI and CMS alone. Our results suggest that MNAI can be used as a potential prognostic biomarker for different tumor types toward different clinical endpoints, and multimodal integration of MNAI and CMS exceeds individual biomarker for precision prognosis.

Acute low back pain (LBP) stands as a leading cause of activity limitation and work absenteeism, and its associated healthcare expenditures are expected to become substantial when acute LBP develops into a chronic and even refractory condition. Therefore, early intervention is crucial to prevent progression to chronic pain whose management is particularly challenging and for which the most effective pharmacological therapy is still controversial. Current guideline treatment recommendations vary and are mostly driven by expertise with opinion differing across different interventions. Thus, it is difficult to formulate evidence-based guidance when relatively few randomized clinical trials did explore the diagnosis and management of LBP while employing different selection criteria, statistical analyses, and outcome measurements. This narrative review aims to provide a critical appraisal of current acute LBP management by discussing the unmet needs and areas of improvement from bench-to-bedside and proposes multimodal analgesia as the way forward to attain an effective and prolonged pain relief and functional recovery in patients with acute LBP.

The aim of the study is to evaluate the predictive factors of response to induction chemotherapy in patients with resectable NSCLC, treated at the unit of Thoracic Surgery of Siena University Hospital with radical surgery. From January 1, 2013 to December 31, 2020, 78 patients were recruited. We analyzed the outcomes in terms of 5-years OS based on the Estimated Regression Rate, N2 downstaging and age; two patients’ subgroups were created by age (Group A: age <66 years-old; Group B: age >66 years-old). No 5-year OS difference was observed based on age, while it was observed in patients with N2 downstaging (p=0.031). Bewtween patients with N2 downstaging, only patients in Group A had a significantly increased 5-year OS (p=0.019), while this was not observed in Group B (p=0-321); the same result was observed with the Estimated Regression Rate > 50% (Group A p=0.005; Group B p=0.391). The percentage of disease regression and the N2 down-staging after induction chemotherapy have great value on the survival, although this advantage seems to be observed mostly in younger patients. A multidisciplinary oncologic discussion of clinical cases could provide support in the careful selection of the ideal patients to undergo neoadjuvant treatment before radical surgery.

Document classification is one of the most critical steps in the document analysis pipeline. There are two types of approaches for document classification, known as image-based and multimodal approaches. The image-based document classification approaches are solely based on the inherent visual cues of the document images. In contrast, the multimodal approach co-learns the visual and textual features, and it has proved to be more effective. Nonetheless, these approaches require a huge amount of data. This paper presents a novel approach for document classification that works with a small amount of data and outperforms other approaches. The proposed approach incorporates a hierarchical attention network(HAN) for the textual stream and the EfficientNet-B0 for the image stream. The hierarchical attention network in the textual stream uses the dynamic word embedding through fine-tuned BERT. HAN incorporates both the word level and sentence level features. While the earlier approaches rely on training on a large corpus (RVL-CDIP), we show that our approach works with a small amount of data (Tobacco-3482). To this end, we trained the neural network at Tobacco-3428 from scratch. Thereby, we outperform state-of-the-art by obtaining an accuracy of 90.3%. This results in a relative error reduction rate of 7.9%.

Advanced MRI methods and PET using radiolabeled amino acids provide valuable information in addition to conventional MR imaging for brain tumor diagnostics. These methods are particularly helpful in challenging situations such as the differentiation of malignant processes from benign lesions, the identification of non-enhancing glioma subregions, the differentiation of tumor progression from treatment-related changes, and the early assessment of response to anticancer therapy. The debate over which of the methods is preferable in which situation is ongoing and has been addressed in numerous studies. Currently, most radiology and nuclear medicine departments perform these examinations independently of each other leading to multiple examinations for the patient. The advent of hybrid PET/MRI allowed a convergence of the methods but to date simultaneous imaging has reached little relevance in clinical neuro-oncology. This is partly due to the limited availability of hybrid PET/MRI scanners, but is also due to the fact that PET is a second-line examination in brain tumors. PET is only required in equivocal situations, and spatial co-registration of PET examinations of the brain to previous MRI is possible without disadvantage. A key factor for the benefit of PET/MRI in neuro-oncology is a multimodal approach that provides decisive improvements in the diagnostics of brain tumors compared with a single modality. This systematic review focuses on studies that were able to demonstrate the additive value of amino acid PET and ‘advanced’ MRI in the diagnosis of brain tumors. Available studies suggest that the combination of amino acid PET and advanced MRI improves grading and the histomolecular characterization of newly diagnosed tumors. However, data concerning the delineation of tumor extent and biopsy guidance are of limited value. A clear additive diagnostic value of amino acid PET and advanced MRI can be achieved regarding the differentiation of tumor recurrence from treatment-related changes. Here, PET-guided evaluation of advanced MR methods seems to be helpful. In summary, there is growing evidence that a multimodal approach can achieve decisive improvements in the diagnostics of brain tumors, for which hybrid PET/MRI offers optimal conditions.

Preliminary clinical data suggest pain reduction through fasting in different diagnoses. This uncontrolled observational clinical study examined the effects of prolonged modified fasting on pain and functional parameters in hip and knee osteoarthritis. Patients admitted to the inpatient department of Internal Medicine and Nature-based Therapies of the Immanuel Hospital Berlin between February 2018 and March 2020, answered questionnaires at the beginning and end of inpatient treatment, as well as 3, 6 and 12 months after discharge. Additionally, selected blood and anthropometric parameters were routinely assessed during the inpatient stay. Fasting was performed as part of a multimodal integrative treatment program, with daily caloric intake of <600 kcal for 7.7 ± 1.7 days. N=125 consecutive patients were included. Results revealed an amelioration of overall symptomatology (WOMAC Index score: -14.8±13.31; p<0.001; d=0.78), and pain alleviation (NRS Pain: -2.7±1.98, p<0.001, d=1.48). Pain medication was reduced, stopped, or replaced by herbal remedies in 36% of patients. Improvements were also observed in secondary outcome parameters, including increased quality of life (WHO-5: +4.5±4.94, p<0.001, d=0.94), reduced anxiety (HADS-A: -2.1±2.91, p<0001, d=0.55) and depression (HADS-D: -2.3±3.01, p<0.001, d=0.65), decreases in body weight (-3.6 kg ± 1.65, p< 0.001, d=0.21), and blood pressure (systolic: -6.2±15.93, p<0.001, d= 0.43; diastolic: -3.7±10.55, p<0.001, d=0.43). Results suggest that patients with osteoarthritis of the lower extremities may profit from a prolonged fast embedded in a multimodal integrative treatment regarding quality of life, pain, and disease-specific functional parameters. Confirmatory RCTs are warranted to further investigate these hypotheses.

Integrating voice assistant technology can potentially enhance accessibility and user interaction in web-based platforms for geospatial data visualisation and processing. This paper presents an open-source development incorporating a voice virtual assistant into the BStreams web application platform, designed specifically for visualising geospatial data. The study aimed to mitigate the observed limitations of virtual assistants in understanding the semantics and context of user commands in the geospatial domain. A discourse framework specialised for geospatial data was developed using a structured prototype methodology. In addition, a survey assisted in identifying commonly used English terminologies for geospatial data interaction. The questionnaire results were subsequently assessed using the ChatGPT natural language processing model. The application used open-source frameworks, including Web Speech API, Leaflet, Mapbox geocoding, and JavaScript. This investigation indicates that voice virtual assistants can potentially enhance human-computer interaction for geospatial data visualisation and processing. However, challenges related to natural language understanding, accents, and domain knowledge were raised. The paper advocates for further research to refine the integration of voice technology in this domain.

The human mind is multimodal. Yet most behavioral studies rely on century-old measures of behavior—task accuracy and latency (response time). Multimodal and multisensory analysis of human behavior creates a better understanding of how the mind works. The problem is that designing and implementing these experiments is technically complex and costly. This paper introduces versatile and economical means of developing multimodal-multisensory human experiments. We provide an experimental design framework that automatically integrates and synchronizes measures including electroencephalogram (EEG), galvanic skin response (GSR), eye-tracking, virtual reality (VR), body movement, mouse/cursor motion and response time. Unlike proprietary systems (e.g., iMotions), our system is free and open-source; it integrates PsychoPy, Unity and Lab Streaming Layer (LSL). The system embeds LSL inside PsychoPy/Unity for the synchronization of multiple sensory signals—gaze motion, electroencephalogram (EEG), galvanic skin response (GSR), mouse/cursor movement, and body motion—with low-cost consumer-grade devices in a simple behavioral task designed by PsychoPy and a virtual reality environment designed by Unity. This tutorial shows a step-by-step process by which a complex multimodal-multisensory experiment can be designed and implemented in a few hours. When conducting the experiment, all of the data synchronization and recoding of the data to disk will be done automatically.

Smart, reliable, and connected multi-modal mobility has been a long-standing goal of transit services. This paper focuses on the smart, seamless, and multi-modal mobility service in the context of “Mobility as a Service” (MaaS). Intelligent mobility is the smarter, greener, and more efficient movement of passengers around the world. Increasingly, mobility is approached as a service. This study first conducts an extensive literature review on mobility behavior and demand pattern of MaaS end-users. It then extends the mechanism of supply chain, MaaS, synergy (i.e., vertical cooperation synergy, horizontal competition synergy), and coopetition to develop the multi-tier closed-loop intelligent mobility service supply chain network. This paper explains the intelligent mobility service supply chain network from following perspectives: (i) mobility service taxonomy of MaaS; (ii) aims of intelligent mobility service supply chain network; (iii) urban rail transit (URT)-centered alternatives for integrated multimodal journey planning, i.e. access + URT + egress, and both access and egress can be served by Mobility-On-Demand (MOD) transport; (iv) node member imperatives. From a synthesis of insights from the ‘during’ journey, this study puts forward the synergetic design of intelligent mobility service supply chain network, including: (i) multi-tier closed-loop structure; (ii) key nodes identification for the physical multimodal transport network in the supply chain; (iii) hybrid synergy mechanisms among the partners, i.e., synergy principle, temporal splitting approach for coopetition synergy; (iv) index systems and evaluation methods for synergy measurement. This study also contributes to the integrated multimodal journey planning. In concluding, the paper highlights the important implications of the proposed intelligent mobility service supply chain network for MaaS bundle design and adverse effects reduction, resulting from 1 + 1 > 2 synergy effects.