Multimodal AI in Healthcare: A Paradigm Shift in Health Intelligence

Description

This book aims to highlight the latest achievements in the use of AI and multimodal artificial intelligence in biomedicine and healthcare. Multimodal AI is a relatively new concept in AI, in which different types of data (e.g. text, image, video, audio, and numerical data) are collected, integrated, and processed through a series of intelligence processing algorithms to improve performance. The edited volume contains selected papers presented at the 2022 Health Intelligence workshop and the associated Data Hackathon/Challenge, co-located with the Thirty-Sixth Association for the Advancement of Artificial Intelligence (AAAI) conference, and presents an overview of the issues, challenges, and potentials in the field, along with new research results. This book provides information for researchers, students, industry professionals, clinicians, and public health agencies interested in the applications of AI and Multimodal AI in public health and medicine. Preface Contents Contributors Abbreviations Multimodal Artificial Intelligence: Next Wave of Innovation in Healthcare and Medicine 1 Introduction 2 Clinical and Biomedical Applications of Multimodal AI and Data Science 3 Advances in AI Technologies and Data Analytics in Healthcare References Unsupervised Numerical Reasoning to Extract Phenotypes from Clinical Text by Leveraging External Knowledge 1 Introduction 2 Related Work 3 Methodology 3.1 External Knowledge 3.2 Number and Lexical Candidates Extraction 3.3 Contextualized Embeddings for Numeric Entities and Lexical Candidates 3.4 Embedding Similarity and Deterministic HPO Assignment 4 Experiment Design 4.1 Datasets 4.2 Implementation Details 4.3 Baselines and Evaluation Methods 5 Results and Discussion 5.1 Quantitative Analysis 5.2 Qualitative Analysis 5.3 Ablation Studies 6 Conclusions and Future Works References Domain-specific Language Pre-training for Dialogue Comprehension on Clinical Inquiry-Answering Conversations 1 Introduction 2 Domain-Specific Language Pre-training 2.1 Conversation-based Sample Construction 2.2 Experiment Setup of Pre-training 3 Dialogue Comprehension on Clinical Inquiry-Answering Conversations 3.1 Task Definition 3.2 Clinical Dialogue Corpus 3.3 Baseline Models 3.4 Training Configuration 3.5 Evaluation: Comparison with Baselines 3.6 Evaluation in Low-Resource Scenarios 3.7 Evaluation: Pre-training Scheme Comparison 4 Conclusions References Clinical Dialogue Transcription Error Correction Using Seq2Seq Models 1 Introduction 2 Related Work 3 Clinical Dialogue Transcription 3.1 Gastrointestinal Clinical Dialogue Dataset 4 Methods 4.1 General Purpose Base Language Models 4.2 PubMed Gastrointestinal Dataset 4.3 Fine-Tune Using Self-Supervision 5 Evaluation 5.1 Performance Metric 5.2 Comparison of Base Language Models 5.3 Comparison of Fine-Tuned Language Models 6 Discussion 7 Conclusion References Customized Training of Pretrained Language Models to Detect Post Intents in Online Health Support Groups 1 Introduction 2 Background and Related Work 3 Tweet2Quit Dataset 3.1 Data Collection 3.2 Identification of the Intents to Annotate 3.3 Reliability 3.4 Annotation Process 4 Models 4.1 Random Forest (Baseline) 4.2 Pretrained Language Models 5 Adapting to the Labels' Relationships 5.1 Customized Loss Functions 6 Experiments 6.1 Experimental Setup 6.2 Pretrained Language Models 6.3 Loss Functions and Adjusted Metrics 7 Conclusion and Discussion References EXPECT-NLP: An Integrated Pipeline and User Interface for Exploring Patient Preferences Directly from Patient-Generated Text 1 Introduction 2 Aspect Based Sentiment Analysis 3 Data Model 4 Aspect Extraction Results 5 EXPECT-NLP Interface 6 Real-World Use Cases 7 Conclusion and Future Work References Medication Error Detection Using Contextual Language Models 1 Introduction 2 Proposed Methodology 2.1 Problem Formalization 2.2 Contextual Language Models 3 Experimentation 3.1 Dataset Generation 3.2 ASR Implementation 3.3 Results and Analysis 4 Conclusions References Latent Representation Weights Learning of the Indefinite Length of Views for Conception Diagnosis 1 Introduction 2 Integration and Methods 2.1 Latent Representation Weight Learning 2.2 Optimization 3 Results and Discussion 3.1 Data Collection 3.2 Experimental Setup 3.3 Comparison with the Baseline 4 Conclusion References Phenotyping with Positive Unlabelled Learning for Genome-Wide Association Studies 1 Introduction 2 Related Work 3 Methods 3.1 Problem Formulation 3.2 Anchor Learning 3.3 Using BERT as Anchor Classifier 3.4 Baselines 3.5 Anchor Performance Metrics 3.6 Hyperparameters 4 Experiments and Results 4.1 UK Biobank data 4.2 Anchor Classifier Performance and Robustness to Control Noise 4.3 Evaluating Phenotypes with GWAS 5 Discussion References Out-of-Distribution Detection for Medical Applications: Guidelines for Practical Evaluation 1 Introduction 2 Related Work 3 Methods 3.1 Considerations 3.2 Designing OOD Tests 3.3 Dataset 3.4 Models 3.5 AUC Score of OOD Detection 4 Experimental Results 5 Discussion References A Robust System to Detect and Explain Public Mask Wearing Behavior 1 Introduction 2 Background and Related Work 2.1 Face Mask Detection 2.2 Explanation 3 Methodology 3.1 Detection Architecture 3.2 Explanation Architecture 3.3 Transfer Learning 3.4 Individual and Aggregate Explanation 4 Experimentation 4.1 Dataset 4.2 Augmentation 4.3 Experiment Setup 5 Evaluation 5.1 Qualitative Results 5.2 Quantitative Results 5.3 Sanity Check 6 Conclusion References A Federated Cox Model with Non-proportional Hazards 1 Introduction 2 Background and Related Work 3 Model 4 Experiments 4.1 Datasets 4.2 Setup 4.3 Results 5 Conclusion Appendix A Additional Figures References A Step Towards Automated Functional Assessment of Activities of Daily Living 1 Introduction 2 Related Works 3 Dataset 4 Proposed Approach 5 Training and Evaluation 6 Experiments and Results 6.1 Ablation Study 7 Conclusion References The Interpretation of Deep Learning Based Analysis of Medical Images—An Examination of Methodological and Practical Challenges Using Chest X-ray Data 1 Introduction 1.1 Previous Work 2 Performance Quantification 3 Model Training 4 Analysis 4.1 Understanding Data and Findings Interpretation 5 Summary and Conclusions References Predicting Drug Functions from Adverse Drug Reactions by Multi-label Deep Neural Network 1 Introduction 2 Related Work 3 Proposed Methodology and Working Architecture 3.1 Description of the Datasets 3.2 Problem Statement 3.3 Proposed Methodology 4 Performance Evaluation of Experiments 4.1 Performance Measurement 4.2 Experimental Results and Discussions 5 Conclusion References Pattern Discovery in Physiological Data with Byte Pair Encoding 1 Introduction 2 Related Work 3 Method 3.1 PAA Transformation 3.2 Discretization 3.3 Identifying Patterns 3.4 Handling Consecutive Identical Symbols 3.5 Post Processing 3.6 Extension to Multivariate Series 3.7 Classification/Regression 3.8 Hyper-Parameter Tuning 4 Data 5 Results 5.1 Computation Speed 5.2 Interpretability 6 Conclusion References Predicting ICU Admissions for Hospitalized COVID-19 Patients with a Factor Graph-based Model 1 Introduction 2 Model 2.1 Model Overview 2.2 Variable Selection 2.3 Factor Function Construction 2.4 Inference Algorithms 3 Experimental Setup 3.1 Dataset 3.2 Model Evaluation 4 Results 4.1 Predictive Biomarkers 4.2 Model Validation 5 Limitations and Future Work 6 Conclusion References Semantic Network Analysis of COVID-19 Vaccine Related Text from Reddit 1 Introduction 2 Data 3 Methods 4 Results 5 Discussion 6 Conclusion References Towards Providing Clinical Insights on Long Covid from Twitter Data 1 Introduction 2 Related Work 2.1 Social Media Platform for COVID-19 2.2 Clinical Information Extraction 2.3 Interpretability 3 Dataset 3.1 Data Acquisition and De-identification 3.2 Filtering Long COVID Self Reports 4 Methodology 5 Results and Discussion References Predicting Infections in the Covid-19 Pandemic—Lessons Learned 1 Introduction 2 Related Work 3 Learning-based Models for Predicting the Number of Infections 4 Experiments 5 Conclusion References Improving Radiology Report Generation with Adaptive Attention 1 Introduction 2 Related Work 3 Method 3.1 Encoder-Decoder Framework 3.2 Multi-Head Adaptive Attention 3.3 Choice of Pretrained Visual Extractor 4 Experiments and Results 4.1 Datasets 4.2 Model Evaluation 4.3 Model Development and Hyper Parameter Tuning 4.4 Ablation Study 4.5 Quantitative Evaluation 4.6 Qualitative Analysis 5 Conclusion References Instantaneous Physiological Estimation Using Video Transformers 1 Introduction 2 Related Work 2.1 Video Based Physiology Extraction 2.2 Transformers 3 Methods 3.1 Optical Basis of Video-Based Bio-Signal Extraction 3.2 Video Transformer for Physiological Estimation 3.3 Loss Formulation 4 Results 4.1 Implementation Details 4.2 Datasets and Evaluation Protocol 4.3 Heart Rate Estimation Results 4.4 Spatial Attention Mask 4.5 Respiration Rate Estimation Results 5 Conclusion References Automated Vision-Based Wellness Analysis for Elderly Care Centers 1 Introduction 2 Related Work 3 Proposed Wellness Analysis System 3.1 Facial Analysis 3.2 Activity Analysis 3.3 Interaction Analysis 3.4 Analysis of Long-Term Pattern and Trend 4 Evaluation 4.1 Data Collection 4.2 Results 5 Conclusion References Efficient Extraction of Pathologies from C-Spine Radiology Reports Using Multi-task Learning 1 Introduction 2 Datasets 3 Description of the Workflow 4 Methods 5 Results 6 Empirical Evidence Behind MultiTasking Models 7 Conclusion References Benchmarking Uncertainty Quantification on Biosignal Classification Tasks Under Dataset Shift 1 Introduction 2 Related Work 3 Uncertainty Quantification Approaches 4 Bencmark Tasks and Experiemnts 4.1 Biosignal Classification Tasks 4.2 Dataset Shift and Evaluation Protocol 4.3 Metrics 5 Results and Analysis 6 Conclusions References Mining Adverse Drug Reactions from Unstructured Mediums at Scale 1 Introduction 2 Related Work 3 Approach 3.1 Classification 3.2 Named Entity Recognition 3.3 Relation Extraction 4 Experimental Setup 4.1 Datasets 4.2 Experiments 4.3 Results 5 Conclusion References A Graph-based Imputation Method for Sparse Medical Records 1 Introduction 2 Method 3 Results 4 Conclusion References Using Nursing Notes to Predict Length of Stay in ICU for Critically Ill Patients 1 Introduction 2 Related Works 3 Dataset 4 Proposed Architecture of ICU LOS Prediction 4.1 Transformer Based Document Representation 4.2 TF-IDF Vector 4.3 Severity of Illness (SOI) Score 4.4 Training the Model 5 Evaluation of the Proposed Predictive Model 5.1 Results and Discussions 6 Conclusion References Automatic Classification of Dementia Using Text and Speech Data 1 Introduction 2 Related Work 3 Materials and Methods 3.1 Dataset: DementiaBank Pitt and WLS Corpora 3.2 Data Pre-processing 3.3 Feature Extraction 4 Ensemble Model 4.1 Deep Learning Model 4.2 Model Evaluation 5 Discussion 6 Conclusion References Unified Tensor Network for Multimodal Dementia Detection 1 Introduction 2 Background 2.1 Alzheimer's Disease and Dementia 2.2 Multimodal Machine Learning 3 Unified Tensor Analysis Pipeline 3.1 Dataset 3.2 Preprocessing 3.3 Representation 3.4 Fusion 3.5 Evaluation 4 Experiments and Results 5 Conclusion References