Informed Machine Learning

Description

This book presents the concept of Informed Machine Learning and demonstrates its practical use with a compelling collection of applications of this paradigm in industrial and business use cases. These range from health care over manufacturing and material science to more advanced combinations with Deep Learning, say, in the form of physical informed neural networks. The book is intended for those interested in modern Informed Machine Learning for a wide range of practical applications where the aspect of small data sets is a challenge. Machine Learning with small amounts of data? After the recent success of Artificial Intelligence based on training with massive amounts of data, this idea may sound exotic. However, it addresses crucial needs of practitioners in industry. While many industrial applications stand to benefit from the use of AI, the amounts of data needed by current learning paradigms are often hard to come by in industrial settings. As an alternative, learning methods and models are called for which integrate other sources of knowledge in order to compensate for the lack of data. This is where the principle of “Informed Machine Learning” comes into play. Informed Machine Learning combines purely data driven learning and knowledge-based techniques to learn from both data and knowledge. This has several advantages. It reduces the need for data, it often results in smaller, less complex and more robust models, and even makes Machine Learning applicable in settings where data is scarce. The kind of knowledge to be incorporated into learning processes can take many different forms, for example, differential equations, analytical models, simulation results, logical rules, knowledge graphs, or human feedback which makes the approach overall very powerful and widely applicable. The past decade has seen substantial progress in the field of Artificial Intelligence (AI). This has primarily been due to the increasingly rapid developments in the field of machine learning (ML) which, in turn, benefited from the confluence of four technological trends: (1) availability of ever-increasing training data sets, (2) comparatively cheap high-performance computing hardware, (3) open source code sharing and access to software for model training or to pre-trained models, and (4) theoretical and practical progress in deep learning and artificial neural networks. As a consequence, there have been significant advancements, say, in natural language processing, image/speech recognition, or autonomous systems. As a result of these developments, AI has now made its way out of academic research into companies and our daily lives. A key feature of today’s cutting-edge AI technologies is their hunger for resources. This is because modern ML models (deep neural networks) have become incredibly large and complex and involve millions if not billions of adjustable parameters. Their training therefore requires enormous amounts of data and considerable computing infrastructures and therefore energy. Alas, in many industries and application domains, data is still scarce or incomplete and there often is limited access to large-scale high performance computing facilities. But even if data availability, compute resources, and energy costs are not an issue, model complexity may still pose challenges with respect to explainability, accountability, or trustworthiness of AI solutions which can be dire in settings where regulatory guidelines have to be met or safety guarantees must be ensured. This is where the paradigm of Informed Machine Learning (Informed ML) comes into play. In a nutshell, the idea of Informed ML is to systematically leverage additional prior knowledge for the design and training of data-driven AI models. The overall goal is to use reliable background knowledge in order to, on the one hand, reduce model complexity and the need for extensive training data and, on the other hand, increase interpretability and explainability of the decisions made by trained models. Preface Contents 1 Introduction and Overview 1.1 Introduction to Informed Machine Learning 1.1.1 Historical Context and Motivation 1.1.2 Concept and Taxonomy 1.1.3 Benefits 1.2 Overview 1.3 Summary References Part I Digital Twins 2 Optimizing Cooling System Operations with Informed ML and a Digital Twin 2.1 Introduction 2.1.1 Related Work 2.1.2 Informed Machine Learning for Cooling System Optimization 2.1.3 Structure 2.2 Cooling System Description and Plant Operation 2.2.1 Components of the Cooling System 2.2.2 Sensors of the Cooling System 2.2.3 Analysis of the Operation Strategy 2.2.4 Cooling Reserve 2.3 Modeling of the Plant Using Machine Learning 2.3.1 Submodels of the Cooling System 2.3.2 Data Processing 2.3.3 Training and Plausibility 2.3.4 Recalculation of the Entire Cooling System 2.4 Optimization Concept 2.4.1 Variable Switchpoint Temperature 2.4.2 Forecast Horizon 2.4.3 Software Implementation as Assistance System 2.5 Conclusion and Outlook References 3 AITwin: A Uniform Digital Twin Interface for Artificial Intelligence Applications 3.1 Introduction 3.1.1 Related Work 3.2 ML/AI and the Digital Twin 3.3 AI Reference Model 3.3.1 Synchronized Data 3.3.2 Prediction-Enabled Models 3.3.3 Causalities 3.3.3.1 System and Product State Causalities 3.3.4 The AITwin Reference Model 3.4 Evaluation 3.4.1 Applying AITwin to a Four Tank Model 3.4.2 Applying AITwin to Tennessee Eastman Process 3.4.3 Applying AITwin to a Quality Assurance Example 3.4.4 Applying AITwin to a Sensor-Based Sorting System 3.5 Discussion and Future Work References Part II Optimization 4 A Regression-Based Predictive Model Hierarchy for Nonwoven Tensile Strength Inference 4.1 Introduction 4.1.1 Literature Overview 4.1.2 New Regression-Based Predictive Model Hierarchy 4.1.3 Structure 4.2 First Principle Oriented Model Chain for Dataset Generation 4.2.1 Fiber Graph Generation and Tensile Strength Simulation 4.2.2 Production Process Class 4.2.3 Stress-Strain Curve Class 4.2.4 Fiber Graph Features 4.2.5 Dataset 4.3 Linear Regression-Based Predictive Models 4.3.1 Linear Regression and Monte Carlo Simulations 4.3.2 Numerical Results 4.4 Sequential Predictive Regression Model 4.4.1 Coupled Polynomial Regression and Errors-In-Variabels Model 4.4.2 Numerical Results 4.5 Conclusion and Future Work References 5 Machine Learning for Optimizing the Homogeneity of Spunbond Nonwovens 5.1 Introduction 5.2 Related Work 5.3 Machine Learning-Based Optimization Workflow Using Simulation Models 5.3.1 Parameter Selection 5.3.1.1 Process Parameters 5.3.1.2 Product Quality: Homogeneity 5.3.2 Data Collection with Knowledge Integration 5.3.2.1 Sample Size Estimation for Simulation Model Setup 5.3.2.2 Influence of Discretization Step Size (ds) 5.3.2.3 Input Data Sampling 5.3.3 Model Selection 5.3.3.1 Linear Regression (LR) 5.3.3.2 Support Vector Regression (SVR) 5.3.3.3 Polynomial Regression (PR) 5.3.3.4 Bayesian Regression (BR) 5.3.3.5 Random Forests (RF) 5.3.3.6 Artificial Neural Networks (ANN) 5.3.4 Training and Testing 5.3.5 Homogeneity Optimization with Human Validation 5.4 Experiments 5.4.1 Models Evaluation Based on the Accuracy 5.4.2 Models Evaluation Based on Computational Performance 5.5 Conclusion References 6 Bayesian Inference for Fatigue Strength Estimation 6.1 Introduction 6.2 Background 6.2.1 Fatigue Testing 6.2.2 Experimental Procedure and Analysis of the Staircase Method 6.2.2.1 Experimental Procedure 6.2.2.2 Analysis of Test Results by the Staircase Method 6.2.2.3 Disadvantages of the Staircase Method 6.2.2.4 Requirements for an Alternative Experimental Approach 6.2.3 Related Work 6.3 Informed Fatigue Strength Estimation 6.3.1 Overview of Approach 6.3.2 Machine Learning Model 6.3.2.1 Gaussian Processes 6.3.2.2 Gaussian Process for Estimating Fatigue Strength 6.3.3 Bayesian Inference on the Distribution Parameters 6.3.3.1 Maximum A Posteriori Estimate 6.3.3.2 Active Learning-Inspired Acquisition Function 6.3.3.3 Stopping Criterion 6.3.4 Details on the Overall Experimental Procedure 6.4 Validation of Approach 6.5 Conclusion References 7 Incorporating Shape Knowledge into Regression Models 7.1 Introduction 7.2 Related Work 7.3 Methods 7.3.1 SIASCOR 7.3.2 ISI 7.4 Application Examples 7.4.1 Press Hardening 7.4.2 Brushing 7.4.3 Milling 7.5 Synthetic Example 7.6 Conclusion References Part III Neural Networks 8 Predicting Properties of Oxide Glasses Using Informed Neural Networks 8.1 Introduction 8.1.1 Related Work 8.1.2 Contributions 8.2 Methodology 8.2.1 Data Collection and Preparation 8.2.2 Model Setups 8.2.2.1 Blind Models 8.2.2.2 Informed Model 8.2.3 Model Training and Evaluation 8.3 Results and Discussion 8.4 Conclusion and Outlook References 9 Graph Neural Networks for Predicting Side Effects and New Indications of Drugs Using Electronic Health Records 9.1 Introduction 9.2 Methods 9.2.1 Overview About Data 9.2.2 Code Normalization and Mapping 9.2.3 Initial Knowledge Graph Construction 9.2.4 Extended Knowledge Graph Construction 9.2.4.1 Chemical Compound Similarities 9.2.4.2 Use of Diagnosis-Diagnosis Relationships 9.2.5 Relation Aware Graph Attention Networks 9.2.6 Evaluation against Alternative Methods 9.2.7 Performance Measures 9.3 Results 9.3.1 Performance Comparison 9.3.1.1 Initial Knowledge Graph 9.3.1.2 Extended Knowledge Graph 9.3.2 Use Case: Trazodone in the Treatment of Bipolar Disorder 9.3.3 Predicted Side Effects of Marketed Drugs 9.4 Discussion 9.5 Conclusion References 10 On the Interplay of Subset Selection and Informed Graph Neural Networks 10.1 Introduction 10.2 Related Work 10.3 Methods and Sampling Strategies 10.3.1 SchNet 10.3.2 Kernel Ridge Regression 10.3.3 Spatial 3-Hop Convolution Network 10.3.4 Graph Rate-Distortion Explanations 10.3.5 Sampling Strategies 10.3.5.1 Diversity 10.3.5.2 Representativeness 10.4 Numerical Experiments 10.4.1 QM9 Dataset 10.4.1.1 Knowledge Based Molecular Representation 10.4.1.2 Diverse and Representative Sets of Molecules 10.4.1.3 Sampling the QM9 Dataset 10.4.1.4 Measuring the Error 10.4.2 SchNet 10.4.3 Kernel Ridge Regression 10.4.4 Spatial 3-Hop Convolution Network 10.4.5 Explanation 10.4.5.1 Setup of the Experiments 10.4.5.2 Results 10.5 Conclusion References 11 Informed Machine Learning Aspects for the Multi-Agent Neural Rewriter 11.1 Introduction 11.2 Related Work 11.2.1 Informed Machine Learning 11.3 Multi-Agent Neural Rewriter (MANR) 11.3.1 Problem Definition 11.3.2 Game Design 11.3.3 Game Workflow 11.3.4 Game Implementation 11.3.4.1 Loss Functions 11.4 Empirical Evaluation 11.4.1 Data Generation 11.4.2 Experiment Results for the MANR 11.4.3 Transfer Learning Investigations 11.5 Conclusion References Part IV Hybrid Methods 12 Training Support Vector Machines by Solving DifferentialEquations 12.1 Introduction 12.1.1 Overview 12.1.2 Mathematical Notation 12.2 Setting the Stage 12.2.1 L2 Support Vector Machines 12.2.2 Invoking the Kernel Trick 12.2.3 A Baseline Training Algorithm 12.3 Gradient Flows for L2 SVM Training 12.4 Practical Examples 12.5 Conclusion Appendix References 13 Informed Machine Learning to Maximize Robustness and Computational Performance of Linear Solvers 13.1 Introduction 13.2 Short Overview on Linear Solvers in Numerical Simulations 13.3 Genetic Optimization of Parameters with Tree Hierarchy 13.4 Pre-evolution via Surrogate Learning Model 13.5 Online vs. Offline Training 13.6 Reproducibility 13.7 Controlling Solver Setup Reusage 13.8 Results: Informed Machine Learning for Linear Solver Parameters in Various Practical Applications 13.8.1 Mere Parameter Optimization: Single Reservoir Simulation Problems 13.8.2 Parameter Optimization: Linear Elasticity Problem 13.8.3 Setup Reusage: Sequence of Reservoir Simulation Problems 13.8.4 Full Simulation Result: Reservoir Application(SPE10) 13.8.5 Full Simulation Result: Groundwater Application 13.8.6 Full Simulation Result: Computational Fluid Dynamics Application 13.8.7 Full Simulation Result: Battery Aging Simulation 13.9 Conclusions and Future Research References 14 Anomaly Detection in Multivariate Time Series Using Uncertainty Estimation 14.1 Introduction 14.2 Background and Related Work 14.2.1 Problem Formulation and Anomaly Categorization 14.2.1.1 Point Anomalies 14.2.1.2 Context Anomalies 14.2.1.3 Collective Anomalies 14.2.2 Unsupervised Anomaly Detection 14.2.3 Bayesian Neural Networks 14.2.3.1 Epistemic Uncertainty 14.2.3.2 Aleatoric Uncertainty 14.2.3.3 Predictive Uncertainty 14.2.4 Related Work 14.3 Detecting Anomalies in Time Series Using Uncertainty Estimation 14.3.1 Window Processing and Forecast Modelling 14.3.2 Formalization of Multivariate Anomaly Detection 14.3.3 Anomaly Scoring 14.3.4 Anomaly Threshold Fitting 14.4 Experimental Setup and Evaluation 14.4.1 Skoltech Anomaly Benchmark Data Set 14.4.2 Experimental Hyperparameters 14.4.3 Evaluation Metrics 14.4.4 Discussion of Utilized Anomaly Detection Metrics 14.4.5 Experimental Results and Analysis 14.4.5.1 Anomaly Detection Analysis 14.5 Discussion of Experimental Results 14.5.1 Quantile Based Threshold Versus Tabulated Control Limits 14.5.2 Competitiveness to Recent Work 14.6 Conclusion References