Computational Intelligence: A Methodological Introduction

Description

Preface Contents 1 Introduction 1.1 Intelligent Systems 1.2 Computational Intelligence 1.3 About the Third Edition of this Book Part I Neural Networks 2 Introduction to Artificial Neural Networks 2.1 Motivation 2.2 Biological Background 3 Threshold Logic Units 3.1 Definition and Examples 3.2 Geometric Interpretation 3.3 Limitations 3.4 Networks of Threshold Logic Units 3.5 Training the Parameters 3.6 Variants 3.7 Training Networks 4 General Neural Networks 4.1 Structure of Neural Networks 4.2 Operation of Neural Networks 4.3 Scale Types and Encoding 4.4 Training Neural Networks 5 Multi-layer Perceptrons 5.1 Definition and Examples 5.2 Why Non-linear Activation Functions? 5.3 Function Approximation 5.4 Logistic Regression 5.5 Gradient Descent 5.6 Error Backpropagation 5.7 Gradient Descent Examples 5.8 Variants of Gradient Descent 5.8.1 Manhattan Training 5.8.2 Momentum Term 5.8.3 Nesterov's Accelerated Gradient (NAG) 5.8.4 Self-adaptive Error Backpropagation (SuperSAB) 5.8.5 Resilient Backpropagation 5.8.6 Quick Propagation 5.8.7 Adaptive Subgradient Descent (AdaGrad) 5.8.8 Root Mean Squared Gradient Descent (RMSProp) 5.8.9 Adaptive Subgradient Descent Over Windows (AdaDelta) 5.8.10 Adaptive Moment Estimation (Adam) 5.8.11 Lifting the Derivative of the Activation Function 5.8.12 Weight Decay 5.8.13 Batch Normalization 5.9 Examples for Some Variants 5.10 Initializing the Parameters 5.11 Number of Hidden Neurons 5.12 Deep Learning 5.13 Sensitivity Analysis 6 Radial Basis Function Networks 6.1 Definition and Examples 6.2 Function Approximation 6.3 Initializing the Parameters 6.4 Training the Parameters 6.5 Example of Training 6.6 Generalized Form 7 Self-organizing Maps 7.1 Definition and Examples 7.2 Learning Vector Quantization 7.3 Neighborhood of the Output Neurons 8 Hopfield Networks 8.1 Definition and Examples 8.2 Convergence of the Computations 8.3 Associative Memory 8.4 Solving Optimization Problems 8.5 Simulated Annealing 8.6 Boltzmann Machines 9 Recurrent Networks 9.1 Simple Examples 9.2 Representing Differential Equations 9.3 Vectorial Neural Networks 9.4 Error Backpropagation Through Time 9.5 Long Short-Term Memory 10 Neural Networks: Mathematical Remarks 10.1 Equations for Straight Lines 10.2 Regression 10.3 Activation Transformation Part II Evolutionary Algorithms 11 Introduction to Evolutionary Algorithms 11.1 Metaheuristics 11.2 Biological Evolution 11.3 Simulated Evolution 11.3.1 Optimization Problems 11.3.2 Basic Notions and Concepts 11.3.3 Building Blocks of an Evolutionary Algorithm 11.4 The n-Queens Problem 11.5 Related Optimization Techniques 11.5.1 Gradient Ascent or Descent 11.5.2 Hill Climbing 11.5.3 Simulated Annealing 11.5.4 Threshold Accepting 11.5.5 Great Deluge Algorithm 11.5.6 Record-to-Record Travel 11.6 The Traveling Salesman Problem 12 Elements of Evolutionary Algorithms 12.1 Encoding of Solution Candidates 12.1.1 Hamming Cliffs 12.1.2 Epistasis 12.1.3 Closedness of the Search Space 12.2 Fitness and Selection 12.2.1 Fitness Proportionate Selection 12.2.2 The Dominance Problem 12.2.3 Vanishing Selective Pressure 12.2.4 Adapting the Fitness Function 12.2.5 The Variance Problem 12.2.6 Rank-Based Selection 12.2.7 Tournament Selection 12.2.8 Elitism 12.2.9 Environmental Selection 12.2.10 Niche Techniques 12.2.11 Characterization of Selection Methods 12.3 Genetic Operators 12.3.1 Mutation Operators 12.3.2 Crossover Operators 12.3.3 Multi-parent Operators 12.3.4 Characteristics of Recombination Operators 12.3.5 Interpolating and Extrapolating Recombination 13 Fundamental Evolutionary Algorithms 13.1 Genetic Algorithms 13.1.1 The Schema Theorem 13.1.2 The Two-Armed Bandit Argument 13.1.3 The Principle of Minimal Alphabets 13.2 Evolution Strategies 13.2.1 Selection 13.2.2 Global Variance Adaptation 13.2.3 Local Variance Adaptation 13.2.4 Covariances 13.2.5 Recombination Operators 13.3 Genetic Programming 13.3.1 Initialization 13.3.2 Genetic Operators 13.3.3 Application Examples 13.3.4 The Problem of Introns 13.3.5 Extensions 13.4 Multi-objective Optimization 13.4.1 Weighted Combination of Objectives 13.4.2 Pareto-Optimal Solutions 13.4.3 Finding Pareto Frontiers with Evolutionary Algorithms 13.5 Special Applications and Techniques 13.5.1 Behavioral Simulation 13.5.2 Parallelization 14 Computational Swarm Intelligence 14.1 Introduction 14.2 Basic Principles of Computational Swarm Intelligence 14.2.1 Swarms in Known Environments 14.2.2 Swarms in Unknown Environments 14.3 Particle Swarm Optimization 14.3.1 Influence of the Parameters 14.3.2 Turbulence Factor 14.3.3 Boundary Handling 14.4 Multi-objective Particle Swarm Optimization 14.4.1 Leader Selection Mechanism 14.4.2 Archiving 14.5 Many-objective Particle Swarm Optimization 14.5.1 Ranking Non-dominated Solutions 14.5.2 Distance Based Ranking 14.6 Ant Colony Optimization Part III Fuzzy Systems 15 Introduction to Fuzzy Sets and Fuzzy Logics 15.1 Natural Languages and Formal Models 15.2 Fuzzy Sets 15.3 Representation of Fuzzy Sets 15.3.1 Definition Based on Functions 15.3.2 α-Cuts 15.4 Fuzzy Logic 15.4.1 Propositions and Truth Values 15.4.2 t-Norms and t-Conorms 15.4.3 Aggregation Functions 15.5 Semantics of Membership Degrees 15.5.1 Membership Degrees as Truth Degrees 15.5.2 Membership Degrees as Similarity to a Reference Value 15.5.3 Membership Degrees as Preferences 15.5.4 Membership Degrees as Possibility 15.5.5 Consistent Interpretations of Fuzzy Sets in Applications 15.6 Operations on Fuzzy Sets 15.6.1 Intersection 15.6.2 Union 15.6.3 Complement 15.6.4 Covering and Partition 15.6.5 Linguistic Modifiers 15.7 Fuzzy Sets of Type 2 16 The Extension Principle 16.1 Mappings of Fuzzy Sets 16.2 Mappings of a-cuts 16.3 Cartesian Product and Cylindrical Extension 16.4 Extension Principle for Multivariate Mappings 17 Fuzzy Relations 17.1 Crisp Relations 17.2 Application of Relations 17.3 Logical Deduction with Relations 17.4 Simple Fuzzy Relations 17.5 Composition of Fuzzy Relations 17.6 Fuzzy Relational Equations 18 Similarity Relations 18.1 Similarity 18.2 Fuzzy Sets and Extensional Hulls 18.3 Scaling Concepts 18.4 Fuzzy Sets and Similarity Relations 19 Approximate Reasoning 19.1 Linguistic Variables 19.2 Computing with Words 19.3 Generalized Logical Inference 19.4 Approximation of Functions Using Linguistic If-Then Rules 19.4.1 Approximation of Functions by Using Rules as Constraints 19.4.2 Approximation of Functions by Solving Fuzzy Relational Equations 19.4.3 Approximation of Functions by Interpolation Between Fuzzy Points 20 Fuzzy Control 20.1 Mamdani Fuzzy Controller 20.2 Design of a Mamdani Fuzzy Controller 20.3 Mamdani Controller and Similarity Relations 20.3.1 Interpretation of a Mamdani Controller Using Similarity Relations 20.3.2 Construction of a Mamdani Controller Using Similarity Relations 20.4 Takagi–Sugeno Controller 21 Hybrid Systems for Tuning and Learning Fuzzy Systems 21.1 Neuro-Fuzzy Control 21.1.1 Models with Supervised Learning Methods 21.1.2 Models with Reinforcement Learning 21.2 Evolutionary Fuzzy Control 21.2.1 Structure of an Evolutionary Fuzzy Controller 21.2.2 Optimizing Parameters of an Evolutionary Fuzzy Controller 21.2.3 Example 22 Fuzzy Data Analysis 22.1 Fuzzy Methods in Data Analysis 22.2 Fuzzy Clustering 22.2.1 Clustering 22.2.2 Presuppositions and Notation 22.2.3 Classical c-Means Clustering 22.2.4 Fuzzification by Membership Transformation 22.2.5 Fuzzification by Membership Regularization 22.2.6 Comparison 22.3 Analysis of Precise Data with Possibility Theory 22.4 Analysis of Imprecise Data Using Random Sets 22.5 Analysis of Fuzzy Data with Fuzzy Random Variables Part IV Bayes and Markov Networks 23 Bayesian Networks 24 Elements of Probability and Graph Theory 24.1 Probability Theory 24.1.1 Random Variables and Random Vectors 24.1.2 Independences 24.2 Graph Theory 24.2.1 Background 24.2.2 Join Graphs 24.2.3 Separations 25 Decompositions 26 Evidence Propagation 26.1 Initialization 26.2 Message Passing 26.3 Update 26.4 Marginalization 27 Learning Graphical Models 27.1 Score-Based Approaches 27.1.1 Likelihood of a Database 27.1.2 K2 Algorithm 27.2 Constraint-Based Approaches 28 Belief Revision 28.1 Introduction 28.2 Revision Procedure 28.3 A Real-World Application 29 Decision Graphs 29.1 Motivation 29.2 Definition 29.3 Policies and Strategies 29.4 Finding Optimal Strategies 29.5 Example Scenario 30 Causal Networks 30.1 Causal and Probabilistic Structure 30.2 The Do Operator Index