Computational Optimization. Success in Practice

Description

Cover Half Title Series Page Title Page Copyright Page Dedication Contents Preface Author Acronyms and Abbreviations List of Algorithms 1. Introduction to Optimization 1.1. Optimization Models 1.2. General Notations for Optimization Problem 1.3. Data Fitting Examples 1.4. Optimization Fundamentals 1.4.1. Feasibility 1.4.2. Optimality 1.4.3. Convexity 1.5. General Optimization Algorithm 1.5.1. Solving Optimization Problems in Iterations 1.5.2. Termination Criteria 1.6. Convergence 1.7. Homework Problems 2. Minimization Approaches for Functions of One Variable 2.1. Minimizing a Function in 1D 2.2. Bisection Method 2.3. Golden Section Search 2.4. Newton's Method 2.5. Brute-Force Search 2.6. Monte Carlo Method 2.7. Practical Examples 2.8. Computational Analysis for Convergence 2.9. Homework Problems 2.10. Lab Assignment #1: Review Chapters 1–2 3. Generalized Optimization Framework 3.1. Parameter Identification for Least-Square Data Fitting 3.2. Generalized Optimization Framework 3.2.1. Computational Components 3.2.2. Choice of Proper Software 3.3. Choosing and Adjusting Optimization Algorithms 3.4. Visualization and Analysis of Obtained Solutions 3.5. Testing and Dealing with Problems (Debugging) 3.6. TEST Mode for the Gradient-based Framework 3.7. Accuracy and Performance 3.8. Communication within the Framework 3.9. Homework Problems 4. Exploring Optimization Algorithms 4.1. Iterative Optimization Algorithms Revisited 4.2. Gradient-based Strategies: Line Search vs. Trust Region 4.2.1. Line Search 4.2.2. Trust Region 4.2.3. Comparing Strategies 4.3. Heuristic Algorithms 4.4. Particle Swarm Optimization 4.5. Overview of Optimization Algorithms 4.6. Homework Problems 5. Line Search Algorithms 5.1. Local and Global Minimums: Theory Review 5.1.1. Necessary Conditions 5.1.2. Necessary vs. Sufficient Conditions 5.1.3. Existence of Global Minimums 5.2. Selecting Search Direction 5.2.1. Principal Generalization 5.2.2. Some Theory for Convergence 5.2.3. Steepest Descent 5.3. Newton's Method and Newton-based Modifications 5.3.1. Pure Newton's Method 5.3.2. Discretized Newton's Method 5.3.3. Modified Newton's Method 5.3.4. Diagonally Scaled SD 5.3.5. Quasi-Newton Methods 5.3.6. Gauss-Newton Approach 5.4. Conjugate Gradient 5.5. Line Search Performance Comparison 5.6. Homework Problems 5.7. Lab Assignment #2: Review Chapters 3–5 6. Choosing Optimal Step Size 6.1. Overview 6.2. Simple Approaches 6.3. Inexact Line Search 6.3.1. Armijo Rule 6.3.2. Wolfe Conditions 6.3.3. Goldstein Conditions 6.3.4. Backtracking Line Search 6.4. Step Size Performance Comparison 6.5. Advanced Methods for 1D Search 6.6. Brent's Method 6.7. Bracketing-Brent Toolbox in MATLAB 6.7.1. General Description 6.7.2. Initial Bracketing with fn_min_brack.m 6.7.3. Brent Minimization with fn_brent.m 6.7.4. Technicalities for MATLAB Implementation 6.8. Bracketing-Brent Toolbox Performance 6.9. Homework Problems 7. Trust Region and Derivative-Free Methods 7.1. Trust Region Outline 7.2. General Algorithm 7.3. Cauchy Point Calculation 7.4. Improving Cauchy Point by Dogleg Method 7.5. Checking and Tuning Performance 7.6. Exploring Derivative-Free Options 7.7. Homework Problems 7.8. Lab Assignment #3: Review Chapters 6–7 7.9. Midterm Assignment: Review Chapters 1–7 8. Large-Scale and Constrained Optimization 8.1. Generalization of Large-Scale Optimization 8.2. LSO Examples 8.2.1. Solving Systems of Equations 8.2.2. Space-Dependent Parameter Reconstruction 8.2.3. Parameter Identification – Another Example 8.2.4. State-Dependent Parameter Reconstruction 8.3. Improving Performance of LSO 8.4. General Theory for Constrained Optimization 8.5. Lagrange Multiplier Approach 8.6. Lagrange Multiplier vs. Penalization 8.7. Extending Complexity – DE-constrained Optimization 8.8. KKT Optimality Conditions 8.9. Homework Problems 9. ODE-based Optimization 9.1. Fitting Data by ODE-based Optimization 9.2. Derivative vs. Directional Derivative: Review 9.3. Optimize–then–Discretize 9.3.1. Deriving Gradient 9.3.2. Solving Problem 9.4. Discretize–then–Optimize 9.5. Numerical ODE Solvers 9.5.1. Runge–Kutta Integration Methods 9.5.2. Solving ODEs by MATLAB 9.6. Dynamics of Biological Systems by Lotka–Volterra Equations 9.7. Optimization Problem Constrained by LV Model 9.7.1. Statement of Problem 9.7.2. Deriving Gradients 9.7.3. Optimization Algorithm 9.8. Optimization Framework in MATLAB 9.8.1. Benchmark Models for Controls 9.8.2. Analytic vs. Synthetic Measurements 9.8.3. Adjusting Framework and Choosing Parameters 9.8.4. Checking and Improving Quality of Discretized Gradients 9.8.5. Optimization Results 9.9. Homework Problems 10. Implementing Regularization Techniques 10.1. Motivation for Regularization 10.2. Some Regularization Theory 10.2.1. Quadratic Penalty Function Method 10.2.2. Barrier Functions Method 10.2.3. Tikhonov-type Regularization 10.2.4. Gradient Preconditioning 10.2.5. Bounds by Simple Projections and Slack Variables 10.3. Examples of Numerical Implementation 10.3.1. From Theory to Practice 10.3.2. Adjusting Framework 10.3.3. Results and Food for Thought 10.4. Homework Problems 10.5. Lab Assignment #4: Review Chapters 8–10 11. Moving to PDE-based Optimization 11.1. Generalized Problem of Fitting Data 11.2. Practice Example 11.2.1. Elliptic Equation as Governing PDE 11.2.2. Deriving Gradient by Optimize–then–Discretize 11.2.3. Optimization Algorithm 11.3. Solving PDEs in Higher Dimensions by FreeFEM 11.4. Brief Introduction to Finite Elements Method 11.5. Solving PDEs by FreeFEM 11.5.1. Poisson's Equation in 2D 11.5.2. Coding with FreeFEM 11.5.3. Solution Analysis 11.5.4. Technicalities for FreeFEM Coding 11.6. Homework Problems 12. Sharing Multiple Software Environments 12.1. Practice Example and Benchmark Model 12.2. Updating and Tuning Optimization Framework 12.2.1. Overview of Changes 12.2.2. Software Communication 12.2.3. Another Round on Measurements 12.2.4. Evaluating Objectives 12.2.5. Evaluating Gradients 12.2.6. Checking Quality of Discretized Gradients 12.3. Analyzing Optimization Results 12.4. Homework Problems 12.5. Lab Assignment #5: Review Chapters 11–12 Appendix: Review of Math with MATLAB Bibliography Index