Generic Multi-Agent Reinforcement Learning Approach for Flexible Job-Shop Scheduling

Description

The production control of flexible manufacturing systems is a relevant component that must go along with the requirements of being flexible in terms of new product variants, new machine skills and reaction to unforeseen events during runtime. This work focuses on developing a reactive job-shop scheduling system for flexible and re-configurable manufacturing systems. Reinforcement Learning approaches are therefore investigated for the concept of multiple agents that control products including transportation and resource allocation. Danksagung Abstract Zusammenfassung Contents Abbreviations List of Figures List of Tables 1 Introduction 1.1 Research Goals 1.2 Methodology 1.3 Structure of the Thesis 2 Requirements for Production Scheduling in Flexible Manufacturing 2.1 Foundations of Flexible Job-Shop Scheduling Problems 2.2 Requirement Analysis of Flexible Scheduling Solutions 2.2.1 Influences on Warehouse Control Systems 2.2.2 Influences on Manufacturing Control Systems 2.2.3 Derived and Ranked Requirements 2.3 State of the Art: Approaches to Solve Job-Shop Scheduling Problems 2.3.1 Conventional Scheduling Solutions 2.3.2 Reinforcement Learning Scheduling Solutions 2.4 Identification of the Research Gap 2.5 Contribution of this Work: Extended Flexible Job-Shop Scheduling Problem 3 Reinforcement Learning as an Approach for Flexible Scheduling 3.1 Understanding the Foundations: Formalization as a Markov Decision Process 3.1.1 Agent-Environment Interaction 3.1.2 Policies and Value Functions 3.1.3 Challenges Arising in Reinforcement Learning 3.2 Deep Q-Learning 3.2.1 Temporal Difference Learning and Q-Learning 3.2.2 Deep Q-Network 3.2.3 Loss Optimization 3.3 State of the Art: Cooperating Agents to Solve Complex Problems 3.3.1 Multi-Agent Learning Methods 3.3.2 Learning in Cooperative Multi-Agent RL Setups 3.4 Summary 4 Concept for Multi-Resources Flexible Job-Shop Scheduling 4.1 Concept for Agent-based Scheduling in FMS 4.1.1 Overall Concept 4.1.2 Job Specification 4.1.3 Petri Net Simulation 4.2 Formalization as a Markov Decision Process 4.2.1 Action Designs 4.2.2 State Designs 4.2.3 Reward Design 4.3 Considered Flexible Manufacturing System 4.4 Evaluation of the Technical Functionalities 4.5 Summary 5 Multi-Agent Approach for Reactive Scheduling in Flexible Manufacturing 5.1 Training Set-up 5.2 Specification of the Reward Design 5.3 Evaluation of Suitable Training Strategies 5.3.1 Evaluation of MARL Algorithms 5.3.2 Selection of MARL Learning Methods 5.3.3 Evaluation of Parameter Sharing and Centralized Learning 5.4 Training Approach to Present Situations 5.5 Summary 6 Empirical Evaluation of the Requirements 6.1 Generalization to Various Products and Machines 6.2 Achieving the Global Objective 6.2.1 Comparison of Dense and Sparse Global Rewards 6.2.2 Cooperative Behavior 6.3 Benchmarking against Heuristic Search Algorithms 6.3.1 Evaluation for Unknown and Known Situations 6.3.2 Evaluation of Real-time Decision-Making 6.4 Consolidated Requirements Evaluation 6.5 Summary 7 Integration into a Flexible Manufacturing System 7.1 Acceptance Criteria for the Integration Concept 7.2 Integration Concept of MARL Scheduling Solution 7.2.1 Integration in the MES 7.2.2 Information Exchange 7.3 Design Cycle 7.3.1 Functioning Scheduling 7.3.2 Efficient Production Flow 7.3.3 Handling of Unforeseen Events 7.3.4 Handling of New Machine Skills 7.3.5 Handling of New Machines 7.4 Summary 8 Critical Discussion and Outlook 9 Summary 1 Bibliography