Big Data Analytics in Cybersecurity and IT Management

Description

Big data analytics provide more accurate, timely, and actionable decisions for both cybersecurity and IT management. This book gives a comprehensive coverage of state-of-the-art big data analytics in cybersecurity and IT management. The topics include threat analysis, vulnerability identification, mission analysis, network monitoring, network management, visualization, and cybertraining. Each topic is studied in detail in a case study. Edited by Dr. Onur Savas and Dr. Julia Deng, and written by leading experts from industry, academia, and government, this book will be of tremendous help for both the starters and the more experienced. Cover Half Title Title Page Copyright Page Table of Contents Preface About the Editors Contributors Section I : Applying Big Data into Different Cybersecurity Aspects Chapter 1: The Power of Big Data in Cybersecurity 1.1 Introduction to Big Data Analytics 1.1.1 What Is Big Data Analytics? 1.1.2 Differences between Traditional Analytics and Big Data Analytics 1.1.2.1 Distributed Storage 1.1.2.2 Support for Unstructured Data 1.1.2.3 Fast Data Processing 1.1.3 Big Data Ecosystem 1.2 The Need for Big Data Analytics in Cybersecurity 1.2.1 Limitations of Traditional Security Mechanisms 1.2.2 The Evolving Threat Landscape Requires New Security Approaches 1.2.3 Big Data Analytics Offers New Opportunities to Cybersecurity 1.3 Applying Big Data Analytics in Cybersecurity 1.3.1 The Category of Current Solutions 1.3.2 Big Data Security Analytics Architecture 1.3.3 Use Cases 1.3.3.1 Data Retention/Access 1.3.3.2 Context Enrichment 1.3.3.3 Anomaly Detection 1.4 Challenges to Big Data Analytics for Cybersecurity References Chapter 2: Big Data for Network Forensics 2.1 Introduction to Network Forensics 2.2 Network Forensics: Terms and Process 2.2.1 Terms 2.2.2 Network Forensics Process 2.2.2.1 Phase 1: Data Collection 2.2.2.2 Phase 2: Data Examination 2.2.2.3 Phase 3: Data Analysis 2.2.2.4 Phase 4: Visualization and Reporting 2.3 Network Forensics: Current Practice 2.3.1 Data Sources for Network Forensics 2.3.2 Most Popular Network Forensic Tools 2.3.2.1 Packet Capture Tools 2.3.2.2 Flow Capture and Analysis Tools 2.3.2.3 Intrusion Detection System 2.3.2.4 Network Monitoring and Management Tools 2.3.2.5 Limitations of Traditional Technologies 2.4 Applying Big Data Analysis for Network Forensics 2.4.1 Available Big Data Software Tools 2.4.1.1 Programming Model: MapReduce [53] 2.4.1.2 Compute Engine: Spark [54], Hadoop [55] 2.4.1.3 Resource Manager: Yarn [56], Mesos [57] 2.4.1.4 Stream Processing: Storm [58], Spark Streaming [54], Apache Flink [59], Beam [60] 2.4.1.5 Real-Time In-Memory Processing: Apache Ignite [61], Hazelcast [62] 2.4.1.6 Fast SQL Analytics (OLAP): Apache Drill [63], Kylin [64] 2.4.1.7 NOSQL (Non-Relational) Databases: HBase [65], Accumulo [66], MongoDB [67], Cassandra [68], Voldmort [69] 2.4.1.8 NOSQL Query Engine: Pheonix [70], Pig [71] 2.4.2 Design Considerations 2.4.2.1 NOSQL Databases 2.4.2.2 Computing Frameworks 2.4.3 State-of-the-Art Big Data Based Cyber Analysis Solutions 2.4.3.1 Cisco OpenSOC [78,79] 2.4.3.2 Sqrrl Enterprise [80] 2.5 Our Experiences 2.5.1 Software Architecture 2.5.2 Services Components 2.5.2.1 Data Processing Using Accumulo 2.5.2.2 Log Service (Message System) Using Kafka 2.5.2.3 Stream Processing Engine Using Storm 2.5.3 Key Features 2.6 Summary References Chapter 3: Dynamic Analytics-Driven Assessment of Vulnerabilities and Exploitation 3.1 Introduction 3.1.1 The Need and Challenges 3.1.2 The Objective and Approach of This Chapter 3.2 Vulnerability Assessment, Attribution, and Exploitation 3.2.1 Vulnerability Assessment 3.2.2 Use Case: Identification and Attribution of Vulnerability Exploitation 3.3 State-of-the-Art Vulnerability Assessment Tools, Data Sources, and Analytics 3.3.1 Vulnerability Assessment Tools 3.3.2 Data Sources, Assessment, and Parsing Methods 3.4 Secure Management of Cyber Events Involved with Vulnerability and Exploitation 3.4.1 Comparison of Current SIEM Tools 3.4.1.1 Open Source SIEM Tools 3.4.1.2 Traditional SIEM Tool 3.4.1.3 Non-Traditional SIEM Tool 3.4.2 Temporal Causality Analysis for Enhancing Management of Cyber Events 3.5 Summary and Future Directions References Chapter 4: Root Cause Analysis for Cybersecurity 4.1 Introduction 4.2 Root Cause Analysis and Attack Attribution 4.3 The Causal Analysis of Security Threats 4.3.1 Challenges in Detecting Security Incidents 4.3.2 Root Cause Analysis for Security Data Mining 4.3.2.1 Feature Selection for Security Events 4.3.2.2 Graph-Based Clustering 4.3.2.3 MCDA-Based Attack Attribution 4.4 Case Studies 4.4.1 Attack Attribution via Multi-Criteria Decision Making 4.4.1.1 Defining Attack Characteristics 4.4.1.2 Extracting Cliques of Attackers 4.4.1.3 Multi-Criteria Decision Making 4.4.2 Large-Scale Log Analysis for Detecting Suspicious Activity 4.4.2.1 Defining Attack Characteristics 4.4.2.2 Discovering Outliers in the Network 4.5 Conclusion References Chapter 5: Data Visualization for Cybersecurity 5.1 Introduction 5.2 Threat Identification, Analysis, and Mitigation 5.3 Vulnerability Management 5.4 Forensics 5.5 Traffic 5.6 Emerging Themes References Chapter 6: Cybersecurity Training 6.1 Specific Characteristics of Training Cybersecurity 6.2 General Principles of Training and Learning 6.2.1 Desired Result of Training: Better Performance 6.2.2 Use of Media in Training 6.2.3 Context in which to Present General Learning Principles 6.2.4 Learning with Understanding 6.2.5 Reflection and Interactions 6.2.6 Immersive Environments of Simulations and Games 6.2.7 Building on What Learners Know 6.2.8 Metacognition 6.2.9 Teamwork 6.2.10 Feedback 6.2.11 Motivation 6.2.12 Transfer 6.2.13 Misconceptions 6.3 Practical Design 6.3.1 Sponsor’s Expectations 6.3.2 Available Resources 6.3.3 Subject Matter Experts and Cognitive Task Analysis 6.3.4 Identify What Trainees Need To Learn 6.3.5 The Underlying Representation That Supports Computerized Assessment and Instruction 6.3.6 Pilot Test the Instruction 6.4 Putting it All Together 6.5 Using Big Data to Inform Cybersecurity Training 6.6 Conclusions References Chapter 7: Machine Unlearning: Repairing Learning Models in Adversarial Environments 7.1 Introduction 7.1.1 The Need for Systems to Forget 7.1.2 Machine Unlearning 7.1.3 Chapter Organization 7.2 Background and Adversarial Model 7.2.1 Machine Learning Background 7.2.2 Adversarial Model 7.2.2.1 System Inference Attacks 7.2.2.2 Training Data Pollution Attacks 7.3 Overview 7.3.1 Unlearning Goals 7.3.1.1 Completeness 7.3.1.2 Timeliness 7.3.2 Unlearning Work Flow 7.4 Unlearning Approach 7.4.1 Nonadaptive SQ Learning 7.4.2 Adaptive SQ Learning 7.5 Unlearning in LensKit 7.5.1 The Attack–System Inference 7.5.2 Analytical Results 7.5.3 Empirical Results 7.6 Related Work 7.6.1 Adversarial Machine Learning 7.6.1.1 Causative Attacks 7.6.1.2 Exploratory Attacks 7.6.2 Defense of Data Pollution and Privacy Leakage 7.6.2.1 Defense of Data Pollution 7.6.2.2 Defense of Privacy Leaks 7.6.3 Incremental Machine Learning Further Reading References Section II : Big Data in Emerging Cybersecurity Domains Chapter 8: Big Data Analytics for Mobile App Security 8.1 Introduction to Mobile App Security Analysis 8.2 Applying Machine Learning (ML) in Triaging App Security Analysis 8.3 The State-of-the-Art ML Approaches for Android Malware Detection 8.4 Challenges in Applying ML for Android Malware Detection 8.4.1 Challenges in Ensuring Proper Evaluation 8.4.2 Challenges in the Algorithm Design 8.4.3 Challenges in Data Collection 8.4.4 Insights Based on Our Own Study 8.5 Recommendations 8.5.1 Data Preparation and Labeling 8.5.2 Learning from Large Data 8.5.3 Imbalanced Data 8.5.4 Expensive Features 8.5.5 Leveraging Static Analysis in Feature Selection 8.5.6 Understanding the Results 8.6 Summary References Chapter 9: Security, Privacy, and Trust in Cloud Computing 9.1 Introduction to Cloud 9.1.1 Deployment Models 9.1.2 Service Models 9.1.3 Distinct Characteristics 9.1.4 Key Technologies 9.2 Security, Privacy, and Trust Challenges in Cloud Computing 9.2.1 Security Attacks against Multi-Tenancy 9.2.2 Security Attacks against Virtualization 9.2.3 Data Security and Privacy in Cloud 9.2.4 Lack of Trust among Multiple Stakeholders in Cloud 9.3 Security, Privacy, and Trust Solutions in Cloud Computing 9.3.1 Logging and Monitoring 9.3.2 Access Control 9.3.3 Encryption-Based Security Solutions 9.3.4 Virtual Isolation 9.3.5 Defense against Co-Resident Attacks 9.3.6 Establishing Trust in Cloud Computing 9.4 Future Directions 9.5 Conclusion References Chapter 10: Cybersecurity in Internet of Things (IoT) 10.1 Introduction 10.2 IoT and Big Data 10.3 Security Requirement and Issues 10.3.1 Heterogeneous Big Data Security and Management 10.3.2 Lightweight Cryptography 10.3.3 Universal Security Infrastructure 10.3.4 Trust Management 10.3.5 Key Management 10.3.6 Privacy Preservation 10.3.6.1 Identity Privacy 10.3.6.2 Location Privacy 10.3.6.3 Profiling Privacy 10.3.6.4 Linkage Privacy 10.3.6.5 Interaction Privacy 10.3.7 Transparency 10.3.8 Fraud Protection 10.3.8.1 Ad Fraud 10.3.8.2 ATM Fraud 10.3.8.3 NTL Fraud 10.3.9 Identity Management 10.3.9.1 Identity and Address 10.3.9.2 Identity and Ownership 10.3.9.3 Identity and Domain 10.3.9.4 Identity and Lifecycle 10.4 Big Data Analytics for Cybersecurity in IoT 10.4.1 Single Big Dataset Security Analysis 10.4.2 Big Amount of Datasets Security Analysis 10.4.3 Big Heterogeneous Security Data 10.4.3.1 Heterogeneous Input Data 10.4.3.2 Heterogeneous Output Data 10.4.4 Information Correlation and Data Fusion 10.4.5 Dynamic Security Feature Selection 10.4.6 Cross-Boundary Intelligence 10.5 Conclusion References Chapter 11: Big Data Analytics for Security in Fog Computing 11.1 Introduction 11.2 Background of Fog Computing 11.2.1 Definitions 11.2.2 Features 11.2.3 Architectures and Existing Implementations 11.2.4 The State-of-the-Art of Data Analytics in Fog Computing 11.3 When Big Data Meets Fog Computing 11.4 Big Data Analytics for Fog Computing Security 11.4.1 Trust Management 11.4.2 Identity and Access Management 11.4.3 Availability Management 11.4.4 Security Information and Event Management 11.4.5 Data Protection 11.5 Conclusion References Chapter 12: Analyzing Deviant Socio-Technical Behaviors Using Social Network Analysis and Cyber Forensics-Based Methodologies 12.1 Introduction 12.2 Literature Review 12.3 Methodology 12.4 Case Studies 12.4.1 DAESH or ISIS/ISIL Case Study: Motivation, Evolution, and Findings 12.4.1.1 Exploring the Network of the Top Disseminators of ISIL 12.4.1.2 Beheading of Innocent People by ISIL 12.4.2 Novorossiya Case Study: Motivation, Evolution, and Findings 12.5 Conclusion and Future Work Acknowledgments References Section III : Tools and Datasets for Cybersecurity Chapter 13: Security Tools 13.1 Introduction 13.2 Defining Areas of Personal Cybersecurity 13.3 Tools for Big Data Analytics 13.4 Boundary Tools 13.4.1 Firewalls 13.4.1.1 ISP Firewalls 13.4.1.2 Home Firewalls 13.4.1.3 Free Software Firewalls 13.4.2 Antivirus 13.4.3 Content Filtering 13.5 Network Monitoring Tools 13.6 Memory Protection Tools 13.7 Memory Forensics Tools 13.8 Password Management 13.9 Conclusion Chapter 14: Data and Research Initiatives for Cybersecurity Analysis 14.1 Cybersecurity Data Sources 14.1.1 Datasets from the Operating System 14.1.2 Datasets from Network Traffic 14.1.3 Datasets from the Application Layer 14.2 Benchmark Datasets 14.2.1 DARPA KDD Cup Dataset 14.2.1.1 Website 14.2.1.2 Short Description 14.2.2 CDX 2009 Dataset 14.2.2.1 Website 14.2.2.2 Short Description 14.2.3 UNB ISCX 2012 14.2.3.1 Website 14.2.3.2 Short Description 14.3 Research Repositories and Data Collection Sites 14.3.1 IMPACT (The Information Marketplace for Policy and Analysis of Cyber-Risk & Trust) 14.3.1.1 Website 14.3.1.2 Short Description 14.3.1.3 Example Datasets 14.3.2 CAIDA (The Center for Applied Internet Data Analysis) 14.3.2.1 Website 14.3.2.2 Short Description 14.3.2.3 Example Datasets 14.3.3 Publicly Available PCAP Repository Collections—netresec.com 14.3.3.1 Website 14.3.3.2 Short Description 14.3.3.3 Example Datasets 14.3.4 Publicly Available Repository Collections—SecRepo.com 14.3.4.1 Website 14.3.4.2 Short Description 14.3.4.3 Example Datasets 14.4 Future Directions on Data Sharing References Index