Multimedia Forensics

Description

Media forensics has never been more relevant to societal life. Not only media content represents an ever-increasing share of the data traveling on the net and the preferred communications means for most users, it has also become integral part of most innovative applications in the digital information ecosystem that serves various sectors of society, from the entertainment, to journalism, to politics. Undoubtedly, the advances in deep learning and computational imaging contributed significantly to this outcome. The underlying technologies that drive this trend, however, also pose a profound challenge in establishing trust in what we see, hear, and read, and make media content the preferred target of malicious attacks. In this new threat landscape powered by innovative imaging technologies and sophisticated tools, based on autoencoders and generative adversarial networks, this book fills an important gap. It presents a comprehensive review of state-of-the-art forensics capabilities that relate to media attribution, integrity and authenticity verification, and counter forensics. Its content is developed to provide practitioners, researchers, photo and video enthusiasts, and students a holistic view of the field. Preface Contents Symbols Notation Part I Present and Challenges 1 What's in This Book and Why? 1.1 Introduction 1.2 Overviews 2 Media Forensics in the Age of Disinformation 2.1 Media and the Human Experience 2.2 The Threat to Democracy 2.3 New Technologies, New Threats 2.3.1 End-to-End Trainable Speech Synthesis 2.3.2 GAN-Based Codecs for Still and Moving Pictures 2.3.3 Improvements in Image Manipulation 2.3.4 Trillion-Param Models 2.3.5 Lottery Tickets and Compression in Generative Models 2.4 New Developments in the Private Sector 2.4.1 Image and Video 2.4.2 Language Models 2.5 Threats in the Wild 2.5.1 User-Generated Manipulations 2.5.2 Corporate Manipulation Services 2.5.3 Nation State Manipulation Examples 2.5.4 Use of AI Techniques for Deception 2019–2020 2.6 Threat Models 2.6.1 Carnegie Mellon BEND Framework 2.6.2 The ABC Framework 2.6.3 The AMITT Framework 2.6.4 The SCOTCH Framework 2.6.5 Deception Model Effects 2.6.6 4Ds 2.6.7 Advanced Persistent Manipulators 2.6.8 Scenarios for Financial Harm 2.7 Investments in Countering False Media 2.7.1 DARPA SEMAFOR 2.7.2 The Partnership on AI Steering Committee on Media Integrity Working Group 2.7.3 JPEG Committee 2.7.4 Content Authenticity Initiative (CAI) 2.7.5 Media Review 2.8 Excerpts on Susceptibility and Resilience to Media Manipulation 2.8.1 Susceptibility and Resilience 2.8.2 Case Studies: Threats and Actors 2.8.3 Dynamics of Exploitative Activities 2.8.4 Meta-Review 2.9 Conclusion References 3 Computational Imaging 3.1 Introduction to Computational Imaging 3.2 Automation of Geometrically Correct Synthetic Blur 3.2.1 Primary Cue: Image Noise 3.2.2 Additional Photo Forensic Cues 3.2.3 Focus Manipulation Detection 3.2.4 Portrait Mode Detection Experiments 3.2.5 Conclusions on Detecting Geometrically Correct Synthetic Blur 3.3 Differences Between Optical and Digital Blur 3.3.1 Authentically Blurred Edges 3.3.2 Authentic Sharp Edge 3.3.3 Forged Blurred Edge 3.3.4 Forged Sharp Edge 3.3.5 Distinguishing IGHs of the Edge Types 3.3.6 Classifying IGHs 3.3.7 Splicing Logo Dataset 3.3.8 Experiments Differentiating Optical and Digital Blur 3.3.9 Conclusions: Differentiating Optical and Digital Blur 3.4 Additional Forensic Challenges from Computational Cameras References Part II Attribution 4 Sensor Fingerprints: Camera Identification and Beyond 4.1 Introduction 4.2 Sensor Noise Fingerprints 4.3 Camera Identification 4.4 Sensor Misalignment 4.5 Image Manipulation Localization 4.6 Counter-Forensics 4.7 Camera Fingerprints and Deep Learning 4.8 Public Datasets 4.9 Concluding Remarks References 5 Source Camera Attribution from Videos 5.1 Introduction 5.2 Challenges in Attributing Videos 5.3 Attribution of Downsized Media 5.3.1 The Effect of In-Camera Downsizing on PRNU 5.3.2 Media with Mismatching Resolutions 5.4 Mitigation of Video Coding Artifacts 5.4.1 Video Coding from Attribution Perspective 5.4.2 Compensation of Loop Filtering 5.4.3 Coping with Quantization-Related Weakening of PRNU 5.5 Tackling Digital Stabilization 5.5.1 Inverting Frame Level Stabilization Transformations 5.5.2 Inverting Spatially Variant Stabilization Transformations 5.6 Datasets 5.7 Conclusions and Outlook References 6 Camera Identification at Large Scale 6.1 Introduction 6.2 Naive Methods 6.2.1 Linear Search 6.2.2 Sequential Trimming 6.3 Efficient Pairwise Correlation 6.3.1 Search over Fingerprint Digests 6.3.2 Pixel Quantization 6.3.3 Downsizing 6.3.4 Dimension Reduction Using PCA and LDA 6.3.5 PRNU Compression via Random Projection 6.3.6 Preprocessing, Quantization, Coding 6.4 Decreasing the Number of Comparisons 6.4.1 Clustering by Cameras 6.4.2 Composite Fingerprints 6.5 Hybrid Methods 6.5.1 Search over Composite-Digest Search Tree 6.5.2 Search over Full Digest Search Tree 6.6 Conclusion References 7 Source Camera Model Identification 7.1 Introduction 7.1.1 Image Acquisition Pipeline 7.1.2 Problem Formulation 7.2 Model-Based Approaches 7.2.1 Color Filter Array (CFA) 7.2.2 Lens Effects 7.2.3 Other Processing and Defects 7.3 Data-Driven Approaches 7.3.1 Hand-Crafted Features 7.3.2 Learned Features 7.4 Datasets and Benchmarks 7.4.1 Template Dataset 7.4.2 State-of-the-art Datasets 7.4.3 Benchmark Protocol 7.5 Case Studies 7.5.1 Experimental Setup 7.5.2 Comparison of Closed-Set Methods 7.5.3 Comparison of Open-Set Methods 7.6 Conclusions and Outlook References 8 GAN Fingerprints in Face Image Synthesis 8.1 Introduction 8.2 Related Work 8.2.1 Generative Adversarial Networks 8.2.2 GAN Detection Techniques 8.3 GAN Fingerprint Removal: GANprintR 8.4 Databases 8.4.1 Real Face Images 8.4.2 Synthetic Face Images 8.5 Experimental Setup 8.5.1 Pre-processing 8.5.2 Facial Manipulation Detection Systems 8.5.3 Protocol 8.6 Experimental Results 8.6.1 Controlled Scenarios 8.6.2 In-the-Wild Scenarios 8.6.3 GAN-Fingerprint Removal 8.6.4 Impact of GANprintR on Other Fake Detectors 8.7 Conclusions and Outlook References Part III Integrity and Authenticity 9 Physical Integrity 9.1 Introduction 9.1.1 Journalistic Fact Checking 9.1.2 Physics-Based Methods in Multimedia Forensics 9.1.3 Outline of This Chapter 9.2 Physics-Based Models for Forensic Analysis 9.2.1 Geometry and Optics 9.2.2 Photometry and Reflectance 9.3 Algorithms for Physics-Based Forensic Analysis 9.3.1 Principal Points and Homographies 9.3.2 Photometric Methods 9.3.3 Point Light Sources and Line Constraints in the Projective Space 9.4 Discussion and Outlook 9.5 Picture Credits References 10 Power Signature for Multimedia Forensics 10.1 Electric Network Frequency (ENF): An Environmental Signature for Multimedia Recordings 10.2 Technical Foundations of ENF-Based Forensics 10.2.1 Reference Signal Acquisition 10.2.2 ENF Signal Estimation 10.2.3 Higher Order Harmonics for ENF Estimation 10.3 ENF Characteristics and Embedding Conditions 10.3.1 Establishing Presence of ENF Traces 10.3.2 Modeling ENF Behavior 10.4 ENF Traces in the Visual Track 10.4.1 Mechanism of ENF Embedding in Videos and Images 10.4.2 ENF Extraction from the Visual Track 10.4.3 ENF Extraction from a Single Image 10.5 Key Applications in Forensics and Security 10.5.1 Joint Time–Location Authentication 10.5.2 Integrity Authentication 10.5.3 ENF-Based Localization 10.5.4 ENF-Based Camera Forensics 10.6 Anti-Forensics and Countermeasures 10.6.1 Anti-Forensics and Detection of Anti-Forensics 10.6.2 Game-Theoretic Analysis on ENF-Based Forensics 10.7 Applications Beyond Forensics and Security 10.7.1 Multimedia Synchronization 10.7.2 Time-Stamping Historical Recordings 10.7.3 Audio Restoration 10.8 Conclusions and Outlook References 11 Data-Driven Digital Integrity Verification 11.1 Introduction 11.2 Forensics Clues 11.2.1 Camera-Based Artifacts 11.2.2 JPEG Artifacts 11.2.3 Editing Artifacts 11.3 Localization Versus Detection 11.3.1 Patch-Based Localization 11.3.2 Image-Based Localization 11.3.3 Detection 11.4 Architectural Solutions 11.4.1 Constrained Networks 11.4.2 Two-Branch Networks 11.4.3 Fully Convolutional Networks 11.4.4 Siamese Networks 11.5 Datasets 11.6 Major Challenges 11.7 Conclusions and Future Directions References 12 DeepFake Detection 12.1 Introduction 12.2 DeepFake Video Generation 12.3 Current DeepFake Detection Methods 12.3.1 General Principles 12.3.2 Categorization Based on Methodology 12.3.3 Categorization Based on Input Types 12.3.4 Categorization Based on Output Types 12.3.5 The DeepFake-o-Meter Platform 12.3.6 Datasets 12.3.7 Challenges 12.4 Future Directions 12.5 Conclusion and Outlook References 13 Video Frame Deletion and Duplication 13.1 Introduction 13.2 Related Work 13.2.1 Frame Deletion Detection 13.2.2 Frame Duplication Detection 13.3 Frame Deletion Detection 13.3.1 Baseline Approaches 13.3.2 C3D Network for Frame Deletion Detection 13.3.3 Experimental Result 13.4 Frame Duplication Detection 13.4.1 Coarse-Level Search for Duplicated Frame Sequences 13.4.2 Fine-Level Search for Duplicated Frames 13.4.3 Inconsistency Detector for Duplication Localization 13.4.4 Experimental Results 13.5 Conclusions and Discussion References 14 Integrity Verification Through File Container Analysis 14.1 Introduction 14.1.1 Main Image File Format Specifications 14.1.2 Main Video File Format Specifications 14.2 Analysis of Image File Formats 14.2.1 Analysis of JPEG Tables and Image Resolution 14.2.2 Analysis of Exif Metadata Parameters 14.2.3 Analysis of the JPEG File Format 14.2.4 Automatic Analysis of JPEG Header Information 14.2.5 Methods for the Identification of Social Networks 14.3 Analysis of Video File Formats 14.3.1 Analysis of the Video File Structure 14.3.2 Automated Analysis of mp4-like Videos 14.3.3 Efficient Video Analysis 14.4 Concluding Remarks References 15 Image Provenance Analysis 15.1 The Problem 15.1.1 The Provenance Framework 15.1.2 Previous Work 15.2 Content Retrieval 15.2.1 Approaches 15.2.2 Datasets and Evaluation 15.2.3 Results 15.3 Graph Construction 15.3.1 Approaches 15.3.2 Datasets and Evaluation 15.3.3 Results 15.4 Content Clustering 15.4.1 Approach 15.4.2 Datasets and Evaluation 15.4.3 Results 15.5 Open Issues and Research Directions 15.6 Summary References Part IV Counter-Forensics 16 Adversarial Examples in Image Forensics 16.1 Introduction 16.2 Adversarial Examples in a Nutshell 16.2.1 Problem Definition and Review of the Most Popular Attacks 16.2.2 Adversarial Examples in the Physical Domain 16.2.3 White Versus Black-Box Attacks 16.3 Adversarial Examples in Multimedia Forensics 16.3.1 Transferability of Adversarial Examples in Multimedia Forensics 16.3.2 Increased-Confidence Adversarial Examples with Improved Transferability 16.4 Defenses 16.4.1 Detect Then Defend 16.4.2 Adversarial Training 16.4.3 Detector Randomization 16.4.4 Multiple-Classifier Architectures 16.5 Final Remarks References 17 Anti-Forensic Attacks Using Generative Adversarial Networks 17.1 Introduction 17.2 Background on GANs 17.2.1 GANs for Image Synthesis 17.3 Brief Overview of Relevant Anti-Forensic Attacks 17.3.1 What Are Anti-Forensic Attacks 17.3.2 Anti-Forensic Attack Objectives and Requirements 17.3.3 Traditional Anti-Forensic Attack Design Procedure and Shortcomings 17.3.4 Anti-Forensic Attacks on Parametric Forensic Models 17.3.5 Anti-Forensic Attacks on Deep Neural Networks 17.4 Using GANs to Make Anti-Forensic Attacks 17.4.1 How GANs Are Used to Construct Anti-Forensic Attacks 17.4.2 Overview of Existing GAN-Based Attacks 17.4.3 Differences Between GAN-Based Anti-Forensic Attacks and Adversarial Examples 17.4.4 Advantages of GAN-Based Anti-Forensic Attacks 17.5 Training Anti-Forensic GANs 17.5.1 Overview of Adversarial Training 17.5.2 Knowledge Levels of the Victim Classifier 17.5.3 White Box Attacks 17.5.4 Black Box Scenario 17.5.5 Zero Knowledge 17.6 Known Problems with GAN-Based Attacks & Future Directions References