Building a Debugger: Write a Native x64 Debugger From Scratch

Description

Master the inner workings of your x64 Linux system and expand your OS expertise by writing your very own debugger using C++. If debuggers seem like magic to you, there is no better way to demystify them than to write your own. This book will show you exactly how to do it, walking you through the entire process of building a debugger for x64 Linux systems using C++. As go from an empty filesystem folder to a fully fledged debugger capable of setting breakpoints, stepping through code, manipulating variables, and more, you’ll learn how to: • Attach to a process • Read and write to registers • Set hardware and software breakpoints • Output disassembly • Support multithreading and other tasks As you add features to your debugger, you’ll also pick up a wealth of knowledge about operating systems, compilers, software testing, and low-level programming that you can use in your day-to-day development. ABOUT THE AUTHOR BRIEF CONTENTS CONTENTS IN DETAIL ACKNOWLEDGMENTS INTRODUCTION What Is a Debugger? What Will We Build? The Source Language Limitations What's in This Book? The Target Platform For Windows Users For macOS Users LIST OF ABBREVIATIONS 1: PROJECT SETUP The Directory Structure Writing the Top-Level CMake File Building the libsdb Library Building the sdb Executable Making the Dependencies Accessible Dependency Management with vcpkg Testing Summary 2: COMPILATION AND COMPUTER ARCHITECTURE Compilation Encoding Debug Information Operating Systems and Debuggers Program Loading User Space vs. Kernel Space Virtual Memory Debug APIs Signals Computer Architecture Registers Program Counter Assembly and Instruction Encodings Endianness Stack Frames Summary Check Your Knowledge 3: ATTACHING TO A PROCESS Process Interaction fork and exec ptrace Launching and Attaching to Processes The main Function The attach Function Adding a User Interface Handling User Input Manual Testing Refactoring into a Library Creating a Process Type Implementing launch and attach Handling Errors Destructing Processes Resuming the Process Waiting on Signals Summary Check Your Knowledge 4: PIPES, PROCFS, AND AUTOMATED TESTING Test Cases Testing Process Launching Pipes for Inter-Process Communication The Linux procfs Testing Process Attaching Testing Process Resuming Summary Check Your Knowledge 5: REGISTERS x64 Registers General Purpose Floating Point and Vector Debug Interactions with ptrace Describing Registers X-Macros Register Tables Register Interactions Reading Writing Troubleshooting Summary Check Your Knowledge 6: TESTING REGISTERS WITH X64 ASSEMBLY Why Test with Assembly? An x64 Assembly Primer Test Setup Issuing Syscalls printf General-Purpose Registers MMX SSE x87 Register Reads Exposing Registers The Help Command Register Reading Register Writing Register Interactions Summary Check Your Knowledge 7: SOFTWARE BREAKPOINTS Hardware vs. Software Breakpoints Implementing Software Breakpoints Representing Locations Creating Sites Managing Sites Testing Site Management Enabling Breakpoints Disabling Breakpoints Adding Breakpoints to the Debugger Listing Setting Enabling, Disabling, and Deleting Providing Help Determining Where to Set Breakpoints Position-Independent Executables Breakpoint Tests Continuing Automated Testing Summary Check Your Knowledge 8: MEMORY AND DISASSEMBLY Memory Operations Reading and Writing Exposing Memory to the User Testing Memory Operations Disassembly Summary Check Your Knowledge 9: HARDWARE BREAKPOINTS AND WATCHPOINTS Debug Registers Implementing Hardware Breakpoints Tracking Setting Clearing Testing Hardware Breakpoints Watchpoints Exposing Watchpoints to the User Testing Watchpoints Summary Check Your Knowledge 10: SIGNALS AND SYSCALLS Signal Handlers Printing Stop Information Tracking Debug Register Assignments Tracking Watchpoint Values Displaying Stop Information Catchpoints Tracing Syscalls Exposing Syscall Catchpoints Testing Syscall Catchpoints Signal and Interrupt Internals Summary Check Your Knowledge 11: OBJECT FILES What Is an ELF? Sections and Segments File Structure ELF Header Parsing Section Header Parsing String Tables Getting Section Names Building a Section Map Parsing the General String Table File Addresses, File Offsets, and Virtual Addresses Parsing the Symbol Table Creating a Target Type Auxiliary Vectors Targets Testing Summary Check Your Knowledge 12: DEBUG INFORMATION An Introduction to DWARF DWARF Sections DIE Binary Encoding Fetching a Constants File Parsing Abbreviation Tables Integer Encoding Entry Structure The DWARF Cursor Extraction Parsing Compile Unit Headers Parsing DIEs Implementing Form Skipping Traversing the DIE Tree Reading Attributes Augmenting DIEs with Attribute Support Optimizing Child Iterators Extracting DIE Address Ranges Checking Offsets and Supporting Range Lists Augmenting the dwarf Type Retrieving DIE Names Indexing DIEs Testing the Parser Summary Check Your Knowledge 13: LINE TABLES Line Table Contents Interpreting the Line Table Program The Program Header The Abstract Machine The Program Instructions Retrieving Entries by Line or File Address DIE Line Attribute Helpers Testing the Interpreter Summary Check Your Knowledge 14: SOURCE-LEVEL BREAKPOINTS AND STEPPING Function Inlining Retrieving Inlined Function Stacks Tracking Inlining Finding the Current Stack Source-Level Stepping Step In Step Over Step Out Exposing Stepping to the User Source-Level Breakpoints Expanding Breakpoint Sites Determining the Breakpoint Site Type Implementing Breakpoint Functions Creating Breakpoint Subtypes Creating Breakpoints Exposing Breakpoints to the User Printing Currently Executing Source Code Testing Breakpoints Stepping Summary Check Your Knowledge 15: CALL FRAME INFORMATION A Backtrace Example DWARF Call Frame Information Common Information Entries Frame Description Entries Parsing Common Information Entries Parsing Frame Description Entries Looking Up Frame Description Entries Parsing the EH Frame Header Searching the EH Frame Table Adding the Parsers to the Debugger Summary Check Your Knowledge 16: STACK UNWINDING Executing Call Frame Information Representing Rules Returning Registers Writing a Stack Unwinder Executing Instructions Executing Register Rules Unwinding the Stack Creating a Stack Frame Type Tracking and Manipulating Frames Understanding the Unwinding Algorithm Writing the Unwinding Function Exposing Stack Unwinding to the User Adding up and down Commands Reading Registers from Other Frames Printing the Backtrace Testing Summary Check Your Knowledge 17: SHARED LIBRARIES Program Loading Static Executables Dynamic Executables Loading Dependencies The .dynamic Section The Rendezvous Structure Relocations Global Offset Table Relocation Records Procedure Linkage Table Tracing Shared Library Loading Adding Breakpoint Hit Callbacks Locating the Rendezvous Structure Handling Multiple ELF Files Reading the Loaded Library List Testing Summary Check Your Knowledge 18: MULTITHREADING Threads on Linux The pthreads Library Race Conditions ptrace and the procfs Tracing Threads Trapping New Threads Identifying Thread Creation Representing Thread States Supporting Multithreaded Processes Multithreaded Signal Handling Updating the Signal Handling Function Stopping and Resuming Threads Cleaning Up Exited Threads Reporting Lifecycle Events Handling the Signals Tracing Threads in the Target Exposing Threads to the User Testing Summary Check Your Knowledge 19: DWARF EXPRESSIONS A Taxonomy of DWARF Expressions Single Location Descriptions Simple Location Descriptions Composite Location Descriptions Executing Simple Location Descriptions Defining Result Types Evaluating Expressions Performing Stack Operations Finding the Current Simple Location Description Handling Opcode Ranges Handling Individual Opcodes Performing Stack Operations Executing Dereferencing Instructions Pushing Specific Entities to the Stack Performing Arithmetic, Bitwise, and Relational Operations Executing Control Flow Instructions Executing Non-Address Location Types Handling Composite Locations Executing Location Lists Exposing Attributes to the User Augmenting the Stack Unwinder Reading Global Variables Reading DWARF Expression Results Indexing Global Variables Adding a Variable Lookup Interface Testing Summary Check Your Knowledge 20: VARIABLES AND TYPES Type DIEs Base Types Array Types Class and Union Types Member Pointer Types Storing Type Information Visualizing Typed Data Member Pointers Pointers Classes Arrays Base Types Finding Local Variables Resolving Indirect Names Exposing Variables to the User Testing Summary Check Your Knowledge 21: EXPRESSION EVALUATION Supporting Expression Evaluation Extending Name Lookup Parsing Arguments Representing Built-in Types Parsing a Single Argument Collecting All Arguments Overload Resolution Calling Conventions Classifying Parameters Passing Arguments Returning Values Calling Functions in the Debugger Performing Low-Level Inferior Calls Allocating Memory for Arguments Classifying Parameters Checking Call Triviality Identifiying Constructors and Destructors Checking for Unaligned Fields Implementing Argument Passing Reading Return Values Indexing Member Functions Performing High-Level Inferior Calls Evaluating Expressions Exposing Expression Evaluation to the User Testing the Evaluator Summary Check Your Knowledge 22: ADVANCED TOPICS Remote Debugging Development Tool Communication GDB Machine Interface Debug Adapter Protocol What You Should Use Thread-Local Storage Time-Travel Debugging Exception Catchpoints Non-Stop Mode Follow-Fork Mode Summary APPENDIX: CHECK YOUR KNOWLEDGE ANSWERS GLOSSARY INDEX