Foundations of Linux Debugging, Disassembling, and Reversing: Analyze Binary Code, Understand Stack Memory Usage, and Reconstruct C/C++ Code with Intel x64

Description

Review topics ranging from Intel x64 assembly language instructions and writing programs in assembly language, to pointers, live debugging, and static binary analysis of compiled C and C++ code. This book is ideal for Linux desktop and cloud developers.  Using the latest version of Debian, you’ll focus on the foundations of the diagnostics of core memory dumps, live and postmortem debugging of Linux applications, services, and systems, memory forensics, malware, and vulnerability analysis. This requires an understanding of x64 Intel assembly language and how C and C++ compilers generate code, including memory layout and pointers. This book provides the back­ground knowledge and practical foundations you’ll need in order to master internal Linux program structure and behavior. It consists of practical step-by-step exercises of increasing complexity with explanations and ample diagrams. You’ll also work with the GDB debugger and use it for disassembly and reversing.   By the end of the book, you will have a solid understanding of how Linux C and C++ compilers generate binary code. In addition, you will be able to analyze such code confidently, understand stack memory usage, and reconstruct original C/C++ code. Foundations of Linux Debugging, Disassembling, and Reversing is the perfect companion to Foundations of ARM64 Linux Debugging, Disassembling, and Reversing for readers interested in the cloud or cybersecurity. What You'll LearnReview the basics of x64 assembly languageExamine the essential GDB debugger commands for debugging and binary analysis Study C and C++ compiler code generation with and without compiler optimizations Look at binary code disassembly and reversing patternsSee how pointers in C and C++ are implemented and usedWho This Book Is For Software support and escalation engineers, cloud security engineers, site reliability engineers, DevSecOps, platform engineers, software testers, Linux C/C++ software engineers and security researchers without Intel x64 assembly language background, beginners learning Linux software reverse engineering techniques, and engineers coming from non-Linux environments. Table of Contents About the Author About the Technical Reviewer Preface Chapter 1: Memory, Registers, and Simple Arithmetic Memory and Registers Inside an Idealized Computer Memory and Registers Inside Intel 64-Bit PC “Arithmetic” Project: Memory Layout and Registers “Arithmetic” Project: A Computer Program “Arithmetic” Project: Assigning Numbers to Memory Locations Assigning Numbers to Registers “Arithmetic” Project: Adding Numbers to Memory Cells Incrementing/Decrementing Numbers in Memory and Registers Multiplying Numbers Summary Chapter 2: Code Optimization “Arithmetic” Project: C/C++ Program Downloading GDB GDB Disassembly Output – No Optimization GDB Disassembly Output – Optimization Summary Chapter 3: Number Representations Numbers and Their Representations Decimal Representation (Base Ten) Ternary Representation (Base Three) Binary Representation (Base Two) Hexadecimal Representation (Base Sixteen) Why Are Hexadecimals Used? Summary Chapter 4: Pointers A Definition “Pointers” Project: Memory Layout and Registers “Pointers” Project: Calculations Using Pointers to Assign Numbers to Memory Cells Adding Numbers Using Pointers Incrementing Numbers Using Pointers Multiplying Numbers Using Pointers Summary Chapter 5: Bytes, Words, Double, and Quad Words Using Hexadecimal Numbers Byte Granularity Bit Granularity Memory Layout Summary Chapter 6: Pointers to Memory Pointers Revisited Addressing Types Registers Revisited NULL Pointers Invalid Pointers Variables As Pointers Pointer Initialization Initialized and Uninitialized Data More Pseudo Notation “MemoryPointers” Project: Memory Layout Summary Chapter 7: Logical Instructions and RIP Instruction Format Logical Shift Instructions Logical Operations Zeroing Memory or Registers Instruction Pointer Code Section Summary Chapter 8: Reconstructing a Program with Pointers Example of Disassembly Output: No Optimization Reconstructing C/C++ Code: Part 1 Reconstructing C/C++ Code: Part 2 Reconstructing C/C++ Code: Part 3 Reconstructing C/C++ Code: C/C++ Program Example of Disassembly Output: Optimized Program Summary Chapter 9: Memory and Stacks Stack: A Definition Stack Implementation in Memory Things to Remember PUSH Instruction POP Instruction Register Review Application Memory Simplified Stack Overflow Jumps Calls Call Stack Exploring Stack in GDB Summary Chapter 10: Frame Pointer and Local Variables Stack Usage Register Review Addressing Array Elements Stack Structure (No Function Parameters) Function Prolog Raw Stack (No Local Variables and Function Parameters) Function Epilog “Local Variables” Project Disassembly of Optimized Executable Summary Chapter 11: Function Parameters “FunctionParameters” Project Stack Structure Function Prolog and Epilog Project Disassembled Code with Comments Parameter Mismatch Problem Summary Chapter 12: More Instructions CPU Flags Register The Fast Way to Fill Memory Testing for 0 TEST – Logical Compare CMP – Compare Two Operands TEST or CMP? Conditional Jumps The Structure of Registers Function Return Value Using Byte Registers Summary Chapter 13: Function Pointer Parameters “FunctionPointerParameters” Project Commented Disassembly Summary Chapter 14: Summary of Code Disassembly Patterns Function Prolog/Epilog LEA (Load Effective Address) Passing Parameters Accessing Parameters and Local Variables Summary Index