Hands-On System Programming With Linux: Explore Linux System Programming Interfaces, Theory, And Practice

Description

The Linux OS and its embedded and server applications are critical components of today’s software infrastructure in a decentralized, networked universe. The industry's demand for proficient Linux developers is only rising with time. Hands-On System Programming with Linux gives you a solid theoretical base and practical industry-relevant descriptions, and covers the Linux system programming domain. It delves into the art and science of Linux application programming— system architecture, process memory and management, signaling, timers, pthreads, and file IO. Title Page......Page 2 Copyright and Credits......Page 3 Packt Upsell......Page 4 Contributors......Page 5 Table of Contents......Page 7 Preface......Page 19 Technical requirements......Page 27 Linux and the Unix operating system......Page 28 The Unix philosophy in a nutshell......Page 29 Everything is a process – if it's not a process, it's a file......Page 30 One tool to do one task......Page 33 Three standard I/O channels......Page 35 Word count ......Page 36 cat......Page 37 Combine tools seamlessly......Page 39 Plain text preferred......Page 41 Modular, designed to be repurposed by others......Page 42 Pseudocode......Page 43 The ABI......Page 45 Accessing a register's content via inline assembly......Page 49 Accessing a control register's content via inline assembly......Page 51 CPU privilege levels......Page 52 Privilege levels or rings on the x86......Page 53 Linux architecture......Page 56 Libraries......Page 57 System calls......Page 58 Linux – a monolithic OS......Page 59 What does that mean?......Page 60 Execution contexts within the kernel......Page 64 Interrupt context......Page 65 Summary......Page 66 Technical requirements......Page 67 Virtual memory......Page 68 No VM – the problem......Page 69 Objective......Page 70 Virtual memory......Page 72 Addressing 1 – the simplistic flawed approach......Page 76 Addressing 2 – paging in brief......Page 79 Paging tables – simplified......Page 81 Address-translation......Page 83 Process-isolation......Page 84 The programmer need not worry about physical memory......Page 85 Memory-region protection......Page 86 SIDEBAR :: Testing the memcpy() C program......Page 87 Process memory layout......Page 91 Segments or mappings......Page 92 Data segments......Page 94 Library segments......Page 95 Why a process stack?......Page 96 Peeking at the stack......Page 99 Advanced – the VM split......Page 102 Summary......Page 107 Resource limits......Page 108 Granularity of resource limits......Page 110 Available resource limits......Page 111 Hard and soft limits......Page 113 Querying and changing resource limit values......Page 116 Caveats......Page 118 Using prlimit(1) – examples......Page 119 API interfaces......Page 122 Code examples......Page 124 Permanence......Page 129 Summary......Page 130 Dynamic Memory Allocation......Page 131 The malloc(3) API......Page 132 malloc(3) – some FAQs......Page 135 The free API......Page 140 The calloc API......Page 142 The realloc API......Page 143 The realloc(3) – corner cases......Page 144 The reallocarray API......Page 145 Using the sbrk() API......Page 146 How malloc(3) really behaves......Page 150 Scenario 1 – default options......Page 151 Scenario 2 – showing malloc statistics......Page 152 Scenario 3 – large allocations option......Page 153 Advanced features......Page 154 Demand-paging......Page 155 Resident or not?......Page 157 Locking memory......Page 158 Limits and privileges......Page 159 Locking all pages......Page 163 Memory protection......Page 164 Memory protection – a code example......Page 165 LSM logs......Page 173 An experiment – running the memprot program on an ARM-32......Page 174 Using alloca to allocate automatic memory......Page 177 Summary......Page 181 Linux Memory Issues......Page 182 Common memory issues......Page 183 Incorrect memory accesses......Page 185 Test case 1: Uninitialized memory access......Page 186 Test case 2......Page 188 Test case 3......Page 189 Test case 4......Page 190 Test case 5......Page 191 Test case 6......Page 192 Test case 7......Page 193 Use-after-free/Use-after-return bugs......Page 194 Test case 8......Page 195 Test case 9......Page 196 Test case 10......Page 197 Test case 11......Page 200 Test case 12......Page 202 Test case 13 ......Page 205 Test case 13.1......Page 206 Test case 13.2......Page 207 Test case 13.3......Page 209 Undefined behavior......Page 210 Fragmentation......Page 211 Miscellaneous......Page 212 Summary......Page 213 Debugging Tools for Memory Issues......Page 214 Tool types......Page 215 Using Valgrind's Memcheck tool......Page 216 Valgrind pros and cons : a quick summary......Page 228 Sanitizer tools......Page 229 Sanitizer toolset......Page 230 Building programs for use with ASan......Page 231 Running the test cases with ASan......Page 232 AddressSanitizer (ASan) summary table......Page 245 AddressSanitizer pros and cons – a quick summary......Page 246 Glibc mallopt......Page 248 Malloc options via the environment ......Page 250 Code coverage while testing......Page 251 A mention of the malloc API helpers......Page 252 Summary......Page 254 Process Credentials......Page 255 The traditional Unix permissions model......Page 256 How the Unix permission model works......Page 257 Determining the access category......Page 260 Real and effective IDs......Page 262 A puzzle – how can a regular user change their password?......Page 265 The setuid and setgid special permission bits......Page 267 Setting the setuid and setgid bits with chmod......Page 268 Hacking attempt 1......Page 269 Querying the process credentials......Page 272 Code example......Page 273 Sudo – how it works......Page 274 Setting the process credentials......Page 275 Hacking attempt 2......Page 276 An aside – a script to identify setuid-root and setgid  installed programs......Page 280 setgid example – wall......Page 282 Giving up privileges......Page 285 Saved-set UID – a quick demo......Page 286 The setres[u|g]id(2) system calls......Page 289 Important security notes......Page 291 Summary......Page 292 Process Capabilities......Page 293 Motivation......Page 294 POSIX capabilities......Page 295 Viewing process capabilities via procfs......Page 298 Thread capability sets......Page 300 File capability sets......Page 301 Embedding capabilities into a program binary......Page 302 Getcap and similar utilities......Page 306 Wireshark – a case in point......Page 307 Setting capabilities programmatically......Page 308 How ls displays different binaries......Page 314 Permission models layering......Page 315 FYI – under the hood, at the level of the Kernel......Page 316 Summary......Page 317 Technical requirements......Page 318 Converting a program to a process......Page 319 The exec Unix axiom......Page 320 Key points during an exec operation......Page 321 Testing the exec axiom......Page 322 Experiment 2 – on the CLI, again......Page 323 The point of no return......Page 324 Family time – the exec family APIs......Page 325 Passing a zero as an argument......Page 328 Specifying the name of the successor......Page 329 The execlp API......Page 332 The execv API......Page 334 Exec at the OS level......Page 335 Summary table – exec family of APIs......Page 336 Code example......Page 337 Summary......Page 340 Process Creation......Page 341 How fork works......Page 342 Using the fork system call......Page 345 Fork rule #1......Page 346 Fork rule #2 – the return......Page 347 Fork rule #3......Page 353 Fork rule #4 – data......Page 355 Fork rule #5 – racing......Page 356 The process and open files......Page 357 Fork rule #6 – open files......Page 359 Open files and security......Page 361 Malloc and the fork......Page 362 COW in a nutshell......Page 364 Waiting and our simpsh project......Page 365 The Unix fork-exec semantic......Page 366 The need to wait......Page 367 Defeating the race after fork......Page 368 Putting it together – our simpsh project......Page 369 The wait API – details......Page 373 The scenarios of wait......Page 376 Wait scenario #2......Page 377 Fork bombs and creating more than one child......Page 378 The waitpid(2)......Page 380 The waitid (2)......Page 383 The actual system call......Page 384 Orphans......Page 386 Zombies......Page 387 Fork rule #7......Page 388 Summary......Page 389 Signaling - Part I......Page 390 The signal mechanism in brief......Page 391 Available signals......Page 394 The standard or Unix signals......Page 395 Handling signals......Page 398 Using the sigaction system call to trap signals......Page 399 The sigaction structure......Page 400 Signal masking with the sigprocmask API......Page 405 Querying the signal mask......Page 406 Reentrant functions......Page 409 Alternate ways to be safe within a signal handler......Page 411 Signal-safe atomic integers......Page 412 Powerful sigaction flags......Page 415 Zombies not invited......Page 416 No zombies! – the classic way......Page 417 No zombies! – the modern way......Page 418 Interrupted system calls and how to fix them with the SA_RESTART......Page 420 The once only SA_RESETHAND flag......Page 422 Signal behavior when masked......Page 423 Case 1 : Default : SA_NODEFER bit cleared......Page 424 Case 2 : SA_NODEFER bit set......Page 425 Running of case 1 – SA_NODEFER bit cleared [default]......Page 429 Running of case 2 – SA_NODEFER bit set......Page 430 Using an alternate signal stack......Page 433 Implementation to handle high-volume signals with an alternate signal stack......Page 434 Case 1 – very small (100 KB) alternate signal stack......Page 436 Case 2 : A large (16 MB) alternate signal stack ......Page 437 Summary......Page 438 Signaling - Part II......Page 439 Detailing information with the SA_SIGINFO......Page 440 The siginfo_t structure......Page 441 Getting system-level details when a process crashes......Page 445 Trapping and extracting information from a crash......Page 446 Register dumping......Page 451 Finding the crash location in source code......Page 455 What does errno do?......Page 457 The errno race......Page 458 Fixing the errno race......Page 459 Sleeping correctly......Page 460 The nanosleep system call......Page 461 Real-time signals......Page 464 Differences from standard signals......Page 465 Real time signals and priority......Page 466 Just kill 'em......Page 470 Agent 00 – permission to kill......Page 471 Are you there?......Page 472 Crude IPC......Page 473 Better IPC – sending a data item......Page 474 Sidebar – LTTng......Page 479 Synchronously waiting for signals......Page 481 Waiting forever or until a signal arrives......Page 482 The sigwait library API......Page 483 The sigwaitinfo and the sigtimedwait system calls......Page 488 The signalfd(2) API......Page 489 Summary......Page 492 Timers......Page 493 The good ol' alarm clock......Page 494 Interval timers......Page 497 A simple CLI digital clock......Page 501 Obtaining the current time......Page 503 Trial runs......Page 505 A word on using the profiling timers......Page 506 The newer POSIX (interval) timers mechanism......Page 508 Creating and using a POSIX (interval) timer......Page 509 The arms race – arming and disarming a POSIX timer......Page 512 Example code snippet showing the workflow......Page 514 Figuring the overrun......Page 517 How fast is fast?......Page 518 Our react game – how it works......Page 519 React – trial runs......Page 521 The react game – code view......Page 523 The run:walk interval timer application......Page 527 A few trial runs......Page 528 The low – level design and code......Page 530 Timer lookup via proc......Page 534 Timers via file descriptors......Page 535 A quick note on watchdog timers......Page 537 Summary......Page 538 Multithreading with Pthreads Part I - Essentials......Page 539 What exactly is a thread?......Page 540 Resource sharing......Page 541 Multiprocess versus multithreaded......Page 545 Example 1 – creation/destruction – process/thread......Page 546 The multithreading model......Page 547 Example 2 – matrix multiplication – process/thread......Page 549 On a VM with 1 GB RAM, two CPU cores and parallelized make -j4......Page 554 On a VM with 1 GB RAM, one CPU core and sequential make -j1......Page 556 Taking advantage of potential parallelism......Page 557 Overlapping CPU with I/O......Page 558 Automatically taking advantage of modern hardware......Page 559 Context switching......Page 560 POSIX threads......Page 561 Pthreads and Linux......Page 562 Thread management – the essential pthread APIs......Page 563 Thread creation......Page 564 Termination......Page 567 The return of the ghost......Page 569 How many threads is too many?......Page 572 How many threads can you create?......Page 574 Code example – creating any number of threads......Page 576 How many threads should one create?......Page 578 Thread attributes......Page 580 Code example – querying the default thread attributes......Page 581 Joining......Page 584 The thread model join and the process model wait......Page 589 Checking for life, timing out......Page 590 Join or not?......Page 591 Parameter passing......Page 592 Passing a structure as a parameter......Page 593 Thread parameters – what not to do......Page 595 Get and set thread stack size......Page 597 Stack location......Page 598 Stack guards......Page 600 Summary......Page 604 Multithreading with Pthreads Part II - Synchronization......Page 605 The racing problem......Page 606 The pedagogical bank account example......Page 607 Critical sections......Page 610 Locking concepts......Page 611 Is it atomic?......Page 613 Dirty reads......Page 617 Locking guidelines......Page 618 Locking granularity......Page 620 Deadlock and its avoidance......Page 621 The ABBA deadlock......Page 622 Avoiding deadlock......Page 623 Using the pthread APIs for synchronization......Page 624 The mutex lock......Page 625 Seeing the race......Page 628 Mutex types......Page 631 The robust mutex attribute......Page 633 IPC, threads, and the process-shared mutex......Page 635 Priority inversion......Page 641 Watchdog timer in brief......Page 643 The Mars Pathfinder mission in brief......Page 645 Priority inheritance – avoiding priority inversion......Page 646 Summary of mutex attribute usage......Page 648 Timing out on a mutex lock attempt......Page 649 The reader-writer mutex lock......Page 650 The spinlock variant......Page 652 A few more mutex usage guidelines......Page 654 Is the mutex locked?......Page 655 Condition variables......Page 656 Using the condition variable......Page 657 A simple CV usage demo application......Page 659 CV broadcast wakeup......Page 663 Summary......Page 665 Thread safety......Page 666 Reentrant-safe versus thread-safe......Page 669 Thread safety via mutex locks......Page 671 Thread safety via function refactoring......Page 674 List of APIs not required to be thread-safe......Page 676 Refactoring glibc APIs from foo to foo_r......Page 677 Some glibc foo and foo_r APIs......Page 679 Thread safety via TLS ......Page 680 Thread safety via TSD ......Page 682 Canceling a thread......Page 683 The cancelability state......Page 684 The cancelability type......Page 685 Canceling a thread – a code example......Page 688 Cleaning up at thread exit......Page 690 Thread cleanup – code example......Page 691 Threads and signaling......Page 693 The POSIX solution to handling signals on MT......Page 694 Code example – handling signals in an MT app......Page 695 Threads vs processes – look again......Page 697 The multiprocess vs the multithreading model – pros of the MT model......Page 698 The multiprocess vs the multithreading model – cons of the MT model......Page 699 Pthreads – some FAQs......Page 700 Debugging multithreaded (pthreads) applications with GDB......Page 701 Summary......Page 703 The Linux OS and the POSIX scheduling model......Page 704 The Linux process state machine......Page 705 The sleep states......Page 706 What is real time?......Page 708 Types of real time......Page 709 Scheduling policies......Page 710 Peeking at the scheduling policy and priority......Page 712 The nice value......Page 713 CPU affinity......Page 714 Scheduling policy and priority APIs......Page 717 Code example – setting a thread scheduling policy and priority......Page 719 Soft real-time – additional considerations......Page 724 RTL – Linux as an RTOS......Page 725 Summary......Page 726 Advanced File I/O......Page 727 I/O performance recommendations......Page 728 The kernel page cache......Page 729 Via the posix_fadvise(2) API......Page 730 Via the readahead(2) API......Page 731 MT app file I/O with the pread, pwrite APIs......Page 732 Discontiguous data file – traditional approach......Page 734 Discontiguous data file – the SG – I/O approach......Page 736 File I/O via memory mapping......Page 739 The Linux I/O code path in brief......Page 740 Memory mapping a file for I/O......Page 743 File and anonymous mappings......Page 746 The mmap advantage......Page 748 Memory mapping – additional points......Page 750 Direct I/O (DIO)......Page 752 Asynchronous I/O (AIO)......Page 753 I/O technologies – a quick comparison......Page 754 Multiplexing or async blocking I/O – a quick note......Page 755 I/O schedulers......Page 756 Ensuring sufficient disk space......Page 758 Utilities for I/O monitoring, analysis, and bandwidth control......Page 759 Summary......Page 760 Troubleshooting and Best Practices......Page 761 perf......Page 762 The Linux proc filesystem......Page 763 Software engineering wisdom in a nutshell......Page 764 A programmer’s checklist – seven rules......Page 765 Using the Linux kernel's control groups......Page 766 Summary......Page 767 Other Books You May Enjoy......Page 768 Index......Page 771