Mastering Apache Pulsar: Cloud Native Event Streaming at Scale

Description

Preface Why I Wrote This Book Who This Book Is For How I Organized This Book Conventions Used in This Book Using Code Examples O’Reilly Online Learning How to Contact Us Acknowledgments 1. The Value of Real-Time Messaging Data in Motion Resource Efficiency Interesting Applications Banking Medical Security Internet of Things Summary 2. Event Streams and Event Brokers Publish/Subscribe Queues Failure Modes Push Versus Poll The Need for Pulsar Unification Modularity Performance Summary 3. Pulsar Origins of Pulsar Pulsar Design Principles Multitenancy Geo-Replication Performance Modularity Pulsar Ecosystem Pulsar Functions Pulsar IO Pulsar SQL Pulsar Success Stories Yahoo! JAPAN Splunk Iterable Summary 4. Pulsar Internals Brokers Message Cache BookKeeper and ZooKeeper Communication Schema Validation Inter-Broker Communication Pulsar Functions and Pulsar IO Apache BookKeeper Write-Ahead Logging Message Storing Object/Blob Storage Pravega Majordodo Apache ZooKeeper Naming Service Configuration Management Leader Election Notification System Apache Kafka Apache Druid Pulsar Proxy Java Virtual Machine (JVM) Netty Apache Spark Apache Lucene Summary 5. Consumers What Does It Mean to Be a Consumer? Subscriptions Exclusive Shared Key_Shared Failover Acknowledgments Individual Ack Cumulative Ack Schemas Consumer Schema Management Consumption Modes Batching Chunking Advanced Configuration Delayed Messages Retention Policy Backlog Quota Configuring a Consumer Replay Dead Letter Topics Retry Letter Topics Summary 6. Producers Synchronous Producers Asynchronous Producers Producer Routing Round-Robin Routing Single Partition Routing Custom Partition Routing Producer Configuration topicName producerName sendTimeoutMs blockIfQueueFull maxPendingMessages maxPendingMessagesAcrossPartitions messageRoutingMode hashingScheme cryptoFailureAction batchingMaxPublishDelayMicros batchingMaxMessages batchingEnabled compressionType Schema on Write Using the Schema Registry Nonpersistent Topics Use Cases Using Nonpersistent Topics Transactions Summary 7. Pulsar IO Pulsar IO Architecture Runtime Performance Considerations Use Cases Simple Event Processing Pipelines Change Data Capture Considerations Message Serialization Pipeline Stability Failure Handling Examples Elasticsearch Netty Writing Your Connector TimescaleDB Summary 8. Pulsar Functions Stream Processing Pulsar Functions Architecture Runtime Isolation Isolation with Kubernetes Function Deployments Use Cases Creating Pulsar Functions Simple Event Processing Topic Hygiene Topic Accounting Summary 9. Tiered Storage Storing Data in the Cloud Object Storage Use Cases Replication CQRS Disaster Recovery Offloading Data Pulsar Offloaders Retrieving Offloaded Data Interacting with Object Store Data Repopulating Topics Utilizing Pulsar Client Summary 10. Pulsar SQL Streams as Tables SQL-on-Anything Engines Apache Flink: An Alternative Perspective Presto/Trino How Pulsar SQL Works Configuring Pulsar SQL Performance Considerations Summary 11. Deploying Pulsar Docker Bare Metal Minimum Requirements Getting Started Deploying ZooKeeper Starting BookKeeper Starting Pulsar Public Cloud Providers AWS Azure Google Cloud Platform Kubernetes Summary 12. Operating Pulsar Apache BookKeeper Metrics Server Metrics Journal Metrics Storage Metrics Apache ZooKeeper Metrics Server Metrics Request Metrics Topic Metrics Consumer Metrics Pulsar Transaction Metrics Pulsar Function Metrics Advanced Operating Techniques Interceptors and Tracing Pulsar SQL Metrics Metrics Forwarding Dashboards Summary 13. The Future Programming Language Support Extension Interface Enhancements to Pulsar Functions Architectural Simplification/Expansion Messaging Platform Bridges Summary A. Pulsar Admin API Use Cases Examples Creating a Partitioned Topic Deleting a Partitioned Topic Creating a Namespace with Specific Policies Deleting a Namespace Summary B. Pulsar Admin CLI CLI API Examples Creating a Partitioned Topic Creating a Pulsar IO Source Creating a Pulsar IO Sink Uploading a Schema Deleting a Schema Creating a Namespace Deleting a Namespace Summary C. Geo-Replication Synchronous Replication Asynchronous Replication Replication Patterns Mesh Aggregation Standby Admin- and Producer-Level Control Summary D. Security, Authentication, and Authorization in Pulsar Encryption in Transit Encryption at Rest Authentication Authorization Summary Index About the Author