← Back to Paths
EXPERT ROADMAP
Apache Kafka & Event Streaming
Apache Kafka & Event Streaming
Build fault-tolerant event-driven systems that process millions of events per second.
CREATED BY
S
Sneha T. ★ 4.9
Business Analyst Lead at ConsultPro | 8+ years of experience
About this Path
For engineers who use Kafka in production but want to master it. Covers internals — log compaction, ISR, exactly-once semantics — alongside Kafka Streams, ksqlDB, schema management, and Kubernetes-native deployment via Strimzi. You will build a real-time analytics pipeline with end-to-end exactly-once guarantees.
Path Overview
Advanced LevelCertificate of CompletionAbout 56 hours to completeEnglish language20+ curated videosLearn online at your own pace6 modules with resourcesGamified & interactive
Path Curriculum
Log Segments, Offsets, and Retention Policies
How Kafka stores messages on disk; time-based vs size-based retention and compaction.
Leader Election and ISR Protocol
Controller election, in-sync replica set management, and preferred leader rebalancing.
KRaft Mode — Removing ZooKeeper Dependency
Architecture of the Raft-based metadata quorum and migration from ZooKeeper.
Network Threads, I/O Threads, and Request Queues
Broker threading model; tuning num.network.threads and num.io.threads for throughput.
Producer Batching, Compression, and Linger Configuration
Trade latency vs throughput with batch.size, linger.ms, and snappy/lz4/zstd.
Idempotent and Transactional Producers
Enable exactly-once delivery; understand epoch fencing and transaction coordinator.
Consumer Group Rebalancing and Cooperative Rebalancing
Static membership, incremental cooperative rebalance, and avoiding stop-the-world pauses.
Manual Offset Management and Dead Letter Topics
Commit offsets after processing; route poison pills to DLT for async reprocessing.
Topology DSL — KStream, KTable, GlobalKTable
Model stream-table duality; choose the right abstraction for your join semantics.
Windowed Aggregations — Tumbling, Hopping, Session
Count and aggregate events in time windows; handle late arrivals with grace periods.
State Stores and RocksDB Tuning
Persistent and in-memory stores; changelog-backed fault recovery and standby replicas.
Interactive Queries and REST Service Layer
Expose local state store contents over HTTP for real-time query use cases.
Avro, Protobuf, and JSON Schema Comparison
Serialisation size, code generation, and schema evolution compatibility rules.
Confluent Schema Registry — Subjects and Compatibility Modes
Configure BACKWARD, FORWARD, and FULL compatibility; automate schema registration in CI.
Schema Migration Strategies Without Downtime
Two-phase publish; default field values and namespace aliases for safe evolution.
Persistent Queries and Push vs Pull Queries
Build real-time materialised views; understand when to use pull queries for low-latency lookup.
Stream-Table Joins and Enrichment Patterns
Enrich click events with user profiles using co-partitioned stream-table joins.
Connector Integration with Kafka Connect
Source and sink connectors for Postgres, S3, and Elasticsearch with SMT transforms.
Strimzi Operator — Kafka Cluster Custom Resources
Define broker, Zookeeper-less cluster, and topic resources declaratively in YAML.
Rack Awareness, Pod Disruption Budgets, and Rolling Upgrades
Maintain availability during broker restarts; spread replicas across AZs.
Prometheus JMX Exporter and Grafana Kafka Dashboards
Track consumer lag, under-replicated partitions, produce/fetch latency percentiles.
Capacity Planning and Partition Count Decisions
Model throughput per partition; avoid over-partitioning pitfalls and coordinator load.
What you'll learn
- ✓Tune partitioning strategy and replication factor for throughput, ordering, and fault-tolerance goals.
- ✓Implement exactly-once semantics using idempotent producers, transactional APIs, and consumer isolation levels.
- ✓Build stateful stream processing topologies in Kafka Streams with windowed aggregations and changelog topics.
- ✓Enforce Avro schemas and manage schema evolution without breaking consumers using Confluent Schema Registry.
- ✓Deploy a multi-broker Kafka cluster on Kubernetes using Strimzi with rack-awareness and rolling upgrades.
- ✓Monitor consumer lag, under-replicated partitions, and broker skew using Prometheus and Grafana dashboards.