Event Streaming Architecture

Event Streaming Architecture is the architectural style in which a durable, replayable, ordered stream of events — a Distributed Log System Design — serves as the central nervous system of the entire data platform. Services do not write to private databases and then notify other services; instead, they publish state changes as events to a shared streaming substrate, and other services consume from the stream to build their own materialized views, react to changes, or feed downstream pipelines. The stream is the source of truth — the canonical record of what happened — and every service’s local state is a derived projection that can be rebuilt from the stream. This is the inversion at the heart of the architecture: traditional systems treat the database as the source of truth and the stream as a notification channel; streaming architecture treats the stream as the source of truth and the databases as derived materialized views. The foundational text is Jay Kreps’s 2013 essay The Log: What every software engineer should know about real-time data’s unifying abstraction, written shortly after Kafka was open-sourced from LinkedIn, which argues that an ordered durable log is the right unifying primitive for data integration, change-data-capture, event sourcing, real-time analytics, and stream processing — they are all the same idea looked at from different angles. The architecture has been refined through Confluent’s “Event Streaming Platform” framing (Stopford’s 2018 Designing Event-Driven Systems is the canonical book), Netflix’s Keystone at trillion-events-per-day scale, LinkedIn’s continued evolution into Brooklin and beyond, and the Apache Pulsar community’s similar story. This note treats event streaming as a whole-system architectural style — distinct from but composing with Event-Driven Architecture (broader umbrella; streaming is one durable implementation), Event Sourcing Pattern (intra-service persistence; streaming is the inter-service spine), and Command Query Responsibility Segregation (CQRS read sides are often fed by streams). The substrate-level details (Kafka internals, partitioning, replication) are covered in Distributed Log System Design; this note focuses on the architectural choice of putting a stream at the center of the platform.

0. The Kreps 2013 Essay in Context — Why “The Log” Matters

The 2013 The Log essay is the founding document for this architecture, and reading it as a historical artifact rather than a stand-alone manifesto helps explain why the resulting design choices look the way they do.

Jay Kreps wrote it in late 2013 from inside LinkedIn, which by 2010 had hit a recognizable wall: roughly 50 internal data systems (search, ads, profile, news feed, network graph, recommendations, A/B testing, Hadoop analytics, monitoring) each needing data from each of the others. Every pair was a one-off integration — a bespoke ETL job, a polling API, a dual write from the application — and the integration matrix had grown to roughly N² edges, of which a substantial fraction broke whenever any source changed schema or any sink fell behind. The on-call burden grew superlinearly with the number of systems; new analytics use cases were blocked for weeks waiting for someone to wire up the data; and the team had no shared understanding of “what is the canonical state of X” because every system had its own slightly-stale copy obtained via a different mechanism.

Kreps’s framing in the essay is explicit: the integration problem is what motivated Kafka, more than any individual use case. Kafka was the LinkedIn team’s response to the realization that an ordered durable log was the right shared abstraction to collapse N² integrations to N producers + M consumers connected through a common bus. Each producer publishes its events once; each consumer subscribes and reads from where it left off; the integration matrix becomes linear in the number of systems rather than quadratic in the number of pairs.

The essay’s bigger move is rhetorical and philosophical: it argues that log is not just a useful piece of infrastructure but a unifying abstraction that subsumes data integration (replication), change-data-capture (database internal logs exposed externally), event sourcing (the application’s persistence), real-time analytics (consumers compute online), and stream processing (consumers emit derived streams). Each of these had been treated as a distinct problem with distinct tooling; Kreps argued they were all manifestations of the same underlying abstraction — an ordered, durable, partitioned log of events — and that recognizing this would simplify both the engineering and the operational story.

Kreps’s later 2014 book I Heart Logs (O’Reilly) and his 2014 essay Questioning the Lambda Architecture extend the same argument; the Confluent product line and the Designing Event-Driven Systems book (Stopford 2018) crystallize it into an architectural style. The intellectual lineage runs further back too — Pat Helland’s 2015 CIDR paper Immutability Changes Everything makes the case that immutability (the defining property of an append-only log) is a fundamental architectural primitive that the CRUD-centric era of databases obscured.

The takeaway for any team evaluating streaming architecture: it is not just a technical decision about messaging infrastructure but a philosophical commitment — that ordered, durable, replayable event logs are the right primitive at the center of a data platform, and that databases and APIs are derived constructs around them. Teams that adopt the architecture without internalizing this commitment often build a “Kafka-shaped” version of a traditional database-centric architecture, missing most of the pattern’s value: they treat Kafka as a fancy queue, accept dual-writes, skip the schema registry, and end up with a more complex version of the integration problems they had before.

0.1 Stream as Source of Truth vs Database as Source of Truth (with Replicate-to-Log)

A frequently muddled question in streaming-architecture design is which thing is the source of truth — the stream, or the database. The two viable answers correspond to materially different operational regimes, and choosing without articulating the choice produces architectures that fall in the gap and get the worst of both.

Regime A — Stream as source of truth. This is the canonical Kreps framing: every state change in the system originates as an event written to the log; service-local databases are derived materialized views fed from the same log; if a database is corrupted or rebuilt from scratch, replaying the relevant streams reconstructs it. The benefit is uniformity: every consumer sees the same canonical events; replay is the universal disaster-recovery tool; the audit trail is intrinsic; new derived systems are added by subscribing.

The cost is that every service must publish events for every state change — the discipline of “no hidden writes” — which means transactional outbox or equivalent at every producing service, and a substantial cultural commitment to never bypassing the log. The cultural commitment matters more than the technical machinery: it requires every team to internalize that “if I update my database without emitting an event, I have created an architectural violation that breaks downstream consumers’ ability to keep up.” Code review, architectural review, and platform-level checks (e.g., monitoring outbox tables for stuck rows, alerting on direct database writes that bypass the outbox) all reinforce the discipline.

Regime B — Database as source of truth, replicate to log via CDC. This regime leaves traditional databases as the canonical record and uses change-data-capture (Debezium reading PostgreSQL/MySQL/Oracle WAL, AWS DMS, Confluent’s CDC connectors) to mirror state changes into the log for downstream consumers. The benefit is that existing services do not need outbox plumbing — the database’s own write-ahead log is the source of events, and the application’s data layer is unchanged.

The cost is that the events are physical row-change events (the database’s idea of a change), not domain events (the business’s idea of what happened). Consumers receive users table updated_at=2026-05-09 with a new email value, not EmailAddressVerified; reconstructing business meaning from physical change events is downstream work. A row-change event also includes a full row image, exposing internal columns (audit flags, timestamps, foreign keys) that the producing service’s API would never have exposed externally — so consumers become coupled to the producer’s database schema, not its API.

There is also a subtle correctness issue: the CDC stream depends on the database keeping its WAL accessible (PostgreSQL replication slots, MySQL binlog retention, Oracle GoldenGate licensing), and if the WAL is truncated before the CDC reader catches up, events are silently lost — a class of incident that has occurred at multiple companies. The defenses (sufficient retention, monitoring CDC lag against retention horizon, alerting before truncation risk materializes) are well-understood but must be implemented explicitly.

Regime C — Dual write (anti-pattern). The application writes to the database AND publishes to the log in two separate operations. Without two-phase commit (which the log generally does not support), one of the writes can succeed while the other fails, producing permanent divergence. Dual write is ubiquitous in early streaming-architecture adoptions because the outbox pattern feels like overhead and “just publish to Kafka after the DB write” looks reasonable. It is invariably the source of the “we’re missing events” production incidents that retrofitting the outbox pattern was invented to prevent.

The pragmatic recommendation. Use stream as source of truth with transactional outbox for new services where the team can adopt the discipline; use CDC for existing legacy systems that cannot be modified; never use dual write. The Kreps essay and Stopford’s book both make this distinction explicit; many production architectures get into trouble by sliding into dual write because the outbox felt like overhead until the first incident, and then by trying to retrofit outbox under emergency pressure rather than as a planned architectural commitment.

1. When to Use / When Not to Use

When event streaming architecture is the right call. The strongest indicator is a heterogeneous data integration problem at scale. A large e-commerce platform has dozens of services and dozens of derived data systems: a search index, a recommendation feature store, a fraud-detection pipeline, an analytics warehouse, a real-time dashboard, a partner-export feed, a CDC pipeline to a backup region. In a non-streaming architecture, every pair of (source service, destination system) is a custom integration: ETL jobs, polling APIs, dual writes from the application, ad-hoc scripts. The integration matrix is O(N²) — every new system requires touching every relevant source. With a streaming spine, every source publishes events once; every destination consumes events from the stream. The integration becomes O(N) — N producers + M consumers connected through the stream. Kreps’s 2013 essay makes this explicit: the log is the integration bus.

A second strong indicator is the need for replayability and history. Building a new derived system (a new search index with different relevance weighting; a new ML feature pipeline with new aggregations) on a non-streaming architecture means writing a backfill job that scrapes historical state from the source databases — slow, error-prone, and incomplete. With a streaming architecture and sufficient retention, the new system just rewinds the stream and rebuilds from history. Replayability is a near-free byproduct of using a durable log.

A third indicator is real-time stream processing as a first-class workload. If your analytics or ML use cases require sub-second freshness — fraud scoring on transactions in flight, real-time pricing adjustments, alerts on metric anomalies — you need a stream that stream-processing frameworks (Apache Flink, Kafka Streams, Spark Structured Streaming) can consume. The architecture naturally supports this; bolt-on stream processing onto a non-streaming architecture is awkward.

A fourth indicator is the team is already operating Kafka or equivalent at scale. If your operational reality already includes a substantial Kafka cluster — perhaps for messaging, perhaps for one specific use case — adopting it as the architectural spine has smaller marginal cost than introducing it from scratch. Many companies that today have a streaming architecture got there incrementally: Kafka was first for messaging, then for CDC, then for event-driven services, then for real-time analytics — and at some point the architecture was de facto streaming-centric without anyone having explicitly decided on the architectural style.

A fifth, more subtle indicator is organizational scale. Streaming architecture aligns with multiple-team independence: each team owns its services and the events those services publish; downstream teams consume and build their own views. Coordination is around event schemas (a contract layer), not around shared databases or synchronous APIs. This scales organizationally to hundreds of teams in a way that monolithic data platforms cannot.

When event streaming architecture is the wrong call. First, when the system is small, single-team, and the data integration matrix is trivial. Two services and one analytics dashboard do not need Kafka. A normal database with a periodic export to the dashboard is fine. The streaming substrate’s operational cost (running Kafka, schema management, monitoring) is real, and small systems do not amortize it.

Second, when the team lacks the operational capacity to run Kafka or equivalent. Kafka is not trivial to operate at scale — broker tuning, partition management, ZooKeeper/KRaft maintenance, replication monitoring, schema-registry coordination. Managed services (Confluent Cloud, AWS MSK, Aiven) reduce this burden but not to zero. Without operational maturity, a streaming architecture’s central bus becomes a single point of operational failure.

Third, when strong cross-stream consistency is required and the team cannot live with eventual consistency. Streaming gives you per-key/per-partition ordering and at-least-once or (with care) exactly-once delivery, but cross-stream consistency is the application’s problem. If your domain requires strict transactional consistency across multiple data products derived from the stream, you may need the streaming primitives plus careful application logic, or you may need a different architecture.

Fourth, when the workload is fundamentally batch and not streaming. A nightly ETL pipeline ingesting customer-uploaded files, transforming them, and loading into a warehouse is naturally batch. Forcing it into a streaming architecture introduces complexity without benefit. Hybrid architectures (streaming for real-time data + batch for periodic large data) are common.

The pragmatic test: does the data integration matrix have substantial N×M complexity, do you need replay or stream processing, is the team operationally ready? If yes to all three, streaming architecture pays off enormously. If no to any, consider lighter alternatives.

2. Structure

flowchart TB
    subgraph "Source Systems"
        SVC1[Order Service]
        SVC2[Payment Service]
        SVC3[Inventory Service]
        DB1[(Customer DB<br/>via CDC)]
        DB2[(Product DB<br/>via CDC)]
    end
    subgraph "Streaming Substrate"
        K[(Distributed Log<br/>Kafka / Pulsar / Kinesis<br/>partitioned, replicated, durable)]
        SR[Schema Registry<br/>Avro / Protobuf / JSON Schema]
    end
    subgraph "Stream Processing"
        SP1[Flink Job<br/>Real-time Fraud Scoring]
        SP2[Kafka Streams<br/>Joins + Aggregations]
        SP3[Materialize / RisingWave<br/>SQL Streaming]
    end
    subgraph "Derived Systems (Materialized Views)"
        SEARCH[Elasticsearch<br/>Search Index]
        WAREHOUSE[(Snowflake / BigQuery<br/>Analytics Warehouse)]
        FS[Feature Store<br/>ML Features]
        CACHE[Redis<br/>Real-time Lookups]
        DASH[Druid / Pinot<br/>Real-time Dashboards]
    end
    SVC1 -- OrderPlaced --> K
    SVC2 -- PaymentCaptured --> K
    SVC3 -- InventoryReserved --> K
    DB1 -- CustomerChanged --> K
    DB2 -- ProductChanged --> K
    K -. validate .-> SR
    K --> SP1
    K --> SP2
    K --> SP3
    SP1 --> K
    SP2 --> K
    SP3 --> K
    K --> SEARCH
    K --> WAREHOUSE
    K --> FS
    K --> CACHE
    K --> DASH

What this diagram shows. The streaming substrate (Kafka or equivalent) sits at the center as the architectural spine. Source systems on the left publish events to the substrate: services emit domain events (OrderPlaced, PaymentCaptured); legacy databases publish change events via change-data-capture (CDC) tooling like Debezium (which reads the database’s transaction log and emits row-change events to the stream). The schema registry validates event shapes against published schemas, ensuring consumers can decode events safely.

In the middle, stream processors (Flink, Kafka Streams, ksqlDB, Materialize, RisingWave) consume events, perform stateful transformations (joins across streams, time-windowed aggregations, filtering, enrichment), and emit derived events back to the substrate. A fraud-scoring Flink job reads transaction events, joins with user history, scores each transaction, and emits a TransactionScored event back to the stream. A Kafka Streams job aggregates orders by product over a 1-hour rolling window. A SQL streaming engine (Materialize) materializes a view that other systems can query.

On the right, derived systems consume from the stream and build their own materialized representations: Elasticsearch builds a full-text search index; the analytics warehouse loads events for offline analysis; the feature store maintains real-time ML features; Redis caches hot lookups; Druid/Pinot serves real-time dashboards. Each derived system is a materialized view — its content is fully derivable from the stream and can be rebuilt by replaying.

The most important structural property is the inversion of source-of-truth. In a traditional architecture, the order service’s database is the source of truth, and notifications about orders are sent to other systems. In a streaming architecture, the stream of order events is the source of truth; the order service’s local database is just one materialized view (the one used by the order service itself for command-handling). Other services build their own materialized views. The stream is the connective tissue and the canonical record.

A second important property: the architecture is bidirectional in the substrate but one-directional per producer-consumer pair. Stream processors both consume from and publish to the substrate, creating chains of events. But within any one pair, the producer publishes and the consumer reads — there is no synchronous request-response.

A third important property: schema is a first-class architectural concern. The schema registry is not optional; without it, producer-consumer compatibility is unmanaged and the architecture breaks the moment anyone changes an event shape. Avro and Protobuf are standard; JSON Schema is the third option. The compatibility rules (backward-compatible: consumers can read old events; forward-compatible: old consumers can read new events) are operational policy.

3. Core Principles

Principle 1: the stream is the source of truth. The defining inversion. Every fact about the system’s state has a corresponding event in the stream. If a derived store is corrupted or lost, it can be rebuilt by replaying. If a new system needs to be added, it subscribes to the stream and projects. The stream’s retention policy determines how far back history is preserved; for “stream as source of truth” semantics, retention must be effectively unlimited (often achieved via tiered storage to S3, cf. Distributed Log System Design §7.5).

The corollary: there are no hidden writes that bypass the stream. Every state change emits an event. Every service’s database is derived from events the service has published, plus its own command-handling state. Direct writes that don’t emit events are an architectural violation that breaks downstream systems’ ability to keep up. This is why the transactional outbox pattern is non-negotiable: you write your state change and your event in one transaction.

Principle 2: services consume from streams to build their own materialized views. A service does not call another service’s API to get data; it subscribes to the relevant stream and maintains a local materialized view of the data it needs. This is the dramatic shift from synchronous architectures: data is pushed to where it’s needed rather than pulled on demand. Latency drops (no synchronous call); availability improves (consumer can serve cached data even if producer is down); coupling decreases (consumer depends on event schemas, not API endpoints).

The cost: every consumer maintains its own copy of the data, leading to substantial storage redundancy across the platform. This is a deliberate tradeoff — storage is cheap, latency and availability are valuable. Stream-processing frameworks (Kafka Streams, Flink) make managing the local materialized views easier (automatic state stores, RocksDB-backed) but the underlying redundancy is fundamental.

Principle 3: events are immutable, ordered per partition, replayable. The stream’s contract: events are appended in order within a partition (typically keyed by entity ID), are immutable once written, and can be replayed by any consumer that holds an offset. This is the contract that makes everything else work — replayability gives you the ability to rebuild materialized views from scratch; immutability gives you auditability; per-partition ordering lets consumers process related events in causal order.

The constraint: cross-partition ordering is not preserved. Two events with different keys are processed in independent timelines. Consumers that need cross-key ordering must implement it (typically via timestamps, vector clocks, or single-partition processing).

Principle 4: schema evolution is the architecture’s hardest discipline. Events live in the stream forever; consumers built today must continue to work as producers evolve their schemas tomorrow. The schema registry enforces compatibility: typically backward-compatible (new events can be read by old consumers — only optional field additions, no field removals or type changes); forward-compatible (old events can be read by new consumers — only optional field additions, default values for missing fields). Major breaking changes require a new event type with both versions coexisting during a migration window.

This is the operational equivalent of long-lived API design: the event is the API, consumers depend on it, breaking changes break consumers. Teams that adopt streaming architecture without this discipline experience constant breakage.

3.1 Schema Management Deep Dive — Avro, Protobuf, JSON Schema

Schema management is the single operational discipline that most distinguishes a production streaming architecture from a prototype one. Three serialization formats dominate practice — Apache Avro, Google’s Protocol Buffers (Protobuf), and JSON Schema — and the choice has substantial operational consequences far beyond the surface “Avro is binary, JSON is human-readable” tradeoff that schema-management blog posts often stop at.

Apache Avro is the format Kafka was originally designed around (the original Confluent Schema Registry was Avro-only; Avro support in Confluent’s tooling remains the most mature).

Avro’s distinguishing property is that the schema is required to deserialize an event: a producer encodes an event with a schema fingerprint embedded in the message header; a consumer fetches the matching schema from the registry on first sight; from then on, deserialization is fast (the schema fingerprint is a 4-byte ID plus the actual schema fetched once and cached).

The compatibility model is schema-resolution: both writer’s and reader’s schemas are passed to the deserializer, which projects the writer’s data into the reader’s expected shape, supplying defaults for fields the reader expects that the writer didn’t include and dropping fields the writer included that the reader doesn’t expect. This makes backward and forward compatibility explicit and checkable: the Confluent compatibility checker validates a candidate new schema against the registered version and rejects incompatible changes at registration time, before any event is produced under the bad schema.

The operational implication: if compatibility is configured to BACKWARD or FULL, the registry is a hard gate — you cannot accidentally publish a breaking schema.

The downside: Avro requires the registry to be online for every new schema fingerprint (deserialization without registry access fails); Avro’s tooling is JVM-centric (excellent in Java/Scala, less mature in Python/Go/Rust); the binary format is not human-readable, complicating debugging. Avro is the right choice when the team is JVM-heavy, when registry-enforced compatibility is a hard operational requirement, and when binary-payload size matters.

Protocol Buffers (Protobuf) has become the dominant choice in newer streaming architectures (LinkedIn’s newer systems, Google’s internal streaming, many Kubernetes-ecosystem projects).

Protobuf’s compatibility model differs from Avro’s: instead of writer’s-schema-vs-reader’s-schema resolution, Protobuf assumes that both producer and consumer have a compiled schema (the .proto file translated to language-specific code) and that compatibility rules are followed by the schema author.

The rules are simpler: never reuse a field number, never change a field’s type, never make an optional field required. Adding new fields with new field numbers is always safe (old consumers ignore them, new consumers read them); removing fields with reserved markers is safe (the field number is permanently retired); changing field types is unsafe and not detected automatically.

Protobuf’s polyglot tooling is excellent — official support for C++, Java, Python, Go, Rust, JavaScript/TypeScript, C#, and many other languages — making it the natural choice in heterogeneous environments. The Confluent Schema Registry, which had been Avro-only, added Protobuf (and JSON Schema) support in Confluent Platform 5.5 (2020), including compatibility checking based on the rules above (Confluent Platform 5.5 announcement).

The downside: Protobuf compatibility relies more on author discipline (the field-number rule is conventional, not machine-enforced unless the registry is configured to check); the binary format is similarly opaque to humans without the schema; large messages with many optional fields have somewhat larger binary representations than Avro. Protobuf is the right choice for polyglot environments, for teams already using gRPC (which uses Protobuf as the wire format), and where the operational simplicity of “compile the schema, ship the code” outweighs Avro’s resolution machinery.

JSON Schema is the third option, and the choice that most teams reach for first because JSON is universal and immediately debuggable.

JSON Schema does not specify a binary serialization — events are simply JSON documents — and the schema is metadata describing the expected shape (field names, types, required-vs-optional, value constraints). The Confluent Schema Registry has supported JSON Schema since 2020, with compatibility checking using JSON Schema’s own composition rules.

The benefits: JSON is universal, every language has a JSON parser, debugging is trivial (kafkacat | jq shows the event content), and JSON Schema’s expressiveness (oneOf, anyOf, conditional schemas) handles polymorphic event types more naturally than Avro’s union types or Protobuf’s oneof.

The costs: JSON is verbose (a JSON event is typically 3–5× the size of the Avro/Protobuf binary equivalent — substantial at trillion-event scale), parsing is slow (tens of microseconds per event vs sub-microsecond for Avro/Protobuf), and the compatibility model is less battle-tested than Avro’s. JSON Schema is the right choice for low-throughput, debuggability-prioritized, polyglot environments where the verbosity overhead is acceptable and human inspection of events is a daily workflow.

The non-negotiable operational implication. The registry is mandatory, regardless of which format is chosen. Without a schema registry enforcing compatibility, schemas drift out of compatibility on the first day a developer adds a required field “just for one consumer.” With a registry plus a sane compatibility configuration (typically BACKWARD or FULL_TRANSITIVE), schema breakage becomes an alert at publish time rather than a production incident at consume time.

The tooling around all three formats has matured substantially since 2020; the choice between them is increasingly a question of language ecosystem and serialization size rather than of fundamental capability.

A worked compatibility-rule example. A team using Protobuf adds int32 user_age = 8; to an OrderPlaced event; the field number 8 was never used before; old consumers ignore it (Protobuf default); new consumers read it; the registry approves the schema change as backward-compatible.

Six months later, someone tries to change user_age from int32 to string because they want to allow “unknown” — the registry rejects the change because field-type changes are not compatible.

The team adds a new field string user_age_text = 12; and deprecates the old one with a comment, eventually issuing a major-version migration after consumers have been updated.

This is the everyday cadence of schema evolution in a healthy streaming architecture; without the registry, the same change goes through silently and breaks consumers in production.

4. Request Flow

4.1 Source service publishes; multiple downstream materialized views update

sequenceDiagram
    participant API as Order API
    participant DB as Order DB
    participant K as Kafka (Stream)
    participant ES as Elasticsearch Indexer
    participant FS as Feature Store Builder
    participant DASH as Real-time Dashboard

    API->>DB: BEGIN TX; INSERT order; INSERT outbox(OrderPlaced); COMMIT
    Note over API,DB: transactional outbox - event durable with state change
    DB-->>API: 201 Created
    Note over DB,K: separate process: outbox poller / Debezium CDC
    DB->>K: publish OrderPlaced { order_id, user_id, items, total, ts }
    par parallel materialization
        K->>ES: deliver
        ES->>ES: index order for full-text search
    and
        K->>FS: deliver
        FS->>FS: update user-features (orders_in_last_24h, total_spend, ...)
    and
        K->>DASH: deliver
        DASH->>DASH: increment per-product counters in time window
    end

Walk-through. The order API commits the order to its DB along with an outbox row in a single transaction. A separate process — Debezium CDC reading the database WAL, or a pollere reading the outbox table — publishes to Kafka. Three downstream consumers (search index builder, feature store, real-time dashboard) consume the event in parallel. Each builds its own materialized view independently. The order service knows nothing about the consumers.

The user got their 201 Created response after the API’s local transaction committed — milliseconds. The downstream materialization is asynchronous. If the search index is down, the event waits in Kafka until it recovers; the order service is unaffected. If a new consumer is added next week (a partner export), it subscribes to the same stream and starts receiving events — no change to the order service.

4.2 Stream processor: stateful join across streams

sequenceDiagram
    participant K as Kafka
    participant FLINK as Flink Job (Fraud Scoring)
    participant STATE as RocksDB State Store
    participant ALERT as Alert Stream

    K->>FLINK: TransactionEvent { user_id, amount, merchant, ts }
    FLINK->>STATE: load user-history-aggregates(user_id)
    STATE-->>FLINK: { recent_txn_count, avg_amount, distinct_merchants }
    FLINK->>FLINK: score = ml_model(transaction, history)
    FLINK->>STATE: update aggregates (sliding window)
    alt score > threshold
        FLINK->>K: publish TransactionFlagged { user_id, txn_id, score, reasons }
    end
    K->>ALERT: deliver to alerting consumer

Walk-through. A Flink job consumes transaction events, joins each with the user’s recent-history aggregates (held in a state store backed by RocksDB and checkpointed for fault-tolerance — see Carbone et al. 2017 for Flink’s state-management details), scores the transaction with an ML model, updates the aggregates with the new transaction, and emits a TransactionFlagged event back to Kafka if the score exceeds a threshold. Downstream alerting consumers pick up the flagged events.

This is the stream as source of truth in action: the Flink job’s state is derived from the stream and can be rebuilt by replaying; the alerting decisions are themselves events in the stream that other systems can consume. Stream processors and the things they emit are first-class participants in the streaming architecture.

4.3 CDC Worked Example — Bringing a Legacy MySQL Database into the Stream

To anchor the integration regime where the database is the source of truth and a CDC pipeline mirrors changes into the stream, walk through a worked example using Debezium (the canonical open-source CDC connector for Kafka). The setup: a legacy customer-service is a Java application backed by a MySQL 8 database with a customers table; the team cannot modify the application to add an outbox, but downstream services (a CRM analytics warehouse, a marketing-segmentation pipeline, a search index) need real-time updates whenever a customer record changes.

The configuration is roughly: enable MySQL’s binary log (log-bin=mysql-bin, binlog_format=ROW, binlog_row_image=FULL) so every row change is recorded with full before/after images; create a dedicated MySQL replication user with REPLICATION SLAVE and REPLICATION CLIENT privileges; deploy Debezium MySQL connector (running in Kafka Connect, typically as a managed service like Confluent Cloud or AWS MSK Connect) configured with the MySQL connection details and a list of tables to capture (database.include.list=customer_db, table.include.list=customer_db.customers); the connector starts by performing an initial snapshot of the table (a consistent SELECT capturing the current state of every row), publishing each row as a c (create) event to a Kafka topic (dbserver1.customer_db.customers by default), and then transitioning to streaming mode where it tails the binlog and publishes c/u/d (create/update/delete) events for each row change. The events have a structured envelope: { before: {...}, after: {...}, source: { ts_ms, position, file, ... }, op: "u", ts_ms: ... } — the before-image and after-image of the row, the source binlog coordinates (so events can be deduplicated or replayed from a specific position), the operation type, and a timestamp.

Downstream consumers transform these physical row-change events into domain events as appropriate. The CRM analytics warehouse consumes raw CDC events directly into Snowflake via Kafka Connect’s Snowflake Sink, building a near-real-time replica of the customer table for analytics queries. The search index consumer reads the events, transforms them into Elasticsearch documents (selecting only the searchable fields, applying language-specific stemming), and bulk-indexes them. The marketing-segmentation pipeline runs a Kafka Streams job that reads CDC events, maintains a per-customer state store (RocksDB-backed) tracking signup date, last-active date, segment tags; on each update the job evaluates segmentation rules and emits CustomerSegmentChanged events to a separate domain-event topic that other services consume.

The operational realities that often surprise teams:

1. The initial snapshot can take hours for large tables. A 500GB table with 2 billion rows requires hours of consistent SELECT; during this period, the binlog must be retained or the connector cannot transition to streaming mode without re-snapshotting. Configuring sufficient binlog retention (expire_logs_days=7 minimum, often longer) is mandatory.

2. Schema changes in the source database must be coordinated with downstream consumers. A column added to MySQL appears in Debezium events as a new field; consumers expecting the old schema either ignore it (safe — Avro/Protobuf forward compatibility helps) or fail (unsafe). The team owning the database may not even know who consumes the resulting CDC stream — a discovery problem solved by good documentation and registry-enforced contracts.

3. Hard deletes are events too. A DELETE FROM customers WHERE id = 42 produces a d event with the before-image; consumers must handle deletes (search index removes the document, segmentation pipeline forgets the customer) or stale data persists. Soft-delete patterns in the database (where rows are flagged deleted_at rather than removed) emit u events, which consumers must distinguish from regular updates by inspecting the deletion flag.

4. The source field’s binlog coordinates are the audit trail for “did the consumer see every event?” — Debezium guarantees exactly-once production into Kafka via its own offset-management, but consumer applications still need their own offset management to ensure no event is processed twice.

5. Schema evolution in Debezium itself. Debezium’s event envelope has evolved across versions; consumers built against Debezium 1.x may need adjustments for 2.x. The operational discipline is to treat Debezium’s event format as a contract and test consumer code against new Debezium versions in a staging environment before upgrading.

The CDC pattern’s broader place in the streaming architecture: it is the integration bridge that lets streaming architecture extend into systems that were not designed for it. Most large companies that adopted streaming after 2015 have a substantial CDC layer because most of their data already lived in relational databases. Confluent’s customer base, by their own product-marketing claim, is dominated by CDC use cases; LinkedIn’s Brooklin is a similar internal tool for cross-database CDC at scale; AWS Database Migration Service (DMS) provides managed CDC into Kinesis. The pattern is mature, well-tooled, and operationally well-understood as of 2026.

5. Variants

5.1 Pure event streaming (no traditional databases)

The aspirational extreme: every service’s state is fully derived from the stream; there are no traditional CRUD databases anywhere. Service A publishes events; Service B consumes them and builds a materialized view that Service B then serves queries from. There is no “Service A’s database” that’s the source of truth.

In practice this is rare. Most production architectures have a hybrid: services use traditional databases internally for command-handling, emit events via outbox, and consumers materialize views from the stream. The stream is the integration source of truth; per-service databases are the command-handling source of truth.

The reasons “pure streaming” is rare are worth understanding because they illustrate the genuine limits of the architecture. First, command-handling typically requires strongly-consistent reads against current state — to validate “is this user allowed to place this order?” the order service needs to read the user’s current credit limit, which is most cleanly served by a local database that the order service writes to transactionally. Building this from a streaming-only architecture requires the order service to maintain its own materialized view of user state by consuming a user-events stream, with the operational complexity of keeping that view consistent and dealing with the lag between user-events publication and the view becoming queryable. Most teams find it simpler to keep the order service’s own database for command-handling and emit OrderPlaced events for downstream consumption. Second, transactional integrity within an aggregate is much easier in a database than in a streaming-only architecture — a single database transaction that updates the user’s credit limit and inserts an order record is a familiar primitive; the equivalent in pure streaming requires careful design with idempotent event handlers and possibly compensation logic. Third, the cognitive load of pure streaming is genuinely higher than hybrid; teams that have not internalized event-driven thinking find the architecture confusing. The hybrid pattern (per-service database for command-handling + outbox for cross-service events) preserves most of streaming’s benefits (decoupled integration, replayability for derived systems, multi-projection support) while keeping intra-service correctness reasoning conventional. The pure-streaming pattern shows up most in event-sourced + CQRS systems where the team has fully committed to the discipline; even there, the read sides typically use traditional databases populated by stream consumers.

5.2 Streaming + traditional databases (hybrid, most common)

The pragmatic standard. Each service has its own database (often the database it always had); state changes are emitted as events via outbox/CDC. Downstream consumers build materialized views from the stream. The stream is the integration bus and the canonical record of inter-service events; the per-service databases are command-handling stores.

This variant is what most companies actually run when they say “streaming architecture.” It requires the outbox pattern or CDC for every source service.

5.3 Streaming + Event Sourcing (write side is event-sourced)

The write side of services is Event Sourcing Pattern: the event log is the persistence model. Events emitted to other services are the same events stored as the service’s source of truth. Strong coherence between intra-service persistence and inter-service communication.

This variant gives the cleanest semantics but requires teams to commit to event sourcing’s evolution discipline. Often paired with Command Query Responsibility Segregation for the read side.

5.4 Stream + materialized view databases (Materialize, RisingWave)

A newer variant where SQL-streaming databases (Materialize, RisingWave) consume from the stream and maintain materialized views as databases. Application services then query the materialized views via SQL, getting fresh data without writing custom projection logic.

This is a substantial simplification: instead of “every consumer writes its own projection handler,” consumers declare SQL views and the streaming database maintains them. As of 2026 this is gaining adoption; production maturity is still building.

The lineage of this variant is worth tracing because it represents a genuine architectural advance over the “every consumer writes its own projection code” pattern that dominated streaming architectures from roughly 2014 to 2020. Materialize was founded in 2019 by Frank McSherry (one of the lead authors of the Differential Dataflow line of research at Microsoft Research) and others, and is built on top of Differential Dataflow — a Rust-based framework for incremental computation that maintains query results as inputs change with provably minimal recomputation. The architectural contract: declare a SQL view (CREATE MATERIALIZED VIEW orders_per_minute AS SELECT minute, COUNT(*) FROM orders GROUP BY minute); Materialize ingests the underlying stream, maintains the view’s results incrementally, and serves PostgreSQL-protocol queries against the always-fresh result. Application services then query Materialize as if it were a regular PostgreSQL database, getting sub-second latency on results that reflect events processed milliseconds ago. RisingWave is a similar product (Rust-based, also founded ~2021, focused on cloud-native deployment with separated compute and storage); ksqlDB (Confluent’s offering) occupies adjacent space but with somewhat different semantics. The strategic significance: if the streaming-architecture’s projection-maintenance work can be expressed as SQL and outsourced to a managed materialized-view database, the cognitive load on application teams drops substantially — they declare what they want, the database maintains it. The asterisks: Materialize and RisingWave are both younger than Kafka or Flink (production maturity in 2026 is real but less battle-tested than the older substrate); the SQL-streaming abstraction works well for declaratively-expressible transformations (filters, joins, aggregations, time-windowed computations) but not for arbitrary stateful logic (custom ML models, complex event processing); the cost model differs from traditional databases (SQL queries that touch large windows can be expensive to maintain incrementally). The pattern’s right fit: read-side projections that are naturally SQL-shaped (analytics dashboards, real-time aggregations) where the team would otherwise be writing equivalent Kafka Streams or Flink jobs by hand.

5.5 Edge streaming with regional aggregation

For globally-distributed systems, regional Kafka clusters (or equivalent) consume local events, asynchronously replicate to a global stream, and serve regional consumers locally. Tools like LinkedIn’s Brooklin handle the cross-cluster replication. Tradeoffs: local low-latency for regional consumers; cross-region eventual consistency.

The architectural rationale becomes clearer with a concrete example. Consider a globally-distributed e-commerce platform with users and order traffic across North America, Europe, and Asia. A naive single-region Kafka deployment forces every event from every region through one cluster — fine for small platforms, but at scale the cross-region producer latency (300+ ms from Asia to a US-East cluster), the regulatory implications of moving customer data across borders (GDPR data residency, China’s PIPL, India’s DPDP Act), and the blast radius of a single-region cluster outage all argue for regional clusters with selective cross-region replication. The standard pattern: each region runs a primary Kafka cluster serving local producers and consumers; cross-region replication tools (Confluent’s Cluster Linking, the older MirrorMaker 2, LinkedIn’s Brooklin, or Pulsar’s built-in geo-replication) asynchronously copy specific topics to specific other regions based on routing policies. Regional consumers see local events at low latency (single-millisecond) plus globally-replicated events at higher latency (hundreds of milliseconds plus replication lag). The hardest design question is which topics replicate to which regions: replicate everything everywhere → enormous bandwidth and storage cost; replicate only what’s needed → topic-routing policy becomes a complex artifact that must be maintained as new topics are added. Pulsar’s tenants and namespaces feature is partly designed to make this routing policy manageable; Confluent Cluster Linking offers similar abstractions. The operational realities: replication lag is observable but often not bounded (a network partition can stall replication indefinitely; recovery copies the backlog); cross-region consumer groups need careful design (a consumer in region A reading a topic replicated from region B sees events later than a consumer in region B); failover semantics are subtle (if region A goes down, consumers there cut over to region B’s replica — but the replica may be behind, producing apparent event re-delivery on cutover). Multi-region streaming is a substantial operational discipline; teams adopting it typically have a dedicated platform team for the cross-region tooling.

5.6 Lambda Architecture (deprecated by streaming-first thinking)

Marz’s Lambda Architecture (~2011): batch layer (long-running, complete, eventually consistent) + speed layer (real-time, partial, immediately consistent) + serving layer (combines both). Used to be the canonical answer for “we need both batch and real-time.” Kreps’s 2014 essay Questioning the Lambda Architecture argued that the streaming architecture (later called Kappa Architecture) makes Lambda obsolete: a single streaming pipeline can serve both real-time and batch needs by replaying historical events for batch-style processing.

Most new architectures in 2026 are Kappa-style (single streaming pipeline) rather than Lambda; legacy Lambda systems persist where the original choice predates good streaming infrastructure.

6. Real-World Examples

LinkedIn’s Kafka-based architecture. Apache Kafka was created at LinkedIn in 2010–2011 specifically to solve the data integration problem at LinkedIn’s scale. The 2011 NetDB paper (Kreps, Narkhede, Rao) describes the original design; the 2013 The Log essay articulates the architectural philosophy. By 2015 LinkedIn was processing over a trillion messages/day on Kafka. Every major data product at LinkedIn (search, recommendations, ads, news feed) is built on top of the streaming architecture. Brooklin handles cross-cluster replication; Samza (LinkedIn’s stream processor, since deprecated in favor of Kafka Streams and Flink elsewhere) was the original processing layer.

Confluent’s Event Streaming Platform framing. Confluent (founded 2014 by the original Kafka team) has framed event streaming as a complete platform: Kafka as the substrate, Schema Registry for contract management, Kafka Connect for source/sink integration, ksqlDB for SQL-style stream processing, Confluent Cloud for managed deployment. Stopford’s 2018 Designing Event-Driven Systems is the canonical book. Confluent has framed many of the architectural patterns (“stream as source of truth,” “materialized views from streams”) that define the discipline.

Netflix Keystone. Netflix’s Keystone pipeline (2018 blog post, evolving since) ingests roughly a trillion events per day through Kafka and routes to Flink stream-processing jobs, S3 for batch analytics, Elasticsearch for operational search, and Druid for real-time dashboards. Every microservice at Netflix emits events about its state changes; Keystone routes them to whoever cares. The pipeline is a textbook streaming architecture at internet scale.

Uber’s Marketplace architecture. Uber’s matching, dispatch, pricing, and billing flows are built on a streaming architecture (Kafka internally, with Cadence/Temporal handling orchestrated workflows). Trip events, driver-location events, surge-pricing events, and billing events all flow through Kafka. Uber Engineering’s blog has many posts on the architecture.

Pinterest’s data platform. Pinterest uses a Kafka-centric architecture for analytics, ML features, search indexing, and operational events. Their engineering blog describes patterns and challenges at scale.

Walmart Labs. Walmart’s pricing, inventory, and order-management systems use a Kafka-based streaming spine; engineering talks (CodeMash, QCon) describe the architecture and the migration from a monolithic data-platform.

Many fintech and trading platforms. Streaming architecture is dominant in fintech because the audit trail, replayability, and real-time processing align with the domain’s needs. Specific deployments are usually confidential, but Confluent’s customer case studies feature many financial-services examples.

Cloud-native managed offerings. Confluent Cloud, AWS MSK, AWS Kinesis Data Streams, Google Cloud Pub/Sub (with the streaming patterns), Azure Event Hubs — all support the streaming architecture for customers. Adoption in 2026 is broad across SaaS, e-commerce, gaming, IoT, and adtech.

6.1 Production Incidents — What Goes Wrong at Scale

A streaming architecture’s failure modes are operationally distinctive and worth understanding through real public-record incidents. The following are sanitized but match the public engineering record.

The LinkedIn Kafka backup story (the canonical near-disaster). LinkedIn’s published engineering history (multiple blog posts and conference talks 2014–2018) describes a class of operational scares around the early years of Kafka being mission-critical: the entire LinkedIn data platform depended on Kafka, and Kafka’s operational tooling (especially backup, cross-region replication, and disaster recovery) was substantially less mature than its read/write hot path. A specific class of incident: a Kafka cluster’s metadata (held in ZooKeeper at the time, before KRaft) became corrupt or inconsistent during a maintenance window, requiring careful recovery procedures that would not have been straightforward without deep institutional knowledge. The team’s response over years was to invest heavily in operational tooling (Cruise Control for cluster rebalancing, Burrow for consumer-lag monitoring, MirrorMaker for cross-cluster replication, eventually Brooklin for richer cross-cluster data movement) — almost all of which became open-source projects that the wider Kafka ecosystem now relies on. The lesson generalizable to any streaming-architecture adoption: the operational tooling is at least as important as the substrate itself, and the maturity gap between “Kafka is running” and “Kafka is operationally robust” is several years of dedicated investment for any team that does not adopt a mature managed service.

The Square 2018 Kafka misconfiguration outage. Square (now Block) published a public post-mortem describing how a Kafka cluster misconfiguration combined with a bursty load pattern produced a multi-hour outage in 2018. The key chain: a configuration push lowered the per-broker file-handle limit; a routine traffic spike drove the cluster to its file-handle ceiling; brokers began rejecting connections; consumer-side reconnect storms exacerbated load; the cluster’s internal coordination (ZooKeeper-mediated at the time) began to time out, triggering further failovers that piled load on remaining brokers. Square’s engineering response was multi-pronged: configuration rollback, per-broker resource ceilings, throttling at the producer side, dead-letter queues on consumer-side processing failures, and a longer-term migration to managed services for some workloads. The lesson: streaming-architecture’s central-bus property means a cluster-level operational issue affects everything; resilience requires multiple layers of throttling and circuit-breaking (producer-side rate limits, broker-side write quotas, consumer-side bounded concurrency), and the configuration of those layers must be exercised in load-testing because they typically only matter during incidents.

The Robinhood 2020 / 2021 streaming pipeline incidents. Robinhood’s engineering blog has described (in posts 2020-2022) incidents where their streaming pipeline (Kafka-based, feeding fraud detection and trade-surveillance systems) experienced consumer lag during high-volume market events (the 2021 GameStop trading volume spike being the most notorious). The pipeline was sized for normal-day volumes; volume spikes of 10× normal exhausted consumer-side capacity; lag grew into the tens of minutes; downstream fraud-detection signals arrived too late to be useful for real-time intervention. The mitigation pattern that emerged is capacity headroom for streaming pipelines must be sized for tail events, not p50 events — at minimum 5× normal capacity, often 10–20× for systems with extreme tail-load profiles like trading platforms. This is materially different from request-response systems where p99 sizing is normally adequate; in streaming pipelines, a 10× volume spike with capacity sized for 2× spawns lag that takes hours to recover even after the spike subsides.

The “schema breakage cascade” pattern. Multiple smaller-scale public reports describe consumer breakage cascades when a producer team made a schema change that the registry approved as backward-compatible but that the consumers had not been updated to handle. The classic example: a producer adds a new optional enum value (shipping_method=DRONE); the schema registry approves the change because adding values to an enum is technically backward-compatible; downstream consumers using strict enum decoding (some Avro/Protobuf code generators do this by default) crash on the new value because their compiled enum doesn’t include DRONE. The event is at the head of the partition; the consumer crashes; the consumer restarts; the consumer crashes on the same event; the lag grows. The mitigation is consumer-side discipline (treat unknown enum values as default/unknown rather than crashing), schema-registry policies that distinguish “wire-compatible” from “code-compatible,” and CI tests that exercise consumers against the latest producer schemas. This pattern shows up roughly quarterly in any organization with substantial streaming adoption.

The “exactly-once that wasn’t” production embarrassments. Kafka’s transactions (introduced in Kafka 0.11, 2017) provide exactly-once semantics for the producer-broker boundary and, with care, for stream-processing pipelines that read-process-write entirely within Kafka. They do not provide exactly-once semantics for arbitrary consumer side effects (sending an email, calling an external API, writing to an external database). Multiple post-mortems describe teams that misread Kafka’s exactly-once claim as end-to-end exactly-once, built consumer code that did not implement idempotent side effects, and discovered duplicate emails / duplicate charges / duplicate notifications during a partition rebalance or broker failover. The architectural lesson: at-least-once + idempotency is the universal regime; exactly-once at the broker boundary is a useful refinement for specific stream-processing pipelines but does not propagate to external side effects without consumer-side idempotency.

7. Tradeoffs

Choice	Pro	Con	When chosen
Streaming spine vs point-to-point integration	O(N) integration; replayable; new consumers cheap	Operational complexity of substrate; schema discipline	Heterogeneous integration at scale
Stream as source of truth	Replay; multi-projection; auditability	Hidden-write discipline; outbox required	Audit-mandated, multi-derived-system
Hybrid (per-service DBs + streams)	Pragmatic; minimal disruption	Outbox per service	Most production deployments
Pure streaming (no DBs)	Maximum coherence	Operationally heavy; cognitive load	Rare; aspirational
Kafka	Mature; ecosystem; ubiquity	Operational tax	Default in 2026
Pulsar	Compute/storage separation; geo-replication	Two-layer ops	Very large or geo-distributed
Kinesis	Managed; simple	AWS-locked; less ecosystem	AWS-native shops
Schema Registry + Avro/Protobuf	Compatibility enforcement	Coordination overhead	Non-negotiable for production
CDC from databases (Debezium)	Bring legacy DBs into the stream	DB log details leak; primary-key drift	Integrating existing databases
Outbox pattern	At-least-once event publication	Per-service implementation	Required for streaming-architecture services
Stream processing (Flink, Kafka Streams)	Stateful real-time processing	Operational and cognitive overhead	Real-time analytics, ML features
Materialized-view DB (Materialize)	SQL queries on streams	Younger ecosystem	Where SQL is preferred
Tiered storage	Effectively unlimited retention	Cold-read latency	Multi-year history needs

8. Migration Path

Migrating from a non-streaming architecture to event streaming:

Step 1: Pick the substrate. For most teams in 2026: Apache Kafka via a managed service (Confluent Cloud, AWS MSK, Aiven). Self-managed Kafka if you have the operational team. Kinesis for AWS-only shops. Pulsar for elastic scale or geo-replication needs. The selection criteria worth weighing: (a) ecosystem breadth — Kafka has by far the largest ecosystem of connectors, stream-processing frameworks, and tooling, which compounds the value of choosing it for a long-term architectural commitment; (b) operational model — Kafka’s broker-based architecture is well-understood, while Pulsar’s broker-plus-BookKeeper separation of compute and storage is more elastic but operationally heavier; (c) cloud-vendor coupling — Kinesis is AWS-only and the managed integration is excellent for AWS-native shops, but the ecosystem outside AWS is sparse; (d) team experience — adopting whatever the team has run before is usually worth a substantial premium over learning a new substrate. Most teams in 2026 land on Kafka unless one of the alternative’s specific strengths (Kinesis’s AWS-native simplicity, Pulsar’s geo-replication, Redpanda’s lower operational footprint) decisively outweighs Kafka’s ecosystem advantage.

Step 2: Stand up the schema registry. Confluent Schema Registry, Apicurio, or AWS Glue Schema Registry. Define your compatibility policy from day one (typically backward + forward compatible).

Step 3: Pick the first integration scenario. A high-pain N×M integration — for example, “five different systems need updates whenever an order is placed.” Start there.

Step 4: Implement transactional outbox in the source service. Add an outbox table, modify the service to write to it transactionally, deploy Debezium or a poller to publish from the outbox to Kafka. The transactional outbox pattern (popularized in Chris Richardson’s Microservices Patterns, 2018, and in microservices.io) works as follows: in the same database transaction that updates business state (UPDATE orders SET status='placed'), insert a row into an outbox table with the event payload and metadata; commit the transaction; a separate process — either an outbox poller (reads WHERE published=false, publishes to Kafka, marks rows as published) or Debezium reading the database’s WAL and emitting outbox-row inserts to Kafka — moves the events into the streaming substrate. The pattern’s correctness property: state and event are atomically committed together; either both happen (database write succeeded → event will eventually publish) or neither does (database write failed → no event). The pattern’s operational property: the poller or Debezium is asynchronous, so events publish slightly after the state change (typically milliseconds, but configurable) — this is the unavoidable cost of transactional consistency without distributed transactions. The pattern’s typical pitfalls: (1) outbox table grows unbounded if cleanup is not implemented; (2) the poller falls behind during traffic spikes and lag accumulates silently without alerting; (3) failure of the publish step is hidden if the published flag is set optimistically rather than after broker acknowledgment. Each of these has known mitigations (TTL-based cleanup, lag dashboards with hard alerts, post-ack flag-setting) but must be implemented explicitly.

Step 5: Define the event schema. Avro or Protobuf, registered in the schema registry. Get the schema right — this is the contract that everyone depends on.

Step 6: Migrate the first consumer. Build a consumer that subscribes to the stream and maintains the relevant materialized view (or feeds a downstream system via Kafka Connect to Elasticsearch, etc.). Validate against the legacy integration; cut over.

Step 7: Migrate more consumers. Each new consumer is incremental work; the substrate, schema, and producer are already in place.

Step 8: Add stream processing where valuable. If real-time aggregations, joins, or ML features are needed, layer in Flink or Kafka Streams jobs.

Step 9: Migrate more producers. As the architecture proves itself, migrate other services to publish via outbox. The N×M integration matrix gradually collapses.

Step 10: Implement CDC for legacy data sources. Databases that aren’t easily modified to emit events can have Debezium running against their transaction logs, bringing them into the stream.

The Strangler Fig pattern (cf. Strangler Fig Pattern) is the discipline: legacy and streaming coexist; migration proceeds source-by-source and consumer-by-consumer; never a Big Bang.

9. Pitfalls

Pitfall 1: skipping the outbox pattern. “We’ll just write the event to Kafka right after we update the database.” But what if the Kafka write fails after the DB commit (or vice versa)? Now state and events are out of sync. Always use transactional outbox or CDC.

Pitfall 2: skipping the schema registry. “JSON is fine, we’ll just be careful.” Then someone adds a field; consumers crash; emergency rollback. Schema enforcement is non-negotiable.

Pitfall 3: schema evolution without compatibility rules. Events from 6 months ago are no longer parseable; rebuild from history is impossible. Compatibility rules + automated checking in CI.

Pitfall 4: events as commands. Naming events SendEmail or ChargeCard rather than OrderPlaced re-introduces tight coupling. Past-tense, domain-language names.

Pitfall 5: hidden writes that bypass the stream. Someone writes directly to a downstream system without emitting an event. The stream is no longer the complete record; replay can’t rebuild that downstream. This is an architectural violation; have automated checks.

Pitfall 6: “we’ll just buy a managed service and ignore the architecture.” Managed services reduce ops burden but don’t change the architectural decisions. You still need outbox, schema registry discipline, idempotent consumers, etc.

Pitfall 7: cross-stream consistency assumed. Events from different streams (different topics) have no global ordering. Code that assumes “stream A’s event N came before stream B’s event M” is wrong.

Pitfall 8: consumer rebalance hiccups. Consumer groups rebalance when consumers join/leave; older Kafka versions had stop-the-world rebalances that crashed p99 latency. Cooperative incremental rebalancing (Kafka 2.4+, KIP-429) mitigates; configure properly.

Pitfall 9: the firehose problem. Every consumer wants to consume every event; the substrate becomes overloaded. Mitigations: per-consumer partitioning (consumers subscribe to specific topics); content-based filtering (consumers receive only relevant events via predicates); hierarchical fanout.

Pitfall 10: legacy systems hard to integrate. A legacy Oracle DB with no transaction-log access (or no Oracle GoldenGate / equivalent licensed). CDC requires log access; without it, you fall back to dual-write (with the consistency problems that brings) or polling (with the latency).

Pitfall 11: stream processor state explosion. A Flink job’s state grows unbounded (windowed aggregates with no eviction; user-state for users that never come back). State store grows; checkpointing slows; eventual failure. State eviction policies must be designed.

Pitfall 12: forgetting at-least-once → idempotency requirement. Default delivery is at-least-once; consumers must dedupe. Idempotency at every consumer. The full picture: Kafka’s exactly-once semantics, introduced in KIP-98 / Kafka 0.11 (2017), guarantee that messages are not duplicated between Kafka producers and Kafka brokers and that read-process-write pipelines that stay within Kafka transactions can avoid duplicates. They do not magically extend exactly-once to arbitrary consumer side effects — sending an email, writing to an external database, calling a third-party API. The way to achieve effective exactly-once for external side effects is consumer-side idempotency: each event carries a unique ID; the consumer maintains a “processed” ledger (in a database, in Redis, in a Kafka compacted topic); duplicate events are detected and skipped. The cost of idempotency machinery is real (storage, lookup latency on every event) but unavoidable. Teams that read “Kafka supports exactly-once” and skip the idempotency layer discover the gap when their first consumer rebalance or broker failover triggers replay of a few minutes of events and customers receive duplicate confirmation emails.

Pitfall 12.1: late-arriving events and watermarking. A subtler correctness issue specific to stream processing with windowed aggregations. Real-time pipelines aggregate events over time windows (“count orders in this 1-minute tumbling window”); events ideally arrive in timestamp order, but in reality network delays, consumer rebalancing, and out-of-order partition processing cause events to arrive late — sometimes minutes or hours after their nominal timestamp. The question: when a 1-minute window closes at 12:01:00, do you emit the result immediately (and risk missing late-arriving events from the 12:00–12:01 window), or wait some grace period (and accept latency)? The framework-level answer is watermarks: a watermark is a timestamp that the stream processor emits saying “I believe I have seen all events with timestamps ≤ W”; windows are closed when the watermark advances past their end. Apache Flink, Apache Beam (Akidau et al. 2015 Dataflow Model paper), Kafka Streams, and Spark Structured Streaming all implement variants of watermark-based windowing. The operational realities: watermarks are heuristic (the processor cannot truly know when “all” events have arrived); allowed-lateness configuration (Flink’s allowedLateness, Beam’s late-firings) controls how long after window close late events can still update results; very-late events typically go to a side output for separate handling. Teams that ignore the watermark question silently produce incorrect aggregates whenever events are out of order, which on a real network is constantly. The Akidau Dataflow paper is the canonical reference and worth reading in full for any team building serious stream-processing pipelines.

Pitfall 12.2: clock skew across producers. Related to watermarking: events carry a timestamp, but whose clock set the timestamp? If producers use their own local clocks and clock skew across producers is bounded only by NTP synchronization (~tens of milliseconds typical, but sometimes seconds during NTP failures), then “ordered by timestamp” is approximately ordered, not strictly ordered. Stream processors that assume strict ordering produce subtly wrong results on cross-producer joins. Mitigations: use Kafka’s broker-assigned timestamps (LogAppendTime) for ordering rather than producer timestamps; use logical timestamps (vector clocks, Lamport clocks) for cross-producer causality; tolerate some out-of-order processing via watermarks. There is no single right answer; the question must be addressed explicitly at architecture-design time.

Pitfall 13: the “stream as queue” misunderstanding. A team treats Kafka as a queue: write event, consume, delete. Misses the entire point of the architecture. Streams retain history; consumers can have independent offsets; replay is a feature.

Pitfall 14: schema overspecification. Events with 100 fields, most optional, used differently by different consumers. The events become a junk drawer. Keep events focused on the producer’s domain change; consumers project what they need.

Pitfall 15: insufficient retention. Retention set to 7 days; a consumer needs to backfill from 30 days ago; can’t. With tiered storage, retention can be effectively unlimited; without it, plan retention conservatively.

Pitfall 16: missing observability. What’s the consumer lag? What’s the schema-version distribution? Where are events being dropped? Without metrics, the streaming architecture is opaque. Observability tooling (Prometheus + Grafana for Kafka metrics, Confluent Control Center, Kafka UI tools like Kowl/Redpanda Console) is mandatory.

10. Comparison with Sibling Architectures

Streaming vs Event-Driven Architecture (EDA). Event-Driven Architecture is the broader umbrella: services communicate via events. Streaming architecture is a durable, replayable implementation of EDA — a specific way to build an EDA on top of a Distributed Log System Design. EDA can also be built on transient pub-sub (Redis, NATS Core) without durability or replay; that’s “lite EDA.” Streaming gives you EDA plus the materialized-view and replay properties.

Streaming vs Pub-Sub (Publish Subscribe System Design). Pub-sub is the message-delivery primitive (one publish, multiple subscribers). Streaming is the architectural style that uses durable, replayable pub-sub as the system spine. Pub-sub on transient infrastructure (Redis Pub/Sub, NATS Core) is not a streaming architecture; pub-sub on durable infrastructure (Kafka, Pulsar) can underpin one.

Streaming vs Event Sourcing (Event Sourcing Pattern). Event sourcing is intra-service persistence (events are the source of truth for a single aggregate). Streaming is inter-service architecture (events are the source of truth for the whole platform). They compose: a service can be event-sourced internally (with its own per-aggregate event store) and also emit events to a streaming spine for inter-service communication.

Streaming vs CQRS (Command Query Responsibility Segregation). CQRS separates write and read models. Streaming provides the substrate for feeding the read models from the write side. They compose naturally: CQRS read sides are typically materialized from streams.

Streaming vs Lambda Architecture (Lambda Architecture). The Lambda Architecture, named by Nathan Marz around 2011 (Big Data: Principles and Best Practices of Scalable Realtime Data Systems, Manning 2015), proposed splitting a data platform into two parallel pipelines:

• A batch layer that processes the full historical dataset on a long cadence (hours to days) producing complete, eventually-consistent views.
• A speed layer that processes only recent events on a short cadence (seconds to minutes) producing partial, approximate views.
• A serving layer that combines them at query time.

The architectural argument for Lambda was that batch processing was the only way to reliably compute correct aggregates over history (because batch frameworks like Hadoop were operationally mature), while the speed layer filled the gap for real-time queries. The pattern was widely adopted at companies like Twitter (in the early 2010s), LinkedIn, and Yahoo, and was the canonical answer to “we need both batch and real-time” through roughly 2014.

Jay Kreps’s 2014 essay Questioning the Lambda Architecture made the pointed counter-argument that Lambda’s two-pipeline structure imposed 2× the engineering cost on every analytics use case (build the logic twice, debug it twice, keep the two implementations in sync) without commensurate benefit if the streaming substrate was operationally mature.

Kreps coined the name Kappa Architecture in that very essay — and notably did so tentatively, writing “Maybe we could call this the Kappa Architecture, though it may be too simple of an idea to merit a Greek letter” (Kreps 2014). The idea: a single streaming pipeline with sufficient retention that “batch reprocessing” becomes “rewind the stream and run the streaming job from scratch.” Kreps’s concrete recipe in the essay is to retain data in Kafka (he uses 30 days as the example), and when the processing code changes, start a second instance of the job reading from the beginning of the retained data, write its output to a new table, and cut over to that table once the new job has caught up to head — then tear down the old job and table. There is exactly one framework, one codebase, and one operational substrate, which is the whole point of the argument against Lambda’s two-pipeline cost.

The same pipeline serves both the streaming-from-now use case and the batch-style reprocessing use case; there is no two-implementations cost; there is only one operational substrate.

The Kappa approach has become dominant in new architectures since roughly 2017, enabled by mature streaming frameworks (Apache Flink especially), Kafka’s tiered storage extending retention to effectively unlimited, and the ergonomic improvements of stream-processing APIs.

Lambda persists where (1) legacy batch infrastructure (Hadoop / Spark batch) was already in place when streaming arrived and would be expensive to retire; (2) the workload is fundamentally batch (nightly customer-data file ingest) and forcing it into streaming would be unnatural; (3) regulatory or auditing requirements specifically mandate batch-style “complete view at end-of-day” computations.

The pragmatic 2026 stance: prefer Kappa for new platforms, treat surviving Lambda deployments as legacy migrations, and accept hybrid patterns (Kappa for real-time-derived data; periodic batch jobs for low-latency-insensitive aggregations like nightly billing reconciliation) where they are operationally simpler than forcing everything through one pipeline. The full debate is captured in Kreps’s essay, in Marz’s response (Marz has continued to advocate for Lambda’s strengths in specific contexts), and in the broader streaming-vs-batch literature.

Streaming vs ETL / data warehouse architectures. Traditional ETL: nightly jobs extract from source databases, transform, load into warehouse. Streaming: events flow continuously into the warehouse via Kafka Connect; “ETL” is replaced by ELT (load raw events, transform with SQL streaming or in-warehouse). Latency drops from hours to seconds.

Streaming vs Service Mesh (Service Mesh System Design). Service mesh handles synchronous service-to-service communication (mTLS, traffic management, observability for HTTP/gRPC). Streaming handles asynchronous service-to-many-services event flow. They are orthogonal and often coexist: synchronous user-facing paths use the mesh; downstream fan-out uses the stream.

11. Common Interview Discussion Points

• “What is event streaming architecture?” Architectural style where a durable, replayable, ordered event log (Kafka or equivalent) is the central nervous system; services publish events about state changes; downstream systems consume and build materialized views; the stream is the source of truth.
• “How does it differ from event-driven architecture?” EDA is the umbrella style; streaming is a durable, replayable implementation. EDA can be built on transient pub-sub; streaming requires a durable log.
• “Why is the stream the source of truth?” Replayability gives you free history; rebuildable derived systems; multi-projection support; auditability. Traditional architecture has a single DB as source of truth and the integration story is O(N²); streaming makes integration O(N).
• “What’s the role of a schema registry?” Enforce compatibility between producers and consumers; make event evolution safe; prevent the “I added a field and broke 12 downstream consumers” disaster. Use Avro or Protobuf with backward+forward compatibility rules.
• “How do you bring legacy databases into the stream?” Change-data-capture (CDC) tools like Debezium read the database transaction log and emit row-change events to Kafka. Bring legacy data into the streaming architecture without modifying the legacy application.
• “How do you handle schema evolution over years?” Strict compatibility rules (backward+forward); only optional field additions; major changes via new event types with both versions coexisting; eventually retire old events.
• “What’s the relationship between streaming and event sourcing?” Streaming is the inter-service substrate; event sourcing is the intra-service persistence pattern. They compose: event-sourced services can emit to streams naturally.
• “What’s the relationship between streaming and CQRS?” CQRS read models are typically materialized from streams. Streaming is the substrate that feeds CQRS read sides.
• “How does streaming compare to traditional ETL?” ETL is batch (nightly extract-transform-load); streaming is continuous (event-by-event). Latency drops from hours to seconds. Both can coexist for hybrid workloads.
• “How do you handle exactly-once semantics?” At the substrate level, Kafka transactions provide exactly-once at the broker boundary. End-to-end exactly-once requires idempotent consumers. Pragmatically: at-least-once + idempotency is universal.
• “What’s the role of stream processors like Flink?” Stateful real-time processing: joins across streams, time-windowed aggregations, ML feature computation, complex event processing. They consume from the stream, maintain state, and emit derived events back.
• “How do you scale a streaming architecture?” Partition topics for parallelism; multiple brokers in a cluster; tiered storage for retention; managed services for operational scale. Cross-region via async replication (MirrorMaker, Confluent Cluster Linking).
• “What are the operational challenges?” Schema management; consumer lag monitoring; rebalance configuration; broker tuning; tiered storage management; cross-cluster replication. Mitigations: managed services; mature tooling.

12. Open Questions / Uncertain

Uncertain

Verify: whether managed materialized-view databases (Materialize, RisingWave, ksqlDB) will broadly replace hand-written application-level projection handlers. Reason: this is a forward-looking adoption prediction, not a settled fact — the products exist and work (verified: Materialize is built on Differential Dataflow, founded 2019; RisingWave ~2021; Confluent CP 5.5 shipped ksqlDB integration), but production maturity for the most demanding stateful workloads in 2026 is still emerging and there is no authoritative measurement of adoption share. To resolve: an industry adoption survey, or vendor-published production-scale case studies, over the next 1–2 years. #uncertain

Uncertain

Verify: best-practice patterns for very-long-retention streaming (5+ years of history retained via tiered storage). Reason: tiered storage GA’d only recently (KIP-405 reached GA in Kafka 3.9, Nov 2024 — see Distributed Log System Design §7.5), so multi-year-retention operational experience is thin; schema-evolution discipline over that horizon, query performance against cold tiers, and cost controls are still being worked out at vendors. To resolve: published multi-year tiered-storage case studies and maturing vendor guidance. #uncertain

Uncertain

Verify: whether the boundary between a “streaming substrate” and an “operational/streaming database” (Materialize, RisingWave, Confluent’s Tableflow) will collapse into a unified abstraction that obsolesces parts of this architecture’s current shape. Reason: a speculative architectural-trend prediction; the convergence is visible but its endpoint is not determined. To resolve: time and the market — track whether one abstraction demonstrably subsumes the others. #uncertain

• Will Kafka remain dominant, or will Pulsar’s compute/storage separation become the norm at very large scale?
• For multi-region geo-distributed streaming architectures, what is the right consistency model? Async replication is current standard; stronger guarantees require careful design.
• How does streaming architecture interact with serverless / function-as-a-service compute? Trigger-based consumers are workable; stateful stream processing on serverless is harder.
• Where is the right boundary between “use a stream” and “use a synchronous API”? Per-call decisions remain craft.
• Will SQL-streaming engines (Materialize, RisingWave, Snowflake Streaming, BigQuery Continuous Queries) eventually consolidate into a single dominant abstraction, or persist as a fragmented ecosystem?

13. See Also

• System Architectures MOC
• SWE Interview Preparation MOC
• Distributed Log System Design — the substrate (Kafka, Pulsar, Kinesis); read this first
• Event-Driven Architecture — broader umbrella; streaming is one durable implementation
• Event Sourcing Pattern — composes naturally; intra-service persistence on the same substrate
• Command Query Responsibility Segregation — read sides fed by streams
• Saga Pattern — choreographed sagas use streams for coordination
• Publish Subscribe System Design — pub-sub primitive that streaming subsumes via consumer groups
• Message Queue System Design — sibling messaging primitive; queues vs streams
• Webhook Delivery System Design — outbound webhook delivery often built on streams
• Distributed Task Scheduler System Design — workflow engines that read from / write to streams
• Real Time Analytics System Design — natural consumer of streams
• Microservices Architecture — streaming architecture is a typical microservices spine
• Lambda Architecture — older batch+speed dual-pipeline; mostly superseded by streaming
• Kappa Architecture — Kreps’s name for the streaming-first alternative to Lambda
• Strangler Fig Pattern — migration discipline
• Distributed Tracing System Design — observability across the streaming pipeline
• CRDTs Basics — convergent state replication; alternative to projection rebuild for some workloads
• Vector Clocks — causality across streams
• Two-Phase Commit — synchronous alternative; what streaming displaces for inter-service coordination
• B+ Tree — what traditional DBs use; contrast with append-only streaming substrate
• LSM Tree — append-friendly storage that aligns with streaming write profile