vault

Kubernetes as a Distributed Monolith

9 min read

Kubernetes as a Distributed Monolith

The distributed monolith is the anti-pattern of adopting Kubernetes and microservices without the organizational and architectural maturity that microservices demand — you split the monolith into N deployable services, but they remain synchronously, tightly coupled: every request fans out into a chain of blocking calls, every service must be deployed in lockstep with its peers, and any one service being down takes the whole flow down. The result is the worst of both worlds: you keep the monolith’s coupling (a change in one place forces changes everywhere) while adding the network’s failure modes (latency, partial failure, partitions) and the operational cost of running, observing, and securing N services instead of one (Gremlin — is your microservice a distributed monolith?). Kubernetes does not cause this — but Kubernetes is the platform on which it is almost always built, because “K8s is for microservices” is the cultural pressure that drives the premature split.

Mental Model

flowchart LR
    subgraph DM["Distributed monolith — synchronous fan-out"]
        A[API gateway] -->|blocking| B[Service A]
        B -->|blocking| C[Service B]
        C -->|blocking| D[Service C]
        D -->|blocking| E[Service D]
        E -->|blocking| F[(DB)]
    end
    DM -.->|"latency stacks: p99 = Σ p99<br/>any node down ⇒ whole flow down<br/>deploys must be coordinated"| BAD[Worst of both worlds]

What this diagram shows. A single user request entering the API gateway fans out into a chain of synchronous, blocking service-to-service calls. The two failure properties to extract: (1) latency stacks — the end-to-end p99 is the sum of every hop’s p99, not the max, because each call blocks on the next; and (2) availability multiplies down — if each of four services has 99.9% availability, the chained flow has 0.999⁴ ≈ 99.6%, almost 4× the downtime of a single monolith at 99.9%. The insight: a synchronous call chain is less available and slower than the monolith it replaced, while costing more to operate. You have distributed the code without distributing the failure domains — every service is still a single point of failure for the whole flow.

Mechanical Walk-through — Anatomy of the Anti-Pattern

Why it’s tempting. The decision looks reasonable at every step. Kubernetes documentation, conference talks, and the entire cloud-native ecosystem are framed around microservices — “K8s is the microservices platform” is received wisdom. There is org pressure to “modernize” and “break up the monolith.” Splitting along the lines of the existing code structure (the users module becomes the users service, the orders module becomes the orders service) feels like the natural decomposition. And the first service extraction works — it deploys, it serves traffic. The trap is that the seams chosen were code-organization seams, not bounded-context seams: the modules were never independent, they just looked separable in the source tree.

What you actually built. Because the services were carved along false seams, they retain every dependency the modules had:

  • Synchronous call chains. A request that used to be an in-process function call is now a network round-trip. A flow that touched four modules now makes four blocking HTTP/gRPC calls. The in-process call cost nanoseconds and could not fail; the network call costs milliseconds and can fail, time out, or be slow.
  • Lockstep deploys. Because the services share data contracts that were never versioned (they were just function signatures), a change to one service’s API forces a coordinated, simultaneous deploy of every caller. The “independently deployable” promise of microservices is gone — you have a distributed deploy of a single logical unit.
  • A shared database. Often the services all still talk to the same database (the monolith’s database, untouched). Now you have N services contending on one schema, with no service actually owning its data — the textbook distributed-monolith smell.
  • Temporal coupling. Service A cannot complete its work until Service B responds, which cannot complete until Service C responds. The services are temporally coupled even if the code is physically separated.

Worked failure scenario

A retail company splits its monolith into web, catalog, pricing, inventory, and checkout services on Kubernetes. The checkout flow synchronously calls all four others. One afternoon, pricing has a slow database query and its p99 latency climbs from 50 ms to 4 s. Because checkout blocks on pricing, every checkout request now takes 4+ s. The checkout Pods’ worker threads are all blocked waiting on pricing, so checkout exhausts its thread pool and starts failing readiness probes. The kubelet removes checkout Pods from the Service. The web service, which calls checkout, now sees checkout as down — and the entire site is offline because one service had a slow query. In the original monolith, a slow pricing query would have slowed pricing pages; here it took down checkout and the homepage. The blast radius grew when the system was “modernized.” This is the canonical cascading failure of the distributed monolith.

Configuration / API Surface — The Wrong Way and the Right Way

The anti-pattern is architectural, not a single YAML field — but it has a manifest signature. The wrong way: every service’s Deployment lists every other service as an environment-variable dependency, and the deploy pipeline applies all of them together.

# WRONG — checkout hard-wired to call four peers synchronously, deployed in lockstep
apiVersion: apps/v1
kind: Deployment
metadata: { name: checkout }
spec:
  template:
    spec:
      containers:
        - name: checkout
          env:
            - { name: CATALOG_URL,   value: http://catalog }     # blocking dependency
            - { name: PRICING_URL,   value: http://pricing }     # blocking dependency
            - { name: INVENTORY_URL, value: http://inventory }   # blocking dependency
            # checkout cannot serve a single request unless ALL THREE are up

The right way is not a different manifest — it is a different interaction style. Where the call must stay synchronous, make it resilient (timeout, retry budget, circuit breaker, fallback); where it can be made asynchronous, do so:

# BETTER — checkout publishes an event; downstream services consume at their own pace
# checkout writes an "OrderPlaced" event to a durable queue and returns immediately.
# inventory/pricing consume asynchronously. checkout's availability no longer
# depends on inventory/pricing being up at request time.
apiVersion: apps/v1
kind: Deployment
metadata: { name: checkout }
spec:
  template:
    spec:
      containers:
        - name: checkout
          env:
            - { name: EVENT_BUS_URL, value: kafka://orders-events }  # async boundary
            # the ONLY synchronous dependency is the durable bus, which is HA

The point is not “Kafka fixes everything” — it is that the failure domain boundary must be drawn deliberately. A synchronous call couples the caller’s availability to the callee’s; an asynchronous boundary (queue, event log) decouples them. A distributed monolith is a system with no such boundaries.

Failure Modes

  • Cascading failure. One slow or down service propagates failure up the entire call chain — the worked scenario above. The blast radius of a single-service incident is the whole flow.
  • Latency stacking. End-to-end latency is the sum of every synchronous hop. Each network hop adds serialization, transport, and deserialization cost that the in-process monolith never paid.
  • Lockstep deploys. Services cannot be released independently; a change ripples across N repos and N pipelines. The team has more coordination overhead than the monolith, not less.
  • Distributed debugging. A single user-facing bug now spans N services, N log streams, and N teams. Without distributed tracing it is nearly undebuggable — and tracing is itself an operational cost the monolith did not have.
  • N× operational cost. N services means N sets of dashboards, alerts, on-call rotations, security boundaries, CI pipelines, and dependency upgrades. This cost is real and recurring whether or not the decomposition delivered any benefit.
  • The “it works in dev” trap. In dev, all services run on one laptop with sub-millisecond loopback latency and no failures — the distributed monolith looks fine. Production is where the network’s latency and partial failure surface.

The Correct Alternative

  1. Monolith-first (Fowler — MonolithFirst). Start with a modular monolith. You do not yet know where the real bounded-context seams are; building microservices before you know the boundaries guarantees you draw them wrong. Fowler’s observation: nearly every successful microservices system was evolved from a monolith, and nearly every system built as microservices from scratch ran into serious trouble.
  2. Split on real bounded-context seams. When you do split, split along domain boundaries (the Bounded Context of Domain-Driven Design), not along source-tree modules. A correctly drawn service owns its data, has a versioned contract, and can be deployed and reasoned about independently. See Microservices Architecture.
  3. Async by default at service boundaries. Where two services interact, prefer events/messaging over synchronous request-response. A synchronous call couples availability; an event does not. Reserve synchronous calls for genuinely synchronous needs (a read the user is waiting on) and make those calls resilient — timeouts, circuit breakers, bulkheads.
  4. Use the Strangler Fig Pattern to extract incrementally. Do not big-bang the decomposition. Route a slice of traffic to a new extracted service behind a façade, prove the seam, and migrate incrementally — keeping the monolith working the entire time.
  5. Earn microservices. Microservices demand mature CI/CD, observability (tracing, structured logs, metrics), and a team-per-service org structure. If you do not have those, microservices on Kubernetes will be slower and less reliable than the monolith. Adopt the maturity first.

Production Notes

  • The Gremlin chaos-engineering team frames the diagnostic test crisply: if you cannot deploy one service without deploying others, if services share a database, or if a single service outage takes down unrelated functionality — you have a distributed monolith regardless of how many Deployment objects you have.
  • The recurring industry pattern (chronicled across the microservices-anti-pattern literature) is a team that “did microservices” by containerizing the monolith’s modules, shipped to Kubernetes, and then spent the next year fighting cascading outages — eventually either recombining services or doing the bounded-context work that should have come first.
  • Kubernetes’ own overview docs describe K8s as a container orchestrator — it runs containers, monoliths included. Nothing in Kubernetes requires or rewards microservices. A modular monolith on Kubernetes (one Deployment, HPA, rolling updates, health probes) is a perfectly legitimate and often better architecture than a premature distributed monolith.
  • The honest summary: Kubernetes makes it cheap to run many services, and that cheapness is exactly the trap — it lowers the friction of splitting before the friction of splitting was telling you not to.

See Also

Graph View