vault

KubeEdge

14 min read

KubeEdge

KubeEdge is a CNCF project that extends Kubernetes-native container orchestration, scheduling, and device management out to edge nodes — devices and gateways that sit far from the datacenter, on slow or intermittent network links, and that must keep running when the link to the cloud is gone (kubeedge.io, Kubernetes blog 2019). It was initiated by Huawei Cloud, accepted into the CNCF as a sandbox project on 2019-03-18, promoted to incubating on 2020-09-16, and graduated on 2024-09-11 (announced 2024-10-15) — making it the first edge-computing project to reach CNCF graduated tier (CNCF graduation announcement, CNCF project page). The architecture splits into two halves: CloudCore, which runs inside a normal Kubernetes cluster in the cloud or datacenter, and EdgeCore, a single lightweight binary (memory footprint on the order of ~70 MiB) that runs on each edge node. The defining property is edge autonomy: EdgeCore caches the desired state in a local SQLite store, so workloads keep running — and can even be restarted — through cloud-edge network partitions, with state re-syncing on reconnect. This is precisely what stock Kubernetes cannot do, because the kubelet assumes a reliable, low-latency connection to the kube-apiserver.

Mental Model

KubeEdge is best understood as Kubernetes with the cloud-edge link cut into an explicit, asynchronous, durable channel. In a normal cluster the kubelet holds an open watch against the apiserver and assumes it is always reachable; lose that connection for more than ~40 s and the node lifecycle controller marks the Kubernetes Node NotReady and starts evicting Pods (see Node Lifecycle Management). At the edge, a connection lapse of minutes or hours is normal operation, not a failure — so KubeEdge inserts a message bus (WebSocket or QUIC) between cloud and edge, and a local metadata store on the edge so the edge node never needs the cloud to keep doing its job.

flowchart TB
    subgraph CLOUD["Cloud / Datacenter Kubernetes cluster"]
        APIS[kube-apiserver]
        subgraph CC["CloudCore (Deployment)"]
            EC[EdgeController]
            DC[DeviceController]
            CH[CloudHub<br/>WebSocket / QUIC server]
        end
        APIS <--> EC
        APIS <--> DC
        EC <--> CH
        DC <--> CH
    end

    CH <-. "intermittent WAN link<br/>(may be down for hours)" .-> EH

    subgraph EDGE["Edge Node (gateway / device, resource-constrained)"]
        subgraph ECORE["EdgeCore (single ~70 MiB binary)"]
            EH[EdgeHub<br/>WebSocket / QUIC client]
            MM["MetaManager<br/>+ local SQLite store"]
            ED["Edged<br/>(lightweight kubelet-equivalent)"]
            EB[EventBus<br/>MQTT client]
            DT[DeviceTwin]
        end
        EH <--> MM
        MM <--> ED
        ED -->|CRI| RT["container runtime<br/>(containerd / etc.)"]
        EB <--> DT
        DT <--> MM
    end

    EB <-. MQTT .-> MQTT["MQTT broker<br/>(mosquitto)"]
    MQTT <-. MQTT .-> DEV["Physical devices<br/>(sensors, actuators)"]

What this diagram shows. The cloud half is an ordinary Kubernetes cluster plus the CloudCore Deployment. The edge half is a constrained machine running only EdgeCore. The single link between them — CloudHub ↔ EdgeHub — is explicitly drawn as intermittent. The insight to extract: everything the edge node needs to keep running (Pod specs, ConfigMaps, Secrets, device desired-state) is mirrored into MetaManager’s SQLite store, so when the dashed link is down, Edged keeps reconciling against the cached desired state. Devices below the edge node speak MQTT to a local broker, decoupled from the WAN entirely.

Mechanical Walk-through

CloudCore runs as a Deployment in the cloud cluster and contains three controllers/modules:

  • EdgeController — the bridge between the apiserver and edge nodes. It watches Pods, ConfigMaps, Secrets, and Node objects scheduled to edge nodes and pushes the relevant slices down through CloudHub; it also writes edge-reported status (Pod status, node status) back into the apiserver. It is essentially an extended node controller that speaks the KubeEdge message protocol instead of the native list-watch.
  • DeviceController — the bridge for IoT devices. It watches the Device and DeviceModel custom resources (below) and synchronizes device desired-state to the edge and device reported-state back to the cloud.
  • CloudHub — a WebSocket (or QUIC) server. It watches changes on the cloud side, caches messages, and ships them to the EdgeHub on each connected edge node. Caching matters: if an edge node is offline, CloudHub holds the pending messages and delivers them on reconnect.

EdgeCore runs on each edge node as one binary with several internal modules (Giant Swarm, Octopus Deploy):

  • EdgeHub — the WebSocket/QUIC client, counterpart to CloudHub. It syncs cloud-side resource updates down to the edge and reports edge host and device status changes up to the cloud. It owns the reconnect logic.
  • Edged — a lightweight kubelet-equivalent. It manages the lifecycle of containerized applications on the edge node: it talks to a CRI-compliant container runtime (see Container Runtime Interface) to pull images and run containers, applies Pod specs, runs probes, and reports Pod status. It is not the upstream kubelet — it is a slimmed-down agent that takes its desired state from MetaManager rather than from a live apiserver watch.
  • MetaManager — the message processor between Edged and EdgeHub, and the component that makes autonomy work. It stores and retrieves Kubernetes metadata (Pod specs, ConfigMap and Secret contents, node state) in a lightweight local SQLite database. When the WAN link drops, Edged keeps reading desired state from MetaManager’s SQLite store; when the link returns, MetaManager syncs the delta.
  • EventBus — an MQTT client that connects to a local MQTT broker (typically mosquitto). It is how the edge node talks to physical devices, offering publish/subscribe to the other EdgeCore modules.
  • DeviceTwin — a software mirror (“twin”) of each connected physical device. It stores device attributes and desired/reported state, syncs that to the cloud via DeviceController, and exposes a query interface to edge applications. Built on the same local SQLite store.
  • ServiceBus — an HTTP client that lets cloud-side applications reach HTTP servers running on the edge node.

These modules do not call each other directly; they communicate over Beehive, KubeEdge’s internal Go-channel-based messaging framework, which gives every module a send / receive / sync-response mailbox abstraction (KubeEdge MetaManager docs). The practical consequence is that EdgeHub can broadcast a “cloud connected” or “cloud disconnected” event to the other modules, letting MetaManager flip between serving reads from the cloud versus serving them from its local SQLite cache — the mechanical hinge on which edge autonomy turns.

The autonomy flow: a Pod scheduled to an edge node flows apiserver → EdgeController → CloudHub → EdgeHub → MetaManager (which writes the spec to SQLite) → Edged (which runs it). If the link then drops, Edged keeps reconciling against the SQLite-cached spec — and critically, if a container crashes, Edged can restart it from the local spec without ever contacting the cloud. Pods on a disconnected edge node are not rescheduled elsewhere the way a NotReady node’s Pods would be in stock Kubernetes; the cloud knows the node is “offline-but-expected” and leaves its workloads in place.

Device management uses two CRDs registered in the cloud cluster:

  • DeviceModel — a reusable template describing a class of device: its properties (e.g., a temperature sensor exposes a temperature property of type float with a unit and range), and the protocol to talk to it.
  • Device — an instance of a DeviceModel, bound to a specific edge node, carrying the device’s desired and reported property values.

DeviceController watches these CRDs; DeviceTwin on the edge keeps the twin in sync; EventBus carries the actual MQTT traffic to/from the hardware. Newer KubeEdge versions formalize the edge-to-device protocol layer as DMI (Device Management Interface), a gRPC contract analogous to CRI/CNI/CSI but for device mappers (CNCF DMI blog 2022).

Configuration / API Surface

A DeviceModel and a Device instance in the current devices.kubeedge.io/v1beta1 API group. The structure below matches the v1beta1 CRD schema shipped in the KubeEdge repository and documented on the official site (KubeEdge — Device CRDs, devices_v1beta1_device.yaml on GitHub):

apiVersion: devices.kubeedge.io/v1beta1
kind: DeviceModel
metadata:
  name: temperature-sensor-model
  namespace: default
spec:
  properties:
    - name: temperature                 # the property the twin tracks
      description: ambient temperature in Celsius
      type: INT                         # v1beta1: a flat scalar type tag (INT/FLOAT/STRING/...)
      accessMode: ReadOnly              # sensor: read-only; an actuator would be ReadWrite
      minimum: "-40"
      maximum: "125"
      unit: "Celsius"
  protocol: modbus                      # the protocol family the mapper speaks
---
apiVersion: devices.kubeedge.io/v1beta1
kind: Device
metadata:
  name: rack-7-temp-sensor              # one physical sensor
  namespace: default
spec:
  deviceModelRef:
    name: temperature-sensor-model      # binds to the model above
  nodeName: edge-node-rack-7            # pins the device to a specific edge node
  properties:
    - name: temperature
      collectCycle: 10000000000         # how often the mapper polls the device (ns)
      reportCycle: 10000000000          # how often the mapper reports upward (ns)
      reportToCloud: true               # mirror this property's value to the cloud twin
      desired:
        value: "22"                     # operator-set desired value (cloud → edge)
      visitors:                         # protocol-specific "how to read this property"
        protocolName: modbus
        configData: {}
  protocol:
    protocolName: modbus                # v1beta1: protocol is {protocolName, configData}
    configData: {}                      # free-form per-protocol connection config
status:
  twins:
    - propertyName: temperature
      reported:
        value: "22.4"                   # last value the device reported (cloud-visible)

Line-by-line, and noting where v1beta1 differs from the older v1alpha2 form so the schema is unambiguous. DeviceModel.spec.properties[] declares the schema of a device class; in v1beta1 each property is flat — name, description, a scalar type tag (INT, FLOAT, STRING, …), accessMode (which distinguishes a ReadOnly sensor from a ReadWrite actuator), and minimum/maximum/unit directly on the property. This replaced the older v1alpha2 layout that nested those fields inside a type: { int: {...} } block; the simplification was part of the v1beta1 rework that also dropped the DeviceModel-as-ConfigMap handling (Device CRDs v1beta1). DeviceModel.spec.protocol names the protocol family (e.g. modbus). On the instance, Device.spec.deviceModelRef binds to the model; nodeName pins the device to one edge node so DeviceController routes its traffic to the right EdgeCore. Device.spec.properties[] carries the per-instance runtime knobs — collectCycle/reportCycle (poll and report cadence, in nanoseconds), reportToCloud, the desired.value (operator intent pushed down), and a visitors block that tells the mapper how to read that property over the wire. Device.spec.protocol in v1beta1 is the generic { protocolName, configData } pair (not a per-protocol key like the old protocol: { modbus: {...} }), with configData holding free-form connection settings. Finally Device.status.twins[].reported.value is the device twin — the cloud-visible mirror of the hardware’s last reported state, updated by DeviceTwin via EventBus/MQTT; the twin also carries a parallel desired value, making status.twins the reconciliation record between operator intent and device reality.

To register an edge node, KubeEdge ships the keadm CLI: keadm init sets up CloudCore in the cloud cluster, and keadm join --cloudcore-ipport=<ip>:10000 --token=<token> installs and starts EdgeCore on an edge machine, which then appears as a Node object in the cloud apiserver.

Failure Modes

Long disconnection then a flood on reconnect. When an edge node has been offline for a long time, the reconnect triggers a burst: CloudHub delivers all queued messages, and EdgeHub reports a backlog of status updates. On large fleets of edge nodes reconnecting simultaneously (e.g., after a regional WAN outage), CloudCore can be overwhelmed. Mitigation is to scale CloudCore and rate-limit, but it is a real operational edge case.

SQLite store corruption on the edge. MetaManager’s autonomy depends on the local SQLite database. An unclean power loss on a constrained edge device (no battery, no graceful shutdown) can corrupt it; EdgeCore then loses its cached desired state and the node must re-sync fully from the cloud — which is impossible if the WAN is also down. Edge hardware with stable power or journaling filesystems mitigates this.

Stale desired state during a long partition. While disconnected, the edge node runs whatever desired state it last cached. If the cloud changed a Deployment during the partition, the edge node is running an old version until reconnect. This is inherent to the model — eventual consistency with an unbounded staleness window — and applications at the edge must tolerate it.

Device mapper / MQTT broker failure. If the local mosquitto broker dies, EventBus loses contact with physical devices even though the edge node itself is healthy. Device twins go stale; the symptom is Device.status not updating while Pods run fine.

Version skew between CloudCore and EdgeCore. KubeEdge has its own compatibility matrix layered on top of Kubernetes’ own version skew. A CloudCore far ahead of an EdgeCore can break the message protocol; upgrading thousands of intermittently-connected edge nodes is itself a hard Day-2 problem.

Alternatives and When to Choose Them

  • Stock Kubernetes with k3s at the edge (Lightweight Kubernetes Distributions) — k3s is a small (~50–100 MiB) full Kubernetes distribution. It runs fine on edge hardware, but it is still Kubernetes: a k3s agent node assumes connectivity to its k3s server, and a k3s server assumes its datastore is reachable. Run an isolated k3s cluster at each edge site and it is autonomous, but you then have N independent clusters with no unified cloud control plane. KubeEdge’s distinguishing feature is exactly that: one cloud control plane, many autonomous edge nodes, with autonomy built into the node agent rather than achieved by giving each site its own full cluster.
  • k0s + remote-controller patterns — similar trade-off to k3s; small, but connectivity-assuming.
  • Akri — a CNCF project for discovering and exposing leaf devices (cameras, sensors) as Kubernetes resources; it solves device discovery, not edge-node autonomy, and is often used alongside, not instead of, an edge framework.
  • OpenYurt — a sibling CNCF edge project (also originating from Alibaba) with a similar “extend upstream Kubernetes to the edge with offline autonomy” goal but a different design (it keeps the real upstream kubelet and adds a per-node caching proxy, YurtHub). Choose OpenYurt if you want to stay closer to vanilla kubelet semantics; choose KubeEdge if you want first-class IoT device management and the smallest edge footprint.
  • AWS IoT Greengrass / Azure IoT Edge — cloud-vendor edge runtimes. Choose these when fully bought into a single cloud’s IoT stack; choose KubeEdge for a cloud-neutral, Kubernetes-native model.

Why ordinary Kubernetes does not work at the edge. The kubelet’s design assumes a reliable, low-latency apiserver connection: it holds a live watch, it relies on the apiserver as its source of truth on every reconcile, and the control plane evicts its Pods within ~40 s of a heartbeat lapse. Edge environments violate all three assumptions — bandwidth is scarce and metered, connectivity is intermittent by design, and a “disconnected” node is healthy, not failed. KubeEdge’s answer is to replace the kubelet with Edged + MetaManager, decoupling the edge node’s reconcile loop from live cloud connectivity.

Production Notes

KubeEdge’s graduation due diligence (announced 2024-10-15, after the project’s maturity level formally moved to Graduated on 2024-09-11) cited a project that had grown to over 1,600 contributors from more than 35 countries and 110 organizations, with maintainers from 15 organizations (CNCF graduation announcement). On the technical-scale side, the graduation materials highlight a tested capability of supporting on the order of 100,000 edge nodes from a single cloud control plane, and name production deployments across CDN, intelligent transportation, smart energy, smart retail, automobiles, logistics, finance, and power — including what KubeEdge describes as “the largest cloud-native cloud-edge collaborative highway toll-station management project” and a cloud-native electric-vehicle deployment (KubeEdge graduation blog). The recurring operational themes from those write-ups: (1) the cloud-edge link is the scarce resource — keep the message protocol lean and rate-limit reconnect storms; (2) edge hardware diversity (ARM boards, x86 gateways) makes multi-architecture image management a first-order concern; (3) upgrading the edge fleet is the dominant Day-2 cost, because edge nodes are physically remote, intermittently connected, and numerous — a far harder problem than upgrading a datacenter cluster’s worker nodes (see Kubernetes Cluster Upgrade).

The “100,000 edge nodes” figure is a tested-scale claim from KubeEdge’s own scalability testing rather than a count of a single live cluster; named adopters (highway tolling, EV) are described qualitatively without published per-deployment node/device counts. The contributor and country/organization counts above are the precise figures stated in the CNCF graduation announcement.

See Also

Graph View

Backlinks