vault

Multus

14 min read

Multus

Multus (from multus-cni, the GitHub project k8snetworkplumbingwg/multus-cni) is a meta-CNI — a CNI plugin that delegates to other CNI plugins in order to attach multiple network interfaces to a single Pod (github.com/k8snetworkplumbingwg/multus-cni). The default Kubernetes networking model assumes one network namespace per Pod with one eth0 reaching the cluster network (provisioned by the cluster’s primary CNI: Calico, Cilium, AWS VPC CNI, etc.); Multus lifts that assumption by chaining additional CNI invocations after the primary one, producing extra interfaces named net1, net2, … inside the Pod, each on a distinct underlying network. The canonical motivating use case is the telco/NFV (Network Function Virtualization) and 5G CNF (Containerized Network Function) world, where a Pod typically needs (a) a control-plane interface on the cluster network for kubelet/probes/cluster traffic, (b) a data-plane interface for high-throughput packet processing via SR-IOV or DPDK, and (c) potentially a management interface on a separate L2 broadcast domain — three independent networks that no single CNI can deliver simultaneously. Multus is maintained by the Kubernetes Network Plumbing Working Group (github.com/k8snetworkplumbingwg) — a community group that also stewards the Network Custom Resource Definition De-facto Standard (multi-net-spec), the CRD schema Multus uses. The configuration model: cluster operators define NetworkAttachmentDefinition CRDs (group k8s.cni.cncf.io/v1) that wrap CNI JSON configs, and workload owners reference them via a Pod annotation k8s.v1.cni.cncf.io/networks. This note covers Multus’ architecture, the NAD CRD, the thick vs thin deployment models, the telco use cases that drive its development, and the limits and gotchas of multi-homed Pods. See Container Network Interface for the spec Multus implements (and delegates to), Pod Networking for the per-Pod plumbing, and the cluster CNI it sits on top of (Calico, Cilium, etc.) for the “primary” interface.

Mental Model

flowchart TB
    subgraph POD["Pod (telco CNF)"]
        ETH0[eth0<br/>10.244.1.5<br/>cluster network<br/>control plane / mgmt]
        NET1[net1<br/>192.168.10.5<br/>macvlan on external VLAN<br/>data plane]
        NET2[net2<br/>192.168.20.5<br/>SR-IOV virtual function<br/>high-throughput packet I/O]
    end
    subgraph NODE["Node CNI stack"]
        MULTUS[/opt/cni/bin/multus<br/>meta-plugin, called by containerd]
        PRIMARY[/opt/cni/bin/calico<br/>(or cilium, aws-cni, etc.)<br/>primary cluster CNI]
        MACVLAN[/opt/cni/bin/macvlan]
        SRIOV[/opt/cni/bin/sriov]
    end
    subgraph K8S["Kubernetes API"]
        NAD1[NetworkAttachmentDefinition<br/>name: macvlan-vlan10<br/>config: { type: macvlan, master: eth1.10, ipam: {...} }]
        NAD2[NetworkAttachmentDefinition<br/>name: sriov-pool-a<br/>config: { type: sriov, resourceName: intel.com/sriov_a, ipam: {...} }]
        POD_SPEC[Pod annotations:<br/>k8s.v1.cni.cncf.io/networks:<br/>macvlan-vlan10, sriov-pool-a]
    end
    MULTUS -- "1. read pod annotation,<br/>find primary CNI from<br/>multus config" --> PRIMARY
    PRIMARY -- "creates eth0" --> ETH0
    MULTUS -- "2. read NAD via K8s API" --> NAD1
    MULTUS -- "3. invoke macvlan CNI" --> MACVLAN
    MACVLAN -- "creates net1" --> NET1
    MULTUS -- "4. read NAD via K8s API" --> NAD2
    MULTUS -- "5. invoke sriov CNI" --> SRIOV
    SRIOV -- "creates net2" --> NET2

What this diagram shows. Multus is invoked by the container runtime (containerd/CRI-O) as the configured CNI plugin — operators replace the cluster’s main CNI plugin binary in /etc/cni/net.d/ with a Multus config that points at Multus itself. Inside Multus, the meta-plugin first invokes the primary CNI (the cluster’s actual CNI, identified in Multus’ own config as clusterNetwork or defaultNetwork) to set up the Pod’s default eth0 on the standard cluster network. Then Multus reads the Pod’s k8s.v1.cni.cncf.io/networks annotation, looks up each named NetworkAttachmentDefinition via the Kubernetes API, and for each one invokes the CNI plugin specified inside the NAD’s config field (macvlan, sriov, bridge, host-device, ipvlan, ovs, etc.) to create additional interfaces named net1, net2, … inside the Pod’s namespace. The Pod ends up with three NICs, each on a distinct underlying network with distinct IPAM, distinct MTU, and distinct routing — exactly what telco/NFV workloads need. The insight to extract: Multus is purely an orchestrator; it does no networking itself. It is a thin layer that turns “one CNI call per Pod” into “N CNI calls per Pod, configured declaratively via CRDs.”

Mechanical Walk-through

Why a meta-CNI exists at all

A single CNI plugin call produces a single interface inside the Pod namespace. The CNI spec (Container Network Interface) is one-Pod-one-network by design — the spec authors deliberately punted on multi-interface use cases. But three categories of real-world workload need multiple interfaces:

  1. Telecom NFV / 5G CNFs. A 5G User Plane Function (UPF) Pod needs (a) a cluster network for management traffic and (b) one or more SR-IOV VFs or DPDK-backed interfaces for line-rate packet processing across separate L2 segments. The data-plane interfaces must bypass the kernel network stack for throughput; the control-plane interface uses the standard CNI.
  2. Network appliances on K8s. Software firewalls, load balancers, VPN gateways — each needs distinct interfaces for “inside” and “outside” networks.
  3. Compliance / segregation. Some banking and gov use cases mandate separation of management traffic from data traffic at the L2 level, not just at the policy level.

The Kubernetes Network Plumbing Working Group’s Network Custom Resource Definition De-facto Standard (github.com/k8snetworkplumbingwg/multi-net-spec) formalized the configuration shape — a CRD called NetworkAttachmentDefinition plus a Pod annotation — and Multus is the reference (and overwhelmingly dominant) implementation.

The NetworkAttachmentDefinition CRD

A NetworkAttachmentDefinition (NAD) is a namespaced CRD in API group k8s.cni.cncf.io/v1 that wraps a CNI configuration JSON blob (multi-net-spec):

apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
  name: macvlan-vlan10
  namespace: telco-workloads
spec:
  config: '{
    "cniVersion": "1.0.0",
    "type": "macvlan",
    "master": "eth1.10",
    "mode": "bridge",
    "ipam": {
      "type": "host-local",
      "subnet": "192.168.10.0/24",
      "rangeStart": "192.168.10.100",
      "rangeEnd": "192.168.10.200",
      "routes": [{"dst": "0.0.0.0/0"}],
      "gateway": "192.168.10.1"
    }
  }'

The spec.config field is a CNI config JSON string — the same format that would live in /etc/cni/net.d/ for a single-network CNI deployment. Multus reads the NAD, parses the JSON, and invokes the named CNI binary (macvlan here, looked up in /opt/cni/bin/).

The Pod annotation

Pods select which NADs to attach via the annotation k8s.v1.cni.cncf.io/networks:

apiVersion: v1
kind: Pod
metadata:
  name: telco-cnf
  namespace: telco-workloads
  annotations:
    k8s.v1.cni.cncf.io/networks: macvlan-vlan10, sriov-pool-a
spec:
  containers:
    - name: cnf
      image: registry.example.com/cnf:v2

The annotation is a comma-separated list of NAD references; the default form is just the NAD name (resolved in the Pod’s own namespace). Cross-namespace references use the form <namespace>/<nad-name>. The richer JSON form lets operators override the interface name (net1 vs data0) and IPAM:

annotations:
  k8s.v1.cni.cncf.io/networks: |
    [
      {"name": "macvlan-vlan10", "interface": "data0"},
      {"name": "sriov-pool-a", "interface": "data1", "ips": ["192.168.20.5/24"]}
    ]

Thick plugin vs thin plugin

Multus has two deployment models (github.com/k8snetworkplumbingwg/multus-cni — Thick Plugin):

  • Thin plugin (original): The /opt/cni/bin/multus binary is a one-shot CNI plugin invoked per Pod by containerd. It reads its config, calls each delegate CNI, and exits. Simple, lightweight, but every invocation re-reads NADs from the API server (caching is minimal), so high-Pod-churn nodes can rate-limit themselves against the API server.

  • Thick plugin (recommended): A daemon (multus-daemon) runs on every node, watching NADs and Pods via the K8s API and caching state. The CNI binary (multus-shim) is now a thin client that talks to the daemon over a Unix socket. Adds metrics, faster invocation, and lower API server load — at the cost of running an additional process per node.

The thick model is recommended for production by both Red Hat and the project maintainers.

Maintainer and project status

Multus is maintained by the Kubernetes Network Plumbing Working Group, a vendor-neutral community group originally seeded by Intel and Red Hat. The dominant downstream consumers are:

  • Red Hat OpenShift — Multus is bundled by default; OpenShift’s “additional networks” feature is Multus.
  • Telco distributions — Mirantis Kubernetes Engine, Nokia, Ericsson, Wind River, all bundle Multus.
  • The CNCF Telecom User Group — Multus is the de facto standard mechanism for telco CNFs.

A common point of confusion is whether Multus is a CNCF project. It is not. Verified against the CNCF Projects page (as of May 2026), Multus appears in no CNCF tier — not graduated, not incubating, not sandbox. It is a community project hosted under the Kubernetes Network Plumbing Working Group, which sits in the Kubernetes SIG-Network orbit but is not itself a separately-hosted CNCF project. The genuinely CNCF-hosted networking projects in this neighborhood are the Container Network Interface (CNI) specification (incubating, accepted 2017-05-23) — the spec Multus both implements and delegates to — and Cilium (graduated, accepted 2021-10-13), one of the primary CNIs Multus can sit on top of. So “Multus is CNCF” is a frequent but incorrect claim; the accurate statement is “Multus builds on the CNCF-incubating CNI spec.”

The current Multus release is v4.2.4, cut 2025-02-18 (github.com/k8snetworkplumbingwg/multus-cni — Releases). That release added support for the CNI STATUS verb and other fixes targeting CNI spec 1.1.0, and was built against Go 1.24 with Kubernetes 1.34 compatibility — useful context that Multus tracks the CNI spec’s own evolution closely, since its whole job is to invoke CNI plugins faithfully.

What is not in scope for Multus

  • Multus does not provide its own dataplane. All actual networking is delegated to other CNIs.
  • Multus does not enforce NetworkPolicy on secondary interfaces. Standard Kubernetes NetworkPolicy only applies to the primary eth0. NetworkPolicy for secondary interfaces requires the MultiNetworkPolicy extension (a separate project under the same working group) — not covered by the core Multus CNI.
  • Multus does not bypass the primary CNI. Every Pod still gets its primary eth0 from the cluster’s main CNI; Multus adds additional interfaces.
  • Multus is not a Service-load-balancing layer. Kubernetes Services and kube-proxy operate on the primary network only.

Configuration / API Surface

A complete end-to-end deployment:

# 1. Install Multus via the project's manifests (thick model)
# kubectl apply -f https://github.com/k8snetworkplumbingwg/multus-cni/raw/master/deployments/multus-daemonset-thick.yml
# This rewrites /etc/cni/net.d/00-multus.conf on every node and runs multus-daemon
#
# Multus's own CNI config (auto-generated on each node):
# /etc/cni/net.d/00-multus.conf
{
  "cniVersion": "1.0.0",
  "name": "multus-cni-network",
  "type": "multus",                                  # the binary in /opt/cni/bin/
  "capabilities": { "portMappings": true },
  "delegates": [
    {                                                # the cluster's primary CNI
      "cniVersion": "1.0.0",                         # (e.g., Calico's config copied in)
      "name": "k8s-pod-network",
      "type": "calico",
      "datastore_type": "kubernetes",
      "ipam": { "type": "calico-ipam" }
    }
  ],
  "kubeconfig": "/etc/cni/net.d/multus.d/multus.kubeconfig"
}
 
---
# 2. Define an additional network (a macvlan attached to host's eth1.10)
apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
  name: macvlan-vlan10
  namespace: telco
spec:
  config: |
    {
      "cniVersion": "1.0.0",
      "type": "macvlan",                             # delegate CNI binary
      "master": "eth1.10",                           # host parent interface
      "mode": "bridge",                              # macvlan mode
      "ipam": {
        "type": "whereabouts",                       # cluster-wide IPAM CRD
        "range": "192.168.10.0/24",
        "exclude": ["192.168.10.1/32", "192.168.10.254/32"]
      }
    }
 
---
# 3. A Pod requesting two networks: cluster (default) + macvlan-vlan10
apiVersion: v1
kind: Pod
metadata:
  name: telco-cnf
  namespace: telco
  annotations:
    k8s.v1.cni.cncf.io/networks: macvlan-vlan10     # space-separated NAD names
spec:
  containers:
    - name: cnf
      image: registry.example.com/cnf:v2
      securityContext:
        capabilities:
          add: ["NET_ADMIN", "NET_RAW"]              # often needed for DPDK / raw sockets

After the Pod starts, inside it:

$ kubectl -n telco exec telco-cnf -- ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> ...
3: eth0@if102: <BROADCAST,MULTICAST,UP,LOWER_UP> ...
   inet 10.244.1.42/24 scope global eth0            # cluster network via Calico
4: net1@if78: <BROADCAST,MULTICAST,UP,LOWER_UP> ...
   inet 192.168.10.123/24 scope global net1         # macvlan on VLAN 10
 
$ kubectl -n telco get pod telco-cnf -o jsonpath='{.metadata.annotations.k8s\.v1\.cni\.cncf\.io/network-status}' | jq
# Multus writes back a status annotation with per-network details:
[
  {
    "name": "calico",
    "interface": "eth0",
    "ips": ["10.244.1.42"],
    "mac": "1a:2b:3c:4d:5e:6f",
    "default": true,
    "dns": {}
  },
  {
    "name": "telco/macvlan-vlan10",
    "interface": "net1",
    "ips": ["192.168.10.123"],
    "mac": "0e:8f:73:a2:b1:c4",
    "dns": {}
  }
]

Failure Modes

  1. Pod stuck ContainerCreating with “failed to attach network”. Most common cause: NAD referenced in the annotation doesn’t exist in the Pod’s namespace. Diagnostic: kubectl describe pod shows the Multus error; kubectl get net-attach-def -n <namespace> should list the NAD. Fix: create the NAD or correct the annotation.

  2. NAD references a CNI binary that’s not installed. Multus invokes the binary named in the NAD’s config.type from /opt/cni/bin/. If macvlan, sriov, etc. aren’t deployed on every node, Pod creation fails on those nodes. Fix: deploy the reference CNI plugins DaemonSet (containernetworking/plugins) on every node, or use NodeSelectors to restrict Multus-using Pods to nodes where the plugins exist.

  3. SR-IOV device plugin not allocating VFs. SR-IOV requires (a) the SR-IOV CNI plugin, (b) the SR-IOV Device Plugin DaemonSet, and (c) the kernel’s SR-IOV driver to be configured on each node. Symptom: Pod stuck Pending with Insufficient resource: intel.com/sriov_a. Fix: deploy the SR-IOV Network Operator, verify VFs are visible (ip link show <PF>), check device plugin logs.

  4. NetworkPolicy enforced on cluster network but not on macvlan/sriov interfaces. As noted, core NetworkPolicy applies only to the primary network. Sensitive workloads on multi-network Pods may inadvertently leak via secondary interfaces. Fix: install MultiNetworkPolicy or apply L2 segmentation at the underlying VLAN level.

  5. Pod-to-Pod traffic on the same macvlan VLAN can’t reach each other through the host. macvlan has a long-standing limitation: the parent interface (master) cannot communicate with its own macvlan sub-interfaces. Two Pods on the same node sharing a macvlan can talk to each other, but a Pod cannot reach a service running on the node itself via the macvlan. Fix: use mode: bridge (default) for Pod-to-Pod, accept that Pod-to-host on the macvlan is broken; for Pod-to-host, use the cluster network or a separate veth.

  6. Thick plugin daemon crash. multus-daemon crashes mean no new Pods can attach (CNI ADD via shim fails). Symptom: Pods stuck ContainerCreating on a specific node. Fix: investigate daemon logs; the thick plugin’s RBAC and node-local CR cache are the usual culprits.

  7. whereabouts IPAM exhaustion across nodes. whereabouts is the canonical cluster-scoped IPAM for Multus-attached networks. If the IP range is too small, allocation fails across nodes. Fix: enlarge the range or add exclude clauses for IPs in use elsewhere.

Alternatives and When to Choose Them

  • Single-CNI deployment. Default Kubernetes. Choose unless you specifically need multiple interfaces per Pod. The overwhelming majority of workloads do not.
  • hostNetwork: true. Bypasses Pod networking entirely; Pod uses the node’s network namespace. Useful for single-interface-but-direct cases (CNI agents, monitoring daemons). Loses Pod IP semantics and Service VIPs.
  • DANM (Damascene Multi-Network). Nokia-led alternative to Multus from the same era; smaller community, mostly used in telco. Has converged with Multus in practice.
  • DRA (Dynamic Resource Allocation). The longer-term direction for network attachment: a Kubernetes-native resource model for device allocation that can include network interfaces. DRA graduated to GA in Kubernetes 1.34 (released September 2025), with the stable API resource.k8s.io/v1 enabled by default (Kubernetes blog — DRA GA in v1.34) — so it is no longer an experimental future, it is a shipped, stable mechanism. The DRANET project on GKE applies DRA specifically to network interfaces. Multus may eventually be supplanted by DRA-based attachment for some use cases, but Multus + NAD remains the dominant production pattern as of mid-2026, and the two are expected to coexist (see Production Notes).
  • Pod’s primary CNI providing rich features. If only one extra interface is needed (e.g., a service-mesh sidecar’s interception), the primary CNI may suffice via per-Pod configuration (e.g., Cilium’s CiliumEndpoint).

Production Notes

  • OpenShift ships Multus on by default. Red Hat’s OpenShift bundles Multus and the cluster-network-operator to manage NADs declaratively at install time. Operators familiar with OpenShift’s multi-network UX are using Multus whether or not they realize it.
  • The dominant non-OpenShift consumer is the telco/CNF world. AT&T, T-Mobile, Verizon, NTT, Deutsche Telekom — all run Multus in production for 5G CNFs. The Linux Foundation’s Anuket / OPNFV reference architectures specify Multus + SR-IOV CNI + DPDK as the standard NFV networking stack.
  • whereabouts solves the IPAM-across-nodes problem. The reference host-local IPAM keeps a per-node JSON file; it cannot prevent two nodes from assigning the same IP to two Pods on the same macvlan VLAN. whereabouts (also under the network-plumbing-wg) is a CRD-backed cluster-scoped IPAM that solves this; it’s the recommended IPAM for any cluster-wide Multus-managed network.
  • MultiNetworkPolicy is alpha and rarely used in practice. Most multi-network Pods rely on the underlying physical network’s segmentation (VLANs, ACLs on the leaf switches) rather than K8s-layer policy on secondary interfaces.
  • Performance: secondary interfaces can be high-throughput. SR-IOV VFs achieve ~100 Gbps line rate with DPDK; macvlan is roughly host-NIC-equivalent. The primary cluster CNI’s overlay overhead does not apply to secondary networks.
  • DRA + Multus coexistence is the near-term roadmap. Both can attach interfaces to a Pod; with DRA now GA (Kubernetes 1.34, Sept 2025), the Kubernetes community is working on coordination so operators can mix and match — DRA for device-backed interfaces (GPUs, SmartNICs, SR-IOV via a DRA driver) and Multus + NAD for the established macvlan/ipvlan/bridge attachments. Don’t be surprised if Multus’ role narrows over the next 2–3 years as DRA-based network drivers mature, but the migration is gradual and Multus’ install base in telco is enormous.

See Also

Graph View