vault

systemd and cgroup Integration

13 min read

systemd and cgroup Integration

Every process systemd supervises lives in a cgroup-v2 control group, and systemd organizes the entire cgroup tree into a disciplined shape: a tree of slices (inner nodes that carve up resources but hold no processes) whose leaves are services (process groups systemd forks itself) and scopes (process groups some other manager forked, which systemd merely adopts). systemd does this as the single privileged writer of the hierarchy — the rule that “each cgroup only has a single writer, i.e. a single process managing it” (systemd.io CGROUP_DELEGATION). Where another manager (a container runtime, a user-session manager) needs to build its own sub-hierarchy, systemd hands it a delegated subtree with Delegate=yes and then keeps its hands off. This note is about that organization — the slice/scope/service tree, the single-writer and delegation contract, and the resource-control knobs (MemoryMax=, CPUWeight=, IOWeight=, TasksMax=) that translate unit settings into the kernel’s cgroup files. The mechanics of how those knobs are enforced (how cpu.weight actually schedules CPU time, how memory.max triggers reclaim and the OOM killer) belong to The cgroup v2 CPU Controller and the kernel notes — cross-linked, not re-derived here.

The cgroup-v2-unified default — dated

As of 2026, systemd uses the cgroup-v2 unified hierarchy exclusively, and this is now a hard requirement rather than a default. The build-time default flipped to unified (-Ddefault-hierarchy=unified) in systemd v243 (September 2019, v243 release notes); previously the default was the hybrid mode. Distributions migrated over the following years (Fedora led, Ubuntu switched its default in the 21.10 era). The legacy options were “fully deprecated” in v252, and finally the legacy and hybrid modes were removed entirely in systemd v258 (September 2025): per the v258 NEWS, support for cgroup v1 was removed and “cgroup v2 (‘unified’ hierarchy) will always be mounted during system bootup” (systemd 258 release announcement, v258 NEWS). v258 also raised the minimum kernel baseline to 5.4. So on any 6.12/6.18-LTS-era machine running current systemd, there is exactly one mounted cgroup filesystem, /sys/fs/cgroup/, and it is cgroup v2.

Uncertain

Verify: that the specific distro a reader runs ships systemd ≥ v258 (and thus has no legacy/hybrid fallback at all). Reason: the unified-only removal landed in v258 (Sept 2025); long-lived LTS distros may still ship an older systemd where systemd.unified_cgroup_hierarchy=0 could force legacy mode. To resolve: check systemctl --version and stat -fc %T /sys/fs/cgroup/ (a value of cgroup2fs confirms unified). uncertain

Mental Model — Slices Branch, Services and Scopes Are Leaves

The cgroup tree systemd builds maps one-to-one onto three unit types, and the mapping is the key to reading any systemd-cgls output at a glance. Slices are the branches: “[slices] do not directly contain any processes … [they] are the inner nodes of part of the cgroup tree” (CGROUP_DELEGATION). Services and scopes are the leaves that actually hold processes — the difference between them is who created the processes. A service holds processes systemd itself forked (ExecStart=). A scope holds processes “forked off by some unrelated manager process” that systemd then adopts — a user login session (created by systemd-logind), a container or VM (registered by systemd-machined), or anything launched with systemd-run --scope.

This shape is forced by the kernel’s no-internal-process rule: in cgroup v2, once a cgroup distributes resources to children, it may not also hold processes — “processes may reside only in leaf nodes” (cgroups(7), and see cgroup Delegation and the No Internal Process Rule for the full mechanism). Slices distribute; therefore slices hold no processes; therefore processes live only in the service/scope leaves. systemd’s tree is not an arbitrary design choice — it is the natural shape any cgroup-v2 manager must adopt.

flowchart TB
  ROOT["-.slice  (root, /sys/fs/cgroup)"]
  ROOT --> SYS["system.slice  (system services)"]
  ROOT --> USR["user.slice  (login sessions)"]
  ROOT --> MACH["machine.slice  (VMs / containers)"]
  SYS --> SSHD["sshd.service  (LEAF: systemd-forked procs)"]
  SYS --> NGINX["nginx.service  (LEAF)"]
  USR --> U1000["user-1000.slice"]
  U1000 --> SESS["session-3.scope  (LEAF: logind-adopted procs)"]
  MACH --> CT["systemd-nspawn container.scope<br/>Delegate=yes to DELEGATED subtree"]
  CT --> CINNER["...container's OWN slices/services<br/>(managed by the container's systemd)"]

systemd’s cgroup-v2 tree. What it shows: the root -.slice branches into the three standard slices; system.slice/user.slice/machine.slice are inner nodes (no processes), and the actual processes sit in .service leaves (systemd-forked) and .scope leaves (externally-forked, adopted). The machine.slice branch shows a delegated container scope whose subtree is managed by the container’s own manager, not host systemd. The insight to take: read any systemd-cgls tree by asking of each node “branch or leaf?” — branches are slices, leaves are services or scopes, and the leaf type tells you whether systemd or some other manager forked the processes inside.

Mechanical Walk-through

The three standard slices and the slice naming scheme

systemd “sets up four slice units” at boot (CGROUP_DELEGATION): the root -.slice (mapping to /sys/fs/cgroup/ itself) and three children:

  • system.slice — “the default location for service and scope units.” Almost every system daemon you start ends up here.
  • user.slice — home of per-user session trees managed by systemd-logind. Each logged-in user gets a nested user-<UID>.slice (e.g. user-1000.slice), and within it a user@1000.service (the user’s own systemd instance) and one session-N.scope per login.
  • machine.slice — “contains virtual machines and containers registered with systemd-machined.”

The slice name encodes its position in the tree using dashes as path separators: per systemd.slice(5), “foo-bar.slice resides within foo.slice, which itself is contained in the root slice -.slice.” So user-1000.slice is literally the path -.slice / user.slice / user-1000.slice. (This dash-as-separator convention is also the source of a subtle escaping gotcha — see Slice Aliasing and append Surprises.) A unit chooses its slice with the Slice= directive; the default for services is system.slice. Slice units gain automatic After= and Requires= on their parent slice, so the tree is also a dependency chain.

The single-writer rule and why systemd owns the root

The discipline that keeps the whole thing coherent is the single-writer rule: “each cgroup only has a single writer, i.e. a single process managing it … only a single process should own a specific cgroup, and when it does that ownership is exclusive, and nothing else should manipulate it at the same time” (CGROUP_DELEGATION). On a systemd machine, PID 1 (systemd) is the writer for the root and for every slice/service/scope it creates. The corollary the doc states bluntly: “the root cgroup is managed by systemd and hence off limits to everybody else.” A container runtime or a homegrown resource manager that pokes cgroup files under systemd’s nodes directly is violating the rule and will fight systemd (systemd reconciles the tree to match its unit state and will undo your changes). The sanctioned way to get write access to part of the tree is delegation.

Delegation — handing a subtree to another manager

Delegation is “some parts of the cgroup tree may be managed by different managers than others” — and crucially “only sub-trees can be delegated” (CGROUP_DELEGATION). You request it by setting Delegate=yes (or Delegate=cpu memory io to delegate only specific controllers) on a service or scope unit. Once set, that unit’s cgroup becomes “the cut-off point for systemd’s cgroup management … systemd won’t fiddle with your sub-tree below it anymore.” The owner can then create child cgroups, move its processes among them, and enable controllers in the delegated subtree — building, for example, a container’s own slice/service hierarchy with a nested systemd inside it.

There is a sharp restriction: delegation is allowed on services and scopes only, never on slices. The doc is explicit: “Delegation is available on scope and service units only. It’s expressly not available on slice units … if we’d allow delegation on slice units … both systemd and your own manager would create/delete cgroups below the slice unit” — i.e. two writers, breaking the single-writer rule. This is why a delegated container appears in the tree as a .scope (with Delegate=yes) under machine.slice, not as a delegated slice. A practical wrinkle from the no-internal-process rule: because a delegated unit will itself spawn child cgroups (becoming an inner node), systemd offers DelegateSubgroup= to automatically place the unit’s own main/supervising process into a subgroup, since “no processes should live in inner nodes of the control group tree” (systemd.resource-control(5)).

The flagship users of delegation are systemd-nspawn (each container is a delegated scope running its own systemd as PID 1) and the per-user user@.service (the user’s session manager gets a delegated subtree so an unprivileged user can run their own user-level services without root). Container runtimes integrating with systemd (the cgroup driver in runc/crun, used by Docker and Kubernetes) ask systemd — over D-Bus — to create a delegated scope or service rather than writing cgroup files directly; see cgroups Integration and Namespaces and cgroups as Container Building Blocks.

Resource-control knobs map unit settings to controller files

The point of putting every unit in a cgroup is that systemd can then apply resource limits declaratively in the unit file, and it translates each directive into a write to the corresponding cgroup-v2 controller file (systemd.resource-control(5)). The mapping:

Unit directivecgroup-v2 fileControllerMeaning (mechanics live elsewhere)
CPUWeight=cpu.weightcpuproportional share among siblings (1-10000, default 100)
CPUQuota=cpu.maxcpuabsolute ceiling, e.g. CPUQuota=20%
MemoryMax=memory.maxmemoryhard cap; exceed → OOM kill within the cgroup
MemoryHigh=memory.highmemorysoft throttle; reclaim pressure, no OOM
MemoryMin= / MemoryLow=memory.min / memory.lowmemoryhard/soft reclaim protection
IOWeight=io.weightioproportional I/O share among siblings
TasksMax=pids.maxpidscap on number of tasks (fork/clone limit)
AllowedCPUs=cpuset.cpuscpusetpin to a CPU set
AllowedMemoryNodes=cpuset.memscpusetpin to NUMA memory nodes

The essential point for this note is the organization, not the enforcement: because limits are hierarchical, a limit set on system.slice caps the sum of every service beneath it, and “regardless of what happens in the delegated sub-hierarchy, nothing can escape the resource restrictions imposed by the parent” (cgroup-v2 docs). So you can throttle all user sessions collectively by limiting user.slice, independent of any per-user setting. The actual scheduling math — how cpu.weight becomes CPU time, how memory.max drives reclaim and the cgroup OOM killer — is The cgroup v2 CPU Controller / The Memory Cgroup memcg / The cgroup io Controller / The cgroup pids Controller, and is deliberately not re-derived here.

Inspecting the tree: systemd-cgls and systemd-cgtop

systemd ships two purpose-built viewers. systemd-cgls “recursively shows the contents of the selected Linux control group hierarchy in a tree” (systemd-cgls(1)) — it renders exactly the slice→service/scope structure above, annotating each leaf with its PIDs and command lines, which makes “which slice is this process in?” a one-glance question. systemd-cgtop is the top(1) analogue: it shows live per-cgroup resource usage (CPU, memory, I/O, task count) sorted like top, so you can find which slice or service is consuming resources in real time. Together they are the standard pair for understanding the live tree; for the static unit-defined limits you’d use systemctl show -p MemoryMax,CPUWeight foo.service.

Configuration / Worked Examples

A capped service

# /etc/systemd/system/batch-job.service
[Unit]
Description=Resource-limited batch job
 
[Service]
ExecStart=/usr/local/bin/batch
Slice=batch.slice        # place this service under a custom slice
MemoryMax=2G             # → memory.max: hard cap, OOM-kill if exceeded
MemoryHigh=1500M         # → memory.high: start throttling under 1.5G pressure
CPUWeight=50             # → cpu.weight: half the default share vs siblings
CPUQuota=40%             # → cpu.max: never exceed 40% of one CPU
IOWeight=30              # → io.weight: low I/O priority among siblings
TasksMax=64              # → pids.max: cannot fork more than 64 tasks

Each line above is a declarative statement of a limit; systemd creates /sys/fs/cgroup/batch.slice/batch-job.service/ and writes 2147483648 to its memory.max, 50 to cpu.weight, and so on. Putting it under batch.slice means a single limit on batch.slice would cap this and any sibling batch jobs collectively.

A transient scope from the command line

# Launch a shell in a brand-new scope under system.slice, with limits:
systemd-run --scope -p MemoryMax=512M -p CPUQuota=25% --slice=adhoc.slice bash

This creates an adhoc.slice (if absent) and a transient run-<rand>.scope leaf inside it holding the new bash — a scope, not a service, because the process is yours and systemd merely adopts and accounts it. Inspect with systemd-cgls adhoc.slice and watch it with systemd-cgtop.

Failure Modes / Common Misunderstandings

  • Writing cgroup files directly under systemd’s nodes. Symptom: limits mysteriously revert, or processes get migrated out from under you. Cause: violating the single-writer rule — systemd reconciles the tree to its unit state and overwrites manual edits. Fix: set the limit via a drop-in (systemctl set-property) or get a delegated subtree.
  • Expecting Delegate=yes on a slice. Symptom: delegation seems ignored. Cause: “Delegation is available on scope and service units only … expressly not available on slice units.” Fix: delegate a service or scope, not a slice.
  • Putting processes in an inner node. Symptom: enabling a controller in cgroup.subtree_control fails with EBUSY. Cause: the cgroup still holds processes while you try to make it a distributing inner node — the no-internal-process rule. Fix: move processes to a leaf child (this is what DelegateSubgroup= automates). Full mechanism: cgroup Delegation and the No Internal Process Rule.
  • Assuming a per-service limit bounds total usage. A MemoryMax= on each service does not bound their sum; only a limit on the enclosing slice does, because limits are hierarchical. To cap a whole class of work, limit its slice.
  • Confusing scopes and services in cgls output. A .scope does not mean systemd started the processes — it adopted them. Misreading this leads to “why is this container a scope, not a service?” confusion; the answer is delegation.

Alternatives and When to Choose Them

Before systemd’s cgroup integration, resource control meant hand-mounting cgroup-v1 controller hierarchies and writing tasks into them with ad-hoc scripts or libcgroup/cgconfig — fragile, multi-writer, and exactly the chaos the single-writer rule exists to prevent. On a modern unified-only system, letting systemd own the tree and asking it (via unit files or D-Bus) for what you need is the only sanctioned path; the only question is which interface: static unit-file directives for persistent services, systemctl set-property --runtime for live tweaks, systemd-run for transient scopes, or Delegate=yes + your own manager when you genuinely need to build a sub-hierarchy systemd shouldn’t touch (containers, nested systemd). Direct cgroupfs manipulation remains technically possible only inside a delegated subtree — i.e. only where systemd has explicitly stepped back.

Production Notes

The systemd cgroup model is the substrate the entire container ecosystem stands on. Kubernetes’ recommended cgroup driver is systemd precisely so the kubelet and the container runtime do not become a second writer fighting PID 1 — running the kubelet with the cgroupfs driver on a systemd host is a classic source of subtle resource-accounting drift and is explicitly discouraged. systemd-nspawn and machinectl-managed containers rely wholly on delegated scopes under machine.slice. The transition pain was real: because v243 flipped the default but distros and container runtimes migrated on their own schedules, there was a multi-year window where “is this host unified, hybrid, or legacy?” was a live operational question (stat -fc %T /sys/fs/cgroup/); the v258 removal of legacy/hybrid finally closes that window for anyone on current systemd. The day-to-day operator tools remain systemd-cgls (structure) and systemd-cgtop (live usage) — learning to read their output as the slice/scope/service tree described here is most of cgroup-on-systemd literacy.

See Also

Graph View