vault

systemctl and Unit Management

13 min read

systemctl and Unit Management

systemctl is the primary command-line interface to systemd — the tool you use to start, stop, inspect, enable, mask, and reconfigure every unit the service manager knows about (systemctl(1)). Its verbs fall into a few families: runtime lifecycle (start, stop, restart, reload, try-restart), persistent enablement (enable/disable, which create or remove [Install]-section symlinks so a unit comes up — or does not — at every boot), masking (mask/unmask, which link a unit to /dev/null to make it un-startable), introspection (status, is-active, is-enabled, list-units, cat, show, list-dependencies), configuration reload (daemon-reload after editing unit files; edit for drop-ins), and whole-system state (isolate, reboot, poweroff, rescue, default). The two most-confused distinctions — enable-vs-start (persistent boot wiring vs runtime activation, which are orthogonal) and mask-vs-disable (un-startable vs merely not-auto-started) — are the conceptual heart of the tool.

Mental Model — Two Independent Axes: Running and Enabled

The single biggest source of confusion with systemctl is conflating running now with configured to run at boot. They are independent. A unit can be:

  • running but not enabled — you systemctl started it this boot; it will not come back after a reboot.
  • enabled but not running — it has a boot-time symlink, but you haven’t started it this session (or it failed).
  • both, or neither.

The man page is unambiguous: “Enabling units should not be confused with starting (activating) units… Enabling and starting units is orthogonal: units may be enabled without being started and started without being enabled” (Arch systemctl(1)). start touches runtime state (systemd’s in-memory job queue); enable touches persistent state (symlinks on disk that systemd reads only at boot or daemon-reload).

flowchart LR
  subgraph RUNTIME["Runtime axis (this boot, in-memory)"]
    direction TB
    STOPPED["inactive"] -->|"start"| ACTIVE["active (running)"]
    ACTIVE -->|"stop"| STOPPED
    ACTIVE -->|"restart"| ACTIVE
  end
  subgraph PERSIST["Persistent axis (across boots, on-disk symlinks)"]
    direction TB
    DIS["disabled<br/>(no .wants/ symlink)"] -->|"enable"| ENA["enabled<br/>(.wants/ symlink exists)"]
    ENA -->|"disable"| DIS
    DIS -->|"mask"| MSK["masked<br/>(symlink → /dev/null)<br/>CANNOT start at all"]
    MSK -->|"unmask"| DIS
  end

The two orthogonal axes systemctl operates on. What it shows: the left axis is runtime activation (start/stop/restart), entirely in-memory and gone at reboot; the right axis is persistent enablement (enable/disable/mask), pure on-disk symlink manipulation that takes effect at the next boot. The insight to take: enable does not start, start does not enable, and mask is a third state on the persistent axis that overrides the runtime axis entirely — a masked unit cannot be started even manually.

Runtime Lifecycle — start, stop, restart, reload

These verbs act on the live system now and persist nothing (systemctl(1)):

  • start — “Start (activate) one or more units.” Enqueues a start job; the unit and its dependencies are brought up. Effect lasts until stop or reboot.
  • stop — “Stop (deactivate) one or more units.”
  • restart — “Stop and then start one or more units… If the units are not running yet, they will be started.” A full teardown and bring-up.
  • reload — “Asks all units listed… to reload their configuration.” This reloads the service’s own config (e.g. nginx re-reading nginx.conf), not the systemd unit file — a critical distinction from daemon-reload. It only works if the service declares an ExecReload=.
  • try-restart — “Stop and then start one or more units… if the units are running. This does nothing if units are not running.” The conditional restart: useful in package post-install scripts so upgrading an inactive service doesn’t spuriously start it.
  • reload-or-restart — reload if the unit supports it, otherwise a full restart; starts the unit if it isn’t running.
  • kill — “Send a UNIX process signal to one or more processes of the unit” (e.g. systemctl kill -s SIGTERM foo); --kill-whom= selects which processes.

Persistent Enablement — enable, disable, and the [Install] Symlinks

enable and disable are pure symlink operations driven by the unit’s [Install] section (systemd.unit(5)). This is the mechanism that decides what comes up at boot.

enable — “Enable one or more units… This will create a set of symlinks, as encoded in the [Install] sections of the indicated unit files.” Concretely, if foo.service declares WantedBy=multi-user.target, then systemctl enable foo.service creates the symlink /etc/systemd/system/multi-user.target.wants/foo.service → the unit file. At boot, when systemd pulls in multi-user.target, it scans multi-user.target.wants/ and adds every symlinked unit as a Wants= dependency — so foo.service gets started. The man page is explicit that this is the only time the [Install] section is consulted: “systemd does not look at the [Install] section at all during normal operation, so any directives in that section only have an effect through the symlinks created during enablement.” Directives:

  • WantedBy= / RequiredBy= — create the symlink in <target>.wants/ or <target>.requires/.
  • Alias= — create an additional symlink under an alternate name.
  • Also= — enable/disable other units alongside this one.
  • DefaultInstance= — the default instance for a template unit.

Critically, enable does not start the unit: “Note that this does not have the effect of also starting any of the units being enabled.” To do both at once, use enable --now (and symmetrically disable --now to disable and stop).

disable — “Disables one or more units. This removes all symlinks to the unit files… and hence undoes any changes made by enable.” It does not stop a running unit (use --now or a separate stop). reenable is disable + enable — it resets a unit’s symlinks back to exactly what its [Install] section specifies, useful after editing WantedBy=.

By default these write to /etc/systemd/system/. With --runtime, the symlinks go under /run/systemd/ instead and “persist only until the next reboot” — a temporary enablement.

Masking — mask and unmask Make a Unit Un-startable

mask is the strongest form of disablement and a different kind of operation from disable. The man page: “Mask one or more units… This will link these unit files to /dev/null, making it impossible to start them. This is a stronger version of disable, since it prohibits all kinds of activation of the unit, even manual activation.” A masked unit’s name resolves to /dev/null in the unit search path, so any attempt to start it — manual systemctl start, a dependency pulling it in, socket activation, a timer — fails. is-enabled reports it as masked (“Completely disabled, so that any start operation on it fails”).

The distinction that trips people up:

  • disable removes the convenience auto-start symlinks. The unit no longer comes up at boot, but you (or a dependency) can still start it manually.
  • mask makes the unit unreferenceable. It cannot be started by anyone, by any mechanism, until you unmask.

mask is the right tool when you must guarantee a unit never runs — e.g. masking systemd-networkd so it cannot be pulled in as a dependency, or masking a shipped service you are replacing. unmask “undoes the effect of mask,” restoring normal resolution.

Introspection — status, is-, list-, cat, show

systemctl is also how you read system state:

  • status — “Show runtime status information about the whole system or about one or more units followed by most recent log data from the journal.” The human-facing dashboard: active state, the main PID, the cgroup tree of the unit’s processes, and the last log lines. The exit code encodes state (0 active, 3 inactive/failed), so it is scriptable too.
  • is-active — exit 0 if at least one named unit is active. The clean predicate for “is it running?”
  • is-enabled — exit 0 if enabled; prints one of a rich set of states (Arch systemctl(1)). The ones worth knowing:
    • enabled — has a .wants//.requires//Alias= symlink.
    • disabled — has an [Install] section but is not currently enabled (so it can be enabled).
    • static — has no [Install] section at all, so it cannot be enabled; it is pulled in only as a dependency of other units. This is the state people most often misread as “broken” — it is normal for helper units like dbus.service or mount helpers.
    • masked — linked to /dev/null, un-startable.
    • linked — the unit file lives outside the search path and was made visible via systemctl link.
    • indirect — not itself enabled, but its Also= lists units that might be, or it has an alias.
    • generated / transient — created by a generator or the runtime API; cannot be enabled (they are pulled in implicitly).
  • is-failed — exit 0 if a unit is in the failed state.
  • list-units — “List units that systemd currently has in memory”; by default only active/pending/failed ones. Add --all for everything loaded, --type=service, --state=failed, etc.
  • list-unit-files — “List unit files installed on the system… with their enablement state.” Unlike list-units this works from disk and includes templates and units that have never been loaded.
  • list-dependencies — “Shows units required and wanted by the specified units,” recursing through Requires=, Requisite=, Wants=, ConsistsOf=, BindsTo=, and Upholds=. The map of what pulls in what.
  • cat — “Show backing files of one or more units. Prints the ‘fragment’ and ‘drop-ins’.” The fast way to see the effective unit definition, base file plus every override.
  • show — “Show properties of one or more units… whenever computer-parsable output is required.” The machine-readable KEY=VALUE dump of every resolved property (use -p to select one, e.g. systemctl show -p MainPID foo).

Reconfiguration — daemon-reload and edit

After you edit a unit file on disk, systemd does not notice automatically — its in-memory model is stale. daemon-reload fixes that: “Reload the systemd manager configuration. This will rerun all generators, reload all unit files, and recreate the entire dependency tree.” Note what this is not: it does not restart any service or reload service-internal config — it only re-reads the unit files and rebuilds systemd’s dependency graph. After daemon-reload you typically still need systemctl restart foo for the new ExecStart= to actually take effect. This is the second arm of the reload confusion: systemctl reload foo reloads foo’s own config; systemctl daemon-reload reloads systemd’s view of the unit files. The heavier relative daemon-reexec re-executes the systemd manager binary itself (serializing and restoring state) — used mainly for systemd upgrades and debugging, rarely needed by hand.

edit is the supported way to modify a unit without touching the distribution’s shipped file: “Edit or replace a drop-in snippet or the main unit file.” Without --full it creates a drop-in — a .conf fragment under /etc/systemd/system/foo.service.d/ (default override.conf) that merges over the base unit, leaving the vendor file intact and upgrade-safe. With --full it makes a full editable copy of the main unit file. Either way, “configuration reloaded after successful edit,” so an explicit daemon-reload is unnecessary. Drop-ins are the correct mechanism for local customization: a Service=-overriding fragment survives a package upgrade that rewrites /usr/lib/systemd/system/foo.service, because /etc/ drop-ins take precedence over /usr/lib/ base files.

Whole-System State — isolate, reboot, poweroff, rescue, default

The last family changes the system’s state rather than an individual unit:

  • isolate — “Start the unit specified… and its dependencies and stop all others, unless they have IgnoreOnIsolate=yes.” This is how you switch the active target: systemctl isolate multi-user.target brings the system to a no-GUI multi-user state and stops everything not wanted there.
  • rescue — “equivalent to systemctl isolate rescue.target”: single-user-like maintenance mode with a root shell and minimal services. emergency is even more minimal (emergency.target: just the root filesystem and a shell).
  • default — “equivalent to systemctl isolate default.target”: return to the system’s normal target.
  • reboot / poweroff / halt — these enqueue the corresponding .target “with --job-mode=replace-irreversibly --no-block”; reboot is “mostly equivalent to systemctl start reboot.target.” suspend, hibernate, and hybrid-sleep trigger the matching sleep targets.

—user — the Per-User Service Manager

Everything above defaults to the system manager (--system, PID 1), operating on units in /etc/systemd/system/ and /usr/lib/systemd/system/. With --user, the same verbs drive a per-user systemd instance that supervises the calling user’s own units from ~/.config/systemd/user/ and /usr/lib/systemd/user/. This is how desktop sessions, user-level timers, and tools like podman’s rootless containers manage themselves: systemctl --user enable mybackup.timer enables a timer that runs under your UID, with no root involved. The user manager is itself started by systemd --user, launched per login session (or lingering, via loginctl enable-linger).

Failure Modes and Common Misunderstandings

The dominant confusion is “I started it but it’s gone after reboot” — start without enable. The fix is enable (or enable --now). The mirror image is “I enabled it but it’s not running” — enable doesn’t start; use --now or a separate start. A third is editing a unit file and seeing no change — you forgot daemon-reload, so systemd is still running the old definition; systemctl cat foo showing your edit but the behavior unchanged is the tell. A fourth is reload vs daemon-reload: running systemctl reload foo after editing foo.service does nothing useful, because reload asks foo to reload its config, not systemd to reload the unit file. And a subtle one is misreading static as an error: a static unit has no [Install] section by design and is meant to be pulled in by dependency, not enabled — systemctl enable on it is a no-op that prints a warning, which is correct, not a failure. Finally, disable does not stop, and enable does not start — both are pure symlink operations; pair them with --now or explicit start/stop when you mean to change the running state too.

Alternatives and When to Choose Them

systemctl is the interface for systemd; the alternatives are either lower-level or for other init systems. Underneath, systemctl talks to PID 1 over D-Bus, and you can drive the same operations programmatically via that API (the org.freedesktop.systemd1 interface) — appropriate for software that manages units, not for interactive use. For per-unit logs you reach for journalctl rather than systemctl status once you need more than the last few lines (see The systemd Journal). For sysvinit systems the historical equivalents are service foo start and chkconfig foo on / update-rc.d — systemctl deliberately subsumes both the runtime (service) and the persistent (chkconfig) roles into one tool, which is exactly why the enable-vs-start split exists as two explicit verbs. On systemd systems the legacy service/chkconfig commands are usually thin shims that redirect to systemctl.

Production Notes

In practice the verbs you reach for daily are status (triage), restart (deploy a new build), enable --now (turn a service on permanently in one step), and daemon-reload (after any unit-file edit). The --now suffix on enable/disable is the most under-used convenience — it collapses the orthogonal axes back into the single intent “make this service on/off both now and forever,” which is what an operator usually means. For debugging boot, systemctl list-units --failed and systemctl status <unit> are the entry points, with systemctl list-dependencies <target> to understand why something was pulled in. The drop-in mechanism via systemctl edit is the correct, upgrade-safe way to customize vendor units — editing files under /usr/lib/systemd/system/ directly is the anti-pattern, because the next package update overwrites them, whereas an /etc/.../foo.service.d/override.conf survives. And mask is the operator’s emergency brake: when a unit must not run regardless of what tries to pull it in, masking it to /dev/null is the only thing disable cannot guarantee.

See Also

Graph View