vault

systemd Service Units

15 min read

systemd Service Units

A service unit (.service) is the systemd unit type that supervises a process — a daemon, a one-shot setup step, a worker. It is the workhorse of the system: when you run systemctl start nginx, systemd forks and execs the binary in ExecStart=, places it in its own cgroup, tracks its main PID, and — for free, without any code in the daemon — restarts it on failure according to Restart=, captures its stdout/stderr into the journal, and reports its health to systemctl status (systemd.service(5)). The most consequential and most misunderstood setting is Type=, which tells systemd how to decide the service has finished starting up — the readiness contract between the daemon and the manager. Get Type= right and dependent units start at exactly the right moment; get it wrong and you get races, false “active” states, and reload bugs.

This note assumes the unit basics — file format, [Unit]/[Install] sections, the search path, drop-ins, templates — and drills into the .service-specific [Service] section.

Mental Model

A service is a small supervision state machine. systemd forks the process, decides when it counts as started (the Type= question), then watches it: if it exits, the Restart= policy decides whether to relaunch it, after a RestartSec= cool-down. The pivotal subtlety is that “started” is not the same as “the process exists” — for a network daemon you usually want “started” to mean “ready to accept connections,” and the different Type= values are precisely different answers to when systemd is allowed to start the units that depend on this one.

flowchart TB
  START["systemctl start foo.service"] --> PRE["run ExecStartPre=<br/>(serially, must succeed)"]
  PRE --> FORK["fork + exec ExecStart="]
  FORK --> RDY{"Type= readiness rule:<br/>when is it 'started'?"}
  RDY -->|"simple: after fork()"| ACT["active<br/>(dependents may start)"]
  RDY -->|"exec: after execve()"| ACT
  RDY -->|"forking: parent exits"| ACT
  RDY -->|"oneshot: process exits"| ACT
  RDY -->|"notify: READY=1 received"| ACT
  RDY -->|"dbus: bus name acquired"| ACT
  ACT --> POST["run ExecStartPost="]
  POST --> WATCH["watch main process"]
  WATCH -->|"exits"| POLICY{"Restart= policy<br/>matches exit?"}
  POLICY -->|"yes"| WAIT["wait RestartSec=,<br/>then restart"]
  WAIT --> FORK
  POLICY -->|"no"| DEAD["inactive / failed"]

The service lifecycle and the role of Type=. What it shows: ExecStartPre= runs first, then the main process is forked; the Type= rule decides the moment the unit becomes active (and thus the moment its dependents are allowed to start); after it exits, Restart= decides whether to relaunch after RestartSec=. The insight to take: every Type= is a different definition of “ready,” and choosing it correctly is what makes ordering dependencies actually mean “after it can serve,” not merely “after the process was spawned.”

Type= — The Readiness Contract

Type= is the heart of a service unit because it answers: at what instant does systemd consider the unit active, and therefore release the units ordered After= it? (systemd.service(5)).

  • simple (the default when ExecStart= is set without Type= or BusName=) — “the service manager will consider the unit started immediately after the main service process has been forked off (i.e. immediately after fork(), and before … the new process has called execve()).” This is the cheapest and most dangerous default: the unit is “active” before the binary has even begun running, so a daemon that needs a second to open its listening socket is reported ready before it can serve. Dependents ordered after it may race ahead.
  • exec — “the service manager will consider the unit started immediately after the main service binary has been executed.” Unlike simple, it waits for execve() to succeed, so a missing executable or a bad User= makes systemctl start actually report failure. It still does not wait for the daemon to be ready, only for it to start running. Prefer exec over simple when you want start failures surfaced but the daemon has no readiness protocol.
  • forking — for classic Unix daemons that double-fork and exit the parent: “the manager will consider the unit started immediately after the binary that forked off by the manager exits.” You should set PIDFile= so systemd can identify which child is the real main process. The manual is blunt: “The use of this type is discouraged, use notify, notify-reload, or dbus instead.” Forking exists for compatibility with daemons written before systemd; it is racy because systemd has to track the main process through a fork it did not control.
  • oneshot — for a command that runs to completion and exits (a migration, a iptables setup step): “the service manager will consider the unit up after the main process exits. It will then start follow-up units.” oneshot is the only type where ExecStart= may be specified multiple times (or zero times if RemainAfterExit=yes and an ExecStop= is given). It is the default when neither Type= nor ExecStart= is set.
  • dbus — “units of this type must have the BusName= specified and the service manager will consider the unit up when the specified bus name has been acquired.” Readiness is signaled by the daemon registering its well-known D-Bus name — a precise, race-free readiness signal for D-Bus services.
  • notify — “it is expected that the service sends a READY=1 notification message via sd_notify(3) … when it has finished starting up. systemd will proceed with starting follow-up units after this notification message has been sent.” This is the gold standard: the daemon itself tells systemd the exact instant it is ready to serve.
  • notify-reload — like notify, but adds a clean reload protocol: on reload, systemd sends SIGHUP (configurable via ReloadSignal=) and waits for the daemon to send RELOADING=1 then READY=1 again. Lets a config reload be synchronous and ordered.
  • idle — execution “is delayed until all active jobs are dispatched,” with a 5-second cap. Purely a cosmetic aid so a service’s console output does not interleave with boot messages; never use it for ordering correctness.

Readiness Protocols — Why notify Beats forking

The reason Type=notify is preferred over Type=forking is that the daemon, not systemd, knows when it is ready — and notify lets it say so exactly. With forking, systemd infers readiness from “the parent exited,” which is a convention, not a guarantee: a daemon might fork early and still be loading config when the parent dies, so the unit goes “active” before it can serve. With notify, the daemon opens its sockets, loads its config, and then calls sd_notify(0, "READY=1") (sd_notify(3)) — and only at that call does systemd mark the unit active and release dependents. READY=1 “Tells the service manager that service startup is finished … This is only used by systemd if the service definition file has Type=notify or Type=notify-reload set.”

The mechanism is delightfully simple: systemd passes the daemon a $NOTIFY_SOCKET environment variable naming an AF_UNIX datagram socket; sd_notify writes newline-separated key=value strings to it. Beyond READY=1, the protocol carries STATUS= (a human status line shown in systemctl status), RELOADING=1 + MONOTONIC_USEC= (the reload handshake), STOPPING=1, WATCHDOG=1 (the keep-alive ping, below), and MAINPID= (tell systemd a different PID is the main process). If $NOTIFY_SOCKET is unset, sd_notify is a no-op and returns 0 — so a notify-aware binary still runs fine when launched outside systemd. systemd controls who may send notifications via NotifyAccess= (none/main/exec/all); when Type=notify is set but NotifyAccess= is not, “it will be implicitly set to main.”

forking is not wrong — it is the compatibility path for pre-systemd daemons — but it is strictly inferior whenever you control the source. The modern advice (echoed for Type=dbus over forking in Poettering’s admin series) is: if the daemon can register a D-Bus name use dbus; if you can add three lines of sd_notify use notify; only fall back to forking for software you cannot modify.

The Exec* Commands

The [Service] section sequences several command families (systemd.service(5)):

  • ExecStart= — “Commands that are executed when this service is started.” Exactly one command is required, except for Type=oneshot, where multiple ExecStart= lines run serially. “Unless Type=forking is set, the process started via this command line will be considered the main process of the daemon.”
  • ExecStartPre= / ExecStartPost= — commands run before and after ExecStart=. Multiple lines allowed for any type, executed “one after the other, serially.” ExecStart= runs only after all non---prefixed ExecStartPre= commands “exit successfully.” Critically, “ExecStartPre= may not be used to start long-running processes. All processes forked off by processes invoked via ExecStartPre= will be killed before the next service process is run” — ExecStartPre= is for preparation, not for launching helpers.
  • ExecStop= / ExecStopPost= — ExecStop= runs to stop the service and is “only executed when the service started successfully first.” ExecStopPost= runs unconditionally after the service stops, “including cases where … the service exited unexpectedly” and “when a service failed to start up correctly” — the right place for cleanup that must always happen. It receives $SERVICE_RESULT, $EXIT_CODE, $EXIT_STATUS.
  • ExecReload= — the command to reload config in place; $MAINPID is exported so you can kill -HUP $MAINPID.

Each command path supports prefixes: a leading - makes a non-zero exit “considered equivalent to success” (ignore failure of an optional step); @ passes the next token as argv[0]; + runs the command with full privileges bypassing User=/sandboxing; : disables environment-variable substitution.

RemainAfterExit= and PIDFile=

RemainAfterExit= “specifies whether the service shall be considered active even when all its processes exited. Defaults to no.” It is the companion to oneshot: a firewall-setup service runs iptables, exits, and would normally go inactive — but with RemainAfterExit=yes it stays active, so “invoking systemctl start on that unit again will cause no action to be taken” (the rules are already in place) and so the unit can be a meaningful After= ordering target.

PIDFile= is the forking companion: a path (typically under /run/) from which “the service manager will read the PID of the main process … after start-up.” systemd never writes it — the daemon does — though it removes a stale one on shutdown. The manual’s verdict: “PID files should be avoided in modern projects. Use Type=notify, Type=notify-reload or Type=simple where possible.”

Restart= — Supervision For Free

This is the headline feature: systemd is a process supervisor, so a crashed service can be relaunched with no external watchdog, no cron respawn loop, no systemctl cron job. Restart= selects the policy (systemd.service(5)):

  • no (default) — never restart.
  • on-success — restart only on a clean exit: exit code 0, or (for non-oneshot) one of SIGHUP/SIGINT/SIGTERM/SIGPIPE, or a status listed in SuccessExitStatus=.
  • on-failure — restart “when the process exits with a non-zero exit code, is terminated by a signal (including on core dump, but excluding the aforementioned four signals), when an operation … times out, and when the configured watchdog timeout is triggered.” This is the usual choice for a long-running daemon.
  • on-abnormal — restart on signal, timeout, or watchdog, but not on a plain non-zero exit (so a deliberate exit(1) is respected, a crash is not).
  • on-abort — only on an uncaught signal not declared a clean exit.
  • on-watchdog — only when the watchdog timeout expires.
  • always — “restarted regardless of whether it exited cleanly or not.” Note “Type=oneshot services will never be restarted on a clean exit status, i.e. always and on-success are rejected for them.”

Timing is governed by RestartSec= — “the time to sleep before restarting … Defaults to 100ms” — so a crash-looping service does not spin the CPU. For backoff, RestartSteps= and RestartMaxDelaySec= add exponential growth: with RestartSec=10s, RestartSteps=4, RestartMaxDelaySec=160s, successive delays become 10s, 20s, 40s, 80s, 160s, then hold at 160s. Independently, StartLimitIntervalSec=/StartLimitBurst= (in [Unit]) cap how many restarts are allowed in a window before systemd gives up and marks the unit failed — preventing an infinite crash loop from masquerading as a healthy service.

Worked Example — A Notify-Style Service

[Unit]
Description=Inventory API
Documentation=https://example.com/inventory
After=network-online.target postgresql.service
Wants=network-online.target
Requires=postgresql.service
 
[Service]
Type=notify
NotifyAccess=main
ExecStartPre=/usr/bin/inventory-migrate --check
ExecStart=/usr/bin/inventory-api --listen 0.0.0.0:8080
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
RestartSec=2s
RestartSteps=5
RestartMaxDelaySec=60s
WatchdogSec=30s
User=inventory
Group=inventory
 
[Install]
WantedBy=multi-user.target

Reading it: [Unit] orders the service after the network and database and requires Postgres (paired After=+Requires=, the correct idiom per systemd Dependencies and Ordering). Type=notify means systemd waits for the binary to call sd_notify(0, "READY=1") before declaring the unit active — so anything ordered after it truly starts after the API can serve. NotifyAccess=main allows only the main process to send notifications. ExecStartPre= runs a migration check and must succeed (no - prefix) before the API launches; it must be short-lived, not a server. ExecStart= is the long-running main process. ExecReload=/bin/kill -HUP $MAINPID reloads config on systemctl reload. Restart=on-failure relaunches on crash/timeout/watchdog but not on a clean shutdown; the first retry waits RestartSec=2s, then backs off exponentially over RestartSteps=5 toward RestartMaxDelaySec=60s. WatchdogSec=30s arms the watchdog: the daemon must send sd_notify(0, "WATCHDOG=1") at least every 30 seconds, or systemd “is placed in a failed state and it will be terminated with SIGABRT” — combined with Restart=on-failure this means a hung (not crashed) daemon is detected and restarted. User=/Group= drop privileges. [Install]’s WantedBy=multi-user.target makes systemctl enable wire it into the normal boot.

A minimal oneshot contrasts sharply:

[Unit]
Description=Apply static firewall rules
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/sbin/iptables-restore /etc/iptables/rules.v4
ExecStart=/usr/sbin/ip6tables-restore /etc/iptables/rules.v6
[Install]
WantedBy=multi-user.target

Here two ExecStart= lines (legal only for oneshot) run serially; RemainAfterExit=yes keeps the unit active after they exit so the rules count as “applied” and the unit is a valid After= target. No Restart= makes sense — there is no process to supervise.

Failure Modes and Common Misunderstandings

  • Type=simple race. The classic bug: a daemon under Type=simple is “active” the instant it is forked, so a service ordered After= it starts before it can serve. Symptom: intermittent “connection refused” at boot. Fix: Type=notify (best), Type=exec (at least catches exec failures), or a readiness-aware client.
  • forking with no/ wrong PIDFile=. systemd cannot find the main process, so systemctl status shows the wrong PID, MainPID=0, or premature “inactive.” Symptom: systemctl stop fails to actually stop the daemon. Fix: correct PIDFile=, or migrate off forking.
  • ExecStartPre= starts a server. It gets killed before ExecStart= runs (“All processes forked off by … ExecStartPre= will be killed”). Put long-running helpers in their own unit, ordered before this one.
  • Restart=always masks a crash loop. Without StartLimitBurst=/StartLimitIntervalSec=, a service that crashes on startup restarts forever; systemctl status flickers active/failed. Set start limits so systemd gives up and the failure is visible.
  • Reload that isn’t. ExecReload= runs a command but the daemon doesn’t actually re-read config, or under Type=notify-reload the daemon never sends RELOADING=1/READY=1, so systemctl reload hangs until timeout. Verify the daemon honors the reload signal.
  • Watchdog kills a healthy-but-slow service. WatchdogSec= too tight for a service that legitimately blocks (e.g. during GC or a long request) triggers SIGABRT. Tune it to the worst-case ping interval, or omit it for services without a watchdog loop.

Alternatives and When to Choose Them

Before systemd, supervision came from SysV init scripts (no supervision at all — a crashed daemon stayed dead until you noticed), or external supervisors like daemontools, runit, supervisord, or monit. systemd folds supervision, logging, socket activation, and resource control into the init system itself, so for a daemon on a systemd host a .service unit is almost always the right tool. The cases for an external supervisor inside a service are narrow: an application that wants to manage its own worker pool (then run the master under .service and let it fork workers), or a container whose runtime is the supervisor. Within systemd, the choice is between types, not tools: notify if you control the source, dbus for D-Bus daemons, oneshot for setup steps, simple/exec for trivial foreground programs, and forking only as a legacy bridge. See The Init System Comparison for the broader landscape.

Production Notes

In practice the rules of thumb are: make services run in the foreground and use Type=exec or Type=notify — fighting a daemon’s own daemonization (--daemon, double-fork) under Type=forking is the most common source of flaky units. Always pair After= with the matching Requires=/Wants= (systemd Dependencies and Ordering) so ordering implies the dependency is actually present. Use Restart=on-failure plus start limits as a default supervision posture, escalating to WatchdogSec= only for daemons with a real liveness loop. Because every service runs in its own cgroup, systemctl status and systemd-cgls show the entire process tree — including children the daemon forked — which is how systemctl stop reliably kills the whole tree where a PID-file approach would leak children. As of systemd 261 (June 2026), the [Service] directives described here — the eight Type= values, the Restart= policy set, and the sd_notify protocol — are the long-stable core; recent releases add tooling and sandboxing options around this model rather than changing it (systemd 261 context).

Uncertain

Verify: that the Type= value set, the Restart= value set, and the RestartSteps=/RestartMaxDelaySec= exponential-backoff feature are present and unchanged in the exact systemd version on a given target host. Reason: the canonical freedesktop systemd.service(5) page returned HTTP 403 during research, so wording here is from the man7.org mirror tracking upstream “latest,” and RestartSteps=/RestartMaxDelaySec= are relatively recent additions whose availability depends on the installed version. To resolve: run man systemd.service on the target host and check its version with systemctl --version. uncertain

See Also

Graph View

Backlinks