vault

CronJob

16 min read

CronJob

A CronJob is the Kubernetes API object for time-based scheduled Job creation. It is the canonical wrapper around Job: the CronJob’s spec.jobTemplate describes the Job to create, spec.schedule is a standard cron expression telling the controller when to create it, and the CronJob controller (inside kube-controller-manager) wakes up periodically and emits a new Job whenever a scheduled time has arrived. CronJobs reached stable in K8s 1.21 (KEP-19); the long-requested spec.timeZone field reached stable in K8s 1.27 (KEP-3140), finally letting users specify schedules in non-UTC time zones without timezone-shifting hacks. The cron expression itself is parsed by github.com/robfig/cron/v3 (Kubernetes switched to v3 in June 2021), which is not identical to the POSIX crontab parser — subtle differences (notably around the optional-seconds-field that v1 supported but v3 dropped, and the strict 5-field interpretation) cause the most common “my CronJob doesn’t fire when I expect” surprises. CronJobs are concurrency-aware via the Allow/Forbid/Replace policies, retention-aware via successfulJobsHistoryLimit/failedJobsHistoryLimit, and rate-limited against control-plane outages via startingDeadlineSeconds and the missed-100-schedules safety cap. They are the right tool for most periodic in-cluster work — but they are not a workflow engine, and the failure modes around missed schedules and concurrency edge cases catch every operator at least once.

Mental Model

gantt
    title CronJob schedule "0 * * * *" (hourly, Allow concurrency)
    dateFormat HH:mm
    axisFormat %H:%M
    section Tick (cron eval)
    eval                       :milestone, 09:00, 0
    eval                       :milestone, 10:00, 0
    eval                       :milestone, 11:00, 0
    eval                       :milestone, 12:00, 0
    section Job instances
    Job hello-2807060800       :a, 09:00, 65m
    Job hello-2807060900       :b, 10:00, 35m
    Job hello-2807061000       :c, 11:00, 20m
    Job hello-2807061100       :d, 12:00, 10m
    section "Forbid policy (alternative)"
    Job (09:00 run, 65min)     :crit, e, 09:00, 65m
    "10:00 SKIPPED (still running)" :crit, milestone, 10:00, 0
    Job (11:00 run)            :crit, f, 11:00, 20m
    section "Replace policy (alternative)"
    Job (09:00 run started)    :crit, g, 09:00, 60m
    "10:00 killed + restarted"  :crit, milestone, 10:00, 0
    Job (10:00 new)            :crit, h, 10:00, 35m

What this diagram shows. A CronJob with schedule: "0 * * * *" (top of every hour). The CronJob controller wakes up and evaluates the schedule on each tick (default poll loop ~10s; misses bounded to 100 missed schedules). At each scheduled time, it constructs a fresh Job named <cronjob-name>-<unix-timestamp> from spec.jobTemplate. The three concurrency policies diverge in how they handle overlap: Allow (default) creates new Jobs at each tick regardless of whether prior Jobs are still running — multiple Jobs can run concurrently; Forbid skips the tick if a previous Job is still running; Replace kills the running Job and creates a fresh one. Completed Jobs are retained per successfulJobsHistoryLimit (default 3) and failedJobsHistoryLimit (default 1); older Jobs are garbage-collected by the CronJob controller. The insight to extract is that CronJob is a creator of Jobs, not an executor itself — every operational property of the actual work (retries, timeouts, parallelism, indexed mode, podFailurePolicy) is inherited from the underlying Job, while CronJob layers schedule, concurrency, and history on top.

Mechanical Walk-through

The controller’s reconcile loop

The CronJob controller (kube-controller-manager’s cronjob-controller) runs the canonical reconcile pattern (see Kubernetes Control Loop Pattern):

  1. Watches CronJobs and the Jobs they own.
  2. On each CronJob reconcile, computes:
    • The current time (in the CronJob’s spec.timeZone, or the controller’s local time if unset).
    • The next scheduled time, using robfig/cron/v3.
    • The set of previously scheduled times since the last successful Job creation that have not yet been honored — the missed schedules.
  3. If at least one missed schedule falls within the startingDeadlineSeconds window from now, the controller honors the most recent one by creating a Job. Older missed schedules are dropped.
  4. Applies the concurrency policy by inspecting the CronJob’s owned Jobs: are any still active? If Forbid, skip; if Replace, delete the active Job (and its Pods); if Allow (default), proceed regardless.
  5. Creates the new Job by templating spec.jobTemplate and patching in an owner reference back to the CronJob.
  6. Garbage-collects owned Jobs beyond successfulJobsHistoryLimit / failedJobsHistoryLimit.

This loop runs much more frequently than the schedule (every ~10 seconds in modern K8s), so a schedule of * * * * * (every minute) reliably fires within seconds of the start of each minute.

The cron syntax and the robfig/cron/v3 parser

The CronJob uses 5-field cron expressions with the standard semantics (kubernetes.io):

# ┌───────────── minute      (0 - 59)
# │ ┌───────────── hour      (0 - 23)
# │ │ ┌───────────── day-of-month (1 - 31)
# │ │ │ ┌───────────── month (1 - 12)
# │ │ │ │ ┌───────────── day-of-week (0 - 6) (Sunday=0)
# │ │ │ │ │
# * * * * *

Supported syntax:

  • Wildcards: * (any value), ? (alias for *).
  • Lists: 1,15,30.
  • Ranges: 1-5.
  • Step values: */15 (every 15), 0-23/2 (every even hour).
  • Predefined macros: @yearly / @annually / @monthly / @weekly / @daily / @midnight / @hourly (@reboot is not supported — it has no meaning in a cluster context).
  • @every <duration> (a robfig extension): @every 1h30m — useful, but not portable if you ever migrate to a different cron implementation.

Kubernetes switched the underlying parser from robfig/cron v1 to v3 in Kubernetes commit 4b36a5cb (June 2021), released initially in K8s 1.22. The behavioral differences that bite users:

  • v1 accepted an optional 6th field for seconds (30 0 * * * * meaning “30 seconds past minute 0”); v3 does not — five fields, the first is minutes. Pre-1.22 CronJobs using the 6-field form silently shifted their interpretation upon upgrade. The @every job interval bug is a famous example of v1→v3 regression.
  • v3 rejects timezone-prefixed schedules like TZ=Asia/Tokyo 30 04 * * * and CRON_TZ=Asia/Tokyo 30 04 * * * — K8s explicitly validates against these and rejects them, recommending the spec.timeZone field instead.

The full parser code path lives in pkg/controller/cronjob/utils.go in the Kubernetes repository and imports github.com/robfig/cron/v3 directly.

Time zones (spec.timeZone, GA in 1.27)

Before K8s 1.27, CronJobs were interpreted in the kube-controller-manager’s local time zone — which on managed K8s is always UTC, surprising every operator outside UTC who expected the schedule to run in their local time. The workaround was to convert times to UTC manually, which broke twice a year for users in DST-observing zones.

spec.timeZone (alpha in 1.24, beta in 1.25, stable in 1.27, per KEP-3140) lets the user declare the schedule’s time zone:

spec:
  timeZone: "America/New_York"
  schedule: "0 9 * * 1-5"   # 9am NYC, weekdays

Validation:

  • The value must be a valid IANA time zone name (e.g., America/New_York, Etc/UTC, Asia/Tokyo). Validated against the Go standard library’s embedded time/tzdata (since Go 1.15, the database is embedded into the binary so the host’s /usr/share/zoneinfo is not required).
  • Invalid values produce an UnknownTimeZone event on the CronJob.
  • DST transitions are handled correctly: a 0 2 * * * schedule in America/New_York may have a missing run on spring-forward and a doubled run on fall-back — known cron-everywhere quirks, not K8s-specific.

If spec.timeZone is unset or empty, the controller falls back to the kube-controller-manager’s local time zone (typically UTC). The recommendation: always set it explicitly, even to "Etc/UTC", to make schedules unambiguous and portable.

concurrencyPolicy

Three values (kubernetes.io — concurrency-policy):

  • Allow (default). Multiple Job instances can run concurrently. Best when each scheduled run is independent and the workload can handle overlap.
  • Forbid. If a prior Job is still running at the scheduled time, the new run is skipped entirely (it does not queue). The skip counts as a missed schedule for startingDeadlineSeconds accounting. Best when overlap would cause data corruption (e.g., a database vacuum that must run alone).
  • Replace. If a prior Job is still running, it is deleted (which cascades to its Pods) and replaced with a new Job. Best when the work is idempotent and freshness matters more than completion (e.g., a metric-aggregation job where late results are useless).

The choice has subtle interaction with startingDeadlineSeconds. With Forbid + a long-running Job, a backlog of missed schedules accumulates; when the running Job finally finishes, only one new Job is created (the most recent missed schedule), not a flood. With Replace, each tick may interrupt the previous one, so very long-running work paired with a tight schedule never completes.

startingDeadlineSeconds and missed schedules

spec.startingDeadlineSeconds (no default; unset means “no deadline”) bounds how stale a missed schedule can be before it is dropped. If the controller is reconciling at 10:30 and the most recent missed schedule was at 10:00 (30 minutes ago), and startingDeadlineSeconds: 600 (10 minutes), the missed schedule is dropped — too stale.

The 100-missed-schedules safety cap. Independent of startingDeadlineSeconds, the CronJob controller refuses to count more than 100 missed schedules when evaluating a CronJob. If the controller has been down for so long that more than 100 schedules were missed (a * * * * * schedule with a >100-minute control-plane outage), the CronJob is marked as failed to start with an error and no Jobs are created until the user manually intervenes. This safety cap prevents a control-plane recovery from triggering a runaway Job creation storm. Diagnose with kubectl get events showing FailedNeedsStart. Fix: either accept the gap and update the CronJob’s lastScheduleTime annotation, or delete and recreate the CronJob.

Without startingDeadlineSeconds set, missed schedules can accumulate up to the 100-cap; the controller will then create a single Job for the most recent missed schedule, not 100 Jobs. With startingDeadlineSeconds: 60, a missed schedule older than 60 seconds is dropped — typical for time-sensitive work where stale runs are useless (and harmful if Allow would otherwise have made them concurrent with the next on-time run).

Job history retention

  • spec.successfulJobsHistoryLimit (default 3). Number of successfully-completed Jobs retained. The controller garbage-collects older successful Jobs and their Pods.
  • spec.failedJobsHistoryLimit (default 1). Number of failed Jobs retained — typically you want at least 1 for post-mortem, more if failures are common.
  • Setting either to 0 disables retention entirely (the Job is deleted immediately on completion).

This is separate from spec.jobTemplate.spec.ttlSecondsAfterFinished — TTL at the Job level deletes the contents of a Job after a delay; history limits at the CronJob level cap the count. They compose: either can trigger deletion first.

spec.suspend

Setting spec.suspend: true prevents the controller from creating new Jobs (the CronJob is paused). Existing Jobs continue to run. Setting back to false:

  • If startingDeadlineSeconds is unset, all missed schedules during the suspension are honored at unsuspend time — up to the 100-cap — creating one Job (the most recent).
  • If startingDeadlineSeconds is set, only schedules within the deadline window from “now” are honored.

This is the right way to “temporarily disable a CronJob” for maintenance.

Configuration / API Surface

A canonical scheduled-backup CronJob, with line-by-line commentary:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: nightly-backup
  namespace: db
spec:
  schedule: "0 2 * * *"                      # 2:00 AM every day
  timeZone: "America/New_York"               # GA in K8s 1.27; explicit > implicit
  concurrencyPolicy: Forbid                  # backups must not overlap
  startingDeadlineSeconds: 1800              # 30 min grace window for missed runs
  suspend: false                             # default; set true to disable temporarily
  successfulJobsHistoryLimit: 7              # keep last 7 successful backups for inspection
  failedJobsHistoryLimit: 3                  # keep last 3 failures for post-mortem
  jobTemplate:
    spec:
      backoffLimit: 2                        # tolerate 2 Pod failures before failing the Job
      activeDeadlineSeconds: 3600            # 1 hour budget for a single backup
      ttlSecondsAfterFinished: 604800        # delete the Job (and its Pods) 7 days post-completion
      template:
        spec:
          restartPolicy: Never
          serviceAccountName: backup-runner
          containers:
            - name: backup
              image: registry.example.com/db-backup:v3
              command: ["/usr/local/bin/run-backup.sh"]
              env:
                - name: BACKUP_BUCKET
                  value: s3://prod-db-backups
              resources:
                requests: { cpu: "200m", memory: "256Mi" }
                limits:   { cpu: "1",    memory: "1Gi" }

Notable surface points:

  • spec.schedule is validated at admission. Invalid expressions (e.g., 0 25 * * * — hour 25 doesn’t exist) are rejected. Note: 0 5 * * 7 (day-of-week 7) was rejected by v1’s parser but is accepted by v3 (which treats 7 as Sunday, an alias for 0).
  • spec.timeZone is independent of the Pod’s runtime time zone. The Pod’s process still sees UTC inside the container unless you mount /etc/localtime from the host or set TZ env var explicitly.
  • spec.jobTemplate is itself a Job spec. Anything a Job can do (Indexed mode, podFailurePolicy, parallelism, etc.) is available via jobTemplate. The shape of metadata inside jobTemplate is interpreted as the Job’s metadata, not the Pod’s.
  • The created Job’s name pattern is <cronjob-name>-<unix-timestamp-of-schedule> — nightly-backup-2807058000. This is stable: the same schedule reliably produces the same Job name, which makes kubectl get jobs filtering by date trivial.

Failure Modes

  1. Schedule fires “an hour late” twice a year. A 0 2 * * * schedule with no spec.timeZone set, on a kube-controller-manager configured for America/New_York’s local time (rare in managed K8s but possible self-hosted), behaves correctly most of the time but on DST transitions, either skips or doubles the 2 AM run. The defensive fix is to set timeZone: "Etc/UTC" explicitly and shift the schedule to UTC.

  2. Control-plane outage > 100 missed schedules. A * * * * * schedule and a 2-hour kube-controller-manager outage means 120 missed schedules; the CronJob hits the 100-cap, marks itself FailedNeedsStart, and creates no Jobs after recovery. Diagnose: kubectl describe cronjob shows FailedNeedsStart event. Fix: typically delete and recreate the CronJob.

  3. Forbid policy + Job that hangs forever. Setting Forbid and forgetting activeDeadlineSeconds on the Job means: one Job hangs → every subsequent scheduled run is skipped → no scheduled work happens for days. Diagnose: kubectl get jobs -l cronjob-name=... shows a single old Job in Running state. Fix: always set activeDeadlineSeconds in jobTemplate for Forbid-policy CronJobs.

  4. Allow concurrency causing database hammering. A 5-minute Allow CronJob whose work occasionally takes 7 minutes runs 2 instances concurrently every other tick, which doubles the DB connection pressure. The pool exhausts, the work starts failing, the failures retry per backoffLimit, the failures pile up. Diagnose: a sudden DB outage correlated with CronJob ticks. Fix: switch to Forbid, or set activeDeadlineSeconds tight.

  5. Time zone validation silently passes for “Etc/UTC” but fails for “UTC”. Both are valid IANA names but "UTC" is an alias for "Etc/UTC"; in some Go versions and tzdb versions, the validation accepts one but not the other. Use the canonical form "Etc/UTC" to avoid surprise.

  6. The TZ=... schedule prefix from pre-1.22 days. Pre-1.22 CronJobs sometimes used TZ=Asia/Tokyo 0 9 * * * or CRON_TZ=Asia/Tokyo 0 9 * * * (parsed by robfig/cron v1). K8s 1.22+ explicitly rejects this at admission. Migrating these CronJobs requires removing the prefix and setting spec.timeZone.

  7. 6-field schedules from old robfig/cron v1. A YAML manifest with schedule: "30 0 0 * * *" (six fields, seconds=30) was valid in K8s pre-1.22 — and even appeared to work, just with shifted semantics. After upgrade to v3, the same expression is rejected as invalid (5-field parser sees the leading 30 0 0 as min/hour/day-of-month, then * as month, then * as day-of-week, then the trailing * is an extra field). Diagnose: events showing unparseable schedule. Fix: remove the seconds field; live with minute granularity.

  8. startingDeadlineSeconds: 0. Setting this to 0 means every schedule is missed by 0 seconds (i.e., immediately) and dropped — no Jobs are ever created. The default of “unset” (no deadline) is almost always correct; if you want a deadline, set ≥10 seconds.

  9. Cron expression with day-of-month and day-of-week both specified. 0 9 1 * 1 means “9 AM on the 1st of the month OR on Monday” in robfig/cron’s interpretation (the OR semantics, matching POSIX). This is not “9 AM on Monday the 1st.” Users coming from system cron sometimes assume AND. The defensive pattern: use * for one of the two fields.

  10. CronJob and kubectl create job --from=cronjob/<name>. Manually triggering a CronJob’s underlying Job creates a Job outside the CronJob’s concurrencyPolicy accounting — meaning a manual trigger followed by a scheduled fire 30s later under Forbid policy will skip the scheduled fire because the manual Job is “still running.” Diagnose: scheduled runs intermittently skipped after manual interventions. Avoid manual triggers under Forbid/Replace, or accept the skip.

Alternatives and When to Choose Them

  • Job alone — choose when the work is one-shot. CronJob is overkill for a manually-triggered backup; just create a Job (or use kubectl create job).
  • @every macro vs scheduled — @every 30m is convenient but not portable outside K8s (or other robfig/cron consumers). For sub-day cadences, 0,30 * * * * (every 30 minutes on the hour and half-hour) is more portable; for irregular cadences (@every 7h13m), @every may be the only option.
  • External orchestrator (Airflow, Dagster, Prefect) — choose when scheduling logic is more complex than cron (data dependencies, backfills, conditional retries, parameter passing). These typically run on K8s themselves and create Pods or Jobs as their primitive. CronJob is fine for simple periodic work; it is not a workflow scheduler.
  • Knative Eventing’s PingSource — choose when the schedule should produce CloudEvents rather than Jobs (event-driven downstream). PingSource emits a CloudEvent on cron schedule; useful when the work should fan out via Knative messaging.
  • OS-level cron on a chosen node — pre-K8s pattern, sometimes still seen. Tied to a specific node, no HA, no observability, no audit trail. Avoid for any production cluster.
  • kubectl create job --from=cronjob — choose for manual ad-hoc trigger of a CronJob’s template (e.g., “run the backup right now, ahead of schedule”). Reuses the jobTemplate so you don’t have to re-specify the image/env/etc.

Production Notes

  • Spotify and Shopify both use CronJobs heavily for ETL pipelines, daily aggregations, and certificate-renewal hooks, typically with concurrencyPolicy: Forbid + tight activeDeadlineSeconds. The “Etc/UTC” + UTC-shifted-schedule pattern is the most common — they explicitly avoid spec.timeZone for now to keep schedules portable across regions.
  • cert-manager (cert-manager.io) does not use CronJob for certificate renewal; it uses its own custom controller with RequeueAfter periodic reconciliation, since the renewal cadence is per-certificate and not cron-shaped. This illustrates an important point: CronJob is for time-of-day work; interval-from-last-event work belongs in a custom controller.
  • k8s.af incidents involving CronJobs (Kubernetes Failure Stories) cluster around: (a) the 100-missed-schedule cap silently blocking work after a control-plane outage; (b) Allow concurrency causing database hammering during slow runs; (c) Forbid + hung Job causing weeks of skipped runs unnoticed; (d) timezone confusion from running schedules in kube-controller-manager’s local time before 1.27.
  • DigitalOcean Managed Kubernetes explicitly warns that CronJobs scheduled too frequently (sub-minute via custom builds, or many CronJobs at the same exact minute) can overwhelm the API server with simultaneous Job creations. The mitigation is to stagger schedules (1 * * * *, 2 * * * *, 3 * * * * instead of 0 * * * * for everything) so the controller’s reconciliation work spreads.
  • The KEP-3140 (timeZone) motivation explicitly cites Stripe and other multi-region operators as having had to write internal cron-translation tools to convert “9 AM Pacific” into the right UTC cron expression accounting for DST. The native spec.timeZone field eliminated this entire class of tooling.

See Also

Graph View

Backlinks