vault

Finalizers

15 min read

Finalizers

Finalizers are the “don’t actually delete me yet” mechanism in Kubernetes. When a delete is requested against an object that has a non-empty metadata.finalizers[] list, the API server records the user’s intent by setting metadata.deletionTimestamp and returns 202 Accepted, but does not actually remove the object from etcd until the finalizer list is empty. Controllers and operators that need to perform external cleanup before an object disappears — deleting a cloud load balancer, detaching a persistent volume, deregistering a workload from an external system — install their own finalizer key on the object at creation time, watch for the deletion timestamp to be set, perform their cleanup, and only then remove their finalizer entry. Once the last finalizer is gone, the API server reaps the object. The Kubernetes documentation states the contract precisely: “Finalizers are namespaced keys that tell Kubernetes to wait until specific conditions are met before it fully deletes resources that are marked for deletion.” (kubernetes.io — Finalizers). Finalizers are the only legitimate way to perform pre-deletion side effects in K8s — and because they are infrastructure, not code, they survive controller restarts, retries, and partitions in exactly the same way control loops do. They are also the leading cause of “why won’t this object delete” confusion in operations: a namespace stuck in Terminating, a PVC that won’t go away, a CR whose deletion hangs for hours — almost always a finalizer that did not (or cannot) clear.

Mental Model

sequenceDiagram
    participant U as User<br/>(kubectl delete X)
    participant API as kube-apiserver
    participant ETCD as etcd
    participant CTRL as Controller<br/>(owns the finalizer)
    participant EXT as External system<br/>(cloud, DB, etc.)

    U->>API: DELETE /apis/.../X
    API->>ETCD: SET metadata.deletionTimestamp=now()
    API-->>U: 202 Accepted<br/>(object still visible)
    Note over API,ETCD: finalizers list non-empty<br/>→ object NOT removed
    CTRL->>API: WATCH sees deletionTimestamp set
    CTRL->>EXT: perform cleanup<br/>(delete LB, drop DB, etc.)
    EXT-->>CTRL: cleanup done
    CTRL->>API: PATCH metadata.finalizers<br/>(remove our key)
    API->>ETCD: write updated finalizers
    Note over API,ETCD: finalizers list now empty<br/>+ deletionTimestamp set
    API->>ETCD: DELETE object
    API-->>U: subsequent GET → 404

What this diagram shows. Deletion in Kubernetes is two-phase: the user’s DELETE request is a declaration of intent, recorded in metadata.deletionTimestamp; the actual removal happens later, gated by the finalizer list. Controllers that wish to participate in deletion ordering register a finalizer on the object (at creation time, typically inside their reconciler) and react to the deletionTimestamp becoming non-zero by performing their cleanup and then patching their own key out of the list. The insight to extract is that finalizers are level-triggered in the same sense that controllers are: a controller crashing mid-cleanup is fine — when it restarts, it sees the deletion timestamp again, retries the cleanup, and removes its finalizer when done. There is no special “deletion event” to lose; the persistent deletionTimestamp is the signal. This is why finalizers are robust against the entire menagerie of distributed-system failures, and also why a misbehaving controller leaves objects stuck indefinitely — the API server has no concept of “this finalizer has been waiting too long, give up.”

Mechanical Walk-through

What a finalizer literally is

A finalizer is a string. The object’s metadata.finalizers[] array is a list of strings, each one a key naming a piece of cleanup logic that some controller has promised to perform before the object is reaped. The Kubernetes documentation makes the “not code” property explicit: “Finalizers don’t usually specify the code to execute. Instead, they are typically lists of keys on a specific resource similar to annotations. Some finalizers are added automatically by Kubernetes, but you can also add your own.” (kubernetes.io — Finalizers).

Custom finalizer names must be qualified with a DNS subdomain prefix:

metadata:
  finalizers:
  - example.com/cleanup-load-balancer        # good — domain-prefixed
  - my-operator.io/dump-state-to-s3          # good
  - some-finalizer                           # rejected by API server
  - mything                                  # rejected

The API server validates this on create / update and rejects unqualified custom finalizers. Built-in finalizers use the kubernetes.io/ or <kind>.kubernetes.io/ prefix or unprefixed names that were grandfathered in (e.g., foregroundDeletion).

The deletion state machine

The lifecycle of a finalized object under deletion (kubernetes.io — Finalizers):

“When you tell Kubernetes to delete an object that has finalizers specified for it, the Kubernetes API marks the object for deletion by populating .metadata.deletionTimestamp, and returns a 202 status code (HTTP ‘Accepted’). The target object remains in a terminating state while the control plane, or other components, take the actions defined by the finalizers. After these actions are complete, the controller removes the relevant finalizers from the target object. When the metadata.finalizers field is empty, Kubernetes considers the deletion complete and deletes the object.”

So the contract is:

StepAPI serverObject visible?
1. User issues DELETEsets deletionTimestamp to nowyes
2. Finalizer list non-emptyrefuses to remove from etcdyes, in Terminating
3. Controller-1 finishes its cleanup, removes its keyfinalizers list shrinksyes
4. … all other controllers do the same…yes
5. Last finalizer is removedAPI server removes object from etcdno — 404 thereafter

Crucially, once deletionTimestamp is set, you cannot un-set it. The object is on a one-way trip to deletion; the only question is when (i.e., when the finalizer list empties). Adding a new finalizer to an already-deleting object is forbidden (the API server rejects the update); existing finalizers can only be removed, not added.

Built-in finalizers

Kubernetes ships several finalizers that are added automatically by the system. The most commonly encountered:

  • kubernetes.io/pv-protection — added by the PV protection controller to PersistentVolume objects whose status is Bound. The docs give the canonical example: “A common example of a finalizer is kubernetes.io/pv-protection, which prevents accidental deletion of PersistentVolume objects. When a PersistentVolume object is in use by a Pod, Kubernetes adds the pv-protection finalizer. If you try to delete the PersistentVolume, it enters a Terminating status, but the controller can’t delete it because the finalizer exists. When the Pod stops using the PersistentVolume, Kubernetes clears the pv-protection finalizer, and the controller deletes the volume.” (kubernetes.io — Finalizers).
  • kubernetes.io/pvc-protection — same idea, on PersistentVolumeClaim objects in use by a Pod.
  • foregroundDeletion — added by the garbage collector to the owner during foreground cascade deletion. Removed by the GC when all blocking dependents are gone.
  • orphan — used during orphan cascade to coordinate dependent cleanup.
  • kubernetes — the Namespace finalizer, used to gate namespace deletion until every namespaced resource within is gone.

Operator-defined finalizers

Custom operators install their own finalizers. The canonical reconcile pattern, in pseudocode:

func (r *MyReconciler) Reconcile(ctx, req) (Result, error) {
    obj := r.Get(req.NamespacedName)
    myFinalizer := "myorg.io/cleanup"
 
    if obj.DeletionTimestamp.IsZero() {
        // Object is alive; ensure our finalizer is present.
        if !contains(obj.Finalizers, myFinalizer) {
            obj.Finalizers = append(obj.Finalizers, myFinalizer)
            r.Update(obj)
            return Result{}, nil               // requeue
        }
        // ...normal reconcile (do the actual work)
        return Result{}, nil
    }
 
    // DeletionTimestamp is set → we are being deleted.
    if contains(obj.Finalizers, myFinalizer) {
        if err := r.cleanupExternalResources(obj); err != nil {
            // Cleanup failed — exponential backoff, try again.
            return Result{}, err
        }
        // Cleanup succeeded — remove our finalizer.
        obj.Finalizers = remove(obj.Finalizers, myFinalizer)
        r.Update(obj)
    }
    return Result{}, nil                       // object will be reaped
                                               // when last finalizer goes
}

Three properties of this pattern are essential:

  1. Finalizer installation on first reconcile. The operator registers its finalizer the first time it sees the object. This ensures that every object the operator manages has the finalizer; objects created before the operator was installed get the finalizer on the next reconcile.
  2. Idempotent cleanup. The cleanupExternalResources function must be idempotent — it will be called every reconcile until it succeeds. If it deletes a cloud LB that’s already gone, that’s fine; the call must be safe to repeat.
  3. No “delete me” event. The operator never receives a deletion event — it sees DeletionTimestamp != 0 on the watched object. This is consistent with the level-triggered design of every K8s controller (see Kubernetes Control Loop Pattern).

Why namespace deletion uses this

Namespaces are deleted via finalizer. The namespace controller adds kubernetes to metadata.finalizers on every namespace; when a delete is requested, the controller transitions the namespace to status.phase: Terminating and proceeds to delete every namespaced resource within (iterating the discovery API). Only when all child resources are confirmed gone does it remove the kubernetes finalizer, allowing the namespace itself to be reaped. This is why deleting a namespace can take minutes (every resource type must be enumerated and deleted) and why a stuck dependent — usually a PVC or a CR with its own finalizer — keeps the namespace in Terminating forever.

Stuck-Finalizer Scenarios — The Most Common K8s Confusion

The single most-asked operations question about Kubernetes — “why won’t my namespace / PVC / CR delete?” — is almost always answered by: a finalizer is holding it. Diagnostic workflow:

# 1. Identify the stuck object.
kubectl get ns my-ns -o yaml | grep -A5 finalizers
kubectl get pvc -n my-ns my-pvc -o yaml | grep -A5 finalizers
 
# 2. Look for the controller that owns the finalizer (the prefix tells you).
#    "kubernetes.io/pv-protection" → the in-tree PV controller is still
#    holding the PV because a Pod is using it.
#    "kubernetes" on a namespace → the namespace controller is still
#    trying to delete contained resources.
#    "myorg.io/something" → your own operator.
 
# 3. List remaining resources in the namespace (if it's a namespace).
kubectl api-resources --verbs=list --namespaced -o name \
  | xargs -n1 kubectl get --show-kind --ignore-not-found -n my-ns
 
# 4. Find the resource that's actually blocking — it has its own finalizer
#    that isn't clearing. Investigate that controller's logs.

The “force delete” anti-pattern

The most common Stack Overflow answer is to patch the finalizers to null:

kubectl patch pvc my-pvc -n my-ns -p '{"metadata":{"finalizers":null}}' --type=merge
kubectl patch namespace my-ns -p '{"metadata":{"finalizers":null}}' --type=merge

This works — the object is reaped — but it is almost always wrong, and the K8s documentation says so explicitly: “In cases where objects are stuck in a deleting state, avoid manually removing finalizers to allow deletion to continue. Finalizers are usually added to resources for a reason, so forcefully removing them can lead to issues in your cluster. This should only be done when the purpose of the finalizer is understood and is accomplished in another way.” (kubernetes.io — Finalizers).

What can go wrong:

  • For a PVC: bypassing pvc-protection deletes the PVC even though a running Pod is still attached, leaving the Pod’s mount in undefined state.
  • For an operator-managed CR: skipping the operator’s cleanup finalizer leaves orphan external resources (cloud LBs, IAM roles, database schemas) that nothing will ever clean up. These are invisible leaks — the K8s side is healthy, but cloud bills grow.
  • For a namespace: skipping the kubernetes finalizer can leave child resources orphaned in etcd, accessible only via direct API calls and invisible to kubectl.

The correct workflow is:

  1. Find the controller. The finalizer prefix names it.
  2. Fix it. Restart the controller, look at its logs, see why it isn’t clearing the finalizer.
  3. If the controller is gone forever (e.g., uninstalled operator, deprecated webhook), and you’ve manually verified the external state, then patch the finalizer. Treat it as a forensic operation, not a routine workflow.

Force-delete is a break-glass tool. Reaching for it routinely is the hallmark of an unmaintained cluster.

Configuration / API Surface

A minimal CRD with an operator-managed finalizer, line-by-line:

apiVersion: example.com/v1
kind: DatabaseBackup
metadata:
  name: nightly-backup
  finalizers:                                  # set by the operator on
  - example.com/dump-to-s3                     # first reconcile; removed
                                                # after S3 upload succeeds
spec:
  source: postgres-primary
  target: s3://my-bucket/backups/
# Inspect the finalizers on a stuck object.
kubectl get databasebackup nightly-backup -o jsonpath='{.metadata.finalizers}'
# → ["example.com/dump-to-s3"]
 
# Check whether deletion has been requested.
kubectl get databasebackup nightly-backup -o jsonpath='{.metadata.deletionTimestamp}'
# → "2026-05-15T11:42:00Z"
 
# The operator's logs will reveal why the finalizer isn't being removed.
kubectl logs -n operators deployment/db-backup-operator | grep nightly-backup

Programmatically removing your own finalizer (Go, using controller-runtime):

import (
    "sigs.k8s.io/controller-runtime/pkg/controller/controllerutil"
)
 
// Add (idempotent — no-op if already present).
controllerutil.AddFinalizer(obj, "example.com/dump-to-s3")
 
// Remove (idempotent — no-op if already absent).
controllerutil.RemoveFinalizer(obj, "example.com/dump-to-s3")
 
// Persist the change.
if err := r.Update(ctx, obj); err != nil { ... }

Failure Modes

  1. Operator crash before finalizer removal. Cleanup completed, finalizer not yet removed, controller crashes. Symptom: object stuck in Terminating indefinitely. Mitigation: the operator must remove the finalizer atomically with (or after, but reliably retried) the cleanup. On restart, the operator’s reconciler sees the deletion timestamp again and re-runs the cleanup (which must be idempotent), then removes the finalizer.

  2. Cleanup never succeeds. External system permanently unreachable; operator retries forever. Symptom: object stuck. Mitigation: surface the failure via the object’s status.conditions[], emit Events, and ideally implement a manual-override annotation that the operator respects (e.g., example.com/force-cleanup-skip: true) — never via patching finalizers from the outside.

  3. Operator uninstalled while finalizer is present. No one is left to clear the finalizer. Symptom: every object the operator managed is stuck forever. Mitigation: operators should provide an uninstall procedure that removes finalizers from all managed objects before the operator itself is removed. Many operator authors miss this step.

  4. Two operators with overlapping finalizers. Rare but possible if the same finalizer string is used by two different controllers. Mitigation: domain-prefix discipline (myorg.io/...) and a one-finalizer-per-controller convention.

  5. Finalizer race with create. If a controller adds its finalizer in the reconcile loop, an object created and immediately deleted may never see the finalizer added. Mitigation: in a mutating admission webhook, add the finalizer at object creation (atomic with the create); or accept the small race window.

  6. Foreground deletion blocked by stuck child finalizer. A Deployment in foreground cascade deletion will not be reaped because its ReplicaSet → Pod has a finalizer that doesn’t clear. The whole tree freezes. Diagnostic: descend the owner tree (kubectl tree) and look at each level’s finalizers.

  7. Finalizers on cluster-scoped objects with cross-namespace cleanup needs. A finalizer on a cluster-scoped CR can drive cleanup of resources in many namespaces; if RBAC isn’t right, the operator can’t remove what it needs to, and gets stuck. Mitigation: give the operator a ClusterRole that covers all the namespaces it touches.

  8. metadata.finalizers immutable after deletionTimestamp set — except by patching null. The API server rejects adding a finalizer to an object whose deletionTimestamp is set; this is to prevent ever-extending deletion. Removal is allowed. Patching the whole array to null bypasses both checks (the patch type matters — strategic merge with explicit null replaces the slice).

Alternatives and When to Choose Them

  • Owner references (Owner References and Garbage Collection). Choose for in-cluster parent-child cleanup. Owner refs and finalizers compose: owner refs identify the dependents to delete; finalizers gate each one’s actual removal. Don’t try to express “delete this dependent when the parent is deleted” with a finalizer; use ownerReferences.
  • Admission webhooks (validating). Choose for “do not allow this object to be deleted at all” — a validating webhook on DELETE can outright reject deletion. This is rarer; the finalizer’s “delay-then-allow” semantics fit more cleanup scenarios than “deny outright.”
  • External state machines / Sagas. When the cleanup is a long-running, multi-step workflow with retries and compensations, a Saga orchestrator (Temporal, Cadence) called by the finalizer-removing controller is appropriate. The finalizer keeps the K8s object alive until the saga reports done.
  • TTL controllers. For objects that should self-delete after some time (Jobs, Events), the TTL-After-Finished controller is the right mechanism — it issues a normal DELETE which then goes through the finalizer machinery. Finalizers are about gating; TTL is about scheduling.
  • No finalizer. If cleanup is purely in-cluster and ownerReferences suffice, don’t add a finalizer. They are not free — they create stuck-object risk.

Production Notes

  • The most-debugged K8s symptom. Every cluster operator eventually encounters “namespace stuck in Terminating.” The diagnostic playbook above is essentially canon. The Kyverno / Gatekeeper communities have policies that warn or block creation of objects with finalizers from unrecognized controllers — defense against orphaned-operator situations.
  • PV / PVC protection finalizers are extremely common. Storage operators add them to gate volume detachment; running kubectl delete pvc while the Pod is still using it does not race-delete the volume — the finalizer holds until the Pod is gone.
  • Cert-manager uses finalizers extensively. Deleting a Certificate resource triggers cleanup of associated Secrets and ACME challenges via finalizer.
  • AWS Load Balancer Controller uses a finalizer on Service objects of type LoadBalancer to ensure the cloud LB is deprovisioned before the Service is reaped. Skipping this finalizer can leak AWS LBs (billable).
  • Operator authors: include the uninstall finalizer dance. When your operator is uninstalled (via helm uninstall or similar), the helm chart should run a hook that removes finalizers from all managed objects before the operator’s Deployment is deleted. Many operator authors miss this and leave users with stuck objects after uninstall.
  • k8s.af stories include outages where a “stuck Namespace” was force-deleted by patching finalizers, only for the underlying CRs (with their own external resources) to be orphaned; the cluster looked healthy and the cloud bill kept climbing for weeks.
  • GitOps interactions. ArgoCD’s “prune” feature must respect finalizers — when ArgoCD is configured to delete an object that has a finalizer, the actual deletion may take time, and ArgoCD must keep showing the object as “deleting” until it’s truly gone. This is occasionally surprising to users expecting instant pruning.

See Also

Graph View

Backlinks