API Priority and Fairness

API Priority and Fairness (APF) is the kube-apiserver’s built-in overload-protection mechanism: instead of one global cap on concurrent requests, it classifies every incoming request into a priority level, fair-queues requests within each level so no single misbehaving client can starve the others, and rejects (with HTTP 429) or queues excess load rather than letting the apiserver fall over (Kubernetes — API Priority and Fairness). It replaces the old, blunt --max-requests-inflight / --max-mutating-requests-inflight flags — which offered exactly two buckets (read and write) and no fairness — with two configurable API objects: PriorityLevelConfiguration (how much concurrency a class gets) and FlowSchema (which requests map to which class). APF reached stable/GA in Kubernetes v1.29, with the API group flowcontrol.apiserver.k8s.io/v1 (KEP-1040); it is enabled by default.

Why APF Replaced Max-In-Flight

The pre-APF apiserver protected itself with two integers. --max-requests-inflight capped concurrent non-mutating (read) requests; --max-mutating-requests-inflight capped concurrent writes. Once either cap was hit, the apiserver returned 429 to whoever asked next — indiscriminately. This has two fatal flaws. First, no isolation: a single buggy controller issuing thousands of expensive LIST requests could consume the entire read budget, and the apiserver would then 429 the kubelet’s node-status heartbeats and the scheduler’s binds — starving the control plane’s own critical traffic behind one client’s flood. Second, no priority: a leader-election renewal (losing which causes a control-plane component to fail over) was treated identically to a kubectl get pods from a developer’s laptop.

APF fixes both. It splits the apiserver’s total concurrency budget across many priority levels, so a flood in one level cannot exhaust another; and within a level it uses fair queuing so that many independent clients (“flows”) get roughly equal shares, and one client’s burst is queued behind its own fair share rather than in front of everyone else’s. Critical system traffic (kubelet heartbeats, leader election) gets its own high-priority levels that user traffic can never crowd out.

Mental Model

Picture the apiserver’s request-handling capacity as a fixed number of seats — concurrency slots. APF does two independent jobs. Classification: every request is matched against FlowSchemas (in priority-number order) and thereby assigned to a priority level and given a flow distinguisher (an identity string, e.g. the requesting user). Dispatch: each priority level owns a share of the total seats and a bank of queues; a request is shuffle-sharded into one of that level’s queues, and a fair-queuing scheduler picks which queued request executes next so that all flows in the level advance evenly.

flowchart TD
    REQ["Incoming request<br/>(user, verb, resource, namespace)"]
    FS["FlowSchema matching<br/>(lowest matchingPrecedence wins)"]
    PL["Priority level<br/>e.g. workload-high"]
    FD["Flow distinguisher<br/>e.g. ByUser -> 'system:serviceaccount:...'"]
    SS["Shuffle-shard into<br/>1 of N queues (handSize picks)"]
    FQ["Fair-queuing scheduler<br/>picks next by virtual finish time"]
    SEATS["Execute — occupies seats<br/>until request completes"]
    REJECT["Queue full / no seats<br/>-> 429 Too Many Requests"]
    REQ --> FS --> PL
    FS --> FD --> SS
    PL --> SS --> FQ
    FQ -->|seat available| SEATS
    FQ -->|over capacity| REJECT

What this diagram shows. Classification (top path: FlowSchema → priority level) and queuing (bottom path: flow distinguisher → shuffle-shard → fair queue) are separate decisions that meet at the scheduler. The insight to extract: priority level provides isolation between categories of traffic, while the flow distinguisher + shuffle sharding provide fairness within a category — you need both, because “give system traffic priority” and “don’t let one user starve other users” are different problems.

Classification: FlowSchema

A FlowSchema is the matcher. Its spec.rules list subject/resource/verb predicates (e.g. “requests from users in group X, doing verbs [get, list, watch] on resource pods”). Every FlowSchema has a spec.matchingPrecedence (a number, lower = checked first); the apiserver evaluates FlowSchemas in that order and the first match wins, assigning the request to the FlowSchema’s spec.priorityLevelConfiguration. A mandatory catch-all FlowSchema guarantees every request matches something.

The FlowSchema also names a flow distinguisher via spec.distinguisherMethod. This is what defines a “flow” (a unit of fairness) within the priority level. The options (docs):

• ByUser — each distinct requesting user is its own flow. Two users sharing a priority level get fair shares relative to each other.
• ByNamespace — the target resource’s namespace is the flow key. Traffic about namespace A is fair-queued against traffic about namespace B.
• (none) — no distinguisher; all requests matching the FlowSchema are treated as one flow (used for genuinely cluster-wide, uniform traffic).

The distinguisher is what makes the fairness meaningful: without it, all traffic in a level is one flow and the fair queuing degenerates to FIFO.

Concurrency: PriorityLevelConfiguration and Seats

A PriorityLevelConfiguration defines a class of service. Its spec.type is either Limited or Exempt.

Exempt levels bypass all APF limits entirely — requests are dispatched immediately regardless of load. This exists for traffic that must never be throttled (notably the built-in exempt level for the most critical system requests). Since v1.29 the exempt level’s own nominal/lendable behavior is tunable via ExemptPriorityLevelConfiguration, but conceptually: exempt = no queuing, no limit.

Limited levels share the apiserver’s finite concurrency. Each carries a LimitedPriorityLevelConfiguration with:

• nominalConcurrencyShares — a weight, not an absolute count. The apiserver’s total concurrency limit (ServerCL, derived from the sum of the old max-in-flight flags) is divided across the Limited levels in proportion to their shares. Per KEP-1040, each level’s nominal concurrency limit is NominalCL(i) = ceil( ServerCL × ACS(i) / (sum of all ACS) ), where ACS(i) is level i’s shares and the denominator sums shares across all Limited levels. So a level with 30 shares out of 100 total gets ~30% of the seats.
• borrowingLimitPercent — how much extra concurrency this level may borrow from other levels that are currently under-utilized, expressed as a percent of its nominal limit. This yields MaxCL(i) = NominalCL(i) + BorrowingCL(i).
• lendablePercent — how much of this level’s nominal concurrency it is willing to lend to other levels when it is idle, giving MinCL(i) = NominalCL(i) − LendableCL(i).

Borrowing and lending (added in the v1beta3 API in v1.26 and carried into GA v1) mean the seat allocation is dynamic: an idle level lends seats to a busy one, and the busy one borrows up to its cap, so the cluster uses its full capacity under skewed load while still guaranteeing each level its minimum when contention returns.

Seats are the unit of concurrency an executing request occupies. Most requests cost 1 seat. But APF widens the cost for expensive requests so they consume a fair share of capacity:

• A LIST request that returns many objects occupies multiple seats, roughly proportional to the estimated number of objects it will read (a LIST of 10,000 pods is not the same load as a GET of one pod).
• A write that triggers many WATCH notifications occupies extra seats for a period, because the apiserver must fan the change out to all watchers — that fan-out is real work that APF accounts for.
• WATCH establishment occupies a seat only briefly (for the initial burst), then releases it, since a long-lived idle watch consumes little ongoing CPU.

This seat-width model is why APF is fairer than counting raw request numbers: it measures work, not just count.

Dispatch: Shuffle Sharding and Fair Queuing

Within a Limited priority level, requests are not one big FIFO. Each level has a configurable number of queues (queuing.queues) and a hand size (queuing.handSize). A request’s flow (from the distinguisher) is hashed, and shuffle sharding deals it a small hand of handSize queues out of the total; the request goes to the shortest queue in its hand. The point of shuffle sharding (vs a plain hash to one queue) is probabilistic isolation: two different flows are very unlikely to be dealt the same hand of queues, so a heavy flow that fills its queues rarely collides with a light flow’s queues. KEP-1040 notes that with 128 queues and a hand size of 6, the chance a given light flow shares all its queues with a specific heavy flow is on the order of one in billions.

The fair-queuing scheduler then chooses which queued request runs next. It is a virtual-time (byte-cost-fair-queuing-style) scheduler: each request is assigned a virtual finish time computed as R(finish) = R(start) + width × cost, where width is the seat count and the scheduler dispatches in order of virtual finish time, breaking ties round-robin across queues. Because the finish time grows with a flow’s own accumulated work, a flow that has already consumed a lot of capacity gets later finish times and yields to flows that have consumed less — the essence of fairness. The implementation tracks a per-queue virtualStart rather than per-request timestamps, keeping the bookkeeping cheap.

If a request’s assigned queues are all full (queuing.queueLengthLimit reached) and no seat is available, the request is rejected with 429 Too Many Requests and a Retry-After header — a well-behaved client backs off and retries. A priority level can also be configured to reject immediately rather than queue (limitResponse.type: Reject) for latency-sensitive traffic that prefers a fast failure to a long wait.

The Default Priority Levels

Kubernetes ships mandatory and suggested FlowSchemas/PriorityLevelConfigurations so a fresh cluster is protected out of the box (docs). The default Limited/Exempt levels, roughly highest to lowest criticality:

Level	Purpose
exempt	Never throttled — the most critical system requests bypass APF entirely
system	Requests from system:nodes (kubelets) other than heartbeats — e.g. node/pod status
node-high	Node heartbeats (kubelet health reporting) — protected from starvation
leader-election	Leader-election requests from built-in controllers; starving these causes failovers
workload-high	Higher-priority requests from cluster workloads/controllers
workload-low	Lower-priority workload requests
global-default	Catch-all for traffic not matched by a more specific FlowSchema
catch-all	The mandatory last-resort match, sized deliberately small

The two mandatory objects — an exempt level and a catch-all FlowSchema→level pair — cannot be deleted; they guarantee that critical traffic is always exempt and that every request matches something. The rest are “suggested” defaults the apiserver reconciles but that admins may tune.

Observability and Debugging

APF is heavily instrumented. Every response carries two headers naming the classification decision: X-Kubernetes-PF-FlowSchema-UID and X-Kubernetes-PF-PriorityLevel-UID — so you can see exactly which FlowSchema and level a request hit. Three debug endpoints dump live state: /debug/api_priority_and_fairness/dump_priority_levels, .../dump_queues, and .../dump_requests (docs).

Key Prometheus metrics: apiserver_flowcontrol_rejected_requests_total (the alarm bell — non-zero means real throttling), apiserver_flowcontrol_dispatched_requests_total, apiserver_flowcontrol_current_inqueue_requests (queuing depth), apiserver_flowcontrol_request_wait_duration_seconds (how long requests wait — latency injected by APF), apiserver_flowcontrol_current_executing_requests, and apiserver_flowcontrol_nominal_limit_seats (the computed per-level seat allocation). The standard triage is: 429s appearing → check rejected_requests_total by priority_level and flow_schema labels → find the noisy flow → either give it its own level or fix the client.

Failure Modes and Tuning

• A controller floods global-default and 429s itself. A custom controller that doesn’t authenticate with a distinct identity, or that hammers LIST, lands in global-default (or catch-all) and exhausts that level’s small budget. Fix: give the controller its own FlowSchema/priority level, or make it use watches + informer caches instead of polling LISTs (see Watch and Informers).
• Priority inversion / deadlock in nested servers. If apiserver A serves a request that itself calls apiserver B (e.g. an Aggregated API Server or admission webhook), and both apply APF, the subsidiary call can be queued behind lower-priority work, deadlocking the parent. KEP-1040’s guidance: classify the subsidiary request into a higher priority level than the originating one, or exempt it.
• Over-throttling from mis-sized defaults. If critical traffic shares a level with a noisy neighbor, heartbeats or leader-election can be delayed. The design deliberately isolates node-high and leader-election for this reason; custom high-priority workloads should get their own level rather than sharing workload-high.
• Controllers should use distinct identities. KEP-1040 recommends that a controller managing many workloads authenticate per-instance (distinct usernames) so the ByUser distinguisher can fairly separate their flows rather than lumping them into one.

Alternatives and Boundaries

APF is server-side admission control on concurrency, and it is orthogonal to several neighbors it’s easy to confuse it with. It is not rate limiting by requests-per-second — it limits in-flight concurrency and fairness, not a token-bucket QPS. Client-side, the older client-go rate limiter (and the newer flow-controlled clients) throttle a single client before it ever hits the wire; APF is the server’s defense against all clients collectively. It is also distinct from Admission Controllers, which run after a request has been let through APF and authorized, and decide whether the object is allowed — APF decides whether the request gets to run at all, based on load, not content. And it is unrelated to Pod Priority and Preemption, which is about scheduling Pods, despite the shared word “priority.”

You can disable APF (--enable-priority-and-fairness=false), reverting to the old max-in-flight behavior, but on any non-trivial cluster this removes the isolation that keeps one bad client from taking down the control plane — it is almost never the right choice on a production cluster.

Uncertain

Verify: the exact removal version of the older beta APIs (v1beta1, v1beta2, v1beta3) of flowcontrol.apiserver.k8s.io. Reason: v1 GA’d in v1.29 and v1beta3 was deprecated then, but the precise release in which each beta is removed was not confirmed against a primary deprecation-schedule source in this task. To resolve: check the Kubernetes deprecated-API-migration guide for the flowcontrol group. uncertain

See Also

• kube-apiserver — APF lives inside it; the seat budget derives from the apiserver’s concurrency flags
• API Server Request Flow — APF is the flow-control stage before authn/authz/admission
• Admission Controllers — run after APF admits a request; content gate vs load gate
• Watch and Informers and List-Watch Semantics — the client patterns that keep traffic out of expensive LIST-heavy flows
• Aggregated API Server — the nested-server case where APF priority inversion can bite
• Pod Priority and Preemption — a different “priority” (scheduling), commonly confused with APF
• RuntimeClass — sibling cluster-scoped API object, consumed by a different control-plane component (the kubelet rather than the apiserver)
• Kubernetes MOC — parent map