Per-tenant API rate limiting
Keep one noisy tenant from starving everyone else — a fair, configurable request budget per tenant.
A single slice of a larger rate-limiting umbrella. This spec is the in-process, per-node, per-tenant limiter; the distributed shared-store limiter and non-tenant budget axes are deferred to later slices.
A per-tenant rate limiter for the public API. Each tenant gets a configurable request budget; requests within budget pass, over-budget requests are rejected with 429 Too Many Requests and a Retry-After. The limiter is cheap enough to sit in the hot path and fails open, so a limiter fault never takes the API down with it.
§1 Context
The public API has no per-tenant throttling. A single tenant running a backfill or a buggy retry loop can saturate shared capacity and degrade latency for every other tenant — we have seen it in three incidents this quarter.
The users are API consumers (who deserve a fair share and a clear signal when throttled), platform operators (who need to tune a tenant's budget without shipping code), and on-call SREs (who need the limiter to be observable and never the cause of an outage).
§2 User scenarios
Each story is independently testable: if you implemented only US1, you would still have a viable MVP.
As an API consumer, I want to stay served up to my tenant's budget and get a clear signal past it, so that one noisy neighbour can't starve me.
Acceptance: requests within budget return normally; the request that first exceeds the budget receives 429 with a Retry-After whose value elapses before the budget refills.
As a platform operator, I want to change a tenant's limit at runtime, so that I can grant a temporary raise or clamp an abuser without shipping code.
Acceptance: updating a tenant's policy takes effect for subsequent requests without a process restart or redeploy.
As an on-call SRE, I want to observe allow/deny decisions and remaining budget, so that I can tell throttling apart from an outage at 3am.
Acceptance: each decision emits a metric tagged by tenant and outcome; a throttled response carries rate-limit headers.
- ·Limiter backend fault mid-request — the request is allowed (fail-open), not rejected.
- ·A request with no resolvable tenant — treated as anonymous and billed to a shared anonymous budget, never allowed to bypass limiting.
§3 Requirements
Conformance keywords (MUST, SHOULD, MAY) follow RFC 2119.
Enforce a per-tenant budget, reject over-budget with 429
The API MUST enforce a per-tenant request budget over a rolling window and reject over-budget requests with 429 Too Many Requests and a Retry-After header.
Rationale: 429 + Retry-After is the interoperable contract clients already implement backoff against.
Runtime-configurable limit and window
Each tenant's limit and window MUST be configurable at runtime, and a change MUST take effect for subsequent requests without a redeploy or restart.
Tenant identity from trusted context, not headers
The tenant identity used for a decision MUST be derived from the authenticated request context, never from a client-supplied header, so a caller cannot spoof another tenant's budget.
Rate-limit headers on every response
Responses SHOULD carry X-RateLimit-Limit, X-RateLimit-Remaining, and X-RateLimit-Reset headers on both allowed and throttled responses.
Negligible latency on the hot path
Enforcement MUST add negligible latency to a request on the hot path.
Added latency the limiter imposes on a request on the hot path (good ÷ valid requests).
Fail open — a limiter fault never rejects traffic
The limiter MUST fail open: a limiter backend fault MUST NOT reject legitimate traffic.
Fraction of requests served without being wrongly rejected by a limiter fault.
- ×A distributed, shared-store limiter (Redis-backed) for multi-node consistency — this slice is in-process per node → defer-to TBD
- ×Global and per-endpoint budgets (limits on axes other than tenant) → defer-to TBD
§4 Data model
Only the entities this feature owns.
- RateLimitPolicy
- tenantId, limit (requests), window (duration), optional burst — the configured budget for a tenant.
- RateLimitState
- tenantId, tokens (remaining), updatedAt — the live counter the limiter mutates per request.
- Decision
- allowed (bool), remaining, retryAfter — the per-request verdict returned to the caller and the metrics pipeline.
Policy is read-mostly and changes rarely; state is write-heavy and per-request. Keeping them separate lets a config change (policy) apply without disturbing live counters (state).
§5 Success criteria
Technology-agnostic, measurable outcomes. These are how we know it worked.
One abuser can't degrade the fleet. Under a load test where one tenant sends 10× its budget, every other tenant's requests continue to succeed at their normal rate, and the abusive tenant receives 429 for its over-budget requests.
Config changes land fast, without a deploy. An operator changes a tenant's limit and the new limit governs decisions within 10 seconds, with no redeploy.
§6 Assumptions
Every authenticated request already carries a resolvable tenant identity in its context.
Per-node, in-process limiting is acceptable for v1; cross-node drift within a tenant's budget is tolerable until the distributed slice lands.
§7 Open questions
None outstanding — the algorithm choice, fail-open behaviour, and identity source were all anchored during the interview and moved to the design or into the requirements above.
§8 Change log
One file. Renders anywhere. Degrades to readable static HTML.