case-study

Distributed Rate Limiting at Plaid

Type: Architecture Case Study Company: Plaid Role: Staff Software Engineer (Designer and Tech Lead) Outcome: Eliminated cascading failure events during peak load

Plaid's payments APIs are called by thousands of fintech applications, many of which have spiky traffic patterns — payroll runs, end-of-month settlement batches, Black Friday surges. Without effective rate limiting, a single high-volume caller could saturate shared database connection pools and cause latency spikes for unrelated callers, a classic "noisy neighbour" failure mode.

The existing rate limiting was per-instance and naïve: each API server maintained its own in-memory counter, which meant limits were effectively multiplied by the number of running instances. A 1,000 req/s limit with 20 instances was actually 20,000 req/s — not meaningful protection.

Sub-millisecond overhead — rate limit checks happen on every API request; anything slow would directly inflate P99 latency
Correct under distributed conditions — limits must hold even as instances scale up or down
Graceful degradation — if the rate limiting service itself is unavailable, we should fail open rather than take down the API
Configurable per caller — different apps have different negotiated limits

Options Considered

Option A: Centralised atomic counter in Redis Simple sliding window or token bucket in Redis, incremented atomically with INCR + EXPIRE. Well-understood pattern. Risk: Redis becomes a single point of failure; adds a Redis roundtrip to every request.

Option B: Token bucket with gossip synchronisation Each instance maintains a local token bucket. Instances periodically gossip their current rate to peers and adjust their local bucket based on aggregate consumption. No single SPOF. Risk: complex to implement correctly; eventual consistency means limits can be temporarily exceeded.

Option C: Lease-based distributed rate limiting Instances request a "lease" (a batch of tokens) from a coordination layer, consume locally until the lease is exhausted, then request another. Coordination is Redis-backed but hit infrequently. Combines the simplicity of centralised counting with the performance of local counters.

We chose Option C. The lease model gave us the correctness of centralised counting with dramatically reduced Redis pressure — instead of one Redis call per API request, we make one call per lease (typically every few hundred requests per instance).

The lease coordinator is a thin Go service backed by Redis Cluster. When an API instance needs tokens, it calls the coordinator with its caller ID and the number of tokens it wants. The coordinator deducts from the shared budget atomically (using a Lua script to ensure the check-and-deduct is atomic) and returns a signed lease with a TTL.

The API instance consumes from the lease locally (in-process, no network calls) until it's exhausted or the lease TTL expires. When either happens, it requests a new lease. Lease TTLs are short (500ms) to bound the maximum overshoot.

If the coordinator is unavailable, instances fall back to a local-only mode with a conservative limit derived from the last known config. This is the "fail open" path — callers continue to be served, but at a reduced rate that avoids overwhelming the database.

Caller configs (limits, burst size, lease TTL) are stored in Postgres and cached in the coordinator with a 60-second TTL, so config changes propagate quickly without requiring a deploy.

API Request │ ▼ API Server (Go) │ ├─── local lease available? ──── Yes ──► consume token, serve request │ └─── No ──► Lease Coordinator (Go service, 3 replicas) │ ├─── Redis Cluster (atomic Lua deduct) │ └─── return signed lease (N tokens, 500ms TTL) │ API Server (cache lease, consume locally)

Fallback (coordinator unreachable): API Server uses local conservative limit from last-known config (no Redis call)

Cascading failure events from noisy-neighbour callers: eliminated in the 6 months post-launch
Rate limit check overhead: < 0.2ms average (vs. ~4ms with per-request Redis calls)
Redis request volume from rate limiting: reduced by 98% (lease batching)
Overshoot worst case: bounded to < 5% above configured limit (tested under adversarial conditions)

The lease-based model has since been adopted as our standard rate limiting primitive. Three other teams have built on top of the coordinator rather than implementing their own.