case-study

Redesigning Plaid's Transaction Processing Pipeline

Type: Architecture Case Study Company: Plaid Role: Staff Software Engineer (Tech Lead) Outcome: P99 latency reduced from 450ms to 62ms at 2B+ events/day

Plaid's transaction enrichment pipeline ingests raw bank transactions and runs them through a series of enrichment steps — merchant normalization, category tagging, recurring payment detection, and fraud signal generation — before fanning the enriched records out to 6,000+ connected applications.

By late 2022, the pipeline was struggling. P99 latency had climbed to 450ms and was trending up as transaction volume grew. Several enrichment steps had been bolted on over the years without a coherent execution model, leading to redundant database reads and inefficient fan-out patterns. On high-traffic days (tax season, Black Friday) the pipeline would visibly degrade and cascade failures into downstream consumer-facing features.

Zero downtime — the pipeline processes transactions for 6,000+ fintech apps; any cutover had to be seamless
Strict ordering guarantees — enriched transactions must arrive at consumers in the same order they were ingested, even under retry
12-person team — needed to ship incrementally; a full rewrite wasn't on the table
Kafka as the backbone — replacing the message bus was out of scope

Options Considered

Option A: Micro-optimise the existing pipeline Profile individual enrichment steps, add caching, reduce redundant DB reads. Low risk, low effort. We estimated a 20–30% latency improvement but not a step-change. Also wouldn't address the structural fan-out problem.

Option B: Parallel enrichment with a merge step Run independent enrichment steps concurrently and merge results before fan-out. Higher complexity but potentially large latency gains. Risk: merge step adds coordination overhead and a new failure mode if any enricher is slow.

Option C: Pre-materialised enrichment caches with async refresh Move enrichment lookup data into Redis-backed caches, refreshed asynchronously. Enrichment steps become cache reads rather than DB queries. Risk: cache staleness; requires careful invalidation logic and a fallback path.

We chose Option C as the primary intervention, paired with a smaller version of Option B for the enrichment steps that were already independent.

The key insight was that most latency was coming from synchronous DB reads during enrichment, not from the enrichment logic itself. Merchant normalization was hitting Postgres on every event; category tagging was doing the same. These tables are large but change slowly — perfect candidates for Redis-backed read caches.

We built an async cache refresh service that subscribes to a separate change-data-capture stream from Postgres and keeps Redis in sync with a typical staleness of < 500ms. Enrichment steps now do Redis lookups (sub-millisecond) with a Postgres fallback on cache miss.

For steps that were genuinely independent (category tagging, recurring payment detection), we parallelised execution within the enrichment worker using Go's goroutines. A merge step collects results with a hard 200ms deadline; if any enricher misses the deadline, it's applied asynchronously in a second pass.

The ordering guarantee is preserved through a sequence-numbered envelope on the Kafka message. The merge step respects the sequence; the async second-pass enricher only emits enriched fields as a patch — never a full record replacement.

Architecture Diagram

Kafka (raw transactions) │ ▼ Enrichment Worker (Go, horizontally scaled) ├── Redis Cache ──── Merchant Normalizer ─────────┐ ├── Redis Cache ──── Category Tagger ─────────────┤ merge (200ms deadline) └── Redis Cache ──── Recurring Detector ──────────┘ │ Fan-out Worker │ ┌────────────┼────────────┐ ▼ ▼ ▼ App Webhook App Webhook App Webhook (6,000+)

Async Cache Refresh Service ├── subscribes to CDC stream (Postgres → Debezium → Kafka) └── writes to Redis with 5-minute TTL (hard expiry safety net)

P99 latency: 450ms → 62ms (86% reduction)
Peak throughput headroom: increased from ~1.8B to ~4B events/day before degradation
Cascading failures during high-traffic days: eliminated (rate-limiting service, separate project, also contributed)
MTTR for pipeline incidents: reduced from 47 minutes to 11 minutes after adding structured observability spans

The parallel enrichment model has since been adopted as the standard pattern for new enrichment steps at Plaid. The async cache refresh architecture was reused for a separate ML feature serving project.