Walkthrough: Payment System

"Design a payment system" is the correctness question. Nothing here is exotic — a service, a database, a third-party API — but every failure mode involves money, so the interview probes exactly the places where distributed systems get ambiguous: retries, timeouts, and disagreement between systems. Your job is to show you know where the traps are and name the standard defenses.

Step 1: Requirements

Functional:

1. Merchants can charge a customer's card and see the outcome.
2. Every money movement is recorded and auditable.
3. Refunds — acknowledge, then scope to the happy path with the interviewer's blessing.

Non-functional:

• Correctness above all: never double-charge, never lose a record. Strong consistency for anything financial.
• Scale: ~10k transactions/sec peak — significant but not the hard part; say so.
• We integrate a PSP (payment service provider — Stripe/Adyen; they talk to card networks). We never see or store raw card numbers — the client tokenizes directly with the PSP. This kills the PCI-compliance scope in one sentence.

Step 2: Core entities

Charge (merchant, amount, currency, status, idempotency key), LedgerEntry (append-only debit/credit records), WebhookEvent (async messages from the PSP).

Step 3: API

POST /v1/charges            { amount, currency, payment_token }
                            headers: Idempotency-Key: <uuid>
GET  /v1/charges/:id        -> { status: pending | succeeded | failed }
POST /v1/psp-webhooks       <- called by the PSP

The charge flow: create a Charge row (status: pending) → call the PSP → the definitive outcome usually arrives later via webhook (the PSP calls us back) → update status and write ledger entries.

Deep dive 1: Idempotency keys, end to end

The founding problem: a client calls POST /charges, the response times out. Did the charge happen? The client can't know — and if it blindly retries, the customer may be charged twice. Idempotency keys make retries safe:

1. The client generates a unique key per logical payment attempt and sends it on every retry of that attempt.
2. The server atomically records the key with a hash of the request on first sight (a unique constraint makes concurrent duplicates collide safely). If the key exists with a stored response → replay the stored response, execute nothing. Same key with a different request body → reject; that's a bug.
3. The server passes its own idempotency key downstream on the PSP call — Stripe et al. support this — so the server's retries can't double-charge either.

Same key → same result, no matter how many times, at every hop. That sentence is the contract; say it.

Deep dive 2: The double-entry ledger

Store money movements as an append-only double-entry ledger: every transaction writes two entries — a debit from one account and a credit to another — and the entries across the books must always sum to zero. Why this instead of a balance column you UPDATE?

• Auditability: an immutable history of every movement; balances are derived (sum of entries, plus periodic snapshots for speed), never overwritten.
• Error detection: if the books don't balance, something is wrong and the invariant tells you — a mutable balance column silently absorbs bugs.
• Corrections are entries too: you never edit history; you append a reversing entry.

Ledger writes are the strongly-consistent heart: a single Postgres with synchronous replication handles the volume (10k TPS is within one well-tuned instance's range; shard by merchant if growth demands). Everything else — notifications, analytics — hangs off asynchronously.

Deep dive 3: The ambiguous timeout

The sharpest probe in this interview: your PSP call times out. Did the charge go through? You genuinely don't know — the request may have died on the way there (no charge) or the response on the way back (charged). Blind retry risks double-charging without an idempotency key; with one it's safe but may still leave you unsure of state. The disciplined sequence:

1. Leave the Charge in pending — never guess an outcome.
2. Query the PSP for the status of that idempotency key/attempt.
3. Still ambiguous? Void/cancel the attempt if the PSP supports it, then retry cleanly with the same key.
4. Whatever slips through gets caught by reconciliation (next dive).

The theme interviewers reward: in payments you resolve ambiguity by asking, never by assuming.

Deep dive 4: Webhooks and reconciliation

Webhooks are the PSP's at-least-once, unordered delivery of truth — so treat them accordingly: verify the signature (anyone can POST to your endpoint), dedupe by event ID (deliveries repeat), and tolerate out-of-order (apply state transitions only if valid — a "succeeded" arriving after you already processed "succeeded" is a no-op; ignore stale transitions).

Even then, two systems recording the same money will drift — dropped webhooks, bugs, edge cases. Reconciliation is the nightly job that pulls the PSP's settlement report and compares it line-by-line with your ledger. Matches confirm; mismatches raise discrepancies for automated or human resolution (that missing "charged" from the ambiguous-timeout scenario surfaces here). This is the safety net under everything above — and mentioning it unprompted is where candidates separate.

Wrap-up

Recap: idempotency keys make every retry safe at both hops; the append-only double-entry ledger is the auditable, strongly-consistent source of truth; ambiguous failures resolve by querying and voiding, never guessing; webhooks are verified, deduped, and order-tolerant; and nightly reconciliation against the PSP catches whatever slipped. With more time: refund flows, multi-currency, and payout scheduling to merchants.