Walkthrough: Notification System

A notification system is internal infrastructure: other teams hand it a message ("your order shipped", a login code, a marketing campaign) and it delivers via push, email, or SMS. It looks simple and is actually a queue-architecture showcase — priorities, retries, dedupe, and flaky third parties.

Step 1: Requirements

Functional:

1. Internal services can send a notification to a user (or a segment of users) on one or more channels.
2. Sends can be immediate, scheduled, or bulk campaigns.
3. User preferences are respected: channel opt-outs, quiet hours, per-user rate caps.

Non-functional:

• An OTP (one-time passcode) must arrive in seconds — latency matters unequally by type.
• At-least-once delivery, with dedupe: never silently drop a notification, never send the same one twice.
• Scale: steady ~100 sends/sec, but campaign bursts of 5,000/sec — spiky by nature.

Step 2: Core entities

Notification (type, channel, payload, priority, idempotency key), UserPreference (opt-outs, quiet hours, caps), DeliveryAttempt (status per channel per try).

Step 3: API

POST /v1/notifications
{ template_id, user_id | segment_id, channels[],
  priority, send_at?, idempotency_key }

One endpoint, internal-facing, called by other services. The idempotency key is supplied by the caller — that decision pays off in the dedupe dive.

Step 4: High-level design

The pipeline: ingestion API → validation → queues → per-channel workers → providers → tracking.

1. Ingestion authenticates the calling service and enqueues. It does not talk to providers — accepting and delivering are decoupled so a provider outage never blocks callers.
2. Validation checks preferences: opted out of this channel? Inside quiet hours? Over their rate cap? Drop or defer accordingly.
3. Queues absorb the spike: a campaign dumps 5,000/sec in, workers drain at their own pace. This is the textbook queue use case — buffering bursty writes.
4. Per-channel workers: push workers speak APNs/FCM (Apple/Google push gateways), SMS workers speak a Twilio-style API, email workers an SES-style API. Channels are isolated — separate queues and worker pools — so an email-provider outage does not delay SMS.
5. Delivery tracking records every attempt and outcome, for both debugging and the observability probe you should expect.

Deep dive 1: Priorities — the OTP behind a campaign

The classic probe: a marketing campaign enqueues 1M emails, and then a login OTP arrives. In a single queue the OTP waits behind a million promos — a user staring at a login screen for 20 minutes. The fix is structural, not clever: separate queues per priority lane (transactional/OTP vs promotional), with dedicated worker pools for the high-priority lane. A priority field in one queue is not enough — a burst still fills the pipe ahead of the OTP. Separate lanes mean the OTP path is never contended.

Deep dive 2: At-least-once + dedupe

Queues deliver at-least-once (a worker can crash after sending but before acknowledging, so the message is redelivered). Unhandled, that means duplicate texts. The fix is an idempotency check at the send boundary: before calling the provider, the worker does an atomic check-and-set on the notification's idempotency key in a dedupe store (Redis SET NX with a TTL covering the retry window). Already set → someone sent it → skip. This is why the caller supplies the key: retries by the caller dedupe too.

Deep dive 3: Flaky providers

Third-party providers fail routinely. Layer the defenses:

• Retries with exponential backoff and jitter (wait 1s, 2s, 4s... plus randomness so retries don't synchronize into a new stampede).
• A circuit breaker per provider: after N consecutive failures, stop calling for a cooldown instead of hammering a dead API — and optionally fail over to a backup provider (a second SMS vendor) for what matters.
• A dead-letter queue (DLQ — a parking lot for messages that exhausted retries) so nothing vanishes silently; alert on DLQ depth.

Deep dive 4: Campaign fanout

"Send to segment X" where X is 10M users must not happen synchronously in the API call. The ingestion service enqueues one campaign job; fanout workers expand it in batches (query 10k user IDs at a time, enqueue individual sends, checkpoint the cursor). A crash resumes from the checkpoint rather than restarting — and rate-limit the expansion so campaigns can't starve the system. Preference checks run at send time, not enqueue time: a user who opts out mid-campaign stays opted out.

Wrap-up

Recap: one ingestion API decoupled from delivery by queues; priority lanes keep OTPs in seconds; at-least-once plus an atomic dedupe check yields effectively-once sends; provider failures are contained by backoff, circuit breakers, and DLQs; campaigns fan out in resumable batches. With more time: open/click tracking feeding back into rate caps, and localization.