Design a notification system

Requirements

Say these out loud before drawing a single box.

Back-of-the-envelope math

Assumptions labeled.

Architecture

The centerpiece: priority queues → per-channel limiters → workers.

flowchart LR
    API[Notification API] --> VAL[Validation
prefs, quiet hours, dedup]
    VAL --> TPL[Template Service
render + localize]
    TPL --> QC[Priority Queues
Kafka: critical/high/normal/low]
    QC --> DISP[Dispatcher
strict priority drain]
    DISP --> RL1[Rate Limiter
SMS token bucket]
    DISP --> RL2[Rate Limiter
Push token bucket]
    DISP --> RL3[Rate Limiter
Email token bucket]
    RL1 --> WS[SMS Workers
Twilio]
    RL2 --> WP[Push Workers
FCM + APNs]
    RL3 --> WE[Email Workers
SES]
    WS --> LOG[(Delivery Log)]
    WP --> LOG
    WE --> LOG
    LOG --> RETRY[Retry with backoff
→ DLQ after N tries]
    RETRY --> QC

Requests are validated (preferences, quiet hours, idempotency-key dedup), rendered from templates, and enqueued by priority. The dispatcher always drains critical before high before normal before low. Each channel has its own token bucket in Redis sized under the provider's limit — a notification only leaves when its channel has a token. Failures go to a retry loop with exponential backoff, then a dead-letter queue for humans.

Component deep-dives

Send — validate, render, enqueue

sequenceDiagram
    participant C as Caller service
    participant A as Notification API
    participant P as Prefs + Dedup (Redis)
    participant T as Template Service
    participant Q as Kafka priority topic
    C->>A: POST /v1/notifications {user, template, vars, priority, channels}
    A->>P: idempotency key seen? prefs allow? quiet hours?
    alt duplicate or opted out or quiet hours
        P-->>A: reject with reason
        A-->>C: 200 {status: skipped, reason}
    else ok
        A->>T: render template + locale
        T-->>A: rendered bodies
        A->>Q: produce to {priority} topic
        A-->>C: 202 {notification_id, status: queued}
    end

Dispatch — the token bucket gate

sequenceDiagram
    participant D as Dispatcher
    participant B as Token Bucket (Redis)
    participant W as Channel Worker
    participant V as Provider (Twilio)
    D->>D: pick highest-priority non-empty queue
    D->>B: take token for sms:+1555 pool?
    alt token available
        B-->>D: granted
        D->>W: send job
        W->>V: API call
        alt success
            V-->>W: delivered
        else 429 / 5xx
            W->>D: retry with backoff + jitter
        end
    else bucket empty
        B-->>D: denied — requeue, try next channel
        Note over D: low-priority waits here.
This is throttling working as designed.
    end

API + data model

POST /v1/notifications
  { "user_id": "u_123", "template_id": "order_shipped",
    "vars": { "order": "A-9918" }, "priority": "normal",
    "channels": ["push", "email"],
    "idempotency_key": "ord-9918-shipped" }
  → 202 { "notification_id": "n_...", "status": "queued" }
  → 200 { "status": "skipped", "reason": "quiet_hours" }

GET  /v1/notifications/{id}   → delivery status + attempts

templates(user-facing) { template_id, channel, locale, body, version }
user_prefs             { user_id PK, sms/push/email_enabled, quiet_hours_start/end }
notification_log       { notification_id PK, user_id, priority, channel, status, attempts, created_at }
idempotency_keys       { key PK, notification_id }  -- 24h TTL in Redis

Trade-offs

Failure modes

What I'd actually build

Opinionated. Steal this for the interview.

Stack: Go API, Kafka with 4 priority topics, Redis token buckets (per channel, per sending number) + idempotency keys + prefs cache. One worker pool per channel with circuit breakers; Twilio primary with a backup SMS provider; FCM + APNs for push; SES for email with low-priority batching. Quiet hours enforced at dispatch, not at enqueue (a queued promo hitting quiet hours waits). Dashboards: delivery rate per channel, bucket utilization, queue age per priority, DLQ depth. Pager fires on bucket saturation, not just errors.

🔬 Interactive: notification pipeline

Compose notifications with priorities, then fire a burst and watch the pipeline triage: critical jumps the queue, low-priority waits for tokens.

Watch: the 🔴 critical items clear instantly while ⚪ low items pile up — and when the SMS bucket runs dry, even high-priority SMS waits. That's throttling working as designed, not a bug.