Notification Service
An internal router and bookkeeper: services hand it a push, email or SMS, it decides when and whether to send, and the providers do the last mile. Watch a one-time code stay fast while a million-user campaign drains beside it.
- Daily volume: 10M/day · ~100/s sustained
- Campaign: 1M users in 5 min · >3,000/s
- Ceiling: ~5,000/s surge across channels
- OTP target: <= 5 s after acceptance
Upstream services
Engineering services that need to reach a user: auth sends OTPs and security alerts, marketing schedules campaigns, and the mobile apps register device tokens. Every send names a priority.
API gateway
Authenticates the internal API key, applies per-service rate limits, and sheds standard traffic before high-priority traffic when the system is saturated.
Notification service
Validates the send, looks up the user's addresses in its user data slice, prechecks preferences, and publishes ONE durable queue message first, then returns the id.
Postgres
The source of truth for status: one row per notification (one user, one channel), campaign definitions, SCHEDULED rows, preferences, and a synced slice of user contacts and device tokens.
User service sync
Keeps the service's slice of user email and phone in sync with the user service, so a send never makes a synchronous call to another team.
Scheduler cron
Every minute it publishes SCHEDULED rows that are due and starts campaign fanouts. It never sits on the OTP path; immediate sends do not wait for it.
Segments
A named set of users. The fanout reads a snapshot of membership at fanout start so the audience is fixed for this campaign.
SQS high queue
A separate durable queue for high-priority sends only, so an OTP never waits behind a million promos.
SQS standard queue
Holds standard sends and campaign fanout. Bursts are expected here; minutes of delay are acceptable.
High workers
An independent pool that consumes only the high queue: claim the dedup key, recheck preferences, call the provider within the high budget, record status, delete the message.
Standard workers
Consumes the standard queue within the standard provider budget, applying quiet hours and opt-outs at dispatch time.
Dedup store
Holds per-key leased claims and ack records so redelivery, POST retries and cron re-fanout rarely become a second send. Best effort, not exactly-once.
Preferences
Checked just before the provider call: channel opt-outs always apply; quiet hours apply to standard only; high priority bypasses quiet hours but not channel opt-outs. Blocked sends are SUPPRESSED with a reason.
Provider budget
Splits the shared ~5,000/s provider ceiling into a high reserve and a standard share, so campaigns cannot consume the capacity OTPs need.
SMS provider
Delivers SMS over carriers. Twilio p99 is normally under 1 s; in a brownout it can reach 30 s.
Email provider
Delivers email; returns bounces and complaints asynchronously.
Push: APNs / FCM
Delivers push to device tokens registered by the apps; invalid-token responses retire the token.
