System Design: Push Notification Service

Requirements

Functional Requirements:

• Send push notifications to iOS (APNs), Android (FCM), and web (Web Push) devices
• Support for targeted (single user), segmented (user group), and broadcast (all users) notifications
• Notification scheduling with timezone-aware delivery windows
• Delivery tracking: sent, delivered, opened, dismissed
• Rate limiting and user preference management (opt-in/opt-out per category)
• Rich notifications with images, action buttons, and deep links

Non-Functional Requirements:

• Process 10 billion notifications per day across all platforms
• End-to-end latency under 5 seconds for targeted notifications
• 99.95% delivery rate to platform gateways (APNs/FCM)
• Handle broadcast to 500M devices within 30 minutes
• Graceful degradation when a platform gateway is unavailable

Scale Estimation

10 billion notifications per day translates to approximately 115,000 notifications per second sustained, with peaks of 500K/sec during scheduled campaigns. Each notification payload averages 2KB (including metadata), producing ~20TB of notification data per day. Device registry: 2 billion device tokens across three platforms, consuming roughly 200GB of storage. Delivery receipts generate an additional 10 billion events per day. A single broadcast to 500M devices at 500K/sec takes approximately 17 minutes, requiring significant parallelism in the sender fleet.

High-Level Architecture

The system has three main phases: ingestion, routing, and delivery. The Ingestion Layer receives notification requests via a REST API or internal event triggers (e.g., a Kafka topic emitted by other services like the messaging or payments system). Each request contains: target (user_id, segment, or broadcast), payload (title, body, data, platform-specific overrides), and scheduling parameters. The Ingestion Service validates the request, applies rate limiting, and writes to a Kafka topic partitioned by priority level.

The Routing Layer reads from Kafka, resolves targets to device tokens using the Device Registry (a Cassandra table mapping user_id → list of device tokens with platform info), applies user preferences (checking opt-in status for the notification category), and produces per-device notification tasks on a delivery Kafka topic partitioned by platform (APNs, FCM, Web Push). For broadcast notifications, the Routing Layer streams through the entire Device Registry using a Cassandra scan with token-range pagination to avoid memory exhaustion.

The Delivery Layer consists of platform-specific sender workers. APNs Senders maintain persistent HTTP/2 connections to Apple's push gateway and send notifications using the APNs provider API. FCM Senders use the Firebase Admin SDK over HTTP to the FCM endpoint. Web Push Senders implement the Web Push Protocol (RFC 8030) with VAPID authentication. Each sender tracks delivery status and writes receipts to a ClickHouse analytics database.

Core Components

Device Registry

The Device Registry maps users to their device tokens. Stored in Cassandra with partition key user_id, each row contains a list of devices: {device_token, platform (ios/android/web), app_version, locale, timezone, registered_at, last_active_at}. Tokens are updated on every app launch and removed when APNs/FCM reports them as invalid (via feedback services). A background job prunes devices inactive for >90 days. The registry supports multi-app scenarios — a single user may have tokens from multiple apps.

Notification Router

The Router resolves abstract targets into concrete device-level tasks. For a targeted notification (user_id: 12345), it looks up the user's devices and creates one task per device. For a segmented notification (segment: premium_users_in_US), it queries a User Segment Service (backed by a precomputed Redis bitmap of user sets) and iterates through matching user IDs. For broadcasts, it scans the entire Device Registry. The Router also handles deduplication — if the same notification is submitted twice (idempotency key), only one delivery is attempted.

Platform Senders

Each platform sender is optimized for its gateway's protocol. The APNs sender maintains a pool of persistent HTTP/2 connections (each supporting 500 concurrent streams) and handles token-based authentication with JWT signed using the app's signing key (rotated hourly). The FCM sender batches up to 500 messages per HTTP request using the FCM v1 batch endpoint. Both senders implement exponential backoff with jitter for retries on 429 (rate limit) or 503 (server error) responses. Failed tokens are fed back to the Device Registry for cleanup.

Database Design

The Device Registry in Cassandra uses: partition key user_id, clustering key device_token. Columns: platform, app_id, locale, timezone, last_active, notification_preferences (a map of category → enabled/disabled). A secondary index on platform enables platform-scoped scans for broadcasts. Notification logs are stored in ClickHouse (columnar store optimized for analytics): notification_id, user_id, device_token, platform, category, status (sent/delivered/opened/dismissed), sent_at, delivered_at, opened_at. This enables queries like 'what was the open rate for category X last week?'

Scheduled notifications use a separate MySQL table: schedule_id, notification_payload (JSON), target, scheduled_time_utc, timezone_rule (send at 9am user-local-time), status (pending/processing/completed), created_at. A Scheduler Worker polls for due notifications every 10 seconds and feeds them into the ingestion pipeline. For timezone-aware delivery, the worker expands the schedule into per-timezone batches, each fired at the appropriate UTC time.

API Design

• POST /api/v1/notifications — Send notification: {target: {user_ids?: [], segment?: string, broadcast?: bool}, payload: {title, body, data?, image_url?}, platform_overrides?: {ios: {sound, badge}, android: {channel_id}}, schedule?: {send_at, timezone_rule?}}
• GET /api/v1/notifications/{id}/status — Fetch delivery status: counts of sent, delivered, opened, failed
• PUT /api/v1/users/{user_id}/preferences — Update notification category preferences: {categories: {marketing: false, transactional: true}}
• DELETE /api/v1/devices/{device_token} — Unregister a device token

Scaling & Bottlenecks

The primary bottleneck is the platform gateway rate limits. APNs allows ~50K notifications/sec per HTTP/2 connection (with multiple connections allowed), and FCM has a per-project rate limit of ~1M messages/minute. For broadcast scenarios, the system must respect these limits while maximizing throughput. The sender fleet uses a token bucket rate limiter per platform, dynamically adjusting based on 429 response rates. Multiple FCM projects can be used to multiply the rate limit.

The Device Registry scan for broadcasts is I/O-intensive. Scanning 2 billion rows in Cassandra at 10K rows/sec per node would take too long. The solution is parallel scanning: each Cassandra node handles its own token range, and the Router fleet processes ranges in parallel, achieving scan throughput of 1M+ rows/sec across the cluster. Additionally, a precomputed broadcast bitmap (stored in Redis) can provide a cached snapshot of all active device tokens, refreshed hourly, avoiding the full Cassandra scan.

Key Trade-offs

• Kafka over a task queue (SQS/RabbitMQ): Kafka provides ordered, replayable, partitioned processing ideal for high-throughput notification pipelines; the trade-off is operational complexity compared to managed queues
• Cassandra for Device Registry over DynamoDB: Cassandra enables token-range scans for broadcasts and avoids per-request pricing that would be prohibitive at 10B notifications/day; trade-off is operational overhead of running Cassandra
• Fire-and-forget delivery to platform gateways: Once handed to APNs/FCM, delivery is best-effort (the platform controls the last mile); the system tracks 'sent to gateway' not 'delivered to device', accepting this limitation as inherent to push notifications
• Timezone-aware scheduling over instant delivery: Respecting user timezone for marketing notifications improves open rates by 40% but adds complexity in schedule expansion and requires accurate timezone data in the device registry