System Design: Disappearing Messages

Requirements

Functional Requirements:

• Messages auto-delete after a configurable timer (5s, 30s, 1m, 5m, 1h, 24h, 7d)
• Timer starts when the recipient views the message (view-once) or when the message is sent (time-based)
• Deletion occurs on both sender and recipient devices and on the server
• Support for disappearing mode at the conversation level (all messages auto-expire)
• View-once media: photos/videos that can only be viewed once before disappearing
• Deletion is best-effort: no cryptographic guarantee against screenshots

Non-Functional Requirements:

• Timely expiration: messages deleted within 5 seconds of timer expiry
• Support 10 billion ephemeral messages per day
• Server must not retain expired message content under any circumstances
• Multi-device consistency: message disappears from all synced devices
• Expiration system must not degrade messaging performance

Scale Estimation

With 10 billion ephemeral messages per day, the system creates 115,740 expiration events per second. Each message has an expiration record: ~32 bytes (message_id + conversation_id + expiration_timestamp). The expiration index must track all pending expirations: at any given moment, assuming an average timer of 24 hours, approximately 10 billion messages are in the 'pending expiration' state, consuming ~320GB of expiration index storage. Server-side deletion throughput: 115K deletes/sec from the message store (Cassandra). Client-side deletion notifications: 115K × 2 (sender + recipient average) = 230K deletion push events per second.

High-Level Architecture

The disappearing messages system adds an expiration layer on top of the standard messaging pipeline. When a disappearing message is sent, it is stored in the message store with an additional expires_at timestamp (computed as sent_at + timer_duration for time-based, or left null for view-once messages where the timer starts on first view). An Expiration Scheduler Service maintains a time-ordered index of all pending expirations.

The Expiration Scheduler uses a delayed queue pattern: a Redis sorted set (expirations) where each entry is scored by its expires_at timestamp. A fleet of Expiration Workers continuously polls the sorted set for entries whose score is ≤ current timestamp, processes them in batch (deletes from message store, emits deletion events), and removes them from the queue. For view-once messages, the timer is set when the recipient's device reports the view event; the Expiration Scheduler creates the expiration entry at that point.

Client-side deletion is triggered by server-pushed events. When the Expiration Worker deletes a message server-side, it emits a message_expired event to all devices in the conversation via the WebSocket infrastructure. Each client, upon receiving this event, deletes the message from its local database and UI. For offline devices, the deletion event is queued; upon reconnection, the client syncs and deletes locally. As a defense-in-depth measure, each client also runs a local expiration timer — if the server event is delayed, the client deletes locally when its own timer fires.

Core Components

Expiration Scheduler

The Scheduler maintains a Redis sorted set of pending expirations scored by expires_at Unix timestamp. When a disappearing message is created, an entry {message_id}:{conversation_id} is added with score = expires_at. Expiration Workers run a continuous loop: ZRANGEBYSCORE expirations -inf {current_timestamp} LIMIT 0 100 to fetch the next batch of expired entries, process them, and remove with ZREM. The sorted set provides O(log N) insertion and O(log N + M) range queries, efficient for 10 billion entries with proper sharding.

View-Once Handler

View-once messages have special lifecycle: the media can only be opened once per recipient. When the recipient opens a view-once message, the client sends a viewed event. The server: (1) marks the message as viewed for that recipient, (2) revokes the media URL (invalidates the CDN-signed URL), (3) sets a short expiration timer (30 seconds) for cleanup. If the message is a group view-once, each recipient gets one view independently. The media is encrypted with a temporary key that is deleted after all recipients have viewed (or after the conversation-level expiry timer, whichever comes first).

Deletion Propagation Service

The Deletion Propagation Service ensures all copies of an expired message are removed. On expiration: (1) delete the message record from Cassandra, (2) delete the media blob from object storage (if applicable), (3) delete the search index entry (if the message was indexed), (4) emit deletion events to all conversation participants' devices, (5) delete any cached copies in Redis. The service uses a checklist pattern — each deletion step is tracked in a deletion task table. A reconciliation job runs hourly to catch any missed deletions (e.g., if the media deletion failed due to a transient S3 error).

Database Design

Disappearing messages are stored in the same message table as regular messages, with additional columns: is_ephemeral (BOOL), expires_at (TIMESTAMP), timer_duration (INT seconds), timer_type (ENUM: send_based, view_based), viewed_by (SET<user_id>). The expires_at column is set at send time for time-based expiry, or updated by the view handler for view-based expiry. Cassandra TTL is NOT used for expiration because it doesn't provide deletion events or guarantee exact timing — instead, the application-level Expiration Scheduler handles deletion.

The expiration index in Redis uses sharded sorted sets: expirations:{shard} where shard = message_id % 64. This distributes the 10B entries across 64 sorted sets, each holding ~156M entries (~5GB). Each shard is served by a dedicated Redis instance. The deletion task log uses a PostgreSQL table: deletion_tasks (task_id, message_id, conversation_id, steps_completed JSONB, created_at, completed_at) for tracking and reconciliation.

API Design

• POST /api/messages — Send disappearing message: {conversation_id, content, ephemeral: {timer: 3600, type: 'send_based'}} or {..., ephemeral: {type: 'view_once'}}
• POST /api/messages/{message_id}/viewed — Report view-once message opened; triggers timer start and media URL revocation
• PUT /api/conversations/{id}/ephemeral_settings — Set conversation-level disappearing mode: {default_timer: 86400} (24 hours for all new messages)
• WebSocket EVENT {type: 'message_expired', message_ids: [id1, id2, id3], conversation_id} — Server push for client-side deletion

Scaling & Bottlenecks

The Expiration Scheduler is the critical path. With 115K expirations per second, the Workers must process entries faster than they arrive to avoid growing backlog. Each Worker processes a batch of 100 entries per cycle with ~50ms per batch (dominated by Cassandra deletes). With 60 Workers, throughput is 120K/sec — sufficient headroom. During spikes (a popular conversation with a 5-second timer), the queue may temporarily grow but catches up within seconds. The sorted set ZRANGEBYSCORE operation is O(log N + M) where M is the batch size, remaining efficient even with billions of entries.

Media deletion from object storage is the slowest step. S3 delete operations average 100ms each; at 115K messages/sec with 20% having media, that is 23K S3 deletes/sec. This is handled by a dedicated media deletion queue that buffers delete requests and issues them in batches using S3's batch delete API (1000 objects per request), achieving throughput of 100K deletes/sec with 100 concurrent batch requests. Failed deletions are retried with exponential backoff and flagged for the reconciliation job.

Key Trade-offs

• Application-level expiration over Cassandra TTL: Application-level deletion provides exact timing control, deletion events for client sync, and multi-store deletion (message + media + index), while Cassandra TTL only handles the database record and doesn't notify clients
• Best-effort deletion over cryptographic enforcement: True disappearing messages (where screenshots are impossible) would require trusted hardware (DRM), which is impractical on general-purpose devices; best-effort deletion with UI hints ('screenshot detected') is the pragmatic choice
• Client-side timer as defense-in-depth: Running a local timer alongside the server timer ensures messages disappear even if the server push is delayed, but means expiration timing may differ by seconds between devices
• View-once with URL revocation over stream-only delivery: Revoking the CDN URL after viewing is simpler than streaming media without caching, but a determined user could save the media during the viewing window — this is an acceptable limitation