System Design: Payment Gateway

Requirements

Functional Requirements:

• Accept and process card-present and card-not-present payments (credit, debit, prepaid)
• Support authorization, capture, void, and refund transaction lifecycle
• Route transactions to optimal acquiring bank based on cost, approval rate, and latency
• Tokenize and vault sensitive cardholder data in PCI DSS Level 1 compliant storage
• Provide merchant dashboards with real-time settlement reporting and reconciliation
• Support 3D Secure authentication and fraud scoring integration

Non-Functional Requirements:

• Process 10,000 transactions per second at peak with p99 latency under 500ms
• 99.999% availability for the authorization path (five nines)
• ACID compliance for all financial state mutations with double-entry bookkeeping
• Full PCI DSS Level 1 compliance with end-to-end encryption and tokenization
• Idempotency guarantees on all payment operations to prevent duplicate charges

Scale Estimation

At 10,000 TPS peak, that is 864M transactions/day. Average transaction payload is 2KB including card data, merchant info, and metadata. Inbound data rate: 20MB/sec at peak. Each transaction triggers 3-5 downstream calls (fraud check, card network authorization, ledger write, webhook notification) producing 30,000-50,000 internal RPC calls/sec. Settlement files are generated nightly per acquiring bank — with 50 acquirers and 864M daily transactions, settlement file generation involves scanning and aggregating billions of rows. Token vault stores 500M+ unique card tokens requiring sub-millisecond lookup.

High-Level Architecture

The payment gateway is architected as a set of isolated security zones aligned with PCI DSS requirements. The outermost zone is the API Gateway (Kong/Envoy) handling TLS termination, authentication via API keys with HMAC signatures, rate limiting per merchant, and request routing. The Cardholder Data Environment (CDE) is a network-isolated zone containing the Tokenization Service and Card Vault — the only components that ever see raw card numbers. All other services operate on opaque tokens.

The transaction flow follows: Merchant sends payment request with card data → API Gateway routes to the Ingestion Service → Ingestion Service calls the Tokenization Service to exchange raw PAN for a token → the Token and payment details are passed to the Transaction Orchestrator, which implements a saga pattern coordinating: (1) Fraud Scoring Service, (2) Routing Engine (selects optimal acquirer), (3) Acquirer Adapter (translates to ISO 8583 or card network-specific protocol), (4) Ledger Service (records double-entry bookkeeping entries). Each step is idempotent, keyed by a merchant-provided idempotency key.

The Ledger Service is the financial source of truth, implementing double-entry bookkeeping where every transaction creates balanced debit and credit entries across accounts (merchant account, platform fees account, acquiring bank settlement account). The ledger uses PostgreSQL with serializable isolation and append-only entries — no updates or deletes are ever permitted on ledger rows.

Core Components

Transaction Orchestrator

The orchestrator implements the payment lifecycle as a state machine with states: CREATED, FRAUD_CHECKED, ROUTED, AUTHORIZED, CAPTURED, SETTLED, VOIDED, REFUNDED. Each transition is guarded by preconditions (e.g., cannot capture without prior authorization, cannot refund more than captured amount). The state machine is persisted in PostgreSQL with optimistic locking on a version column. Idempotency is enforced by storing a hash of (merchant_id, idempotency_key) in a unique index — duplicate requests return the cached response from the first execution. The orchestrator uses the saga pattern with compensating transactions: if acquirer authorization succeeds but ledger write fails, a compensating void is issued to the acquirer.

Smart Routing Engine

The routing engine selects the optimal acquiring bank for each transaction based on a scoring model considering: (1) historical approval rates by card BIN range and acquirer, (2) transaction cost (interchange + acquirer markup), (3) current acquirer latency and error rates (circuit breaker status), (4) merchant-specific routing rules (e.g., prefer Acquirer A for international cards). The engine maintains a real-time scoring table updated by a Flink job consuming transaction outcome events from Kafka. Failover routing is automatic: if the primary acquirer rejects or times out, the engine retries with the next-best acquirer (up to 2 retries), ensuring higher overall authorization rates.

Card Vault & Tokenization Service

The Card Vault is a hardened, network-isolated service storing encrypted PANs with AES-256-GCM using envelope encryption (data encryption keys wrapped by a master key in an HSM). Tokenization maps each unique PAN to a format-preserving token that passes Luhn validation (enabling downstream systems to perform BIN lookups on the token). The vault exposes only two operations: tokenize (PAN in, token out) and detokenize (token in, PAN out, restricted to the Acquirer Adapter service). Access is controlled via mutual TLS with certificate pinning and requires a signed JWT from the Transaction Orchestrator. All vault access is logged to an immutable audit trail in a separate compliance database.

Database Design

The Ledger database is PostgreSQL with strict serializable isolation. The core table is ledger_entries with columns: entry_id (UUID), transaction_id, account_id, entry_type (DEBIT/CREDIT), amount (BIGINT in smallest currency unit — cents), currency (ISO 4217), created_at, and metadata (JSONB). A constraint trigger ensures every transaction_id has balanced debits and credits (sum of debits = sum of credits). The accounts table tracks account_id, account_type (MERCHANT, PLATFORM_FEE, SETTLEMENT, RESERVE), and current_balance (maintained via a trigger on ledger_entries inserts). The transactions table stores transaction_id, merchant_id, idempotency_key_hash, status, amount, currency, card_token, acquirer_id, acquirer_reference, and timestamps.

The token vault uses a dedicated PostgreSQL instance with TDE (Transparent Data Encryption) and column-level encryption. Schema: token_id, encrypted_pan, pan_hash (SHA-256 for dedup), card_brand, last_four, expiry_month, expiry_year, created_at. The pan_hash column enables checking if a card is already tokenized without decrypting.

API Design

• POST /v1/payments — Create and authorize a payment; body contains amount, currency, card (number, exp, cvv) or token, merchant_id, idempotency_key; returns payment_id, status, acquirer_response_code
• POST /v1/payments/{payment_id}/capture — Capture a previously authorized payment; optional amount for partial capture
• POST /v1/payments/{payment_id}/refund — Refund a captured payment; body contains amount (partial refunds supported), reason, idempotency_key
• GET /v1/payments/{payment_id} — Retrieve payment details including full state machine history and ledger entries

Scaling & Bottlenecks

The Ledger Service is the primary bottleneck due to serializable isolation requirements. Write throughput is scaled via horizontal sharding by merchant_id — each shard handles a subset of merchants with independent serializable guarantees. Cross-shard transactions (e.g., platform fee aggregation) use eventual consistency with a reconciliation job. PostgreSQL connection pooling via PgBouncer in transaction mode is essential — at 10K TPS with 5 shards, each shard handles 2K TPS requiring 200+ concurrent connections.

The Card Vault is latency-sensitive and cannot be sharded easily due to deduplication requirements (same PAN must always map to same token). This is solved with a distributed cache (Redis cluster) storing the token-to-PAN-hash mapping, with PostgreSQL as the durable backing store. HSM operations (key wrapping) are the true bottleneck at ~1,000 operations/sec per HSM — addressed by caching unwrapped data encryption keys in memory with a 1-hour TTL and using a pool of 10+ HSMs.

Key Trade-offs

• Serializable isolation for ledger over read-committed: Prevents phantom reads and write skew that could cause financial discrepancies, but limits write throughput to ~2K TPS per shard — acceptable given sharding strategy
• Saga pattern over distributed transactions (2PC): Sagas with compensating voids/refunds provide better availability than 2PC across acquirer and ledger, but introduce windows where authorization exists without a ledger entry — reconciliation jobs detect and resolve these
• Format-preserving tokenization over random tokens: Tokens that pass Luhn and preserve BIN range enable downstream BIN-based routing without detokenization, but reduce the token keyspace — mitigated by adding a merchant-scoped namespace to tokens
• Append-only ledger over mutable balances: Immutable ledger entries create a complete audit trail required by financial regulators, but balance queries require aggregation — solved with materialized balance columns updated by database triggers