System Design: LinkedIn

Requirements

Functional Requirements:

• Users create professional profiles (work history, skills, education)
• 1st/2nd/3rd degree connection graph
• Job listings with personalized recommendations
• News Feed of professional updates from connections
• Messaging between connections (LinkedIn Messages)
• 'People You May Know' (PYMK) recommendations
• Company pages and follower relationships

Non-Functional Requirements:

• 950M registered users, 300M MAU; 50M companies listed
• Profile load under 200ms; connection degree lookup under 100ms
• Job recommendation freshness: new matches within 5 minutes of posting
• 99.9% availability; messaging requires strong durability

Scale Estimation

300M MAU, ~100M DAU. Each user has on average 400 connections → 300M × 400 / 2 = 60 billion edges in the social graph. Feed reads: 100M DAU × 5 feed loads = 500M feeds/day = 5,800 reads/sec. Job listings: 15M open jobs; 100K new jobs posted daily. PYMK computation for 300M users runs as a batch job. Profile views: 3B profile views/day = 34,700 views/sec — this is the hottest read path.

High-Level Architecture

LinkedIn's platform is built around three key systems. The Social Graph Service (built on LinkedIn's open-source Voldemort KV store, later migrated to Espresso, an MVCC document store) stores the connection graph. Profile data lives in Espresso sharded by member_id. The Feed service uses a pull-based architecture (unlike Twitter's push): at feed load time, the Feed Aggregator fetches updates from the user's top ~300 connections (ranked by affinity score), scores them using a gradient boosting model, and assembles the feed. Updates older than 7 days are excluded from consideration.

The Jobs subsystem runs on a separate infrastructure: job postings are indexed in a custom Elasticsearch cluster (Galene, LinkedIn's in-house search engine). Job recommendations run a batch + streaming pipeline: a batch job (Spark on Hadoop) computes member-job affinity scores nightly; a streaming layer (using Samza, LinkedIn's stream processing framework) handles real-time signals like profile updates triggering immediate re-ranking. PYMK recommendations use graph algorithms (common connections count, profile similarity cosine distance) computed as a nightly Spark job, with results cached in a member's PYMK Redis list.

Core Components

Graph Service (Connection Degrees)

Determining 1st/2nd/3rd degree connections at query time is computationally expensive for graph traversal. LinkedIn pre-computes adjacency lists stored in a sharded in-memory graph store. First-degree connections are a direct lookup (adjacency list for member_id). Second-degree connections (friends of friends) are computed using a two-hop BFS capped at the top 5,000 results by affinity score. The graph is loaded into RAM across a cluster of high-memory machines (2TB RAM each), partitioned by member_id range.

Profile Service

Profiles are stored in Espresso (LinkedIn's MVCC document store over MySQL) with member_id as the partition key. Hot profiles (visited >1,000 times/day) are cached in Couchbase with a 10-minute TTL. Profile completeness score (used in search ranking) is computed asynchronously and updated via a Change Data Capture pipeline on profile writes. A Profile View event triggers a Kafka message processed by the Analytics Service and the 'Who Viewed Your Profile' feature.

Feed Ranking Service

LinkedIn's feed uses a pull-based model to avoid write amplification for high-follower influencers. At feed load time, the system fetches posts from the user's top connections (ranked by an affinity model trained on click and dwell time signals), scores each post using a gradient boosting model (GBM trained on 50+ features including post type, author affinity, engagement velocity), and applies diversity constraints. A separate injection system adds sponsored content and algorithm-suggested posts.

Database Design

Member profiles and connection data live in Espresso, a distributed document store with MVCC for concurrent reads. The schema: Members table (member_id, name, headline, location, profile_data JSONB, created_at), Connections table (member_id_a, member_id_b, connected_at, connection_type). The Connections table is replicated to an in-memory graph store for fast BFS traversal.

Job postings use a separate Galene (Elasticsearch-based) index for full-text search. Job recommendations are stored in a per-member Redis sorted set (job_id scored by relevance). Activity feed events (job changes, posts, likes) are stored in a Kafka-backed event stream consumed by both the Feed Service and Analytics. A Pinot (real-time OLAP) cluster serves analytics queries on member activity with sub-second latency.

API Design

• GET /v2/connections?q=viewer&viewerUrn={member_urn}&start=0&count=20 — Fetch a member's 1st-degree connections
• GET /v2/feed?count=20&start=0 — Fetch ranked professional feed for the authenticated member
• GET /v2/jobRecommendations?member_id={id}&count=25 — Fetch personalized job recommendations
• POST /v2/connections — Send a connection request; body contains invitee_urn and message

Scaling & Bottlenecks

The in-memory social graph is the most memory-intensive component. With 60B edges at ~20 bytes each, the full graph requires ~1.2TB of RAM. LinkedIn distributes this across a cluster of high-memory machines with consistent hashing. The graph is partitioned by member_id; cross-partition edges are stored on both partitions for locality. Graph updates (new connections) are propagated via Kafka to all graph replicas within seconds.

Profile read scaling uses a multi-tier cache: Couchbase L1 cache (hot profiles), a CDN edge cache for public profile HTML (30-minute TTL), and database read replicas for cold profiles. The 'thundering herd' problem on cache expiry is handled with probabilistic early expiration — the cache extends TTL randomly within a window to stagger refreshes. Feed generation is the most latency-sensitive path; it uses speculative execution (fire requests to 3 nodes, take the fastest response) for connection data fetches.

Key Trade-offs

• Pull-based feed over push: Avoids write amplification for high-follower members (celebrities/influencers) at the cost of higher read-time computation — acceptable since feed latency SLA is 300ms
• Pre-computed PYMK vs real-time: Nightly batch computation of PYMK is much cheaper than real-time graph traversal; stale recommendations (1 day old) are acceptable for this feature
• Espresso MVCC over simple MySQL: MVCC allows high-concurrency profile reads without read locks, critical for the 34K profile views/sec workload
• Galene (Elasticsearch) for job search: Full-text search with field boosting and geo-filtering is core to job discovery; a custom inverted index outperforms a general-purpose DB for this use case