Heartbeat Mechanism

A heartbeat is a periodic signal sent between nodes in a distributed system to indicate liveness, enabling other nodes to detect failures and trigger recovery actions like failover or re-replication.

What It Really Means

In a distributed system, nodes fail. Servers crash, networks partition, processes hang. The question is not whether failures happen, but how quickly you detect them. A heartbeat is the simplest answer: every node periodically says "I am alive" to its peers or to a central coordinator. If the signal stops, the node is presumed dead.

The challenge is distinguishing between a dead node and a slow one. If your heartbeat timeout is 5 seconds and a node takes 6 seconds to respond due to a garbage collection pause or network congestion, you will falsely declare it dead. This false positive triggers unnecessary failovers, re-elections, and data re-replication — all expensive operations. Set the timeout too high, and you wait too long to detect actual failures, leaving the system in a degraded state.

This is the fundamental tension in failure detection: speed vs accuracy. Every heartbeat configuration is a bet on how long a healthy node might be silent before you should assume it is dead.

How It Works in Practice

Push vs Pull Heartbeats

Push model: Each node actively sends heartbeat messages to a coordinator or peers at regular intervals. If the coordinator does not receive a heartbeat within the timeout, it marks the node as dead.

Apache Kafka uses push heartbeats. Consumer group members send heartbeats to the group coordinator broker. The heartbeat.interval.ms (default 3 seconds) controls how often heartbeats are sent. The session.timeout.ms (default 45 seconds) controls how long the coordinator waits before considering the consumer dead and rebalancing.

Pull model: A coordinator periodically polls each node with a health check request. If the node does not respond, it is marked as unhealthy.

Kubernetes uses pull-based liveness probes. The kubelet periodically sends HTTP requests, TCP connections, or exec commands to containers. If a container fails failureThreshold consecutive probes, Kubernetes restarts it.

Gossip-Based Failure Detection

Cassandra uses a gossip protocol for failure detection. Every second, each node picks a random peer and exchanges state information (including heartbeat timestamps). If a node's heartbeat timestamp has not been updated for a configurable period, it is marked as DOWN.

The advantage of gossip over centralized heartbeats is that there is no single point of failure. Every node participates in failure detection, and the information propagates organically through the cluster.

Phi Accrual Failure Detector

Instead of a binary alive/dead threshold, the Phi Accrual Failure Detector (used by Akka and Cassandra) outputs a suspicion level (phi) based on the statistical distribution of heartbeat arrival times. If heartbeats normally arrive every 1 second with a standard deviation of 50ms, a heartbeat arriving after 3 seconds produces a very high phi value — the node is almost certainly dead. A heartbeat arriving after 1.2 seconds produces a low phi value — probably just network jitter.

Implementation

Trade-offs

Timeout Tuning

Advantages

• Simple to implement: Send/receive periodic messages
• Universal: Works for any type of node (database, service, container)
• Configurable: Tune timeout and interval for your reliability requirements

Disadvantages

• Network overhead: N nodes sending heartbeats every second generates N messages/second (or N^2 for peer-to-peer)
• False positives: GC pauses, CPU spikes, or network congestion can trigger false failure detection
• Detection delay: You only detect failure after the timeout expires, not when the failure actually occurs
• Not a guarantee: A node can send heartbeats but still be functionally broken (e.g., deadlocked, serving errors)

Common Misconceptions

• "Heartbeat = health check" — A heartbeat proves a process is running. A health check proves the application is functioning correctly. A service can send heartbeats while its database connection is broken. Use both.
• "Shorter timeouts are always better" — Shorter timeouts detect failures faster but trigger more false positives. In cloud environments with variable network latency, a 1-second timeout will generate constant false alarms.
• "Missing one heartbeat means the node is dead" — Production systems typically require 2-3 consecutive missed heartbeats before declaring a node dead. A single missed heartbeat could be a dropped packet.
• "Heartbeats prevent split-brain" — Heartbeats detect failures but do not prevent split-brain. A network partition can make both sides think the other is dead. You need fencing mechanisms to prevent split-brain.
• "All nodes need to heartbeat to all other nodes" — Full mesh heartbeating scales as O(N^2). Gossip-based systems only require each node to contact a few random peers per round, scaling much better.

How This Appears in Interviews

Heartbeat mechanisms appear in many system design scenarios:

• "How does Kubernetes know when to restart a pod?" — Liveness probes (heartbeat checks) detect unresponsive containers. Readiness probes determine if a pod should receive traffic.
• "Your distributed cache has a node that died. How long until traffic is redirected?" — Depends on the heartbeat timeout. Explain the speed vs accuracy trade-off and how you would tune it.
• "Design a service health monitoring system" — Central monitor with pull-based health checks, escalating alerts (warning after 1 miss, critical after 3 misses), and integration with load balancing to remove unhealthy instances.
• "How does Kafka detect a dead consumer?" — Consumer sends heartbeats to group coordinator. After session.timeout.ms with no heartbeat, the coordinator triggers a rebalance.

See our interview questions on distributed systems for more practice.

Related Concepts

• Leader Election — heartbeat timeouts trigger new leader elections
• Split-Brain Problem — heartbeat failures can cause false split-brain
• Load Balancing — health checks remove dead servers from rotation
• Partition Tolerance — heartbeats fail during network partitions
• System Design Interview Guide
• Algoroq Pricing — practice distributed systems concepts