Design a Distributed Cache: Architecture and Data Flow

Once you’ve decided a single cache node can’t handle your traffic or dataset size, the real engineering work begins: how do you route a key to the right node, how do you rebalance when nodes join or leave, and how do you keep serving reads while a node recovers from failure? This piece goes past the “what is a cache” framing and walks through the actual architecture and data flow of a production-grade distributed cache.

The Core Problem: Where Does This Key Live?

With N cache nodes, the naive approach is hash(key) % N. It’s fast and simple, but it has a fatal flaw for distributed systems: if you add or remove a single node, N changes, and nearly every key remaps to a different node. That means a near-total cache flush every time your cluster scales, which defeats the purpose of caching.

Consistent Hashing

Consistent hashing solves this by mapping both nodes and keys onto a fixed hash ring (typically 0 to 2^32-1). A key belongs to the first node found by walking clockwise from the key’s position on the ring. When a node is added or removed, only the keys between that node and its neighbor on the ring need to move — typically around 1/N of total keys, not nearly all of them.

The trade-off: naive consistent hashing can distribute keys unevenly, especially with a small number of nodes, because node positions on the ring are essentially random.

Virtual Nodes

The fix for uneven distribution is virtual nodes: instead of placing each physical node once on the ring, you place it at many points (often 100-300 virtual positions per physical node). This smooths out the distribution because the law of large numbers evens out the randomness of hash placement. It also makes rebalancing more even when a node is added or removed — its share of keys is spread across many small ranges instead of one large one.

Redis Cluster takes a related but distinct approach: rather than a continuous ring, it divides the keyspace into 16,384 fixed hash slots, and each physical node owns a set of slots. This is simpler to reason about operationally (you can see exactly which slots a node owns) while achieving similar rebalancing properties.

Replication Topology

Sharding solves capacity, but not availability — if a shard’s only node dies, that slice of data disappears. Replication addresses this:

• Primary-replica per shard. Each shard has one primary (handles writes) and 1-2 replicas (handle reads, and can be promoted on primary failure). This is the standard Redis Cluster and most managed-cache setups.
• Read replicas for hot shards. Even without failover concerns, adding replicas to a shard that’s disproportionately hot lets you spread read load across more nodes.
• Cross-region replication. For globally distributed applications, asynchronous replication to a secondary region trades a small replication lag for disaster recovery and lower read latency for distant users.

Failover typically works through a consensus mechanism (Redis Sentinel, or Cluster’s built-in gossip protocol) that detects a primary is unreachable and promotes a replica. The key interview point: failover is not instantaneous, and there’s a window (often a few seconds) where writes to that shard fail or queue.

Client-Side Sharding vs Proxy Layer

There are two architectural patterns for how application code finds the right cache node.

Client-side sharding embeds the hashing/routing logic in a client library. The application computes which node owns a key and connects directly. This is lower latency (no extra network hop) but couples every application service to the cluster topology — every client needs to be updated or reloaded when nodes change, and every language/service needs a compatible client library.

Proxy layer (e.g., mcrouter for Memcached, Twemproxy, or Redis Cluster’s own client-side redirect protocol) puts a stateless routing tier between application and cache nodes. Applications just talk to the proxy as if it were a single cache; the proxy handles routing, connection pooling, and failover. This adds one network hop of latency but massively simplifies application code and lets you evolve the cluster topology without touching every service.

Comparison Table

Full Request Data Flow

Walking through a single GET request end to end makes the architecture concrete:

1. Application issues a GET for user:12345:profile.
2. Routing layer (client library or proxy) hashes the key and determines which shard owns it — say, via consistent hashing, shard 7.
3. Connection pool picks an established TCP connection to shard 7’s current primary (or a replica, if reads are configured to prefer replicas).
4. Cache node looks up the key in its in-memory hash table. On a hit, it returns the value immediately, typically in under a millisecond.
5. On a miss, the routing layer (or application logic, depending on where cache-aside is implemented) falls through to the database, fetches the row, and issues a SET back to shard 7 to populate the cache for next time.
6. Write path: for a SET, the same routing determines the shard, the primary applies the write, and — if using synchronous replication — waits for acknowledgment from at least one replica before confirming success to the client. Asynchronous replication returns success immediately and streams the update to replicas afterward.

Handling Node Failure Mid-Flow

If shard 7’s primary becomes unreachable during step 3-4:

• With a proxy layer, the proxy detects the failed connection (via health checks or a failed request) and either retries against a promoted replica or returns an error the application can handle with a database fallback.
• With client-side sharding, the client library needs its own retry/failover logic, which is why most teams prefer a proxy at scale — reimplementing this correctly in every service is a common source of production bugs.

Rebalancing in Practice

When you add a new node to the cluster:

1. The new node is assigned a set of virtual node positions (or hash slots).
2. The cluster coordinator (or an operator-triggered migration) moves the affected key ranges from existing nodes to the new one, typically with a “migrating” state so reads/writes during the transition are still routed correctly.
3. Once migration completes, the new node serves its full assigned range and the source nodes drop the migrated keys.

This process should be gradual and throttled — moving too much data too fast can spike network and CPU usage on nodes that are still serving live production traffic.

Related Reading

For the interview-answer framing of these same concepts — including how to pick Redis vs Memcached and structure your response under time pressure — see Design a Distributed Cache: System Design Interview Guide.

Key Takeaways

• Consistent hashing with virtual nodes is the standard solution to the rebalancing problem; Redis Cluster’s fixed hash slots achieve the same goal with a different mechanism.
• Replication (primary-replica per shard) is what turns a fast cache into a highly available one — plan for a brief failover window.
• Proxy-based sharding trades one network hop for dramatically simpler application code and easier topology changes; client-side sharding trades that simplicity for lower latency.
• Always be ready to trace a request end to end: hash the key, route to the shard, hit or miss, populate on miss, handle node failure mid-flight.

Go Deeper

For more architecture walkthroughs like this one across caching, queuing, and storage systems, The 0-to-1 SWE Interview Playbook (Amazon: https://www.amazon.com/dp/B0H256Z1MF?tag=sirjohnnymai-20) covers the data-flow-level detail that separates a passing system design answer from a standout one.