A-10 — Resharding, Node Addition, and Live Slot Migration

Why Resharding?

After initial cluster creation, hash slots are distributed evenly. But over time, the distribution may become uneven:

• Capacity scaling: adding a new node requires migrating slots to it from existing nodes
• Data imbalance: some slots hold more data than others due to access patterns
• Node removal: removing a node requires migrating its slots elsewhere first
• Hotspot correction: if one node handles most traffic, rebalancing moves some slots to less-loaded nodes

Resharding is live — traffic continues flowing during migration. Keys in migrating slots are briefly accessible from both the source and destination during the transition.

Slot Migration: The States

A slot can be in one of three states during migration:

Normal:         Slot 7638 is on Node A — all reads/writes go to Node A

MIGRATING:      Slot 7638 is migrating FROM Node A to Node B
                - Keys still in Node A answer requests
                - Keys already moved to Node B → Node A issues ASK redirection

IMPORTING:      Slot 7638 is being imported TO Node B
                - Node B accepts keys for this slot only if the ASKING command was sent
                - Normal requests for slot 7638 still go to Node A

The MIGRATE Command

The low-level operation that moves a single key from one node to another:

MIGRATE host port key destination-db timeout [COPY] [REPLACE] [KEYS key [key ...]]

# Move key "user:1001" from Node A to Node B
MIGRATE 10.0.1.52 6379 user:1001 0 5000

MIGRATE:

1. Dumps the key's value (serializes to RDB format)
2. Sends it to the destination node with RESTORE semantics
3. If the destination acknowledges: deletes the key from the source
4. Atomic from the perspective of each node (the key exists on exactly one node at any moment)

Bulk migration with KEYS option:

MIGRATE 10.0.1.52 6379 "" 0 5000 KEYS user:1001 user:1002 user:1003

Slot Migration Protocol

The full protocol for migrating slot 7638 from Node A to Node B:

bash
# Step 1: Prepare the destination (Node B) to accept the slot
CLUSTER SETSLOT 7638 IMPORTING <node-A-id>

# Step 2: Prepare the source (Node A) to migrate the slot
CLUSTER SETSLOT 7638 MIGRATING <node-B-id>

# Step 3: Get all keys in slot 7638 on Node A, in batches
CLUSTER GETKEYSINSLOT 7638 100   → returns up to 100 keys in this slot

# Step 4: For each batch of keys, MIGRATE to Node B
MIGRATE 10.0.1.52 6379 "" 0 5000 KEYS user:1001 session:abc123

# Repeat steps 3-4 until CLUSTER GETKEYSINSLOT returns empty

# Step 5: Finalize: tell all nodes the slot now belongs to Node B
CLUSTER SETSLOT 7638 NODE <node-B-id>
# This must be sent to ALL nodes in the cluster, not just A and B

After CLUSTER SETSLOT 7638 NODE <node-B-id> is propagated to all nodes, the migration is complete. Normal MOVED redirections take over for slot 7638 pointing to Node B.

The ASKING Redirection During Migration

While slot 7638 is in MIGRATING state on Node A:

• Keys that haven't been migrated yet are still on Node A → served normally
• Keys that have been migrated are on Node B → Node A returns -ASK 7638 10.0.1.52:6379

The client receiving an ASK redirection must:

1. Send ASKING to Node B (one-time flag — tells Node B "I know you're importing this slot, accept this command")
2. Resend the original command to Node B
3. Not update its slot map — slot 7638 is still officially on Node A until migration completes

Client → Node A: GET user:1001  (user:1001 already migrated)
Node A → -ASK 7638 10.0.1.52:6379

Client → Node B: ASKING
Client → Node B: GET user:1001  → "Jatin"
# Client's slot map: 7638 → Node A  (unchanged)

# Next request for a different key in slot 7638:
Client → Node A: GET user:1002  (still on Node A)
→ served normally

This is what makes resharding zero-downtime: every key is always accessible, just the route changes depending on whether the specific key has been migrated yet.

Using redis-cli for Resharding

The redis-cli tool automates the multi-step migration protocol:

Add a Node and Rebalance

bash
# Add a new empty primary node to the cluster
redis-cli --cluster add-node 10.0.1.60:6379 10.0.1.50:6379
# new-node-address:port  existing-cluster-node:port

# The new node has 0 slots — rebalance to distribute evenly
redis-cli --cluster rebalance 10.0.1.50:6379 --cluster-use-empty-masters

rebalance automatically calculates how many slots to move from each existing node to the new one and runs the migration.

Manual Resharding

bash
# Move 1000 slots to the new node
redis-cli --cluster reshard 10.0.1.50:6379 \
  --cluster-from <source-node-id> \
  --cluster-to <destination-node-id> \
  --cluster-slots 1000 \
  --cluster-yes   → skip confirmation prompt

reshard handles the full SETSLOT MIGRATING/IMPORTING protocol, GETKEYSINSLOT, MIGRATE, and SETSLOT NODE sequence automatically.

Adding a Node as a Replica

bash
redis-cli --cluster add-node 10.0.1.61:6379 10.0.1.50:6379 \
  --cluster-slave \
  --cluster-master-id <primary-node-id>

Remove a Node

You must first migrate all its slots away, then remove it:

bash
# Move all slots from the node to be removed to other nodes
redis-cli --cluster reshard 10.0.1.50:6379 \
  --cluster-from <node-to-remove-id> \
  --cluster-to <receiving-node-id> \
  --cluster-slots <all-slots-count>

# Then remove the empty node
redis-cli --cluster del-node 10.0.1.50:6379 <node-id>

Resharding Performance and Impact

Migration speed is limited by:

• Key size: larger values take longer to serialize and transmit
• Key count: more keys in the migrating slot = longer migration
• Network bandwidth between source and destination
• MIGRATE timeout: keys that cannot be migrated within the timeout are left in place and retried

During migration:

• Read and write commands continue to work (via MOVED/ASK redirections)
• Latency may increase slightly due to extra redirections
• The source node's CPU and I/O load increases during active migration

To minimise impact:

• Migrate during off-peak hours for large datasets
• Use --cluster-pipeline option to batch MIGRATE commands
• Monitor source node CPU and network during migration

bash
# Use pipeline option for faster migration (batch 10 keys per MIGRATE call)
redis-cli --cluster reshard 10.0.1.50:6379 \
  --cluster-from <source-id> \
  --cluster-to <dest-id> \
  --cluster-slots 1000 \
  --cluster-pipeline 10

Verifying Migration

bash
# Check cluster health and slot distribution
redis-cli --cluster check 10.0.1.50:6379

# Expected output:
# [OK] All nodes agree about slots configuration.
# [OK] All 16384 slots covered.

# Detailed slot distribution per node
redis-cli --cluster info 10.0.1.50:6379

127.0.0.1:6379> CLUSTER INFO
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0

Any value > 0 for cluster_slots_pfail or cluster_slots_fail indicates nodes in a degraded state.

When Resharding Time Is Proportional to Key Count

The most common surprise with resharding: it is slow for large datasets. Each key must be serialized, sent over the network, restored on the destination, and deleted from the source. A slot with 1 million keys takes much longer than a slot with 1,000 keys.

For very large datasets:

• Estimate migration time before starting: measure MIGRATE speed (keys/second) on a test slot, extrapolate
• Plan for migrations to take hours on multi-billion-key clusters
• Consider creating the new node as a replica first, letting it sync fully, then resharding — the replica sync is faster than MIGRATE for bulk data transfer

Summary

• Resharding moves hash slots between nodes while traffic continues — zero downtime via MOVING/IMPORTING slot states and ASKING redirections
• MIGRATING on source + IMPORTING on destination: keys not yet moved served from source; keys moved served from destination via ASK
• ASK redirection: temporary — send ASKING then the command; do not update slot map until CLUSTER SETSLOT NODE finalises
• MIGRATE moves individual keys atomically — key exists on exactly one node at any moment during migration
• redis-cli --cluster reshard automates the full migration protocol; --cluster rebalance distributes slots evenly after adding nodes
• Migration speed = keys/second × key-count; large datasets require hours and should be planned for off-peak
• After migration: redis-cli --cluster check to verify all 16,384 slots are covered and healthy

Next: A-11 — Gossip Protocol and Network Partition Handling — how cluster nodes discover each other, propagate topology changes, and handle split scenarios where portions of the cluster become unreachable.