P-4 — Memory Profiling and Optimization

The Memory Audit Starting Point: INFO memory

Every Redis memory investigation starts here:

INFO memory

used_memory:              1528000000   → allocations from Redis's perspective (bytes)
used_memory_human:        1.42G
used_memory_rss:          2097152000   → RSS reported by OS (includes fragmentation)
used_memory_rss_human:    1.95G
used_memory_peak:         1600000000   → peak allocation since server start
used_memory_peak_human:   1.49G
used_memory_peak_perc:    95.50%       → current / peak
used_memory_overhead:     852000       → internal overhead (dicts, expiry table, etc.)
used_memory_startup:      864000       → baseline memory at startup
used_memory_dataset:      1527136000   → data memory (used_memory - overhead)
used_memory_dataset_perc: 99.94%       → dataset / (peak - startup)
allocator_allocated:      1528100000   → bytes allocated from jemalloc
allocator_active:         1953300000   → bytes in active jemalloc pages
allocator_resident:       2097100000   → bytes in resident jemalloc pages
total_system_memory:      8589934592   → total RAM on the machine
maxmemory:                2147483648   → configured maxmemory (2GB)
maxmemory_human:          2.00G
maxmemory_policy:         allkeys-lru
mem_fragmentation_ratio:  1.37         → RSS / used_memory
mem_fragmentation_bytes:  569152000    → bytes "lost" to fragmentation
mem_not_counted_for_evict: 0
mem_replication_backlog:  1048576      → replication backlog size
mem_clients_slaves:       20512
mem_clients_normal:       84000
mem_cluster_links:        0
mem_aof_buffer:           8
active_defrag_running:    0
lazyfree_pending_objects: 0
lazyfreed_objects:        42831

Interpreting the Key Fields

mem_fragmentation_ratio = used_memory_rss / used_memory

• < 1.0 → Redis is using swap (critical, investigate immediately)
• 1.0–1.2 → healthy
• 1.2–1.5 → moderate fragmentation (normal for dynamic workloads)

• 1.5 → high fragmentation — consider activedefrag or restart

used_memory_overhead = memory used by Redis's internal data structures (the global keyspace dict, expiry table, per-client buffers). If this is a large fraction of used_memory, you have very small values (overhead dominates) — consider consolidating keys into Hashes.

mem_clients_normal = memory used by client output buffers. If this is large (> 10MB), you may have slow clients receiving data faster than they can consume it.

MEMORY USAGE: Per-Key Cost

MEMORY USAGE key [SAMPLES count]

Returns the exact number of bytes allocated for a key and its value, including all internal structures (robj, SDS, listpack nodes, etc.).

127.0.0.1:6379> MEMORY USAGE user:1001
(integer) 128

127.0.0.1:6379> MEMORY USAGE large:hash
(integer) 4194304   ← this hash is using 4MB

For collections, SAMPLES controls how many elements are sampled to estimate total cost (default 5). Use SAMPLES 0 for exact measurement on small collections.

Using MEMORY USAGE to find expensive keys:

bash
#!/bin/bash
# Find the 20 most memory-hungry keys
redis-cli --scan | while read key; do
  size=$(redis-cli MEMORY USAGE "$key" 2>/dev/null)
  echo "$size $key"
done | sort -n -r | head -20

On a production instance with millions of keys, sample a representative subset:

bash
redis-cli --scan --count 1000 | shuf | head -1000 | while read key; do
  size=$(redis-cli MEMORY USAGE "$key" 2>/dev/null)
  echo "$size $key"
done | sort -n -r | head -20

MEMORY DOCTOR

MEMORY DOCTOR

Returns a human-readable diagnosis. Possible outputs:

"Sam, I detected a few problems: 
 * High total allocator fragmentation: The RSS reported by the allocator is 
   suspicious. This could be caused by ...
 * High rss overhead: ..."

Or for a healthy instance:

"Sam, I have detected no problems in the server memory subsystem."

Not a substitute for INFO memory, but a quick sanity check.

MEMORY MALLOC-STATS

MEMORY MALLOC-STATS

Dumps the full jemalloc allocator statistics — bin sizes, fragmentation per bin, active vs retained pages. Useful when you suspect allocator-level fragmentation rather than Redis-level issues.

Finding the Memory Culprits

Pattern 1: Large Hashes in hashtable Encoding

A Hash with > 128 fields (or any field > 64 bytes) switches from listpack to hashtable encoding. The memory cost jumps roughly 5x per element. Find them:

bash
redis-cli --scan --pattern "user:*" | while read key; do
  type=$(redis-cli TYPE "$key")
  if [ "$type" = "hash" ]; then
    len=$(redis-cli HLEN "$key")
    encoding=$(redis-cli OBJECT ENCODING "$key")
    if [ "$encoding" = "hashtable" ]; then
      size=$(redis-cli MEMORY USAGE "$key")
      echo "$size $key $len fields"
    fi
  fi
done | sort -n -r | head -20

If you find 1,000 user Hashes in hashtable encoding that should be in listpack encoding (they have < 128 fields), your encoding threshold is wrong. Check:

CONFIG GET hash-max-listpack-entries
CONFIG GET hash-max-listpack-value

If the threshold is already 128 but hashes have 50 fields in hashtable encoding, some field values exceed 64 bytes. Identify them with HGETALL on a sample key.

Pattern 2: Keys Without TTL (Orphaned Data)

bash
# Count keys with and without TTL
redis-cli --scan | while read key; do
  ttl=$(redis-cli TTL "$key")
  if [ "$ttl" -eq -1 ]; then
    echo "no-ttl $key"
  fi
done | wc -l

Or in a Redis script:

INFO keyspace
→ db0:keys=500000,expires=50000,avg_ttl=3600000

If expires is much less than keys, most of your keys have no TTL. For a cache, this means eviction will eventually clear them — but you are paying for that memory until then. For an application database, this is expected.

Pattern 3: String Keys Storing JSON When Hashes Would Be Better

OBJECT ENCODING tells you if a String key is in raw encoding (large string). If it is storing a JSON blob, consider whether you update individual fields — if so, a Hash is more efficient and enables atomic partial updates.

bash
redis-cli --scan --pattern "user:*" | while read key; do
  encoding=$(redis-cli OBJECT ENCODING "$key")
  if [ "$encoding" = "raw" ]; then
    size=$(redis-cli MEMORY USAGE "$key")
    echo "$size $key raw-string"
  fi
done | sort -n -r | head -20

Pattern 4: Sorted Set Keys in skiplist Encoding

Sorted Sets with > 128 members use skiplist encoding. A skiplist + hashtable for 1,000 members uses ~250KB; listpack for the same 1,000 members uses ~55KB. If you have many small-to-medium sorted sets exceeding the listpack threshold by a few members:

CONFIG SET zset-max-listpack-entries 256   → raise threshold if members are ≤ 64 bytes

Active Defragmentation

When mem_fragmentation_ratio > 1.5, enable active defragmentation:

CONFIG SET activedefrag yes
CONFIG SET active-defrag-ignore-bytes 100mb    → don't defrag if fragmentation bytes < 100MB
CONFIG SET active-defrag-threshold-lower 10    → start at 10% fragmentation
CONFIG SET active-defrag-threshold-upper 100   → max effort at 100%
CONFIG SET active-defrag-cycle-min 1           → min CPU % for defrag
CONFIG SET active-defrag-cycle-max 25          → max CPU % for defrag

Active defragmentation runs a background scan, finding allocations that can be moved to compacted jemalloc pages. It uses 1–25% of a CPU core and can recover significant memory without restarting Redis.

When it is not enough: If mem_fragmentation_ratio > 2.0 and the instance has been running for months with heavy churn, active defragmentation may be slow to converge. A Redis restart (graceful shutdown → dump.rdb / AOF flush → restart → reload) resets memory layout and eliminates fragmentation instantly. Plan this during a low-traffic window.

Encoding Threshold Tuning

The single most impactful memory optimization is ensuring data structures use compact encodings.

Hash Thresholds

hash-max-listpack-entries 128   → listpack if ≤ 128 fields
hash-max-listpack-value 64      → listpack if all values ≤ 64 bytes

If your user Hashes have 50 fields with values averaging 30 bytes: raise to entries 256, value 64 to keep them in listpack. Memory reduction: ~5x per hash.

Sorted Set Thresholds

zset-max-listpack-entries 128
zset-max-listpack-value 64

If your leaderboard sorted sets have up to 200 members under 40 bytes each: raise to entries 256.

Set Thresholds (integer sets)

set-max-intset-entries 512    → intset (sorted integer array) if all members are integers

If you are storing user IDs (integers) in Sets: intset is the most compact encoding. Ensure all members are integers to keep the intset encoding.

Testing Threshold Changes

After changing thresholds:

1. New keys will use the new thresholds
2. Existing keys will NOT automatically convert (they were already promoted to the larger encoding)
3. To convert existing keys: DUMP + RESTORE or use redis-cli --pipe to reload the data

The safest approach for large deployments: change thresholds and let natural key churn (TTL expiry + re-creation) gradually adopt the new encoding.

The Hash-for-Small-Objects Pattern

For reference, here is the memory comparison that drives this pattern:

# 100 users stored as individual String keys with JSON
user:1     → raw string (80 bytes JSON) → ~160 bytes in Redis
user:2     → raw string (80 bytes JSON) → ~160 bytes in Redis
...
user:100   → raw string (80 bytes JSON) → ~160 bytes in Redis
Total: ~16,000 bytes (16KB)

# Same 100 users stored as fields in a single Hash
users      → listpack Hash → ~20 bytes overhead + 11 bytes/field-pair
           → 100 * (name + value + overhead) → ~2,200 bytes
Total: ~2.2KB — 7x more efficient

The per-key overhead (robj, SDS for key, hash table entry) adds ~80 bytes per key. When values are small, this overhead dominates. Grouping small objects under a single Hash key eliminates most of it.

Implementation: Instead of SET user:1001:name "Jatin", use HSET users 1001:name "Jatin" (or one Hash per user: HSET user:1001 name "Jatin" email "...".

This pattern has limits: a Hash cannot have per-field TTLs, and you cannot atomically query across multiple user Hashes. For most use cases, the memory savings outweigh these constraints.

Memory Optimization Checklist

• Run INFO memory — check mem_fragmentation_ratio, used_memory_overhead
• Find top 20 keys by memory: MEMORY USAGE + SCAN
• Check encoding of large collections: OBJECT ENCODING
• Identify Hashes in hashtable encoding that should be listpack: HLEN + OBJECT ENCODING
• Check for keys without TTL in a cache context: INFO keyspace (expires vs keys count)
• Review encoding thresholds: CONFIG GET hash-max-listpack-*, zset-max-listpack-*
• Consider active defragmentation if mem_fragmentation_ratio > 1.5
• Evaluate Hash-for-small-objects pattern for high-cardinality small-value datasets
• Set maxmemory and maxmemory-policy if not set (do not let Redis use unbounded RAM)

Summary

• INFO memory is the starting point: check mem_fragmentation_ratio, used_memory_dataset, mem_clients_normal
• MEMORY USAGE key gives exact per-key RAM cost including all internal structures
• MEMORY DOCTOR for a quick health check; MEMORY MALLOC-STATS for allocator-level detail
• Encoding inspection: OBJECT ENCODING reveals whether a key is in compact (listpack, intset) or large (hashtable, skiplist) encoding
• Tuning encoding thresholds (hash-max-listpack-entries, zset-max-listpack-entries) is the highest-leverage memory optimization
• Active defragmentation (activedefrag yes) recovers memory from fragmentation without restarting
• The Hash-for-small-objects pattern reduces per-key overhead for high-cardinality small datasets by 5–10x

Next: P-5 — Atomic Counters, Rate Limiters, and Sliding Windows — building lock-free counters, fixed and sliding window rate limiters, and the token bucket algorithm using Redis's atomic operations.