Bahriya Tersane

Documentation / valkey

Topologies

The chart supports three deployment modes. Pick the smallest one that meets your durability, availability, and scale requirements — moving up the ladder costs operational complexity.

Decision matrix

Question If yes Mode
Is this a dev/local environment? → standalone
Is the data fully rebuildable from another source of truth (e.g. a cache backed by MySQL)? → standalone or sentinel/minimal
Do you need automatic failover with no data loss tolerance beyond ~1s of unsynced writes? → sentinel
Does the working set fit in the memory of a single node, with some headroom? → sentinel
Will the working set exceed a single node’s memory, or do you want per-shard CPU isolation? → cluster
Do clients use multi-key commands or transactions that touch arbitrary keys? → sentinel (cluster restricts cross-slot ops)

Standalone

┌────────────┐
│  valkey-0  │   (1 Pod, 1 PVC)
└────────────┘

One StatefulSet replica. The single Pod handles all reads and writes; data lives on its PVC. There is no failover — if the Pod is rescheduled, ValKey restarts on the same PVC and replays the RDB/AOF.

Trade-offs: simplest topology and lowest cost. No HA. PVC corruption is unrecoverable without a backup.

Sentinel (HA)

┌────────────┐    ┌────────────┐    ┌────────────┐
│  valkey-0  │ ←  │  valkey-1  │ ←  │  valkey-2  │
│  primary   │    │  replica   │    │  replica   │
│ + sentinel │    │ + sentinel │    │ + sentinel │
└────────────┘    └────────────┘    └────────────┘
       ▲                 ▲                 ▲
       └─── Sentinel quorum (3, agreement = 2) ──┘

A single logical primary with N replicas. Every Pod runs a Sentinel sidecar that gossips state and votes on failover. When the primary fails, the Sentinels promote a replica to primary within down-after-milliseconds (5s default).

Clients: use a Sentinel-aware driver. The driver queries the Sentinel Service for the current primary and reconnects on failover. Examples — valkey-py (Sentinel(...)), ioredis (new Redis({ sentinels, name })), lettuce (RedisURI.Builder.sentinel(...)).

The chart’s primary <release> Service selects valkey.bahriya.io/role=primary, a label each Pod maintains on itself (role self-labelling — see the Failover section of the README), so writes routed through it reach the live primary even after a failover. Allow a window of up to ~10 seconds after a failover before the Service follows the new topology; a Sentinel-aware driver learns of the promotion faster and remains the recommended client for latency-sensitive write paths.

Preset choices

Preset Pods Quorum When
minimal 2 2 Dev HA. Quorum equals Pod count → no tolerance for a Sentinel offline.
standard 3 2 Default. Survives one Pod loss; quorum holds with 2/3.
hardened 5 3 Survives two Pod losses; useful when nodes are flaky.
custom sentinel.replicas sentinel.quorum Anything else.

Always pick an odd Pod count for sentinel to avoid split-brain on quorum. The presets honour this.

Failover timeline

  1. Primary becomes unreachable.
  2. After downAfterMilliseconds (5s), a Sentinel marks it +sdown.
  3. Sentinels gossip and agree it’s +odown (objectively down).
  4. A replica is promoted; its config is rewritten with slaveof no one.
  5. Surviving replicas are reconfigured to slaveof <new-primary>.
  6. Clients learn the new primary via the Sentinel SDK and reconnect.

Total time: typically 5–10 seconds.

Cluster (sharded)

┌──────────────────────┐  ┌──────────────────────┐  ┌──────────────────────┐
│       shard-0        │  │       shard-1        │  │       shard-2        │
│ ┌──────┐  ┌────────┐ │  │ ┌──────┐  ┌────────┐ │  │ ┌──────┐  ┌────────┐ │
│ │prim-0│  │ repl-1 │ │  │ │prim-0│  │ repl-1 │ │  │ │prim-0│  │ repl-1 │ │
│ └──────┘  └────────┘ │  │ └──────┘  └────────┘ │  │ └──────┘  └────────┘ │
└──────────────────────┘  └──────────────────────┘  └──────────────────────┘
        slots 0-5460             slots 5461-10922          slots 10923-16383
                              gossip + automatic failover

M shards. Each shard is a StatefulSet with 1 + replicasPerShard Pods. Keys are sharded across 16384 hash slots, distributed evenly across shards on bootstrap. Each shard handles failover via the cluster bus — no Sentinel needed.

The chart’s bootstrap Job runs valkey-cli --cluster create on install/upgrade. It’s idempotent: it checks CLUSTER INFO first and skips if the cluster is already formed.

Clients: use a cluster-aware driver. Examples — valkey-py (ValkeyCluster(startup_nodes=[...])), ioredis (new Redis.Cluster([...])), lettuce (RedisClusterClient.create(...)).

Preset choices

Preset Pods Layout When
minimal 3 3 × 0 Cache-only. Losing any shard takes ~1/3 of slots offline.
standard 6 3 × 1 Default. Survives any single Pod loss with automatic failover.
hardened 9 3 × 2 More read replicas, two-Pod-failure tolerance per shard.
custom shards × (1+replicasPerShard) — Set both values.

Multi-key restrictions

Cluster mode restricts multi-key operations to keys that hash to the same slot. Use hash tags ({tag}key) to colocate related keys:

SET {user:42}:name "alice"
SET {user:42}:email "alice@example.com"
MGET {user:42}:name {user:42}:email   # works — same slot
MGET user:42:name user:43:email       # CROSSSLOT error

MULTI/EXEC transactions, EVAL scripts, and pipelines also obey this rule. Plan key schemas with hash tags from day one if you intend to run multi-key ops.

Scaling the cluster up (automated)

Increasing cluster.shards or cluster.replicasPerShard is automated. On helm upgrade:

  1. The StatefulSets for new shards are created and Pods come up empty.
  2. The post-upgrade bootstrap Job runs, sees cluster_state:ok but new Pods missing from CLUSTER NODES, and:
    • valkey-cli --cluster add-node <new-primary> for each new shard primary.
    • valkey-cli --cluster add-node <new-replica> --cluster-slave --cluster-master-id <shard-primary-id> for each new replica.
    • valkey-cli --cluster rebalance --cluster-use-empty-masters --cluster-yes to migrate slots onto the new shards.
# Grow from 3 shards to 6
helm upgrade vk charts/valkey --set cluster.shards=6 --reuse-values
kubectl logs -n <ns> job/vk-valkey-cluster-bootstrap

You can monitor the rebalance progress with valkey-cli CLUSTER INFO — cluster_slots_assigned stays at 16384 throughout, while CLUSTER SHARDS shows slots moving.

Scaling the cluster down (manual)

Shrinking is intentionally not automated because slot migration must complete before StatefulSet deletion, and an interrupted shrink leaves the cluster unable to accept writes on the orphaned slots (with cluster.requireFullCoverage: true, the default, that halts the whole cluster).

The playbook below shrinks from 4 shards back to 3. Adapt the indices for other shrinks. Always remove the highest-ordinal shard first — the StatefulSet’s scale-down order is highest-first, and the cluster’s slot migration must drain that shard before the Pods disappear.

0. Prereqs

NS=<namespace>
RELEASE=vk
FULL=${RELEASE}-valkey                     # release fullname
SHARD_TO_REMOVE=3                          # the shard being decommissioned
EXISTING_NODE=${FULL}-shard-0-0.${FULL}-headless.${NS}.svc.cluster.local
PORT=6379
PASS=$(kubectl get secret ${FULL} -n ${NS} -o jsonpath='{.data.password}' | base64 -d)   # if auth.enabled
AUTH="-a $PASS --no-auth-warning"                                                        # else: AUTH=""

# Always run valkey-cli from inside one of the cluster Pods so DNS works.
VC() { kubectl exec -n $NS ${FULL}-shard-0-0 -c valkey -- valkey-cli $AUTH "$@"; }

1. Identify the doomed shard’s node IDs

VC -c -h ${EXISTING_NODE} -p $PORT CLUSTER NODES > /tmp/nodes.txt
grep "${FULL}-shard-${SHARD_TO_REMOVE}-" /tmp/nodes.txt

Output looks like:

<id-primary>  vk-valkey-shard-3-0...:6379@16379  master   - 0 ... connected 12289-16383
<id-replica>  vk-valkey-shard-3-1...:6379@16379  slave    <id-primary> 0 ... connected

Note both IDs. The slot range on the master line is the range you must reshard away.

2. Reshard the doomed shard’s slots onto surviving shards

valkey-cli --cluster reshard only accepts one destination per invocation. To distribute evenly across the remaining shards, run it once per destination, splitting the source slot count.

For a 4 → 3 shrink, you have ~4096 slots on shard 3 to distribute across shards 0/1/2:

SLOTS_TO_MOVE=4096           # adjust from your CLUSTER NODES output
PER_DEST=$(( SLOTS_TO_MOVE / 3 ))
REMAINDER=$(( SLOTS_TO_MOVE - PER_DEST * 3 ))

SOURCE=<id-primary-of-shard-3>
DEST_0=<id-primary-of-shard-0>
DEST_1=<id-primary-of-shard-1>
DEST_2=<id-primary-of-shard-2>

VC --cluster reshard ${EXISTING_NODE}:$PORT \
   --cluster-from $SOURCE --cluster-to $DEST_0 \
   --cluster-slots $(( PER_DEST + REMAINDER )) --cluster-yes

VC --cluster reshard ${EXISTING_NODE}:$PORT \
   --cluster-from $SOURCE --cluster-to $DEST_1 \
   --cluster-slots $PER_DEST --cluster-yes

VC --cluster reshard ${EXISTING_NODE}:$PORT \
   --cluster-from $SOURCE --cluster-to $DEST_2 \
   --cluster-slots $PER_DEST --cluster-yes

3. Verify the source is now empty

VC -c -h ${FULL}-shard-${SHARD_TO_REMOVE}-0.${FULL}-headless.${NS}.svc.cluster.local -p $PORT DBSIZE
# Should print (integer) 0
VC -c -h ${EXISTING_NODE} -p $PORT CLUSTER NODES | grep "${FULL}-shard-${SHARD_TO_REMOVE}-0"
# The master line should show no slot range.

If DBSIZE > 0, stop — there are still keys on the doomed shard. Re-run reshard for the remaining slots before continuing.

4. Forget the replicas, then the primary

Replicas first; otherwise the primary’s removal leaves them orphaned.

for r in 1 2; do                                    # iterate over replicasPerShard
  REPLICA_ID=$(grep "${FULL}-shard-${SHARD_TO_REMOVE}-${r}" /tmp/nodes.txt | awk '{print $1}')
  [ -n "$REPLICA_ID" ] && VC --cluster del-node ${EXISTING_NODE}:$PORT $REPLICA_ID
done

PRIMARY_ID=$(grep "${FULL}-shard-${SHARD_TO_REMOVE}-0" /tmp/nodes.txt | awk '{print $1}')
VC --cluster del-node ${EXISTING_NODE}:$PORT $PRIMARY_ID

After del-node, the cluster forgets these nodes. The Pods are still running but unused.

5. Confirm cluster is healthy at the new size

VC -c -h ${EXISTING_NODE} -p $PORT CLUSTER INFO
# Look for:
#   cluster_state:ok
#   cluster_slots_assigned:16384
#   cluster_known_nodes:9   (3 shards × (1 + 2 replicas) for a 3x3 layout, etc.)

If cluster_state is anything but ok, do not proceed — investigate before shrinking the StatefulSet.

6. Shrink the chart

Only now is it safe to delete the StatefulSet for the removed shard:

helm upgrade $RELEASE charts/valkey --set cluster.shards=3 --reuse-values

Helm deletes the ${FULL}-shard-3 StatefulSet. Its Pods terminate gracefully. The PVCs stay (chart default persistentVolumeClaimRetentionPolicy.whenDeleted: Retain).

7. Reclaim the PVCs (optional)

kubectl get pvc -n $NS | grep ${FULL}-shard-${SHARD_TO_REMOVE}
# data-vk-valkey-shard-3-0   Bound   ...
# data-vk-valkey-shard-3-1   Bound   ...
# Delete only when you are sure you won't need the data:
kubectl delete pvc -n $NS data-${FULL}-shard-${SHARD_TO_REMOVE}-0 \
                          data-${FULL}-shard-${SHARD_TO_REMOVE}-1

Recovery from a failed shrink

If the shrink halted before step 6 and the doomed shard still has slots assigned, do not helm upgrade --set cluster.shards=N-1 — that would delete a shard that owns live slots and make those keys unreachable. Instead:

  1. Re-run step 2 (reshard) to drain remaining slots.
  2. Verify step 3 (DBSIZE 0, no slot range).
  3. Resume step 4 onwards.

If the StatefulSet was already deleted before draining (the catastrophic case): restore from the backup CronJob’s most recent RDB for the lost shard. See docs/backups-restore.md — the cluster restore path can take just one shard’s RDB.

Why shrink isn’t automated

Shrink could be wrapped in a helm hook, but:

  • A reshard mid-flight that fails leaves the cluster in a half-migrated state where some slots are owned by a node that’s about to be deleted. Recovering requires the operator to look at CLUSTER NODES and decide which slots to drain manually.
  • valkey-cli --cluster reshard doesn’t support multi-destination in one call, and even-distribution math depends on the exact slot count of the source.
  • Most shrinks happen as part of a downsizing exercise where the operator wants to watch each step (DBSIZE, slot counts, CLUSTER INFO) before continuing.

The chart could ship a pre-upgrade Job that detects a shrink and refuses with an error message — that’s a future improvement. For now the guard is operator discipline plus this playbook.

Picking a preset, summarised

  • Start with standalone if you can rebuild the data.
  • Go sentinel/standard for HA caches and session stores.
  • Go cluster/standard when you outgrow one node, or want per-shard isolation.
  • Reach for hardened only when you’ve measured node-loss frequency and need the extra margin.