Bahriya Tersane

Documentation / valkey

Backups and restore

The chart ships a scheduled BGSAVE + S3 upload CronJob and an init-container restore path. This guide covers operational details: S3 layout, how restores actually run, sentinel/cluster nuances, and disaster-recovery patterns.

What the CronJob does

  1. Picks a target Pod based on mode:
    • standalone: ordinal 0 (the only Pod).
    • sentinel: backup.fromReplicaOrdinal (default 1) — a replica, never the primary.
    • cluster: one Pod per shard at the same ordinal, falling back to ordinal 0 if the chosen replica is unreachable. When backup.allShards: false, only shard-0 is dumped.
  2. Issues BGSAVE on the target Pod.
  3. Polls LASTSAVE until it changes — confirming the new RDB has been flushed.
  4. Pulls the RDB locally via valkey-cli --rdb /tmp/...rdb.
  5. Uploads to S3 with s3cmd put at a deterministic key.

The Pod runs in the chart’s namespace, uses the same TLS cert and auth password as ValKey, and finishes when all targets have uploaded. concurrencyPolicy: Forbid prevents overlapping runs.

S3 key layout

s3://<bucket>/<prefix>/<release-fullname>/<label>/<UTC-timestamp>.rdb

When clusterRegion is set, it is appended to the prefix — s3://<bucket>/<prefix>/<clusterRegion>/<release-fullname>/… — so multi-region installs sharing one bucket and prefix can never collide, and each key names the region it came from. Region-less installs keep the plain layout above.

Mode Label
standalone <release-fullname>
sentinel <release-fullname>
cluster (per shard) <release-fullname>-shard-<N>

Example (release vk in mode cluster, prefix valkey):

s3://my-backups/valkey/vk-valkey/vk-valkey-shard-0/20260521T020000Z.rdb
s3://my-backups/valkey/vk-valkey/vk-valkey-shard-1/20260521T020000Z.rdb
s3://my-backups/valkey/vk-valkey/vk-valkey-shard-2/20260521T020000Z.rdb

The timestamp is the time the CronJob started, in UTC. All shards of a single cluster run share the same timestamp because the CronJob iterates them in a single Pod.

S3 lifecycle

The CronJob only uploads — retention is a bucket lifecycle rule you apply once per bucket, or snapshots accumulate indefinitely. A flat expiry on the valkey/ prefix, applied with s3cmd setlifecycle, is enough. Choose the expiry (or add storage-class transition rules) to suit your own RPO/RTO — any S3-compatible store with lifecycle support (UpCloud Object Storage, MinIO, AWS) works the same way.

Restore

The chart attaches a restore init container to every primary Pod in the deployment. The init container:

  1. Checks for /data/.restore-complete. If present, exits 0 — restores are idempotent within a PVC lifetime. (To re-restore, delete the marker first.)
  2. For non-primary Pods (sentinel replicas; cluster replicas), writes the marker and exits. They resync from their primary via replication.
  3. Moves any existing AOF material (appendonlydir / appendonly.aof*) aside to /data/.pre-restore-<ts>/ — valkey-server boots from the AOF whenever one exists, so on a live-instance restore stale AOF data would otherwise silently win over the restored RDB. The previous generation is kept for operator recovery; older ones are swept.
  4. Downloads the RDB from S3 to /data/dump.rdb via s3cmd get.
  5. Writes /data/.restore-complete.
  6. The main valkey-server container then starts and loads the RDB on boot.

This works both for first boot on a fresh PVC (disaster recovery) and as an in-place rollback on a running instance (enable restore.*, roll the pods, then converge the values away).

Standalone / Sentinel

restore:
  enabled: true
  s3Key: "valkey/vk-valkey/vk-valkey/20260521T020000Z.rdb"

The single primary (standalone Pod, or sentinel ordinal 0) downloads the literal key. All other Pods skip the download.

Cluster (automated)

Cluster restore is automated end-to-end. The S3 key uses a {shard} placeholder substituted at runtime with <fullname>-shard-<N> to match the per-shard layout the backup CronJob writes:

restore:
  enabled: true
  s3Key: "valkey/vk-valkey/{shard}/20260521T020000Z.rdb"

Each shard’s ordinal-0 Pod downloads its own per-shard RDB. Replicas skip the download.

After the Pods boot with restored data, the bootstrap Job runs and inspects the cluster:

  1. If cluster_state:ok on shard-0-0 → already formed, no-op.
  2. Else, DBSIZE is checked on each shard’s primary. If any primary has data, the bootstrap switches from --cluster create (which refuses non-empty nodes) to a manual join:
    • Slot assignment — each shard’s primary takes its slot range via CLUSTER ADDSLOTSRANGE <start> <end>. Slot ranges are computed deterministically by shard count, matching the layout the backup was taken from.
    • Gossip — every other primary issues CLUSTER MEET shard-0-0 so the cluster forms a single mesh.
    • Replica attach — each replica issues CLUSTER MEET for gossip and then CLUSTER REPLICATE <primary-id> to attach to its shard primary. Stale data on the replica is wiped during the initial sync from the primary.
  3. Otherwise, the bootstrap runs the usual --cluster create --cluster-replicas N.

The slot-range arithmetic is slot range k = [floor(k × 16384/N), floor((k+1) × 16384/N) - 1], with the final shard getting the remainder. Because CRC16 is deterministic, keys land in the same slots they did before the backup — so a per-shard RDB taken from shard-N restores cleanly into shard-N of a cluster with the same shard count.

Constraints worth knowing

  • Shard count must match. A cluster backed up at 3 shards can only be restored into a 3-shard cluster. Changing shard count requires a manual --cluster reshard after restore — the chart will not do this for you.
  • Replica count can change. Backing up 3×1 and restoring into 3×2 (or 3×0) is fine; the manual join attaches whatever replicas the new topology has.
  • nodes.conf is not backed up. The cluster’s node IDs are regenerated on restore. Clients should always go through the cluster’s hostname-based discovery, not pinned node IDs.
  • In-flight writes between BGSAVE and upload are lost. The CronJob targets a replica, so writes during the backup window are captured if they’re replicated before BGSAVE completes, but not after.

Forcing a re-restore

Delete the marker on the PVC and restart the Pod:

kubectl exec -it -n <ns> vk-valkey-0 -c valkey -- rm -f /data/.restore-complete
kubectl delete pod -n <ns> vk-valkey-0

# Or, for cluster shard-1:
kubectl exec -it -n <ns> vk-valkey-shard-1-0 -c valkey -- rm -f /data/.restore-complete
kubectl delete pod -n <ns> vk-valkey-shard-1-0

Disaster recovery patterns

Same-region restore

CronJob writes to a regional bucket. On loss, re-deploy the chart with restore.enabled: true pointing at the latest key. Standalone/sentinel restores complete in minutes; data caught up via AOF afterwards.

Cross-region replication

The chart writes to a single S3 target. For geo-redundancy, either:

  1. Use a bucket with cross-region replication enabled at the storage provider (Hetzner, AWS S3 CRR, MinIO).
  2. Run the CronJob in two clusters that write to different buckets, accepting double the BGSAVE load.

The MySQL chart’s secondaryS3 option is not mirrored here — geo-redundant S3 buckets are the cleaner approach for ValKey, and BGSAVE load is heavier than mysqldump.

Point-in-time recovery

The chart does not ship PITR for ValKey. AOF gives sub-second durability but is not shipped off-Pod. If you need PITR:

  • Increase AOF fsync to always (writes hit the kernel per command).
  • Schedule the backup CronJob more aggressively (e.g. every 15 minutes).
  • Accept the RPO that gives you.

True PITR for ValKey would require streaming the AOF to S3 continuously. That’s a future chart feature and is not implemented today.