Documentation / mysql
Self-healing reseed
Self-healing recovery from the “binlog horizon” failure: a replica’s GTID position is older than the oldest binlog on the primary, so it cannot catch up via replication.
Without this feature, fixing a horizon-stuck replica is a manual delete-PVC-and-wait dance. With it, the replica detects the failure, self-terminates, and the init container re-clones from the donor on pod recreation. End-to-end: detect → reclone → back online, no human in the loop.
This pairs with:
clone-based-seeding.md— the clone path that actually re-seeds the replica.
Modes
| Mode | How self-healing works |
|---|---|
| HA (Group Replication) | Built-in. GR’s distributed recovery uses the clone plugin automatically when binlogs are insufficient. No sidecar, no script — the plugin + clone_user grants we set up in Wave 1 step 2 are enough. |
| Standard (async replication) | A reseed-monitor sidecar on each pod polls replication state. On Last_IO_Errno = 1236, it deletes the seed sentinel and self-deletes the pod via the K8s API. StatefulSet recreates the pod; clone-seed init container re-clones from pod-0. |
Sidecar architecture (Standard mode)
┌─ pod (standard mode replica, ordinal > 0) ───────────────────────┐
│ │
│ ┌──────────────────┐ ┌──────────────────────────────────┐ │
│ │ mysql container │◄──┐│ reseed-monitor sidecar │ │
│ │ (mysqld) │ ││ │ │
│ │ │ ││ every 30s: │ │
│ └────────┬─────────┘ ││ mysql -e "SELECT │ │
│ │ ││ LAST_ERROR_NUMBER FROM │ │
│ │ shares ───► ││ perf_schema.replication_*" │ │
│ │ /var/lib/ ││ │ │
│ │ mysql PVC ││ if errno == 1236: │ │
│ │ ││ rm /var/lib/mysql/.bahriya- │ │
│ │ ││ seeded │ │
│ │ ││ DELETE /api/.../pods/self │ │
│ │ ││ │ │
│ ┌────────▼─────────┐ │└──────────────────┬───────────────┘ │
│ │ data PVC │◄──┘ │ │
│ │ /var/lib/mysql │ │ │
│ └──────────────────┘ │ │
│ │ │
└─────────────────────────────────────────────┼───────────────────┘
│
▼
K8s deletes the pod;
StatefulSet recreates it;
clone-seed init container sees missing sentinel;
re-clones from pod-0;
new replica online with current data.
Why a sidecar and not a liveness probe
A liveness-probe-driven approach was the original card spec, but:
- Liveness probe failure restarts the container, not the pod.
- Init containers only run on pod startup, not container restart.
- So a failed liveness probe would never re-trigger
clone-seed.
A sidecar that self-deletes the pod gives us the pod recreation we need.
Files
| File | Role |
|---|---|
templates/scripts-configmap.yaml |
Now renders for both modes. Contains post-start.sh (HA only) and reseed-monitor.sh (Standard mode replicas only). |
templates/statefulset.yaml |
Adds the reseed-monitor container to the pod template when mode=standard AND selfHealingReseed.enabled AND standard.readReplicas > 0. |
templates/rbac.yaml |
Role grants delete on pods for Standard mode (in addition to the existing get+patch for HA’s role-labeler). |
values.yaml mysql.selfHealingReseed.* |
Toggle, poll interval, sidecar resource limits. |
Values reference
| Key | Default | Notes |
|---|---|---|
mysql.selfHealingReseed.enabled |
true |
Set false to disable the sidecar entirely. HA mode is unaffected (its self-healing is via GR). |
mysql.selfHealingReseed.intervalSeconds |
30 |
Polling interval. Each cycle = 1 SQL query + sleep. |
mysql.selfHealingReseed.resources |
cpu 50m / mem 64Mi limits |
Sidecar is idle most of the time. |
Failure modes & guardrails
| Scenario | Behaviour |
|---|---|
| Local mysql not reachable | Sidecar logs and retries next cycle. No false-positive self-delete. |
Genuine Last_IO_Errno=1236 |
Sidecar deletes sentinel + DELETEs the pod. Recovery flow kicks in. |
| Other replication errors (auth failure, network blip, schema mismatch) | Sidecar ignores them. Only 1236 (binlog horizon) triggers reseed — 1236 is the only error a re-clone actually fixes. |
pod-0 (primary) |
Sidecar self-detects ordinal, sleeps forever. |
| K8s API unreachable | Sidecar logs the HTTP error from DELETE /pods/self. Sentinel is already gone; on next normal pod restart (manual or otherwise), init container will re-clone. |
| ServiceAccount token missing | Sidecar logs and exits the cycle. Same recovery path as above. |
| Donor (pod-0) also broken | clone-seed waits 5 min for donor, then exits 0 (graceful fallback). Replica comes up empty; admin must restore from backup. |
Verification
# 1. Deploy standard mode with 1 replica
helm install m helm-charts/charts/mysql -f k8s/mysql/tests/02-standard-replicas.yaml -n mtest --create-namespace
# 2. Verify reseed-monitor sidecar is running on the replica
kubectl get pod m-mysql-1 -n mtest -o jsonpath='{.spec.containers[*].name}'
# Expect: "mysql reseed-monitor"
# 3. Force a binlog horizon failure by purging binlogs on primary past replica's position
PRIMARY_PW=$(kubectl get secret -n mtest m-mysql -o jsonpath='{.data.root-password}' | base64 -d)
kubectl exec -n mtest m-mysql-0 -- mysql -uroot -p$PRIMARY_PW -e "
STOP REPLICA;
-- write a lot, flush, purge so replica's position is past the horizon
RESET MASTER;
"
# Now restart pod-1 — its replica thread will see 1236
kubectl exec -n mtest m-mysql-1 -- mysql -uroot -p$PRIMARY_PW -e "START REPLICA;"
# 4. Watch the sidecar trigger reseed
kubectl logs -n mtest m-mysql-1 -c reseed-monitor -f
# Expect: "Last_IO_Errno=1236 (binlog horizon) — triggering reseed"
# Expect: "pod DELETE returned HTTP 200"
# 5. Watch the pod come back and clone-seed re-run
kubectl get pods -n mtest -w
kubectl logs -n mtest m-mysql-1 -c clone-seed -f
# Expect: "clone-seed: clone completed successfully"
# 6. Verify replica is back ONLINE
kubectl exec -n mtest m-mysql-1 -- mysql -uroot -p$PRIMARY_PW -e "SHOW REPLICA STATUS\G"
# Expect: Replica_IO_Running=Yes, Replica_SQL_Running=Yes
Disabling
If you don’t want self-deletion (e.g. you have your own pod controller):
mysql:
selfHealingReseed:
enabled: false
The sidecar will not be added, the RBAC delete verb will not be granted,
and replicas stuck at the binlog horizon will require manual intervention
(delete the PVC, let the StatefulSet recreate, clone-seed runs).