Documentation / mysql
Clone-based seeding & self-healing reseed
This doc covers how the chart uses the MySQL CLONE plugin to seed fresh-PVC replicas without binlog-from-zero replay, and how it gives Group Replication a free self-healing reseed path when binlogs are insufficient.
It pairs with self-healing-reseed.md, the
reseed path for a replica whose binlogs have run out.
What it does
| Mode | Behaviour |
|---|---|
| Standard (1 primary + N async replicas) | On a fresh-PVC replica boot, the clone-seed init container clones the entire data dir from pod-0 over the network using CLONE INSTANCE FROM. Replication is auto-configured against the donor’s GTID position. No multi-hour binlog replay. |
| HA (Group Replication, ≥3 members) | The clone-seed init container is a no-op. GR’s built-in distributed-recovery uses the clone plugin automatically when a joining member’s GTID set is older than the oldest binlog on the existing primary. Self-healing reseed comes for free. |
Without clone seeding, a fresh replica in either mode has to replay every binlog the primary still has. For active databases this can take hours or fail outright if the primary has purged binlogs past the replica’s position.
How it works (Standard mode replica)
┌─ Pod boot (replica, fresh PVC) ────────────────────────────────────────┐
│ │
│ initContainers: │
│ 1. init-mysql │
│ - Generates 99-replication.sh (used only on legacy first boot) │
│ - Generates 00-users.sql with clone_user grant │
│ │
│ 2. clone-seed ◄── new │
│ - Sentinel check → /var/lib/mysql/.bahriya-seeded │
│ - Wait for donor (pod-0) reachable │
│ - mysqld --initialize-insecure (empty data dir) │
│ - Start temp mysqld (socket-only, --skip-networking) │
│ - CLONE INSTANCE FROM 'clone_user'@pod-0 │
│ - (clone wipes & repopulates /var/lib/mysql with donor's data) │
│ - Verify performance_schema.clone_status = 'Completed' │
│ - CHANGE REPLICATION SOURCE TO ... pod-0 │
│ - touch .bahriya-seeded │
│ │
│ containers: │
│ mysql: │
│ - docker-entrypoint detects populated data dir → skips initdb │
│ - Starts mysqld with cloned data + configured replication channel │
│ - START REPLICA runs automatically; catches up from donor GTID │
│ │
└─────────────────────────────────────────────────────────────────────────┘
Why a temp mysqld inside an init container?
CLONE INSTANCE FROM is run on the recipient: you connect to the
recipient’s local mysqld and tell it to clone from a remote donor. The clone
plugin needs a running mysqld to execute against — there is no offline tool.
So clone-seed boots a minimal socket-only mysqld inside the init container,
runs CLONE INSTANCE, and exits. The data dir on the shared PVC is left
populated with the donor’s state, and the main mysql container then starts
normally on that data dir.
Why socket-only:
--skip-networking --port=0prevents the temp mysqld from accepting any TCP connections. The only consumer is the same-process bash script.- The mysql client connects via Unix domain socket at
/tmp/clone-mysql.sock.
How it works (HA mode)
HA mode uses Group Replication’s distributed recovery. When a joining member’s GTID set is older than the oldest binlog on the existing group, GR automatically falls back to a clone-based recovery:
- The joining member queries the group for a donor.
- The donor runs
CLONE INSTANCEon behalf of the joiner. - The joiner’s data dir is overwritten with the donor’s state.
- The joiner restarts and rejoins the group at the cloned GTID position.
This requires only two things, both of which the chart now provides:
mysql_clone.soloaded on every member (viaplugin_load_add).clone_userwithBACKUP_ADMINandCLONE_ADMINgrants existing on every member.clone_valid_donor_listset to all members (loose-clone_valid_donor_listin custom.cnf).
So clone-seed in HA mode just writes the sentinel and exits — there is
nothing for it to do at pod boot. GR triggers clone on demand.
Files involved
| File | Role |
|---|---|
templates/configmap.yaml |
Adds plugin_load_add = mysql_clone.so and loose-clone_valid_donor_list = <all-pods>:3306 to [mysqld]. The loose- prefix matters: mysqld reads custom.cnf during --initialize too, before the plugin loads, and an unknown variable there aborts the first boot (earlier versions of the chart set the list through an init_file and failed every fresh install exactly that way). |
templates/secret.yaml |
Adds clone-password key; auto-generated; preserved across helm upgrade via lookup. |
templates/statefulset.yaml |
Adds the clone-seed init container; injects MYSQL_CLONE_PASSWORD into init-mysql. |
templates/_helpers.tpl |
Defines mysql.cloneDonorList helper — comma-separated list of all member host:port. |
values.yaml |
Exposes mysql.clone.user (default clone_user) and mysql.clone.password (auto-generated if empty). |
Failure modes and what happens
| Failure | Behaviour | Recovery |
|---|---|---|
| Donor (pod-0) unreachable for >5 minutes during clone-seed boot | clone-seed logs a WARN and exits 0 (graceful). Docker-entrypoint then runs initdb normally — replica starts with an empty data dir and replicates from primary via binlogs. | Delete the replica’s PVC and let the StatefulSet recreate it once pod-0 is healthy. |
CLONE INSTANCE returns failure (network drop mid-clone, donor crashed) |
clone-seed verifies via performance_schema.clone_status. If state ≠ Completed, exits 1. Pod stays in Init:Error. |
Delete the pod (sentinel not written; on retry clone-seed runs again with a fresh attempt). If repeated failures: check donor connectivity and clone_user grants. |
clone_user does not exist on donor |
CLONE INSTANCE fails with auth error. Same recovery as above. |
The reconcile Job creates clone_user on the next install or upgrade. |
| Binlogs purged past replica’s position (Standard mode, after clone-seed already wrote sentinel) | Replica’s Last_IO_Errno = 1236. Replication breaks. |
Auto-recovered by the reseed-monitor sidecar (Wave 2 4c — see self-healing-reseed.md). The sidecar deletes the sentinel and self-deletes the pod; StatefulSet recreates → clone-seed re-clones. To force a reseed manually instead: kubectl delete pod <pod> after kubectl exec ... rm /var/lib/mysql/.bahriya-seeded. |
| GR member falls behind purged binlogs (HA mode) | GR distributed recovery automatically uses clone. No manual intervention. | Built-in; no action needed. |
Sentinel semantics
/var/lib/mysql/.bahriya-seeded is a marker the chart uses to remember that a
pod has completed its initial seeding. It is not a marker that the data is
“good” or “current” — only that the chart’s seed flow has run.
- Created by
clone-seedafter successful clone (Standard mode replicas), or immediately on first boot for pod-0 / HA pods / legacy populated PVCs. - Lives on the data PVC, so it survives pod restarts.
- Wiped only when the data PVC itself is deleted (then a fresh seed is triggered on next pod start).
- Wave 2’s binlog-horizon self-healing will additionally delete the sentinel whenever it detects irrecoverable replication breakage.
If you want to force a reseed on a Standard mode replica:
kubectl exec -n NS POD -- rm /var/lib/mysql/.bahriya-seeded
kubectl delete pod -n NS POD
The replacement pod will re-run clone-seed and re-clone from pod-0.
Never delete the sentinel on pod-0 in Standard mode — pod-0 is the donor and has no upstream to clone from. The chart treats pod-0 + sentinel-missing as a no-op (writes sentinel, exits 0), so it won’t break, but it’s a confusing state.
Verification
Standard mode replica clone (manual test against a dev cluster):
# 1. Deploy with 2 replicas
helm install m helm-charts/charts/mysql -f k8s/mysql/tests/02-standard-replicas.yaml -n mtest --create-namespace
# 2. Write some data on primary
kubectl exec -n mtest m-mysql-0 -- mysql -uroot -p$PW -e "CREATE TABLE testdb.t (id INT); INSERT INTO testdb.t VALUES (1),(2),(3);"
# 3. Force a reseed on a replica by wiping its PVC
kubectl delete pod -n mtest m-mysql-1
kubectl delete pvc -n mtest data-m-mysql-1 binlog-m-mysql-1
# StatefulSet will recreate pod-1 with fresh PVCs
# 4. Watch clone-seed run
kubectl logs -n mtest m-mysql-1 -c clone-seed -f
# Expect: "clone-seed: clone completed successfully"
# Expect: "clone-seed: replication channel configured"
# Expect: "clone-seed: sentinel /var/lib/mysql/.bahriya-seeded written"
# 5. Verify data + replication
kubectl exec -n mtest m-mysql-1 -- mysql -uroot -p$PW -e "SELECT * FROM testdb.t; SHOW REPLICA STATUS\G"
# Expect: 3 rows; Replica_IO_Running=Yes, Replica_SQL_Running=Yes