Documentation / mysql

MySQL Helm Chart

A Helm chart for deploying MySQL on Kubernetes with two deployment modes: Standard (single primary with optional read replicas) and HA Cluster (Group Replication with automatic failover).

Prerequisites

Kubernetes 1.23+
Helm 3.x
Longhorn storage class (default) or any other ReadWriteOnce-capable storage class
The chart’s MySQL image (1x.ax/bahriya/mysql, the default) and a pull secret for 1x.ax (imagePullSecrets). The chart’s pod-side scripts — clone seeding, the Group Replication post-start, the reseed monitor and the reconcile Job — live in that image (repository bahriya/mysql), so stock mysql images do not work with this chart.

Architecture

This chart supports two deployment modes, designed for different reliability and cost requirements.

Standard Mode

                    ┌──────────────┐
                    │   Clients    │
                    └──────┬───────┘
                           │
              ┌────────────┴────────────┐
              │                         │
     ┌────────▼────────┐     ┌─────────▼─────────┐
     │  mysql (svc)     │     │  mysql-readonly    │
     │  targets pod-0   │     │  (svc, all pods)   │
     └────────┬─────────┘     └────────┬──────────┘
              │                        │
     ┌────────▼────────┐    ┌──────────▼──────────┐
     │   mysql-0       │    │  mysql-1   mysql-2   │
     │   PRIMARY       │───▶│  REPLICA   REPLICA   │
     │   (read-write)  │    │  (read-only)         │
     └─────────────────┘    └──────────────────────┘
            GTID async replication ──────▶

Pod-0 is always the primary (read-write).
Pod-1..N are read replicas using GTID-based asynchronous replication.
A dedicated Kubernetes Service targets pod-0 for writes.
A separate read-only Service round-robins across all pods (including the primary).
If the primary pod crashes, Kubernetes restarts it on the same PVC. Data is intact. Secondaries reconnect automatically via GTID.
Typical recovery time: 30-90 seconds.

Pros

Simple to operate and reason about.
Any number of read replicas (including zero for a standalone primary).
Lower resource requirements.

Cons

No automatic failover. If the primary dies, writes are unavailable until Kubernetes restarts it.
If the node running the primary fails, recovery depends on Kubernetes rescheduling (up to 5-7 minutes with default tolerations).
Read replicas may lag behind the primary under heavy write load (asynchronous replication).

When to use Standard mode

Development and staging environments.
Applications that can tolerate brief write unavailability.
Workloads where read scaling is more important than write availability.
Cost-sensitive deployments.

HA Cluster Mode

                    ┌──────────────┐
                    │   Clients    │
                    └──────┬───────┘
                           │
              ┌────────────┴────────────┐
              │                         │
     ┌────────▼────────┐     ┌─────────▼─────────┐
     │  mysql (svc)     │     │  mysql-readonly    │
     │  label: primary  │     │  (svc, all pods)   │
     └────────┬─────────┘     └────────┬──────────┘
              │                        │
        ┌─────┼────────────────────────┼────┐
        │     │                        │    │
   ┌────▼────┐      ┌─────────┐      ┌▼────────┐
   │ mysql-0 │◄────▶│ mysql-1 │◄────▶│ mysql-2 │
   │ PRIMARY │      │SECONDARY│      │SECONDARY│
   └─────────┘      └─────────┘      └─────────┘
         Group Replication (single-primary mode)

Uses MySQL Group Replication in single-primary mode with communication_stack = MYSQL (all GR traffic on port 3306).
Minimum 3 members (enforced by the chart). Odd numbers recommended for quorum (3, 5, 7).
The group automatically elects a new primary if the current one fails. No external orchestrator needed.
A background monitoring loop on each pod checks its GR role every 5 seconds and labels the pod with mysql.bahriya.cloud/role: primary or secondary.
The mysql Kubernetes Service selects pods with role: primary, so it always routes to the current GR primary.
Clients connect to the mysql service for writes and mysql-readonly for reads — no application-level primary discovery needed.

How failover works

The primary pod (e.g., mysql-0) becomes unreachable.
The remaining group members detect the failure via the Group Replication protocol.
A new primary is automatically elected from the remaining ONLINE members.
The new primary begins accepting writes immediately.
The monitoring loop on the new primary detects the role change and patches the pod label within ~5 seconds.
The mysql Kubernetes Service routes new connections to the new primary.
When the failed pod recovers, it rejoins the group as a secondary.

Typical failover time: 5-15 seconds (GR election + pod label update).

Pros

Automatic failover with no external tooling or extra components.
All committed transactions are guaranteed to be on at least a majority of members before acknowledgement (virtually synchronous).
No data loss on failover (within the group’s consistency guarantees).
Uses native Kubernetes services for routing — no MySQL Router or external proxy needed.

Cons

Minimum 3 nodes required (higher base cost).
Slightly higher write latency due to group consensus protocol.
More complex to debug when group membership issues occur.
Large transactions (bulk inserts, DDL) are slower due to group certification.

When to use HA Cluster mode

Production workloads where write availability is critical.
Applications that cannot tolerate minutes of write downtime.
Environments where you need automatic recovery without operator intervention.

Installation

# Standard mode (single primary, no replicas)
helm install my-db charts/mysql

# Standard mode with 2 read replicas
helm install my-db charts/mysql --set standard.readReplicas=2

# HA Cluster mode (3 members)
helm install my-db charts/mysql --set mode=ha

# HA Cluster mode with 5 members
helm install my-db charts/mysql --set mode=ha --set ha.members=5

MySQL Configuration

All MySQL server variables can be configured via mysql.config in values.yaml. Each key-value pair is rendered into a custom.cnf file mounted at /etc/mysql/conf.d/custom.cnf.

mysql:
  config:
    max_connections: 500
    innodb_buffer_pool_size: "2G"
    innodb_log_file_size: "256M"
    innodb_flush_method: "O_DIRECT"
    innodb_flush_log_at_trx_commit: 1
    sync_binlog: 1
    character_set_server: "utf8mb4"
    collation_server: "utf8mb4_unicode_ci"

For settings not modelled in the config map, use the escape hatch:

mysql:
  extraConfig: |
    innodb_io_capacity = 2000
    innodb_io_capacity_max = 4000
    innodb_read_io_threads = 8
    innodb_write_io_threads = 8

Longhorn storage defaults

The chart defaults are optimised for Longhorn volume storage:

Setting	Value	Reason
`innodb_flush_method`	`O_DIRECT`	Bypasses OS page cache, avoids double-buffering with Longhorn’s own caching
`innodb_flush_log_at_trx_commit`	`1`	Crash-safe: flush redo log on every commit
`sync_binlog`	`1`	Crash-safe replication: sync binlog on every commit

Logging

mysql:
  logging:
    slowQuery: true        # Enable slow query log
    slowQueryTime: 2       # Queries longer than 2 seconds
    generalLog: false      # Very verbose, disable in production
    errorLog: true         # Always recommended

Storage

The chart creates two separate PersistentVolumeClaims per pod:

PVC	Default size	Mount path	Purpose
`data`	10Gi	`/var/lib/mysql`	Database files, InnoDB tablespace, indexes
`binlog`	5Gi	`/var/lib/mysql-binlog`	Binary logs for replication

Separate PVCs allow independent sizing. Binary logs can be smaller and use different retention policies than the main data volume.

Increasing storage

You can increase PVC size by updating storage.data.size or storage.binlog.size and running helm upgrade. Longhorn (and most CSI drivers) support online volume expansion for ReadWriteOnce volumes. The expansion happens without pod restart.

Important: Volume reduction is not supported by Kubernetes. PVCs can only grow, never shrink. This is a Kubernetes limitation, not specific to this chart or Longhorn.

Secrets

The chart manages four credentials:

Key	Purpose
`root-password`	MySQL root user
`replication-password`	Replication user (used by replicas/GR members)
`clone-password`	Clone user (used by `clone-seed` init container and GR distributed recovery — see `docs/clone-based-seeding.md`)
`user-password`	Application user (optional, only if `mysql.user` is set)

If no passwords are provided and no existingSecret is set, the chart auto-generates 24-character alphanumeric passwords. On helm upgrade, the chart uses Helm’s lookup function to read the existing Secret and reuse the passwords, preventing regeneration.

Values-authoritative credentials (mysql.credentialsFromValues, default true, effective with reconcile.enabled): for the credentials the reconcile Job converges — rootPassword, databases[].password and clone.password — a value given in values replaces what the Secret holds, and the reconcile Job applies it to the running instance. That is how a rotation reaches MySQL. A root change also writes the outgoing password to root-password-previous, so the Job can still log in to apply the new one. The replication user, the legacy mysql.user and extraUsers keep the lookup behaviour unconditionally (nothing converges them).

To use an existing Secret:

mysql:
  existingSecret: my-mysql-credentials

The Secret must contain the keys listed above.

Seeding & self-healing

The chart provisions replicas via the MySQL CLONE plugin and self-heals horizon-stuck replicas without manual intervention.

Mode	Fresh replica boot	Recovery from binlog-horizon failure
Standard	`clone-seed` init container clones from pod-0; replica starts at the donor’s GTID position.	`reseed-monitor` sidecar detects `Last_IO_Errno=1236`, deletes the seed sentinel, and self-deletes the pod via the K8s API. StatefulSet recreates the pod; `clone-seed` re-clones from pod-0.
HA (Group Replication)	Pod joins the group; GR’s distributed recovery uses clone automatically when binlogs are insufficient.	Same path as fresh boot — GR distributed recovery handles it.

Practical implications:

Deleting a replica’s PVC triggers an automatic reseed on next pod start — the clone-seed init container sees no .bahriya-seeded sentinel and re-clones the data dir from a donor. Useful for force-rebuilding a stuck replica.
clone_user is maintained by the reconcile Job on every install and upgrade, so a data directory that predates it gains it on the next upgrade.

Full design and verification recipes:

docs/clone-based-seeding.md — clone plugin, init container, sentinel semantics, failure modes.
docs/self-healing-reseed.md — sidecar architecture, why-sidecar-not-liveness, manual binlog-purge verification recipe.

Databases, users and the reconcile Job

mysql.databases[] entries with a user are converged on every install and upgrade by a post-install/post-upgrade hook Job running /opt/bahriya/reconcile.sh against the primary:

each entry: schema and user created if missing, the user’s password set to the Secret’s, ALL PRIVILEGES on its own schema;
an entry removed from the list: its schema and user are dropped — but only schemas the Job created, recorded in a bahriya_managed registry inside the instance. Schemas created by hand over SQL are never touched;
platform users: clone_user and, when monitoring.existingSecret is set, the read-only monitoring user (monitoring.user, default netdata: PROCESS, REPLICATION CLIENT, SELECT on performance_schema).

A failing reconcile fails the release. Managed entries are kept out of the init container, so the database list is not part of the pod template: adding, removing or rotating a database never restarts MySQL. Entries without a user or with custom grants, and the legacy mysql.database / mysql.user / extraUsers, stay on the first-boot initdb path. Disable with reconcile.enabled: false (which also turns off values-authoritative credentials).

Backups

Two layouts (backup.layout, used by both the dump and binlog-shipping jobs):

Layout	Image	Keys
`platform` (default)	`backup.tools.image` (`1x.ax/bahriya/mysql-backup`, client binaries pinned to the server version)	`<backup.s3.prefix>/<clusterRegion>/dumps/<db>/<UTC-ts>.sql.gz` and `.../binlogs/<UTC-ts>-<server_uuid>-<file>.gz`
`legacy`	`backup.image` (db-backup) and `pitr.binlogShipping.image`	`<db>/<db>-DD-Mon-YYYY-HH-MM.sql.tar.gz` and `<pitr prefix>/<release>/<file>.gz`

The platform layout requires backup.s3.prefix and clusterRegion, dumps each declared database (no “all” dump), and stages and verifies every dump before upload.

The examples below use the legacy layout:

backup:
  layout: legacy

Enable automated S3 backups via a Kubernetes CronJob:

backup:
  enabled: true
  schedule: "0 2 * * *"    # Daily at 02:00
  s3:
    accessKey: "AKIAXXXXXXXX"
    secretKey: "xxxxxxxxxxxxx"
    host: "s3.us-east-1.amazonaws.com"
    bucket: "my-backups"
    region: "us-east-1"

The backup CronJob uses mysqldump to create a compressed SQL dump and uploads it to the specified S3-compatible storage. Backups are stored at s3://bucket/database-name/database-name-DD-Mon-YYYY-HH-MM.sql.tar.gz.

To use an existing Secret for S3 credentials:

backup:
  enabled: true
  s3:
    existingSecret: my-s3-credentials

The Secret must contain keys: S3_ACCESS_KEY, S3_SECRET_KEY, S3_HOST, S3_BUCKET, S3_REGION.

Secondary S3 target

For geo-redundant backups or shipping to a separate account/provider, configure a secondary S3 target. The backup runs once — the dump is performed a single time and the same compressed file is uploaded to both targets sequentially within the same job.

backup:
  enabled: true
  s3:
    accessKey: "primary-key"
    secretKey: "primary-secret"
    host: "s3.us-east-1.amazonaws.com"
    bucket: "primary-backups"
    region: "us-east-1"
  secondaryS3:
    enabled: true
    accessKey: "secondary-key"
    secretKey: "secondary-secret"
    host: "s3.eu-west-1.amazonaws.com"
    bucket: "dr-backups"
    region: "eu-west-1"

The secondary target credentials are passed to the backup container as S3_SECONDARY_* environment variables. The db-backup image handles the dual upload internally.

Metrics

Enable Prometheus metrics via a mysqld-exporter sidecar:

metrics:
  enabled: true

The exporter sidecar exposes /metrics on port 9104 with Prometheus annotations for auto-discovery:

prometheus.io/scrape: "true"
prometheus.io/port: "9104"
prometheus.io/path: "/metrics"

Web client (phpMyAdmin)

The chart can deploy a phpMyAdmin web interface for browser-based MySQL management. It is exposed via Kong ingress with automatic TLS (cert-manager) and IP allowlisting for security.

This is designed for on-demand access — enable when you need to inspect or manage the database, disable when done.

client:
  enabled: true
  ipAllowlist:
    - "203.0.113.0/24"       # office network
    - "198.51.100.42/32"     # developer VPN

When enabled, the chart creates:

A phpMyAdmin Deployment (1 replica) that connects to the MySQL primary service.
A ClusterIP Service fronting the phpMyAdmin pod.
A Kong Ingress with HTTPS enforcement and automatic Let’s Encrypt TLS via cert-manager.
A KongPlugin (ip-restriction) that restricts access to the configured IP allowlist.
A cert-manager Certificate for the auto-generated hostname.

Hostname

Set client.ingress.host explicitly, or give client.defaultHostname + client.domain (with clusterRegion) and the chart generates the name pma-<defaultHostname>.<clusterRegion>.<domain>.

client:
  enabled: true
  ingress:
    host: "db-myapp.example.com"
    certIssuer: "letsencrypt-prod"
  ipAllowlist:
    - "10.0.0.0/8"

Security

The IP allowlist is strongly recommended whenever the client is enabled. Without it, the phpMyAdmin login page would be exposed to the internet (though still protected by MySQL credentials).

The Kong ip-restriction plugin is applied at the ingress level, so blocked IPs never reach phpMyAdmin. Combined with TLS and MySQL authentication, this provides three layers of security:

Network: IP allowlisting via Kong
Transport: TLS encryption via cert-manager
Application: MySQL username/password authentication

External access (TCPIngress by port)

MySQL negotiates TLS inside its own protocol after the server greets, so an edge proxy never sees a hostname to route on. exposure.external binds one pre-provisioned edge port to the primary Service:

tls:
  enabled: true            # required: Kong forwards the stream, MySQL offers TLS
exposure:
  external:
    enabled: true
    port: 30117            # allocated from the edge port pool
    ipAllowlist: ["198.51.100.0/24"]   # required

The chart renders a port-only Kong TCPIngress, an ip-restriction KongPlugin, and a NetworkPolicy rule admitting Kong’s proxy pods. The port must already be open on the edge.

Network policy egress

networkPolicy.platformEgress (default on) adds pod-scoped egress allows for what the chart itself needs — intra-namespace (replication, Group Replication, clone, the Jobs), DNS, and 443/6443 (Kubernetes API, S3) — so a namespace egress allowlist cannot silently break replication, failover labelling or backups. With it on, this policy also governs the MySQL pods’ egress when nothing else does.

TLS (encrypted connections)

Enables TLS encryption for all MySQL client-to-server and inter-node connections.

tls:
  enabled: true
  requireSecureTransport: true    # reject plaintext connections

How it works

When TLS is enabled:

Helm generates a self-signed CA and server certificate at template time using genCA and genSignedCert. The certificates are stored in a Kubernetes Secret and preserved across upgrades via lookup (no regeneration on helm upgrade).
MySQL is configured with ssl_ca, ssl_cert, and ssl_key pointing to the mounted certificates.
The CA certificate is published as a ConfigMap (<release>-tls-ca) so any app in the namespace can mount it.
In HA mode, Group Replication is additionally configured with group_replication_ssl_mode=REQUIRED and encrypted recovery channels, so all inter-node replication traffic is encrypted.
If requireSecureTransport is true, MySQL rejects any plaintext connection — all clients must use TLS.
phpMyAdmin (when enabled) automatically connects to MySQL over TLS using the CA certificate.

CA certificate for client applications

The chart publishes the CA certificate as a ConfigMap named <release>-tls-ca. Any application in the namespace can mount this ConfigMap and use the ca.crt to verify the MySQL server.

# In your application's Deployment spec:
volumeMounts:
  - name: mysql-ca
    mountPath: /etc/mysql-tls
    readOnly: true
volumes:
  - name: mysql-ca
    configMap:
      name: my-db-tls-ca     # replace with your MySQL release name

Then configure your MySQL driver to use the CA:

Language / Driver	Configuration
PHP PDO	`new PDO($dsn, $user, $pass, [PDO::MYSQL_ATTR_SSL_CA => '/etc/mysql-tls/ca.crt'])`
Node.js mysql2	`createConnection({ ssl: { ca: fs.readFileSync('/etc/mysql-tls/ca.crt') } })`
Python mysqlclient	`connect(ssl={'ca': '/etc/mysql-tls/ca.crt'})`
Go database/sql	`tls.Config{ RootCAs: pool }` registered via `mysql.RegisterTLSConfig`
CLI	`mysql --ssl-ca=/etc/mysql-tls/ca.crt -h <host> -u <user> -p`

Bring your own certificates

To use certificates from your own CA (or cert-manager):

tls:
  enabled: true
  existingSecret: my-mysql-tls

The Secret must contain keys: ca.crt, tls.crt, tls.key. The CA is still published as a ConfigMap for client apps.

TLS modes

Configuration	Behaviour
`tls.enabled: false`	No TLS. All connections are plaintext.
`tls.enabled: true`, `requireSecureTransport: false`	TLS available but optional. Clients can connect with or without TLS.
`tls.enabled: true`, `requireSecureTransport: true`	TLS mandatory. MySQL rejects plaintext connections. Recommended for production.

Self-signed certificate SANs

The auto-generated certificate includes Subject Alternative Names for all internal DNS names:

<release>, <release>.<namespace>, <release>.<namespace>.svc.<clusterDomain>
<release>-headless, *.<release>-headless.<namespace>.svc.<clusterDomain> (for individual pods)
<release>-readonly (HA and standard with replicas)
localhost, 127.0.0.1

This ensures TLS verification works regardless of which service or pod DNS name the client uses.

Connecting your application over TLS

When TLS is enabled, your application containers need two things:

The CA certificate mounted from the <release>-tls-ca ConfigMap.
The MySQL driver configured to use the CA for server verification.

Step 1: Mount the CA certificate

In your application’s Deployment (or Safeena Helm values), add a volume and volume mount:

# Safeena values.yaml
extraVolumes:
  - name: mysql-ca
    configMap:
      name: my-db-tls-ca          # replace with your MySQL release name + "-tls-ca"

extraVolumeMounts:
  - name: mysql-ca
    mountPath: /etc/mysql-tls
    readOnly: true

env:
  - name: MYSQL_SSL_CA
    value: /etc/mysql-tls/ca.crt

Or in a raw Kubernetes Deployment:

spec:
  containers:
    - name: my-app
      env:
        - name: MYSQL_SSL_CA
          value: /etc/mysql-tls/ca.crt
      volumeMounts:
        - name: mysql-ca
          mountPath: /etc/mysql-tls
          readOnly: true
  volumes:
    - name: mysql-ca
      configMap:
        name: my-db-tls-ca

Step 2: Configure your MySQL driver

PHP (Kipchak / PDO)

In config/kipchak.doctrine.php:

'primary' => [
    'dbname' => env('DB_NAME', 'api'),
    'user' => env('DB_USER', 'api'),
    'password' => env('DB_PASSWORD', 'api'),
    'host' => env('DB_HOST', 'mysql'),
    'port' => env('DB_PORT', 3306),
    'driver' => 'pdo_mysql',
    'charset' => 'utf8',
    'driverOptions' => [
        \PDO::MYSQL_ATTR_SSL_CA => env('MYSQL_SSL_CA', '/etc/mysql-tls/ca.crt'),
        \PDO::MYSQL_ATTR_SSL_VERIFY_SERVER_CERT => true,
    ],
],

Requires kipchak/driver-doctrine v1.2+ which passes driverOptions through to Doctrine DBAL.

Node.js (mysql2)

const connection = mysql.createConnection({
  host: process.env.DB_HOST,
  user: process.env.DB_USER,
  password: process.env.DB_PASSWORD,
  database: process.env.DB_NAME,
  ssl: {
    ca: fs.readFileSync(process.env.MYSQL_SSL_CA || '/etc/mysql-tls/ca.crt'),
  },
});

Python (mysql-connector-python)

import mysql.connector
import os

conn = mysql.connector.connect(
    host=os.environ['DB_HOST'],
    user=os.environ['DB_USER'],
    password=os.environ['DB_PASSWORD'],
    database=os.environ['DB_NAME'],
    ssl_ca=os.environ.get('MYSQL_SSL_CA', '/etc/mysql-tls/ca.crt'),
)

Go (go-sql-driver/mysql)

rootCertPool := x509.NewCertPool()
pem, _ := os.ReadFile(os.Getenv("MYSQL_SSL_CA"))
rootCertPool.AppendCertsFromPEM(pem)
mysql.RegisterTLSConfig("custom", &tls.Config{RootCAs: rootCertPool})
// DSN: user:password@tcp(host:3306)/dbname?tls=custom

CLI (mysql client)

mysql --ssl-ca=/etc/mysql-tls/ca.crt -h <host> -u <user> -p

Encryption at rest

The chart supports two independent layers of encryption that can be used together for defense-in-depth.

Volume-level encryption (Longhorn LUKS)

Encrypts the entire block device underlying each PVC using LUKS (dm-crypt). MySQL is completely unaware of this encryption — it operates on a transparently decrypted block device. This is the simplest and most comprehensive form of encryption at rest.

encryption:
  volume:
    enabled: true
    passphrase: "my-super-secret-passphrase"   # auto-generated if empty

How it works:

The chart creates a Kubernetes Secret containing the LUKS passphrase (or uses an existing one).
A dedicated Longhorn StorageClass is created with encrypted: "true" and references the Secret.
Both data and binlog PVCs use this encrypted StorageClass instead of the default.
Longhorn encrypts the volume at the block level using dm-crypt/LUKS before writing to disk.
Decryption happens transparently when the volume is attached to a node.

BYOK: Provide your own passphrase via encryption.volume.passphrase or an existing Secret. If no passphrase is provided, the chart generates a 64-character random passphrase and preserves it across upgrades via lookup.

encryption:
  volume:
    enabled: true
    existingSecret: my-luks-credentials
    # Secret must contain: CRYPTO_KEY_VALUE, CRYPTO_KEY_PROVIDER ("secret")

MySQL InnoDB encryption

Encrypts the data MySQL writes, using its own encryption engine and the component_keyring_file keyring component (MySQL 8.4 removed the keyring_file plugin). The image loads the component; this switch gives it a keyring and turns encryption on.

encryption:
  innodb:
    enabled: true
    keyring:
      size: 64Mi          # per pod
      storageClass: ""    # defaults to storage.data.storageClass

How it works:

Each pod gets a small keyring volume (a volumeClaimTemplate) mounted at /var/lib/mysql-keyring. It is separate from the data volume, so a data-volume snapshot never carries the key that decrypts it.
On first boot the component creates the keyring and MySQL generates the pod’s master key in it. Each server has its own master key; a clone or Group Replication recovery re-encrypts the tablespace keys for the recipient.
The chart sets default_table_encryption, table_encryption_privilege_check, innodb_redo_log_encrypt, innodb_undo_log_encrypt and binlog_encryption. Every new schema and table is encrypted, and opting a table out needs TABLE_ENCRYPTION_ADMIN.
Rotate a pod’s master key online with ALTER INSTANCE ROTATE INNODB MASTER KEY; the keyring keeps the earlier keys.

Decide at install: the keyring volumeClaimTemplate cannot be added to an existing StatefulSet.

WARNING: Without a copy of a pod’s keyring, its encrypted data is permanently unrecoverable. Copy each pod’s keyring off and keep it somewhere the cluster’s failure cannot reach:

kubectl exec -n <namespace> <release>-mysql-0 -c mysql -- cat /var/lib/mysql-keyring/component_keyring_file > keyring-0.bin

Logical backups and shipped binlogs are not encrypted by either layer: mysqldump and mysqlbinlog --read-from-remote-server read decrypted data.

Combining both layers

encryption:
  volume:
    enabled: true
    passphrase: "volume-layer-passphrase"
  innodb:
    enabled: true

This provides:

Volume layer: everything on the disk is encrypted (data, binlogs, temporary files, logs, the keyring volume too).
InnoDB layer: MySQL’s own files are encrypted with keys MySQL holds, so the data stays encrypted even to someone who can read the decrypted block device.

Point-in-time recovery (PITR)

The chart provides two complementary recovery mechanisms beyond full backups.

Binlog shipping to S3

Binary log shipping uploads completed binary log files to S3-compatible storage on a schedule. Combined with a full backup, this enables recovery to any specific second.

pitr:
  binlogShipping:
    enabled: true
    schedule: "*/15 * * * *"    # every 15 minutes
    # Uses backup.s3 credentials by default, or specify separate:
    s3:
      prefix: "binlogs"        # S3 path prefix

How it works:

A CronJob runs on the configured schedule.
It connects to the MySQL primary, flushes binary logs, and identifies completed log files.
Each completed binlog is compressed (gzip) and uploaded to S3.
Already-shipped binlogs are skipped (idempotent via S3 HEAD check).

Recovery workflow:

# 1. Restore the most recent full backup
#    (from the backup CronJob's S3 output)

# 2. Download binlogs created after the full backup
aws s3 ls s3://my-bucket/binlogs/<release-name>/

# 3. Replay binlogs up to the target timestamp
mysqlbinlog --stop-datetime="2026-04-15 14:30:00" \
  mysql-bin.000042.gz mysql-bin.000043.gz | mysql -u root -p

# Result: database state at exactly 2026-04-15 14:30:00

S3 credentials default to the backup.s3 settings. To ship binlogs to a separate bucket:

pitr:
  binlogShipping:
    enabled: true
    s3:
      accessKey: "AKIA..."
      secretKey: "..."
      host: "s3.eu-west-1.amazonaws.com"
      bucket: "my-binlog-archive"
      region: "eu-west-1"

Longhorn volume snapshots

Longhorn RecurringJob snapshots (or backups) of the instance’s volumes, in addition to the instance’s own dumps and binlog shipping.

pitr:
  volumeSnapshots:
    enabled: true
    task: snapshot              # snapshot (in-volume) or backup (off-cluster, Longhorn backup target)
    schedule: "0 */6 * * *"     # every 6 hours
    retain: 24                  # keep 24 (~6 days)
    namespace: longhorn-system  # where Longhorn reads RecurringJobs

How it works:

A Longhorn RecurringJob is created in namespace (Longhorn reads RecurringJobs from its own namespace only). Its name, which is also its volume group, is <first 27 characters of namespace-fullname-task>-<12 hex of sha256("<namespace>/<release>/<task>")>: at most 40 characters (Longhorn’s limit) and unique across tenants, because / cannot occur in a namespace or release name. Its labels carry the release namespace, instance and task in full.
A post-install/post-upgrade hook (<fullname>-volume-labeler, distroless kubectl, a PVC-only Role) labels the release’s PVCs with recurring-job.longhorn.io/source=enabled, the job’s group and default, and Longhorn syncs those labels onto the volumes. Keeping default keeps the volumes in the cluster’s own daily backup.
Snapshots are taken on the schedule with zero downtime; older ones beyond retain are cleaned up.

Install and upgrade with helm --wait: the hook runs once, after the release is ready, and labels the PVCs that exist then. Installing needs permission to create a RecurringJob in namespace. Disabling removes the RecurringJob; the PVC labels stay, harmlessly.

snapshot copies live inside the volume, on the same disks: they undo a bad write, but do not survive losing the volume. backup sends an off-cluster copy to the cluster’s Longhorn backup target.

Recovery workflow:

Identify the target snapshot in the Longhorn UI or CLI.
Restore the PVC from the snapshot (Longhorn’s built-in restore).
Optionally replay binlogs from S3 to recover changes after the snapshot.

Recovery strategy comparison

Method	Granularity	Recovery speed	Storage cost	Data location
Full backup (mysqldump)	Daily (or per schedule)	Slow (full import)	Low (compressed SQL)	S3 (off-cluster)
Binlog shipping	Per-second	Medium (replay logs)	Medium (compressed binlogs)	S3 (off-cluster)
Volume snapshots	Per-schedule (e.g., 6h)	Fast (block-level restore)	High (Longhorn storage)	Local (same cluster)

Recommended production setup: Enable all three for comprehensive coverage:

Full backups for disaster recovery (daily, off-site).
Binlog shipping for precise point-in-time recovery (every 15 min, off-site).
Volume snapshots for fast operational recovery (every 6h, local).

MySQL version

The chart defaults to MySQL 8.4 LTS. To use a different 8.4 release, set its tag:

image:
  tag: "<8.4 release>"

MySQL 8.4 LTS is the recommended and tested version. The chart uses GTID replication and Group Replication features available in MySQL 8.4+.

Examples

Minimal standalone (no replication)

mode: "standard"
mysql:
  rootPassword: "my-root-password"
  database: "myapp"
  user: "myapp"
  password: "myapp-password"
storage:
  data:
    size: 20Gi

Standard mode with read replicas

mode: "standard"
standard:
  readReplicas: 2
mysql:
  rootPassword: "my-root-password"
  database: "myapp"
  user: "myapp"
  password: "myapp-password"
  config:
    max_connections: 300
    innodb_buffer_pool_size: "1G"
storage:
  data:
    size: 50Gi
  binlog:
    size: 10Gi
resources:
  limits:
    cpu: "2"
    memory: 2Gi
  requests:
    cpu: 500m
    memory: 1Gi

HA Cluster (production)

mode: "ha"
ha:
  members: 3
mysql:
  rootPassword: "my-root-password"
  database: "myapp"
  user: "myapp"
  password: "myapp-password"
  config:
    max_connections: 500
    innodb_buffer_pool_size: "2G"
    innodb_log_file_size: "256M"
storage:
  data:
    size: 100Gi
  binlog:
    size: 20Gi
resources:
  limits:
    cpu: "4"
    memory: 4Gi
  requests:
    cpu: "1"
    memory: 2Gi
podDisruptionBudget:
  enabled: true
  minAvailable: 2
backup:
  enabled: true
  schedule: "0 2 * * *"
  s3:
    existingSecret: backup-s3-credentials
metrics:
  enabled: true

HA with TLS, encryption and PITR (full production)

mode: "ha"
ha:
  members: 3
mysql:
  rootPassword: "my-root-password"
  database: "myapp"
  user: "myapp"
  password: "myapp-password"
  config:
    max_connections: 500
    innodb_buffer_pool_size: "2G"
storage:
  data:
    size: 100Gi
  binlog:
    size: 20Gi
tls:
  enabled: true
  requireSecureTransport: true
encryption:
  volume:
    enabled: true
    passphrase: "my-luks-passphrase-keep-this-safe"
  innodb:
    enabled: true
backup:
  enabled: true
  schedule: "0 2 * * *"
  s3:
    existingSecret: backup-s3-credentials
pitr:
  binlogShipping:
    enabled: true
    schedule: "*/15 * * * *"
    # Uses backup.s3 credentials by default
  volumeSnapshots:
    enabled: true
    schedule: "0 */6 * * *"
    retain: 24
metrics:
  enabled: true
podDisruptionBudget:
  enabled: true
  minAvailable: 2

With web client (phpMyAdmin)

mode: "standard"
mysql:
  rootPassword: "my-root-password"
  database: "myapp"
client:
  enabled: true
  ingress:
    host: "db-myapp.example.com"
  ipAllowlist:
    - "203.0.113.0/24"
    - "198.51.100.42/32"

Access at the configured hostname. Log in with the MySQL root credentials.

With network policy

NetworkPolicy is enabled by default. The chart’s policy is additive on top of any namespace-level default-deny — it punches targeted holes for MySQL pods. Three allow channels (see docs/network-policy.md for full reference):

Channel	How to set
Same-namespace (always on when enabled)	Built-in — no config needed
Ingress controller (Kong)	`allowFromIngressController.{enabled,namespace,podLabels}`
Arbitrary peers (cross-namespace, IP blocks, labelled pods)	`allowFrom[]`

networkPolicy:
  enabled: true

  # Enable when MySQL is fronted by Kong (e.g. TCPIngress for external clients)
  allowFromIngressController:
    enabled: true
    namespace: kong                       # Kong's namespace in your cluster
    podLabels:
      app.kubernetes.io/name: kong

  # Cross-namespace tenants, monitoring pods, IP blocks etc.
  allowFrom:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: monitoring
    - podSelector:
        matchLabels:
          app: my-cross-ns-app

To opt out entirely (e.g. cluster CNI without NetworkPolicy support):

networkPolicy:
  enabled: false

Parameters

Global

Parameter	Description	Default
`mode`	Deployment mode: `standard` or `ha`	`standard`
`clusterDomain`	Kubernetes cluster DNS domain used to compose Service FQDNs. Set it if your cluster does not use `cluster.local`.	`cluster.local`
`image.repository`	MySQL image repository	`mysql`
`image.tag`	MySQL image tag (defaults to appVersion)	Set in each release
`image.pullPolicy`	Image pull policy	`IfNotPresent`
`imagePullSecrets`	Image pull secrets	`[{name: 1x-ax}]`

MySQL

Parameter	Description	Default
`mysql.rootPassword`	Root password (auto-generated if empty)	`""`
`mysql.database`	Database to create on first boot	`""`
`mysql.user`	Application user to create	`""`
`mysql.password`	Application user password	`""`
`mysql.replication.user`	Replication user name	`replicator`
`mysql.replication.password`	Replication user password (auto-generated if empty)	`""`
`mysql.clone.user`	Clone user name (BACKUP_ADMIN + CLONE_ADMIN) — see clone-based-seeding.md	`clone_user`
`mysql.clone.password`	Clone user password (auto-generated if empty; preserved across upgrade)	`""`
`mysql.databases`	Multi-DB schema (see multi-database.md)	`[]`
`mysql.selfHealingReseed.enabled`	Self-healing sidecar for Standard mode replicas (see self-healing-reseed.md)	`true`
`mysql.selfHealingReseed.intervalSeconds`	Sidecar polling interval	`30`
`mysql.selfHealingReseed.resources`	Sidecar resource limits	`cpu 50m / mem 64Mi` limits
`mysql.existingSecret`	Name of existing Secret with credentials	`""`
`mysql.config`	MySQL variables rendered into custom.cnf	See values.yaml
`mysql.extraConfig`	Raw lines appended to custom.cnf	`""`
`mysql.logging.slowQuery`	Enable slow query log	`false`
`mysql.logging.slowQueryTime`	Slow query threshold (seconds)	`2`
`mysql.logging.generalLog`	Enable general query log	`false`
`mysql.logging.errorLog`	Enable error log	`true`

Standard mode

Parameter	Description	Default
`standard.readReplicas`	Number of read replicas (0 = primary only)	`0`

HA mode

Parameter	Description	Default
`ha.members`	Number of GR members (minimum 3 enforced)	`3`
`ha.groupName`	GR group name UUID (auto-generated if empty)	`""`

Storage

Parameter	Description	Default
`storage.data.storageClass`	Storage class for data PVC	`longhorn`
`storage.data.size`	Data PVC size	`10Gi`
`storage.data.accessModes`	Data PVC access modes	`[ReadWriteOnce]`
`storage.binlog.storageClass`	Storage class for binlog PVC	`longhorn`
`storage.binlog.size`	Binlog PVC size	`5Gi`
`storage.binlog.accessModes`	Binlog PVC access modes	`[ReadWriteOnce]`

Deployment

Parameter	Description	Default
`resources.limits.cpu`	CPU limit	`1`
`resources.limits.memory`	Memory limit	`1Gi`
`resources.requests.cpu`	CPU request	`250m`
`resources.requests.memory`	Memory request	`512Mi`
`terminationGracePeriodSeconds`	Shutdown grace period	`60`
`podDisruptionBudget.enabled`	Enable PDB	`false`
`podDisruptionBudget.minAvailable`	Minimum available pods	`1`

Backup

Parameter	Description	Default
`backup.enabled`	Enable backup CronJob	`false`
`backup.schedule`	Cron schedule	`0 2 * * *`
`backup.image`	Backup container image	Set in each release
`backup.s3.accessKey`	S3 access key	`""`
`backup.s3.secretKey`	S3 secret key	`""`
`backup.s3.host`	S3 host	`""`
`backup.s3.bucket`	S3 bucket	`""`
`backup.s3.region`	S3 region	`""`
`backup.s3.existingSecret`	Existing Secret for S3 credentials	`""`
`backup.secondaryS3.enabled`	Enable secondary S3 target	`false`
`backup.secondaryS3.accessKey`	Secondary S3 access key	`""`
`backup.secondaryS3.secretKey`	Secondary S3 secret key	`""`
`backup.secondaryS3.host`	Secondary S3 host	`""`
`backup.secondaryS3.bucket`	Secondary S3 bucket	`""`
`backup.secondaryS3.region`	Secondary S3 region	`""`
`backup.secondaryS3.existingSecret`	Existing Secret for secondary S3	`""`

Metrics

Parameter	Description	Default
`metrics.enabled`	Enable mysqld-exporter sidecar	`false`
`metrics.image.repository`	Exporter image	`prom/mysqld-exporter`
`metrics.image.tag`	Exporter image tag	Set in each release
`metrics.port`	Exporter port	`9104`

Web client

Parameter	Description	Default
`client.enabled`	Enable phpMyAdmin web client	`false`
`client.image.repository`	phpMyAdmin image repository	`phpmyadmin/phpmyadmin`
`client.image.tag`	phpMyAdmin image tag	Set in each release
`client.uploadLimit`	Max file upload size for SQL imports (PHP `upload_max_filesize` + `post_max_size`)	`128M`
`client.maxExecutionTime`	Max PHP execution time in seconds (for long imports/exports)	`600`
`client.memoryLimit`	Max PHP memory per request (for large result sets/exports)	`512M`
`client.resources`	phpMyAdmin resource limits/requests	See values.yaml
`client.ingress.className`	Ingress class	`kong`
`client.ingress.host`	Hostname (required when client is enabled)	`""`
`client.ingress.certIssuer`	cert-manager ClusterIssuer name	`letsencrypt-prod`
`client.ipAllowlist`	IP/CIDR allowlist for access	`[]`
`client.ttl`	Auto-expiry duration (informational)	`""`

TLS

Parameter	Description	Default
`tls.enabled`	Enable TLS for MySQL connections	`false`
`tls.requireSecureTransport`	Reject plaintext connections (require TLS)	`false`
`tls.existingSecret`	Existing TLS Secret (keys: `ca.crt`, `tls.crt`, `tls.key`)	`""`
`tls.selfSigned.validity`	Certificate validity in days	`3650`
`tls.selfSigned.caSubject`	CA certificate subject	`/CN=MySQL CA`

Encryption

Parameter	Description	Default
`encryption.volume.enabled`	Enable Longhorn LUKS volume encryption	`false`
`encryption.volume.passphrase`	LUKS passphrase (auto-generated if empty)	`""`
`encryption.volume.existingSecret`	Existing Secret for LUKS passphrase	`""`
`encryption.volume.storageClass.reclaimPolicy`	Encrypted StorageClass reclaim policy	`Delete`
`encryption.volume.storageClass.numberOfReplicas`	Longhorn replica count	`3`
`encryption.volume.storageClass.dataLocality`	Longhorn data locality	`disabled`
`encryption.innodb.enabled`	InnoDB encryption at rest (tables, redo/undo logs, binlogs) with a per-pod keyring volume	`false`
`encryption.innodb.keyring.size`	Keyring volume size per pod	`64Mi`
`encryption.innodb.keyring.storageClass`	Keyring volume StorageClass (defaults to `storage.data.storageClass`)	`""`

PITR

Parameter	Description	Default
`pitr.localBinlogRetentionSeconds`	How long the server keeps its own binary logs (`binlog_expire_logs_seconds`). Keep `1209600` (14 days) on a `backup.layout: legacy` release, unless its binlog shipping is monitored	`604800`
`pitr.binlogShipping.enabled`	Enable binlog shipping to S3	`false`
`pitr.binlogShipping.schedule`	Cron schedule for binlog shipping	`/15 * * *`
`pitr.binlogShipping.s3.accessKey`	S3 access key (falls back to backup.s3)	`""`
`pitr.binlogShipping.s3.secretKey`	S3 secret key	`""`
`pitr.binlogShipping.s3.host`	S3 host	`""`
`pitr.binlogShipping.s3.bucket`	S3 bucket	`""`
`pitr.binlogShipping.s3.region`	S3 region	`""`
`pitr.binlogShipping.s3.prefix`	S3 path prefix for binlog files	`binlogs`
`pitr.binlogShipping.s3.existingSecret`	Existing Secret for S3 credentials	`""`
`pitr.volumeSnapshots.enabled`	Enable Longhorn volume snapshots	`false`
`pitr.volumeSnapshots.schedule`	Cron schedule for snapshots	`0 /6 * *`
`pitr.volumeSnapshots.retain`	Number of snapshots to retain	`24`
`pitr.volumeSnapshots.task`	`snapshot` (in-volume) or `backup` (off-cluster)	`snapshot`
`pitr.volumeSnapshots.concurrency`	Volumes processed at once	`1`
`pitr.volumeSnapshots.namespace`	Namespace of the RecurringJob (Longhorn’s)	`longhorn-system`
`pitr.volumeSnapshots.labeler.image`	Image of the PVC-labelling hook	Set in each release

Network policy

Parameter	Description	Default
`networkPolicy.enabled`	Render a NetworkPolicy (default = allow same-namespace ingress)	`true`
`networkPolicy.allowFromIngressController.enabled`	Allow ingress from the cluster’s ingress controller (e.g. Kong)	`false`
`networkPolicy.allowFromIngressController.namespace`	Namespace where the ingress controller pods live	`"kong"`
`networkPolicy.allowFromIngressController.podLabels`	Selector for ingress controller pods (ANDed with namespace)	`{app.kubernetes.io/name: kong}`
`networkPolicy.allowFrom`	Arbitrary additional sources (full NetworkPolicyPeer schema)	`[]`

PITR — Restore (see `docs/pitr-restore.md`)

Parameter	Description	Default
`pitr.restore.enabled`	Render a one-shot Job that restores into `targetHost`	`false`
`pitr.restore.targetDatetime`	Point in time to recover to (UTC, `YYYY-MM-DD HH:MM:SS`)	`""`
`pitr.restore.targetHost`	Staging MySQL primary service. Render-time refused if it is this release’s primary, short or qualified.	`""`
`pitr.restore.targetPort`	MySQL port on `targetHost`	`3306`
`pitr.restore.id`	Platform layout: unique per restore (1-20 lowercase letters/digits); names the Job `<release>-restore-<id>`	`""`
`pitr.restore.targetDatabases`	Platform layout: the databases to restore	`[]`
`pitr.restore.platformResources`	Platform layout: the restore container’s requests and limits	100m/256Mi requests, 1/1Gi limits
`pitr.restore.targetDatabase`	Legacy layout: database to restore (defaults to `mysql.database`)	`""`
`pitr.restore.targetRootPasswordSecret`	Secret holding the root password on `targetHost` (defaults to `<targetHost>`)	`""`
`pitr.restore.targetRootPasswordKey`	Key inside that Secret	`"root-password"`
`pitr.restore.s3BackupPrefix`	S3 prefix for full backups (defaults to `<targetDatabase>/`)	`""`
`pitr.restore.s3BinlogPrefix`	S3 prefix for shipped binlogs (defaults to `<binlogShipping.s3.prefix>/<release>/`)	`""`
`pitr.restore.image`	Restore image (mysql-client 8.4 + mysqlbinlog + s3cmd)	Set in each release
`pitr.restore.keepWorkDir`	Leave `/tmp/pitr-restore` on Job exit (debug)	`false`
`pitr.restore.extraEnv`	Extra env vars for the restore container	`[]`
`pitr.restore.resources`	Resource limits	`{}`

Upgrading

helm upgrade my-db charts/mysql -f my-values.yaml

Passwords are preserved across upgrades via Helm lookup.
Storage can be expanded by increasing storage.data.size or storage.binlog.size (requires CSI driver support for volume expansion).
Storage cannot be reduced. This is a Kubernetes limitation on PersistentVolumeClaims.
Mode switching (standard to HA or vice versa) is not supported as an in-place upgrade. It requires a fresh deployment with data migration.
Adding read replicas (standard mode) is supported via helm upgrade with an increased standard.readReplicas value. New replicas will bootstrap from the primary via GTID.
Adding HA members is supported via helm upgrade with an increased ha.members value. New members join the group automatically.

Testing

helm test my-db

Runs a mysqladmin ping against the primary service to verify connectivity.

MySQL Helm Chart

Prerequisites

Architecture

Standard Mode

Pros

Cons

When to use Standard mode

HA Cluster Mode

How failover works

Pros

Cons

When to use HA Cluster mode

Installation

MySQL Configuration

Longhorn storage defaults

Logging

Storage

Increasing storage

Secrets

Seeding & self-healing

Databases, users and the reconcile Job

Backups

Secondary S3 target

Metrics

Web client (phpMyAdmin)

Hostname

Security

External access (TCPIngress by port)

Network policy egress

TLS (encrypted connections)

How it works

CA certificate for client applications

Bring your own certificates

TLS modes

Self-signed certificate SANs

Connecting your application over TLS

Step 1: Mount the CA certificate

Step 2: Configure your MySQL driver

Encryption at rest

Volume-level encryption (Longhorn LUKS)

MySQL InnoDB encryption

Combining both layers

Point-in-time recovery (PITR)

Binlog shipping to S3

Longhorn volume snapshots

Recovery strategy comparison

MySQL version

Examples

Minimal standalone (no replication)

Standard mode with read replicas

HA Cluster (production)

HA with TLS, encryption and PITR (full production)

With web client (phpMyAdmin)

With network policy

Parameters

Global

MySQL

Standard mode

HA mode

Storage

Deployment

Backup

Metrics

Web client

TLS

Encryption

PITR

Network policy

PITR — Restore (see docs/pitr-restore.md)

Upgrading

Testing

PITR — Restore (see `docs/pitr-restore.md`)