Running OpsAPI on Kubernetes: the Helm chart, piece by piece

OpsAPI ships as one container image, bwalia/opsapi, and one Helm chart. The image is OpenResty with the Lapis application inside it; there's no separate app server. The chart turns that image into a release: pods, a Service, an Ingress, the secrets the pods need, a database, the migration step and the backups. This note goes through each piece in the order you meet it when something goes wrong.

What the chart creates

Secrets never live in git

The values file names a vault path and a ClusterSecretStore; it never holds a password. The ExternalSecret extracts every key at that path into a Kubernetes Secret, and its template adds one more key, LAPIS_CONFIG_LUA_FILE, which becomes the pod’s config.lua. That file reads everything from the environment rather than having values templated into it:

config("production", {
  secret = os.getenv("JWT_SECRET_KEY"),
  postgres = {
    host = "<release-db>.<namespace>.svc.cluster.local",
    user = os.getenv("POSTGRES_USER"),
    password = os.getenv("POSTGRES_PASSWORD"),
    database = os.getenv("POSTGRES_DB") or "opsapi",
    ssl = os.getenv("POSTGRES_SSL") == "true"
  },
})

The pod has a reloader.stakater.com/auto annotation. When the vault value changes, External Secrets updates the Secret within a minute and Reloader rolls the pods, so rotating a key is a vault write, not a deploy.

Migrations: once per deploy, before the pods roll

Migrations used to run in every pod’s start-up hook. With two replicas, both pods could see migration N as not applied and race to run it. They now run in a Job with these annotations:

metadata:
  annotations:
    helm.sh/hook: pre-upgrade
    helm.sh/hook-weight: "-5"
    helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
spec:
  backoffLimit: 2
  activeDeadlineSeconds: 900

Helm waits for the Job before touching the Deployment, so a failed migration stops the rollout instead of shipping pods against a half-migrated schema. Which migrations run depends on PROJECT_CODE: a release built for core_only creates only the core tables.

Rollouts that don't drop requests

• RollingUpdate with maxUnavailable: 0 and maxSurge: 1: a new pod must pass /ready before an old one goes.
• preStop: the old pod sleeps a few seconds (default 8) so the Service and Ingress drop its endpoint, then sends nginx SIGQUIT to finish in-flight requests. PID 1 ignores SIGTERM, so without this every stop waited out the grace period and was killed mid-request.
• /ready checks the database, that every plugin loaded, and, when chat requires it, that the Redis subscriber is connected. /live only says the process is up, so a database blip never restarts pods.

Placement

Values carry a nodeSelector by label rather than a single host name, for example a network region and kubernetes.io/arch: amd64. Pinning to one node kept failing in practice: a node hit its pod limit, another went NotReady, and pinned pods sat Pending. Selecting a pool lets the scheduler route around a bad node. The image is amd64, so arm64 nodes are excluded.

The database

Each release gets its own PostgreSQL StatefulSet using the pgvector image, because the tax classifier stores vector(384) embeddings. An init ConfigMap runs CREATE EXTENSION IF NOT EXISTS vector on a fresh volume, so a rebuilt volume heals itself before the first migration. The password comes from the same vault path the app reads.

readinessProbe:
  exec:
    command: ["sh", "-c", "pg_isready -U pguser -d <db> -h 127.0.0.1"]

Name the database in pg_isready. Without -d it connects to a database named after the user, and the Postgres log fills with “database does not exist” every probe.

Backups, and switching them off

Two CronJobs (0 * * * * and 0 4 * * *) pg_dump the release’s database and upload the gzip to object storage. A release with no object storage would fail every hour with a missing-key error, so the chart has an opt-out:

backup:
  enabled: false   # unset = on, so existing releases render the same

The public demo uses it. A production release should keep backups on and test a restore.

Lesson: a start-up hook isn't a pre-start hook

The chart’s postStart hook copies config.lua from the mounted Secret into the app directory. Kubernetes runs that hook alongside the container’s entrypoint, not before it. On a slow node the copy usually won. On a fast node nginx started first, its workers loaded the image’s default configuration (database on localhost) and the pod stayed unready with “connection refused” until someone restarted it.

if [ -s /app/logs/nginx.pid ]; then
  /usr/local/openresty/nginx/sbin/nginx -p /app/ -c nginx.conf.compiled -s reload \
    || echo "[bootstrap] WARN: nginx reload failed; workers may hold a stale config"
fi

The reload is non-fatal on purpose: on nodes whose AppArmor profile denies the signal, the pod keeps the old behaviour instead of crash-looping. The general rule holds for any image: if a hook writes configuration, either make the entrypoint wait for it or reload after it.

Checklist for a new release

1. Write the release’s keys to its vault path (database password, JWT_SECRET_KEY, encryption key and IV, any OAuth and mail settings).
2. Apply the database StatefulSet and wait for it to be ready.
3. helm upgrade --install with the release’s values file, then run the same upgrade again so the migrate Job runs.
4. Restart the Deployment and check /ready returns {"ready":true}.
5. Register the hostname at the edge (see the edge technote).
6. Decide on backups: object storage and a restore test, or an explicit backup.enabled: false.