fix(deploy): load production environment before backup

deploy.sh symlinked /opt/job-tracker/shared/.env for docker compose but
never loaded it into its own shell. Its own decisions therefore ran
against an empty environment: DATABASE_PROVIDER fell back to sqlite on a
MariaDB host, so the deploy tarred the data volume, printed "Backup
verified" and continued with no database dump. The operator saw a green
backup line and a new file in the backups directory, and had no restore
point.

Load the shared env before any decision. Parsed line by line rather than
sourced, because a compose .env is not a shell script and an unquoted
value containing spaces would execute as a command. Values already in
the environment win, so CI-provided APP_VERSION and friends still
override the file. No value is echoed.

Remove the sqlite default. DATABASE_PROVIDER must be stated; missing or
unrecognised aborts the deploy.

Validate deployment configuration before the backup, and so before
anything is built, stopped or replaced: the connection string when the
provider needs one, AI_SERVICE_TOKEN (compose declares it with :?) and
AUTH_JWT_KEY (the backend throws on a blank key). Names in the output,
never values.

Verify each backup against its own format. A dump must be valid gzip,
contain CREATE TABLE, and carry the "Dump completed" trailer, so a dump
that died partway through is rejected. An archive must contain
jobtracker.db. A tar can no longer pass the dump check.

Also resolve the SQLite volume by its project-prefixed name and fail if
absent. The bare jobtracker_data name would have silently created an
empty volume and backed that up -- the same class of bug, found while
testing this fix.

Verified against a seeded MariaDB 11 container and real Docker volumes:
provider selection, all four validation failures, both backup formats
and their failure paths, truncated and trailer-stripped dumps, and zero
secret occurrences across every test's output.

Docs updated for the drift: deploy/README.md, deploy/first-production-
deployment.md, docs/release-candidate-review.md (B1 closed) and
.env.example, which now names the two database variables.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
cesnimda
2026-07-19 18:44:23 +02:00
parent ab53582c71
commit 66b02bcab8
5 changed files with 380 additions and 49 deletions
+57 -7
View File
@@ -109,22 +109,72 @@ hat links open the correct job/tab
only the rollback. Restore the database only if the data itself is wrong or lost — it discards
everything written since the dump.
## Environment loading
`deploy/deploy.sh` **loads `/opt/job-tracker/shared/.env` into its own shell** before it decides
anything. The symlink it creates in the checkout is for docker compose, which reads `.env` itself;
the script needs the values too, to pick the right backup and to check its own configuration.
- Parsed line by line, not `source`d — a compose `.env` is not a shell script, so an unquoted value
containing spaces would execute as a command.
- **Variables already set in the environment win**, so CI-provided `APP_VERSION`, `APP_COMMIT_SHA`
and `APP_BUILD_STAMP` still override the file.
- No value is ever echoed. Error messages name variables, never their contents.
This was added on 2026-07-19. Before it, the script read an empty environment: `DATABASE_PROVIDER`
fell back to `sqlite` on a MariaDB host, so the deploy tarred the data volume, printed
`Backup verified`, and continued with no database dump at all. See
`docs/release-candidate-review.md` (B1).
## Required production variables
`validate_deploy_config` runs **before** the backup, and therefore before anything is built, stopped
or replaced. A missing variable aborts the deploy while the running stack is still untouched.
| Variable | Required | Why it is checked here |
|---|---|---|
| `DATABASE_PROVIDER` | **Always** — no default | Selects the backup. Guessing it wrong backs up the wrong database and reports success |
| `JOBTRACKER_CONNECTION_STRING` | When provider is `mariadb`/`mysql` | Without it there is no way to dump the database |
| `AI_SERVICE_TOKEN` | Always | `docker-compose.yml` declares it with `:?`; missing it kills the stack *after* the images are built |
| `AUTH_JWT_KEY` | Always | Compose sets `Auth__Require=true`, and the backend throws at startup on a blank key — after the containers have been replaced |
| `APP_PUBLIC_BASE_URL` | Optional | If unset the post-deploy public smoke check is skipped, and the script says so rather than skipping silently |
`DATABASE_PROVIDER` deliberately has **no default**. An unset value used to mean "sqlite"; it now
means "stop and tell me".
## Backup creation
`deploy/deploy.sh` takes a backup **before** it builds, stops or replaces anything, and **aborts the
deploy if the backup fails**. Nothing else in the deploy runs without a restore point.
- **Location:** `/opt/job-tracker/backups` — override with `BACKUP_DIR`.
- **Naming:** `jobtracker-<database>-<UTC timestamp>.sql.gz`, e.g.
`jobtracker-jobtracker-20260719T153759Z.sql.gz`. The timestamp makes every file unique, so a deploy
never overwrites an earlier backup.
- **SQLite deployments** (`DATABASE_PROVIDER` unset or `sqlite`) get the data volume instead:
`jobtracker-sqlite-<UTC timestamp>.tar.gz`.
- **Selection:** driven solely by `DATABASE_PROVIDER`, which must be set.
- `mariadb` / `mysql` → SQL dump, `jobtracker-<database>-<UTC timestamp>.sql.gz`, e.g.
`jobtracker-jobtracker-20260719T153759Z.sql.gz`
- `sqlite` data volume archive, `jobtracker-sqlite-<UTC timestamp>.tar.gz`
- anything else → the deploy stops
- **The filename tells you which path ran.** If you expect a MariaDB deploy and find a
`jobtracker-sqlite-*.tar.gz`, the environment is wrong — that is the exact failure this check exists
to make visible.
- **Naming:** the UTC timestamp makes every file unique, so a deploy never overwrites an earlier backup.
- **Credentials** come from `JOBTRACKER_CONNECTION_STRING` and are passed via `MYSQL_PWD`, never on the
command line, so they cannot appear in the process list or the deploy log.
- **Compression:** gzip. A small database compresses to a few KB.
- **Verification:** the script rejects a dump that is empty or missing `CREATE TABLE`, because a
truncated file that *looks* like a restore point is worse than none.
- **SQLite volume resolution:** compose prefixes volume names with the project name, so the script
resolves `<project>_jobtracker_data` and **fails if that volume does not exist**. Naming the bare
volume would silently create an empty one and back *that* up.
### Verification — each provider gets the check that proves its own format
A backup that exists but is empty, truncated, or the wrong *kind* is worse than none, because it looks
like a restore point.
| Provider | Checks |
|---|---|
| MariaDB | non-empty; valid gzip; contains `CREATE TABLE`; contains the `Dump completed` trailer that `mariadb-dump` writes last, so a dump that died partway through is rejected |
| SQLite | non-empty; valid gzip; the archive actually contains `jobtracker.db` |
A failed check deletes the file rather than leaving something that looks like a backup.
### Taking one by hand