fix(deploy): load production environment before backup
deploy.sh symlinked /opt/job-tracker/shared/.env for docker compose but never loaded it into its own shell. Its own decisions therefore ran against an empty environment: DATABASE_PROVIDER fell back to sqlite on a MariaDB host, so the deploy tarred the data volume, printed "Backup verified" and continued with no database dump. The operator saw a green backup line and a new file in the backups directory, and had no restore point. Load the shared env before any decision. Parsed line by line rather than sourced, because a compose .env is not a shell script and an unquoted value containing spaces would execute as a command. Values already in the environment win, so CI-provided APP_VERSION and friends still override the file. No value is echoed. Remove the sqlite default. DATABASE_PROVIDER must be stated; missing or unrecognised aborts the deploy. Validate deployment configuration before the backup, and so before anything is built, stopped or replaced: the connection string when the provider needs one, AI_SERVICE_TOKEN (compose declares it with :?) and AUTH_JWT_KEY (the backend throws on a blank key). Names in the output, never values. Verify each backup against its own format. A dump must be valid gzip, contain CREATE TABLE, and carry the "Dump completed" trailer, so a dump that died partway through is rejected. An archive must contain jobtracker.db. A tar can no longer pass the dump check. Also resolve the SQLite volume by its project-prefixed name and fail if absent. The bare jobtracker_data name would have silently created an empty volume and backed that up -- the same class of bug, found while testing this fix. Verified against a seeded MariaDB 11 container and real Docker volumes: provider selection, all four validation failures, both backup formats and their failure paths, truncated and trailer-stripped dumps, and zero secret occurrences across every test's output. Docs updated for the drift: deploy/README.md, deploy/first-production- deployment.md, docs/release-candidate-review.md (B1 closed) and .env.example, which now names the two database variables. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
+57
-7
@@ -109,22 +109,72 @@ hat links open the correct job/tab
|
||||
only the rollback. Restore the database only if the data itself is wrong or lost — it discards
|
||||
everything written since the dump.
|
||||
|
||||
## Environment loading
|
||||
|
||||
`deploy/deploy.sh` **loads `/opt/job-tracker/shared/.env` into its own shell** before it decides
|
||||
anything. The symlink it creates in the checkout is for docker compose, which reads `.env` itself;
|
||||
the script needs the values too, to pick the right backup and to check its own configuration.
|
||||
|
||||
- Parsed line by line, not `source`d — a compose `.env` is not a shell script, so an unquoted value
|
||||
containing spaces would execute as a command.
|
||||
- **Variables already set in the environment win**, so CI-provided `APP_VERSION`, `APP_COMMIT_SHA`
|
||||
and `APP_BUILD_STAMP` still override the file.
|
||||
- No value is ever echoed. Error messages name variables, never their contents.
|
||||
|
||||
This was added on 2026-07-19. Before it, the script read an empty environment: `DATABASE_PROVIDER`
|
||||
fell back to `sqlite` on a MariaDB host, so the deploy tarred the data volume, printed
|
||||
`Backup verified`, and continued with no database dump at all. See
|
||||
`docs/release-candidate-review.md` (B1).
|
||||
|
||||
## Required production variables
|
||||
|
||||
`validate_deploy_config` runs **before** the backup, and therefore before anything is built, stopped
|
||||
or replaced. A missing variable aborts the deploy while the running stack is still untouched.
|
||||
|
||||
| Variable | Required | Why it is checked here |
|
||||
|---|---|---|
|
||||
| `DATABASE_PROVIDER` | **Always** — no default | Selects the backup. Guessing it wrong backs up the wrong database and reports success |
|
||||
| `JOBTRACKER_CONNECTION_STRING` | When provider is `mariadb`/`mysql` | Without it there is no way to dump the database |
|
||||
| `AI_SERVICE_TOKEN` | Always | `docker-compose.yml` declares it with `:?`; missing it kills the stack *after* the images are built |
|
||||
| `AUTH_JWT_KEY` | Always | Compose sets `Auth__Require=true`, and the backend throws at startup on a blank key — after the containers have been replaced |
|
||||
| `APP_PUBLIC_BASE_URL` | Optional | If unset the post-deploy public smoke check is skipped, and the script says so rather than skipping silently |
|
||||
|
||||
`DATABASE_PROVIDER` deliberately has **no default**. An unset value used to mean "sqlite"; it now
|
||||
means "stop and tell me".
|
||||
|
||||
## Backup creation
|
||||
|
||||
`deploy/deploy.sh` takes a backup **before** it builds, stops or replaces anything, and **aborts the
|
||||
deploy if the backup fails**. Nothing else in the deploy runs without a restore point.
|
||||
|
||||
- **Location:** `/opt/job-tracker/backups` — override with `BACKUP_DIR`.
|
||||
- **Naming:** `jobtracker-<database>-<UTC timestamp>.sql.gz`, e.g.
|
||||
`jobtracker-jobtracker-20260719T153759Z.sql.gz`. The timestamp makes every file unique, so a deploy
|
||||
never overwrites an earlier backup.
|
||||
- **SQLite deployments** (`DATABASE_PROVIDER` unset or `sqlite`) get the data volume instead:
|
||||
`jobtracker-sqlite-<UTC timestamp>.tar.gz`.
|
||||
- **Selection:** driven solely by `DATABASE_PROVIDER`, which must be set.
|
||||
- `mariadb` / `mysql` → SQL dump, `jobtracker-<database>-<UTC timestamp>.sql.gz`, e.g.
|
||||
`jobtracker-jobtracker-20260719T153759Z.sql.gz`
|
||||
- `sqlite` → data volume archive, `jobtracker-sqlite-<UTC timestamp>.tar.gz`
|
||||
- anything else → the deploy stops
|
||||
- **The filename tells you which path ran.** If you expect a MariaDB deploy and find a
|
||||
`jobtracker-sqlite-*.tar.gz`, the environment is wrong — that is the exact failure this check exists
|
||||
to make visible.
|
||||
- **Naming:** the UTC timestamp makes every file unique, so a deploy never overwrites an earlier backup.
|
||||
- **Credentials** come from `JOBTRACKER_CONNECTION_STRING` and are passed via `MYSQL_PWD`, never on the
|
||||
command line, so they cannot appear in the process list or the deploy log.
|
||||
- **Compression:** gzip. A small database compresses to a few KB.
|
||||
- **Verification:** the script rejects a dump that is empty or missing `CREATE TABLE`, because a
|
||||
truncated file that *looks* like a restore point is worse than none.
|
||||
- **SQLite volume resolution:** compose prefixes volume names with the project name, so the script
|
||||
resolves `<project>_jobtracker_data` and **fails if that volume does not exist**. Naming the bare
|
||||
volume would silently create an empty one and back *that* up.
|
||||
|
||||
### Verification — each provider gets the check that proves its own format
|
||||
|
||||
A backup that exists but is empty, truncated, or the wrong *kind* is worse than none, because it looks
|
||||
like a restore point.
|
||||
|
||||
| Provider | Checks |
|
||||
|---|---|
|
||||
| MariaDB | non-empty; valid gzip; contains `CREATE TABLE`; contains the `Dump completed` trailer that `mariadb-dump` writes last, so a dump that died partway through is rejected |
|
||||
| SQLite | non-empty; valid gzip; the archive actually contains `jobtracker.db` |
|
||||
|
||||
A failed check deletes the file rather than leaving something that looks like a backup.
|
||||
|
||||
### Taking one by hand
|
||||
|
||||
|
||||
Reference in New Issue
Block a user