fix(deploy): load production environment before backup

deploy.sh symlinked /opt/job-tracker/shared/.env for docker compose but
never loaded it into its own shell. Its own decisions therefore ran
against an empty environment: DATABASE_PROVIDER fell back to sqlite on a
MariaDB host, so the deploy tarred the data volume, printed "Backup
verified" and continued with no database dump. The operator saw a green
backup line and a new file in the backups directory, and had no restore
point.

Load the shared env before any decision. Parsed line by line rather than
sourced, because a compose .env is not a shell script and an unquoted
value containing spaces would execute as a command. Values already in
the environment win, so CI-provided APP_VERSION and friends still
override the file. No value is echoed.

Remove the sqlite default. DATABASE_PROVIDER must be stated; missing or
unrecognised aborts the deploy.

Validate deployment configuration before the backup, and so before
anything is built, stopped or replaced: the connection string when the
provider needs one, AI_SERVICE_TOKEN (compose declares it with :?) and
AUTH_JWT_KEY (the backend throws on a blank key). Names in the output,
never values.

Verify each backup against its own format. A dump must be valid gzip,
contain CREATE TABLE, and carry the "Dump completed" trailer, so a dump
that died partway through is rejected. An archive must contain
jobtracker.db. A tar can no longer pass the dump check.

Also resolve the SQLite volume by its project-prefixed name and fail if
absent. The bare jobtracker_data name would have silently created an
empty volume and backed that up -- the same class of bug, found while
testing this fix.

Verified against a seeded MariaDB 11 container and real Docker volumes:
provider selection, all four validation failures, both backup formats
and their failure paths, truncated and trailer-stripped dumps, and zero
secret occurrences across every test's output.

Docs updated for the drift: deploy/README.md, deploy/first-production-
deployment.md, docs/release-candidate-review.md (B1 closed) and
.env.example, which now names the two database variables.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

.env.example

@@ -1,6 +1,18 @@
 # Copy this file to `.env` (same folder as docker-compose.yml) and fill in values.
 #
 # Used by docker-compose.yml
+#
+# Database. deploy/deploy.sh REQUIRES DATABASE_PROVIDER to be set explicitly and
+# refuses to deploy without it — it selects which backup to take, and guessing it
+# wrong means backing up the wrong database. Use `mariadb` (or `mysql`) for a
+# server deployment, `sqlite` for a single-file local one.
+DATABASE_PROVIDER=sqlite
+# Required when DATABASE_PROVIDER is mariadb/mysql. Ignored for sqlite, which
+# stores its file in the jobtracker_data volume.
+# The host resolves from INSIDE the backend container: 127.0.0.1 means the
+# container, not the Docker host.
+JOBTRACKER_CONNECTION_STRING=
+
 AUTH_JWT_KEY=CHANGE_ME_LONG_RANDOM_SECRET
 AUTH_ADMIN_EMAIL=admin@example.com
 AUTH_ADMIN_PASSWORD=CHANGE_ME_STRONG_PASSWORD

deploy/README.md

@@ -109,22 +109,72 @@ hat links open the correct job/tab
 only the rollback. Restore the database only if the data itself is wrong or lost — it discards
 everything written since the dump.
 
+## Environment loading
+
+`deploy/deploy.sh` **loads `/opt/job-tracker/shared/.env` into its own shell** before it decides
+anything. The symlink it creates in the checkout is for docker compose, which reads `.env` itself;
+the script needs the values too, to pick the right backup and to check its own configuration.
+
+- Parsed line by line, not `source`d — a compose `.env` is not a shell script, so an unquoted value
+  containing spaces would execute as a command.
+- **Variables already set in the environment win**, so CI-provided `APP_VERSION`, `APP_COMMIT_SHA`
+  and `APP_BUILD_STAMP` still override the file.
+- No value is ever echoed. Error messages name variables, never their contents.
+
+This was added on 2026-07-19. Before it, the script read an empty environment: `DATABASE_PROVIDER`
+fell back to `sqlite` on a MariaDB host, so the deploy tarred the data volume, printed
+`Backup verified`, and continued with no database dump at all. See
+`docs/release-candidate-review.md` (B1).
+
+## Required production variables
+
+`validate_deploy_config` runs **before** the backup, and therefore before anything is built, stopped
+or replaced. A missing variable aborts the deploy while the running stack is still untouched.
+
+| Variable | Required | Why it is checked here |
+|---|---|---|
+| `DATABASE_PROVIDER` | **Always** — no default | Selects the backup. Guessing it wrong backs up the wrong database and reports success |
+| `JOBTRACKER_CONNECTION_STRING` | When provider is `mariadb`/`mysql` | Without it there is no way to dump the database |
+| `AI_SERVICE_TOKEN` | Always | `docker-compose.yml` declares it with `:?`; missing it kills the stack *after* the images are built |
+| `AUTH_JWT_KEY` | Always | Compose sets `Auth__Require=true`, and the backend throws at startup on a blank key — after the containers have been replaced |
+| `APP_PUBLIC_BASE_URL` | Optional | If unset the post-deploy public smoke check is skipped, and the script says so rather than skipping silently |
+
+`DATABASE_PROVIDER` deliberately has **no default**. An unset value used to mean "sqlite"; it now
+means "stop and tell me".
+
 ## Backup creation
 
 `deploy/deploy.sh` takes a backup **before** it builds, stops or replaces anything, and **aborts the
 deploy if the backup fails**. Nothing else in the deploy runs without a restore point.
 
 - **Location:** `/opt/job-tracker/backups` — override with `BACKUP_DIR`.
-- **Naming:** `jobtracker-<database>-<UTC timestamp>.sql.gz`, e.g.
+- **Selection:** driven solely by `DATABASE_PROVIDER`, which must be set.
-  `jobtracker-jobtracker-20260719T153759Z.sql.gz`. The timestamp makes every file unique, so a deploy
+  - `mariadb` / `mysql` → SQL dump, `jobtracker-<database>-<UTC timestamp>.sql.gz`, e.g.
-  never overwrites an earlier backup.
+    `jobtracker-jobtracker-20260719T153759Z.sql.gz`
-- **SQLite deployments** (`DATABASE_PROVIDER` unset or `sqlite`) get the data volume instead:
+  - `sqlite` → data volume archive, `jobtracker-sqlite-<UTC timestamp>.tar.gz`
-  `jobtracker-sqlite-<UTC timestamp>.tar.gz`.
+  - anything else → the deploy stops
+- **The filename tells you which path ran.** If you expect a MariaDB deploy and find a
+  `jobtracker-sqlite-*.tar.gz`, the environment is wrong — that is the exact failure this check exists
+  to make visible.
+- **Naming:** the UTC timestamp makes every file unique, so a deploy never overwrites an earlier backup.
 - **Credentials** come from `JOBTRACKER_CONNECTION_STRING` and are passed via `MYSQL_PWD`, never on the
   command line, so they cannot appear in the process list or the deploy log.
 - **Compression:** gzip. A small database compresses to a few KB.
-- **Verification:** the script rejects a dump that is empty or missing `CREATE TABLE`, because a
+- **SQLite volume resolution:** compose prefixes volume names with the project name, so the script
-  truncated file that *looks* like a restore point is worse than none.
+  resolves `<project>_jobtracker_data` and **fails if that volume does not exist**. Naming the bare
+  volume would silently create an empty one and back *that* up.
+
+### Verification — each provider gets the check that proves its own format
+
+A backup that exists but is empty, truncated, or the wrong *kind* is worse than none, because it looks
+like a restore point.
+
+| Provider | Checks |
+|---|---|
+| MariaDB | non-empty; valid gzip; contains `CREATE TABLE`; contains the `Dump completed` trailer that `mariadb-dump` writes last, so a dump that died partway through is rejected |
+| SQLite | non-empty; valid gzip; the archive actually contains `jobtracker.db` |
+
+A failed check deletes the file rather than leaving something that looks like a backup.
 
 ### Taking one by hand
 

deploy/deploy.sh

@@ -19,6 +19,48 @@ if [ ! -L "$ENV_TARGET" ] && [ ! -f "$ENV_TARGET" ]; then
   exit 1
 fi
 
+# ---------------------------------------------------------------------------
+# Load the shared environment into THIS shell.
+#
+# The symlink above is for docker compose, which reads .env itself. The script's
+# own decisions — which backup to take, which variables are missing — ran against
+# an empty environment until this existed, so DATABASE_PROVIDER defaulted to
+# sqlite and a MariaDB host silently got a volume tar instead of a dump, with a
+# "Backup verified" line to match. See docs/release-candidate-review.md (B1).
+#
+# Parsed line by line rather than sourced: a compose .env is not a shell script,
+# so an unquoted value containing spaces would execute as a command under `.`.
+# Values already present in the environment win, so CI-provided APP_VERSION and
+# friends still override the file. No value is ever echoed.
+# ---------------------------------------------------------------------------
+load_env_file() {
+  local file="$1" line key value
+
+  while IFS= read -r line || [ -n "$line" ]; do
+    line="${line%$'\r'}"
+    case "$line" in ''|'#'*) continue ;; esac
+    case "$line" in *=*) ;; *) continue ;; esac
+
+    key="${line%%=*}"
+    value="${line#*=}"
+    key="${key#export }"
+    key="${key//[[:space:]]/}"
+    case "$key" in ''|*[!A-Za-z0-9_]*) continue ;; esac
+
+    case "$value" in
+      \"*\") value="${value#\"}"; value="${value%\"}" ;;
+      \'*\') value="${value#\'}"; value="${value%\'}" ;;
+    esac
+
+    # Already set (CI, or an explicit override on the command line) wins.
+    [ -n "${!key+x}" ] && continue
+    export "$key=$value"
+  done < "$file"
+}
+
+load_env_file "$ENV_TARGET"
+echo "Loaded deployment environment from ${ENV_SOURCE}"
+
 export APP_VERSION="${APP_VERSION:-0.0.0}"
 export APP_COMMIT_SHA="${APP_COMMIT_SHA:-unknown}"
 export APP_BUILD_STAMP="${APP_BUILD_STAMP:-unknown}"
@@ -28,6 +70,74 @@ compose() {
   docker compose "$@"
 }
 
+# ---------------------------------------------------------------------------
+# Configuration validation, before anything is built, stopped or replaced.
+#
+# Everything checked here is fatal later anyway — compose declares AI_SERVICE_TOKEN
+# with `:?`, and the backend throws on a blank Auth__JwtKey. Failing here costs an
+# aborted deploy; failing there costs a half-replaced stack.
+#
+# Names only in the output. Never values.
+# ---------------------------------------------------------------------------
+require_var() {
+  local name="$1" hint="$2"
+
+  if [ -z "${!name:-}" ]; then
+    echo "Missing required deployment variable: ${name}"
+    echo "    ${hint}"
+    echo "    Set it in ${ENV_SOURCE}."
+    return 1
+  fi
+  return 0
+}
+
+validate_deploy_config() {
+  local failed=0
+
+  # Deliberately no default. Guessing this wrong means backing up the wrong
+  # database and reporting success — the exact failure this block exists to stop.
+  if [ -z "${DATABASE_PROVIDER:-}" ]; then
+    echo "Missing required deployment variable: DATABASE_PROVIDER"
+    echo "    Set to 'mariadb' (or 'mysql') for a MariaDB deployment, or 'sqlite'."
+    echo "    There is no default: the wrong value backs up the wrong database."
+    echo "    Set it in ${ENV_SOURCE}."
+    failed=1
+  else
+    case "$(printf '%s' "$DATABASE_PROVIDER" | tr '[:upper:]' '[:lower:]')" in
+      mysql|mariadb)
+        require_var JOBTRACKER_CONNECTION_STRING \
+          "MariaDB connection string. Required to take a database dump." || failed=1
+        ;;
+      sqlite)
+        ;;
+      *)
+        echo "DATABASE_PROVIDER is not a recognised value. Use mariadb, mysql or sqlite."
+        failed=1
+        ;;
+    esac
+  fi
+
+  require_var AI_SERVICE_TOKEN \
+    "Shared secret between backend and ai-service. docker compose refuses to start without it." || failed=1
+
+  # docker-compose.yml sets Auth__Require=true unconditionally, and the backend
+  # throws at startup rather than issuing tokens signed with a blank key.
+  require_var AUTH_JWT_KEY \
+    "JWT signing key. With Auth__Require=true the backend refuses to start without it." || failed=1
+
+  if [ -z "${APP_PUBLIC_BASE_URL:-}" ]; then
+    echo "Note: APP_PUBLIC_BASE_URL is not set — the post-deploy public smoke check will be skipped."
+  fi
+
+  if [ "$failed" -ne 0 ]; then
+    echo "Deployment configuration is incomplete. Nothing was built, stopped or replaced."
+    return 1
+  fi
+
+  echo "Deployment configuration validated (database provider: ${DATABASE_PROVIDER})."
+  return 0
+}
+
 # ---------------------------------------------------------------------------
 # Database backup, taken BEFORE anything is stopped, built or replaced.
 #
@@ -44,27 +154,43 @@ backup_database() {
     return 0
   fi
 
-  local provider="${DATABASE_PROVIDER:-sqlite}"
+  # validate_deploy_config has already established this is set and recognised.
+  local provider
+  provider="$(printf '%s' "$DATABASE_PROVIDER" | tr '[:upper:]' '[:lower:]')"
+
   local stamp
   stamp="$(date -u +%Y%m%dT%H%M%SZ)"
   mkdir -p "$BACKUP_DIR"
 
-  if [ "$provider" != "mysql" ] && [ "$provider" != "mariadb" ]; then
+  if [ "$provider" = "sqlite" ]; then
     # SQLite lives in the jobtracker_data volume. Tar it from a throwaway container
     # so the host needs no sqlite tooling and no knowledge of the volume layout.
+    #
+    # docker compose prefixes volume names with the project name, which defaults to
+    # the directory name. Naming the bare volume here would silently CREATE an empty
+    # one and back that up instead, so resolve it and fail if it is not there.
+    local volume="${COMPOSE_PROJECT_NAME:-$(basename "$PWD")}_jobtracker_data"
+    if ! docker volume inspect "$volume" >/dev/null 2>&1; then
+      echo "SQLite data volume '${volume}' does not exist. Aborting deploy."
+      echo "    Volumes are prefixed with the compose project name; set COMPOSE_PROJECT_NAME if it differs."
+      echo "    Available: $(docker volume ls -q | grep jobtracker_data | tr '\n' ' ')"
+      return 1
+    fi
+
     local target="$BACKUP_DIR/jobtracker-sqlite-${stamp}.tar.gz"
-    echo "Backing up SQLite data volume to ${target}"
+    echo "Backing up SQLite data volume '${volume}' to ${target}"
     # The archive path is built inside the container: passing /backup/... as an argument
     # gets rewritten by MSYS path translation when the script is run from Git Bash.
     if ! docker run --rm \
-        -v jobtracker_data:/data:ro \
+        -v "$volume":/data:ro \
         -v "$BACKUP_DIR":/backup \
         -e ARCHIVE_NAME="$(basename "$target")" \
         alpine:3 sh -c 'tar czf "/backup/$ARCHIVE_NAME" -C /data .'; then
       echo "SQLite volume backup FAILED. Aborting deploy."
+      rm -f "$target"
       return 1
     fi
-    verify_backup "$target"
+    verify_volume_backup "$target"
     return $?
   fi
 
@@ -120,14 +246,20 @@ backup_database() {
     return 1
   fi
 
-  verify_backup "$target" "CREATE TABLE"
+  verify_sql_backup "$target"
 }
 
-# A dump that exists but is empty or truncated is worse than none, because it
+# A backup that exists but is empty, truncated, or is the wrong KIND of backup is
-# looks like a restore point. Check size, and content when we know what to expect.
+# worse than none, because it looks like a restore point. Each provider gets the
-verify_backup() {
+# check that proves its own format — a tar cannot pass the dump check and a dump
-  local target="$1"
+# cannot pass the archive check.
-  local expect="${2:-}"
+#
+# Note on `set +o pipefail` below: `grep -q` exits at the first match, which
+# SIGPIPEs the decompressor upstream. Under pipefail that fails the pipeline and
+# would reject a perfectly good backup.
+
+verify_backup_file_shape() {
+  local target="$1" kind="$2"
 
   if [ ! -s "$target" ]; then
     echo "Backup file ${target} is missing or empty. Aborting deploy."
@@ -135,20 +267,83 @@ verify_backup() {
     return 1
   fi
 
-  local size
+  if ! gzip -t "$target" 2>/dev/null; then
-  size="$(du -h "$target" | cut -f1)"
+    echo "Backup ${target} is not a valid gzip archive (truncated ${kind}?). Aborting deploy."
-
-  if [ -n "$expect" ] && ! gzip -dc "$target" | grep -q "$expect"; then
-    echo "Backup ${target} does not contain '${expect}' — it is not a usable dump. Aborting deploy."
     rm -f "$target"
     return 1
   fi
 
+  return 0
+}
+
+report_backup() {
+  local target="$1" size
+  size="$(du -h "$target" | cut -f1)"
   echo "Backup verified: ${target} (${size})"
   echo "Retention is manual — old backups in ${BACKUP_DIR} are never deleted automatically."
   return 0
 }
 
+verify_sql_backup() {
+  local target="$1"
+
+  verify_backup_file_shape "$target" "dump" || return 1
+
+  local has_schema=1 has_trailer=1
+  set +o pipefail
+  if gzip -dc "$target" | grep -q "CREATE TABLE"; then has_schema=0; fi
+  # mariadb-dump / mysqldump write "-- Dump completed on ..." as the last line.
+  # Its absence means the dump died partway through.
+  if gzip -dc "$target" | grep -q "Dump completed"; then has_trailer=0; fi
+  set -o pipefail
+
+  if [ "$has_schema" -ne 0 ]; then
+    echo "Backup ${target} contains no CREATE TABLE statement — it is not a usable dump. Aborting deploy."
+    rm -f "$target"
+    return 1
+  fi
+
+  if [ "$has_trailer" -ne 0 ]; then
+    echo "Backup ${target} has no dump trailer — it is truncated. Aborting deploy."
+    rm -f "$target"
+    return 1
+  fi
+
+  report_backup "$target"
+}
+
+verify_volume_backup() {
+  local target="$1"
+
+  verify_backup_file_shape "$target" "archive" || return 1
+
+  # Prove this is the DATABASE volume, not merely a non-empty tar. This is what
+  # catches an archive taken from the wrong volume, or from a freshly created
+  # empty one.
+  # Listed in the same container that wrote it, not with the host's tar: GNU tar
+  # reads a leading "C:/" as an rsh host and fails outright when this script runs
+  # from Git Bash. Docker is already required to have produced the archive.
+  local has_db=1
+  if docker run --rm \
+      -v "$(dirname "$target")":/backup:ro \
+      -e ARCHIVE_NAME="$(basename "$target")" \
+      alpine:3 sh -c 'tar tzf "/backup/$ARCHIVE_NAME" | grep -q "jobtracker\.db"' >/dev/null 2>&1; then
+    has_db=0
+  fi
+
+  if [ "$has_db" -ne 0 ]; then
+    echo "Archive ${target} contains no jobtracker.db — it is not a SQLite data backup. Aborting deploy."
+    rm -f "$target"
+    return 1
+  fi
+
+  report_backup "$target"
+}
+
+if ! validate_deploy_config; then
+  exit 1
+fi
+
 if ! backup_database; then
   exit 1
 fi

deploy/first-production-deployment.md

@@ -11,16 +11,25 @@
 
 Verified by reading `deploy/deploy.sh` and `JobTrackerApi/Program.cs`:
 
-1. `deploy.sh` links `/opt/job-tracker/shared/.env` into the checkout as `.env`.
+1. `deploy.sh` links `/opt/job-tracker/shared/.env` into the checkout as `.env`, **and loads it into
-2. **It takes a database backup and aborts if that fails.** Nothing else runs without a restore point.
+   its own shell.** The link is for docker compose; the script needs the values itself to pick the
-3. `docker compose pull`, then builds `backend` and `frontend` (with one prune-and-retry on failure).
+   right backup. Values already in the environment (CI's `APP_VERSION` and friends) win.
-4. `docker compose up -d --force-recreate --remove-orphans backend frontend`.
+2. **It validates the deployment configuration**, before anything is built, stopped or replaced:
+   `DATABASE_PROVIDER` (required, no default), the connection string when that provider needs one,
+   `AI_SERVICE_TOKEN` and `AUTH_JWT_KEY`. A missing variable aborts the deploy with the running stack
+   untouched. Names are printed, never values.
+3. **It takes a database backup and aborts if that fails.** Nothing else runs without a restore point.
+   The provider chooses the backup: `mariadb`/`mysql` gives a `.sql.gz` dump, `sqlite` gives a
+   `.tar.gz` of the data volume. Each is verified against its own format — the dump must contain
+   `CREATE TABLE` and the `Dump completed` trailer; the archive must contain `jobtracker.db`.
+4. `docker compose pull`, then builds `backend` and `frontend` (with one prune-and-retry on failure).
+5. `docker compose up -d --force-recreate --remove-orphans backend frontend`.
    **There is no `compose down`** — containers are replaced in place, so the window is short.
-5. On backend start, `InitializeJobTrackerAsync` runs: **reconcile → `Database.Migrate()` → reconcile**.
+6. On backend start, `InitializeJobTrackerAsync` runs: **reconcile → `Database.Migrate()` → reconcile**.
    Every Phase 4/5 migration is a no-op; the reconciler creates those tables with correct per-provider
    DDL. `Migrate()` throws on failure, so a schema problem exits the container rather than limping on.
-6. `deploy.sh` waits, then fails the deploy if `backend` is not running, and runs a public smoke check
+7. `deploy.sh` waits, then fails the deploy if `backend` is not running, and runs a public smoke check
-   against `APP_PUBLIC_BASE_URL` if it is set.
+   against `APP_PUBLIC_BASE_URL`.
 
 ---
 
@@ -31,12 +40,17 @@ Verified by reading `deploy/deploy.sh` and `JobTrackerApi/Program.cs`:
       restored is a hypothesis.
 - [ ] **Disk space checked.** `df -h` on the host. You need room for the backup, two image sets during
       the build, and the build cache. `docker system df` shows what Docker is holding.
-- [ ] **Environment variables present.** Check `/opt/job-tracker/shared/.env` contains:
+- [ ] **Environment variables present.** `deploy.sh` now checks these itself and aborts before it
-      - `AI_SERVICE_TOKEN` — **compose refuses to start without it**
+      builds or replaces anything, so a miss costs an aborted deploy rather than a broken one. Check
+      first anyway and skip the round trip — `/opt/job-tracker/shared/.env` must contain:
+      - `DATABASE_PROVIDER=mariadb` (or `mysql`) — **required, no default.** An absent value used to
+        mean "sqlite", which silently produced the wrong backup; it now stops the deploy
+      - `JOBTRACKER_CONNECTION_STRING` — see the host-resolution note below
+      - `AI_SERVICE_TOKEN` — compose refuses to start without it
       - `AUTH_JWT_KEY` — with `Auth__Require=true`, a blank key **throws at startup** (this is good;
         it fails loud rather than silently invalidating every session on restart)
-      - `DATABASE_PROVIDER=mysql` — **defaults to `sqlite` if absent**
+      - `APP_PUBLIC_BASE_URL` — optional; without it the post-deploy public smoke check is skipped,
-      - `JOBTRACKER_CONNECTION_STRING` — see the host-resolution note below
+        and the script prints that it is skipping
       - `AUTH_ADMIN_EMAIL` / `AUTH_ADMIN_PASSWORD` only if you want admin seeding on this boot
 - [ ] **Connection string host resolves from inside the container.** `Server=127.0.0.1` means *the
       backend container*, not the host — this bit me during validation. Use the host's LAN address, a
@@ -65,8 +79,12 @@ Automatic — `deploy.sh` runs it first and aborts on failure. Confirm afterward
 ls -lt /opt/job-tracker/backups | head -3
 ```
 
-Expect a new `jobtracker-<db>-<UTC timestamp>.sql.gz`. The script already rejected it if it were empty
+Expect a new `jobtracker-<db>-<UTC timestamp>.sql.gz`. The script already rejected it if it were empty,
-or missing `CREATE TABLE`.
+not valid gzip, missing `CREATE TABLE`, or missing the `Dump completed` trailer.
+
+**Check the filename, not just that a file appeared.** A `jobtracker-sqlite-<stamp>.tar.gz` on a
+MariaDB deploy means the environment is wrong — that is the exact failure the provider check exists to
+prevent, and it should now abort rather than reach this point.
 
 ### 2. Pull code
 
@@ -105,7 +123,7 @@ docker compose logs -f backend
 | `Unhandled exception ... Table '...' doesn't exist` | Reconciler ordering problem |
 | `no such table: INFORMATION_SCHEMA.TABLES` | `DATABASE_PROVIDER=mysql` but the connection string is **empty** — verified failure mode |
 | `Unable to connect to any of the specified MySQL hosts` | Connection string host unreachable from inside the container |
-| `Auth is required but Auth:JwtKey is not configured` | `AUTH_JWT_KEY` missing from `.env` |
+| `Auth is required but Auth:JwtKey is not configured` | `AUTH_JWT_KEY` missing from `.env` — `deploy.sh` should now catch this before the build |
 
 ### 8. Health verification
 
@@ -236,6 +254,11 @@ Against MariaDB 11 containers, no production data:
 | Backup against a seeded MariaDB | ✅ verified dump written |
 | Restore into a clean MariaDB | ✅ rows identical |
 | Backup failure paths | ✅ bad credentials and missing connection string both abort, no partial file |
+| Shared `.env` loaded into the deploy shell | ✅ MariaDB path selected from the file alone; dump contains schema, rows and trailer |
+| `DATABASE_PROVIDER` missing or unrecognised | ✅ deploy stops before build/replace, names the variable, leaves no backup file |
+| `AI_SERVICE_TOKEN` / `AUTH_JWT_KEY` missing | ✅ both reported in one pass, deploy stops |
+| SQLite volume backup, and its failure paths | ✅ valid archive verified; a volume with no `jobtracker.db`, and a volume name that does not exist, both abort |
+| Secret leakage in deploy output | ✅ zero occurrences of any password, token or key across every test |
 | Empty connection string with `provider=mysql` | ✅ fails loudly (`no such table: INFORMATION_SCHEMA.TABLES`) rather than silently serving an empty database |
 | Backend with unreachable database | ✅ exits, reported `unhealthy` |
 

docs/release-candidate-review.md

@@ -9,8 +9,13 @@
 > `docs/infrastructure/database-ownership.md` (who creates which table),
 > `docs/release-checklist.md` (state of the build).
 
-**Verdict: do not deploy yet.** Two blocking items, one of them new and material — the pre-deploy
+**Updated 2026-07-19 (second pass).** **B1 is fixed and verified** — `deploy.sh` now loads the shared
-database backup does not do what every other document assumes it does.
+environment before it decides anything, requires `DATABASE_PROVIDER` explicitly, validates the rest of
+the deployment configuration before touching the stack, and verifies each backup against its own
+format. Details under *Blocking*.
+
+**Verdict: one blocking item remains — CI (B2), which is external.** The deployment path itself is now
+sound.
 
 ---
 
@@ -73,7 +78,9 @@ database backup does not do what every other document assumes it does.
 
 ## Blocking
 
-### B1. The pre-deploy database backup does not back up the database — it silently backs up the wrong thing
+### B1. ~~The pre-deploy database backup does not back up the database~~ — **CLOSED 2026-07-19**
+
+*Original finding, kept because the failure mode is worth understanding:*
 
 **Severity: critical. This invalidates the restore point that the entire deployment plan depends on.**
 
@@ -113,8 +120,47 @@ MariaDB dump. If the deploy then damages the schema, there is nothing to restore
 `/opt/job-tracker/backups` is named `jobtracker-<dbname>-<stamp>.sql.gz`, not
 `jobtracker-sqlite-<stamp>.tar.gz`. The filename alone distinguishes the two paths.
 
-**Interim manual workaround if you deploy before this is fixed:** take the MariaDB dump by hand,
+**Closed.** `deploy/deploy.sh` now:
-verify it restores into a scratch database, and treat `deploy.sh`'s backup line as meaningless.
+
+1. **Loads `/opt/job-tracker/shared/.env` into its own shell** before any decision. Parsed line by
+   line rather than sourced, because a compose `.env` is not a shell script. Variables already set in
+   the environment win, so CI-provided `APP_VERSION` and friends still override the file. No value is
+   echoed.
+2. **Requires `DATABASE_PROVIDER` explicitly.** The `:-sqlite` default is gone. Missing means stop and
+   say which variable is missing; an unrecognised value means stop.
+3. **Validates the rest of the deployment configuration before the backup**, and therefore before
+   anything is built, stopped or replaced: the connection string when the provider needs one,
+   `AI_SERVICE_TOKEN` (compose declares it with `:?`, so missing it would otherwise kill the stack
+   after the images are built) and `AUTH_JWT_KEY` (the backend throws at startup on a blank key, after
+   the containers have been replaced). Names only in the output, never values.
+4. **Verifies each backup against its own format.** A MariaDB dump must be valid gzip, contain
+   `CREATE TABLE`, and carry the `Dump completed` trailer that `mariadb-dump` writes last — so a dump
+   that died partway through is rejected. A SQLite archive must be valid gzip and actually contain
+   `jobtracker.db`. A tar can no longer pass the dump check, which is precisely what went wrong.
+5. **Resolves the SQLite volume by its real, project-prefixed name** and fails if it does not exist.
+   The old code named the bare `jobtracker_data`, which on a real deploy would have silently *created*
+   an empty volume and backed that up — a second instance of the same class of bug, found while
+   testing the fix.
+
+Two side effects of the same root cause are also resolved: `APP_PUBLIC_BASE_URL` now reaches the
+script, so the post-deploy public smoke check actually runs (and the script says so explicitly when it
+is unset rather than skipping in silence), as does `OLLAMA_MODEL` for the warmup.
+
+*Verified against a seeded MariaDB 11 container and real Docker volumes:*
+
+| Scenario | Result |
+|---|---|
+| MariaDB production-style `.env` | ✅ env loaded, MariaDB path selected, `.sql.gz` written containing `CREATE TABLE`, the seeded row, and the `Dump completed` trailer |
+| `DATABASE_PROVIDER` missing | ✅ exits 1 naming the variable; nothing built, stopped or replaced; no backup file |
+| `DATABASE_PROVIDER` unrecognised | ✅ exits 1 |
+| `AI_SERVICE_TOKEN` / `AUTH_JWT_KEY` missing | ✅ both reported in one pass, exits 1 |
+| SQLite with a populated volume | ✅ `.tar.gz` written and verified |
+| SQLite volume containing no `jobtracker.db` | ✅ rejected, file deleted |
+| SQLite volume name not present | ✅ rejected before any container ran; no stray empty volume created |
+| Broken database credentials | ✅ exits 1, no partial file left behind |
+| Truncated dump (no `CREATE TABLE`) | ✅ rejected |
+| Dump with the trailer stripped | ✅ rejected as truncated |
+| Secret leakage across every test's output | ✅ zero occurrences of any password, token or key |
 
 ### B2. CI is red — deployment is gated on it
 
@@ -134,7 +180,7 @@ Known and accepted for the first release. None of these should stop a deploy; al
 
 | # | Finding | Why it is not blocking |
 |---|---|---|
-| N1 | **`.env.example` omits `DATABASE_PROVIDER` and `JOBTRACKER_CONNECTION_STRING`** — both are consumed by `docker-compose.yml`, and the second is the only way to reach a database at all | The production `.env` already has them, and `deploy/README.md:48-49` documents both. Only bites someone building a new environment from the template |
+| N1 | ~~**`.env.example` omits `DATABASE_PROVIDER` and `JOBTRACKER_CONNECTION_STRING`**~~ — **CLOSED 2026-07-19** | Both added to the template with the connection-string host caveat. `deploy.sh` now hard-fails without `DATABASE_PROVIDER`, so the template had to name it |
 | N2 | **`/health` always reports `"version":"unknown"` under Docker** — the endpoint reads the `APP_VERSION` environment variable, but compose passes it as `App__Version` | Liveness is unaffected; only the version string is wrong. `deploy/first-production-deployment.md` shows a populated version in its expected output, which will not match reality |
 | N3 | **The post-deploy gate in `deploy.sh` checks `.State == running`, not health** — a container can be `running` while `starting` or `unhealthy` | Harmless in practice: `compose up` already blocks on `service_healthy` for the frontend's dependency, so an unhealthy backend aborts the deploy before this check is reached. The check is weaker than it looks, not wrong |
 | N4 | **Table-count discrepancy across documents** — `database-ownership.md` records 40 tables on MariaDB; `release-checklist.md` and the runbook say ~42 | Both were measured, at different points in Phase 5. Use "the tables listed in `database-ownership.md` all exist" as the check, not a number |
@@ -154,12 +200,17 @@ them — every automated check stops at the authentication boundary.
 
 ### Before deploying
 
-- [ ] **Take a MariaDB dump by hand and restore it into a scratch database.** Given B1, this is not
+- [ ] **Take a MariaDB dump by hand and restore it into a scratch database.** B1 is fixed and the
-      optional and not a formality. `deploy.sh`'s backup line cannot currently be trusted on a MariaDB
+      automatic backup is verified against containers, but the first production deploy is still the
-      host.
+      wrong moment to discover that a backup does not restore *on your data*.
-- [ ] **Confirm `/opt/job-tracker/shared/.env` contains `DATABASE_PROVIDER` and
+- [ ] **Confirm `/opt/job-tracker/shared/.env` contains `DATABASE_PROVIDER=mariadb` and
-      `JOBTRACKER_CONNECTION_STRING`**, and that the `Server=` host resolves *from inside the backend
+      `JOBTRACKER_CONNECTION_STRING`.** `deploy.sh` now aborts without them, so a missing value costs
-      container* — `127.0.0.1` there means the container, not the host.
+      an aborted deploy rather than a bad backup — but check first and skip the round trip.
+- [ ] **Confirm the connection-string host resolves *from inside the backend container*** —
+      `127.0.0.1` there means the container, not the Docker host.
+- [ ] **After the deploy, check the backup filename.** It must be
+      `jobtracker-<database>-<stamp>.sql.gz`. A `jobtracker-sqlite-<stamp>.tar.gz` means the
+      environment is wrong — though the provider check should now stop that before it happens.
 - [ ] **Record the current commit** (`git rev-parse HEAD`) and the current row counts for
       `JobApplications` and `Companies`. The row counts are the check that matters most afterwards.