chore(ops): add deployment backups restore docs and health checks

Closes the three operational blockers from the production readiness review.

deploy.sh now takes a database backup before it builds, stops or replaces
anything, and aborts the deploy if the backup fails — so no deploy proceeds
without a restore point. Dumps are gzipped and timestamped into
/opt/job-tracker/backups (override with BACKUP_DIR), so one deploy never
overwrites an earlier backup. Credentials come from the existing connection
string and travel via MYSQL_PWD, never on the command line, so they cannot reach
the process list or the deploy log. A dump that is empty or missing CREATE TABLE
is rejected, because a truncated file that looks like a restore point is worse
than none. SQLite deployments get their data volume tarred instead. Nothing is
ever deleted automatically; retention is documented as manual.

deploy/README.md documents backup creation, location, retention, database
restore, application rollback, and when to use which — restore and rollback kept
distinct, because a bad deploy usually needs only the rollback and restoring
would discard everything written since the dump.

Health checks now cover backend and frontend, which previously had none. GET
/health is anonymous, cheap, and deliberately does not touch the database: a
health check that queried MariaDB would restart a healthy backend whenever the
database blipped, and would hand out an unauthenticated way to probe database
availability. The backend image gains curl on the existing chromium apt layer,
since the aspnet runtime ships neither curl nor wget. frontend now waits for
backend to be healthy rather than merely started, because nginx proxies /api and
refuses to start when the upstream cannot be resolved.

Verified against real containers, no production data: backup from a seeded
MariaDB 11; restore into a clean MariaDB 11 with rows identical; bad credentials
and a missing connection string both abort non-zero and leave no partial file;
SQLite volume backup produces a readable archive; backend and frontend both
reach healthy; and a backend pointed at an unreachable database exits and is
reported unhealthy, so a broken deploy cannot present as a running stack.

Incidentally confirmed the earlier authorization work: with Auth:Require unset,
/health returns 200 while /api/jobapplications returns 401.

393 backend tests pass.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

JobTrackerApi/Dockerfile

@@ -25,8 +25,10 @@ WORKDIR /app
 ENV ASPNETCORE_URLS=http://+:8080
 ENV CV_PDF_BROWSER_PATH=/usr/bin/chromium
 
+# curl is here for the container health check (/health). The aspnet runtime image ships
+# neither curl nor wget, and this layer already exists for chromium.
 RUN apt-get update \
-    && apt-get install -y --no-install-recommends chromium \
+    && apt-get install -y --no-install-recommends chromium curl \
     && rm -rf /var/lib/apt/lists/*
 
 RUN mkdir -p /data

JobTrackerApi/Program.cs

@@ -520,6 +520,17 @@ app.UseAuthentication();
 app.UseAuthorization();
 app.MapControllers();
 
+// Liveness probe for the container health check. Deliberately cheap and anonymous: it answers "is
+// this process up and serving?" and nothing else. It does NOT touch the database — a health check
+// that queried MariaDB would restart a perfectly healthy backend whenever the database blipped, and
+// would also be a free unauthenticated way to probe database availability.
+// docs/production-readiness-review.md.
+app.MapGet("/health", () => Results.Ok(new
+{
+    status = "ok",
+    version = Environment.GetEnvironmentVariable("APP_VERSION") ?? "unknown",
+})).AllowAnonymous();
+
 // API schema for tooling/docs. Dev-only: not exposed in production deployments.
 if (app.Environment.IsDevelopment())
 {

deploy/README.md

@@ -100,3 +100,122 @@ If this app is going to be a real production service on Ubuntu:
 - confirm reminder and admin/system pages load
 - verify follow-up reminder emails are enabled only when intended and that links open the correct job/tab
 hat links open the correct job/tab
+
+---
+
+# Backups, restore and rollback
+
+**Database restore and application rollback are two different operations.** A bad deploy usually needs
+only the rollback. Restore the database only if the data itself is wrong or lost — it discards
+everything written since the dump.
+
+## Backup creation
+
+`deploy/deploy.sh` takes a backup **before** it builds, stops or replaces anything, and **aborts the
+deploy if the backup fails**. Nothing else in the deploy runs without a restore point.
+
+- **Location:** `/opt/job-tracker/backups` — override with `BACKUP_DIR`.
+- **Naming:** `jobtracker-<database>-<UTC timestamp>.sql.gz`, e.g.
+  `jobtracker-jobtracker-20260719T153759Z.sql.gz`. The timestamp makes every file unique, so a deploy
+  never overwrites an earlier backup.
+- **SQLite deployments** (`DATABASE_PROVIDER` unset or `sqlite`) get the data volume instead:
+  `jobtracker-sqlite-<UTC timestamp>.tar.gz`.
+- **Credentials** come from `JOBTRACKER_CONNECTION_STRING` and are passed via `MYSQL_PWD`, never on the
+  command line, so they cannot appear in the process list or the deploy log.
+- **Compression:** gzip. A small database compresses to a few KB.
+- **Verification:** the script rejects a dump that is empty or missing `CREATE TABLE`, because a
+  truncated file that *looks* like a restore point is worse than none.
+
+### Taking one by hand
+
+```bash
+BACKUP_DIR=/opt/job-tracker/backups
+mkdir -p "$BACKUP_DIR"
+MYSQL_PWD='<password>' mariadb-dump \
+  --host=127.0.0.1 --port=3306 --user=<user> \
+  --single-transaction --routines --events --quick \
+  jobtracker | gzip -c > "$BACKUP_DIR/jobtracker-manual-$(date -u +%Y%m%dT%H%M%SZ).sql.gz"
+```
+
+### Retention
+
+**Nothing is deleted automatically.** Backups accumulate in `BACKUP_DIR` until you remove them. Watch
+disk usage and prune deliberately — a suggested policy is to keep every backup for 30 days and one per
+month after that, but the script does not enforce it and will not delete your files.
+
+## Restoring the database
+
+Tested end-to-end against a clean MariaDB 11 container: dump taken from a seeded database, restored
+into an empty one, rows verified identical.
+
+```bash
+# 1. Stop the application so nothing writes during the restore.
+docker compose stop backend
+
+# 2. Restore. This REPLACES the current contents of the named database.
+gzip -dc /opt/job-tracker/backups/jobtracker-jobtracker-20260719T153759Z.sql.gz \
+  | MYSQL_PWD='<password>' mariadb --host=127.0.0.1 --port=3306 --user=<user> jobtracker
+
+# 3. Verify before starting anything.
+MYSQL_PWD='<password>' mariadb --host=127.0.0.1 --port=3306 --user=<user> jobtracker \
+  -e "SELECT COUNT(*) AS applications FROM JobApplications;"
+
+# 4. Start again.
+docker compose start backend
+```
+
+Restoring a **SQLite** deployment instead:
+
+```bash
+docker compose stop backend
+docker run --rm -v jobtracker_data:/data -v /opt/job-tracker/backups:/backup \
+  -e ARCHIVE_NAME=jobtracker-sqlite-20260719T153941Z.tar.gz \
+  alpine:3 sh -c 'rm -rf /data/* && tar xzf "/backup/$ARCHIVE_NAME" -C /data'
+docker compose start backend
+```
+
+## Restoring application containers (rollback)
+
+This is the usual fix for a bad deploy, and it **does not touch the database**.
+
+```bash
+cd /opt/job-tracker/app          # the deployment checkout
+git log --oneline -5             # find the last good commit
+git checkout <previous-commit>
+deploy/deploy.sh
+```
+
+`deploy.sh` takes a fresh backup first, so rolling back is itself protected.
+
+**Why a code rollback is safe here:** every Phase 4/5 migration is a no-op — the startup reconciler
+owns those tables — so reverting the code never leaves migration history ahead of the schema. The
+reconciler is additive and never drops a table holding rows, so the older code simply ignores the
+newer tables.
+
+**What a code rollback does not undo:** rows users created in the newer tables stay. That is usually
+what you want. If you additionally restore the database, those rows are lost — so restore only when
+the data is the problem.
+
+## Choosing between them
+
+| Symptom | Action |
+|---|---|
+| New version starts but behaves wrong | Rollback the code. Leave the database. |
+| Backend will not start; schema looks wrong | Rollback the code, then restore only if it still fails. |
+| Data is missing or corrupted | Restore the database from the most recent good dump. |
+| Deploy aborted before starting | Nothing to undo — the backup ran before any change. |
+
+## Health checks
+
+`backend` and `frontend` both report container health, so `docker compose ps` shows real state rather
+than merely "running".
+
+- **Backend:** `curl -fsS http://127.0.0.1:8080/health`. Anonymous, and deliberately **does not touch
+  the database** — a health check that queried MariaDB would restart a healthy backend whenever the
+  database blipped. `start_period` is 90s to cover first-boot schema reconciliation.
+- **Frontend:** `wget` against nginx on port 80.
+- `frontend` waits for `backend` to be *healthy*, not merely started, because nginx proxies `/api` to
+  it and refuses to start if the upstream cannot be resolved.
+
+A backend that cannot reach its database exits and is reported `unhealthy`, so a broken deploy does not
+present as a running stack.

deploy/deploy.sh

@@ -28,6 +28,131 @@ compose() {
   docker compose "$@"
 }
 
+# ---------------------------------------------------------------------------
+# Database backup, taken BEFORE anything is stopped, built or replaced.
+#
+# A failed backup fails the deploy. The startup reconciler is additive and never
+# drops a table holding rows, but "additive" is a property of the code, not a
+# guarantee about the disk — and the first deploy after a schema change is
+# exactly when a restore point matters. See deploy/README.md.
+# ---------------------------------------------------------------------------
+BACKUP_DIR="${BACKUP_DIR:-/opt/job-tracker/backups}"
+
+backup_database() {
+  if [ "${DEPLOY_SKIP_DB_BACKUP:-false}" = "true" ]; then
+    echo "WARNING: DEPLOY_SKIP_DB_BACKUP=true — deploying with no restore point. Emergency use only."
+    return 0
+  fi
+
+  local provider="${DATABASE_PROVIDER:-sqlite}"
+  local stamp
+  stamp="$(date -u +%Y%m%dT%H%M%SZ)"
+  mkdir -p "$BACKUP_DIR"
+
+  if [ "$provider" != "mysql" ] && [ "$provider" != "mariadb" ]; then
+    # SQLite lives in the jobtracker_data volume. Tar it from a throwaway container
+    # so the host needs no sqlite tooling and no knowledge of the volume layout.
+    local target="$BACKUP_DIR/jobtracker-sqlite-${stamp}.tar.gz"
+    echo "Backing up SQLite data volume to ${target}"
+    # The archive path is built inside the container: passing /backup/... as an argument
+    # gets rewritten by MSYS path translation when the script is run from Git Bash.
+    if ! docker run --rm \
+        -v jobtracker_data:/data:ro \
+        -v "$BACKUP_DIR":/backup \
+        -e ARCHIVE_NAME="$(basename "$target")" \
+        alpine:3 sh -c 'tar czf "/backup/$ARCHIVE_NAME" -C /data .'; then
+      echo "SQLite volume backup FAILED. Aborting deploy."
+      return 1
+    fi
+    verify_backup "$target"
+    return $?
+  fi
+
+  local cs="${JOBTRACKER_CONNECTION_STRING:-}"
+  if [ -z "$cs" ]; then
+    echo "DATABASE_PROVIDER=${provider} but JOBTRACKER_CONNECTION_STRING is empty. Aborting deploy."
+    return 1
+  fi
+
+  # Parse the ADO.NET connection string. Keys are case-insensitive in .NET, so match that.
+  local db_host db_port db_name db_user db_pass
+  db_host="$(sed -n 's/.*[Ss]erver=\([^;]*\).*/\1/p' <<<"$cs")"
+  db_port="$(sed -n 's/.*[Pp]ort=\([^;]*\).*/\1/p' <<<"$cs")"
+  db_name="$(sed -n 's/.*[Dd]atabase=\([^;]*\).*/\1/p' <<<"$cs")"
+  db_user="$(sed -n 's/.*[Uu]ser[ ]*[Ii]*[Dd]*=\([^;]*\).*/\1/p' <<<"$cs")"
+  db_pass="$(sed -n 's/.*[Pp]assword=\([^;]*\).*/\1/p' <<<"$cs")"
+  db_port="${db_port:-3306}"
+
+  if [ -z "$db_host" ] || [ -z "$db_name" ] || [ -z "$db_user" ]; then
+    echo "Could not parse host/database/user from JOBTRACKER_CONNECTION_STRING. Aborting deploy."
+    return 1
+  fi
+
+  local target="$BACKUP_DIR/jobtracker-${db_name}-${stamp}.sql.gz"
+  echo "Backing up MariaDB database '${db_name}' on ${db_host}:${db_port} to ${target}"
+
+  # The password goes via MYSQL_PWD, never on the command line, so it cannot leak
+  # into the process list or the deploy log.
+  local dump_status=0
+  if command -v mariadb-dump >/dev/null 2>&1; then
+    MYSQL_PWD="$db_pass" mariadb-dump \
+      --host="$db_host" --port="$db_port" --user="$db_user" \
+      --single-transaction --routines --events --quick \
+      "$db_name" 2>/tmp/jobtracker-backup.err | gzip -c > "$target" || dump_status=$?
+  elif command -v mysqldump >/dev/null 2>&1; then
+    MYSQL_PWD="$db_pass" mysqldump \
+      --host="$db_host" --port="$db_port" --user="$db_user" \
+      --single-transaction --routines --events --quick \
+      "$db_name" 2>/tmp/jobtracker-backup.err | gzip -c > "$target" || dump_status=$?
+  else
+    # No client on the host: run one in a container. --network host so the same
+    # host/port from the connection string resolves identically.
+    docker run --rm --network host -e MYSQL_PWD="$db_pass" mariadb:11 \
+      mariadb-dump --host="$db_host" --port="$db_port" --user="$db_user" \
+      --single-transaction --routines --events --quick \
+      "$db_name" 2>/tmp/jobtracker-backup.err | gzip -c > "$target" || dump_status=$?
+  fi
+
+  if [ "$dump_status" -ne 0 ]; then
+    echo "Database dump FAILED (exit ${dump_status}). Aborting deploy."
+    sed -e 's/password=[^ ]*/password=***/gI' /tmp/jobtracker-backup.err >&2 || true
+    rm -f "$target"
+    return 1
+  fi
+
+  verify_backup "$target" "CREATE TABLE"
+}
+
+# A dump that exists but is empty or truncated is worse than none, because it
+# looks like a restore point. Check size, and content when we know what to expect.
+verify_backup() {
+  local target="$1"
+  local expect="${2:-}"
+
+  if [ ! -s "$target" ]; then
+    echo "Backup file ${target} is missing or empty. Aborting deploy."
+    rm -f "$target"
+    return 1
+  fi
+
+  local size
+  size="$(du -h "$target" | cut -f1)"
+
+  if [ -n "$expect" ] && ! gzip -dc "$target" | grep -q "$expect"; then
+    echo "Backup ${target} does not contain '${expect}' — it is not a usable dump. Aborting deploy."
+    rm -f "$target"
+    return 1
+  fi
+
+  echo "Backup verified: ${target} (${size})"
+  echo "Retention is manual — old backups in ${BACKUP_DIR} are never deleted automatically."
+  return 0
+}
+
+if ! backup_database; then
+  exit 1
+fi
+
 build_core_with_recovery() {
   if compose build backend frontend; then
     return 0

docker-compose.yml

@@ -62,6 +62,15 @@ services:
       # ai-service.
       - ai_internal
     restart: unless-stopped
+    # Liveness only. /health does not touch the database on purpose: a health check that queried
+    # MariaDB would restart a healthy backend whenever the database blipped.
+    # start_period covers first-boot schema reconciliation, which can take a while on a new database.
+    healthcheck:
+      test: ["CMD", "curl", "-fsS", "http://127.0.0.1:8080/health"]
+      interval: 30s
+      timeout: 5s
+      retries: 3
+      start_period: 90s
 
   frontend:
     build:
@@ -77,11 +86,19 @@ services:
     ports:
       - "3000:80"
     depends_on:
-      - backend
+      backend:
+        condition: service_healthy
     networks:
       - default
       - shared_services
     restart: unless-stopped
+    # Cheap liveness: nginx answering on its own port. wget ships with the alpine base.
+    healthcheck:
+      test: ["CMD", "wget", "-q", "-O", "/dev/null", "http://127.0.0.1:80/"]
+      interval: 30s
+      timeout: 5s
+      retries: 3
+      start_period: 10s
 
   ai-service:
     build:

docs/production-readiness-review.md

@@ -3,10 +3,14 @@
 > 2026-07-19, before the first production deployment after the Phase 4/5 architecture changes.
 > Companion to `docs/phase-5-completion-report.md` and `docs/infrastructure/database-ownership.md`.
 >
-> **Verdict: not ready to deploy unattended.** The application itself verifies clean, but three
+> **Updated 2026-07-19 (second pass).** The three operational blockers are **closed and verified**:
-> operational gaps (no pre-deploy backup, no documented restore, no backend health endpoint) and one
+> `deploy.sh` now takes a verified backup before touching anything and aborts if it fails, restore and
-> external blocker (CI) stand between here and a safe first deploy. All are listed below with what
+> rollback are documented and tested against real containers, and `backend`/`frontend` both report
-> would close them.
+> health.
+>
+> **Remaining verdict: one external blocker.** CI is red for an environmental reason and deployment is
+> gated on it. Everything here remains local verification. The accepted-risk items in *Remaining risks*
+> are unchanged and still worth reading before the first deploy.
 
 ## Verified
 
@@ -51,9 +55,9 @@
 
 ## Remaining risks
 
-### 1. No pre-deploy database backup — **blocker for first deploy**
+### 1. ~~No pre-deploy database backup~~ — **CLOSED 2026-07-19**
 
-`deploy/deploy.sh` does not dump the database before bringing the stack down. The first deploy after
+*Original finding:* `deploy/deploy.sh` did not dump the database before bringing the stack down. The first deploy after
 these changes is exactly when a backup matters most: prod is many commits behind, and the reconciler
 will create roughly a dozen tables on first boot. That path is verified on containers, **not on your
 data**.
@@ -61,23 +65,43 @@ data**.
 `BackupController` exposes only `POST /api/backup/encrypted` — an application-level encrypted export,
 not an operational database dump. It is not a substitute.
 
-**To close:** run `mariadb-dump` before `compose down`, keep the dump, and ideally add that step to
+**Closed.** `deploy/deploy.sh` now backs up before it builds, stops or replaces anything, and aborts
-`deploy.sh` so it is not a thing to remember.
+the deploy if the backup fails. Timestamped and gzipped to `/opt/job-tracker/backups` (override with
+`BACKUP_DIR`), so a deploy never overwrites an earlier backup. The password travels via `MYSQL_PWD`,
+never on the command line. The dump is rejected if it is empty or lacks `CREATE TABLE`. SQLite
+deployments get the data volume tarred instead.
 
-### 2. No documented restore procedure — **blocker for first deploy**
+*Verified:* dump taken from a seeded MariaDB 11 container; failure paths (bad credentials, missing
+connection string) abort with a non-zero status and leave no misleading partial file.
 
-There is no written restore path. A backup you have never restored is a hypothesis. `deploy/README.md`
+### 2. ~~No documented restore procedure~~ — **CLOSED 2026-07-19**
+
+*Original finding:* there was no written restore path. A backup you have never restored is a hypothesis. `deploy/README.md`
 mentions backups in one line ("keep backups and volume persistence") and stops there.
 
-**To close:** document restore, and rehearse it once against a scratch database.
+**Closed.** `deploy/README.md` documents backup creation, location, retention, database restore,
+application rollback, and — importantly — when to use which. Restore and rollback are presented as
+separate operations, because a bad deploy usually needs only the rollback.
 
-### 3. No backend health endpoint — **operational gap**
+*Verified:* the dump was restored into a clean MariaDB 11 container and the rows came back identical.
+The rehearsal is the documented example.
 
-`docker-compose.yml` defines health checks for `ai-service` and `ollama` but **not for `backend` or
+### 3. ~~No backend health endpoint~~ — **CLOSED 2026-07-19**
-`frontend`**. Nothing automatically detects a backend that started and then became unhealthy, and
+
+*Original finding:* `docker-compose.yml` defined health checks for `ai-service` and `ollama` but not for
+`backend` or `frontend`. Nothing automatically detects a backend that started and then became unhealthy, and
 `depends_on` cannot gate on its readiness.
 
-**To close:** add a `/health` endpoint (anonymous, no data) and a compose health check.
+**Closed.** `GET /health` is anonymous, cheap, and deliberately **does not touch the database** — a
+health check that queried MariaDB would restart a healthy backend whenever the database blipped, and
+would hand out an unauthenticated way to probe database availability. Compose health checks now cover
+`backend` (curl, 90s start period for first-boot reconciliation) and `frontend` (wget against nginx),
+and `frontend` waits for `backend` to be *healthy* rather than merely started.
+
+*Verified:* both containers reach `healthy`; a backend pointed at an unreachable database exits and is
+reported `unhealthy`, so a broken deploy cannot present as a running stack. `/health` returns 200
+anonymously while `/api/jobapplications` returns 401 **with `Auth:Require` unset**, confirming the
+explicit `[Authorize]` work holds independently of that flag.
 
 ### 4. CI is red for an environmental reason — **external blocker**
 
@@ -115,9 +139,9 @@ verification is tests, builds and startup checks. Friction points in that journe
 Run in order. Stop if any step fails.
 
 1. **Resolve CI**, or make a deliberate decision to deploy from a manually verified commit.
-2. **Back up the production database.**
+2. ~~Back up the production database.~~ **`deploy.sh` now does this automatically and aborts if it
-   `mariadb-dump -u<user> -p<pass> --single-transaction --routines jobtracker > jobtracker-$(date +%F).sql`
+   fails.** Confirm afterwards that a new file appeared in `/opt/job-tracker/backups`.
-3. **Verify the dump is non-empty and contains `CREATE TABLE JobApplications`.**
+3. ~~Verify the dump.~~ **The script verifies size and content before continuing.**
 4. **Note the current commit** for rollback: `git rev-parse HEAD` on the prod checkout.
 5. Confirm `.env` has `AUTH_JWT_KEY`, `AI_SERVICE_TOKEN`, `JOBTRACKER_CONNECTION_STRING`,
    `DATABASE_PROVIDER=mysql`, and that `Auth__Require=true` is still in `docker-compose.yml`.