chore(ops): add deployment backups restore docs and health checks

Closes the three operational blockers from the production readiness review.

deploy.sh now takes a database backup before it builds, stops or replaces
anything, and aborts the deploy if the backup fails — so no deploy proceeds
without a restore point. Dumps are gzipped and timestamped into
/opt/job-tracker/backups (override with BACKUP_DIR), so one deploy never
overwrites an earlier backup. Credentials come from the existing connection
string and travel via MYSQL_PWD, never on the command line, so they cannot reach
the process list or the deploy log. A dump that is empty or missing CREATE TABLE
is rejected, because a truncated file that looks like a restore point is worse
than none. SQLite deployments get their data volume tarred instead. Nothing is
ever deleted automatically; retention is documented as manual.

deploy/README.md documents backup creation, location, retention, database
restore, application rollback, and when to use which — restore and rollback kept
distinct, because a bad deploy usually needs only the rollback and restoring
would discard everything written since the dump.

Health checks now cover backend and frontend, which previously had none. GET
/health is anonymous, cheap, and deliberately does not touch the database: a
health check that queried MariaDB would restart a healthy backend whenever the
database blipped, and would hand out an unauthenticated way to probe database
availability. The backend image gains curl on the existing chromium apt layer,
since the aspnet runtime ships neither curl nor wget. frontend now waits for
backend to be healthy rather than merely started, because nginx proxies /api and
refuses to start when the upstream cannot be resolved.

Verified against real containers, no production data: backup from a seeded
MariaDB 11; restore into a clean MariaDB 11 with rows identical; bad credentials
and a missing connection string both abort non-zero and leave no partial file;
SQLite volume backup produces a readable archive; backend and frontend both
reach healthy; and a backend pointed at an unreachable database exits and is
reported unhealthy, so a broken deploy cannot present as a running stack.

Incidentally confirmed the earlier authorization work: with Auth:Require unset,
/health returns 200 while /api/jobapplications returns 401.

393 backend tests pass.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

docs/production-readiness-review.md

@@ -3,10 +3,14 @@
 > 2026-07-19, before the first production deployment after the Phase 4/5 architecture changes.
 > Companion to `docs/phase-5-completion-report.md` and `docs/infrastructure/database-ownership.md`.
 >
-> **Verdict: not ready to deploy unattended.** The application itself verifies clean, but three
-> operational gaps (no pre-deploy backup, no documented restore, no backend health endpoint) and one
-> external blocker (CI) stand between here and a safe first deploy. All are listed below with what
-> would close them.
+> **Updated 2026-07-19 (second pass).** The three operational blockers are **closed and verified**:
+> `deploy.sh` now takes a verified backup before touching anything and aborts if it fails, restore and
+> rollback are documented and tested against real containers, and `backend`/`frontend` both report
+> health.
+>
+> **Remaining verdict: one external blocker.** CI is red for an environmental reason and deployment is
+> gated on it. Everything here remains local verification. The accepted-risk items in *Remaining risks*
+> are unchanged and still worth reading before the first deploy.
 
 ## Verified
 
@@ -51,9 +55,9 @@
 
 ## Remaining risks
 
-### 1. No pre-deploy database backup — **blocker for first deploy**
+### 1. ~~No pre-deploy database backup~~ — **CLOSED 2026-07-19**
 
-`deploy/deploy.sh` does not dump the database before bringing the stack down. The first deploy after
+*Original finding:* `deploy/deploy.sh` did not dump the database before bringing the stack down. The first deploy after
 these changes is exactly when a backup matters most: prod is many commits behind, and the reconciler
 will create roughly a dozen tables on first boot. That path is verified on containers, **not on your
 data**.
@@ -61,23 +65,43 @@ data**.
 `BackupController` exposes only `POST /api/backup/encrypted` — an application-level encrypted export,
 not an operational database dump. It is not a substitute.
 
-**To close:** run `mariadb-dump` before `compose down`, keep the dump, and ideally add that step to
-`deploy.sh` so it is not a thing to remember.
+**Closed.** `deploy/deploy.sh` now backs up before it builds, stops or replaces anything, and aborts
+the deploy if the backup fails. Timestamped and gzipped to `/opt/job-tracker/backups` (override with
+`BACKUP_DIR`), so a deploy never overwrites an earlier backup. The password travels via `MYSQL_PWD`,
+never on the command line. The dump is rejected if it is empty or lacks `CREATE TABLE`. SQLite
+deployments get the data volume tarred instead.
 
-### 2. No documented restore procedure — **blocker for first deploy**
+*Verified:* dump taken from a seeded MariaDB 11 container; failure paths (bad credentials, missing
+connection string) abort with a non-zero status and leave no misleading partial file.
 
-There is no written restore path. A backup you have never restored is a hypothesis. `deploy/README.md`
+### 2. ~~No documented restore procedure~~ — **CLOSED 2026-07-19**
+
+*Original finding:* there was no written restore path. A backup you have never restored is a hypothesis. `deploy/README.md`
 mentions backups in one line ("keep backups and volume persistence") and stops there.
 
-**To close:** document restore, and rehearse it once against a scratch database.
+**Closed.** `deploy/README.md` documents backup creation, location, retention, database restore,
+application rollback, and — importantly — when to use which. Restore and rollback are presented as
+separate operations, because a bad deploy usually needs only the rollback.
 
-### 3. No backend health endpoint — **operational gap**
+*Verified:* the dump was restored into a clean MariaDB 11 container and the rows came back identical.
+The rehearsal is the documented example.
 
-`docker-compose.yml` defines health checks for `ai-service` and `ollama` but **not for `backend` or
-`frontend`**. Nothing automatically detects a backend that started and then became unhealthy, and
+### 3. ~~No backend health endpoint~~ — **CLOSED 2026-07-19**
+
+*Original finding:* `docker-compose.yml` defined health checks for `ai-service` and `ollama` but not for
+`backend` or `frontend`. Nothing automatically detects a backend that started and then became unhealthy, and
 `depends_on` cannot gate on its readiness.
 
-**To close:** add a `/health` endpoint (anonymous, no data) and a compose health check.
+**Closed.** `GET /health` is anonymous, cheap, and deliberately **does not touch the database** — a
+health check that queried MariaDB would restart a healthy backend whenever the database blipped, and
+would hand out an unauthenticated way to probe database availability. Compose health checks now cover
+`backend` (curl, 90s start period for first-boot reconciliation) and `frontend` (wget against nginx),
+and `frontend` waits for `backend` to be *healthy* rather than merely started.
+
+*Verified:* both containers reach `healthy`; a backend pointed at an unreachable database exits and is
+reported `unhealthy`, so a broken deploy cannot present as a running stack. `/health` returns 200
+anonymously while `/api/jobapplications` returns 401 **with `Auth:Require` unset**, confirming the
+explicit `[Authorize]` work holds independently of that flag.
 
 ### 4. CI is red for an environmental reason — **external blocker**
 
@@ -115,9 +139,9 @@ verification is tests, builds and startup checks. Friction points in that journe
 Run in order. Stop if any step fails.
 
 1. **Resolve CI**, or make a deliberate decision to deploy from a manually verified commit.
-2. **Back up the production database.**
-   `mariadb-dump -u<user> -p<pass> --single-transaction --routines jobtracker > jobtracker-$(date +%F).sql`
-3. **Verify the dump is non-empty and contains `CREATE TABLE JobApplications`.**
+2. ~~Back up the production database.~~ **`deploy.sh` now does this automatically and aborts if it
+   fails.** Confirm afterwards that a new file appeared in `/opt/job-tracker/backups`.
+3. ~~Verify the dump.~~ **The script verifies size and content before continuing.**
 4. **Note the current commit** for rollback: `git rev-parse HEAD` on the prod checkout.
 5. Confirm `.env` has `AUTH_JWT_KEY`, `AI_SERVICE_TOKEN`, `JOBTRACKER_CONNECTION_STRING`,
    `DATABASE_PROVIDER=mysql`, and that `Auth__Require=true` is still in `docker-compose.yml`.