docs: production readiness review

Audited migrations, reconciler ownership, authentication, authorization, AI
security, file storage, public CV access, backups, logging, error handling and
configuration defaults before the first production deployment after the Phase
4/5 architecture changes.

Verdict: not ready to deploy unattended. The application verifies clean —
fresh MariaDB, populated MariaDB restart, existing SQLite upgrade, 393 backend
tests, 128 frontend tests, both Docker images — but four things stand in the way,
and the review lists them rather than declaring success.

Two are deployment blockers found by this audit. deploy.sh takes no database dump
before bringing the stack down, which is exactly backwards for a first deploy
where the reconciler will create roughly a dozen tables on a database many
commits behind; BackupController offers only an application-level encrypted
export, not an operational dump. And there is no documented restore procedure —
a backup nobody has restored is a hypothesis.

One is an operational gap: compose defines health checks for ai-service and
ollama but not for backend or frontend, so nothing detects a backend that starts
and then goes unhealthy.

One is the standing external blocker: CI is red for an environmental reason, and
deployment is gated on it.

Also recorded as accepted rather than fixed: console-only logging, no global
exception handler, the intentionally anonymous client-error endpoint, and the
fact that nobody has walked the authenticated end-to-end journey.

Includes a ten-step deployment checklist and a rollback plan. Rollback is safe
because every Phase 4/5 migration is a no-op, so reverting the code never leaves
migration state ahead of the schema — but it does not recover data users create
in the new tables during the window, which the review says plainly.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

docs/production-readiness-review.md

@@ -0,0 +1,150 @@
+# Production readiness review
+
+> 2026-07-19, before the first production deployment after the Phase 4/5 architecture changes.
+> Companion to `docs/phase-5-completion-report.md` and `docs/infrastructure/database-ownership.md`.
+>
+> **Verdict: not ready to deploy unattended.** The application itself verifies clean, but three
+> operational gaps (no pre-deploy backup, no documented restore, no backend health endpoint) and one
+> external blocker (CI) stand between here and a safe first deploy. All are listed below with what
+> would close them.
+
+## Verified
+
+### Database
+
+| Item | Status | Evidence |
+|---|---|---|
+| Fresh empty MariaDB starts | ✅ | 42 tables created, 0 exceptions, app listening |
+| Existing populated MariaDB restarts | ✅ | 42 tables, rows preserved, reconciler idempotent |
+| Existing partially-migrated SQLite upgrades | ✅ | 34 → 44 tables, 13 applications and 8 companies intact |
+| Migration path | ✅ | All Phase 4/5 migrations are no-ops; the reconciler owns their DDL per provider |
+| Ownership model documented | ✅ | `docs/infrastructure/database-ownership.md` |
+| Reconciler never destroys data | ✅ | `DropMalformedMySqlTable` checks row count and skips any table holding rows |
+| Column types on MariaDB | ✅ | `int AUTO_INCREMENT` PKs, `varchar` owner keys, `datetime(6)`, indexes inside the 3072-byte limit |
+
+### Security
+
+| Item | Status | Evidence |
+|---|---|---|
+| Explicit authorization on every controller | ✅ | Reflection test over the whole assembly; public endpoints are an allow-list |
+| No reliance on `Auth:Require` alone | ✅ | The five previously-implicit controllers now declare `[Authorize]` |
+| Admin separation | ✅ | `AdminAudit`, `AdminSystem`, `Users` require `Roles = "Admin"`, asserted by test |
+| Tenant isolation | ✅ | Global query filters are deny-on-null (`CurrentUserId != null && OwnerUserId == CurrentUserId`); every Phase 5 service scopes by owner and returns 404 across tenants |
+| AI endpoints protected | ✅ | `AiWorkspaceController` requires auth; generation refuses another user's application |
+| AI cannot mutate user content | ✅ | Suggestion-only, pinned by four tests |
+| Public CV is user-controlled | ✅ | Off by default, per-variant opt-in, unguessable slug, `noindex` |
+| Uploaded documents are private | ✅ | `AttachmentsController` requires auth and scopes by owner |
+| Secrets are not committed | ✅ | 28 env-var references in compose; `.env` and `.env.*` gitignored; nothing sensitive tracked |
+| Stack traces not leaked | ✅ | No developer exception page outside Development |
+
+### Application
+
+| Item | Status |
+|---|---|
+| Backend tests (Release) | ✅ 393 passed |
+| Frontend tests | ✅ 128 passed, 36 suites |
+| TypeScript | ✅ clean |
+| Frontend production build | ✅ clean |
+| Backend Docker image | ✅ builds |
+| Frontend Docker image | ✅ builds |
+| Backend container against empty MariaDB | ✅ starts, builds schema |
+
+## Remaining risks
+
+### 1. No pre-deploy database backup — **blocker for first deploy**
+
+`deploy/deploy.sh` does not dump the database before bringing the stack down. The first deploy after
+these changes is exactly when a backup matters most: prod is many commits behind, and the reconciler
+will create roughly a dozen tables on first boot. That path is verified on containers, **not on your
+data**.
+
+`BackupController` exposes only `POST /api/backup/encrypted` — an application-level encrypted export,
+not an operational database dump. It is not a substitute.
+
+**To close:** run `mariadb-dump` before `compose down`, keep the dump, and ideally add that step to
+`deploy.sh` so it is not a thing to remember.
+
+### 2. No documented restore procedure — **blocker for first deploy**
+
+There is no written restore path. A backup you have never restored is a hypothesis. `deploy/README.md`
+mentions backups in one line ("keep backups and volume persistence") and stops there.
+
+**To close:** document restore, and rehearse it once against a scratch database.
+
+### 3. No backend health endpoint — **operational gap**
+
+`docker-compose.yml` defines health checks for `ai-service` and `ollama` but **not for `backend` or
+`frontend`**. Nothing automatically detects a backend that started and then became unhealthy, and
+`depends_on` cannot gate on its readiness.
+
+**To close:** add a `/health` endpoint (anonymous, no data) and a compose health check.
+
+### 4. CI is red for an environmental reason — **external blocker**
+
+Unresolved and proven outside the repository: commit `8f73548` changed one markdown file and its test
+job failed in the same duration band as every other run. Deployment is gated on CI, so nothing ships
+until this is resolved. Everything in this review is **local verification**.
+
+Blocked on: job logs (a read-scoped Gitea token), `journalctl -u act_runner`, and the runner container
+config. See `docs/infrastructure/runner-investigation.md`.
+
+### 5. Logging is console-only — **accepted for now**
+
+`builder.Logging` clears providers and adds Console and Debug. Adequate under Docker (captured by
+`docker logs`), but there is no retention, no structure, and no aggregation. Acceptable for a
+self-hosted single-node deployment; revisit if that changes.
+
+### 6. No global exception handler — **minor**
+
+There is no `UseExceptionHandler`, so unhandled exceptions return a bare 500. No stack leaks outside
+Development, so this is not a security issue — but errors reach the client with no correlation id,
+which makes support harder.
+
+### 7. `ClientErrorsController` is anonymous and unauthenticated — **accepted, by design**
+
+It accepts browser error reports and must work on pages reached before sign-in. It is size-limited
+(32 KB) and field-truncated. Worth knowing it is an unauthenticated write path; it stores nothing.
+
+### 8. End-to-end journey unwalked — **verification gap**
+
+No one has driven the real authenticated flow from empty profile through to recorded outcome. All
+verification is tests, builds and startup checks. Friction points in that journey remain unmapped.
+
+## Deployment checklist
+
+Run in order. Stop if any step fails.
+
+1. **Resolve CI**, or make a deliberate decision to deploy from a manually verified commit.
+2. **Back up the production database.**
+   `mariadb-dump -u<user> -p<pass> --single-transaction --routines jobtracker > jobtracker-$(date +%F).sql`
+3. **Verify the dump is non-empty and contains `CREATE TABLE JobApplications`.**
+4. **Note the current commit** for rollback: `git rev-parse HEAD` on the prod checkout.
+5. Confirm `.env` has `AUTH_JWT_KEY`, `AI_SERVICE_TOKEN`, `JOBTRACKER_CONNECTION_STRING`,
+   `DATABASE_PROVIDER=mysql`, and that `Auth__Require=true` is still in `docker-compose.yml`.
+6. Deploy: `deploy/deploy.sh`.
+7. **Watch backend startup logs.** The reconciler logs what it creates. Any unhandled exception at
+   startup means stop and roll back — `Database.Migrate()` throws rather than continuing.
+8. Confirm the expected tables exist:
+   `SELECT COUNT(*) FROM information_schema.tables WHERE table_schema='jobtracker';` — expect ~42.
+9. Sign in and load one application workspace. Check the Overview, Checklist, Timeline, Analysis and
+   Match sections render.
+10. Confirm `/cv/{slug}` still resolves for an existing public variant.
+
+## Rollback plan
+
+**If startup fails or the schema is wrong:**
+
+1. `docker compose down`
+2. Restore the dump: `mariadb -u<user> -p<pass> jobtracker < jobtracker-<date>.sql`
+3. `git checkout <previous-commit>` on the prod checkout
+4. `deploy/deploy.sh`
+5. Verify sign-in works
+
+**Why this is safe:** every Phase 4/5 migration is a no-op, so rolling the code back does not leave
+migration state ahead of the schema. The reconciler only adds tables, columns and indexes — it never
+drops a table holding rows — so the restored database is compatible with the older code, which simply
+ignores the extra tables.
+
+**What rollback does not recover:** anything a user created in the new tables during the window
+(checklist items, cover letter versions, interview prep, AI history). Those tables are dropped by the
+restore. Keep the deploy window short and quiet.