jobtrackingapp/docs/production-readiness-review.md

# Production readiness review

> 2026-07-19, before the first production deployment after the Phase 4/5 architecture changes.
> Companion to `docs/phase-5-completion-report.md` and `docs/infrastructure/database-ownership.md`.
>
> **Updated 2026-07-19 (second pass).** The three operational blockers are **closed and verified**:
> `deploy.sh` now takes a verified backup before touching anything and aborts if it fails, restore and
> rollback are documented and tested against real containers, and `backend`/`frontend` both report
> health.
>
> **Remaining verdict: one external blocker.** CI is red for an environmental reason and deployment is
> gated on it. Everything here remains local verification. The accepted-risk items in *Remaining risks*
> are unchanged and still worth reading before the first deploy.

## Verified

### Database

| Item | Status | Evidence |
|---|---|---|
| Fresh empty MariaDB starts | ✅ | 42 tables created, 0 exceptions, app listening |
| Existing populated MariaDB restarts | ✅ | 42 tables, rows preserved, reconciler idempotent |
| Existing partially-migrated SQLite upgrades | ✅ | 34 → 44 tables, 13 applications and 8 companies intact |
| Migration path | ✅ | All Phase 4/5 migrations are no-ops; the reconciler owns their DDL per provider |
| Ownership model documented | ✅ | `docs/infrastructure/database-ownership.md` |
| Reconciler never destroys data | ✅ | `DropMalformedMySqlTable` checks row count and skips any table holding rows |
| Column types on MariaDB | ✅ | `int AUTO_INCREMENT` PKs, `varchar` owner keys, `datetime(6)`, indexes inside the 3072-byte limit |

### Security

| Item | Status | Evidence |
|---|---|---|
| Explicit authorization on every controller | ✅ | Reflection test over the whole assembly; public endpoints are an allow-list |
| No reliance on `Auth:Require` alone | ✅ | The five previously-implicit controllers now declare `[Authorize]` |
| Admin separation | ✅ | `AdminAudit`, `AdminSystem`, `Users` require `Roles = "Admin"`, asserted by test |
| Tenant isolation | ✅ | Global query filters are deny-on-null (`CurrentUserId != null && OwnerUserId == CurrentUserId`); every Phase 5 service scopes by owner and returns 404 across tenants |
| AI endpoints protected | ✅ | `AiWorkspaceController` requires auth; generation refuses another user's application |
| AI cannot mutate user content | ✅ | Suggestion-only, pinned by four tests |
| Public CV is user-controlled | ✅ | Off by default, per-variant opt-in, unguessable slug, `noindex` |
| Uploaded documents are private | ✅ | `AttachmentsController` requires auth and scopes by owner |
| Secrets are not committed | ✅ | 28 env-var references in compose; `.env` and `.env.*` gitignored; nothing sensitive tracked |
| Stack traces not leaked | ✅ | No developer exception page outside Development |

### Application

| Item | Status |
|---|---|
| Backend tests (Release) | ✅ 393 passed |
| Frontend tests | ✅ 128 passed, 36 suites |
| TypeScript | ✅ clean |
| Frontend production build | ✅ clean |
| Backend Docker image | ✅ builds |
| Frontend Docker image | ✅ builds |
| Backend container against empty MariaDB | ✅ starts, builds schema |

## Remaining risks

### 1. ~~No pre-deploy database backup~~ — **CLOSED 2026-07-19**

*Original finding:* `deploy/deploy.sh` did not dump the database before bringing the stack down. The first deploy after
these changes is exactly when a backup matters most: prod is many commits behind, and the reconciler
will create roughly a dozen tables on first boot. That path is verified on containers, **not on your
data**.

`BackupController` exposes only `POST /api/backup/encrypted` — an application-level encrypted export,
not an operational database dump. It is not a substitute.

**Closed.** `deploy/deploy.sh` now backs up before it builds, stops or replaces anything, and aborts
the deploy if the backup fails. Timestamped and gzipped to `/opt/job-tracker/backups` (override with
`BACKUP_DIR`), so a deploy never overwrites an earlier backup. The password travels via `MYSQL_PWD`,
never on the command line. The dump is rejected if it is empty or lacks `CREATE TABLE`. SQLite
deployments get the data volume tarred instead.

*Verified:* dump taken from a seeded MariaDB 11 container; failure paths (bad credentials, missing
connection string) abort with a non-zero status and leave no misleading partial file.

### 2. ~~No documented restore procedure~~ — **CLOSED 2026-07-19**

*Original finding:* there was no written restore path. A backup you have never restored is a hypothesis. `deploy/README.md`
mentions backups in one line ("keep backups and volume persistence") and stops there.

**Closed.** `deploy/README.md` documents backup creation, location, retention, database restore,
application rollback, and — importantly — when to use which. Restore and rollback are presented as
separate operations, because a bad deploy usually needs only the rollback.

*Verified:* the dump was restored into a clean MariaDB 11 container and the rows came back identical.
The rehearsal is the documented example.

### 3. ~~No backend health endpoint~~ — **CLOSED 2026-07-19**

*Original finding:* `docker-compose.yml` defined health checks for `ai-service` and `ollama` but not for
`backend` or `frontend`. Nothing automatically detects a backend that started and then became unhealthy, and
`depends_on` cannot gate on its readiness.

**Closed.** `GET /health` is anonymous, cheap, and deliberately **does not touch the database** — a
health check that queried MariaDB would restart a healthy backend whenever the database blipped, and
would hand out an unauthenticated way to probe database availability. Compose health checks now cover
`backend` (curl, 90s start period for first-boot reconciliation) and `frontend` (wget against nginx),
and `frontend` waits for `backend` to be *healthy* rather than merely started.

*Verified:* both containers reach `healthy`; a backend pointed at an unreachable database exits and is
reported `unhealthy`, so a broken deploy cannot present as a running stack. `/health` returns 200
anonymously while `/api/jobapplications` returns 401 **with `Auth:Require` unset**, confirming the
explicit `[Authorize]` work holds independently of that flag.

### 4. CI is red for an environmental reason — **external blocker**

Unresolved and proven outside the repository: commit `8f73548` changed one markdown file and its test
job failed in the same duration band as every other run. Deployment is gated on CI, so nothing ships
until this is resolved. Everything in this review is **local verification**.

Blocked on: job logs (a read-scoped Gitea token), `journalctl -u act_runner`, and the runner container
config. See `docs/infrastructure/runner-investigation.md`.

### 5. Logging is console-only — **accepted for now**

`builder.Logging` clears providers and adds Console and Debug. Adequate under Docker (captured by
`docker logs`), but there is no retention, no structure, and no aggregation. Acceptable for a
self-hosted single-node deployment; revisit if that changes.

### 6. No global exception handler — **minor**

There is no `UseExceptionHandler`, so unhandled exceptions return a bare 500. No stack leaks outside
Development, so this is not a security issue — but errors reach the client with no correlation id,
which makes support harder.

### 7. `ClientErrorsController` is anonymous and unauthenticated — **accepted, by design**

It accepts browser error reports and must work on pages reached before sign-in. It is size-limited
(32 KB) and field-truncated. Worth knowing it is an unauthenticated write path; it stores nothing.

### 8. End-to-end journey unwalked — **verification gap**

No one has driven the real authenticated flow from empty profile through to recorded outcome. All
verification is tests, builds and startup checks. Friction points in that journey remain unmapped.

## Deployment checklist

Run in order. Stop if any step fails.

1. **Resolve CI**, or make a deliberate decision to deploy from a manually verified commit.
2. ~~Back up the production database.~~ **`deploy.sh` now does this automatically and aborts if it
   fails.** Confirm afterwards that a new file appeared in `/opt/job-tracker/backups`.
3. ~~Verify the dump.~~ **The script verifies size and content before continuing.**
4. **Note the current commit** for rollback: `git rev-parse HEAD` on the prod checkout.
5. Confirm `.env` has `AUTH_JWT_KEY`, `AI_SERVICE_TOKEN`, `JOBTRACKER_CONNECTION_STRING`,
   `DATABASE_PROVIDER=mysql`, and that `Auth__Require=true` is still in `docker-compose.yml`.
6. Deploy: `deploy/deploy.sh`.
7. **Watch backend startup logs.** The reconciler logs what it creates. Any unhandled exception at
   startup means stop and roll back — `Database.Migrate()` throws rather than continuing.
8. Confirm the expected tables exist:
   `SELECT COUNT(*) FROM information_schema.tables WHERE table_schema='jobtracker';` — expect ~42.
9. Sign in and load one application workspace. Check the Overview, Checklist, Timeline, Analysis and
   Match sections render.
10. Confirm `/cv/{slug}` still resolves for an existing public variant.

## Rollback plan

**If startup fails or the schema is wrong:**

1. `docker compose down`
2. Restore the dump: `mariadb -u<user> -p<pass> jobtracker < jobtracker-<date>.sql`
3. `git checkout <previous-commit>` on the prod checkout
4. `deploy/deploy.sh`
5. Verify sign-in works

**Why this is safe:** every Phase 4/5 migration is a no-op, so rolling the code back does not leave
migration state ahead of the schema. The reconciler only adds tables, columns and indexes — it never
drops a table holding rows — so the restored database is compatible with the older code, which simply
ignores the extra tables.

**What rollback does not recover:** anything a user created in the new tables during the window
(checklist items, cover letter versions, interview prep, AI history). Those tables are dropped by the
restore. Keep the deploy window short and quiet.