jobtrackingapp/docs/production-readiness-review.md

Production readiness review

Historical snapshot (2026-07-19). The current production blockers are maintained in BLOCKERS.md; current implementation status is in docs/implementation-roadmap.md.

2026-07-19, before the first production deployment after the Phase 4/5 architecture changes. Companion to docs/phase-5-completion-report.md and docs/infrastructure/database-ownership.md.

Updated 2026-07-19 (second pass). The three operational blockers are closed and verified: deploy.sh now takes a verified backup before touching anything and aborts if it fails, restore and rollback are documented and tested against real containers, and backend/frontend both report health.

Remaining verdict: one external blocker. CI is red for an environmental reason and deployment is gated on it. Everything here remains local verification. The accepted-risk items in Remaining risks are unchanged and still worth reading before the first deploy.

Verified

Database

Item	Status	Evidence
Fresh empty MariaDB starts	✅	42 tables created, 0 exceptions, app listening
Existing populated MariaDB restarts	✅	42 tables, rows preserved, reconciler idempotent
Existing partially-migrated SQLite upgrades	✅	34 → 44 tables, 13 applications and 8 companies intact
Migration path	✅	All Phase 4/5 migrations are no-ops; the reconciler owns their DDL per provider
Ownership model documented	✅	docs/infrastructure/database-ownership.md
Reconciler never destroys data	✅	DropMalformedMySqlTable checks row count and skips any table holding rows
Column types on MariaDB	✅	int AUTO_INCREMENT PKs, varchar owner keys, datetime(6), indexes inside the 3072-byte limit

Security

Item	Status	Evidence
Explicit authorization on every controller	✅	Reflection test over the whole assembly; public endpoints are an allow-list
No reliance on Auth:Require alone	✅	The five previously-implicit controllers now declare [Authorize]
Admin separation	✅	AdminAudit, AdminSystem, Users require Roles = "Admin", asserted by test
Tenant isolation	✅	Global query filters are deny-on-null (CurrentUserId != null && OwnerUserId == CurrentUserId); every Phase 5 service scopes by owner and returns 404 across tenants
AI endpoints protected	✅	AiWorkspaceController requires auth; generation refuses another user's application
AI cannot mutate user content	✅	Suggestion-only, pinned by four tests
Public CV is user-controlled	✅	Off by default, per-variant opt-in, unguessable slug, noindex
Uploaded documents are private	✅	AttachmentsController requires auth and scopes by owner
Secrets are not committed	✅	28 env-var references in compose; .env and .env.* gitignored; nothing sensitive tracked
Stack traces not leaked	✅	No developer exception page outside Development

Application

Item	Status
Backend tests (Release)	✅ 393 passed
Frontend tests	✅ 128 passed, 36 suites
TypeScript	✅ clean
Frontend production build	✅ clean
Backend Docker image	✅ builds
Frontend Docker image	✅ builds
Backend container against empty MariaDB	✅ starts, builds schema

Remaining risks

1. No pre-deploy database backup — CLOSED 2026-07-19

Original finding: deploy/deploy.sh did not dump the database before bringing the stack down. The first deploy after these changes is exactly when a backup matters most: prod is many commits behind, and the reconciler will create roughly a dozen tables on first boot. That path is verified on containers, not on your data.

BackupController exposes only POST /api/backup/encrypted — an application-level encrypted export, not an operational database dump. It is not a substitute.

Closed. deploy/deploy.sh now backs up before it builds, stops or replaces anything, and aborts the deploy if the backup fails. Timestamped and gzipped to /opt/job-tracker/backups (override with BACKUP_DIR), so a deploy never overwrites an earlier backup. The password travels via MYSQL_PWD, never on the command line. The dump is rejected if it is empty or lacks CREATE TABLE. SQLite deployments get the data volume tarred instead.

Verified: dump taken from a seeded MariaDB 11 container; failure paths (bad credentials, missing connection string) abort with a non-zero status and leave no misleading partial file.

2. No documented restore procedure — CLOSED 2026-07-19

Original finding: there was no written restore path. A backup you have never restored is a hypothesis. deploy/README.md mentions backups in one line ("keep backups and volume persistence") and stops there.

Closed. deploy/README.md documents backup creation, location, retention, database restore, application rollback, and — importantly — when to use which. Restore and rollback are presented as separate operations, because a bad deploy usually needs only the rollback.

Verified: the dump was restored into a clean MariaDB 11 container and the rows came back identical. The rehearsal is the documented example.

3. No backend health endpoint — CLOSED 2026-07-19

Original finding: docker-compose.yml defined health checks for ai-service and ollama but not for backend or frontend. Nothing automatically detects a backend that started and then became unhealthy, and depends_on cannot gate on its readiness.

Closed. GET /health is anonymous, cheap, and deliberately does not touch the database — a health check that queried MariaDB would restart a healthy backend whenever the database blipped, and would hand out an unauthenticated way to probe database availability. Compose health checks now cover backend (curl, 90s start period for first-boot reconciliation) and frontend (wget against nginx), and frontend waits for backend to be healthy rather than merely started.

Verified: both containers reach healthy; a backend pointed at an unreachable database exits and is reported unhealthy, so a broken deploy cannot present as a running stack. /health returns 200 anonymously while /api/jobapplications returns 401 with Auth:Require unset, confirming the explicit [Authorize] work holds independently of that flag.

4. CI is red for an environmental reason — external blocker

Unresolved and proven outside the repository: commit 8f73548 changed one markdown file and its test job failed in the same duration band as every other run. Deployment is gated on CI, so nothing ships until this is resolved. Everything in this review is local verification.

Blocked on: job logs (a read-scoped Gitea token), journalctl -u act_runner, and the runner container config. See docs/infrastructure/runner-investigation.md.

5. Logging is console-only — accepted for now

builder.Logging clears providers and adds Console and Debug. Adequate under Docker (captured by docker logs), and production output is structured JSON, but there is no configured retention or aggregation. Acceptable for a self-hosted single-node deployment; revisit if that changes.

6. No global exception handler — minor

There is no UseExceptionHandler, so unhandled exceptions return a bare 500. No stack leaks outside Development, so this is not a security issue — but errors reach the client with no correlation id, which makes support harder.

7. ClientErrorsController is anonymous and unauthenticated — accepted, by design

It accepts browser error reports and must work on pages reached before sign-in. It is size-limited (32 KB) and field-truncated. Worth knowing it is an unauthenticated write path; it stores nothing.

8. End-to-end journey unwalked — verification gap

No one has driven the real authenticated flow from empty profile through to recorded outcome. All verification is tests, builds and startup checks. Friction points in that journey remain unmapped.

Deployment checklist

Run in order. Stop if any step fails.

1. Resolve CI, or make a deliberate decision to deploy from a manually verified commit.
2. Back up the production database. deploy.sh now does this automatically and aborts if it fails. Confirm afterwards that a new file appeared in /opt/job-tracker/backups.
3. Verify the dump. The script verifies size and content before continuing.
4. Note the current commit for rollback: git rev-parse HEAD on the prod checkout.
5. Confirm .env has AUTH_JWT_KEY, AI_SERVICE_TOKEN, JOBTRACKER_CONNECTION_STRING, DATABASE_PROVIDER=mysql, and that Auth__Require=true is still in docker-compose.yml.
6. Deploy: deploy/deploy.sh.
7. Watch backend startup logs. The reconciler logs what it creates. Any unhandled exception at startup means stop and roll back — Database.Migrate() throws rather than continuing.
8. Confirm the expected tables exist: SELECT COUNT(*) FROM information_schema.tables WHERE table_schema='jobtracker'; — expect ~42.
9. Sign in and load one application workspace. Check the Overview, Checklist, Timeline, Analysis and Match sections render.
10. Confirm /cv/{slug} still resolves for an existing public variant.

Rollback plan

If startup fails or the schema is wrong:

1. docker compose down
2. Restore the dump: mariadb -u<user> -p<pass> jobtracker < jobtracker-<date>.sql
3. git checkout <previous-commit> on the prod checkout
4. deploy/deploy.sh
5. Verify sign-in works

Why this is safe: every Phase 4/5 migration is a no-op, so rolling the code back does not leave migration state ahead of the schema. The reconciler only adds tables, columns and indexes — it never drops a table holding rows — so the restored database is compatible with the older code, which simply ignores the extra tables.

What rollback does not recover: anything a user created in the new tables during the window (checklist items, cover letter versions, interview prep, AI history). Those tables are dropped by the restore. Keep the deploy window short and quiet.