jobtrackingapp/docs/production-readiness-review.md

Production readiness review

2026-07-19, before the first production deployment after the Phase 4/5 architecture changes. Companion to docs/phase-5-completion-report.md and docs/infrastructure/database-ownership.md.

Verdict: not ready to deploy unattended. The application itself verifies clean, but three operational gaps (no pre-deploy backup, no documented restore, no backend health endpoint) and one external blocker (CI) stand between here and a safe first deploy. All are listed below with what would close them.

Verified

Database

Item	Status	Evidence
Fresh empty MariaDB starts	✅	42 tables created, 0 exceptions, app listening
Existing populated MariaDB restarts	✅	42 tables, rows preserved, reconciler idempotent
Existing partially-migrated SQLite upgrades	✅	34 → 44 tables, 13 applications and 8 companies intact
Migration path	✅	All Phase 4/5 migrations are no-ops; the reconciler owns their DDL per provider
Ownership model documented	✅	docs/infrastructure/database-ownership.md
Reconciler never destroys data	✅	DropMalformedMySqlTable checks row count and skips any table holding rows
Column types on MariaDB	✅	int AUTO_INCREMENT PKs, varchar owner keys, datetime(6), indexes inside the 3072-byte limit

Security

Item	Status	Evidence
Explicit authorization on every controller	✅	Reflection test over the whole assembly; public endpoints are an allow-list
No reliance on Auth:Require alone	✅	The five previously-implicit controllers now declare [Authorize]
Admin separation	✅	AdminAudit, AdminSystem, Users require Roles = "Admin", asserted by test
Tenant isolation	✅	Global query filters are deny-on-null (CurrentUserId != null && OwnerUserId == CurrentUserId); every Phase 5 service scopes by owner and returns 404 across tenants
AI endpoints protected	✅	AiWorkspaceController requires auth; generation refuses another user's application
AI cannot mutate user content	✅	Suggestion-only, pinned by four tests
Public CV is user-controlled	✅	Off by default, per-variant opt-in, unguessable slug, noindex
Uploaded documents are private	✅	AttachmentsController requires auth and scopes by owner
Secrets are not committed	✅	28 env-var references in compose; .env and .env.* gitignored; nothing sensitive tracked
Stack traces not leaked	✅	No developer exception page outside Development

Application

Item	Status
Backend tests (Release)	✅ 393 passed
Frontend tests	✅ 128 passed, 36 suites
TypeScript	✅ clean
Frontend production build	✅ clean
Backend Docker image	✅ builds
Frontend Docker image	✅ builds
Backend container against empty MariaDB	✅ starts, builds schema

Remaining risks

1. No pre-deploy database backup — blocker for first deploy

deploy/deploy.sh does not dump the database before bringing the stack down. The first deploy after these changes is exactly when a backup matters most: prod is many commits behind, and the reconciler will create roughly a dozen tables on first boot. That path is verified on containers, not on your data.

BackupController exposes only POST /api/backup/encrypted — an application-level encrypted export, not an operational database dump. It is not a substitute.

To close: run mariadb-dump before compose down, keep the dump, and ideally add that step to deploy.sh so it is not a thing to remember.

2. No documented restore procedure — blocker for first deploy

There is no written restore path. A backup you have never restored is a hypothesis. deploy/README.md mentions backups in one line ("keep backups and volume persistence") and stops there.

To close: document restore, and rehearse it once against a scratch database.

3. No backend health endpoint — operational gap

docker-compose.yml defines health checks for ai-service and ollama but not for backend or frontend. Nothing automatically detects a backend that started and then became unhealthy, and depends_on cannot gate on its readiness.

To close: add a /health endpoint (anonymous, no data) and a compose health check.

4. CI is red for an environmental reason — external blocker

Unresolved and proven outside the repository: commit 8f73548 changed one markdown file and its test job failed in the same duration band as every other run. Deployment is gated on CI, so nothing ships until this is resolved. Everything in this review is local verification.

Blocked on: job logs (a read-scoped Gitea token), journalctl -u act_runner, and the runner container config. See docs/infrastructure/runner-investigation.md.

5. Logging is console-only — accepted for now

builder.Logging clears providers and adds Console and Debug. Adequate under Docker (captured by docker logs), but there is no retention, no structure, and no aggregation. Acceptable for a self-hosted single-node deployment; revisit if that changes.

6. No global exception handler — minor

There is no UseExceptionHandler, so unhandled exceptions return a bare 500. No stack leaks outside Development, so this is not a security issue — but errors reach the client with no correlation id, which makes support harder.

7. ClientErrorsController is anonymous and unauthenticated — accepted, by design

It accepts browser error reports and must work on pages reached before sign-in. It is size-limited (32 KB) and field-truncated. Worth knowing it is an unauthenticated write path; it stores nothing.

8. End-to-end journey unwalked — verification gap

No one has driven the real authenticated flow from empty profile through to recorded outcome. All verification is tests, builds and startup checks. Friction points in that journey remain unmapped.

Deployment checklist

Run in order. Stop if any step fails.

1. Resolve CI, or make a deliberate decision to deploy from a manually verified commit.
2. Back up the production database. mariadb-dump -u<user> -p<pass> --single-transaction --routines jobtracker > jobtracker-$(date +%F).sql
3. Verify the dump is non-empty and contains CREATE TABLE JobApplications.
4. Note the current commit for rollback: git rev-parse HEAD on the prod checkout.
5. Confirm .env has AUTH_JWT_KEY, AI_SERVICE_TOKEN, JOBTRACKER_CONNECTION_STRING, DATABASE_PROVIDER=mysql, and that Auth__Require=true is still in docker-compose.yml.
6. Deploy: deploy/deploy.sh.
7. Watch backend startup logs. The reconciler logs what it creates. Any unhandled exception at startup means stop and roll back — Database.Migrate() throws rather than continuing.
8. Confirm the expected tables exist: SELECT COUNT(*) FROM information_schema.tables WHERE table_schema='jobtracker'; — expect ~42.
9. Sign in and load one application workspace. Check the Overview, Checklist, Timeline, Analysis and Match sections render.
10. Confirm /cv/{slug} still resolves for an existing public variant.

Rollback plan

If startup fails or the schema is wrong:

1. docker compose down
2. Restore the dump: mariadb -u<user> -p<pass> jobtracker < jobtracker-<date>.sql
3. git checkout <previous-commit> on the prod checkout
4. deploy/deploy.sh
5. Verify sign-in works

Why this is safe: every Phase 4/5 migration is a no-op, so rolling the code back does not leave migration state ahead of the schema. The reconciler only adds tables, columns and indexes — it never drops a table holding rows — so the restored database is compatible with the older code, which simply ignores the extra tables.

What rollback does not recover: anything a user created in the new tables during the window (checklist items, cover letter versions, interview prep, AI history). Those tables are dropped by the restore. Keep the deploy window short and quiet.