Written against the actual implementation rather than the existing docs, and
validated locally against MariaDB 11 containers. No application behaviour
changed — this commit adds two documents.
deploy/first-production-deployment.md covers pre-deployment checks, the eight
deployment steps, smoke tests for backend, database and application, and
rollback. It documents what deploy.sh really does: it backs up first and aborts
on failure, and it replaces containers with up -d --force-recreate rather than
running compose down, so the window is container start time. It also records the
startup sequence as implemented — reconcile, migrate, reconcile — and that
Database.Migrate() throws rather than limping on.
Validation surfaced things worth writing down. The connection string resolves
from inside the backend container, so Server=127.0.0.1 means the container and
not the host; this broke a validation run before it could have broken a deploy.
DATABASE_PROVIDER defaults to sqlite, and if it goes missing the backend does not
quietly serve an empty database — it exits with "no such table:
INFORMATION_SCHEMA.TABLES", which is loud but baffling if unexplained. A blank
AUTH_JWT_KEY throws at startup when auth is required, which is the right
behaviour. The runbook maps each of these log lines to its cause.
Rollback is documented with the distinction stated plainly: a code rollback
keeps all data and is almost always the whole fix, while a database restore
discards everything written since the dump. Restore only when the data itself is
wrong.
docs/release-checklist.md records the completed architecture work, local
verification results, known risks with severities, the unresolved CI runner
blocker and what would unblock it, and seven first-deployment warnings.
Validated: compose build; compose up on a fresh MariaDB (42 tables, backend
healthy); the depends_on health gate holding the frontend until the backend is
healthy; restart against the populated database with rows preserved; backup and
restore; and the failure paths. Not validated, and said so in both documents: the
authenticated end-to-end journey, because signing in needs a password.
393 backend tests pass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Closes the three operational blockers from the production readiness review.
deploy.sh now takes a database backup before it builds, stops or replaces
anything, and aborts the deploy if the backup fails — so no deploy proceeds
without a restore point. Dumps are gzipped and timestamped into
/opt/job-tracker/backups (override with BACKUP_DIR), so one deploy never
overwrites an earlier backup. Credentials come from the existing connection
string and travel via MYSQL_PWD, never on the command line, so they cannot reach
the process list or the deploy log. A dump that is empty or missing CREATE TABLE
is rejected, because a truncated file that looks like a restore point is worse
than none. SQLite deployments get their data volume tarred instead. Nothing is
ever deleted automatically; retention is documented as manual.
deploy/README.md documents backup creation, location, retention, database
restore, application rollback, and when to use which — restore and rollback kept
distinct, because a bad deploy usually needs only the rollback and restoring
would discard everything written since the dump.
Health checks now cover backend and frontend, which previously had none. GET
/health is anonymous, cheap, and deliberately does not touch the database: a
health check that queried MariaDB would restart a healthy backend whenever the
database blipped, and would hand out an unauthenticated way to probe database
availability. The backend image gains curl on the existing chromium apt layer,
since the aspnet runtime ships neither curl nor wget. frontend now waits for
backend to be healthy rather than merely started, because nginx proxies /api and
refuses to start when the upstream cannot be resolved.
Verified against real containers, no production data: backup from a seeded
MariaDB 11; restore into a clean MariaDB 11 with rows identical; bad credentials
and a missing connection string both abort non-zero and leave no partial file;
SQLite volume backup produces a readable archive; backend and frontend both
reach healthy; and a backend pointed at an unreachable database exits and is
reported unhealthy, so a broken deploy cannot present as a running stack.
Incidentally confirmed the earlier authorization work: with Auth:Require unset,
/health returns 200 while /api/jobapplications returns 401.
393 backend tests pass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The compose file shipped its own ollama service, so 'docker compose pull'
during deploy re-downloaded the Ollama image and a deploy that starts the
AI stack would spin up a second Ollama alongside an existing/shared one.
- ollama service moved behind a 'bundled-ollama' compose profile, so it is
excluded from the default pull/up (no duplicate, no re-download)
- ai-service no longer depends_on ollama and is documented to point at a
shared instance via OLLAMA_BASE_URL (e.g. http://<host>:11435)
- deploy.sh no longer names ollama in 'compose up'
To run a self-contained Ollama: docker compose --profile bundled-ollama up
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
cesnimda
cesnimda