jobtrackingapp/deploy/first-production-deployment.md

First production deployment runbook

Written 2026-07-19 against the actual implementation, not against the other docs. Every command and failure mode below was exercised locally against MariaDB 11 containers. No production data was used.

This is the first deploy after the Phase 4/5 architecture changes. Production is many commits behind and the startup reconciler will create roughly a dozen tables against real data for the first time. Read Pre-deployment fully before starting.

What actually happens on deploy

Verified by reading deploy/deploy.sh and JobTrackerApi/Program.cs:

1. deploy.sh links /opt/job-tracker/shared/.env into the checkout as .env, and loads it into its own shell. The link is for docker compose; the script needs the values itself to pick the right backup. Values already in the environment (CI's APP_VERSION and friends) win.
2. It validates the deployment configuration, before anything is built, stopped or replaced: DATABASE_PROVIDER (required, no default), the connection string when that provider needs one, AI_SERVICE_TOKEN and AUTH_JWT_KEY. A missing variable aborts the deploy with the running stack untouched. Names are printed, never values.
3. It takes a database backup and aborts if that fails. Nothing else runs without a restore point. The provider chooses the backup: mariadb/mysql gives a .sql.gz dump, sqlite gives a .tar.gz of the data volume. Each is verified against its own format — the dump must contain CREATE TABLE and the Dump completed trailer; the archive must contain jobtracker.db.
4. docker compose pull, then builds backend and frontend (with one prune-and-retry on failure).
5. docker compose up -d --force-recreate --remove-orphans backend frontend. There is no compose down — containers are replaced in place, so the window is short.
6. On backend start, InitializeJobTrackerAsync runs: reconcile → Database.Migrate() → reconcile. Every Phase 4/5 migration is a no-op; the reconciler creates those tables with correct per-provider DDL. Migrate() throws on failure, so a schema problem exits the container rather than limping on.
7. deploy.sh waits, then fails the deploy if backend is not running, and runs a public smoke check against APP_PUBLIC_BASE_URL.

---

Pre-deployment

• Database backup verified. deploy.sh takes one automatically, but for the first deploy take one by hand as well and confirm it restores — see deploy/README.md. A backup you have never restored is a hypothesis.
• Disk space checked. df -h on the host. You need room for the backup, two image sets during the build, and the build cache. docker system df shows what Docker is holding.
• Environment variables present. deploy.sh now checks these itself and aborts before it builds or replaces anything, so a miss costs an aborted deploy rather than a broken one. Check first anyway and skip the round trip — /opt/job-tracker/shared/.env must contain: - DATABASE_PROVIDER=mariadb (or mysql) — required, no default. An absent value used to mean "sqlite", which silently produced the wrong backup; it now stops the deploy - JOBTRACKER_CONNECTION_STRING — see the host-resolution note below - AI_SERVICE_TOKEN — compose refuses to start without it - AUTH_JWT_KEY — with Auth__Require=true, a blank key throws at startup (this is good; it fails loud rather than silently invalidating every session on restart) - APP_PUBLIC_BASE_URL — optional; without it the post-deploy public smoke check is skipped, and the script prints that it is skipping - AUTH_ADMIN_EMAIL / AUTH_ADMIN_PASSWORD only if you want admin seeding on this boot
• Connection string host resolves from inside the container. Server=127.0.0.1 means the backend container, not the host — this bit me during validation. Use the host's LAN address, a shared Docker network alias, or host.docker.internal where supported.
• Secrets available. Confirm .env is the real shared file and not a stale copy: ls -l /opt/job-tracker/shared/.env.
• Docker healthy. docker info succeeds; docker ps shows the current stack running.
• Current version recorded — you need this to roll back: bash cd /opt/job-tracker/app git rev-parse HEAD | tee /tmp/jobtracker-rollback-commit docker compose ps
• Quiet window chosen. Rows users create in the new tables during the deploy are lost if you later restore the database.

---

Deployment steps

1. Backup

Automatic — deploy.sh runs it first and aborts on failure. Confirm afterwards:

ls -lt /opt/job-tracker/backups | head -3

Expect a new jobtracker-<db>-<UTC timestamp>.sql.gz. The script already rejected it if it were empty, not valid gzip, missing CREATE TABLE, or missing the Dump completed trailer.

Check the filename, not just that a file appeared. A jobtracker-sqlite-<stamp>.tar.gz on a MariaDB deploy means the environment is wrong — that is the exact failure the provider check exists to prevent, and it should now abort rather than reach this point.

2. Pull code

cd /opt/job-tracker/app
git fetch --all
git log --oneline HEAD..origin/main | head -20   # read what you are about to deploy
git checkout main && git pull --ff-only

3–5. Build, replace containers

deploy/deploy.sh

This builds, then up -d --force-recreate for backend and frontend. Old containers are replaced, not stopped first, so downtime is roughly container start time.

6–7. Database startup and reconciler

Watch it. This is the step that matters on a first deploy:

docker compose logs -f backend

Healthy looks like: EF migration lines, then Now listening on: http://[::]:8080 and Application started.

Stop and roll back if you see:

Log line	Meaning
Unhandled exception ... Specified key was too long	A migration ran that should be a no-op
Unhandled exception ... Table '...' doesn't exist	Reconciler ordering problem
no such table: INFORMATION_SCHEMA.TABLES	DATABASE_PROVIDER=mysql but the connection string is empty — verified failure mode
Unable to connect to any of the specified MySQL hosts	Connection string host unreachable from inside the container
Auth is required but Auth:JwtKey is not configured	AUTH_JWT_KEY missing from .env — deploy.sh should now catch this before the build

8. Health verification

docker compose ps

Both backend and frontend should read (healthy), not merely Up. The frontend waits for the backend to be healthy before it starts, because nginx proxies /api and refuses to boot if the upstream cannot be resolved.

---

Verification

Backend

# Health endpoint — anonymous, does not touch the database
curl -fsS https://<host>/health
# expect: {"status":"ok","version":"<the APP_VERSION you deployed>"}
#
# The version comes from App:Version, which compose passes as App__Version from APP_VERSION.
# CI sets it to the workflow run number. A version of "1.0.0.0" (the assembly fallback) means
# APP_VERSION did not reach the container — harmless in itself, but it tells you the build
# metadata is not flowing, so the admin system page will be vague about what is deployed.

# Auth still enforced (this is the check that proves the API is not open)
curl -s -o /dev/null -w '%{http_code}\n' https://<host>/api/jobapplications
# expect: 401

# Auth config responds as JSON
curl -fsS https://<host>/api/auth/config | head -c 200
# expect JSON containing requireAuth

• Existing user login works. Sign in with a real account in a browser. Do this yourself — it needs a password, and no automated step here should handle one.
• API access after login. The applications list loads with your real data.

Database

MYSQL_PWD='<password>' mariadb --host=<host> --user=<user> jobtracker -e "
  SELECT COUNT(*) AS tables FROM information_schema.tables WHERE table_schema='jobtracker';
  SELECT COUNT(*) AS applications FROM JobApplications;
  SELECT COUNT(*) AS companies FROM Companies;
  SHOW TABLES LIKE 'InterviewPrepItems';
"

• Tables created — expect ~42, including CvVariants, AiInteractions, ApplicationChecklistItems, CoverLetterVersions, InterviewPrepItems.
• Existing rows preserved — application and company counts match what you saw before the deploy. This is the single most important check.

Application

• Frontend loads at the public URL.
• Existing applications visible with the correct count.
• CV system available — the CV builder lists existing variants; open one.
• Workspace available — open an application, check Overview, Checklist, Timeline, Analysis and Match render. New sections start empty for existing applications; that is correct, not a fault.
• Public CV route works — open /cv/<slug> for a variant already marked public. If none is public, mark one, check it, then unmark it.

---

Rollback

When to roll back

Roll back if any of these are true:

• Backend unavailable — container exits, restarts in a loop, or never reports healthy
• Migration or reconciler failure — any unhandled exception in the startup log
• Data integrity issue — row counts dropped, or existing applications are missing
• Frontend unusable — will not load, or cannot reach the API

Do not roll back for a cosmetic problem or an empty new section. Empty is expected.

Code rollback is not database rollback

These are different operations and you usually want only the first.

A code rollback reverts the application and leaves all data intact. A database restore discards everything written since the dump — including anything users did during and after the deploy.

Restore the database only if the data itself is wrong. If the backend simply will not start, the code rollback is almost certainly the whole fix.

Why a code rollback is safe here: every Phase 4/5 migration is a no-op — the reconciler owns those tables — so reverting code never leaves migration history ahead of the schema. The reconciler is additive and never drops a table holding rows, so the older code simply ignores the newer tables.

Procedure

# 1. Stop the new version
cd /opt/job-tracker/app
docker compose stop backend frontend

# 2. Restore the previous containers (code rollback)
git checkout "$(cat /tmp/jobtracker-rollback-commit)"
deploy/deploy.sh          # takes a fresh backup first, so the rollback is itself protected

# 3. Database restore decision — ONLY if data is wrong. Skip otherwise.
#    docker compose stop backend
#    gzip -dc /opt/job-tracker/backups/<file>.sql.gz \
#      | MYSQL_PWD='<password>' mariadb --host=<host> --user=<user> jobtracker
#    docker compose start backend

# 4. Verification
docker compose ps                                   # both healthy
curl -fsS https://<host>/health

Then re-run the Verification section above. Confirm row counts and sign-in before walking away.

---

Validation performed for this runbook

Against MariaDB 11 containers, no production data:

Check	Result
docker compose build backend frontend	✅ both images built
docker compose up -d on a fresh MariaDB	✅ 42 tables created, backend healthy
depends_on: service_healthy gate	✅ frontend waited for backend health, then started healthy
Restart against the now-populated database	✅ still 42 tables, seeded row preserved
Backend health endpoint	✅ 200 anonymously; /api/jobapplications 401 with Auth:Require unset
Backup against a seeded MariaDB	✅ verified dump written
Restore into a clean MariaDB	✅ rows identical
Backup failure paths	✅ bad credentials and missing connection string both abort, no partial file
Shared .env loaded into the deploy shell	✅ MariaDB path selected from the file alone; dump contains schema, rows and trailer
DATABASE_PROVIDER missing or unrecognised	✅ deploy stops before build/replace, names the variable, leaves no backup file
AI_SERVICE_TOKEN / AUTH_JWT_KEY missing	✅ both reported in one pass, deploy stops
SQLite volume backup, and its failure paths	✅ valid archive verified; a volume with no jobtracker.db, and a volume name that does not exist, both abort
Secret leakage in deploy output	✅ zero occurrences of any password, token or key across every test
Empty connection string with provider=mysql	✅ fails loudly (no such table: INFORMATION_SCHEMA.TABLES) rather than silently serving an empty database
Backend with unreachable database	✅ exits, reported unhealthy

Not validated: the authenticated end-to-end user journey. Signing in requires a password, so the browser checks in Verification are yours to perform.