# First production deployment runbook > Written 2026-07-19 against the actual implementation, not against the other docs. Every command and > failure mode below was exercised locally against MariaDB 11 containers. No production data was used. > > This is the **first** deploy after the Phase 4/5 architecture changes. Production is many commits > behind and the startup reconciler will create roughly a dozen tables against real data for the first > time. Read *Pre-deployment* fully before starting. ## What actually happens on deploy Verified by reading `deploy/deploy.sh` and `JobTrackerApi/Program.cs`: 1. `deploy.sh` links `/opt/job-tracker/shared/.env` into the checkout as `.env`, **and loads it into its own shell.** The link is for docker compose; the script needs the values itself to pick the right backup. Values already in the environment (CI's `APP_VERSION` and friends) win. 2. **It validates the deployment configuration**, before anything is built, stopped or replaced: `DATABASE_PROVIDER` (required, no default), the connection string when that provider needs one, `AI_SERVICE_TOKEN` and `AUTH_JWT_KEY`. A missing variable aborts the deploy with the running stack untouched. Names are printed, never values. 3. **It takes a database backup and aborts if that fails.** Nothing else runs without a restore point. The provider chooses the backup: `mariadb`/`mysql` gives a `.sql.gz` dump, `sqlite` gives a `.tar.gz` of the data volume. Each is verified against its own format — the dump must contain `CREATE TABLE` and the `Dump completed` trailer; the archive must contain `jobtracker.db`. 4. `docker compose pull`, then builds `backend` and `frontend` (with one prune-and-retry on failure). 5. `docker compose up -d --force-recreate --remove-orphans backend frontend`. **There is no `compose down`** — containers are replaced in place, so the window is short. 6. On backend start, `InitializeJobTrackerAsync` runs: **reconcile → `Database.Migrate()` → reconcile**. Every Phase 4/5 migration is a no-op; the reconciler creates those tables with correct per-provider DDL. `Migrate()` throws on failure, so a schema problem exits the container rather than limping on. 7. `deploy.sh` waits, then fails the deploy if `backend` is not running, and runs a public smoke check against `APP_PUBLIC_BASE_URL`. --- # Pre-deployment - [ ] **Database backup verified.** `deploy.sh` takes one automatically, but for the first deploy take one by hand as well and confirm it restores — see `deploy/README.md`. A backup you have never restored is a hypothesis. - [ ] **Disk space checked.** `df -h` on the host. You need room for the backup, two image sets during the build, and the build cache. `docker system df` shows what Docker is holding. - [ ] **Environment variables present.** `deploy.sh` now checks these itself and aborts before it builds or replaces anything, so a miss costs an aborted deploy rather than a broken one. Check first anyway and skip the round trip — `/opt/job-tracker/shared/.env` must contain: - `DATABASE_PROVIDER=mariadb` (or `mysql`) — **required, no default.** An absent value used to mean "sqlite", which silently produced the wrong backup; it now stops the deploy - `JOBTRACKER_CONNECTION_STRING` — see the host-resolution note below - `AI_SERVICE_TOKEN` — compose refuses to start without it - `AUTH_JWT_KEY` — with `Auth__Require=true`, a blank key **throws at startup** (this is good; it fails loud rather than silently invalidating every session on restart) - `APP_PUBLIC_BASE_URL` — optional; without it the post-deploy public smoke check is skipped, and the script prints that it is skipping - `AUTH_ADMIN_EMAIL` / `AUTH_ADMIN_PASSWORD` only if you want admin seeding on this boot - [ ] **Connection string host resolves from inside the container.** `Server=127.0.0.1` means *the backend container*, not the host — this bit me during validation. Use the host's LAN address, a shared Docker network alias, or `host.docker.internal` where supported. - [ ] **Secrets available.** Confirm `.env` is the real shared file and not a stale copy: `ls -l /opt/job-tracker/shared/.env`. - [ ] **Docker healthy.** `docker info` succeeds; `docker ps` shows the current stack running. - [ ] **Current version recorded** — you need this to roll back: ```bash cd /opt/job-tracker/app git rev-parse HEAD | tee /tmp/jobtracker-rollback-commit docker compose ps ``` - [ ] **Quiet window chosen.** Rows users create in the new tables during the deploy are lost if you later restore the database. --- # Deployment steps ### 1. Backup Automatic — `deploy.sh` runs it first and aborts on failure. Confirm afterwards: ```bash ls -lt /opt/job-tracker/backups | head -3 ``` Expect a new `jobtracker--.sql.gz`. The script already rejected it if it were empty, not valid gzip, missing `CREATE TABLE`, or missing the `Dump completed` trailer. **Check the filename, not just that a file appeared.** A `jobtracker-sqlite-.tar.gz` on a MariaDB deploy means the environment is wrong — that is the exact failure the provider check exists to prevent, and it should now abort rather than reach this point. ### 2. Pull code ```bash cd /opt/job-tracker/app git fetch --all git log --oneline HEAD..origin/main | head -20 # read what you are about to deploy git checkout main && git pull --ff-only ``` ### 3–5. Build, replace containers ```bash deploy/deploy.sh ``` This builds, then `up -d --force-recreate` for `backend` and `frontend`. Old containers are replaced, not stopped first, so downtime is roughly container start time. ### 6–7. Database startup and reconciler Watch it. This is the step that matters on a first deploy: ```bash docker compose logs -f backend ``` **Healthy looks like:** EF migration lines, then `Now listening on: http://[::]:8080` and `Application started`. **Stop and roll back if you see:** | Log line | Meaning | |---|---| | `Unhandled exception ... Specified key was too long` | A migration ran that should be a no-op | | `Unhandled exception ... Table '...' doesn't exist` | Reconciler ordering problem | | `no such table: INFORMATION_SCHEMA.TABLES` | `DATABASE_PROVIDER=mysql` but the connection string is **empty** — verified failure mode | | `Unable to connect to any of the specified MySQL hosts` | Connection string host unreachable from inside the container | | `Auth is required but Auth:JwtKey is not configured` | `AUTH_JWT_KEY` missing from `.env` — `deploy.sh` should now catch this before the build | ### 8. Health verification ```bash docker compose ps ``` Both `backend` and `frontend` should read `(healthy)`, not merely `Up`. The frontend waits for the backend to be healthy before it starts, because nginx proxies `/api` and refuses to boot if the upstream cannot be resolved. --- # Verification ## Backend ```bash # Health endpoint — anonymous, does not touch the database curl -fsS https:///health # expect: {"status":"ok","version":""} # # The version comes from App:Version, which compose passes as App__Version from APP_VERSION. # CI sets it to the workflow run number. A version of "1.0.0.0" (the assembly fallback) means # APP_VERSION did not reach the container — harmless in itself, but it tells you the build # metadata is not flowing, so the admin system page will be vague about what is deployed. # Auth still enforced (this is the check that proves the API is not open) curl -s -o /dev/null -w '%{http_code}\n' https:///api/jobapplications # expect: 401 # Auth config responds as JSON curl -fsS https:///api/auth/config | head -c 200 # expect JSON containing requireAuth ``` - [ ] **Existing user login works.** Sign in with a real account in a browser. Do this yourself — it needs a password, and no automated step here should handle one. - [ ] **API access after login.** The applications list loads with your real data. ## Database ```bash MYSQL_PWD='' mariadb --host= --user= jobtracker -e " SELECT COUNT(*) AS tables FROM information_schema.tables WHERE table_schema='jobtracker'; SELECT COUNT(*) AS applications FROM JobApplications; SELECT COUNT(*) AS companies FROM Companies; SHOW TABLES LIKE 'InterviewPrepItems'; " ``` - [ ] **Tables created** — expect ~42, including `CvVariants`, `AiInteractions`, `ApplicationChecklistItems`, `CoverLetterVersions`, `InterviewPrepItems`. - [ ] **Existing rows preserved** — application and company counts match what you saw before the deploy. This is the single most important check. ## Application - [ ] **Frontend loads** at the public URL. - [ ] **Existing applications visible** with the correct count. - [ ] **CV system available** — the CV builder lists existing variants; open one. - [ ] **Workspace available** — open an application, check Overview, Checklist, Timeline, Analysis and Match render. New sections start empty for existing applications; that is correct, not a fault. - [ ] **Public CV route works** — open `/cv/` for a variant already marked public. If none is public, mark one, check it, then unmark it. --- # Rollback ## When to roll back Roll back if any of these are true: - **Backend unavailable** — container exits, restarts in a loop, or never reports healthy - **Migration or reconciler failure** — any unhandled exception in the startup log - **Data integrity issue** — row counts dropped, or existing applications are missing - **Frontend unusable** — will not load, or cannot reach the API Do **not** roll back for a cosmetic problem or an empty new section. Empty is expected. ## Code rollback is not database rollback **These are different operations and you usually want only the first.** A code rollback reverts the application and leaves all data intact. A database restore discards everything written since the dump — including anything users did during and after the deploy. **Restore the database only if the data itself is wrong.** If the backend simply will not start, the code rollback is almost certainly the whole fix. Why a code rollback is safe here: every Phase 4/5 migration is a no-op — the reconciler owns those tables — so reverting code never leaves migration history ahead of the schema. The reconciler is additive and never drops a table holding rows, so the older code simply ignores the newer tables. ## Procedure ```bash # 1. Stop the new version cd /opt/job-tracker/app docker compose stop backend frontend # 2. Restore the previous containers (code rollback) git checkout "$(cat /tmp/jobtracker-rollback-commit)" deploy/deploy.sh # takes a fresh backup first, so the rollback is itself protected # 3. Database restore decision — ONLY if data is wrong. Skip otherwise. # docker compose stop backend # gzip -dc /opt/job-tracker/backups/.sql.gz \ # | MYSQL_PWD='' mariadb --host= --user= jobtracker # docker compose start backend # 4. Verification docker compose ps # both healthy curl -fsS https:///health ``` Then re-run the **Verification** section above. Confirm row counts and sign-in before walking away. --- # Validation performed for this runbook Against MariaDB 11 containers, no production data: | Check | Result | |---|---| | `docker compose build backend frontend` | ✅ both images built | | `docker compose up -d` on a fresh MariaDB | ✅ 42 tables created, backend healthy | | `depends_on: service_healthy` gate | ✅ frontend waited for backend health, then started healthy | | Restart against the now-populated database | ✅ still 42 tables, seeded row preserved | | Backend health endpoint | ✅ 200 anonymously; `/api/jobapplications` 401 with `Auth:Require` unset | | Backup against a seeded MariaDB | ✅ verified dump written | | Restore into a clean MariaDB | ✅ rows identical | | Backup failure paths | ✅ bad credentials and missing connection string both abort, no partial file | | Shared `.env` loaded into the deploy shell | ✅ MariaDB path selected from the file alone; dump contains schema, rows and trailer | | `DATABASE_PROVIDER` missing or unrecognised | ✅ deploy stops before build/replace, names the variable, leaves no backup file | | `AI_SERVICE_TOKEN` / `AUTH_JWT_KEY` missing | ✅ both reported in one pass, deploy stops | | SQLite volume backup, and its failure paths | ✅ valid archive verified; a volume with no `jobtracker.db`, and a volume name that does not exist, both abort | | Secret leakage in deploy output | ✅ zero occurrences of any password, token or key across every test | | Empty connection string with `provider=mysql` | ✅ fails loudly (`no such table: INFORMATION_SCHEMA.TABLES`) rather than silently serving an empty database | | Backend with unreachable database | ✅ exits, reported `unhealthy` | **Not validated:** the authenticated end-to-end user journey. Signing in requires a password, so the browser checks in *Verification* are yours to perform.