jobtrackingapp/deploy/first-production-deployment.md

# First production deployment runbook

> Written 2026-07-19 against the actual implementation, not against the other docs. Every command and
> failure mode below was exercised locally against MariaDB 11 containers. No production data was used.
>
> This is the **first** deploy after the Phase 4/5 architecture changes. Production is many commits
> behind and the startup reconciler will create roughly a dozen tables against real data for the first
> time. Read *Pre-deployment* fully before starting.

## What actually happens on deploy

Verified by reading `deploy/deploy.sh` and `JobTrackerApi/Program.cs`:

1. `deploy.sh` links `/opt/job-tracker/shared/.env` into the checkout as `.env`, **and loads it into
   its own shell.** The link is for docker compose; the script needs the values itself to pick the
   right backup. Values already in the environment (CI's `APP_VERSION` and friends) win.
2. **It validates the deployment configuration**, before anything is built, stopped or replaced:
   `DATABASE_PROVIDER` (required, no default), the connection string when that provider needs one,
   `AI_SERVICE_TOKEN` and `AUTH_JWT_KEY`. A missing variable aborts the deploy with the running stack
   untouched. Names are printed, never values.
3. **It takes a database backup and aborts if that fails.** Nothing else runs without a restore point.
   The provider chooses the backup: `mariadb`/`mysql` gives a `.sql.gz` dump, `sqlite` gives a
   `.tar.gz` of the data volume. Each is verified against its own format — the dump must contain
   `CREATE TABLE` and the `Dump completed` trailer; the archive must contain `jobtracker.db`.
4. `docker compose pull`, then builds `backend` and `frontend` (with one prune-and-retry on failure).
5. `docker compose up -d --force-recreate --remove-orphans backend frontend`.
   **There is no `compose down`** — containers are replaced in place, so the window is short.
6. On backend start, `InitializeJobTrackerAsync` runs: **reconcile → `Database.Migrate()` → reconcile**.
   Every Phase 4/5 migration is a no-op; the reconciler creates those tables with correct per-provider
   DDL. `Migrate()` throws on failure, so a schema problem exits the container rather than limping on.
7. `deploy.sh` waits, then fails the deploy if `backend` is not running, and runs a public smoke check
   against `APP_PUBLIC_BASE_URL`.

---

# Pre-deployment

- [ ] **Database backup verified.** `deploy.sh` takes one automatically, but for the first deploy take
      one by hand as well and confirm it restores — see `deploy/README.md`. A backup you have never
      restored is a hypothesis.
- [ ] **Disk space checked.** `df -h` on the host. You need room for the backup, two image sets during
      the build, and the build cache. `docker system df` shows what Docker is holding.
- [ ] **Environment variables present.** `deploy.sh` now checks these itself and aborts before it
      builds or replaces anything, so a miss costs an aborted deploy rather than a broken one. Check
      first anyway and skip the round trip — `/opt/job-tracker/shared/.env` must contain:
      - `DATABASE_PROVIDER=mariadb` (or `mysql`) — **required, no default.** An absent value used to
        mean "sqlite", which silently produced the wrong backup; it now stops the deploy
      - `JOBTRACKER_CONNECTION_STRING` — see the host-resolution note below
      - `AI_SERVICE_TOKEN` — compose refuses to start without it
      - `AUTH_JWT_KEY` — with `Auth__Require=true`, a blank key **throws at startup** (this is good;
        it fails loud rather than silently invalidating every session on restart)
      - `APP_PUBLIC_BASE_URL` — required canonical HTTPS origin with no path, query, fragment or
        credentials; it controls generated links, OAuth callbacks, secure cookies and Host validation
      - `WEB_PROXY_SUBNET` — required dedicated IPv4 CIDR for nginx-to-backend traffic; confirm it
        does not overlap any existing Docker network before deployment
      - `AUTH_ADMIN_EMAIL` / `AUTH_ADMIN_PASSWORD` only if you want admin seeding on this boot
- [ ] **Connection string host resolves from inside the container.** `Server=127.0.0.1` means *the
      backend container*, not the host — this bit me during validation. Use the host's LAN address, a
      shared Docker network alias, or `host.docker.internal` where supported.
- [ ] **Secrets available.** Confirm `.env` is the real shared file and not a stale copy:
      `ls -l /opt/job-tracker/shared/.env`.
- [ ] **Docker healthy.** `docker info` succeeds; `docker ps` shows the current stack running.
- [ ] **Current version recorded** — you need this to roll back:
      ```bash
      cd /opt/job-tracker/app
      git rev-parse HEAD | tee /tmp/jobtracker-rollback-commit
      docker compose ps
      ```
- [ ] **Quiet window chosen.** Rows users create in the new tables during the deploy are lost if you
      later restore the database.

---

# Deployment steps

### 1. Backup

Automatic — `deploy.sh` runs it first and aborts on failure. Confirm afterwards:

```bash
ls -lt /opt/job-tracker/backups | head -3
```

Expect a new `jobtracker-<db>-<UTC timestamp>.sql.gz`. The script already rejected it if it were empty,
not valid gzip, missing `CREATE TABLE`, or missing the `Dump completed` trailer.

**Check the filename, not just that a file appeared.** A `jobtracker-sqlite-<stamp>.tar.gz` on a
MariaDB deploy means the environment is wrong — that is the exact failure the provider check exists to
prevent, and it should now abort rather than reach this point.

### 2. Pull code

```bash
cd /opt/job-tracker/app
git fetch --all
git log --oneline HEAD..origin/main | head -20   # read what you are about to deploy
git checkout main && git pull --ff-only
```

### 3–5. Build, replace containers

```bash
deploy/deploy.sh
```

This builds, then `up -d --force-recreate` for `backend` and `frontend`. Old containers are replaced,
not stopped first, so downtime is roughly container start time.

### 6–7. Database startup and reconciler

Watch it. This is the step that matters on a first deploy:

```bash
docker compose logs -f backend
```

**Healthy looks like:** EF migration lines, then `Now listening on: http://[::]:8080` and
`Application started`.

**Stop and roll back if you see:**

| Log line | Meaning |
|---|---|
| `Unhandled exception ... Specified key was too long` | A migration ran that should be a no-op |
| `Unhandled exception ... Table '...' doesn't exist` | Reconciler ordering problem |
| `no such table: INFORMATION_SCHEMA.TABLES` | `DATABASE_PROVIDER=mysql` but the connection string is **empty** — verified failure mode |
| `Unable to connect to any of the specified MySQL hosts` | Connection string host unreachable from inside the container |
| `Auth is required but Auth:JwtKey is not configured` | `AUTH_JWT_KEY` missing from `.env` — `deploy.sh` should now catch this before the build |

### 8. Health verification

```bash
docker compose ps
```

Both `backend` and `frontend` should read `(healthy)`, not merely `Up`. The frontend waits for the
backend to be healthy before it starts, because nginx proxies `/api` and refuses to boot if the
upstream cannot be resolved.

---

# Verification

## Backend

```bash
# Health endpoint — anonymous, does not touch the database
curl -fsS https://<host>/health
# expect: {"status":"ok","version":"<the APP_VERSION you deployed>"}
#
# The version comes from App:Version, which compose passes as App__Version from APP_VERSION.
# CI sets it to the workflow run number. A version of "1.0.0.0" (the assembly fallback) means
# APP_VERSION did not reach the container — harmless in itself, but it tells you the build
# metadata is not flowing, so the admin system page will be vague about what is deployed.

# Auth still enforced (this is the check that proves the API is not open)
curl -s -o /dev/null -w '%{http_code}\n' https://<host>/api/jobapplications
# expect: 401

# Auth config responds as JSON
curl -fsS https://<host>/api/auth/config | head -c 200
# expect JSON containing requireAuth
```

- [ ] **Existing user login works.** Sign in with a real account in a browser. Do this yourself — it
      needs a password, and no automated step here should handle one.
- [ ] **API access after login.** The applications list loads with your real data.

## Database

```bash
MYSQL_PWD='<password>' mariadb --host=<host> --user=<user> jobtracker -e "
  SELECT COUNT(*) AS tables FROM information_schema.tables WHERE table_schema='jobtracker';
  SELECT COUNT(*) AS applications FROM JobApplications;
  SELECT COUNT(*) AS companies FROM Companies;
  SHOW TABLES LIKE 'InterviewPrepItems';
"
```

- [ ] **Tables created** — expect ~42, including `CvVariants`, `AiInteractions`,
      `ApplicationChecklistItems`, `CoverLetterVersions`, `InterviewPrepItems`.
- [ ] **Existing rows preserved** — application and company counts match what you saw before the
      deploy. This is the single most important check.

## Application

- [ ] **Frontend loads** at the public URL.
- [ ] **Existing applications visible** with the correct count.
- [ ] **CV system available** — the CV builder lists existing variants; open one.
- [ ] **Workspace available** — open an application, check Overview, Checklist, Timeline, Analysis and
      Match render. New sections start empty for existing applications; that is correct, not a fault.
- [ ] **Public CV route works** — open `/cv/<slug>` for a variant already marked public. If none is
      public, mark one, check it, then unmark it.

---

# Rollback

## When to roll back

Roll back if any of these are true:

- **Backend unavailable** — container exits, restarts in a loop, or never reports healthy
- **Migration or reconciler failure** — any unhandled exception in the startup log
- **Data integrity issue** — row counts dropped, or existing applications are missing
- **Frontend unusable** — will not load, or cannot reach the API

Do **not** roll back for a cosmetic problem or an empty new section. Empty is expected.

## Code rollback is not database rollback

**These are different operations and you usually want only the first.**

A code rollback reverts the application and leaves all data intact. A database restore discards
everything written since the dump — including anything users did during and after the deploy.

**Restore the database only if the data itself is wrong.** If the backend simply will not start, the
code rollback is almost certainly the whole fix.

Why a code rollback is safe here: every Phase 4/5 migration is a no-op — the reconciler owns those
tables — so reverting code never leaves migration history ahead of the schema. The reconciler is
additive and never drops a table holding rows, so the older code simply ignores the newer tables.

## Procedure

```bash
# 1. Stop the new version
cd /opt/job-tracker/app
docker compose stop backend frontend

# 2. Restore the previous containers (code rollback)
git checkout "$(cat /tmp/jobtracker-rollback-commit)"
deploy/deploy.sh          # takes a fresh backup first, so the rollback is itself protected

# 3. Database restore decision — ONLY if data is wrong. Skip otherwise.
#    docker compose stop backend
#    gzip -dc /opt/job-tracker/backups/<file>.sql.gz \
#      | MYSQL_PWD='<password>' mariadb --host=<host> --user=<user> jobtracker
#    docker compose start backend

# 4. Verification
docker compose ps                                   # both healthy
curl -fsS https://<host>/health
```

Then re-run the **Verification** section above. Confirm row counts and sign-in before walking away.

---

# Validation performed for this runbook

Against MariaDB 11 containers, no production data:

| Check | Result |
|---|---|
| `docker compose build backend frontend` | ✅ both images built |
| `docker compose up -d` on a fresh MariaDB | ✅ 42 tables created, backend healthy |
| `depends_on: service_healthy` gate | ✅ frontend waited for backend health, then started healthy |
| Restart against the now-populated database | ✅ still 42 tables, seeded row preserved |
| Backend health endpoint | ✅ 200 anonymously; `/api/jobapplications` 401 with `Auth:Require` unset |
| Backup against a seeded MariaDB | ✅ verified dump written |
| Restore into a clean MariaDB | ✅ rows identical |
| Backup failure paths | ✅ bad credentials and missing connection string both abort, no partial file |
| Shared `.env` loaded into the deploy shell | ✅ MariaDB path selected from the file alone; dump contains schema, rows and trailer |
| `DATABASE_PROVIDER` missing or unrecognised | ✅ deploy stops before build/replace, names the variable, leaves no backup file |
| `AI_SERVICE_TOKEN` / `AUTH_JWT_KEY` missing | ✅ both reported in one pass, deploy stops |
| SQLite volume backup, and its failure paths | ✅ valid archive verified; a volume with no `jobtracker.db`, and a volume name that does not exist, both abort |
| Secret leakage in deploy output | ✅ zero occurrences of any password, token or key across every test |
| Empty connection string with `provider=mysql` | ✅ fails loudly (`no such table: INFORMATION_SCHEMA.TABLES`) rather than silently serving an empty database |
| Backend with unreachable database | ✅ exits, reported `unhealthy` |

**Not validated:** the authenticated end-to-end user journey. Signing in requires a password, so the
browser checks in *Verification* are yours to perform.