cesnimda/jobtrackingapp
1
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity

docs(ops): add first production deployment runbook
CI and Deploy / test (push) Failing after 1m3s
Details
CI and Deploy / deploy (push) Has been skipped
Details

Browse Source 
Written against the actual implementation rather than the existing docs, and
validated locally against MariaDB 11 containers. No application behaviour
changed — this commit adds two documents.

deploy/first-production-deployment.md covers pre-deployment checks, the eight
deployment steps, smoke tests for backend, database and application, and
rollback. It documents what deploy.sh really does: it backs up first and aborts
on failure, and it replaces containers with up -d --force-recreate rather than
running compose down, so the window is container start time. It also records the
startup sequence as implemented — reconcile, migrate, reconcile — and that
Database.Migrate() throws rather than limping on.

Validation surfaced things worth writing down. The connection string resolves
from inside the backend container, so Server=127.0.0.1 means the container and
not the host; this broke a validation run before it could have broken a deploy.
DATABASE_PROVIDER defaults to sqlite, and if it goes missing the backend does not
quietly serve an empty database — it exits with "no such table:
INFORMATION_SCHEMA.TABLES", which is loud but baffling if unexplained. A blank
AUTH_JWT_KEY throws at startup when auth is required, which is the right
behaviour. The runbook maps each of these log lines to its cause.

Rollback is documented with the distinction stated plainly: a code rollback
keeps all data and is almost always the whole fix, while a database restore
discards everything written since the dump. Restore only when the data itself is
wrong.

docs/release-checklist.md records the completed architecture work, local
verification results, known risks with severities, the unresolved CI runner
blocker and what would unblock it, and seven first-deployment warnings.

Validated: compose build; compose up on a fresh MariaDB (42 tables, backend
healthy); the depends_on health gate holding the frontend until the backend is
healthy; restart against the populated database with rows preserved; backup and
restore; and the failure paths. Not validated, and said so in both documents: the
authenticated end-to-end journey, because signing in needs a password.

393 backend tests pass.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
cesnimda
2026-07-19 18:02:49 +02:00
parent 93462b799c
commit 25b64bee8a
2 changed files with 325 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
deploy/first-production-deployment.md
+243
View File
@@ -0,0 +1,243 @@
# First production deployment runbook
> Written 2026-07-19 against the actual implementation, not against the other docs. Every command and
> failure mode below was exercised locally against MariaDB 11 containers. No production data was used.
>
> This is the **first** deploy after the Phase 4/5 architecture changes. Production is many commits
> behind and the startup reconciler will create roughly a dozen tables against real data for the first
> time. Read *Pre-deployment* fully before starting.
## What actually happens on deploy
Verified by reading `deploy/deploy.sh` and `JobTrackerApi/Program.cs`:
1. `deploy.sh` links `/opt/job-tracker/shared/.env` into the checkout as `.env`.
2. **It takes a database backup and aborts if that fails.** Nothing else runs without a restore point.
3. `docker compose pull`, then builds `backend` and `frontend` (with one prune-and-retry on failure).
4. `docker compose up -d --force-recreate --remove-orphans backend frontend`.
**There is no `compose down`** — containers are replaced in place, so the window is short.
5. On backend start, `InitializeJobTrackerAsync` runs: **reconcile → `Database.Migrate()` → reconcile**.
Every Phase 4/5 migration is a no-op; the reconciler creates those tables with correct per-provider
DDL. `Migrate()` throws on failure, so a schema problem exits the container rather than limping on.
6. `deploy.sh` waits, then fails the deploy if `backend` is not running, and runs a public smoke check
against `APP_PUBLIC_BASE_URL` if it is set.
---
# Pre-deployment
- [ ] **Database backup verified.** `deploy.sh` takes one automatically, but for the first deploy take
one by hand as well and confirm it restores — see `deploy/README.md`. A backup you have never
restored is a hypothesis.
- [ ] **Disk space checked.** `df -h` on the host. You need room for the backup, two image sets during
the build, and the build cache. `docker system df` shows what Docker is holding.
- [ ] **Environment variables present.** Check `/opt/job-tracker/shared/.env` contains:
- `AI_SERVICE_TOKEN` — **compose refuses to start without it**
- `AUTH_JWT_KEY` — with `Auth__Require=true`, a blank key **throws at startup** (this is good;
it fails loud rather than silently invalidating every session on restart)
- `DATABASE_PROVIDER=mysql` — **defaults to `sqlite` if absent**
- `JOBTRACKER_CONNECTION_STRING` — see the host-resolution note below
- `AUTH_ADMIN_EMAIL` / `AUTH_ADMIN_PASSWORD` only if you want admin seeding on this boot
- [ ] **Connection string host resolves from inside the container.** `Server=127.0.0.1` means *the
backend container*, not the host — this bit me during validation. Use the host's LAN address, a
shared Docker network alias, or `host.docker.internal` where supported.
- [ ] **Secrets available.** Confirm `.env` is the real shared file and not a stale copy:
`ls -l /opt/job-tracker/shared/.env`.
- [ ] **Docker healthy.** `docker info` succeeds; `docker ps` shows the current stack running.
- [ ] **Current version recorded** — you need this to roll back:
```bash
cd /opt/job-tracker/app
git rev-parse HEAD | tee /tmp/jobtracker-rollback-commit
docker compose ps
```
- [ ] **Quiet window chosen.** Rows users create in the new tables during the deploy are lost if you
later restore the database.
---
# Deployment steps
### 1. Backup
Automatic — `deploy.sh` runs it first and aborts on failure. Confirm afterwards:
```bash
ls -lt /opt/job-tracker/backups | head -3
```
Expect a new `jobtracker-<db>-<UTC timestamp>.sql.gz`. The script already rejected it if it were empty
or missing `CREATE TABLE`.
### 2. Pull code
```bash
cd /opt/job-tracker/app
git fetch --all
git log --oneline HEAD..origin/main | head -20 # read what you are about to deploy
git checkout main && git pull --ff-only
```
### 3–5. Build, replace containers
```bash
deploy/deploy.sh
```
This builds, then `up -d --force-recreate` for `backend` and `frontend`. Old containers are replaced,
not stopped first, so downtime is roughly container start time.
### 6–7. Database startup and reconciler
Watch it. This is the step that matters on a first deploy:
```bash
docker compose logs -f backend
```
**Healthy looks like:** EF migration lines, then `Now listening on: http://[::]:8080` and
`Application started`.
**Stop and roll back if you see:**
| Log line | Meaning |
|---|---|
| `Unhandled exception ... Specified key was too long` | A migration ran that should be a no-op |
| `Unhandled exception ... Table '...' doesn't exist` | Reconciler ordering problem |
| `no such table: INFORMATION_SCHEMA.TABLES` | `DATABASE_PROVIDER=mysql` but the connection string is **empty** — verified failure mode |
| `Unable to connect to any of the specified MySQL hosts` | Connection string host unreachable from inside the container |
| `Auth is required but Auth:JwtKey is not configured` | `AUTH_JWT_KEY` missing from `.env` |
### 8. Health verification
```bash
docker compose ps
```
Both `backend` and `frontend` should read `(healthy)`, not merely `Up`. The frontend waits for the
backend to be healthy before it starts, because nginx proxies `/api` and refuses to boot if the
upstream cannot be resolved.
---
# Verification
## Backend
```bash
# Health endpoint — anonymous, does not touch the database
curl -fsS https://<host>/health
# expect: {"status":"ok","version":"..."}
# Auth still enforced (this is the check that proves the API is not open)
curl -s -o /dev/null -w '%{http_code}\n' https://<host>/api/jobapplications
# expect: 401
# Auth config responds as JSON
curl -fsS https://<host>/api/auth/config | head -c 200
# expect JSON containing requireAuth
```
- [ ] **Existing user login works.** Sign in with a real account in a browser. Do this yourself — it
needs a password, and no automated step here should handle one.
- [ ] **API access after login.** The applications list loads with your real data.
## Database
```bash
MYSQL_PWD='<password>' mariadb --host=<host> --user=<user> jobtracker -e "
SELECT COUNT(*) AS tables FROM information_schema.tables WHERE table_schema='jobtracker';
SELECT COUNT(*) AS applications FROM JobApplications;
SELECT COUNT(*) AS companies FROM Companies;
SHOW TABLES LIKE 'InterviewPrepItems';
"
```
- [ ] **Tables created** — expect ~42, including `CvVariants`, `AiInteractions`,
`ApplicationChecklistItems`, `CoverLetterVersions`, `InterviewPrepItems`.
- [ ] **Existing rows preserved** — application and company counts match what you saw before the
deploy. This is the single most important check.
## Application
- [ ] **Frontend loads** at the public URL.
- [ ] **Existing applications visible** with the correct count.
- [ ] **CV system available** — the CV builder lists existing variants; open one.
- [ ] **Workspace available** — open an application, check Overview, Checklist, Timeline, Analysis and
Match render. New sections start empty for existing applications; that is correct, not a fault.
- [ ] **Public CV route works** — open `/cv/<slug>` for a variant already marked public. If none is
public, mark one, check it, then unmark it.
---
# Rollback
## When to roll back
Roll back if any of these are true:
- **Backend unavailable** — container exits, restarts in a loop, or never reports healthy
- **Migration or reconciler failure** — any unhandled exception in the startup log
- **Data integrity issue** — row counts dropped, or existing applications are missing
- **Frontend unusable** — will not load, or cannot reach the API
Do **not** roll back for a cosmetic problem or an empty new section. Empty is expected.
## Code rollback is not database rollback
**These are different operations and you usually want only the first.**
A code rollback reverts the application and leaves all data intact. A database restore discards
everything written since the dump — including anything users did during and after the deploy.
**Restore the database only if the data itself is wrong.** If the backend simply will not start, the
code rollback is almost certainly the whole fix.
Why a code rollback is safe here: every Phase 4/5 migration is a no-op — the reconciler owns those
tables — so reverting code never leaves migration history ahead of the schema. The reconciler is
additive and never drops a table holding rows, so the older code simply ignores the newer tables.
## Procedure
```bash
# 1. Stop the new version
cd /opt/job-tracker/app
docker compose stop backend frontend
# 2. Restore the previous containers (code rollback)
git checkout "$(cat /tmp/jobtracker-rollback-commit)"
deploy/deploy.sh # takes a fresh backup first, so the rollback is itself protected
# 3. Database restore decision — ONLY if data is wrong. Skip otherwise.
# docker compose stop backend
# gzip -dc /opt/job-tracker/backups/<file>.sql.gz \
# | MYSQL_PWD='<password>' mariadb --host=<host> --user=<user> jobtracker
# docker compose start backend
# 4. Verification
docker compose ps # both healthy
curl -fsS https://<host>/health
```
Then re-run the **Verification** section above. Confirm row counts and sign-in before walking away.
---
# Validation performed for this runbook
Against MariaDB 11 containers, no production data:
| Check | Result |
|---|---|
| `docker compose build backend frontend` | ✅ both images built |
| `docker compose up -d` on a fresh MariaDB | ✅ 42 tables created, backend healthy |
| `depends_on: service_healthy` gate | ✅ frontend waited for backend health, then started healthy |
| Restart against the now-populated database | ✅ still 42 tables, seeded row preserved |
| Backend health endpoint | ✅ 200 anonymously; `/api/jobapplications` 401 with `Auth:Require` unset |
| Backup against a seeded MariaDB | ✅ verified dump written |
| Restore into a clean MariaDB | ✅ rows identical |
| Backup failure paths | ✅ bad credentials and missing connection string both abort, no partial file |
| Empty connection string with `provider=mysql` | ✅ fails loudly (`no such table: INFORMATION_SCHEMA.TABLES`) rather than silently serving an empty database |
| Backend with unreachable database | ✅ exits, reported `unhealthy` |
**Not validated:** the authenticated end-to-end user journey. Signing in requires a password, so the
browser checks in *Verification* are yours to perform.