From 25b64bee8afccc383e39a01409ecd0977d61d78a Mon Sep 17 00:00:00 2001 From: cesnimda Date: Sun, 19 Jul 2026 18:02:49 +0200 Subject: [PATCH] docs(ops): add first production deployment runbook MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Written against the actual implementation rather than the existing docs, and validated locally against MariaDB 11 containers. No application behaviour changed — this commit adds two documents. deploy/first-production-deployment.md covers pre-deployment checks, the eight deployment steps, smoke tests for backend, database and application, and rollback. It documents what deploy.sh really does: it backs up first and aborts on failure, and it replaces containers with up -d --force-recreate rather than running compose down, so the window is container start time. It also records the startup sequence as implemented — reconcile, migrate, reconcile — and that Database.Migrate() throws rather than limping on. Validation surfaced things worth writing down. The connection string resolves from inside the backend container, so Server=127.0.0.1 means the container and not the host; this broke a validation run before it could have broken a deploy. DATABASE_PROVIDER defaults to sqlite, and if it goes missing the backend does not quietly serve an empty database — it exits with "no such table: INFORMATION_SCHEMA.TABLES", which is loud but baffling if unexplained. A blank AUTH_JWT_KEY throws at startup when auth is required, which is the right behaviour. The runbook maps each of these log lines to its cause. Rollback is documented with the distinction stated plainly: a code rollback keeps all data and is almost always the whole fix, while a database restore discards everything written since the dump. Restore only when the data itself is wrong. docs/release-checklist.md records the completed architecture work, local verification results, known risks with severities, the unresolved CI runner blocker and what would unblock it, and seven first-deployment warnings. Validated: compose build; compose up on a fresh MariaDB (42 tables, backend healthy); the depends_on health gate holding the frontend until the backend is healthy; restart against the populated database with rows preserved; backup and restore; and the failure paths. Not validated, and said so in both documents: the authenticated end-to-end journey, because signing in needs a password. 393 backend tests pass. Co-Authored-By: Claude Opus 4.8 --- deploy/first-production-deployment.md | 243 ++++++++++++++++++++++++++ docs/release-checklist.md | 82 +++++++++ 2 files changed, 325 insertions(+) create mode 100644 deploy/first-production-deployment.md create mode 100644 docs/release-checklist.md diff --git a/deploy/first-production-deployment.md b/deploy/first-production-deployment.md new file mode 100644 index 0000000..2c3c6fe --- /dev/null +++ b/deploy/first-production-deployment.md @@ -0,0 +1,243 @@ +# First production deployment runbook + +> Written 2026-07-19 against the actual implementation, not against the other docs. Every command and +> failure mode below was exercised locally against MariaDB 11 containers. No production data was used. +> +> This is the **first** deploy after the Phase 4/5 architecture changes. Production is many commits +> behind and the startup reconciler will create roughly a dozen tables against real data for the first +> time. Read *Pre-deployment* fully before starting. + +## What actually happens on deploy + +Verified by reading `deploy/deploy.sh` and `JobTrackerApi/Program.cs`: + +1. `deploy.sh` links `/opt/job-tracker/shared/.env` into the checkout as `.env`. +2. **It takes a database backup and aborts if that fails.** Nothing else runs without a restore point. +3. `docker compose pull`, then builds `backend` and `frontend` (with one prune-and-retry on failure). +4. `docker compose up -d --force-recreate --remove-orphans backend frontend`. + **There is no `compose down`** — containers are replaced in place, so the window is short. +5. On backend start, `InitializeJobTrackerAsync` runs: **reconcile → `Database.Migrate()` → reconcile**. + Every Phase 4/5 migration is a no-op; the reconciler creates those tables with correct per-provider + DDL. `Migrate()` throws on failure, so a schema problem exits the container rather than limping on. +6. `deploy.sh` waits, then fails the deploy if `backend` is not running, and runs a public smoke check + against `APP_PUBLIC_BASE_URL` if it is set. + +--- + +# Pre-deployment + +- [ ] **Database backup verified.** `deploy.sh` takes one automatically, but for the first deploy take + one by hand as well and confirm it restores — see `deploy/README.md`. A backup you have never + restored is a hypothesis. +- [ ] **Disk space checked.** `df -h` on the host. You need room for the backup, two image sets during + the build, and the build cache. `docker system df` shows what Docker is holding. +- [ ] **Environment variables present.** Check `/opt/job-tracker/shared/.env` contains: + - `AI_SERVICE_TOKEN` — **compose refuses to start without it** + - `AUTH_JWT_KEY` — with `Auth__Require=true`, a blank key **throws at startup** (this is good; + it fails loud rather than silently invalidating every session on restart) + - `DATABASE_PROVIDER=mysql` — **defaults to `sqlite` if absent** + - `JOBTRACKER_CONNECTION_STRING` — see the host-resolution note below + - `AUTH_ADMIN_EMAIL` / `AUTH_ADMIN_PASSWORD` only if you want admin seeding on this boot +- [ ] **Connection string host resolves from inside the container.** `Server=127.0.0.1` means *the + backend container*, not the host — this bit me during validation. Use the host's LAN address, a + shared Docker network alias, or `host.docker.internal` where supported. +- [ ] **Secrets available.** Confirm `.env` is the real shared file and not a stale copy: + `ls -l /opt/job-tracker/shared/.env`. +- [ ] **Docker healthy.** `docker info` succeeds; `docker ps` shows the current stack running. +- [ ] **Current version recorded** — you need this to roll back: + ```bash + cd /opt/job-tracker/app + git rev-parse HEAD | tee /tmp/jobtracker-rollback-commit + docker compose ps + ``` +- [ ] **Quiet window chosen.** Rows users create in the new tables during the deploy are lost if you + later restore the database. + +--- + +# Deployment steps + +### 1. Backup + +Automatic — `deploy.sh` runs it first and aborts on failure. Confirm afterwards: + +```bash +ls -lt /opt/job-tracker/backups | head -3 +``` + +Expect a new `jobtracker--.sql.gz`. The script already rejected it if it were empty +or missing `CREATE TABLE`. + +### 2. Pull code + +```bash +cd /opt/job-tracker/app +git fetch --all +git log --oneline HEAD..origin/main | head -20 # read what you are about to deploy +git checkout main && git pull --ff-only +``` + +### 3–5. Build, replace containers + +```bash +deploy/deploy.sh +``` + +This builds, then `up -d --force-recreate` for `backend` and `frontend`. Old containers are replaced, +not stopped first, so downtime is roughly container start time. + +### 6–7. Database startup and reconciler + +Watch it. This is the step that matters on a first deploy: + +```bash +docker compose logs -f backend +``` + +**Healthy looks like:** EF migration lines, then `Now listening on: http://[::]:8080` and +`Application started`. + +**Stop and roll back if you see:** + +| Log line | Meaning | +|---|---| +| `Unhandled exception ... Specified key was too long` | A migration ran that should be a no-op | +| `Unhandled exception ... Table '...' doesn't exist` | Reconciler ordering problem | +| `no such table: INFORMATION_SCHEMA.TABLES` | `DATABASE_PROVIDER=mysql` but the connection string is **empty** — verified failure mode | +| `Unable to connect to any of the specified MySQL hosts` | Connection string host unreachable from inside the container | +| `Auth is required but Auth:JwtKey is not configured` | `AUTH_JWT_KEY` missing from `.env` | + +### 8. Health verification + +```bash +docker compose ps +``` + +Both `backend` and `frontend` should read `(healthy)`, not merely `Up`. The frontend waits for the +backend to be healthy before it starts, because nginx proxies `/api` and refuses to boot if the +upstream cannot be resolved. + +--- + +# Verification + +## Backend + +```bash +# Health endpoint — anonymous, does not touch the database +curl -fsS https:///health +# expect: {"status":"ok","version":"..."} + +# Auth still enforced (this is the check that proves the API is not open) +curl -s -o /dev/null -w '%{http_code}\n' https:///api/jobapplications +# expect: 401 + +# Auth config responds as JSON +curl -fsS https:///api/auth/config | head -c 200 +# expect JSON containing requireAuth +``` + +- [ ] **Existing user login works.** Sign in with a real account in a browser. Do this yourself — it + needs a password, and no automated step here should handle one. +- [ ] **API access after login.** The applications list loads with your real data. + +## Database + +```bash +MYSQL_PWD='' mariadb --host= --user= jobtracker -e " + SELECT COUNT(*) AS tables FROM information_schema.tables WHERE table_schema='jobtracker'; + SELECT COUNT(*) AS applications FROM JobApplications; + SELECT COUNT(*) AS companies FROM Companies; + SHOW TABLES LIKE 'InterviewPrepItems'; +" +``` + +- [ ] **Tables created** — expect ~42, including `CvVariants`, `AiInteractions`, + `ApplicationChecklistItems`, `CoverLetterVersions`, `InterviewPrepItems`. +- [ ] **Existing rows preserved** — application and company counts match what you saw before the + deploy. This is the single most important check. + +## Application + +- [ ] **Frontend loads** at the public URL. +- [ ] **Existing applications visible** with the correct count. +- [ ] **CV system available** — the CV builder lists existing variants; open one. +- [ ] **Workspace available** — open an application, check Overview, Checklist, Timeline, Analysis and + Match render. New sections start empty for existing applications; that is correct, not a fault. +- [ ] **Public CV route works** — open `/cv/` for a variant already marked public. If none is + public, mark one, check it, then unmark it. + +--- + +# Rollback + +## When to roll back + +Roll back if any of these are true: + +- **Backend unavailable** — container exits, restarts in a loop, or never reports healthy +- **Migration or reconciler failure** — any unhandled exception in the startup log +- **Data integrity issue** — row counts dropped, or existing applications are missing +- **Frontend unusable** — will not load, or cannot reach the API + +Do **not** roll back for a cosmetic problem or an empty new section. Empty is expected. + +## Code rollback is not database rollback + +**These are different operations and you usually want only the first.** + +A code rollback reverts the application and leaves all data intact. A database restore discards +everything written since the dump — including anything users did during and after the deploy. + +**Restore the database only if the data itself is wrong.** If the backend simply will not start, the +code rollback is almost certainly the whole fix. + +Why a code rollback is safe here: every Phase 4/5 migration is a no-op — the reconciler owns those +tables — so reverting code never leaves migration history ahead of the schema. The reconciler is +additive and never drops a table holding rows, so the older code simply ignores the newer tables. + +## Procedure + +```bash +# 1. Stop the new version +cd /opt/job-tracker/app +docker compose stop backend frontend + +# 2. Restore the previous containers (code rollback) +git checkout "$(cat /tmp/jobtracker-rollback-commit)" +deploy/deploy.sh # takes a fresh backup first, so the rollback is itself protected + +# 3. Database restore decision — ONLY if data is wrong. Skip otherwise. +# docker compose stop backend +# gzip -dc /opt/job-tracker/backups/.sql.gz \ +# | MYSQL_PWD='' mariadb --host= --user= jobtracker +# docker compose start backend + +# 4. Verification +docker compose ps # both healthy +curl -fsS https:///health +``` + +Then re-run the **Verification** section above. Confirm row counts and sign-in before walking away. + +--- + +# Validation performed for this runbook + +Against MariaDB 11 containers, no production data: + +| Check | Result | +|---|---| +| `docker compose build backend frontend` | ✅ both images built | +| `docker compose up -d` on a fresh MariaDB | ✅ 42 tables created, backend healthy | +| `depends_on: service_healthy` gate | ✅ frontend waited for backend health, then started healthy | +| Restart against the now-populated database | ✅ still 42 tables, seeded row preserved | +| Backend health endpoint | ✅ 200 anonymously; `/api/jobapplications` 401 with `Auth:Require` unset | +| Backup against a seeded MariaDB | ✅ verified dump written | +| Restore into a clean MariaDB | ✅ rows identical | +| Backup failure paths | ✅ bad credentials and missing connection string both abort, no partial file | +| Empty connection string with `provider=mysql` | ✅ fails loudly (`no such table: INFORMATION_SCHEMA.TABLES`) rather than silently serving an empty database | +| Backend with unreachable database | ✅ exits, reported `unhealthy` | + +**Not validated:** the authenticated end-to-end user journey. Signing in requires a password, so the +browser checks in *Verification* are yours to perform. diff --git a/docs/release-checklist.md b/docs/release-checklist.md new file mode 100644 index 0000000..b78ecb7 --- /dev/null +++ b/docs/release-checklist.md @@ -0,0 +1,82 @@ +# Release checklist + +> 2026-07-19. State of the build before the first production deployment after the Phase 4/5 +> architecture changes. Companion to `deploy/first-production-deployment.md` (how to deploy), +> `docs/production-readiness-review.md` (what was audited) and +> `docs/phase-5-completion-report.md` (what was built and why). + +## Completed architecture work + +| Area | State | +|---|---| +| **Career Profile** | Single source of truth. Relational children with stable item keys; `ProfileJson` is a derived projection. Nothing downstream writes to it. | +| **CV Builder** | Data-driven theme engine — themes are data, not hardcoded templates. Eight themes; adding one is appending an object. Variants are lenses over the master profile, never copies. Autosave, version history, restore. | +| **Public CV** | `/cv/{slug}`, off by default, per-variant opt-in, unguessable slug, `noindex`. SPA deep-link routing fixed so direct navigation, refresh and shared links resolve. | +| **Application Workspace** | `/applications/{id}`: Overview, Checklist, Timeline, Analysis, Match, CV, Cover Letter, Documents, Interview Prep, Communication. Owns almost no data — it composes systems that already existed. | +| **AI integrations** | Suggestion-only. Every generation appends to `AiInteraction` (that history *is* the versioning). Interview generation now receives the analysis and match context. Nothing AI-driven writes to the profile, a variant, a cover letter or prep content without the user saving it. | +| **Interview preparation** | User-owned `InterviewPrepItem`, distinct from the two AI *caches* that regenerate. Six categories in one table. | +| **Production hardening** | Explicit `[Authorize]` on every controller with a reflection test; clean MariaDB initialization; database ownership documented; pre-deploy backups; restore and rollback docs; health checks on backend and frontend. | + +## Local verification + +| Check | Result | +|---|---| +| Backend tests (Release) | 393 passed | +| Frontend tests | 128 passed, 36 suites | +| TypeScript | clean | +| Frontend production build | clean | +| Docker images | backend and frontend build | +| Fresh empty MariaDB 11 | 42 tables, starts healthy | +| Populated MariaDB restart | idempotent, rows preserved | +| Existing SQLite dev database | upgraded 34 → 44 tables, rows preserved | +| Backup and restore | dump verified, restored into a clean database, rows identical | + +## Unresolved external blocker + +### CI runner — deployment is gated on this + +CI is red for a reason **proven to be outside the repository**: commit `8f73548` changed a single +markdown file — no code, no test, no dependency, no workflow — and its test job failed in the same +duration band as every other run. A change that cannot affect compilation cannot fail a test job. + +Consequence: **everything above is local verification.** Nothing has been proven in CI, and the +deployment pipeline will not promote a build until this is fixed. + +Blocked on, and needing you: + +- Job logs — the API returns 401 unauthenticated; a read-scoped Gitea token would unblock this +- `journalctl -u act_runner --since '2 hours ago'` on the runner host +- The act_runner container/service configuration + +Full evidence: `docs/infrastructure/runner-investigation.md`. + +## Known risks + +| Risk | Severity | Notes | +|---|---|---| +| Production is many commits behind | High | The first deploy applies the whole Phase 4/5 change at once. Reconciler builds ~12 tables against real data for the first time. | +| Reconciler path unproven on your data | Medium | Verified on containers and on a real SQLite dev database — not on production. This is what the backup is for. | +| End-to-end journey unwalked | Medium | No one has driven the authenticated flow from empty profile to recorded outcome. Sign-in needs a password, so the browser checks belong to you. | +| Rollback loses new-table data | Medium | A code rollback keeps everything. A *database restore* discards rows created since the dump. Keep the deploy window quiet. | +| Logging is console-only | Low | Fine under Docker; no retention or aggregation. | +| No global exception handler | Low | Unhandled errors return a bare 500 with no correlation id. No stack leaks outside Development. | +| `ClientErrorsController` is anonymous | Low | By design — browser error reports must work before sign-in. Size-limited and truncated; stores nothing. | +| Prompt quality unmeasured | Low | Interview generation gets richer context; whether the questions are *better* is a judgement no test makes. | + +## First deployment warnings + +1. **Take a manual backup as well, and restore it once to a scratch database.** `deploy.sh` backs up + automatically, but the first deploy is the wrong time to discover a backup does not restore. +2. **Watch `docker compose logs -f backend` live.** `Database.Migrate()` throws on failure, so a schema + problem exits the container. The runbook lists the exact log lines that mean stop. +3. **`DATABASE_PROVIDER` defaults to `sqlite`.** If it goes missing from `.env`, the backend does not + quietly serve an empty database — it crashes with `no such table: INFORMATION_SCHEMA.TABLES`. + Loud, but confusing if you have not seen it before. +4. **The connection string resolves from inside the container.** `Server=127.0.0.1` means the backend + container, not the host. +5. **Empty new sections are expected.** Existing applications will show an empty Checklist, Timeline + and Interview Prep until used. The checklist seeds on first open. That is correct behaviour. +6. **Verify row counts before walking away.** Applications and companies must match the pre-deploy + numbers. This is the check that matters most. +7. **Do not restore the database for a code problem.** Roll the code back first; it is almost always + the whole fix, and it costs no data.