diff --git a/deploy/first-production-deployment.md b/deploy/first-production-deployment.md new file mode 100644 index 0000000..2c3c6fe --- /dev/null +++ b/deploy/first-production-deployment.md @@ -0,0 +1,243 @@ +# First production deployment runbook + +> Written 2026-07-19 against the actual implementation, not against the other docs. Every command and +> failure mode below was exercised locally against MariaDB 11 containers. No production data was used. +> +> This is the **first** deploy after the Phase 4/5 architecture changes. Production is many commits +> behind and the startup reconciler will create roughly a dozen tables against real data for the first +> time. Read *Pre-deployment* fully before starting. + +## What actually happens on deploy + +Verified by reading `deploy/deploy.sh` and `JobTrackerApi/Program.cs`: + +1. `deploy.sh` links `/opt/job-tracker/shared/.env` into the checkout as `.env`. +2. **It takes a database backup and aborts if that fails.** Nothing else runs without a restore point. +3. `docker compose pull`, then builds `backend` and `frontend` (with one prune-and-retry on failure). +4. `docker compose up -d --force-recreate --remove-orphans backend frontend`. + **There is no `compose down`** — containers are replaced in place, so the window is short. +5. On backend start, `InitializeJobTrackerAsync` runs: **reconcile → `Database.Migrate()` → reconcile**. + Every Phase 4/5 migration is a no-op; the reconciler creates those tables with correct per-provider + DDL. `Migrate()` throws on failure, so a schema problem exits the container rather than limping on. +6. `deploy.sh` waits, then fails the deploy if `backend` is not running, and runs a public smoke check + against `APP_PUBLIC_BASE_URL` if it is set. + +--- + +# Pre-deployment + +- [ ] **Database backup verified.** `deploy.sh` takes one automatically, but for the first deploy take + one by hand as well and confirm it restores — see `deploy/README.md`. A backup you have never + restored is a hypothesis. +- [ ] **Disk space checked.** `df -h` on the host. You need room for the backup, two image sets during + the build, and the build cache. `docker system df` shows what Docker is holding. +- [ ] **Environment variables present.** Check `/opt/job-tracker/shared/.env` contains: + - `AI_SERVICE_TOKEN` — **compose refuses to start without it** + - `AUTH_JWT_KEY` — with `Auth__Require=true`, a blank key **throws at startup** (this is good; + it fails loud rather than silently invalidating every session on restart) + - `DATABASE_PROVIDER=mysql` — **defaults to `sqlite` if absent** + - `JOBTRACKER_CONNECTION_STRING` — see the host-resolution note below + - `AUTH_ADMIN_EMAIL` / `AUTH_ADMIN_PASSWORD` only if you want admin seeding on this boot +- [ ] **Connection string host resolves from inside the container.** `Server=127.0.0.1` means *the + backend container*, not the host — this bit me during validation. Use the host's LAN address, a + shared Docker network alias, or `host.docker.internal` where supported. +- [ ] **Secrets available.** Confirm `.env` is the real shared file and not a stale copy: + `ls -l /opt/job-tracker/shared/.env`. +- [ ] **Docker healthy.** `docker info` succeeds; `docker ps` shows the current stack running. +- [ ] **Current version recorded** — you need this to roll back: + ```bash + cd /opt/job-tracker/app + git rev-parse HEAD | tee /tmp/jobtracker-rollback-commit + docker compose ps + ``` +- [ ] **Quiet window chosen.** Rows users create in the new tables during the deploy are lost if you + later restore the database. + +--- + +# Deployment steps + +### 1. Backup + +Automatic — `deploy.sh` runs it first and aborts on failure. Confirm afterwards: + +```bash +ls -lt /opt/job-tracker/backups | head -3 +``` + +Expect a new `jobtracker--.sql.gz`. The script already rejected it if it were empty +or missing `CREATE TABLE`. + +### 2. Pull code + +```bash +cd /opt/job-tracker/app +git fetch --all +git log --oneline HEAD..origin/main | head -20 # read what you are about to deploy +git checkout main && git pull --ff-only +``` + +### 3–5. Build, replace containers + +```bash +deploy/deploy.sh +``` + +This builds, then `up -d --force-recreate` for `backend` and `frontend`. Old containers are replaced, +not stopped first, so downtime is roughly container start time. + +### 6–7. Database startup and reconciler + +Watch it. This is the step that matters on a first deploy: + +```bash +docker compose logs -f backend +``` + +**Healthy looks like:** EF migration lines, then `Now listening on: http://[::]:8080` and +`Application started`. + +**Stop and roll back if you see:** + +| Log line | Meaning | +|---|---| +| `Unhandled exception ... Specified key was too long` | A migration ran that should be a no-op | +| `Unhandled exception ... Table '...' doesn't exist` | Reconciler ordering problem | +| `no such table: INFORMATION_SCHEMA.TABLES` | `DATABASE_PROVIDER=mysql` but the connection string is **empty** — verified failure mode | +| `Unable to connect to any of the specified MySQL hosts` | Connection string host unreachable from inside the container | +| `Auth is required but Auth:JwtKey is not configured` | `AUTH_JWT_KEY` missing from `.env` | + +### 8. Health verification + +```bash +docker compose ps +``` + +Both `backend` and `frontend` should read `(healthy)`, not merely `Up`. The frontend waits for the +backend to be healthy before it starts, because nginx proxies `/api` and refuses to boot if the +upstream cannot be resolved. + +--- + +# Verification + +## Backend + +```bash +# Health endpoint — anonymous, does not touch the database +curl -fsS https:///health +# expect: {"status":"ok","version":"..."} + +# Auth still enforced (this is the check that proves the API is not open) +curl -s -o /dev/null -w '%{http_code}\n' https:///api/jobapplications +# expect: 401 + +# Auth config responds as JSON +curl -fsS https:///api/auth/config | head -c 200 +# expect JSON containing requireAuth +``` + +- [ ] **Existing user login works.** Sign in with a real account in a browser. Do this yourself — it + needs a password, and no automated step here should handle one. +- [ ] **API access after login.** The applications list loads with your real data. + +## Database + +```bash +MYSQL_PWD='' mariadb --host= --user= jobtracker -e " + SELECT COUNT(*) AS tables FROM information_schema.tables WHERE table_schema='jobtracker'; + SELECT COUNT(*) AS applications FROM JobApplications; + SELECT COUNT(*) AS companies FROM Companies; + SHOW TABLES LIKE 'InterviewPrepItems'; +" +``` + +- [ ] **Tables created** — expect ~42, including `CvVariants`, `AiInteractions`, + `ApplicationChecklistItems`, `CoverLetterVersions`, `InterviewPrepItems`. +- [ ] **Existing rows preserved** — application and company counts match what you saw before the + deploy. This is the single most important check. + +## Application + +- [ ] **Frontend loads** at the public URL. +- [ ] **Existing applications visible** with the correct count. +- [ ] **CV system available** — the CV builder lists existing variants; open one. +- [ ] **Workspace available** — open an application, check Overview, Checklist, Timeline, Analysis and + Match render. New sections start empty for existing applications; that is correct, not a fault. +- [ ] **Public CV route works** — open `/cv/` for a variant already marked public. If none is + public, mark one, check it, then unmark it. + +--- + +# Rollback + +## When to roll back + +Roll back if any of these are true: + +- **Backend unavailable** — container exits, restarts in a loop, or never reports healthy +- **Migration or reconciler failure** — any unhandled exception in the startup log +- **Data integrity issue** — row counts dropped, or existing applications are missing +- **Frontend unusable** — will not load, or cannot reach the API + +Do **not** roll back for a cosmetic problem or an empty new section. Empty is expected. + +## Code rollback is not database rollback + +**These are different operations and you usually want only the first.** + +A code rollback reverts the application and leaves all data intact. A database restore discards +everything written since the dump — including anything users did during and after the deploy. + +**Restore the database only if the data itself is wrong.** If the backend simply will not start, the +code rollback is almost certainly the whole fix. + +Why a code rollback is safe here: every Phase 4/5 migration is a no-op — the reconciler owns those +tables — so reverting code never leaves migration history ahead of the schema. The reconciler is +additive and never drops a table holding rows, so the older code simply ignores the newer tables. + +## Procedure + +```bash +# 1. Stop the new version +cd /opt/job-tracker/app +docker compose stop backend frontend + +# 2. Restore the previous containers (code rollback) +git checkout "$(cat /tmp/jobtracker-rollback-commit)" +deploy/deploy.sh # takes a fresh backup first, so the rollback is itself protected + +# 3. Database restore decision — ONLY if data is wrong. Skip otherwise. +# docker compose stop backend +# gzip -dc /opt/job-tracker/backups/.sql.gz \ +# | MYSQL_PWD='' mariadb --host= --user= jobtracker +# docker compose start backend + +# 4. Verification +docker compose ps # both healthy +curl -fsS https:///health +``` + +Then re-run the **Verification** section above. Confirm row counts and sign-in before walking away. + +--- + +# Validation performed for this runbook + +Against MariaDB 11 containers, no production data: + +| Check | Result | +|---|---| +| `docker compose build backend frontend` | ✅ both images built | +| `docker compose up -d` on a fresh MariaDB | ✅ 42 tables created, backend healthy | +| `depends_on: service_healthy` gate | ✅ frontend waited for backend health, then started healthy | +| Restart against the now-populated database | ✅ still 42 tables, seeded row preserved | +| Backend health endpoint | ✅ 200 anonymously; `/api/jobapplications` 401 with `Auth:Require` unset | +| Backup against a seeded MariaDB | ✅ verified dump written | +| Restore into a clean MariaDB | ✅ rows identical | +| Backup failure paths | ✅ bad credentials and missing connection string both abort, no partial file | +| Empty connection string with `provider=mysql` | ✅ fails loudly (`no such table: INFORMATION_SCHEMA.TABLES`) rather than silently serving an empty database | +| Backend with unreachable database | ✅ exits, reported `unhealthy` | + +**Not validated:** the authenticated end-to-end user journey. Signing in requires a password, so the +browser checks in *Verification* are yours to perform. diff --git a/docs/release-checklist.md b/docs/release-checklist.md new file mode 100644 index 0000000..b78ecb7 --- /dev/null +++ b/docs/release-checklist.md @@ -0,0 +1,82 @@ +# Release checklist + +> 2026-07-19. State of the build before the first production deployment after the Phase 4/5 +> architecture changes. Companion to `deploy/first-production-deployment.md` (how to deploy), +> `docs/production-readiness-review.md` (what was audited) and +> `docs/phase-5-completion-report.md` (what was built and why). + +## Completed architecture work + +| Area | State | +|---|---| +| **Career Profile** | Single source of truth. Relational children with stable item keys; `ProfileJson` is a derived projection. Nothing downstream writes to it. | +| **CV Builder** | Data-driven theme engine — themes are data, not hardcoded templates. Eight themes; adding one is appending an object. Variants are lenses over the master profile, never copies. Autosave, version history, restore. | +| **Public CV** | `/cv/{slug}`, off by default, per-variant opt-in, unguessable slug, `noindex`. SPA deep-link routing fixed so direct navigation, refresh and shared links resolve. | +| **Application Workspace** | `/applications/{id}`: Overview, Checklist, Timeline, Analysis, Match, CV, Cover Letter, Documents, Interview Prep, Communication. Owns almost no data — it composes systems that already existed. | +| **AI integrations** | Suggestion-only. Every generation appends to `AiInteraction` (that history *is* the versioning). Interview generation now receives the analysis and match context. Nothing AI-driven writes to the profile, a variant, a cover letter or prep content without the user saving it. | +| **Interview preparation** | User-owned `InterviewPrepItem`, distinct from the two AI *caches* that regenerate. Six categories in one table. | +| **Production hardening** | Explicit `[Authorize]` on every controller with a reflection test; clean MariaDB initialization; database ownership documented; pre-deploy backups; restore and rollback docs; health checks on backend and frontend. | + +## Local verification + +| Check | Result | +|---|---| +| Backend tests (Release) | 393 passed | +| Frontend tests | 128 passed, 36 suites | +| TypeScript | clean | +| Frontend production build | clean | +| Docker images | backend and frontend build | +| Fresh empty MariaDB 11 | 42 tables, starts healthy | +| Populated MariaDB restart | idempotent, rows preserved | +| Existing SQLite dev database | upgraded 34 → 44 tables, rows preserved | +| Backup and restore | dump verified, restored into a clean database, rows identical | + +## Unresolved external blocker + +### CI runner — deployment is gated on this + +CI is red for a reason **proven to be outside the repository**: commit `8f73548` changed a single +markdown file — no code, no test, no dependency, no workflow — and its test job failed in the same +duration band as every other run. A change that cannot affect compilation cannot fail a test job. + +Consequence: **everything above is local verification.** Nothing has been proven in CI, and the +deployment pipeline will not promote a build until this is fixed. + +Blocked on, and needing you: + +- Job logs — the API returns 401 unauthenticated; a read-scoped Gitea token would unblock this +- `journalctl -u act_runner --since '2 hours ago'` on the runner host +- The act_runner container/service configuration + +Full evidence: `docs/infrastructure/runner-investigation.md`. + +## Known risks + +| Risk | Severity | Notes | +|---|---|---| +| Production is many commits behind | High | The first deploy applies the whole Phase 4/5 change at once. Reconciler builds ~12 tables against real data for the first time. | +| Reconciler path unproven on your data | Medium | Verified on containers and on a real SQLite dev database — not on production. This is what the backup is for. | +| End-to-end journey unwalked | Medium | No one has driven the authenticated flow from empty profile to recorded outcome. Sign-in needs a password, so the browser checks belong to you. | +| Rollback loses new-table data | Medium | A code rollback keeps everything. A *database restore* discards rows created since the dump. Keep the deploy window quiet. | +| Logging is console-only | Low | Fine under Docker; no retention or aggregation. | +| No global exception handler | Low | Unhandled errors return a bare 500 with no correlation id. No stack leaks outside Development. | +| `ClientErrorsController` is anonymous | Low | By design — browser error reports must work before sign-in. Size-limited and truncated; stores nothing. | +| Prompt quality unmeasured | Low | Interview generation gets richer context; whether the questions are *better* is a judgement no test makes. | + +## First deployment warnings + +1. **Take a manual backup as well, and restore it once to a scratch database.** `deploy.sh` backs up + automatically, but the first deploy is the wrong time to discover a backup does not restore. +2. **Watch `docker compose logs -f backend` live.** `Database.Migrate()` throws on failure, so a schema + problem exits the container. The runbook lists the exact log lines that mean stop. +3. **`DATABASE_PROVIDER` defaults to `sqlite`.** If it goes missing from `.env`, the backend does not + quietly serve an empty database — it crashes with `no such table: INFORMATION_SCHEMA.TABLES`. + Loud, but confusing if you have not seen it before. +4. **The connection string resolves from inside the container.** `Server=127.0.0.1` means the backend + container, not the host. +5. **Empty new sections are expected.** Existing applications will show an empty Checklist, Timeline + and Interview Prep until used. The checklist seeds on first open. That is correct behaviour. +6. **Verify row counts before walking away.** Applications and companies must match the pre-deploy + numbers. This is the check that matters most. +7. **Do not restore the database for a code problem.** Roll the code back first; it is almost always + the whole fix, and it costs no data.