diff --git a/docs/deployment/backup-restore.md b/docs/deployment/backup-restore.md new file mode 100644 index 0000000..c7193aa --- /dev/null +++ b/docs/deployment/backup-restore.md @@ -0,0 +1,102 @@ +# Production backup & restore + +> 2026-07-19. The backup **mechanism** is verified end to end against MariaDB 11 containers, including a +> byte-exact Norwegian-character round trip. **No production database has been backed up from this +> environment** — see *Production access*. This document is the checklist to run against production, +> plus the evidence for what is already proven. + +Companion to `deploy/deploy.sh` (the implementation), `deploy/README.md` (backup location/retention), +and `docs/operations/production-backup-verification.md` (the earlier container rehearsal). + +## Production backup checklist + +Run top to bottom. Every item is checked automatically by `deploy.sh` **before** it builds or replaces +anything — this list is for a manual pre-flight and for an out-of-band backup. + +- [ ] **`DATABASE_PROVIDER=mariadb`** in `/opt/job-tracker/shared/.env`. No default — `deploy.sh` + aborts if it is missing. If it were wrong, the SQLite path would run and back up the wrong thing. +- [ ] **`JOBTRACKER_CONNECTION_STRING` present** and pointing at the production database. The host + resolves from the deploy shell, not from inside a container, for the backup step. +- [ ] **`deploy.sh` loads the environment.** It parses `/opt/job-tracker/shared/.env` into its own + shell before deciding anything (verified: the "Loaded deployment environment" line prints first). +- [ ] **Backup runs before replacement.** `validate_deploy_config` → `backup_database` → build → + `up -d --force-recreate`. Confirmed by line order in `deploy.sh` (validate/backup precede + build/replace). A failed backup aborts the deploy with the running stack untouched. +- [ ] **Backup validation works.** The dump must be valid gzip, contain `CREATE TABLE`, and end with + the `-- Dump completed` trailer; otherwise the file is deleted and the deploy stops. A SQLite + archive must contain `jobtracker.db`. +- [ ] **Filename confirms the type.** `jobtracker--.sql.gz` = MariaDB dump. + A `jobtracker-sqlite-*.tar.gz` on a MariaDB host means the environment is wrong. +- [ ] **Password never on the command line** — `deploy.sh` passes it via `MYSQL_PWD`. + +## Restore procedure + +Restore is a **separate** operation from code rollback. A bad deploy usually needs only the rollback; +restore the database **only if the data itself is wrong**, because it discards everything written since +the dump. + +```bash +# 1. Restore into a SEPARATE, empty database first — never straight over the live one. +gzip -dc /opt/job-tracker/backups/jobtracker--.sql.gz \ + | MYSQL_PWD='' mariadb --host= --user= --default-character-set=utf8mb4 jobtracker_scratch + +# 2. Sanity-check row counts and character fidelity (below), THEN, if replacing production: +docker compose stop backend +gzip -dc | MYSQL_PWD='' mariadb --host= --user= --default-character-set=utf8mb4 jobtracker +docker compose start backend +``` + +Always pass `--default-character-set=utf8mb4` on both dump and restore so multibyte text is not +mangled. + +## UTF-8 / Norwegian character verification — **VERIFIED 2026-07-19** + +The earlier rehearsal used ASCII-only seed data, so character fidelity was unproven. It has now been +tested explicitly through the real `deploy.sh` backup path. + +Seeded into a MariaDB 11 database on the real 42-table schema: + +| Field | Value | +|---|---| +| Company name | `Ærøskøbing Systemutvikling AS` | +| CV variant name | `Søknad – Bjørn Håkonsen` (note the em-dash `–`) | +| User email | `bjørn@dåg.no` | +| Career profile JSON | `Erfaren utvikler frå Tromsø … Språk: norsk … flåten … Morsmål` | + +Backed up with the real `backup_database` function, restored into a **clean** MariaDB 11 container, +compared byte-for-byte: + +``` +source HEX(Name): C38672C3B8736BC3B862696E672053797374656D757476696B6C696E67204153 +restored HEX(Name): C38672C3B8736BC3B862696E672053797374656D757476696B6C696E67204153 +BYTE-EXACT MATCH +``` + +`C386` = `Æ`, `C3B8` = `ø`, `C3A5` = `å` — correct UTF-8, not double-encoded or stripped. **æ ø å**, +plus the em-dash, survived the full dump → gzip → restore cycle unchanged. + +To repeat this against production after a restore: + +```bash +MYSQL_PWD='' mariadb --host= --user= --default-character-set=utf8mb4 -N -B jobtracker_scratch \ + -e "SELECT Name FROM Companies WHERE Name LIKE '%ø%' OR Name LIKE '%æ%' OR Name LIKE '%å%' LIMIT 5;" +# Read the output in a UTF-8 terminal. Mangled output (æ, ø) = charset problem in the pipeline. +``` + +## Production access + +**This environment has no route to the production database** — no `/opt/job-tracker`, no production +connection string, and the local stack runs SQLite. Per the task constraints, **no credential +discovery and no SSH guessing were attempted.** + +**Manual step the owner must perform** (only the owner has production access): + +1. On the production host, run one out-of-band backup: `deploy/deploy.sh` takes one automatically, or + dump by hand with the command in `deploy/README.md`. +2. Restore that dump into a **scratch** database (not production) and confirm: + - table count (~42) and row counts for `AspNetUsers`, `JobApplications`, `Companies`, + `CareerProfiles` match production; + - a real record containing `æ`/`ø`/`å` reads back correctly (the check above). + +Until that is done, backup/restore is proven **on the mechanism and on synthetic Norwegian data**, not +on the production dataset. diff --git a/docs/deployment/manual-smoke-test.md b/docs/deployment/manual-smoke-test.md new file mode 100644 index 0000000..1e6e9bb --- /dev/null +++ b/docs/deployment/manual-smoke-test.md @@ -0,0 +1,72 @@ +# Manual smoke test — post-deployment + +> Run this after every production deploy, in a real browser, signed in as a real user. +> +> **Authentication requires the owner.** No step here is automated and no password is handled by any +> tool or script — signing in is the owner's job. The automated suite verifies the authorization +> *boundary* exists (every user endpoint returns 401 unauthenticated); only a human with credentials +> can verify what is behind it. + +Each item names what **wrong** looks like, because "it loaded" is not a check. Stop and consider +rollback on any ✗. + +## Authentication + +- [ ] **Login succeeds** with an existing account. ✗ = password rejected, or a 5xx on submit. +- [ ] **Existing session works** — reload the page after login and stay signed in. ✗ = session drops + on refresh, which means `AUTH_JWT_KEY` changed between deploys. +- [ ] **Logout works** and returns to the signed-out state. ✗ = still authenticated after logout. + +## Applications + +- [ ] **Existing applications load.** ✗ = empty list for a user who had applications. +- [ ] **Counts are correct** — the number matches what you saw before the deploy. ✗ = any drop. This + is the single most important check. +- [ ] **Workspace opens** for one application — Overview, Checklist, Timeline, Analysis, Match render. + ✗ = a section erroring. Empty new sections on old applications are **correct**, not a fault. + +## Career profile + +- [ ] **Profile loads** with real experience, education and skills. +- [ ] **Languages are present.** ✗ = languages missing, or reduced to English only. That is the + `9681618` / `fba858e` ICU-dependence class of bug; if it reappears the container's ICU differs + from what was tested. Check a profile that lists Norwegian specifically. +- [ ] **Structured career data is intact** — experiences, education, projects, certifications all + show their fields, not blanks. +- [ ] **No data loss from the relational migration** — spot-check a profile edited before the deploy + against what you remember. ✗ = fields silently emptied. + +## CV builder + +- [ ] **Existing CV variants load** in the builder list; open one. +- [ ] **Editing works** — change a field, confirm autosave persists after reload. +- [ ] **Preview works** — the themed preview renders the variant. +- [ ] **PDF export works** — export produces a valid PDF (this exercises the in-container Chromium). + ✗ = export hangs or errors, usually a Chromium/`CV_PDF_BROWSER_PATH` problem. +- [ ] **Public CV loads directly after refresh** — open `/cv/` for an already-public variant, + then **hard-refresh**. ✗ = 404 on refresh, which is SPA deep-link routing, not the CV itself. + +## AI features + +- [ ] **AI generation works** — run one generation (interview prep or cover letter). ✗ = 5xx or an + indefinite hang. If `ai-service` is down the *deploy* still succeeds (AI is not a deploy gate), + so this must be checked by hand. +- [ ] **Suggestions are generated** and shown for review. +- [ ] **No unwanted writes occur** — the generation does **not** modify the Career Profile, a CV + variant, or application fields until you explicitly save. Confirm the source records are + unchanged after generating but before saving. ✗ = anything written without your action. + +## Files + +- [ ] **Attachment upload** — upload a file to an application; it appears in the list. +- [ ] **Attachment download** — download an existing attachment from an old application. ✗ = 404, + which means the `jobtracker_data` volume did not survive the deploy. +- [ ] **Permissions** — confirm you cannot reach another user's attachment. Signed in as user A, + requesting user B's attachment id must return 404/403, never the file. (The automated suite + already asserts tenant scoping; this is the human confirmation.) + +## After the checklist + +- [ ] `/health` reports the version you deployed, not `1.0.0.0` (the assembly fallback means + `APP_VERSION` did not reach the container). +- [ ] Row counts for applications and companies still match the pre-deploy numbers.