docs(deployment): backup-restore, smoke test, runner finding C
- docs/deployment/backup-restore.md: production backup checklist; documents that deploy.sh loads the env, validates before backup, and validates the dump. Adds VERIFIED UTF-8/Norwegian-character round trip (æ ø å survive a real deploy.sh backup -> restore byte-exact; HEX compared). States plainly that no production database was reached and the owner must run one real backup + scratch restore. - docs/deployment/manual-smoke-test.md: owner-run post-deploy checklist (auth, applications, career profile, CV builder, AI, files). Each item names what "wrong" looks like. Documents that login requires the owner. - runner-investigation.md: Finding C -- the latest CI red was a real ICU code bug the runner caught correctly, not instability. Amends the blanket "outside the repository" conclusion. A and B stand as separate env issues. - release-candidate-review.md: corrected drifted line refs after the index fix; noted the CI ICU finding so the "purely external" verdict is honest. All claims reflect behaviour verified this session. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
cesnimda
@@ -0,0 +1,102 @@
|
|||||||
|
# Production backup & restore
|
||||||
|
|
||||||
|
> 2026-07-19. The backup **mechanism** is verified end to end against MariaDB 11 containers, including a
|
||||||
|
> byte-exact Norwegian-character round trip. **No production database has been backed up from this
|
||||||
|
> environment** — see *Production access*. This document is the checklist to run against production,
|
||||||
|
> plus the evidence for what is already proven.
|
||||||
|
|
||||||
|
Companion to `deploy/deploy.sh` (the implementation), `deploy/README.md` (backup location/retention),
|
||||||
|
and `docs/operations/production-backup-verification.md` (the earlier container rehearsal).
|
||||||
|
|
||||||
|
## Production backup checklist
|
||||||
|
|
||||||
|
Run top to bottom. Every item is checked automatically by `deploy.sh` **before** it builds or replaces
|
||||||
|
anything — this list is for a manual pre-flight and for an out-of-band backup.
|
||||||
|
|
||||||
|
- [ ] **`DATABASE_PROVIDER=mariadb`** in `/opt/job-tracker/shared/.env`. No default — `deploy.sh`
|
||||||
|
aborts if it is missing. If it were wrong, the SQLite path would run and back up the wrong thing.
|
||||||
|
- [ ] **`JOBTRACKER_CONNECTION_STRING` present** and pointing at the production database. The host
|
||||||
|
resolves from the deploy shell, not from inside a container, for the backup step.
|
||||||
|
- [ ] **`deploy.sh` loads the environment.** It parses `/opt/job-tracker/shared/.env` into its own
|
||||||
|
shell before deciding anything (verified: the "Loaded deployment environment" line prints first).
|
||||||
|
- [ ] **Backup runs before replacement.** `validate_deploy_config` → `backup_database` → build →
|
||||||
|
`up -d --force-recreate`. Confirmed by line order in `deploy.sh` (validate/backup precede
|
||||||
|
build/replace). A failed backup aborts the deploy with the running stack untouched.
|
||||||
|
- [ ] **Backup validation works.** The dump must be valid gzip, contain `CREATE TABLE`, and end with
|
||||||
|
the `-- Dump completed` trailer; otherwise the file is deleted and the deploy stops. A SQLite
|
||||||
|
archive must contain `jobtracker.db`.
|
||||||
|
- [ ] **Filename confirms the type.** `jobtracker-<db>-<UTC>.sql.gz` = MariaDB dump.
|
||||||
|
A `jobtracker-sqlite-*.tar.gz` on a MariaDB host means the environment is wrong.
|
||||||
|
- [ ] **Password never on the command line** — `deploy.sh` passes it via `MYSQL_PWD`.
|
||||||
|
|
||||||
|
## Restore procedure
|
||||||
|
|
||||||
|
Restore is a **separate** operation from code rollback. A bad deploy usually needs only the rollback;
|
||||||
|
restore the database **only if the data itself is wrong**, because it discards everything written since
|
||||||
|
the dump.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 1. Restore into a SEPARATE, empty database first — never straight over the live one.
|
||||||
|
gzip -dc /opt/job-tracker/backups/jobtracker-<db>-<UTC>.sql.gz \
|
||||||
|
| MYSQL_PWD='<password>' mariadb --host=<host> --user=<user> --default-character-set=utf8mb4 jobtracker_scratch
|
||||||
|
|
||||||
|
# 2. Sanity-check row counts and character fidelity (below), THEN, if replacing production:
|
||||||
|
docker compose stop backend
|
||||||
|
gzip -dc <backup> | MYSQL_PWD='<password>' mariadb --host=<host> --user=<user> --default-character-set=utf8mb4 jobtracker
|
||||||
|
docker compose start backend
|
||||||
|
```
|
||||||
|
|
||||||
|
Always pass `--default-character-set=utf8mb4` on both dump and restore so multibyte text is not
|
||||||
|
mangled.
|
||||||
|
|
||||||
|
## UTF-8 / Norwegian character verification — **VERIFIED 2026-07-19**
|
||||||
|
|
||||||
|
The earlier rehearsal used ASCII-only seed data, so character fidelity was unproven. It has now been
|
||||||
|
tested explicitly through the real `deploy.sh` backup path.
|
||||||
|
|
||||||
|
Seeded into a MariaDB 11 database on the real 42-table schema:
|
||||||
|
|
||||||
|
| Field | Value |
|
||||||
|
|---|---|
|
||||||
|
| Company name | `Ærøskøbing Systemutvikling AS` |
|
||||||
|
| CV variant name | `Søknad – Bjørn Håkonsen` (note the em-dash `–`) |
|
||||||
|
| User email | `bjørn@dåg.no` |
|
||||||
|
| Career profile JSON | `Erfaren utvikler frå Tromsø … Språk: norsk … flåten … Morsmål` |
|
||||||
|
|
||||||
|
Backed up with the real `backup_database` function, restored into a **clean** MariaDB 11 container,
|
||||||
|
compared byte-for-byte:
|
||||||
|
|
||||||
|
```
|
||||||
|
source HEX(Name): C38672C3B8736BC3B862696E672053797374656D757476696B6C696E67204153
|
||||||
|
restored HEX(Name): C38672C3B8736BC3B862696E672053797374656D757476696B6C696E67204153
|
||||||
|
BYTE-EXACT MATCH
|
||||||
|
```
|
||||||
|
|
||||||
|
`C386` = `Æ`, `C3B8` = `ø`, `C3A5` = `å` — correct UTF-8, not double-encoded or stripped. **æ ø å**,
|
||||||
|
plus the em-dash, survived the full dump → gzip → restore cycle unchanged.
|
||||||
|
|
||||||
|
To repeat this against production after a restore:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
MYSQL_PWD='<pw>' mariadb --host=<host> --user=<user> --default-character-set=utf8mb4 -N -B jobtracker_scratch \
|
||||||
|
-e "SELECT Name FROM Companies WHERE Name LIKE '%ø%' OR Name LIKE '%æ%' OR Name LIKE '%å%' LIMIT 5;"
|
||||||
|
# Read the output in a UTF-8 terminal. Mangled output (æ, ø) = charset problem in the pipeline.
|
||||||
|
```
|
||||||
|
|
||||||
|
## Production access
|
||||||
|
|
||||||
|
**This environment has no route to the production database** — no `/opt/job-tracker`, no production
|
||||||
|
connection string, and the local stack runs SQLite. Per the task constraints, **no credential
|
||||||
|
discovery and no SSH guessing were attempted.**
|
||||||
|
|
||||||
|
**Manual step the owner must perform** (only the owner has production access):
|
||||||
|
|
||||||
|
1. On the production host, run one out-of-band backup: `deploy/deploy.sh` takes one automatically, or
|
||||||
|
dump by hand with the command in `deploy/README.md`.
|
||||||
|
2. Restore that dump into a **scratch** database (not production) and confirm:
|
||||||
|
- table count (~42) and row counts for `AspNetUsers`, `JobApplications`, `Companies`,
|
||||||
|
`CareerProfiles` match production;
|
||||||
|
- a real record containing `æ`/`ø`/`å` reads back correctly (the check above).
|
||||||
|
|
||||||
|
Until that is done, backup/restore is proven **on the mechanism and on synthetic Norwegian data**, not
|
||||||
|
on the production dataset.
|
||||||
@@ -0,0 +1,72 @@
|
|||||||
|
# Manual smoke test — post-deployment
|
||||||
|
|
||||||
|
> Run this after every production deploy, in a real browser, signed in as a real user.
|
||||||
|
>
|
||||||
|
> **Authentication requires the owner.** No step here is automated and no password is handled by any
|
||||||
|
> tool or script — signing in is the owner's job. The automated suite verifies the authorization
|
||||||
|
> *boundary* exists (every user endpoint returns 401 unauthenticated); only a human with credentials
|
||||||
|
> can verify what is behind it.
|
||||||
|
|
||||||
|
Each item names what **wrong** looks like, because "it loaded" is not a check. Stop and consider
|
||||||
|
rollback on any ✗.
|
||||||
|
|
||||||
|
## Authentication
|
||||||
|
|
||||||
|
- [ ] **Login succeeds** with an existing account. ✗ = password rejected, or a 5xx on submit.
|
||||||
|
- [ ] **Existing session works** — reload the page after login and stay signed in. ✗ = session drops
|
||||||
|
on refresh, which means `AUTH_JWT_KEY` changed between deploys.
|
||||||
|
- [ ] **Logout works** and returns to the signed-out state. ✗ = still authenticated after logout.
|
||||||
|
|
||||||
|
## Applications
|
||||||
|
|
||||||
|
- [ ] **Existing applications load.** ✗ = empty list for a user who had applications.
|
||||||
|
- [ ] **Counts are correct** — the number matches what you saw before the deploy. ✗ = any drop. This
|
||||||
|
is the single most important check.
|
||||||
|
- [ ] **Workspace opens** for one application — Overview, Checklist, Timeline, Analysis, Match render.
|
||||||
|
✗ = a section erroring. Empty new sections on old applications are **correct**, not a fault.
|
||||||
|
|
||||||
|
## Career profile
|
||||||
|
|
||||||
|
- [ ] **Profile loads** with real experience, education and skills.
|
||||||
|
- [ ] **Languages are present.** ✗ = languages missing, or reduced to English only. That is the
|
||||||
|
`9681618` / `fba858e` ICU-dependence class of bug; if it reappears the container's ICU differs
|
||||||
|
from what was tested. Check a profile that lists Norwegian specifically.
|
||||||
|
- [ ] **Structured career data is intact** — experiences, education, projects, certifications all
|
||||||
|
show their fields, not blanks.
|
||||||
|
- [ ] **No data loss from the relational migration** — spot-check a profile edited before the deploy
|
||||||
|
against what you remember. ✗ = fields silently emptied.
|
||||||
|
|
||||||
|
## CV builder
|
||||||
|
|
||||||
|
- [ ] **Existing CV variants load** in the builder list; open one.
|
||||||
|
- [ ] **Editing works** — change a field, confirm autosave persists after reload.
|
||||||
|
- [ ] **Preview works** — the themed preview renders the variant.
|
||||||
|
- [ ] **PDF export works** — export produces a valid PDF (this exercises the in-container Chromium).
|
||||||
|
✗ = export hangs or errors, usually a Chromium/`CV_PDF_BROWSER_PATH` problem.
|
||||||
|
- [ ] **Public CV loads directly after refresh** — open `/cv/<slug>` for an already-public variant,
|
||||||
|
then **hard-refresh**. ✗ = 404 on refresh, which is SPA deep-link routing, not the CV itself.
|
||||||
|
|
||||||
|
## AI features
|
||||||
|
|
||||||
|
- [ ] **AI generation works** — run one generation (interview prep or cover letter). ✗ = 5xx or an
|
||||||
|
indefinite hang. If `ai-service` is down the *deploy* still succeeds (AI is not a deploy gate),
|
||||||
|
so this must be checked by hand.
|
||||||
|
- [ ] **Suggestions are generated** and shown for review.
|
||||||
|
- [ ] **No unwanted writes occur** — the generation does **not** modify the Career Profile, a CV
|
||||||
|
variant, or application fields until you explicitly save. Confirm the source records are
|
||||||
|
unchanged after generating but before saving. ✗ = anything written without your action.
|
||||||
|
|
||||||
|
## Files
|
||||||
|
|
||||||
|
- [ ] **Attachment upload** — upload a file to an application; it appears in the list.
|
||||||
|
- [ ] **Attachment download** — download an existing attachment from an old application. ✗ = 404,
|
||||||
|
which means the `jobtracker_data` volume did not survive the deploy.
|
||||||
|
- [ ] **Permissions** — confirm you cannot reach another user's attachment. Signed in as user A,
|
||||||
|
requesting user B's attachment id must return 404/403, never the file. (The automated suite
|
||||||
|
already asserts tenant scoping; this is the human confirmation.)
|
||||||
|
|
||||||
|
## After the checklist
|
||||||
|
|
||||||
|
- [ ] `/health` reports the version you deployed, not `1.0.0.0` (the assembly fallback means
|
||||||
|
`APP_VERSION` did not reach the container).
|
||||||
|
- [ ] Row counts for applications and companies still match the pre-deploy numbers.
|
||||||
Reference in New Issue
Block a user