docs(deployment): backup-restore, smoke test, runner finding C
- docs/deployment/backup-restore.md: production backup checklist; documents that deploy.sh loads the env, validates before backup, and validates the dump. Adds VERIFIED UTF-8/Norwegian-character round trip (æ ø å survive a real deploy.sh backup -> restore byte-exact; HEX compared). States plainly that no production database was reached and the owner must run one real backup + scratch restore. - docs/deployment/manual-smoke-test.md: owner-run post-deploy checklist (auth, applications, career profile, CV builder, AI, files). Each item names what "wrong" looks like. Documents that login requires the owner. - runner-investigation.md: Finding C -- the latest CI red was a real ICU code bug the runner caught correctly, not instability. Amends the blanket "outside the repository" conclusion. A and B stand as separate env issues. - release-candidate-review.md: corrected drifted line refs after the index fix; noted the CI ICU finding so the "purely external" verdict is honest. All claims reflect behaviour verified this session. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
cesnimda
@@ -0,0 +1,102 @@
|
||||
# Production backup & restore
|
||||
|
||||
> 2026-07-19. The backup **mechanism** is verified end to end against MariaDB 11 containers, including a
|
||||
> byte-exact Norwegian-character round trip. **No production database has been backed up from this
|
||||
> environment** — see *Production access*. This document is the checklist to run against production,
|
||||
> plus the evidence for what is already proven.
|
||||
|
||||
Companion to `deploy/deploy.sh` (the implementation), `deploy/README.md` (backup location/retention),
|
||||
and `docs/operations/production-backup-verification.md` (the earlier container rehearsal).
|
||||
|
||||
## Production backup checklist
|
||||
|
||||
Run top to bottom. Every item is checked automatically by `deploy.sh` **before** it builds or replaces
|
||||
anything — this list is for a manual pre-flight and for an out-of-band backup.
|
||||
|
||||
- [ ] **`DATABASE_PROVIDER=mariadb`** in `/opt/job-tracker/shared/.env`. No default — `deploy.sh`
|
||||
aborts if it is missing. If it were wrong, the SQLite path would run and back up the wrong thing.
|
||||
- [ ] **`JOBTRACKER_CONNECTION_STRING` present** and pointing at the production database. The host
|
||||
resolves from the deploy shell, not from inside a container, for the backup step.
|
||||
- [ ] **`deploy.sh` loads the environment.** It parses `/opt/job-tracker/shared/.env` into its own
|
||||
shell before deciding anything (verified: the "Loaded deployment environment" line prints first).
|
||||
- [ ] **Backup runs before replacement.** `validate_deploy_config` → `backup_database` → build →
|
||||
`up -d --force-recreate`. Confirmed by line order in `deploy.sh` (validate/backup precede
|
||||
build/replace). A failed backup aborts the deploy with the running stack untouched.
|
||||
- [ ] **Backup validation works.** The dump must be valid gzip, contain `CREATE TABLE`, and end with
|
||||
the `-- Dump completed` trailer; otherwise the file is deleted and the deploy stops. A SQLite
|
||||
archive must contain `jobtracker.db`.
|
||||
- [ ] **Filename confirms the type.** `jobtracker-<db>-<UTC>.sql.gz` = MariaDB dump.
|
||||
A `jobtracker-sqlite-*.tar.gz` on a MariaDB host means the environment is wrong.
|
||||
- [ ] **Password never on the command line** — `deploy.sh` passes it via `MYSQL_PWD`.
|
||||
|
||||
## Restore procedure
|
||||
|
||||
Restore is a **separate** operation from code rollback. A bad deploy usually needs only the rollback;
|
||||
restore the database **only if the data itself is wrong**, because it discards everything written since
|
||||
the dump.
|
||||
|
||||
```bash
|
||||
# 1. Restore into a SEPARATE, empty database first — never straight over the live one.
|
||||
gzip -dc /opt/job-tracker/backups/jobtracker-<db>-<UTC>.sql.gz \
|
||||
| MYSQL_PWD='<password>' mariadb --host=<host> --user=<user> --default-character-set=utf8mb4 jobtracker_scratch
|
||||
|
||||
# 2. Sanity-check row counts and character fidelity (below), THEN, if replacing production:
|
||||
docker compose stop backend
|
||||
gzip -dc <backup> | MYSQL_PWD='<password>' mariadb --host=<host> --user=<user> --default-character-set=utf8mb4 jobtracker
|
||||
docker compose start backend
|
||||
```
|
||||
|
||||
Always pass `--default-character-set=utf8mb4` on both dump and restore so multibyte text is not
|
||||
mangled.
|
||||
|
||||
## UTF-8 / Norwegian character verification — **VERIFIED 2026-07-19**
|
||||
|
||||
The earlier rehearsal used ASCII-only seed data, so character fidelity was unproven. It has now been
|
||||
tested explicitly through the real `deploy.sh` backup path.
|
||||
|
||||
Seeded into a MariaDB 11 database on the real 42-table schema:
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Company name | `Ærøskøbing Systemutvikling AS` |
|
||||
| CV variant name | `Søknad – Bjørn Håkonsen` (note the em-dash `–`) |
|
||||
| User email | `bjørn@dåg.no` |
|
||||
| Career profile JSON | `Erfaren utvikler frå Tromsø … Språk: norsk … flåten … Morsmål` |
|
||||
|
||||
Backed up with the real `backup_database` function, restored into a **clean** MariaDB 11 container,
|
||||
compared byte-for-byte:
|
||||
|
||||
```
|
||||
source HEX(Name): C38672C3B8736BC3B862696E672053797374656D757476696B6C696E67204153
|
||||
restored HEX(Name): C38672C3B8736BC3B862696E672053797374656D757476696B6C696E67204153
|
||||
BYTE-EXACT MATCH
|
||||
```
|
||||
|
||||
`C386` = `Æ`, `C3B8` = `ø`, `C3A5` = `å` — correct UTF-8, not double-encoded or stripped. **æ ø å**,
|
||||
plus the em-dash, survived the full dump → gzip → restore cycle unchanged.
|
||||
|
||||
To repeat this against production after a restore:
|
||||
|
||||
```bash
|
||||
MYSQL_PWD='<pw>' mariadb --host=<host> --user=<user> --default-character-set=utf8mb4 -N -B jobtracker_scratch \
|
||||
-e "SELECT Name FROM Companies WHERE Name LIKE '%ø%' OR Name LIKE '%æ%' OR Name LIKE '%å%' LIMIT 5;"
|
||||
# Read the output in a UTF-8 terminal. Mangled output (æ, ø) = charset problem in the pipeline.
|
||||
```
|
||||
|
||||
## Production access
|
||||
|
||||
**This environment has no route to the production database** — no `/opt/job-tracker`, no production
|
||||
connection string, and the local stack runs SQLite. Per the task constraints, **no credential
|
||||
discovery and no SSH guessing were attempted.**
|
||||
|
||||
**Manual step the owner must perform** (only the owner has production access):
|
||||
|
||||
1. On the production host, run one out-of-band backup: `deploy/deploy.sh` takes one automatically, or
|
||||
dump by hand with the command in `deploy/README.md`.
|
||||
2. Restore that dump into a **scratch** database (not production) and confirm:
|
||||
- table count (~42) and row counts for `AspNetUsers`, `JobApplications`, `Companies`,
|
||||
`CareerProfiles` match production;
|
||||
- a real record containing `æ`/`ø`/`å` reads back correctly (the check above).
|
||||
|
||||
Until that is done, backup/restore is proven **on the mechanism and on synthetic Norwegian data**, not
|
||||
on the production dataset.
|
||||
Reference in New Issue
Block a user