cesnimda
57fabe9a97
- docs/deployment/backup-restore.md: production backup checklist; documents that deploy.sh loads the env, validates before backup, and validates the dump. Adds VERIFIED UTF-8/Norwegian-character round trip (æ ø å survive a real deploy.sh backup -> restore byte-exact; HEX compared). States plainly that no production database was reached and the owner must run one real backup + scratch restore. - docs/deployment/manual-smoke-test.md: owner-run post-deploy checklist (auth, applications, career profile, CV builder, AI, files). Each item names what "wrong" looks like. Documents that login requires the owner. - runner-investigation.md: Finding C -- the latest CI red was a real ICU code bug the runner caught correctly, not instability. Amends the blanket "outside the repository" conclusion. A and B stand as separate env issues. - release-candidate-review.md: corrected drifted line refs after the index fix; noted the CI ICU finding so the "purely external" verdict is honest. All claims reflect behaviour verified this session. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
103 lines
5.4 KiB
Markdown
103 lines
5.4 KiB
Markdown
# Production backup & restore
|
||
|
||
> 2026-07-19. The backup **mechanism** is verified end to end against MariaDB 11 containers, including a
|
||
> byte-exact Norwegian-character round trip. **No production database has been backed up from this
|
||
> environment** — see *Production access*. This document is the checklist to run against production,
|
||
> plus the evidence for what is already proven.
|
||
|
||
Companion to `deploy/deploy.sh` (the implementation), `deploy/README.md` (backup location/retention),
|
||
and `docs/operations/production-backup-verification.md` (the earlier container rehearsal).
|
||
|
||
## Production backup checklist
|
||
|
||
Run top to bottom. Every item is checked automatically by `deploy.sh` **before** it builds or replaces
|
||
anything — this list is for a manual pre-flight and for an out-of-band backup.
|
||
|
||
- [ ] **`DATABASE_PROVIDER=mariadb`** in `/opt/job-tracker/shared/.env`. No default — `deploy.sh`
|
||
aborts if it is missing. If it were wrong, the SQLite path would run and back up the wrong thing.
|
||
- [ ] **`JOBTRACKER_CONNECTION_STRING` present** and pointing at the production database. The host
|
||
resolves from the deploy shell, not from inside a container, for the backup step.
|
||
- [ ] **`deploy.sh` loads the environment.** It parses `/opt/job-tracker/shared/.env` into its own
|
||
shell before deciding anything (verified: the "Loaded deployment environment" line prints first).
|
||
- [ ] **Backup runs before replacement.** `validate_deploy_config` → `backup_database` → build →
|
||
`up -d --force-recreate`. Confirmed by line order in `deploy.sh` (validate/backup precede
|
||
build/replace). A failed backup aborts the deploy with the running stack untouched.
|
||
- [ ] **Backup validation works.** The dump must be valid gzip, contain `CREATE TABLE`, and end with
|
||
the `-- Dump completed` trailer; otherwise the file is deleted and the deploy stops. A SQLite
|
||
archive must contain `jobtracker.db`.
|
||
- [ ] **Filename confirms the type.** `jobtracker-<db>-<UTC>.sql.gz` = MariaDB dump.
|
||
A `jobtracker-sqlite-*.tar.gz` on a MariaDB host means the environment is wrong.
|
||
- [ ] **Password never on the command line** — `deploy.sh` passes it via `MYSQL_PWD`.
|
||
|
||
## Restore procedure
|
||
|
||
Restore is a **separate** operation from code rollback. A bad deploy usually needs only the rollback;
|
||
restore the database **only if the data itself is wrong**, because it discards everything written since
|
||
the dump.
|
||
|
||
```bash
|
||
# 1. Restore into a SEPARATE, empty database first — never straight over the live one.
|
||
gzip -dc /opt/job-tracker/backups/jobtracker-<db>-<UTC>.sql.gz \
|
||
| MYSQL_PWD='<password>' mariadb --host=<host> --user=<user> --default-character-set=utf8mb4 jobtracker_scratch
|
||
|
||
# 2. Sanity-check row counts and character fidelity (below), THEN, if replacing production:
|
||
docker compose stop backend
|
||
gzip -dc <backup> | MYSQL_PWD='<password>' mariadb --host=<host> --user=<user> --default-character-set=utf8mb4 jobtracker
|
||
docker compose start backend
|
||
```
|
||
|
||
Always pass `--default-character-set=utf8mb4` on both dump and restore so multibyte text is not
|
||
mangled.
|
||
|
||
## UTF-8 / Norwegian character verification — **VERIFIED 2026-07-19**
|
||
|
||
The earlier rehearsal used ASCII-only seed data, so character fidelity was unproven. It has now been
|
||
tested explicitly through the real `deploy.sh` backup path.
|
||
|
||
Seeded into a MariaDB 11 database on the real 42-table schema:
|
||
|
||
| Field | Value |
|
||
|---|---|
|
||
| Company name | `Ærøskøbing Systemutvikling AS` |
|
||
| CV variant name | `Søknad – Bjørn Håkonsen` (note the em-dash `–`) |
|
||
| User email | `bjørn@dåg.no` |
|
||
| Career profile JSON | `Erfaren utvikler frå Tromsø … Språk: norsk … flåten … Morsmål` |
|
||
|
||
Backed up with the real `backup_database` function, restored into a **clean** MariaDB 11 container,
|
||
compared byte-for-byte:
|
||
|
||
```
|
||
source HEX(Name): C38672C3B8736BC3B862696E672053797374656D757476696B6C696E67204153
|
||
restored HEX(Name): C38672C3B8736BC3B862696E672053797374656D757476696B6C696E67204153
|
||
BYTE-EXACT MATCH
|
||
```
|
||
|
||
`C386` = `Æ`, `C3B8` = `ø`, `C3A5` = `å` — correct UTF-8, not double-encoded or stripped. **æ ø å**,
|
||
plus the em-dash, survived the full dump → gzip → restore cycle unchanged.
|
||
|
||
To repeat this against production after a restore:
|
||
|
||
```bash
|
||
MYSQL_PWD='<pw>' mariadb --host=<host> --user=<user> --default-character-set=utf8mb4 -N -B jobtracker_scratch \
|
||
-e "SELECT Name FROM Companies WHERE Name LIKE '%ø%' OR Name LIKE '%æ%' OR Name LIKE '%å%' LIMIT 5;"
|
||
# Read the output in a UTF-8 terminal. Mangled output (æ, ø) = charset problem in the pipeline.
|
||
```
|
||
|
||
## Production access
|
||
|
||
**This environment has no route to the production database** — no `/opt/job-tracker`, no production
|
||
connection string, and the local stack runs SQLite. Per the task constraints, **no credential
|
||
discovery and no SSH guessing were attempted.**
|
||
|
||
**Manual step the owner must perform** (only the owner has production access):
|
||
|
||
1. On the production host, run one out-of-band backup: `deploy/deploy.sh` takes one automatically, or
|
||
dump by hand with the command in `deploy/README.md`.
|
||
2. Restore that dump into a **scratch** database (not production) and confirm:
|
||
- table count (~42) and row counts for `AspNetUsers`, `JobApplications`, `Companies`,
|
||
`CareerProfiles` match production;
|
||
- a real record containing `æ`/`ø`/`å` reads back correctly (the check above).
|
||
|
||
Until that is done, backup/restore is proven **on the mechanism and on synthetic Norwegian data**, not
|
||
on the production dataset.
|