diff --git a/docs/operations/production-backup-verification.md b/docs/operations/production-backup-verification.md new file mode 100644 index 0000000..11b5ae9 --- /dev/null +++ b/docs/operations/production-backup-verification.md @@ -0,0 +1,141 @@ +# Backup and restore verification + +> **2026-07-19 19:32 UTC.** Full backup → verify → restore → start-the-app rehearsal of the +> `deploy/deploy.sh` backup path, exercised end to end against MariaDB 11. +> +> **This is a rehearsal, not a verification of production data.** See *Limitations* — that section is +> the most important part of this document, and the checklist at the end is what actually closes the +> gap. + +## Scope — read this first + +| | | +|---|---| +| **What was verified** | The backup *mechanism*: the real `backup_database` function from `deploy.sh`, against a real MariaDB 11 database carrying the real 42-table schema, restored into a separate clean MariaDB 11 container, with the real application started against the result | +| **What was NOT verified** | **Your production database.** No production host was contacted, no production credentials were used, and no production data was read, copied or restored | +| **Why** | The machine this ran on has no route to production: no `/opt/job-tracker`, no `DATABASE_PROVIDER` or `JOBTRACKER_CONNECTION_STRING` in its `.env`, and the local stack runs SQLite. The production host, user and key are CI secrets (`PROD_HOST`, `PROD_USER`, `PROD_SSH_KEY`) that are not available here | + +The commands below are the ones to run against production. They are recorded so the owner can execute +the same sequence with production values substituted. + +## 1. Backup configuration + +Read from `deploy/deploy.sh` and `docker-compose.yml`: + +- `deploy.sh` loads `/opt/job-tracker/shared/.env` into its own shell before any decision. +- `validate_deploy_config` runs **before** the backup, and therefore before any build, stop or replace. + It requires `DATABASE_PROVIDER` (no default), `JOBTRACKER_CONNECTION_STRING` when the provider is + MariaDB, plus `AI_SERVICE_TOKEN` and `AUTH_JWT_KEY`. +- Backups land in `/opt/job-tracker/backups` (override with `BACKUP_DIR`), UTC-timestamped, gzipped. + Nothing is ever overwritten and nothing is ever auto-deleted. +- The password travels via `MYSQL_PWD`, never on the command line. + +## 2. Provider resolution + +`DATABASE_PROVIDER=mariadb` selected the MariaDB dump path. Confirmed by the script's own output: + +``` +Deployment configuration validated (database provider: mariadb). +Backing up MariaDB database 'jobtracker' on : to /jobtracker-jobtracker-20260719T193022Z.sql.gz +Backup verified: /jobtracker-jobtracker-20260719T193022Z.sql.gz (8.0K) +``` + +**The filename is the check that matters.** `jobtracker--.sql.gz` means the MariaDB +path ran. A `jobtracker-sqlite-.tar.gz` on a MariaDB host would mean the environment is wrong. + +## 3. Commands used + +Schema built by the real application, not by hand — `Database.Migrate()` plus the startup reconciler +against an empty MariaDB, producing 42 tables. Representative rows were then seeded across users, +companies, applications, career profile and CV variants. + +```bash +# Backup — the real function from deploy.sh, not a hand-written dump +DATABASE_PROVIDER=mariadb \ +JOBTRACKER_CONNECTION_STRING='Server=;Port=;Database=jobtracker;User Id=;Password=;' \ +BACKUP_DIR=/opt/job-tracker/backups \ + deploy/deploy.sh # takes the backup first and aborts the deploy if it fails + +# Integrity and content +gzip -t "$BACKUP" # valid archive +gzip -dc "$BACKUP" | grep -c 'CREATE TABLE' # schema present +gzip -dc "$BACKUP" | tail -1 # "-- Dump completed on ..." trailer + +# Restore into a SEPARATE, empty database — never over a live one +gzip -dc "$BACKUP" | MYSQL_PWD='' mariadb --host= --user= jobtracker +``` + +Passwords are supplied via `MYSQL_PWD` so they never reach the process list or the shell history. + +## 4. Backup verification result + +| Check | Result | +|---|---| +| Correct backup type | ✅ `.sql.gz` MariaDB dump, not a volume archive | +| File integrity | ✅ `gzip -t` passed | +| Schema markers | ✅ 42 `CREATE TABLE` statements | +| Dump trailer | ✅ `-- Dump completed on 2026-07-19 19:30:23` — proves the dump was not truncated | +| Expected tables | ✅ `AspNetUsers`, `Companies`, `JobApplications`, `CareerProfiles`, `CvVariants`, `JobEvents`, `ApplicationChecklistItems`, `AiInteractions` all present | +| Data present | ✅ 7 `INSERT INTO` statements | + +## 5. Restore result + +Restored into a **separate, empty** MariaDB 11 container (0 tables before, 42 after). + +| Entity | Source | Restored | | +|---|---|---|---| +| Users | 1 | 1 | ✅ | +| Companies | 2 | 2 | ✅ | +| Applications | 2 | 2 | ✅ | +| Career profiles | 1 | 1 | ✅ | +| CV variants | 1 | 1 | ✅ | + +Beyond counts: + +- **All 42 tables compared** — table lists identical, and **every table's row count matched**. +- **Content survived, not just cardinality** — the restored applications resolve their company + foreign keys (`Senior Backend Engineer @ Northwind Consulting`), the CV variant kept its name and + `IsPublic` flag, and the career profile JSON still contained its languages. +- **The application starts against the restored database.** `/health` returned + `{"status":"ok","version":"restore-rehearsal"}`, the schema stayed at 42 tables (the reconciler + correctly found nothing to do) and rows were preserved. A restore that produces a database the app + cannot boot against is not a restore. + +## Limitations + +Read these before treating the deployment as backed up. + +1. **No production data was touched.** Everything above ran against a locally built MariaDB with + seeded rows. It proves the mechanism; it proves nothing about your database. +2. **The seeded dataset is tiny** (7 rows). It does not exercise dump duration, disk headroom, lock + behaviour under load, or timeout limits on a real dataset. A production database large enough to + make `mariadb-dump` slow could behave differently. +3. **Character-set and collation fidelity was not stress-tested.** The seeded data was mostly ASCII. + Real CV content contains non-ASCII text — Norwegian `æøå`, accents, CJK. The dump defaults should + handle this, but it was not proven here. +4. **No restore was performed over a populated database.** The destination was empty. Restoring over + an existing database is a different operation with different failure modes. +5. **`deploy.sh` was exercised up to and including the backup**, not through the build and container + replacement, which would have required a full deployment. +6. **The `mariadb-dump` client came from a container** (`mariadb:11`). If the production host has its + own client installed, `deploy.sh` uses that instead, and version differences are possible. + +## What the owner still needs to do + +This is the checklist that turns a rehearsal into a verified backup. + +- [ ] **Run one backup by hand against production** and confirm the filename is + `jobtracker--.sql.gz`. +- [ ] **Check the dump size is plausible** for the amount of data you have. A suspiciously small file + is the signal worth catching. +- [ ] **Restore it into a scratch database** — never over the live one — and confirm row counts for + `AspNetUsers`, `JobApplications`, `Companies` and `CareerProfiles` match production. +- [ ] **Check non-ASCII text survived.** Open one CV or career profile containing `æ`, `ø` or `å` in + the restored copy and confirm it is not mangled. This is the most likely silent failure. +- [ ] **Confirm `/opt/job-tracker/backups` has disk headroom** for several dumps. +- [ ] **Note how long the dump takes.** It runs before every deploy and blocks it. + +## Cleanup + +Both MariaDB containers and all rehearsal files were removed. No test artefacts remain, and the +existing local `jobtracker-backend-1` / `jobtracker-frontend-1` stack was left untouched.