cesnimda
155 lines
8.7 KiB
Markdown
155 lines
8.7 KiB
Markdown
# Backup and restore verification
|
|
|
|
> **2026-07-19 19:32 UTC.** Full backup → verify → restore → start-the-app rehearsal of the
|
|
> `deploy/deploy.sh` backup path, exercised end to end against MariaDB 11.
|
|
>
|
|
> **This is a rehearsal, not a verification of production data.** See *Limitations* — that section is
|
|
> the most important part of this document, and the checklist at the end is what actually closes the
|
|
> gap.
|
|
|
|
## Scope — read this first
|
|
|
|
| | |
|
|
|---|---|
|
|
| **What was verified** | The backup *mechanism*: the real `backup_database` function from `deploy.sh`, against a real MariaDB 11 database carrying the real 42-table schema, restored into a separate clean MariaDB 11 container, with the real application started against the result |
|
|
| **What was NOT verified** | **Your production database.** No production host was contacted, no production credentials were used, and no production data was read, copied or restored |
|
|
| **Why** | The machine this ran on has no route to production: no `/opt/job-tracker`, no `DATABASE_PROVIDER` or `JOBTRACKER_CONNECTION_STRING` in its `.env`, and the local stack runs SQLite. The production host, user and key are CI secrets (`PROD_HOST`, `PROD_USER`, `PROD_SSH_KEY`) that are not available here |
|
|
|
|
The commands below are the ones to run against production. They are recorded so the owner can execute
|
|
the same sequence with production values substituted.
|
|
|
|
## 2026-08-15 read-only production checkpoint
|
|
|
|
The earlier access statement above remains historically accurate for the 2026-07-19 rehearsal, but access is now available. A strictly read-only production inventory found:
|
|
|
|
- 21 MariaDB `.sql.gz` dumps (about 5.2 MB combined), all passing `gzip -t`;
|
|
- oldest observed dump: 2026-07-19 21:41 UTC; newest: 2026-08-02 17:32 UTC;
|
|
- no JobTracker systemd timer or current-user cron entry;
|
|
- 1.4 TiB free on the Docker/data filesystem and 36 GiB free on the 83%-used root filesystem;
|
|
- database-only dump files in the backup directory: no owner-file volume, data-protection-key, protected tombstone, or configuration recovery bundle;
|
|
- no restore, row/content read, count comparison, non-ASCII check, or production mutation.
|
|
|
|
This upgrades production dump *presence and compressed-stream integrity* from unknown to observed, but it does not close the restore/RPO/completeness checklist. The newest observed dump was 13 days old at capture. See `docs/production/production-ai-hardware-assessment.md` and `docs/production/production-ai-rollout-and-rollback.md`.
|
|
|
|
## 1. Backup configuration
|
|
|
|
Read from `deploy/deploy.sh` and `docker-compose.yml`:
|
|
|
|
- `deploy.sh` loads `/opt/job-tracker/shared/.env` into its own shell before any decision.
|
|
- `validate_deploy_config` runs **before** the backup, and therefore before any build, stop or replace.
|
|
It requires `DATABASE_PROVIDER` (no default), `JOBTRACKER_CONNECTION_STRING` when the provider is
|
|
MariaDB, plus `AI_SERVICE_TOKEN` and `AUTH_JWT_KEY`.
|
|
- Backups land in `/opt/job-tracker/backups` (override with `BACKUP_DIR`), UTC-timestamped, gzipped.
|
|
Nothing is ever overwritten and nothing is ever auto-deleted.
|
|
- The password travels via `MYSQL_PWD`, never on the command line.
|
|
|
|
## 2. Provider resolution
|
|
|
|
`DATABASE_PROVIDER=mariadb` selected the MariaDB dump path. Confirmed by the script's own output:
|
|
|
|
```
|
|
Deployment configuration validated (database provider: mariadb).
|
|
Backing up MariaDB database 'jobtracker' on <host>:<port> to <dir>/jobtracker-jobtracker-20260719T193022Z.sql.gz
|
|
Backup verified: <dir>/jobtracker-jobtracker-20260719T193022Z.sql.gz (8.0K)
|
|
```
|
|
|
|
**The filename is the check that matters.** `jobtracker-<database>-<stamp>.sql.gz` means the MariaDB
|
|
path ran. A `jobtracker-sqlite-<stamp>.tar.gz` on a MariaDB host would mean the environment is wrong.
|
|
|
|
## 3. Commands used
|
|
|
|
Schema built by the real application, not by hand — `Database.Migrate()` plus the startup reconciler
|
|
against an empty MariaDB, producing 42 tables. Representative rows were then seeded across users,
|
|
companies, applications, career profile and CV variants.
|
|
|
|
```bash
|
|
# Backup — the real function from deploy.sh, not a hand-written dump
|
|
DATABASE_PROVIDER=mariadb \
|
|
JOBTRACKER_CONNECTION_STRING='Server=<host>;Port=<port>;Database=jobtracker;User Id=<user>;Password=<password>;' \
|
|
BACKUP_DIR=/opt/job-tracker/backups \
|
|
deploy/deploy.sh # takes the backup first and aborts the deploy if it fails
|
|
|
|
# Integrity and content
|
|
gzip -t "$BACKUP" # valid archive
|
|
gzip -dc "$BACKUP" | grep -c 'CREATE TABLE' # schema present
|
|
gzip -dc "$BACKUP" | tail -1 # "-- Dump completed on ..." trailer
|
|
|
|
# Restore into a SEPARATE, empty database — never over a live one
|
|
gzip -dc "$BACKUP" | MYSQL_PWD='<password>' mariadb --host=<host> --user=<user> jobtracker
|
|
```
|
|
|
|
Passwords are supplied via `MYSQL_PWD` so they never reach the process list or the shell history.
|
|
|
|
## 4. Backup verification result
|
|
|
|
| Check | Result |
|
|
|---|---|
|
|
| Correct backup type | ✅ `.sql.gz` MariaDB dump, not a volume archive |
|
|
| File integrity | ✅ `gzip -t` passed |
|
|
| Schema markers | ✅ 42 `CREATE TABLE` statements |
|
|
| Dump trailer | ✅ `-- Dump completed on 2026-07-19 19:30:23` — proves the dump was not truncated |
|
|
| Expected tables | ✅ `AspNetUsers`, `Companies`, `JobApplications`, `CareerProfiles`, `CvVariants`, `JobEvents`, `ApplicationChecklistItems`, `AiInteractions` all present |
|
|
| Data present | ✅ 7 `INSERT INTO` statements |
|
|
|
|
## 5. Restore result
|
|
|
|
Restored into a **separate, empty** MariaDB 11 container (0 tables before, 42 after).
|
|
|
|
| Entity | Source | Restored | |
|
|
|---|---|---|---|
|
|
| Users | 1 | 1 | ✅ |
|
|
| Companies | 2 | 2 | ✅ |
|
|
| Applications | 2 | 2 | ✅ |
|
|
| Career profiles | 1 | 1 | ✅ |
|
|
| CV variants | 1 | 1 | ✅ |
|
|
|
|
Beyond counts:
|
|
|
|
- **All 42 tables compared** — table lists identical, and **every table's row count matched**.
|
|
- **Content survived, not just cardinality** — the restored applications resolve their company
|
|
foreign keys (`Senior Backend Engineer @ Northwind Consulting`), the CV variant kept its name and
|
|
`IsPublic` flag, and the career profile JSON still contained its languages.
|
|
- **The application starts against the restored database.** `/health` returned
|
|
`{"status":"ok","version":"restore-rehearsal"}`, the schema stayed at 42 tables (the reconciler
|
|
correctly found nothing to do) and rows were preserved. A restore that produces a database the app
|
|
cannot boot against is not a restore.
|
|
|
|
## Limitations
|
|
|
|
Read these before treating the deployment as backed up.
|
|
|
|
1. **No production data was touched.** Everything above ran against a locally built MariaDB with
|
|
seeded rows. It proves the mechanism; it proves nothing about your database.
|
|
2. **The seeded dataset is tiny** (7 rows). It does not exercise dump duration, disk headroom, lock
|
|
behaviour under load, or timeout limits on a real dataset. A production database large enough to
|
|
make `mariadb-dump` slow could behave differently.
|
|
3. **Character-set and collation fidelity was not stress-tested.** The seeded data was mostly ASCII.
|
|
Real CV content contains non-ASCII text — Norwegian `æøå`, accents, CJK. The dump defaults should
|
|
handle this, but it was not proven here.
|
|
4. **No restore was performed over a populated database.** The destination was empty. Restoring over
|
|
an existing database is a different operation with different failure modes.
|
|
5. **`deploy.sh` was exercised up to and including the backup**, not through the build and container
|
|
replacement, which would have required a full deployment.
|
|
6. **The `mariadb-dump` client came from a container** (`mariadb:11`). If the production host has its
|
|
own client installed, `deploy.sh` uses that instead, and version differences are possible.
|
|
|
|
## What the owner still needs to do
|
|
|
|
This is the checklist that turns a rehearsal into a verified backup.
|
|
|
|
- [ ] **Run one backup by hand against production** and confirm the filename is
|
|
`jobtracker-<database>-<stamp>.sql.gz`.
|
|
- [ ] **Check the dump size is plausible** for the amount of data you have. A suspiciously small file
|
|
is the signal worth catching.
|
|
- [ ] **Restore it into a scratch database** — never over the live one — and confirm row counts for
|
|
`AspNetUsers`, `JobApplications`, `Companies` and `CareerProfiles` match production.
|
|
- [ ] **Check non-ASCII text survived.** Open one CV or career profile containing `æ`, `ø` or `å` in
|
|
the restored copy and confirm it is not mangled. This is the most likely silent failure.
|
|
- [ ] **Confirm `/opt/job-tracker/backups` has disk headroom** for several dumps.
|
|
- [ ] **Note how long the dump takes.** It runs before every deploy and blocks it.
|
|
|
|
## Cleanup
|
|
|
|
Both MariaDB containers and all rehearsal files were removed. No test artefacts remain, and the
|
|
existing local `jobtracker-backend-1` / `jobtracker-frontend-1` stack was left untouched.
|