jobtrackingapp/docs/deployment/backup-restore.md

# Production backup & restore

> 2026-07-19. The backup **mechanism** is verified end to end against MariaDB 11 containers, including a
> byte-exact Norwegian-character round trip. **No production database has been backed up from this
> environment** — see *Production access*. This document is the checklist to run against production,
> plus the evidence for what is already proven.

Companion to `deploy/deploy.sh` (the implementation), `deploy/README.md` (backup location/retention),
and `docs/operations/production-backup-verification.md` (the earlier container rehearsal).

## Production backup checklist

Run top to bottom. Every item is checked automatically by `deploy.sh` **before** it builds or replaces
anything — this list is for a manual pre-flight and for an out-of-band backup.

- [ ] **`DATABASE_PROVIDER=mariadb`** in `/opt/job-tracker/shared/.env`. No default — `deploy.sh`
      aborts if it is missing. If it were wrong, the SQLite path would run and back up the wrong thing.
- [ ] **`JOBTRACKER_CONNECTION_STRING` present** and pointing at the production database. The host
      resolves from the deploy shell, not from inside a container, for the backup step.
- [ ] **`deploy.sh` loads the environment.** It parses `/opt/job-tracker/shared/.env` into its own
      shell before deciding anything (verified: the "Loaded deployment environment" line prints first).
- [ ] **Backup runs before replacement.** `validate_deploy_config` → `backup_database` → build →
      `up -d --force-recreate`. Confirmed by line order in `deploy.sh` (validate/backup precede
      build/replace). A failed backup aborts the deploy with the running stack untouched.
- [ ] **Backup validation works.** The dump must be valid gzip, contain `CREATE TABLE`, and end with
      the `-- Dump completed` trailer; otherwise the file is deleted and the deploy stops. A SQLite
      archive must contain `jobtracker.db`.
- [ ] **Filename confirms the type.** `jobtracker-<db>-<UTC>.sql.gz` = MariaDB dump.
      A `jobtracker-sqlite-*.tar.gz` on a MariaDB host means the environment is wrong.
- [ ] **Password never on the command line** — `deploy.sh` passes it via `MYSQL_PWD`.

## Restore procedure

Restore is a **separate** operation from code rollback. A bad deploy usually needs only the rollback;
restore the database **only if the data itself is wrong**, because it discards everything written since
the dump.

```bash
# 1. Restore into a SEPARATE, empty database first — never straight over the live one.
gzip -dc /opt/job-tracker/backups/jobtracker-<db>-<UTC>.sql.gz \
  | MYSQL_PWD='<password>' mariadb --host=<host> --user=<user> --default-character-set=utf8mb4 jobtracker_scratch

# 2. Sanity-check row counts and character fidelity (below), THEN, if replacing production:
docker compose stop backend
gzip -dc <backup> | MYSQL_PWD='<password>' mariadb --host=<host> --user=<user> --default-character-set=utf8mb4 jobtracker
docker compose start backend
```

Always pass `--default-character-set=utf8mb4` on both dump and restore so multibyte text is not
mangled.

## UTF-8 / Norwegian character verification — **VERIFIED 2026-07-19**

The earlier rehearsal used ASCII-only seed data, so character fidelity was unproven. It has now been
tested explicitly through the real `deploy.sh` backup path.

Seeded into a MariaDB 11 database on the real 42-table schema:

| Field | Value |
|---|---|
| Company name | `Ærøskøbing Systemutvikling AS` |
| CV variant name | `Søknad – Bjørn Håkonsen` (note the em-dash `–`) |
| User email | `bjørn@dåg.no` |
| Career profile JSON | `Erfaren utvikler frå Tromsø … Språk: norsk … flåten … Morsmål` |

Backed up with the real `backup_database` function, restored into a **clean** MariaDB 11 container,
compared byte-for-byte:

```
source  HEX(Name): C38672C3B8736BC3B862696E672053797374656D757476696B6C696E67204153
restored HEX(Name): C38672C3B8736BC3B862696E672053797374656D757476696B6C696E67204153
BYTE-EXACT MATCH
```

`C386` = `Æ`, `C3B8` = `ø`, `C3A5` = `å` — correct UTF-8, not double-encoded or stripped. **æ ø å**,
plus the em-dash, survived the full dump → gzip → restore cycle unchanged.

To repeat this against production after a restore:

```bash
MYSQL_PWD='<pw>' mariadb --host=<host> --user=<user> --default-character-set=utf8mb4 -N -B jobtracker_scratch \
  -e "SELECT Name FROM Companies WHERE Name LIKE '%ø%' OR Name LIKE '%æ%' OR Name LIKE '%å%' LIMIT 5;"
# Read the output in a UTF-8 terminal. Mangled output (æ, ø) = charset problem in the pipeline.
```

## Production access

A sanitized read-only host inventory was completed on 2026-08-15. It verified the integrity of the
existing compressed database dumps without reading private rows or restoring data. The newest
observed dump was dated 2026-08-02, and the set is database-only: it does not prove recovery of owned
files, data-protection keys, configuration, or account-deletion tombstones.

**Authorized production steps still required:**

1. On the production host, run one out-of-band backup: `deploy/deploy.sh` takes one automatically, or
   dump by hand with the command in `deploy/README.md`.
2. Restore that dump into a **scratch** database (not production) and confirm:
   - table count (~42) and row counts for `AspNetUsers`, `JobApplications`, `Companies`,
     `CareerProfiles` match production;
   - a real record containing `æ`/`ø`/`å` reads back correctly (the check above).
3. Build and restore a complete recovery bundle covering the `jobtracker_data` volume and deployment
   data-protection keys as well as MariaDB. Preserve the separately mounted
   `jobtracker_deletion_tombstones` volume across application-data restores; never overwrite it with
   an older backup capable of resurrecting a deleted identity.

Until that is done, backup/restore is proven **on the mechanism and on synthetic Norwegian data**, not
on the production dataset.