jobtrackingapp/docs/deployment/backup-restore.md

Production backup & restore

2026-07-19. The backup mechanism is verified end to end against MariaDB 11 containers, including a byte-exact Norwegian-character round trip. No production database has been backed up from this environment — see Production access. This document is the checklist to run against production, plus the evidence for what is already proven.

Companion to deploy/deploy.sh (the implementation), deploy/README.md (backup location/retention), and docs/operations/production-backup-verification.md (the earlier container rehearsal).

Production backup checklist

Run top to bottom. Every item is checked automatically by deploy.sh before it builds or replaces anything — this list is for a manual pre-flight and for an out-of-band backup.

• DATABASE_PROVIDER=mariadb in /opt/job-tracker/shared/.env. No default — deploy.sh aborts if it is missing. If it were wrong, the SQLite path would run and back up the wrong thing.
• JOBTRACKER_CONNECTION_STRING present and pointing at the production database. The host resolves from the deploy shell, not from inside a container, for the backup step.
• deploy.sh loads the environment. It parses /opt/job-tracker/shared/.env into its own shell before deciding anything (verified: the "Loaded deployment environment" line prints first).
• Backup runs before replacement. validate_deploy_config → backup_database → build → up -d --force-recreate. Confirmed by line order in deploy.sh (validate/backup precede build/replace). A failed backup aborts the deploy with the running stack untouched.
• Backup validation works. The dump must be valid gzip, contain CREATE TABLE, and end with the -- Dump completed trailer; otherwise the file is deleted and the deploy stops. A SQLite archive must contain jobtracker.db.
• Filename confirms the type. jobtracker-<db>-<UTC>.sql.gz = MariaDB dump. A jobtracker-sqlite-*.tar.gz on a MariaDB host means the environment is wrong.
• Password never on the command line — deploy.sh passes it via MYSQL_PWD.

Restore procedure

Restore is a separate operation from code rollback. A bad deploy usually needs only the rollback; restore the database only if the data itself is wrong, because it discards everything written since the dump.

# 1. Restore into a SEPARATE, empty database first — never straight over the live one.
gzip -dc /opt/job-tracker/backups/jobtracker-<db>-<UTC>.sql.gz \
  | MYSQL_PWD='<password>' mariadb --host=<host> --user=<user> --default-character-set=utf8mb4 jobtracker_scratch

# 2. Sanity-check row counts and character fidelity (below), THEN, if replacing production:
docker compose stop backend
gzip -dc <backup> | MYSQL_PWD='<password>' mariadb --host=<host> --user=<user> --default-character-set=utf8mb4 jobtracker
docker compose start backend

Always pass --default-character-set=utf8mb4 on both dump and restore so multibyte text is not mangled.

UTF-8 / Norwegian character verification — VERIFIED 2026-07-19

The earlier rehearsal used ASCII-only seed data, so character fidelity was unproven. It has now been tested explicitly through the real deploy.sh backup path.

Seeded into a MariaDB 11 database on the real 42-table schema:

Field	Value
Company name	Ærøskøbing Systemutvikling AS
CV variant name	Søknad – Bjørn Håkonsen (note the em-dash –)
User email	bjørn@dåg.no
Career profile JSON	Erfaren utvikler frå Tromsø … Språk: norsk … flåten … Morsmål

Backed up with the real backup_database function, restored into a clean MariaDB 11 container, compared byte-for-byte:

source  HEX(Name): C38672C3B8736BC3B862696E672053797374656D757476696B6C696E67204153
restored HEX(Name): C38672C3B8736BC3B862696E672053797374656D757476696B6C696E67204153
BYTE-EXACT MATCH

C386 = Æ, C3B8 = ø, C3A5 = å — correct UTF-8, not double-encoded or stripped. æ ø å, plus the em-dash, survived the full dump → gzip → restore cycle unchanged.

To repeat this against production after a restore:

MYSQL_PWD='<pw>' mariadb --host=<host> --user=<user> --default-character-set=utf8mb4 -N -B jobtracker_scratch \
  -e "SELECT Name FROM Companies WHERE Name LIKE '%ø%' OR Name LIKE '%æ%' OR Name LIKE '%å%' LIMIT 5;"
# Read the output in a UTF-8 terminal. Mangled output (æ, ø) = charset problem in the pipeline.

Production access

A sanitized read-only host inventory was completed on 2026-08-15. It verified the integrity of the existing compressed database dumps without reading private rows or restoring data. The newest observed dump was dated 2026-08-02, and the set is database-only: it does not prove recovery of owned files, data-protection keys, configuration, or account-deletion tombstones.

Authorized production steps still required:

1. On the production host, run one out-of-band backup: deploy/deploy.sh takes one automatically, or dump by hand with the command in deploy/README.md.
2. Restore that dump into a scratch database (not production) and confirm:
   • table count (~42) and row counts for AspNetUsers, JobApplications, Companies, CareerProfiles match production;
   • a real record containing æ/ø/å reads back correctly (the check above).
3. Build and restore a complete recovery bundle covering the jobtracker_data volume and deployment data-protection keys as well as MariaDB. Preserve the separately mounted jobtracker_deletion_tombstones volume across application-data restores; never overwrite it with an older backup capable of resurrecting a deleted identity.

Until that is done, backup/restore is proven on the mechanism and on synthetic Norwegian data, not on the production dataset.