jobtrackingapp/docs/deployment/backup-restore.md

Production backup & restore

2026-07-19. The backup mechanism is verified end to end against MariaDB 11 containers, including a byte-exact Norwegian-character round trip. No production database has been backed up from this environment — see Production access. This document is the checklist to run against production, plus the evidence for what is already proven.

Companion to deploy/deploy.sh (the implementation), deploy/README.md (backup location/retention), and docs/operations/production-backup-verification.md (the earlier container rehearsal).

Production backup checklist

Run top to bottom. Every item is checked automatically by deploy.sh before it builds or replaces anything — this list is for a manual pre-flight and for an out-of-band backup.

• DATABASE_PROVIDER=mariadb in /opt/job-tracker/shared/.env. No default — deploy.sh aborts if it is missing. If it were wrong, the SQLite path would run and back up the wrong thing.
• JOBTRACKER_CONNECTION_STRING present and pointing at the production database. The host resolves from the deploy shell, not from inside a container, for the backup step.
• deploy.sh loads the environment. It parses /opt/job-tracker/shared/.env into its own shell before deciding anything (verified: the "Loaded deployment environment" line prints first).
• Backup runs before replacement. validate_deploy_config → backup_database → build → up -d --force-recreate. Confirmed by line order in deploy.sh (validate/backup precede build/replace). A failed backup aborts the deploy with the running stack untouched.
• Backup validation works. The dump must be valid gzip, contain CREATE TABLE, and end with the -- Dump completed trailer; otherwise the file is deleted and the deploy stops. A SQLite archive must contain jobtracker.db.
• Filename confirms the type. jobtracker-<db>-<UTC>.sql.gz = MariaDB dump. A jobtracker-sqlite-*.tar.gz on a MariaDB host means the environment is wrong.
• Password never on the command line — deploy.sh passes it via MYSQL_PWD.

Restore procedure

Restore is a separate operation from code rollback. A bad deploy usually needs only the rollback; restore the database only if the data itself is wrong, because it discards everything written since the dump.

# 1. Restore into a SEPARATE, empty database first — never straight over the live one.
gzip -dc /opt/job-tracker/backups/jobtracker-<db>-<UTC>.sql.gz \
  | MYSQL_PWD='<password>' mariadb --host=<host> --user=<user> --default-character-set=utf8mb4 jobtracker_scratch

# 2. Sanity-check row counts and character fidelity (below), THEN, if replacing production:
docker compose stop backend
gzip -dc <backup> | MYSQL_PWD='<password>' mariadb --host=<host> --user=<user> --default-character-set=utf8mb4 jobtracker
docker compose start backend

Always pass --default-character-set=utf8mb4 on both dump and restore so multibyte text is not mangled.

UTF-8 / Norwegian character verification — VERIFIED 2026-07-19

The earlier rehearsal used ASCII-only seed data, so character fidelity was unproven. It has now been tested explicitly through the real deploy.sh backup path.

Seeded into a MariaDB 11 database on the real 42-table schema:

Field	Value
Company name	Ærøskøbing Systemutvikling AS
CV variant name	Søknad – Bjørn Håkonsen (note the em-dash –)
User email	bjørn@dåg.no
Career profile JSON	Erfaren utvikler frå Tromsø … Språk: norsk … flåten … Morsmål

Backed up with the real backup_database function, restored into a clean MariaDB 11 container, compared byte-for-byte:

source  HEX(Name): C38672C3B8736BC3B862696E672053797374656D757476696B6C696E67204153
restored HEX(Name): C38672C3B8736BC3B862696E672053797374656D757476696B6C696E67204153
BYTE-EXACT MATCH

C386 = Æ, C3B8 = ø, C3A5 = å — correct UTF-8, not double-encoded or stripped. æ ø å, plus the em-dash, survived the full dump → gzip → restore cycle unchanged.

To repeat this against production after a restore:

MYSQL_PWD='<pw>' mariadb --host=<host> --user=<user> --default-character-set=utf8mb4 -N -B jobtracker_scratch \
  -e "SELECT Name FROM Companies WHERE Name LIKE '%ø%' OR Name LIKE '%æ%' OR Name LIKE '%å%' LIMIT 5;"
# Read the output in a UTF-8 terminal. Mangled output (æ, ø) = charset problem in the pipeline.

Production access

This environment has no route to the production database — no /opt/job-tracker, no production connection string, and the local stack runs SQLite. Per the task constraints, no credential discovery and no SSH guessing were attempted.

Manual step the owner must perform (only the owner has production access):

1. On the production host, run one out-of-band backup: deploy/deploy.sh takes one automatically, or dump by hand with the command in deploy/README.md.
2. Restore that dump into a scratch database (not production) and confirm:
   • table count (~42) and row counts for AspNetUsers, JobApplications, Companies, CareerProfiles match production;
   • a real record containing æ/ø/å reads back correctly (the check above).

Until that is done, backup/restore is proven on the mechanism and on synthetic Norwegian data, not on the production dataset.