jobtrackingapp/docs/operations/production-backup-verification.md

Backup and restore verification

2026-07-19 19:32 UTC. Full backup → verify → restore → start-the-app rehearsal of the deploy/deploy.sh backup path, exercised end to end against MariaDB 11.

This is a rehearsal, not a verification of production data. See Limitations — that section is the most important part of this document, and the checklist at the end is what actually closes the gap.

Scope — read this first

What was verified	The backup mechanism: the real backup_database function from deploy.sh, against a real MariaDB 11 database carrying the real 42-table schema, restored into a separate clean MariaDB 11 container, with the real application started against the result
What was NOT verified	Your production database. No production host was contacted, no production credentials were used, and no production data was read, copied or restored
Why	The machine this ran on has no route to production: no /opt/job-tracker, no DATABASE_PROVIDER or JOBTRACKER_CONNECTION_STRING in its .env, and the local stack runs SQLite. The production host, user and key are CI secrets (PROD_HOST, PROD_USER, PROD_SSH_KEY) that are not available here

The commands below are the ones to run against production. They are recorded so the owner can execute the same sequence with production values substituted.

2026-08-15 read-only production checkpoint

The earlier access statement above remains historically accurate for the 2026-07-19 rehearsal, but access is now available. A strictly read-only production inventory found:

• 21 MariaDB .sql.gz dumps (about 5.2 MB combined), all passing gzip -t;
• oldest observed dump: 2026-07-19 21:41 UTC; newest: 2026-08-02 17:32 UTC;
• no JobTracker systemd timer or current-user cron entry;
• 1.4 TiB free on the Docker/data filesystem and 36 GiB free on the 83%-used root filesystem;
• database-only dump files in the backup directory: no owner-file volume, data-protection-key, protected tombstone, or configuration recovery bundle;
• no restore, row/content read, count comparison, non-ASCII check, or production mutation.

This upgrades production dump presence and compressed-stream integrity from unknown to observed, but it does not close the restore/RPO/completeness checklist. The newest observed dump was 13 days old at capture. See docs/production/production-ai-hardware-assessment.md and docs/production/production-ai-rollout-and-rollback.md.

1. Backup configuration

Read from deploy/deploy.sh and docker-compose.yml:

• deploy.sh loads /opt/job-tracker/shared/.env into its own shell before any decision.
• validate_deploy_config runs before the backup, and therefore before any build, stop or replace. It requires DATABASE_PROVIDER (no default), JOBTRACKER_CONNECTION_STRING when the provider is MariaDB, plus AI_SERVICE_TOKEN and AUTH_JWT_KEY.
• Backups land in /opt/job-tracker/backups (override with BACKUP_DIR), UTC-timestamped, gzipped. Nothing is ever overwritten and nothing is ever auto-deleted.
• The password travels via MYSQL_PWD, never on the command line.

2. Provider resolution

DATABASE_PROVIDER=mariadb selected the MariaDB dump path. Confirmed by the script's own output:

Deployment configuration validated (database provider: mariadb).
Backing up MariaDB database 'jobtracker' on <host>:<port> to <dir>/jobtracker-jobtracker-20260719T193022Z.sql.gz
Backup verified: <dir>/jobtracker-jobtracker-20260719T193022Z.sql.gz (8.0K)

The filename is the check that matters. jobtracker-<database>-<stamp>.sql.gz means the MariaDB path ran. A jobtracker-sqlite-<stamp>.tar.gz on a MariaDB host would mean the environment is wrong.

3. Commands used

Schema built by the real application, not by hand — Database.Migrate() plus the startup reconciler against an empty MariaDB, producing 42 tables. Representative rows were then seeded across users, companies, applications, career profile and CV variants.

# Backup — the real function from deploy.sh, not a hand-written dump
DATABASE_PROVIDER=mariadb \
JOBTRACKER_CONNECTION_STRING='Server=<host>;Port=<port>;Database=jobtracker;User Id=<user>;Password=<password>;' \
BACKUP_DIR=/opt/job-tracker/backups \
  deploy/deploy.sh          # takes the backup first and aborts the deploy if it fails

# Integrity and content
gzip -t  "$BACKUP"                                  # valid archive
gzip -dc "$BACKUP" | grep -c 'CREATE TABLE'         # schema present
gzip -dc "$BACKUP" | tail -1                        # "-- Dump completed on ..." trailer

# Restore into a SEPARATE, empty database — never over a live one
gzip -dc "$BACKUP" | MYSQL_PWD='<password>' mariadb --host=<host> --user=<user> jobtracker

Passwords are supplied via MYSQL_PWD so they never reach the process list or the shell history.

4. Backup verification result

Check	Result
Correct backup type	✅ .sql.gz MariaDB dump, not a volume archive
File integrity	✅ gzip -t passed
Schema markers	✅ 42 CREATE TABLE statements
Dump trailer	✅ -- Dump completed on 2026-07-19 19:30:23 — proves the dump was not truncated
Expected tables	✅ AspNetUsers, Companies, JobApplications, CareerProfiles, CvVariants, JobEvents, ApplicationChecklistItems, AiInteractions all present
Data present	✅ 7 INSERT INTO statements

5. Restore result

Restored into a separate, empty MariaDB 11 container (0 tables before, 42 after).

Entity	Source	Restored
Users	1	1	✅
Companies	2	2	✅
Applications	2	2	✅
Career profiles	1	1	✅
CV variants	1	1	✅

Beyond counts:

• All 42 tables compared — table lists identical, and every table's row count matched.
• Content survived, not just cardinality — the restored applications resolve their company foreign keys (Senior Backend Engineer @ Northwind Consulting), the CV variant kept its name and IsPublic flag, and the career profile JSON still contained its languages.
• The application starts against the restored database. /health returned {"status":"ok","version":"restore-rehearsal"}, the schema stayed at 42 tables (the reconciler correctly found nothing to do) and rows were preserved. A restore that produces a database the app cannot boot against is not a restore.

Limitations

Read these before treating the deployment as backed up.

1. No production data was touched. Everything above ran against a locally built MariaDB with seeded rows. It proves the mechanism; it proves nothing about your database.
2. The seeded dataset is tiny (7 rows). It does not exercise dump duration, disk headroom, lock behaviour under load, or timeout limits on a real dataset. A production database large enough to make mariadb-dump slow could behave differently.
3. Character-set and collation fidelity was not stress-tested. The seeded data was mostly ASCII. Real CV content contains non-ASCII text — Norwegian æøå, accents, CJK. The dump defaults should handle this, but it was not proven here.
4. No restore was performed over a populated database. The destination was empty. Restoring over an existing database is a different operation with different failure modes.
5. deploy.sh was exercised up to and including the backup, not through the build and container replacement, which would have required a full deployment.
6. The mariadb-dump client came from a container (mariadb:11). If the production host has its own client installed, deploy.sh uses that instead, and version differences are possible.

What the owner still needs to do

This is the checklist that turns a rehearsal into a verified backup.

• Run one backup by hand against production and confirm the filename is jobtracker-<database>-<stamp>.sql.gz.
• Check the dump size is plausible for the amount of data you have. A suspiciously small file is the signal worth catching.
• Restore it into a scratch database — never over the live one — and confirm row counts for AspNetUsers, JobApplications, Companies and CareerProfiles match production.
• Check non-ASCII text survived. Open one CV or career profile containing æ, ø or å in the restored copy and confirm it is not mangled. This is the most likely silent failure.
• Confirm /opt/job-tracker/backups has disk headroom for several dumps.
• Note how long the dump takes. It runs before every deploy and blocks it.

Cleanup

Both MariaDB containers and all rehearsal files were removed. No test artefacts remain, and the existing local jobtracker-backend-1 / jobtracker-frontend-1 stack was left untouched.