cesnimda/jobtrackingapp
1
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity

chore(ops): add deployment backups restore docs and health checks
CI and Deploy / test (push) Failing after 1m8s
Details
CI and Deploy / deploy (push) Has been skipped
Details

Browse Source 
Closes the three operational blockers from the production readiness review.

deploy.sh now takes a database backup before it builds, stops or replaces
anything, and aborts the deploy if the backup fails — so no deploy proceeds
without a restore point. Dumps are gzipped and timestamped into
/opt/job-tracker/backups (override with BACKUP_DIR), so one deploy never
overwrites an earlier backup. Credentials come from the existing connection
string and travel via MYSQL_PWD, never on the command line, so they cannot reach
the process list or the deploy log. A dump that is empty or missing CREATE TABLE
is rejected, because a truncated file that looks like a restore point is worse
than none. SQLite deployments get their data volume tarred instead. Nothing is
ever deleted automatically; retention is documented as manual.

deploy/README.md documents backup creation, location, retention, database
restore, application rollback, and when to use which — restore and rollback kept
distinct, because a bad deploy usually needs only the rollback and restoring
would discard everything written since the dump.

Health checks now cover backend and frontend, which previously had none. GET
/health is anonymous, cheap, and deliberately does not touch the database: a
health check that queried MariaDB would restart a healthy backend whenever the
database blipped, and would hand out an unauthenticated way to probe database
availability. The backend image gains curl on the existing chromium apt layer,
since the aspnet runtime ships neither curl nor wget. frontend now waits for
backend to be healthy rather than merely started, because nginx proxies /api and
refuses to start when the upstream cannot be resolved.

Verified against real containers, no production data: backup from a seeded
MariaDB 11; restore into a clean MariaDB 11 with rows identical; bad credentials
and a missing connection string both abort non-zero and leave no partial file;
SQLite volume backup produces a readable archive; backend and frontend both
reach healthy; and a backend pointed at an unreachable database exits and is
reported unhealthy, so a broken deploy cannot present as a running stack.

Incidentally confirmed the earlier authorization work: with Auth:Require unset,
/health returns 200 while /api/jobapplications returns 401.

393 backend tests pass.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
cesnimda
2026-07-19 17:49:31 +02:00
parent b2b87f39a5
commit 93462b799c
6 changed files with 318 additions and 20 deletions
Download Patch File Download Diff File Expand all files Collapse all files
deploy/README.md
+119
View File
@@ -100,3 +100,122 @@ If this app is going to be a real production service on Ubuntu:
- confirm reminder and admin/system pages load
- verify follow-up reminder emails are enabled only when intended and that links open the correct job/tab
hat links open the correct job/tab
---
# Backups, restore and rollback
**Database restore and application rollback are two different operations.** A bad deploy usually needs
only the rollback. Restore the database only if the data itself is wrong or lost — it discards
everything written since the dump.
## Backup creation
`deploy/deploy.sh` takes a backup **before** it builds, stops or replaces anything, and **aborts the
deploy if the backup fails**. Nothing else in the deploy runs without a restore point.
- **Location:** `/opt/job-tracker/backups` — override with `BACKUP_DIR`.
- **Naming:** `jobtracker-<database>-<UTC timestamp>.sql.gz`, e.g.
`jobtracker-jobtracker-20260719T153759Z.sql.gz`. The timestamp makes every file unique, so a deploy
never overwrites an earlier backup.
- **SQLite deployments** (`DATABASE_PROVIDER` unset or `sqlite`) get the data volume instead:
`jobtracker-sqlite-<UTC timestamp>.tar.gz`.
- **Credentials** come from `JOBTRACKER_CONNECTION_STRING` and are passed via `MYSQL_PWD`, never on the
command line, so they cannot appear in the process list or the deploy log.
- **Compression:** gzip. A small database compresses to a few KB.
- **Verification:** the script rejects a dump that is empty or missing `CREATE TABLE`, because a
truncated file that *looks* like a restore point is worse than none.
### Taking one by hand
```bash
BACKUP_DIR=/opt/job-tracker/backups
mkdir -p "$BACKUP_DIR"
MYSQL_PWD='<password>' mariadb-dump \
--host=127.0.0.1 --port=3306 --user=<user> \
--single-transaction --routines --events --quick \
jobtracker | gzip -c > "$BACKUP_DIR/jobtracker-manual-$(date -u +%Y%m%dT%H%M%SZ).sql.gz"
```
### Retention
**Nothing is deleted automatically.** Backups accumulate in `BACKUP_DIR` until you remove them. Watch
disk usage and prune deliberately — a suggested policy is to keep every backup for 30 days and one per
month after that, but the script does not enforce it and will not delete your files.
## Restoring the database
Tested end-to-end against a clean MariaDB 11 container: dump taken from a seeded database, restored
into an empty one, rows verified identical.
```bash
# 1. Stop the application so nothing writes during the restore.
docker compose stop backend
# 2. Restore. This REPLACES the current contents of the named database.
gzip -dc /opt/job-tracker/backups/jobtracker-jobtracker-20260719T153759Z.sql.gz \
| MYSQL_PWD='<password>' mariadb --host=127.0.0.1 --port=3306 --user=<user> jobtracker
# 3. Verify before starting anything.
MYSQL_PWD='<password>' mariadb --host=127.0.0.1 --port=3306 --user=<user> jobtracker \
-e "SELECT COUNT(*) AS applications FROM JobApplications;"
# 4. Start again.
docker compose start backend
```
Restoring a **SQLite** deployment instead:
```bash
docker compose stop backend
docker run --rm -v jobtracker_data:/data -v /opt/job-tracker/backups:/backup \
-e ARCHIVE_NAME=jobtracker-sqlite-20260719T153941Z.tar.gz \
alpine:3 sh -c 'rm -rf /data/* && tar xzf "/backup/$ARCHIVE_NAME" -C /data'
docker compose start backend
```
## Restoring application containers (rollback)
This is the usual fix for a bad deploy, and it **does not touch the database**.
```bash
cd /opt/job-tracker/app # the deployment checkout
git log --oneline -5 # find the last good commit
git checkout <previous-commit>
deploy/deploy.sh
```
`deploy.sh` takes a fresh backup first, so rolling back is itself protected.
**Why a code rollback is safe here:** every Phase 4/5 migration is a no-op — the startup reconciler
owns those tables — so reverting the code never leaves migration history ahead of the schema. The
reconciler is additive and never drops a table holding rows, so the older code simply ignores the
newer tables.
**What a code rollback does not undo:** rows users created in the newer tables stay. That is usually
what you want. If you additionally restore the database, those rows are lost — so restore only when
the data is the problem.
## Choosing between them
| Symptom | Action |
|---|---|
| New version starts but behaves wrong | Rollback the code. Leave the database. |
| Backend will not start; schema looks wrong | Rollback the code, then restore only if it still fails. |
| Data is missing or corrupted | Restore the database from the most recent good dump. |
| Deploy aborted before starting | Nothing to undo — the backup ran before any change. |
## Health checks
`backend` and `frontend` both report container health, so `docker compose ps` shows real state rather
than merely "running".
- **Backend:** `curl -fsS http://127.0.0.1:8080/health`. Anonymous, and deliberately **does not touch
the database** — a health check that queried MariaDB would restart a healthy backend whenever the
database blipped. `start_period` is 90s to cover first-boot schema reconciliation.
- **Frontend:** `wget` against nginx on port 80.
- `frontend` waits for `backend` to be *healthy*, not merely started, because nginx proxies `/api` to
it and refuses to start if the upstream cannot be resolved.
A backend that cannot reach its database exits and is reported `unhealthy`, so a broken deploy does not
present as a running stack.
deploy/deploy.sh
+125
View File
@@ -28,6 +28,131 @@ compose() {
docker compose "$@"
}
# ---------------------------------------------------------------------------
# Database backup, taken BEFORE anything is stopped, built or replaced.
#
# A failed backup fails the deploy. The startup reconciler is additive and never
# drops a table holding rows, but "additive" is a property of the code, not a
# guarantee about the disk — and the first deploy after a schema change is
# exactly when a restore point matters. See deploy/README.md.
# ---------------------------------------------------------------------------
BACKUP_DIR="${BACKUP_DIR:-/opt/job-tracker/backups}"
backup_database() {
if [ "${DEPLOY_SKIP_DB_BACKUP:-false}" = "true" ]; then
echo "WARNING: DEPLOY_SKIP_DB_BACKUP=true — deploying with no restore point. Emergency use only."
return 0
fi
local provider="${DATABASE_PROVIDER:-sqlite}"
local stamp
stamp="$(date -u +%Y%m%dT%H%M%SZ)"
mkdir -p "$BACKUP_DIR"
if [ "$provider" != "mysql" ] && [ "$provider" != "mariadb" ]; then
# SQLite lives in the jobtracker_data volume. Tar it from a throwaway container
# so the host needs no sqlite tooling and no knowledge of the volume layout.
local target="$BACKUP_DIR/jobtracker-sqlite-${stamp}.tar.gz"
echo "Backing up SQLite data volume to ${target}"
# The archive path is built inside the container: passing /backup/... as an argument
# gets rewritten by MSYS path translation when the script is run from Git Bash.
if ! docker run --rm \
-v jobtracker_data:/data:ro \
-v "$BACKUP_DIR":/backup \
-e ARCHIVE_NAME="$(basename "$target")" \
alpine:3 sh -c 'tar czf "/backup/$ARCHIVE_NAME" -C /data .'; then
echo "SQLite volume backup FAILED. Aborting deploy."
return 1
fi
verify_backup "$target"
return $?
fi
local cs="${JOBTRACKER_CONNECTION_STRING:-}"
if [ -z "$cs" ]; then
echo "DATABASE_PROVIDER=${provider} but JOBTRACKER_CONNECTION_STRING is empty. Aborting deploy."
return 1
fi
# Parse the ADO.NET connection string. Keys are case-insensitive in .NET, so match that.
local db_host db_port db_name db_user db_pass
db_host="$(sed -n 's/.*[Ss]erver=\([^;]*\).*/\1/p' <<<"$cs")"
db_port="$(sed -n 's/.*[Pp]ort=\([^;]*\).*/\1/p' <<<"$cs")"
db_name="$(sed -n 's/.*[Dd]atabase=\([^;]*\).*/\1/p' <<<"$cs")"
db_user="$(sed -n 's/.*[Uu]ser[ ]*[Ii]*[Dd]*=\([^;]*\).*/\1/p' <<<"$cs")"
db_pass="$(sed -n 's/.*[Pp]assword=\([^;]*\).*/\1/p' <<<"$cs")"
db_port="${db_port:-3306}"
if [ -z "$db_host" ] || [ -z "$db_name" ] || [ -z "$db_user" ]; then
echo "Could not parse host/database/user from JOBTRACKER_CONNECTION_STRING. Aborting deploy."
return 1
fi
local target="$BACKUP_DIR/jobtracker-${db_name}-${stamp}.sql.gz"
echo "Backing up MariaDB database '${db_name}' on ${db_host}:${db_port} to ${target}"
# The password goes via MYSQL_PWD, never on the command line, so it cannot leak
# into the process list or the deploy log.
local dump_status=0
if command -v mariadb-dump >/dev/null 2>&1; then
MYSQL_PWD="$db_pass" mariadb-dump \
--host="$db_host" --port="$db_port" --user="$db_user" \
--single-transaction --routines --events --quick \
"$db_name" 2>/tmp/jobtracker-backup.err | gzip -c > "$target" || dump_status=$?
elif command -v mysqldump >/dev/null 2>&1; then
MYSQL_PWD="$db_pass" mysqldump \
--host="$db_host" --port="$db_port" --user="$db_user" \
--single-transaction --routines --events --quick \
"$db_name" 2>/tmp/jobtracker-backup.err | gzip -c > "$target" || dump_status=$?
else
# No client on the host: run one in a container. --network host so the same
# host/port from the connection string resolves identically.
docker run --rm --network host -e MYSQL_PWD="$db_pass" mariadb:11 \
mariadb-dump --host="$db_host" --port="$db_port" --user="$db_user" \
--single-transaction --routines --events --quick \
"$db_name" 2>/tmp/jobtracker-backup.err | gzip -c > "$target" || dump_status=$?
fi
if [ "$dump_status" -ne 0 ]; then
echo "Database dump FAILED (exit ${dump_status}). Aborting deploy."
sed -e 's/password=[^ ]*/password=***/gI' /tmp/jobtracker-backup.err >&2 || true
rm -f "$target"
return 1
fi
verify_backup "$target" "CREATE TABLE"
}
# A dump that exists but is empty or truncated is worse than none, because it
# looks like a restore point. Check size, and content when we know what to expect.
verify_backup() {
local target="$1"
local expect="${2:-}"
if [ ! -s "$target" ]; then
echo "Backup file ${target} is missing or empty. Aborting deploy."
rm -f "$target"
return 1
fi
local size
size="$(du -h "$target" | cut -f1)"
if [ -n "$expect" ] && ! gzip -dc "$target" | grep -q "$expect"; then
echo "Backup ${target} does not contain '${expect}' — it is not a usable dump. Aborting deploy."
rm -f "$target"
return 1
fi
echo "Backup verified: ${target} (${size})"
echo "Retention is manual — old backups in ${BACKUP_DIR} are never deleted automatically."
return 0
}
if ! backup_database; then
exit 1
fi
build_core_with_recovery() {
if compose build backend frontend; then
return 0