diff --git a/JobTrackerApi/Dockerfile b/JobTrackerApi/Dockerfile index 8597334..f9962d5 100644 --- a/JobTrackerApi/Dockerfile +++ b/JobTrackerApi/Dockerfile @@ -25,8 +25,10 @@ WORKDIR /app ENV ASPNETCORE_URLS=http://+:8080 ENV CV_PDF_BROWSER_PATH=/usr/bin/chromium +# curl is here for the container health check (/health). The aspnet runtime image ships +# neither curl nor wget, and this layer already exists for chromium. RUN apt-get update \ - && apt-get install -y --no-install-recommends chromium \ + && apt-get install -y --no-install-recommends chromium curl \ && rm -rf /var/lib/apt/lists/* RUN mkdir -p /data diff --git a/JobTrackerApi/Program.cs b/JobTrackerApi/Program.cs index 74d4c6b..dc82a65 100644 --- a/JobTrackerApi/Program.cs +++ b/JobTrackerApi/Program.cs @@ -520,6 +520,17 @@ app.UseAuthentication(); app.UseAuthorization(); app.MapControllers(); +// Liveness probe for the container health check. Deliberately cheap and anonymous: it answers "is +// this process up and serving?" and nothing else. It does NOT touch the database — a health check +// that queried MariaDB would restart a perfectly healthy backend whenever the database blipped, and +// would also be a free unauthenticated way to probe database availability. +// docs/production-readiness-review.md. +app.MapGet("/health", () => Results.Ok(new +{ + status = "ok", + version = Environment.GetEnvironmentVariable("APP_VERSION") ?? "unknown", +})).AllowAnonymous(); + // API schema for tooling/docs. Dev-only: not exposed in production deployments. if (app.Environment.IsDevelopment()) { diff --git a/deploy/README.md b/deploy/README.md index f4ac278..d67bdd9 100644 --- a/deploy/README.md +++ b/deploy/README.md @@ -100,3 +100,122 @@ If this app is going to be a real production service on Ubuntu: - confirm reminder and admin/system pages load - verify follow-up reminder emails are enabled only when intended and that links open the correct job/tab hat links open the correct job/tab + +--- + +# Backups, restore and rollback + +**Database restore and application rollback are two different operations.** A bad deploy usually needs +only the rollback. Restore the database only if the data itself is wrong or lost — it discards +everything written since the dump. + +## Backup creation + +`deploy/deploy.sh` takes a backup **before** it builds, stops or replaces anything, and **aborts the +deploy if the backup fails**. Nothing else in the deploy runs without a restore point. + +- **Location:** `/opt/job-tracker/backups` — override with `BACKUP_DIR`. +- **Naming:** `jobtracker--.sql.gz`, e.g. + `jobtracker-jobtracker-20260719T153759Z.sql.gz`. The timestamp makes every file unique, so a deploy + never overwrites an earlier backup. +- **SQLite deployments** (`DATABASE_PROVIDER` unset or `sqlite`) get the data volume instead: + `jobtracker-sqlite-.tar.gz`. +- **Credentials** come from `JOBTRACKER_CONNECTION_STRING` and are passed via `MYSQL_PWD`, never on the + command line, so they cannot appear in the process list or the deploy log. +- **Compression:** gzip. A small database compresses to a few KB. +- **Verification:** the script rejects a dump that is empty or missing `CREATE TABLE`, because a + truncated file that *looks* like a restore point is worse than none. + +### Taking one by hand + +```bash +BACKUP_DIR=/opt/job-tracker/backups +mkdir -p "$BACKUP_DIR" +MYSQL_PWD='' mariadb-dump \ + --host=127.0.0.1 --port=3306 --user= \ + --single-transaction --routines --events --quick \ + jobtracker | gzip -c > "$BACKUP_DIR/jobtracker-manual-$(date -u +%Y%m%dT%H%M%SZ).sql.gz" +``` + +### Retention + +**Nothing is deleted automatically.** Backups accumulate in `BACKUP_DIR` until you remove them. Watch +disk usage and prune deliberately — a suggested policy is to keep every backup for 30 days and one per +month after that, but the script does not enforce it and will not delete your files. + +## Restoring the database + +Tested end-to-end against a clean MariaDB 11 container: dump taken from a seeded database, restored +into an empty one, rows verified identical. + +```bash +# 1. Stop the application so nothing writes during the restore. +docker compose stop backend + +# 2. Restore. This REPLACES the current contents of the named database. +gzip -dc /opt/job-tracker/backups/jobtracker-jobtracker-20260719T153759Z.sql.gz \ + | MYSQL_PWD='' mariadb --host=127.0.0.1 --port=3306 --user= jobtracker + +# 3. Verify before starting anything. +MYSQL_PWD='' mariadb --host=127.0.0.1 --port=3306 --user= jobtracker \ + -e "SELECT COUNT(*) AS applications FROM JobApplications;" + +# 4. Start again. +docker compose start backend +``` + +Restoring a **SQLite** deployment instead: + +```bash +docker compose stop backend +docker run --rm -v jobtracker_data:/data -v /opt/job-tracker/backups:/backup \ + -e ARCHIVE_NAME=jobtracker-sqlite-20260719T153941Z.tar.gz \ + alpine:3 sh -c 'rm -rf /data/* && tar xzf "/backup/$ARCHIVE_NAME" -C /data' +docker compose start backend +``` + +## Restoring application containers (rollback) + +This is the usual fix for a bad deploy, and it **does not touch the database**. + +```bash +cd /opt/job-tracker/app # the deployment checkout +git log --oneline -5 # find the last good commit +git checkout +deploy/deploy.sh +``` + +`deploy.sh` takes a fresh backup first, so rolling back is itself protected. + +**Why a code rollback is safe here:** every Phase 4/5 migration is a no-op — the startup reconciler +owns those tables — so reverting the code never leaves migration history ahead of the schema. The +reconciler is additive and never drops a table holding rows, so the older code simply ignores the +newer tables. + +**What a code rollback does not undo:** rows users created in the newer tables stay. That is usually +what you want. If you additionally restore the database, those rows are lost — so restore only when +the data is the problem. + +## Choosing between them + +| Symptom | Action | +|---|---| +| New version starts but behaves wrong | Rollback the code. Leave the database. | +| Backend will not start; schema looks wrong | Rollback the code, then restore only if it still fails. | +| Data is missing or corrupted | Restore the database from the most recent good dump. | +| Deploy aborted before starting | Nothing to undo — the backup ran before any change. | + +## Health checks + +`backend` and `frontend` both report container health, so `docker compose ps` shows real state rather +than merely "running". + +- **Backend:** `curl -fsS http://127.0.0.1:8080/health`. Anonymous, and deliberately **does not touch + the database** — a health check that queried MariaDB would restart a healthy backend whenever the + database blipped. `start_period` is 90s to cover first-boot schema reconciliation. +- **Frontend:** `wget` against nginx on port 80. +- `frontend` waits for `backend` to be *healthy*, not merely started, because nginx proxies `/api` to + it and refuses to start if the upstream cannot be resolved. + +A backend that cannot reach its database exits and is reported `unhealthy`, so a broken deploy does not +present as a running stack. diff --git a/deploy/deploy.sh b/deploy/deploy.sh index 644595c..34be8cc 100644 --- a/deploy/deploy.sh +++ b/deploy/deploy.sh @@ -28,6 +28,131 @@ compose() { docker compose "$@" } +# --------------------------------------------------------------------------- +# Database backup, taken BEFORE anything is stopped, built or replaced. +# +# A failed backup fails the deploy. The startup reconciler is additive and never +# drops a table holding rows, but "additive" is a property of the code, not a +# guarantee about the disk — and the first deploy after a schema change is +# exactly when a restore point matters. See deploy/README.md. +# --------------------------------------------------------------------------- +BACKUP_DIR="${BACKUP_DIR:-/opt/job-tracker/backups}" + +backup_database() { + if [ "${DEPLOY_SKIP_DB_BACKUP:-false}" = "true" ]; then + echo "WARNING: DEPLOY_SKIP_DB_BACKUP=true — deploying with no restore point. Emergency use only." + return 0 + fi + + local provider="${DATABASE_PROVIDER:-sqlite}" + local stamp + stamp="$(date -u +%Y%m%dT%H%M%SZ)" + mkdir -p "$BACKUP_DIR" + + if [ "$provider" != "mysql" ] && [ "$provider" != "mariadb" ]; then + # SQLite lives in the jobtracker_data volume. Tar it from a throwaway container + # so the host needs no sqlite tooling and no knowledge of the volume layout. + local target="$BACKUP_DIR/jobtracker-sqlite-${stamp}.tar.gz" + echo "Backing up SQLite data volume to ${target}" + # The archive path is built inside the container: passing /backup/... as an argument + # gets rewritten by MSYS path translation when the script is run from Git Bash. + if ! docker run --rm \ + -v jobtracker_data:/data:ro \ + -v "$BACKUP_DIR":/backup \ + -e ARCHIVE_NAME="$(basename "$target")" \ + alpine:3 sh -c 'tar czf "/backup/$ARCHIVE_NAME" -C /data .'; then + echo "SQLite volume backup FAILED. Aborting deploy." + return 1 + fi + verify_backup "$target" + return $? + fi + + local cs="${JOBTRACKER_CONNECTION_STRING:-}" + if [ -z "$cs" ]; then + echo "DATABASE_PROVIDER=${provider} but JOBTRACKER_CONNECTION_STRING is empty. Aborting deploy." + return 1 + fi + + # Parse the ADO.NET connection string. Keys are case-insensitive in .NET, so match that. + local db_host db_port db_name db_user db_pass + db_host="$(sed -n 's/.*[Ss]erver=\([^;]*\).*/\1/p' <<<"$cs")" + db_port="$(sed -n 's/.*[Pp]ort=\([^;]*\).*/\1/p' <<<"$cs")" + db_name="$(sed -n 's/.*[Dd]atabase=\([^;]*\).*/\1/p' <<<"$cs")" + db_user="$(sed -n 's/.*[Uu]ser[ ]*[Ii]*[Dd]*=\([^;]*\).*/\1/p' <<<"$cs")" + db_pass="$(sed -n 's/.*[Pp]assword=\([^;]*\).*/\1/p' <<<"$cs")" + db_port="${db_port:-3306}" + + if [ -z "$db_host" ] || [ -z "$db_name" ] || [ -z "$db_user" ]; then + echo "Could not parse host/database/user from JOBTRACKER_CONNECTION_STRING. Aborting deploy." + return 1 + fi + + local target="$BACKUP_DIR/jobtracker-${db_name}-${stamp}.sql.gz" + echo "Backing up MariaDB database '${db_name}' on ${db_host}:${db_port} to ${target}" + + # The password goes via MYSQL_PWD, never on the command line, so it cannot leak + # into the process list or the deploy log. + local dump_status=0 + if command -v mariadb-dump >/dev/null 2>&1; then + MYSQL_PWD="$db_pass" mariadb-dump \ + --host="$db_host" --port="$db_port" --user="$db_user" \ + --single-transaction --routines --events --quick \ + "$db_name" 2>/tmp/jobtracker-backup.err | gzip -c > "$target" || dump_status=$? + elif command -v mysqldump >/dev/null 2>&1; then + MYSQL_PWD="$db_pass" mysqldump \ + --host="$db_host" --port="$db_port" --user="$db_user" \ + --single-transaction --routines --events --quick \ + "$db_name" 2>/tmp/jobtracker-backup.err | gzip -c > "$target" || dump_status=$? + else + # No client on the host: run one in a container. --network host so the same + # host/port from the connection string resolves identically. + docker run --rm --network host -e MYSQL_PWD="$db_pass" mariadb:11 \ + mariadb-dump --host="$db_host" --port="$db_port" --user="$db_user" \ + --single-transaction --routines --events --quick \ + "$db_name" 2>/tmp/jobtracker-backup.err | gzip -c > "$target" || dump_status=$? + fi + + if [ "$dump_status" -ne 0 ]; then + echo "Database dump FAILED (exit ${dump_status}). Aborting deploy." + sed -e 's/password=[^ ]*/password=***/gI' /tmp/jobtracker-backup.err >&2 || true + rm -f "$target" + return 1 + fi + + verify_backup "$target" "CREATE TABLE" +} + +# A dump that exists but is empty or truncated is worse than none, because it +# looks like a restore point. Check size, and content when we know what to expect. +verify_backup() { + local target="$1" + local expect="${2:-}" + + if [ ! -s "$target" ]; then + echo "Backup file ${target} is missing or empty. Aborting deploy." + rm -f "$target" + return 1 + fi + + local size + size="$(du -h "$target" | cut -f1)" + + if [ -n "$expect" ] && ! gzip -dc "$target" | grep -q "$expect"; then + echo "Backup ${target} does not contain '${expect}' — it is not a usable dump. Aborting deploy." + rm -f "$target" + return 1 + fi + + echo "Backup verified: ${target} (${size})" + echo "Retention is manual — old backups in ${BACKUP_DIR} are never deleted automatically." + return 0 +} + +if ! backup_database; then + exit 1 +fi + build_core_with_recovery() { if compose build backend frontend; then return 0 diff --git a/docker-compose.yml b/docker-compose.yml index 200b9e1..1c4a51f 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -62,6 +62,15 @@ services: # ai-service. - ai_internal restart: unless-stopped + # Liveness only. /health does not touch the database on purpose: a health check that queried + # MariaDB would restart a healthy backend whenever the database blipped. + # start_period covers first-boot schema reconciliation, which can take a while on a new database. + healthcheck: + test: ["CMD", "curl", "-fsS", "http://127.0.0.1:8080/health"] + interval: 30s + timeout: 5s + retries: 3 + start_period: 90s frontend: build: @@ -77,11 +86,19 @@ services: ports: - "3000:80" depends_on: - - backend + backend: + condition: service_healthy networks: - default - shared_services restart: unless-stopped + # Cheap liveness: nginx answering on its own port. wget ships with the alpine base. + healthcheck: + test: ["CMD", "wget", "-q", "-O", "/dev/null", "http://127.0.0.1:80/"] + interval: 30s + timeout: 5s + retries: 3 + start_period: 10s ai-service: build: diff --git a/docs/production-readiness-review.md b/docs/production-readiness-review.md index eeac318..21f2a61 100644 --- a/docs/production-readiness-review.md +++ b/docs/production-readiness-review.md @@ -3,10 +3,14 @@ > 2026-07-19, before the first production deployment after the Phase 4/5 architecture changes. > Companion to `docs/phase-5-completion-report.md` and `docs/infrastructure/database-ownership.md`. > -> **Verdict: not ready to deploy unattended.** The application itself verifies clean, but three -> operational gaps (no pre-deploy backup, no documented restore, no backend health endpoint) and one -> external blocker (CI) stand between here and a safe first deploy. All are listed below with what -> would close them. +> **Updated 2026-07-19 (second pass).** The three operational blockers are **closed and verified**: +> `deploy.sh` now takes a verified backup before touching anything and aborts if it fails, restore and +> rollback are documented and tested against real containers, and `backend`/`frontend` both report +> health. +> +> **Remaining verdict: one external blocker.** CI is red for an environmental reason and deployment is +> gated on it. Everything here remains local verification. The accepted-risk items in *Remaining risks* +> are unchanged and still worth reading before the first deploy. ## Verified @@ -51,9 +55,9 @@ ## Remaining risks -### 1. No pre-deploy database backup — **blocker for first deploy** +### 1. ~~No pre-deploy database backup~~ — **CLOSED 2026-07-19** -`deploy/deploy.sh` does not dump the database before bringing the stack down. The first deploy after +*Original finding:* `deploy/deploy.sh` did not dump the database before bringing the stack down. The first deploy after these changes is exactly when a backup matters most: prod is many commits behind, and the reconciler will create roughly a dozen tables on first boot. That path is verified on containers, **not on your data**. @@ -61,23 +65,43 @@ data**. `BackupController` exposes only `POST /api/backup/encrypted` — an application-level encrypted export, not an operational database dump. It is not a substitute. -**To close:** run `mariadb-dump` before `compose down`, keep the dump, and ideally add that step to -`deploy.sh` so it is not a thing to remember. +**Closed.** `deploy/deploy.sh` now backs up before it builds, stops or replaces anything, and aborts +the deploy if the backup fails. Timestamped and gzipped to `/opt/job-tracker/backups` (override with +`BACKUP_DIR`), so a deploy never overwrites an earlier backup. The password travels via `MYSQL_PWD`, +never on the command line. The dump is rejected if it is empty or lacks `CREATE TABLE`. SQLite +deployments get the data volume tarred instead. -### 2. No documented restore procedure — **blocker for first deploy** +*Verified:* dump taken from a seeded MariaDB 11 container; failure paths (bad credentials, missing +connection string) abort with a non-zero status and leave no misleading partial file. -There is no written restore path. A backup you have never restored is a hypothesis. `deploy/README.md` +### 2. ~~No documented restore procedure~~ — **CLOSED 2026-07-19** + +*Original finding:* there was no written restore path. A backup you have never restored is a hypothesis. `deploy/README.md` mentions backups in one line ("keep backups and volume persistence") and stops there. -**To close:** document restore, and rehearse it once against a scratch database. +**Closed.** `deploy/README.md` documents backup creation, location, retention, database restore, +application rollback, and — importantly — when to use which. Restore and rollback are presented as +separate operations, because a bad deploy usually needs only the rollback. -### 3. No backend health endpoint — **operational gap** +*Verified:* the dump was restored into a clean MariaDB 11 container and the rows came back identical. +The rehearsal is the documented example. -`docker-compose.yml` defines health checks for `ai-service` and `ollama` but **not for `backend` or -`frontend`**. Nothing automatically detects a backend that started and then became unhealthy, and +### 3. ~~No backend health endpoint~~ — **CLOSED 2026-07-19** + +*Original finding:* `docker-compose.yml` defined health checks for `ai-service` and `ollama` but not for +`backend` or `frontend`. Nothing automatically detects a backend that started and then became unhealthy, and `depends_on` cannot gate on its readiness. -**To close:** add a `/health` endpoint (anonymous, no data) and a compose health check. +**Closed.** `GET /health` is anonymous, cheap, and deliberately **does not touch the database** — a +health check that queried MariaDB would restart a healthy backend whenever the database blipped, and +would hand out an unauthenticated way to probe database availability. Compose health checks now cover +`backend` (curl, 90s start period for first-boot reconciliation) and `frontend` (wget against nginx), +and `frontend` waits for `backend` to be *healthy* rather than merely started. + +*Verified:* both containers reach `healthy`; a backend pointed at an unreachable database exits and is +reported `unhealthy`, so a broken deploy cannot present as a running stack. `/health` returns 200 +anonymously while `/api/jobapplications` returns 401 **with `Auth:Require` unset**, confirming the +explicit `[Authorize]` work holds independently of that flag. ### 4. CI is red for an environmental reason — **external blocker** @@ -115,9 +139,9 @@ verification is tests, builds and startup checks. Friction points in that journe Run in order. Stop if any step fails. 1. **Resolve CI**, or make a deliberate decision to deploy from a manually verified commit. -2. **Back up the production database.** - `mariadb-dump -u -p --single-transaction --routines jobtracker > jobtracker-$(date +%F).sql` -3. **Verify the dump is non-empty and contains `CREATE TABLE JobApplications`.** +2. ~~Back up the production database.~~ **`deploy.sh` now does this automatically and aborts if it + fails.** Confirm afterwards that a new file appeared in `/opt/job-tracker/backups`. +3. ~~Verify the dump.~~ **The script verifies size and content before continuing.** 4. **Note the current commit** for rollback: `git rev-parse HEAD` on the prod checkout. 5. Confirm `.env` has `AUTH_JWT_KEY`, `AI_SERVICE_TOKEN`, `JOBTRACKER_CONNECTION_STRING`, `DATABASE_PROVIDER=mysql`, and that `Auth__Require=true` is still in `docker-compose.yml`.