fix(deploy): restore previous release on failure

BLOCKERS.md

@@ -27,6 +27,7 @@ Updated: 2026-08-31
 - **Recommended:** Verify backup/restore before deployment, then exercise login, existing application counts, Career Workspace, public CV refresh/download, AI, and attachments in order.
 - **Current access check:** Read-only SSH access is confirmed. All four JobTracker containers are healthy with zero observed restarts, but root free space is now 36 GiB (83% used). The production checkout is at `de937d25dc5e` / app version `157` and has an unreviewed mode-only change to `deploy/deploy.sh`. No production change or deployment was attempted.
 - **Current status:** On 2026-08-31 the anonymous public liveness endpoint `https://jobs.cesnimda.uk/health` returned HTTP 200 with version `276`, and the Gitea host returned HTTP 200; the earlier public 502 is no longer present. Anonymous `https://jobs.cesnimda.uk/api/health` returned HTTP 401 as expected for an authenticated API route. This proves the proxy/frontend liveness path only, not authenticated application behavior or database readiness. Gitea run 696 failed after repeated truncated Playwright downloads and then a runner-level exit 139 before any test assertion; the workflow now installs through the lockfile CLI and retries only that transient process failure without hiding real test failures. Current remote CI, backup/restore, authenticated smoke and rollback still require operator-controlled environments. Read-only PROD-001 inventory found the JobTracker Ollama and frontend published on all host interfaces, the newest gzip-valid MariaDB backup dated 2026-08-02, no observed scheduled JobTracker backup, and no owner-file/key/tombstone recovery bundle. Close these rollout gates before deployment; see `docs/production/production-ai-hardware-assessment.md`.
+- **Repository-side rollback status:** Core images are now tagged by commit, the prior running backend/frontend images are retained, and any failure after replacement triggers an automatic core rollback while preserving the failed CI status. Compose configuration validates locally. A real rollback rehearsal remains blocked because the local Linux Docker daemon is unavailable and production mutation is not authorised; run the failure-injection rehearsal on a disposable Linux Docker host before relying on it in production.
 
 ## Account deletion retention and restore policy
 

deploy/README.md

@@ -270,6 +270,15 @@ docker compose start backend
 
 This is the usual fix for a bad deploy, and it **does not touch the database**.
 
+Normal deployments tag backend and frontend images with `APP_COMMIT_SHA` and retain the images from
+the currently running containers as `jobtracker-backend:previous` and `jobtracker-frontend:previous`.
+If any step fails after core replacement—including public liveness, readiness, or auth-configuration
+checks—`deploy.sh` automatically recreates both core services from those retained images and still
+returns a failed status so CI reports the rejected release. The previous images are replaced only at
+the start of the next deployment.
+
+Use the manual procedure below if automatic restoration itself fails or an older release is required.
+
 ```bash
 cd /opt/job-tracker/app          # the deployment checkout
 git log --oneline -5             # find the last good commit

deploy/deploy.sh

@@ -65,6 +65,8 @@ export APP_VERSION="${APP_VERSION:-0.0.0}"
 export APP_COMMIT_SHA="${APP_COMMIT_SHA:-unknown}"
 export APP_BUILD_STAMP="${APP_BUILD_STAMP:-unknown}"
 export DEPLOY_BUILD_AI_SERVICE="${DEPLOY_BUILD_AI_SERVICE:-false}"
+export BACKEND_IMAGE="${BACKEND_IMAGE:-jobtracker-backend:${APP_COMMIT_SHA}}"
+export FRONTEND_IMAGE="${FRONTEND_IMAGE:-jobtracker-frontend:${APP_COMMIT_SHA}}"
 
 compose() {
   docker compose -f docker-compose.yml "$@"
@@ -477,6 +479,50 @@ if ! backup_database; then
   exit 1
 fi
 
+previous_backend_ref=""
+previous_frontend_ref=""
+capture_previous_core_image() {
+  local service="$1" previous_ref="$2" container_id image_id
+  container_id="$(compose ps -q "$service" 2>/dev/null || true)"
+  [ -n "$container_id" ] || return 0
+  image_id="$(docker inspect -f '{{.Image}}' "$container_id" 2>/dev/null || true)"
+  [ -n "$image_id" ] || return 0
+  docker tag "$image_id" "$previous_ref"
+  printf '%s' "$previous_ref"
+}
+
+previous_backend_ref="$(capture_previous_core_image backend jobtracker-backend:previous)"
+previous_frontend_ref="$(capture_previous_core_image frontend jobtracker-frontend:previous)"
+replacement_started=false
+auth_config_body_file=""
+auth_config_headers_file=""
+
+finish_deploy() {
+  local status=$?
+  trap - EXIT
+  [ -z "$auth_config_body_file" ] || rm -f "$auth_config_body_file"
+  [ -z "$auth_config_headers_file" ] || rm -f "$auth_config_headers_file"
+
+  if [ "$status" -ne 0 ] && [ "$replacement_started" = true ]; then
+    if [ -n "$previous_backend_ref" ] && [ -n "$previous_frontend_ref" ]; then
+      echo "Deployment failed after core replacement; restoring previous healthy images."
+      if BACKEND_IMAGE="$previous_backend_ref" FRONTEND_IMAGE="$previous_frontend_ref" \
+          compose up -d --force-recreate --remove-orphans backend frontend; then
+        echo "Previous core release restored. Deployment remains failed; inspect the logs above."
+      else
+        echo "Automatic core rollback failed. Manual recovery is required."
+        compose ps || true
+        compose logs --tail=200 backend frontend || true
+      fi
+    else
+      echo "Deployment failed, but no complete previous core release was available for automatic rollback."
+    fi
+  fi
+
+  exit "$status"
+}
+trap finish_deploy EXIT
+
 build_core_with_recovery() {
   if compose build backend frontend; then
     return 0
@@ -500,7 +546,6 @@ build_ai_with_recovery() {
   compose build --no-cache ai-service
 }
 
-compose pull || true
 build_core_with_recovery
 if [ "$DEPLOY_BUILD_AI_SERVICE" = "true" ]; then
   build_ai_with_recovery
@@ -508,6 +553,7 @@ else
   echo "Skipping ai-service rebuild during deploy (set DEPLOY_BUILD_AI_SERVICE=true to rebuild it)."
 fi
 # Force recreation so updated port mappings, env vars, and container config always apply on deploy.
+replacement_started=true
 compose up -d --force-recreate --remove-orphans backend frontend
 if [ "$DEPLOY_BUILD_AI_SERVICE" = "true" ]; then
   # Ollama is opt-in (compose "bundled-ollama" profile). Deploys reuse an
@@ -539,10 +585,6 @@ fi
 public_base="${APP_PUBLIC_BASE_URL%/}"
 auth_config_body_file="$(mktemp)"
 auth_config_headers_file="$(mktemp)"
-cleanup_public_check() {
-  rm -f "$auth_config_body_file" "$auth_config_headers_file"
-}
-trap cleanup_public_check EXIT
 
 echo "Running public smoke check against ${public_base}"
 if ! curl -fsS "${public_base}/" >/dev/null; then
@@ -579,10 +621,8 @@ if ! grep -q 'requireAuth' "$auth_config_body_file"; then
   exit 1
 fi
 
-trap - EXIT
-cleanup_public_check
-
 # Clean up old legacy container name if it still exists from pre-rename deployments.
 docker rm -f app-summarizer-1 2>/dev/null || true
 
+replacement_started=false
 echo "Deployment complete: ${APP_VERSION} ${APP_COMMIT_SHA}"

docker-compose.yml

@@ -1,6 +1,7 @@
 services:
 
   backend:
+    image: ${BACKEND_IMAGE:-jobtracker-backend:local}
     build:
       context: .
       dockerfile: JobTrackerApi/Dockerfile
@@ -112,6 +113,7 @@ services:
       start_period: 90s
 
   frontend:
+    image: ${FRONTEND_IMAGE:-jobtracker-frontend:local}
     build:
       context: ./job-tracker-ui
       # Next's build type-checker needs more than Docker's default 64MB /dev/shm; too little

docs/work-programmes/master-progress.md

@@ -50,6 +50,7 @@ Updated: 2026-08-30
 - Began the JT-019 schema-ownership retirement with an executable 49-table ownership partition and transferred the leaf `SystemEmailSettings` table from MariaDB-only startup DDL to an additive provider-aware migration. Fresh SQLite now receives the table; legacy rows are preserved and startup no longer creates it.
 - Continued the job-application controller split by moving canonical pipeline metadata, soft delete/restore, follow-up scheduling, and event history into `JobApplicationLifecycleController` without changing routes, authorization, tenant filtering, response shapes, or event behavior. Status mutation remains with core updates until its shared applied-date invariant has a single service owner.
 - Added a minimal public `/ready` dependency probe alongside the existing `/health` liveness probe. Nginx exposes both, deployment validation now checks the frontend, API liveness, database readiness, and public auth configuration separately, while detailed dependency metadata remains restricted to Admin/System.
+- Hardened deployment replacement semantics: backend/frontend images receive exact commit tags, the prior running core images are retained as the rollback release, and any post-replacement failure automatically restores both previous services while keeping the deployment result failed. Removed the non-actionable blanket `compose pull || true` suppression.
 - Transferred the independent `UserRuleSettings` table from both provider startup paths to its own provider-aware migration; owner-keyed settings survive adoption, downgrade and retry.
 - Moved `GmailReviewDecisions` into a provider-aware migration, preserving existing SQLite decisions and closing the previously missing MariaDB table path.
 - Moved recovery codes, trusted devices, and revocable user sessions into one provider-aware authentication-support migration; populated legacy rows and indexes survive adoption, downgrade, and retry.
@@ -96,6 +97,7 @@ Updated: 2026-08-30
 - Analytics-controller extraction: Release build passed with 0 warnings/errors and the complete backend suite passed 736/736.
 - Lifecycle-controller extraction: locked restore passed, Release build passed with 0 warnings/errors, and the complete backend suite passed 736/736.
 - Health/readiness split: Release build passed with 0 warnings/errors, the complete backend suite passed 736/736, and the focused Playwright probe passed 1/1 against a real disposable SQLite-backed API process.
+- Deployment rollback configuration: Docker Compose configuration validation passed with non-secret fixture values. Runtime rollback rehearsal remains pending because the local Linux Docker daemon is offline.
 - Focused frontend: 2 suites, 6 tests passed.
 - Full frontend: 64 suites, 272 tests passed.
 - Next production build and TypeScript: passed.