From 5f0c0b05499426880e69fa160053ebf34d9487f4 Mon Sep 17 00:00:00 2001 From: cesnimda Date: Mon, 31 Aug 2026 11:57:03 +0200 Subject: [PATCH] fix(deploy): restore previous release on failure --- BLOCKERS.md | 1 + deploy/README.md | 9 ++++ deploy/deploy.sh | 56 +++++++++++++++++++++---- docker-compose.yml | 2 + docs/work-programmes/master-progress.md | 2 + 5 files changed, 62 insertions(+), 8 deletions(-) diff --git a/BLOCKERS.md b/BLOCKERS.md index 354041d..11db887 100644 --- a/BLOCKERS.md +++ b/BLOCKERS.md @@ -27,6 +27,7 @@ Updated: 2026-08-31 - **Recommended:** Verify backup/restore before deployment, then exercise login, existing application counts, Career Workspace, public CV refresh/download, AI, and attachments in order. - **Current access check:** Read-only SSH access is confirmed. All four JobTracker containers are healthy with zero observed restarts, but root free space is now 36 GiB (83% used). The production checkout is at `de937d25dc5e` / app version `157` and has an unreviewed mode-only change to `deploy/deploy.sh`. No production change or deployment was attempted. - **Current status:** On 2026-08-31 the anonymous public liveness endpoint `https://jobs.cesnimda.uk/health` returned HTTP 200 with version `276`, and the Gitea host returned HTTP 200; the earlier public 502 is no longer present. Anonymous `https://jobs.cesnimda.uk/api/health` returned HTTP 401 as expected for an authenticated API route. This proves the proxy/frontend liveness path only, not authenticated application behavior or database readiness. Gitea run 696 failed after repeated truncated Playwright downloads and then a runner-level exit 139 before any test assertion; the workflow now installs through the lockfile CLI and retries only that transient process failure without hiding real test failures. Current remote CI, backup/restore, authenticated smoke and rollback still require operator-controlled environments. Read-only PROD-001 inventory found the JobTracker Ollama and frontend published on all host interfaces, the newest gzip-valid MariaDB backup dated 2026-08-02, no observed scheduled JobTracker backup, and no owner-file/key/tombstone recovery bundle. Close these rollout gates before deployment; see `docs/production/production-ai-hardware-assessment.md`. +- **Repository-side rollback status:** Core images are now tagged by commit, the prior running backend/frontend images are retained, and any failure after replacement triggers an automatic core rollback while preserving the failed CI status. Compose configuration validates locally. A real rollback rehearsal remains blocked because the local Linux Docker daemon is unavailable and production mutation is not authorised; run the failure-injection rehearsal on a disposable Linux Docker host before relying on it in production. ## Account deletion retention and restore policy diff --git a/deploy/README.md b/deploy/README.md index 80188d7..7f28c10 100644 --- a/deploy/README.md +++ b/deploy/README.md @@ -270,6 +270,15 @@ docker compose start backend This is the usual fix for a bad deploy, and it **does not touch the database**. +Normal deployments tag backend and frontend images with `APP_COMMIT_SHA` and retain the images from +the currently running containers as `jobtracker-backend:previous` and `jobtracker-frontend:previous`. +If any step fails after core replacement—including public liveness, readiness, or auth-configuration +checks—`deploy.sh` automatically recreates both core services from those retained images and still +returns a failed status so CI reports the rejected release. The previous images are replaced only at +the start of the next deployment. + +Use the manual procedure below if automatic restoration itself fails or an older release is required. + ```bash cd /opt/job-tracker/app # the deployment checkout git log --oneline -5 # find the last good commit diff --git a/deploy/deploy.sh b/deploy/deploy.sh index d991b1d..c37641c 100644 --- a/deploy/deploy.sh +++ b/deploy/deploy.sh @@ -65,6 +65,8 @@ export APP_VERSION="${APP_VERSION:-0.0.0}" export APP_COMMIT_SHA="${APP_COMMIT_SHA:-unknown}" export APP_BUILD_STAMP="${APP_BUILD_STAMP:-unknown}" export DEPLOY_BUILD_AI_SERVICE="${DEPLOY_BUILD_AI_SERVICE:-false}" +export BACKEND_IMAGE="${BACKEND_IMAGE:-jobtracker-backend:${APP_COMMIT_SHA}}" +export FRONTEND_IMAGE="${FRONTEND_IMAGE:-jobtracker-frontend:${APP_COMMIT_SHA}}" compose() { docker compose -f docker-compose.yml "$@" @@ -477,6 +479,50 @@ if ! backup_database; then exit 1 fi +previous_backend_ref="" +previous_frontend_ref="" +capture_previous_core_image() { + local service="$1" previous_ref="$2" container_id image_id + container_id="$(compose ps -q "$service" 2>/dev/null || true)" + [ -n "$container_id" ] || return 0 + image_id="$(docker inspect -f '{{.Image}}' "$container_id" 2>/dev/null || true)" + [ -n "$image_id" ] || return 0 + docker tag "$image_id" "$previous_ref" + printf '%s' "$previous_ref" +} + +previous_backend_ref="$(capture_previous_core_image backend jobtracker-backend:previous)" +previous_frontend_ref="$(capture_previous_core_image frontend jobtracker-frontend:previous)" +replacement_started=false +auth_config_body_file="" +auth_config_headers_file="" + +finish_deploy() { + local status=$? + trap - EXIT + [ -z "$auth_config_body_file" ] || rm -f "$auth_config_body_file" + [ -z "$auth_config_headers_file" ] || rm -f "$auth_config_headers_file" + + if [ "$status" -ne 0 ] && [ "$replacement_started" = true ]; then + if [ -n "$previous_backend_ref" ] && [ -n "$previous_frontend_ref" ]; then + echo "Deployment failed after core replacement; restoring previous healthy images." + if BACKEND_IMAGE="$previous_backend_ref" FRONTEND_IMAGE="$previous_frontend_ref" \ + compose up -d --force-recreate --remove-orphans backend frontend; then + echo "Previous core release restored. Deployment remains failed; inspect the logs above." + else + echo "Automatic core rollback failed. Manual recovery is required." + compose ps || true + compose logs --tail=200 backend frontend || true + fi + else + echo "Deployment failed, but no complete previous core release was available for automatic rollback." + fi + fi + + exit "$status" +} +trap finish_deploy EXIT + build_core_with_recovery() { if compose build backend frontend; then return 0 @@ -500,7 +546,6 @@ build_ai_with_recovery() { compose build --no-cache ai-service } -compose pull || true build_core_with_recovery if [ "$DEPLOY_BUILD_AI_SERVICE" = "true" ]; then build_ai_with_recovery @@ -508,6 +553,7 @@ else echo "Skipping ai-service rebuild during deploy (set DEPLOY_BUILD_AI_SERVICE=true to rebuild it)." fi # Force recreation so updated port mappings, env vars, and container config always apply on deploy. +replacement_started=true compose up -d --force-recreate --remove-orphans backend frontend if [ "$DEPLOY_BUILD_AI_SERVICE" = "true" ]; then # Ollama is opt-in (compose "bundled-ollama" profile). Deploys reuse an @@ -539,10 +585,6 @@ fi public_base="${APP_PUBLIC_BASE_URL%/}" auth_config_body_file="$(mktemp)" auth_config_headers_file="$(mktemp)" -cleanup_public_check() { - rm -f "$auth_config_body_file" "$auth_config_headers_file" -} -trap cleanup_public_check EXIT echo "Running public smoke check against ${public_base}" if ! curl -fsS "${public_base}/" >/dev/null; then @@ -579,10 +621,8 @@ if ! grep -q 'requireAuth' "$auth_config_body_file"; then exit 1 fi -trap - EXIT -cleanup_public_check - # Clean up old legacy container name if it still exists from pre-rename deployments. docker rm -f app-summarizer-1 2>/dev/null || true +replacement_started=false echo "Deployment complete: ${APP_VERSION} ${APP_COMMIT_SHA}" diff --git a/docker-compose.yml b/docker-compose.yml index 850d2c6..84187da 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -1,6 +1,7 @@ services: backend: + image: ${BACKEND_IMAGE:-jobtracker-backend:local} build: context: . dockerfile: JobTrackerApi/Dockerfile @@ -112,6 +113,7 @@ services: start_period: 90s frontend: + image: ${FRONTEND_IMAGE:-jobtracker-frontend:local} build: context: ./job-tracker-ui # Next's build type-checker needs more than Docker's default 64MB /dev/shm; too little diff --git a/docs/work-programmes/master-progress.md b/docs/work-programmes/master-progress.md index c96f4b1..5f3a21c 100644 --- a/docs/work-programmes/master-progress.md +++ b/docs/work-programmes/master-progress.md @@ -50,6 +50,7 @@ Updated: 2026-08-30 - Began the JT-019 schema-ownership retirement with an executable 49-table ownership partition and transferred the leaf `SystemEmailSettings` table from MariaDB-only startup DDL to an additive provider-aware migration. Fresh SQLite now receives the table; legacy rows are preserved and startup no longer creates it. - Continued the job-application controller split by moving canonical pipeline metadata, soft delete/restore, follow-up scheduling, and event history into `JobApplicationLifecycleController` without changing routes, authorization, tenant filtering, response shapes, or event behavior. Status mutation remains with core updates until its shared applied-date invariant has a single service owner. - Added a minimal public `/ready` dependency probe alongside the existing `/health` liveness probe. Nginx exposes both, deployment validation now checks the frontend, API liveness, database readiness, and public auth configuration separately, while detailed dependency metadata remains restricted to Admin/System. +- Hardened deployment replacement semantics: backend/frontend images receive exact commit tags, the prior running core images are retained as the rollback release, and any post-replacement failure automatically restores both previous services while keeping the deployment result failed. Removed the non-actionable blanket `compose pull || true` suppression. - Transferred the independent `UserRuleSettings` table from both provider startup paths to its own provider-aware migration; owner-keyed settings survive adoption, downgrade and retry. - Moved `GmailReviewDecisions` into a provider-aware migration, preserving existing SQLite decisions and closing the previously missing MariaDB table path. - Moved recovery codes, trusted devices, and revocable user sessions into one provider-aware authentication-support migration; populated legacy rows and indexes survive adoption, downgrade, and retry. @@ -96,6 +97,7 @@ Updated: 2026-08-30 - Analytics-controller extraction: Release build passed with 0 warnings/errors and the complete backend suite passed 736/736. - Lifecycle-controller extraction: locked restore passed, Release build passed with 0 warnings/errors, and the complete backend suite passed 736/736. - Health/readiness split: Release build passed with 0 warnings/errors, the complete backend suite passed 736/736, and the focused Playwright probe passed 1/1 against a real disposable SQLite-backed API process. +- Deployment rollback configuration: Docker Compose configuration validation passed with non-secret fixture values. Runtime rollback rehearsal remains pending because the local Linux Docker daemon is offline. - Focused frontend: 2 suites, 6 tests passed. - Full frontend: 64 suites, 272 tests passed. - Next production build and TypeScript: passed.