docs(release): scope legacy rollback guidance to its migration baseline

deploy/deploy.sh

@@ -660,10 +660,18 @@ finish_deploy() {
 
   if [ "$status" -ne 0 ] && [ "$replacement_started" = true ]; then
     if [ -n "$previous_backend_ref" ] && [ -n "$previous_frontend_ref" ]; then
-      echo "Deployment failed after core replacement; restoring previous healthy images."
+      echo "Deployment failed after core replacement; restarting previous core images."
       if BACKEND_IMAGE="$previous_backend_ref" FRONTEND_IMAGE="$previous_frontend_ref" \
           compose up -d --force-recreate --remove-orphans backend frontend; then
-        echo "Previous core release restored. Deployment remains failed; inspect the logs above."
+        if curl -fsS --connect-timeout 5 --max-time 5 --retry 5 --retry-delay 2 --retry-connrefused --retry-max-time 30 "${APP_PUBLIC_BASE_URL%/}/" >/dev/null &&
+            curl -fsS --connect-timeout 5 --max-time 5 --retry 5 --retry-delay 2 --retry-connrefused --retry-max-time 30 "${APP_PUBLIC_BASE_URL%/}/health" >/dev/null &&
+            curl -fsS --connect-timeout 5 --max-time 5 --retry 5 --retry-delay 2 --retry-connrefused --retry-max-time 30 "${APP_PUBLIC_BASE_URL%/}/ready" >/dev/null; then
+          echo "Previous core images restarted and public core checks passed. Deployment remains failed; inspect the logs above."
+        else
+          echo "Previous core images restarted, but rollback health verification failed. Manual recovery is required."
+          compose ps || true
+          compose logs --tail=200 backend frontend || true
+        fi
       else
         echo "Automatic core rollback failed. Manual recovery is required."
         compose ps || true

deploy/first-production-deployment.md

@@ -1,5 +1,11 @@
 # First production deployment runbook
 
+> Historical July 2026 baseline. Current release preparation follows
+> [the release qualification matrix](../docs/release-readiness/14_RELEASE_ACCEPTANCE_MATRIX.md) and
+> [recovery specification](../docs/release-readiness/11_BACKUP_RESTORE_ROLLBACK.md).
+> Reconcile the examples below with the exact candidate, migration history and isolated rehearsal
+> before operational use. Current rollback health checks are source-tested offline.
+
 > Written 2026-07-19 against the actual implementation, not against the other docs. Every command and
 > failure mode below was exercised locally against MariaDB 11 containers. No production data was used.
 >
@@ -220,9 +226,11 @@ everything written since the dump — including anything users did during and af
 **Restore the database only if the data itself is wrong.** If the backend simply will not start, the
 code rollback is almost certainly the whole fix.
 
-Why a code rollback is safe here: every Phase 4/5 migration is a no-op — the reconciler owns those
+The July 2026 baseline used no-op Phase 4/5 migrations around reconciler-owned tables. Current
-tables — so reverting code never leaves migration history ahead of the schema. The reconciler is
+migrations also perform schema changes, including `AddJobRevision` adding the application Version
-additive and never drops a table holding rows, so the older code simply ignores the newer tables.
+column and `AddUserOperationEvents` creating its table and indexes. Qualify the exact old/new image
+pair against the current migration history and representative post-upgrade writes. The recovery
+specification above includes ownership, revision, document and restore checks for that rehearsal.
 
 ## Procedure
 

docs/release-readiness/11_BACKUP_RESTORE_ROLLBACK.md

@@ -11,7 +11,7 @@
 | `deploy/deploy.sh:backup_database` | Requires explicit provider; SQL dump for MariaDB, volume archive for SQLite; rejects missing/empty/malformed/trailerless dumps | Format checks do not restore or compare records; emergency skip exists and is not approved here |
 | `backup_recovery_volumes` | MariaDB additionally archives required application data volume; independently archives tombstones when present | Consistency between dump/files/tombstones and current key availability still needs isolated rehearsal |
 | Retention/off-host copy | Daily/monthly retention and optional rclone copy | Source message calls remote encrypted but source does not establish actual remote encryption configuration; off-host configuration and recovery remain unverified |
-| `finish_deploy` | On failure after replacement, restarts previous backend/frontend image references when both exist; retains failed exit | No database reversal, no previous AI image restoration, and no post-rollback health check in this handler |
+| `finish_deploy` | On failure after replacement, restarts previous backend/frontend references, checks public frontend/liveness/readiness with bounded retries and retains failed exit | Offline stubs verify control flow. Actual recovery, image identity, database reversal and previous AI image restoration remain unqualified |
 | Tombstone store and replay | Validates ledger records; Program invokes `StageRestoredAccountsAsync` at startup; deleted identities are staged again | Test restored data with latest independent tombstones; corrupt ledger must fail closed. Existing AccountDeletionTests include restore replay, not a complete operational restore |
 
 Build retry paths prune Docker build/system cache and deployment may warm an Ollama model. Therefore even a proposed "local deploy smoke" must not run this script against the shared workstation. No such action was taken.
@@ -71,3 +71,11 @@ The fix explicitly checks preparation and copy. It writes the stamp to a tempora
 - Red run: **2 failures and 1 passing control** against the pre-fix function. The partial-copy case directly reproduced false completion; the marker-publication case required an atomic rename the old function did not perform.
 - Green run: **12/12 backup tests** including four monthly cases (successful copy and stamp, partial copy, failed marker rename, existing completed month). Rollback handler and hosted quarantine checks passed **5/5** and **4/4**. Bash syntax validation passed.
 - Only temporary synthetic files were created or copied. Pruning and upload were stubbed. Approved rehearsal still needs independent checks of artifacts for the marker stamp, encryption, retrieval and actual restore before any release decision.
+
+## RR-12 post-rollback core health checks
+
+11 November 2026. Source `a39ac847298e1fbe7667cf42671e7ec265500fcd` on `codex/release-rollback-health`, based on `24a62997`. After a successful previous-image restart, the handler now checks the configured public frontend, `/health` and `/ready`. Each curl request has connection/request timeouts and bounded retries for startup failures. A failed check reports that manual recovery is required and collects status/logs. The deployment's original nonzero status is retained even if diagnostics fail. The success message is limited to restarted core images and passing public core checks.
+
+The red test run executed six methods and reported four assertion failures because the old handler performed no probes. The green run passed six methods, including failed-probe subcases for all three endpoints, Compose failure, incomplete references, pre-replacement failure and successful deployment. Probe failures and Compose failure also use failing diagnostic stubs to verify original-status preservation. Twelve backup tests and shell syntax validation passed separately. Earlier five-test receipts remain historical and overlap this run.
+
+Only the extracted exit handler/trap ran with shell functions substituting Compose, Docker, curl and cleanup. The full deployment script was never sourced or executed, and no network/containers/models were used. Raw red/green/backup/syntax receipts are in `evidence/rollback-health/`. Actual startup timing, proxy routing, exact image identity, post-migration writes and database/files/keys/tombstone recovery still need the authorised isolated rehearsal. Passing public probes alone does not establish recovery correctness.

docs/release-readiness/13_REMEDIATION_REGISTER.md

@@ -15,7 +15,8 @@ Discovery only, 2026-11-02. These are qualification gaps unless a concrete defec
 | RR-09 | Backup/restore and rollback | Existing procedures and historical seeded rehearsal are not a current restore. Need isolated recovery-set completeness, decryption, tombstone behavior, post-write rollback compatibility and agreed RPO/RTO |
 | RR-10 | AI quality/activation | Deterministic Docling fixtures are distinct from real-model semantic accuracy. Optional unqualified adapter remains disabled; no AI invocation authorised here |
 | RR-11 | Backup host parsing rejected supported `Host=` alias and uppercase spelling | FIXED AND OFFLINE-TESTED. See [bounded remediation](RR-11_IMPLEMENTATION.md). Real dump/restore qualification remains open |
-| RR-12 | Rollback recreates core images without post-rollback health qualification; no schema/data/AI reversal | OFFLINE HANDLER CHECKS PASSED 5/5. See [failure-injection evidence](11_BACKUP_RESTORE_ROLLBACK.md#rr-12-offline-failure-injection). Actual image recovery and health/readiness after migration/writes still require isolated rehearsal |
+| RR-12 | Previous handler omitted post-rollback health checks; schema/data/AI recovery remains separate | Health probes FIXED in `a39ac847` and six offline handler methods passed, including failed endpoints and diagnostics. See [health follow-up](11_BACKUP_RESTORE_ROLLBACK.md#rr-12-post-rollback-core-health-checks). Actual image identity, startup timing and recovery after migration/writes still require isolated rehearsal |
 | RR-13 | Off-host encryption and recovery-set consistency need independent proof. Failed transfer and partial monthly copy previously let deployment continue | Transfer and monthly marker failure gates FIXED AND OFFLINE-TESTED. Actual encryption, retrieval/decryption and database/files/key/tombstone consistency require approved isolation |
+| RR-14 | July first-deploy runbook inferred rollback safety from its no-op migration baseline, while current migration Up methods add schema | Historical scope and current qualification links clarified. `AddJobRevision` adds Version and `AddUserOperationEvents` creates a table/indexes. Exact old/new image compatibility after migration/writes remains an isolated rehearsal gate |
 
 Continue evidence work against the [source endpoint and environment census](03_ENDPOINT_ENVIRONMENT_CENSUS.md). Actual qualification requires an accepted candidate and isolated environment. Every fix needs minimal scope, targeted verification and the applicable stage approval.

docs/release-readiness/HANDOFF.md

@@ -1,5 +1,9 @@
 # Release discovery handoff
 
+11 November. Branch `codex/release-rollback-health`, baseline `24a62997`, source `a39ac847` adds bounded post-rollback frontend/liveness/readiness probes and failing-diagnostics coverage while preserving the original exit status. Six extracted-handler tests, twelve backup tests and shell syntax validation passed. Full deploy, Docker, network, models and real restore were not executed. [Evidence and remaining qualification](11_BACKUP_RESTORE_ROLLBACK.md#rr-12-post-rollback-core-health-checks).
+
+RR-14 clarifies that the first-production runbook is a July baseline. Current Up migrations add the Version column and operation-event schema. Its old no-op argument cannot qualify a current old/new image pair. Current candidate, migration/write and recovery rehearsal requirements are now linked from that runbook. This was source/document review only.
+
 RR-12 offline failure injection passed 5/5 against the unchanged deploy exit handler. The evidence covers synthetic prior-image recreation, failed rollback diagnostics, original exit status and conditions that skip rollback. [Recovery report](11_BACKUP_RESTORE_ROLLBACK.md#rr-12-offline-failure-injection) distinguishes this from a real post-migration/post-write rehearsal. RR-13 off-host retrieval/decryption and consistent restore remain open. No production qualification.
 
 RR-13 transfer guard [reproduced and fixed offline](11_BACKUP_RESTORE_ROLLBACK.md#rr-13-off-host-upload-failure-handling). Daily and monthly copy errors stop deployment, and the success log makes no encryption claim. Actual remote encryption and restore remain open.

docs/release-readiness/evidence/rollback-health/BACKUP.txt

@@ -0,0 +1,5 @@
+............
+----------------------------------------------------------------------
+Ran 12 tests in 5.884s
+
+OK

docs/release-readiness/evidence/rollback-health/RED.txt

@@ -0,0 +1,73 @@
+FFFF....
+======================================================================
+FAIL: test_each_failed_health_probe_preserves_failure_and_requires_manual_recovery (__main__.DeployRollbackTests.test_each_failed_health_probe_preserves_failure_and_requires_manual_recovery) (path='/')
+----------------------------------------------------------------------
+Traceback (most recent call last):
+  File "C:\Users\Connor\.codex\worktrees\release-readiness-discovery\Job tracker\scripts\test-deploy-rollback.py", line 72, in test_each_failed_health_probe_preserves_failure_and_requires_manual_recovery
+    self.assertEqual(self.probes()[:index + 1], calls[1:-2])
+AssertionError: Lists differ: ['CALL curl -fsS --connect-timeout 5 --max[97 chars]id/'] != []
+
+First list contains 1 additional elements.
+First extra element 0:
+'CALL curl -fsS --connect-timeout 5 --max-time 5 --retry 5 --retry-delay 2 --retry-connrefused --retry-max-time 30 https://synthetic.invalid/'
+
++ []
+- ['CALL curl -fsS --connect-timeout 5 --max-time 5 --retry 5 --retry-delay 2 '
+-  '--retry-connrefused --retry-max-time 30 https://synthetic.invalid/']
+
+======================================================================
+FAIL: test_each_failed_health_probe_preserves_failure_and_requires_manual_recovery (__main__.DeployRollbackTests.test_each_failed_health_probe_preserves_failure_and_requires_manual_recovery) (path='/health')
+----------------------------------------------------------------------
+Traceback (most recent call last):
+  File "C:\Users\Connor\.codex\worktrees\release-readiness-discovery\Job tracker\scripts\test-deploy-rollback.py", line 72, in test_each_failed_health_probe_preserves_failure_and_requires_manual_recovery
+    self.assertEqual(self.probes()[:index + 1], calls[1:-2])
+AssertionError: Lists differ: ['CALL curl -fsS --connect-timeout 5 --max[247 chars]lth'] != []
+
+First list contains 2 additional elements.
+First extra element 0:
+'CALL curl -fsS --connect-timeout 5 --max-time 5 --retry 5 --retry-delay 2 --retry-connrefused --retry-max-time 30 https://synthetic.invalid/'
+
++ []
+- ['CALL curl -fsS --connect-timeout 5 --max-time 5 --retry 5 --retry-delay 2 '
+-  '--retry-connrefused --retry-max-time 30 https://synthetic.invalid/',
+-  'CALL curl -fsS --connect-timeout 5 --max-time 5 --retry 5 --retry-delay 2 '
+-  '--retry-connrefused --retry-max-time 30 https://synthetic.invalid/health']
+
+======================================================================
+FAIL: test_each_failed_health_probe_preserves_failure_and_requires_manual_recovery (__main__.DeployRollbackTests.test_each_failed_health_probe_preserves_failure_and_requires_manual_recovery) (path='/ready')
+----------------------------------------------------------------------
+Traceback (most recent call last):
+  File "C:\Users\Connor\.codex\worktrees\release-readiness-discovery\Job tracker\scripts\test-deploy-rollback.py", line 72, in test_each_failed_health_probe_preserves_failure_and_requires_manual_recovery
+    self.assertEqual(self.probes()[:index + 1], calls[1:-2])
+AssertionError: Lists differ: ['CALL curl -fsS --connect-timeout 5 --max[396 chars]ady'] != []
+
+First list contains 3 additional elements.
+First extra element 0:
+'CALL curl -fsS --connect-timeout 5 --max-time 5 --retry 5 --retry-delay 2 --retry-connrefused --retry-max-time 30 https://synthetic.invalid/'
+
++ []
+- ['CALL curl -fsS --connect-timeout 5 --max-time 5 --retry 5 --retry-delay 2 '
+-  '--retry-connrefused --retry-max-time 30 https://synthetic.invalid/',
+-  'CALL curl -fsS --connect-timeout 5 --max-time 5 --retry 5 --retry-delay 2 '
+-  '--retry-connrefused --retry-max-time 30 https://synthetic.invalid/health',
+-  'CALL curl -fsS --connect-timeout 5 --max-time 5 --retry 5 --retry-delay 2 '
+-  '--retry-connrefused --retry-max-time 30 https://synthetic.invalid/ready']
+
+======================================================================
+FAIL: test_failed_replacement_recreates_only_prior_core_images (__main__.DeployRollbackTests.test_failed_replacement_recreates_only_prior_core_images)
+----------------------------------------------------------------------
+Traceback (most recent call last):
+  File "C:\Users\Connor\.codex\worktrees\release-readiness-discovery\Job tracker\scripts\test-deploy-rollback.py", line 55, in test_failed_replacement_recreates_only_prior_core_images
+    self.assertEqual([
+AssertionError: Lists differ: ['CAL[121 chars]ous]', 'CALL curl -fsS --connect-timeout 5 --m[398 chars]ady'] != ['CAL[121 chars]ous]']
+
+First list contains 3 additional elements.
+First extra element 1:
+'CALL curl -fsS --connect-timeout 5 --max-time 5 --retry 5 --retry-delay 2 --retry-connrefused --retry-max-time 30 https://synthetic.invalid/'
+
+Diff is 808 characters long. Set self.maxDiff to None to see it.
+
+----------------------------------------------------------------------
+Ran 6 tests in 0.420s
+
+FAILED (failures=4)

docs/release-readiness/evidence/rollback-health/SYNTAX.txt

@@ -0,0 +1 @@
+Git Bash bash -n deploy/deploy.sh exited 0. Full script was not executed.

docs/release-readiness/evidence/rollback-health/TESTS.txt

@@ -0,0 +1,5 @@
+......
+----------------------------------------------------------------------
+Ran 6 tests in 0.374s
+
+OK

scripts/test-deploy-rollback.py

@@ -10,12 +10,14 @@ BASH = 'C:/Program Files/Git/bin/bash.exe' if os.name == 'nt' else shutil.which(
 
 
 class DeployRollbackTests(unittest.TestCase):
-    def exit_deploy(self, status, *, replaced=True, backend=True, frontend=True, compose_status=0):
+    def exit_deploy(self, status, *, replaced=True, backend=True, frontend=True, compose_status=0, failed_probe='', diagnostics_status=0):
         source = (ROOT / 'deploy/deploy.sh').read_text(encoding='utf-8')
         start = source.index('finish_deploy() {')
         handler = source[start:source.index('\ntrap finish_deploy EXIT\n', start) + len('\ntrap finish_deploy EXIT\n')]
         script = '\n'.join([
             'set -euo pipefail',
+            'exec 3>&1',
+            'APP_PUBLIC_BASE_URL=https://synthetic.invalid/',
             f'replacement_started={str(replaced).lower()}',
             'previous_backend_ref=' + ('synthetic/backend:previous' if backend else ''),
             'previous_frontend_ref=' + ('synthetic/frontend:previous' if frontend else ''),
@@ -23,11 +25,15 @@ class DeployRollbackTests(unittest.TestCase):
             'auth_config_headers_file=synthetic-headers',
             'rm() { printf "CALL rm %s\\n" "$*"; }',
             'docker() { echo "CALL unexpected docker"; return 99; }',
-            'curl() { echo "CALL unexpected curl"; return 99; }',
+            'curl() {',
+            '  printf "CALL curl %s\\n" "$*" >&3;',
+            f'  if [ "${{@: -1}}" = "https://synthetic.invalid{failed_probe}" ]; then return 22; fi;',
+            '  return 0;',
+            '}',
             'compose() {',
             '  printf "CALL compose %s [%s] [%s]\\n" "$*" "${BACKEND_IMAGE:-}" "${FRONTEND_IMAGE:-}";',
             f'  if [ "$1" = up ]; then return {compose_status}; fi;',
-            '  return 0;',
+            f'  return {diagnostics_status};',
             '}',
             handler,
             f'exit {status}',
@@ -49,11 +55,28 @@ class DeployRollbackTests(unittest.TestCase):
         self.assertEqual([
             'CALL compose up -d --force-recreate --remove-orphans backend frontend '
             '[synthetic/backend:previous] [synthetic/frontend:previous]',
+            *self.probes(),
         ], calls)
+        self.assertIn('public core checks passed', stdout)
         self.assertIn('Deployment remains failed', stdout)
 
+    @staticmethod
+    def probes():
+        flags = '-fsS --connect-timeout 5 --max-time 5 --retry 5 --retry-delay 2 --retry-connrefused --retry-max-time 30'
+        return [f'CALL curl {flags} https://synthetic.invalid{path}' for path in ['/', '/health', '/ready']]
+
+    def test_each_failed_health_probe_preserves_failure_and_requires_manual_recovery(self):
+        for index, path in enumerate(['/', '/health', '/ready']):
+            with self.subTest(path=path):
+                stdout, calls = self.exit_deploy(41, failed_probe=path, diagnostics_status=17)
+                self.assertEqual(self.probes()[:index + 1], calls[1:-2])
+                self.assertEqual(['CALL compose ps [] []', 'CALL compose logs --tail=200 backend frontend [] []'], calls[-2:])
+                self.assertIn('rollback health verification failed', stdout)
+                self.assertIn('Manual recovery is required', stdout)
+                self.assertNotIn('public core checks passed', stdout)
+
     def test_rollback_failure_retains_original_status_and_collects_diagnostics(self):
-        stdout, calls = self.exit_deploy(41, compose_status=19)
+        stdout, calls = self.exit_deploy(41, compose_status=19, diagnostics_status=17)
         self.assertEqual([
             'CALL compose up -d --force-recreate --remove-orphans backend frontend '
             '[synthetic/backend:previous] [synthetic/frontend:previous]',