diff --git a/docs/release-readiness/11_BACKUP_RESTORE_ROLLBACK.md b/docs/release-readiness/11_BACKUP_RESTORE_ROLLBACK.md index f765c3a5..63f8b50c 100644 --- a/docs/release-readiness/11_BACKUP_RESTORE_ROLLBACK.md +++ b/docs/release-readiness/11_BACKUP_RESTORE_ROLLBACK.md @@ -34,3 +34,21 @@ Proposed future remediation: parse with the same supported connection-string sem 6. Measure recovery duration and data-loss window against owner-approved RTO/RPO. No target or measurement exists in this checkpoint. Preserve cleanup and evidence identities; no production execution or release approval implied. Fresh checks: Git Bash `bash -n deploy/deploy.sh` PASSED (syntax only); `python scripts/test-mariadb-workflow.py` PASSED 4/4 (source quarantine guards only). These are separate from the earlier 5/5 offline runner tests. Restore, failure injection, hosted isolation, live encryption and rollback health remain NOT TESTED. + +## RR-12 offline failure injection + +Tested the unchanged `finish_deploy` function and EXIT trap from baseline `47c6834cee192a5f1f054ddcd500b4d45c48c985`, `deploy/deploy.sh` blob `8341110de5a827698efc6eeb6951f296770562da`. The test extracts only the handler and trap and supplies synthetic image refs and mock Compose and cleanup functions. The full deploy script was never sourced or executed. + +| Case | Observed behavior | +| --- | --- | +| Replacement started, both previous core image references exist | Recreates backend and frontend with those refs only. The failed deployment status is preserved. No health probe follows the successful Compose call. | +| Compose rejects rollback | Attempts status and backend/frontend log collection. The original failure status is preserved, even when the rollback command returns a different status. | +| Either or both previous references are missing | Skips automatic recreation and reports that no complete prior core release is available. | +| Deployment fails before replacement, or succeeds | Skips automatic recreation. | + +Both auth-config temporary-file cleanup calls occur before the rollback decision. The test stubs deletion and checks the exact call sequence. + +- `python scripts/test-deploy-rollback.py -v` **PASSED 5/5**, including three missing-reference combinations. +- `python scripts/test-deploy-backup.py -v` **PASSED 5/5** and `python scripts/test-mariadb-workflow.py -v` **PASSED 4/4**. Git Bash `bash -n deploy/deploy.sh` passed syntax validation. + +RR-12 remains open for an authorised isolated rehearsal. After the prior images are recreated, verify exact image identity, `/health`, `/ready`, and authenticated reads and writes against representative migrated and post-write data. A successful Compose call does not establish service health or recover schema, data, or the AI image. RR-13 still needs authorised off-host retrieval/decryption and recovery-set consistency verification. No Docker, network service, database, remote storage, deployment or restore was invoked by these checks. diff --git a/docs/release-readiness/13_REMEDIATION_REGISTER.md b/docs/release-readiness/13_REMEDIATION_REGISTER.md index 3d7845e9..becc4acd 100644 --- a/docs/release-readiness/13_REMEDIATION_REGISTER.md +++ b/docs/release-readiness/13_REMEDIATION_REGISTER.md @@ -4,7 +4,7 @@ Discovery only, 2026-11-02. These are qualification gaps unless a concrete defec | ID | Historical item / current finding | Status / closure evidence needed | | --- | --- | --- | -| RR-01 | Phase 9/10 and production NO-GO | Phase 10 assessment now independently published, implementation still awaits exact approval. Later roadmap acceptance absent. BLOCKED for final RC | +| RR-01 | Phase 9/10 and production NO-GO | CLEAN-20 implementation and verification are published. Independent Phase 10 acceptance and later roadmap acceptance remain open. BLOCKED for final RC | | RR-02 | Old Linux 11/24 and later S01 failure narratives | Latest handoff records approved S01 36/36 per platform and owner visual acceptance for unchanged development output. Do not reopen these metrics without affected change; physical print/device remain NOT TESTED | | RR-03 | Python advisory/native parser/image qualification | Latest scoped conditional disposition retained; current advisories and production-equivalent native isolation NOT TESTED. No new scan or risk waiver | | RR-04 | Hosted MariaDB/isolation | Source explicitly fails before execution until isolated runner proof. BLOCKED; fresh local 5 guard tests do not close it | @@ -14,8 +14,8 @@ Discovery only, 2026-11-02. These are qualification gaps unless a concrete defec | RR-08 | MySQL TLS / dependency / production identity | Latest handoff retains MYSQL-TLS-01 and OPS-01. No live configuration read; current advisory disposition and deployed equivalence unverified | | RR-09 | Backup/restore and rollback | Existing procedures and historical seeded rehearsal are not a current restore. Need isolated recovery-set completeness, decryption, tombstone behavior, post-write rollback compatibility and agreed RPO/RTO | | RR-10 | AI quality/activation | Deterministic Docling fixtures are distinct from real-model semantic accuracy. Optional unqualified adapter remains disabled; no AI invocation authorised here | -| RR-11 | Backup host parsing rejects supported `Host=` alias; uppercase spelling also unrecognised | REPRODUCED with synthetic input and shell-native positive control. Deployment aborts before backup; proposed P2 compatibility defect. Fix scope awaits authorised release remediation | -| RR-12 | Rollback restores core images without post-rollback health qualification; no schema/data/AI reversal | VERIFIED source limitation, runtime impact NOT TESTED. Requires isolated post-migration/post-write recovery rehearsal; no rollback guarantee | +| RR-11 | Backup host parsing rejected supported `Host=` alias and uppercase spelling | FIXED AND OFFLINE-TESTED. See [bounded remediation](RR-11_IMPLEMENTATION.md). Real dump/restore qualification remains open | +| RR-12 | Rollback recreates core images without post-rollback health qualification; no schema/data/AI reversal | OFFLINE HANDLER CHECKS PASSED 5/5. See [failure-injection evidence](11_BACKUP_RESTORE_ROLLBACK.md#rr-12-offline-failure-injection). Actual image recovery and health/readiness after migration/writes still require isolated rehearsal | | RR-13 | Off-host encryption and recovery-set consistency not established by script success messages | NOT TESTED operational qualification. Verify remote encryption, retrieval/decryption, database/files/key/tombstone consistency in approved isolation | -Next remediation is evidence work: complete the source/endpoint/environment census and resolve historical records against their exact latest evidence. RR-11 now has a bounded future fix proposal in 11_BACKUP_RESTORE_ROLLBACK.md; no runtime implementation is authorised by this discovery checkpoint. Every fix needs minimal scope, targeted verification and the applicable stage approval. +Continue evidence work against the [source endpoint and environment census](03_ENDPOINT_ENVIRONMENT_CENSUS.md). Actual qualification requires an accepted candidate and isolated environment. Every fix needs minimal scope, targeted verification and the applicable stage approval. diff --git a/docs/release-readiness/HANDOFF.md b/docs/release-readiness/HANDOFF.md index 2fc4671c..2d27615a 100644 --- a/docs/release-readiness/HANDOFF.md +++ b/docs/release-readiness/HANDOFF.md @@ -1,5 +1,7 @@ # Release discovery handoff +RR-12 offline failure injection passed 5/5 against the unchanged deploy exit handler. The evidence covers synthetic prior-image recreation, failed rollback diagnostics, original exit status and conditions that skip rollback. [Recovery report](11_BACKUP_RESTORE_ROLLBACK.md#rr-12-offline-failure-injection) distinguishes this from a real post-migration/post-write rehearsal. RR-13 off-host retrieval/decryption and consistent restore remain open. No production qualification. + 10 November: [endpoint and environment census](03_ENDPOINT_ENVIRONMENT_CENSUS.md) maps 41 source route families, anonymous exceptions and failure cases to qualification owners on the backup-parser branch. No runtime qualification. Carry the map into Stage 2 only after the release candidate and environment are accepted. Branch `planning/release-readiness-discovery`, baseline `5cb903b989742b7a96cb22ddf4c0e24993b18980`; documentation only. Stage 1 IN_PROGRESS, not complete. Full authoritative release prompt read. No final GO/NO-GO completion report or deployable release claimed. diff --git a/scripts/test-deploy-rollback.py b/scripts/test-deploy-rollback.py new file mode 100644 index 00000000..b522ef6c --- /dev/null +++ b/scripts/test-deploy-rollback.py @@ -0,0 +1,82 @@ +"""Exercise the deployment exit handler with synthetic image references and no Docker.""" +import os +from pathlib import Path +import shutil +import subprocess +import unittest + +ROOT = Path(__file__).resolve().parents[1] +BASH = 'C:/Program Files/Git/bin/bash.exe' if os.name == 'nt' else shutil.which('bash') + + +class DeployRollbackTests(unittest.TestCase): + def exit_deploy(self, status, *, replaced=True, backend=True, frontend=True, compose_status=0): + source = (ROOT / 'deploy/deploy.sh').read_text(encoding='utf-8') + start = source.index('finish_deploy() {') + handler = source[start:source.index('\ntrap finish_deploy EXIT\n', start) + len('\ntrap finish_deploy EXIT\n')] + script = '\n'.join([ + 'set -euo pipefail', + f'replacement_started={str(replaced).lower()}', + 'previous_backend_ref=' + ('synthetic/backend:previous' if backend else ''), + 'previous_frontend_ref=' + ('synthetic/frontend:previous' if frontend else ''), + 'auth_config_body_file=synthetic-body', + 'auth_config_headers_file=synthetic-headers', + 'rm() { printf "CALL rm %s\\n" "$*"; }', + 'docker() { echo "CALL unexpected docker"; return 99; }', + 'curl() { echo "CALL unexpected curl"; return 99; }', + 'compose() {', + ' printf "CALL compose %s [%s] [%s]\\n" "$*" "${BACKEND_IMAGE:-}" "${FRONTEND_IMAGE:-}";', + f' if [ "$1" = up ]; then return {compose_status}; fi;', + ' return 0;', + '}', + handler, + f'exit {status}', + ]) + # Exclude inherited deployment settings from the fixture. + environment = {key: value for key, value in os.environ.items() + if key.upper() in {'PATH', 'SYSTEMROOT', 'WINDIR', 'TEMP', 'TMP'}} + result = subprocess.run([BASH, '--noprofile', '--norc'], input=script, + text=True, encoding='utf-8', capture_output=True, + cwd=ROOT, env=environment, timeout=10) + self.assertEqual(status, result.returncode, result.stdout + result.stderr) + self.assertEqual('', result.stderr) + calls = [line for line in result.stdout.splitlines() if line.startswith('CALL ')] + self.assertEqual(['CALL rm -f synthetic-body', 'CALL rm -f synthetic-headers'], calls[:2]) + return result.stdout, calls[2:] + + def test_failed_replacement_recreates_only_prior_core_images(self): + stdout, calls = self.exit_deploy(41) + self.assertEqual([ + 'CALL compose up -d --force-recreate --remove-orphans backend frontend ' + '[synthetic/backend:previous] [synthetic/frontend:previous]', + ], calls) + self.assertIn('Deployment remains failed', stdout) + + def test_rollback_failure_retains_original_status_and_collects_diagnostics(self): + stdout, calls = self.exit_deploy(41, compose_status=19) + self.assertEqual([ + 'CALL compose up -d --force-recreate --remove-orphans backend frontend ' + '[synthetic/backend:previous] [synthetic/frontend:previous]', + 'CALL compose ps [] []', + 'CALL compose logs --tail=200 backend frontend [] []', + ], calls) + self.assertIn('Manual recovery is required', stdout) + + def test_incomplete_previous_release_does_not_recreate_any_service(self): + for backend, frontend in [(False, True), (True, False), (False, False)]: + with self.subTest(backend=backend, frontend=frontend): + stdout, calls = self.exit_deploy(41, backend=backend, frontend=frontend) + self.assertEqual([], calls) + self.assertIn('no complete previous core release', stdout) + + def test_failure_before_replacement_does_not_recreate_any_service(self): + _, calls = self.exit_deploy(41, replaced=False) + self.assertEqual([], calls) + + def test_success_does_not_recreate_any_service(self): + _, calls = self.exit_deploy(0) + self.assertEqual([], calls) + + +if __name__ == '__main__': + unittest.main()