test(release): exercise rollback handler offline

This commit is contained in:
cesnimda
2026-11-10 13:21:56 +01:00
parent 47c6834cee
commit e73681008a
4 changed files with 106 additions and 4 deletions
@@ -34,3 +34,21 @@ Proposed future remediation: parse with the same supported connection-string sem
6. Measure recovery duration and data-loss window against owner-approved RTO/RPO. No target or measurement exists in this checkpoint. Preserve cleanup and evidence identities; no production execution or release approval implied.
Fresh checks: Git Bash `bash -n deploy/deploy.sh` PASSED (syntax only); `python scripts/test-mariadb-workflow.py` PASSED 4/4 (source quarantine guards only). These are separate from the earlier 5/5 offline runner tests. Restore, failure injection, hosted isolation, live encryption and rollback health remain NOT TESTED.
## RR-12 offline failure injection
Tested the unchanged `finish_deploy` function and EXIT trap from baseline `47c6834cee192a5f1f054ddcd500b4d45c48c985`, `deploy/deploy.sh` blob `8341110de5a827698efc6eeb6951f296770562da`. The test extracts only the handler and trap and supplies synthetic image refs and mock Compose and cleanup functions. The full deploy script was never sourced or executed.
| Case | Observed behavior |
| --- | --- |
| Replacement started, both previous core image references exist | Recreates backend and frontend with those refs only. The failed deployment status is preserved. No health probe follows the successful Compose call. |
| Compose rejects rollback | Attempts status and backend/frontend log collection. The original failure status is preserved, even when the rollback command returns a different status. |
| Either or both previous references are missing | Skips automatic recreation and reports that no complete prior core release is available. |
| Deployment fails before replacement, or succeeds | Skips automatic recreation. |
Both auth-config temporary-file cleanup calls occur before the rollback decision. The test stubs deletion and checks the exact call sequence.
- `python scripts/test-deploy-rollback.py -v` **PASSED 5/5**, including three missing-reference combinations.
- `python scripts/test-deploy-backup.py -v` **PASSED 5/5** and `python scripts/test-mariadb-workflow.py -v` **PASSED 4/4**. Git Bash `bash -n deploy/deploy.sh` passed syntax validation.
RR-12 remains open for an authorised isolated rehearsal. After the prior images are recreated, verify exact image identity, `/health`, `/ready`, and authenticated reads and writes against representative migrated and post-write data. A successful Compose call does not establish service health or recover schema, data, or the AI image. RR-13 still needs authorised off-host retrieval/decryption and recovery-set consistency verification. No Docker, network service, database, remote storage, deployment or restore was invoked by these checks.
@@ -4,7 +4,7 @@ Discovery only, 2026-11-02. These are qualification gaps unless a concrete defec
| ID | Historical item / current finding | Status / closure evidence needed |
| --- | --- | --- |
| RR-01 | Phase 9/10 and production NO-GO | Phase 10 assessment now independently published, implementation still awaits exact approval. Later roadmap acceptance absent. BLOCKED for final RC |
| RR-01 | Phase 9/10 and production NO-GO | CLEAN-20 implementation and verification are published. Independent Phase 10 acceptance and later roadmap acceptance remain open. BLOCKED for final RC |
| RR-02 | Old Linux 11/24 and later S01 failure narratives | Latest handoff records approved S01 36/36 per platform and owner visual acceptance for unchanged development output. Do not reopen these metrics without affected change; physical print/device remain NOT TESTED |
| RR-03 | Python advisory/native parser/image qualification | Latest scoped conditional disposition retained; current advisories and production-equivalent native isolation NOT TESTED. No new scan or risk waiver |
| RR-04 | Hosted MariaDB/isolation | Source explicitly fails before execution until isolated runner proof. BLOCKED; fresh local 5 guard tests do not close it |
@@ -14,8 +14,8 @@ Discovery only, 2026-11-02. These are qualification gaps unless a concrete defec
| RR-08 | MySQL TLS / dependency / production identity | Latest handoff retains MYSQL-TLS-01 and OPS-01. No live configuration read; current advisory disposition and deployed equivalence unverified |
| RR-09 | Backup/restore and rollback | Existing procedures and historical seeded rehearsal are not a current restore. Need isolated recovery-set completeness, decryption, tombstone behavior, post-write rollback compatibility and agreed RPO/RTO |
| RR-10 | AI quality/activation | Deterministic Docling fixtures are distinct from real-model semantic accuracy. Optional unqualified adapter remains disabled; no AI invocation authorised here |
| RR-11 | Backup host parsing rejects supported `Host=` alias; uppercase spelling also unrecognised | REPRODUCED with synthetic input and shell-native positive control. Deployment aborts before backup; proposed P2 compatibility defect. Fix scope awaits authorised release remediation |
| RR-12 | Rollback restores core images without post-rollback health qualification; no schema/data/AI reversal | VERIFIED source limitation, runtime impact NOT TESTED. Requires isolated post-migration/post-write recovery rehearsal; no rollback guarantee |
| RR-11 | Backup host parsing rejected supported `Host=` alias and uppercase spelling | FIXED AND OFFLINE-TESTED. See [bounded remediation](RR-11_IMPLEMENTATION.md). Real dump/restore qualification remains open |
| RR-12 | Rollback recreates core images without post-rollback health qualification; no schema/data/AI reversal | OFFLINE HANDLER CHECKS PASSED 5/5. See [failure-injection evidence](11_BACKUP_RESTORE_ROLLBACK.md#rr-12-offline-failure-injection). Actual image recovery and health/readiness after migration/writes still require isolated rehearsal |
| RR-13 | Off-host encryption and recovery-set consistency not established by script success messages | NOT TESTED operational qualification. Verify remote encryption, retrieval/decryption, database/files/key/tombstone consistency in approved isolation |
Next remediation is evidence work: complete the source/endpoint/environment census and resolve historical records against their exact latest evidence. RR-11 now has a bounded future fix proposal in 11_BACKUP_RESTORE_ROLLBACK.md; no runtime implementation is authorised by this discovery checkpoint. Every fix needs minimal scope, targeted verification and the applicable stage approval.
Continue evidence work against the [source endpoint and environment census](03_ENDPOINT_ENVIRONMENT_CENSUS.md). Actual qualification requires an accepted candidate and isolated environment. Every fix needs minimal scope, targeted verification and the applicable stage approval.
+2
View File
@@ -1,5 +1,7 @@
# Release discovery handoff
RR-12 offline failure injection passed 5/5 against the unchanged deploy exit handler. The evidence covers synthetic prior-image recreation, failed rollback diagnostics, original exit status and conditions that skip rollback. [Recovery report](11_BACKUP_RESTORE_ROLLBACK.md#rr-12-offline-failure-injection) distinguishes this from a real post-migration/post-write rehearsal. RR-13 off-host retrieval/decryption and consistent restore remain open. No production qualification.
10 November: [endpoint and environment census](03_ENDPOINT_ENVIRONMENT_CENSUS.md) maps 41 source route families, anonymous exceptions and failure cases to qualification owners on the backup-parser branch. No runtime qualification. Carry the map into Stage 2 only after the release candidate and environment are accepted.
Branch `planning/release-readiness-discovery`, baseline `5cb903b989742b7a96cb22ddf4c0e24993b18980`; documentation only. Stage 1 IN_PROGRESS, not complete. Full authoritative release prompt read. No final GO/NO-GO completion report or deployable release claimed.
+82
View File
@@ -0,0 +1,82 @@
"""Exercise the deployment exit handler with synthetic image references and no Docker."""
import os
from pathlib import Path
import shutil
import subprocess
import unittest
ROOT = Path(__file__).resolve().parents[1]
BASH = 'C:/Program Files/Git/bin/bash.exe' if os.name == 'nt' else shutil.which('bash')
class DeployRollbackTests(unittest.TestCase):
def exit_deploy(self, status, *, replaced=True, backend=True, frontend=True, compose_status=0):
source = (ROOT / 'deploy/deploy.sh').read_text(encoding='utf-8')
start = source.index('finish_deploy() {')
handler = source[start:source.index('\ntrap finish_deploy EXIT\n', start) + len('\ntrap finish_deploy EXIT\n')]
script = '\n'.join([
'set -euo pipefail',
f'replacement_started={str(replaced).lower()}',
'previous_backend_ref=' + ('synthetic/backend:previous' if backend else ''),
'previous_frontend_ref=' + ('synthetic/frontend:previous' if frontend else ''),
'auth_config_body_file=synthetic-body',
'auth_config_headers_file=synthetic-headers',
'rm() { printf "CALL rm %s\\n" "$*"; }',
'docker() { echo "CALL unexpected docker"; return 99; }',
'curl() { echo "CALL unexpected curl"; return 99; }',
'compose() {',
' printf "CALL compose %s [%s] [%s]\\n" "$*" "${BACKEND_IMAGE:-}" "${FRONTEND_IMAGE:-}";',
f' if [ "$1" = up ]; then return {compose_status}; fi;',
' return 0;',
'}',
handler,
f'exit {status}',
])
# Exclude inherited deployment settings from the fixture.
environment = {key: value for key, value in os.environ.items()
if key.upper() in {'PATH', 'SYSTEMROOT', 'WINDIR', 'TEMP', 'TMP'}}
result = subprocess.run([BASH, '--noprofile', '--norc'], input=script,
text=True, encoding='utf-8', capture_output=True,
cwd=ROOT, env=environment, timeout=10)
self.assertEqual(status, result.returncode, result.stdout + result.stderr)
self.assertEqual('', result.stderr)
calls = [line for line in result.stdout.splitlines() if line.startswith('CALL ')]
self.assertEqual(['CALL rm -f synthetic-body', 'CALL rm -f synthetic-headers'], calls[:2])
return result.stdout, calls[2:]
def test_failed_replacement_recreates_only_prior_core_images(self):
stdout, calls = self.exit_deploy(41)
self.assertEqual([
'CALL compose up -d --force-recreate --remove-orphans backend frontend '
'[synthetic/backend:previous] [synthetic/frontend:previous]',
], calls)
self.assertIn('Deployment remains failed', stdout)
def test_rollback_failure_retains_original_status_and_collects_diagnostics(self):
stdout, calls = self.exit_deploy(41, compose_status=19)
self.assertEqual([
'CALL compose up -d --force-recreate --remove-orphans backend frontend '
'[synthetic/backend:previous] [synthetic/frontend:previous]',
'CALL compose ps [] []',
'CALL compose logs --tail=200 backend frontend [] []',
], calls)
self.assertIn('Manual recovery is required', stdout)
def test_incomplete_previous_release_does_not_recreate_any_service(self):
for backend, frontend in [(False, True), (True, False), (False, False)]:
with self.subTest(backend=backend, frontend=frontend):
stdout, calls = self.exit_deploy(41, backend=backend, frontend=frontend)
self.assertEqual([], calls)
self.assertIn('no complete previous core release', stdout)
def test_failure_before_replacement_does_not_recreate_any_service(self):
_, calls = self.exit_deploy(41, replaced=False)
self.assertEqual([], calls)
def test_success_does_not_recreate_any_service(self):
_, calls = self.exit_deploy(0)
self.assertEqual([], calls)
if __name__ == '__main__':
unittest.main()