From cedf33a358e4b8df202c0f7038313bba8e67de61 Mon Sep 17 00:00:00 2001 From: cesnimda Date: Tue, 10 Nov 2026 13:24:30 +0100 Subject: [PATCH] fix(release): stop deployment when off-host backup transfer fails --- deploy/deploy.sh | 12 +++- .../11_BACKUP_RESTORE_ROLLBACK.md | 9 +++ .../13_REMEDIATION_REGISTER.md | 2 +- docs/release-readiness/HANDOFF.md | 2 + scripts/test-deploy-backup.py | 70 ++++++++++++++++++- 5 files changed, 90 insertions(+), 5 deletions(-) diff --git a/deploy/deploy.sh b/deploy/deploy.sh index 8341110d..8c9ab29d 100644 --- a/deploy/deploy.sh +++ b/deploy/deploy.sh @@ -574,14 +574,20 @@ upload_current_backup_set() { remote="${remote%/}" local current_files=("$BACKUP_DIR"/jobtracker-*"-$BACKUP_STAMP".*) file for file in "${current_files[@]}"; do - rclone copyto "$file" "$remote/daily/$(basename "$file")" --immutable + if ! rclone copyto "$file" "$remote/daily/$(basename "$file")" --immutable; then + echo "Daily off-host backup upload failed. Aborting deploy." + return 1 + fi done local month="$(TZ="${BACKUP_TIMEZONE:-Europe/Oslo}" date +%Y-%m)" if [ -d "$BACKUP_DIR/monthly/$month" ]; then - rclone copy "$BACKUP_DIR/monthly/$month" "$remote/monthly/$month" --immutable + if ! rclone copy "$BACKUP_DIR/monthly/$month" "$remote/monthly/$month" --immutable; then + echo "Monthly off-host backup upload failed. Aborting deploy." + return 1 + fi fi - echo "Current recovery set copied to configured encrypted off-host remote." + echo "Current recovery set copied to configured off-host remote." } resolve_existing_web_proxy_subnet diff --git a/docs/release-readiness/11_BACKUP_RESTORE_ROLLBACK.md b/docs/release-readiness/11_BACKUP_RESTORE_ROLLBACK.md index 63f8b50c..2ff88f0b 100644 --- a/docs/release-readiness/11_BACKUP_RESTORE_ROLLBACK.md +++ b/docs/release-readiness/11_BACKUP_RESTORE_ROLLBACK.md @@ -52,3 +52,12 @@ Both auth-config temporary-file cleanup calls occur before the rollback decision - `python scripts/test-deploy-backup.py -v` **PASSED 5/5** and `python scripts/test-mariadb-workflow.py -v` **PASSED 4/4**. Git Bash `bash -n deploy/deploy.sh` passed syntax validation. RR-12 remains open for an authorised isolated rehearsal. After the prior images are recreated, verify exact image identity, `/health`, `/ready`, and authenticated reads and writes against representative migrated and post-write data. A successful Compose call does not establish service health or recover schema, data, or the AI image. RR-13 still needs authorised off-host retrieval/decryption and recovery-set consistency verification. No Docker, network service, database, remote storage, deployment or restore was invoked by these checks. + +## RR-13 off-host upload failure handling + +Reproduced two transfer failures in the deployment gate using the real `upload_current_backup_set` function, synthetic files and a stub `rclone`. The caller uses `! upload_current_backup_set`, so Bash suppresses errexit inside that function. When either daily `copyto` or monthly `copy` returned 23, the old code still printed success and the gate returned 0. The healthy control selected only current-stamp daily files, then the monthly directory. + +The bounded fix explicitly returns failure on either copy error. A daily error stops before subsequent files and the monthly copy. A monthly error stops before the deployment continues. The successful path still uses immutable transfers and no longer claims the configured remote is encrypted. A failed upload may leave some remote files already transferred. + +- Red run: **2 failures and 1 passing control**. Green run: **8/8 backup tests** including the three upload cases. The independent rollback handler and hosted quarantine guard checks passed **5/5** and **4/4**. Bash syntax validation passed. +- Encrypted off-host retrieval, decryption, completeness and consistent restore of database/files/keys/tombstones remain required. No actual transfer, restore, production configuration, keys or backup files were accessed. diff --git a/docs/release-readiness/13_REMEDIATION_REGISTER.md b/docs/release-readiness/13_REMEDIATION_REGISTER.md index becc4acd..2437e25f 100644 --- a/docs/release-readiness/13_REMEDIATION_REGISTER.md +++ b/docs/release-readiness/13_REMEDIATION_REGISTER.md @@ -16,6 +16,6 @@ Discovery only, 2026-11-02. These are qualification gaps unless a concrete defec | RR-10 | AI quality/activation | Deterministic Docling fixtures are distinct from real-model semantic accuracy. Optional unqualified adapter remains disabled; no AI invocation authorised here | | RR-11 | Backup host parsing rejected supported `Host=` alias and uppercase spelling | FIXED AND OFFLINE-TESTED. See [bounded remediation](RR-11_IMPLEMENTATION.md). Real dump/restore qualification remains open | | RR-12 | Rollback recreates core images without post-rollback health qualification; no schema/data/AI reversal | OFFLINE HANDLER CHECKS PASSED 5/5. See [failure-injection evidence](11_BACKUP_RESTORE_ROLLBACK.md#rr-12-offline-failure-injection). Actual image recovery and health/readiness after migration/writes still require isolated rehearsal | -| RR-13 | Off-host encryption and recovery-set consistency not established by script success messages | NOT TESTED operational qualification. Verify remote encryption, retrieval/decryption, database/files/key/tombstone consistency in approved isolation | +| RR-13 | Off-host encryption and recovery-set consistency need independent proof. A failed transfer previously let the deployment continue | Upload failure gate FIXED AND OFFLINE-TESTED. Actual encryption, retrieval/decryption and database/files/key/tombstone consistency require approved isolation | Continue evidence work against the [source endpoint and environment census](03_ENDPOINT_ENVIRONMENT_CENSUS.md). Actual qualification requires an accepted candidate and isolated environment. Every fix needs minimal scope, targeted verification and the applicable stage approval. diff --git a/docs/release-readiness/HANDOFF.md b/docs/release-readiness/HANDOFF.md index 2d27615a..18120f50 100644 --- a/docs/release-readiness/HANDOFF.md +++ b/docs/release-readiness/HANDOFF.md @@ -2,6 +2,8 @@ RR-12 offline failure injection passed 5/5 against the unchanged deploy exit handler. The evidence covers synthetic prior-image recreation, failed rollback diagnostics, original exit status and conditions that skip rollback. [Recovery report](11_BACKUP_RESTORE_ROLLBACK.md#rr-12-offline-failure-injection) distinguishes this from a real post-migration/post-write rehearsal. RR-13 off-host retrieval/decryption and consistent restore remain open. No production qualification. +RR-13 transfer guard [reproduced and fixed offline](11_BACKUP_RESTORE_ROLLBACK.md#rr-13-off-host-upload-failure-handling). Daily and monthly copy errors stop deployment, and the success log makes no encryption claim. Actual remote encryption and restore remain open. + 10 November: [endpoint and environment census](03_ENDPOINT_ENVIRONMENT_CENSUS.md) maps 41 source route families, anonymous exceptions and failure cases to qualification owners on the backup-parser branch. No runtime qualification. Carry the map into Stage 2 only after the release candidate and environment are accepted. Branch `planning/release-readiness-discovery`, baseline `5cb903b989742b7a96cb22ddf4c0e24993b18980`; documentation only. Stage 1 IN_PROGRESS, not complete. Full authoritative release prompt read. No final GO/NO-GO completion report or deployable release claimed. diff --git a/scripts/test-deploy-backup.py b/scripts/test-deploy-backup.py index 5053be50..8ddcb652 100644 --- a/scripts/test-deploy-backup.py +++ b/scripts/test-deploy-backup.py @@ -1,4 +1,4 @@ -"""Exercise the real backup function with a synthetic dump command; never deploy.""" +"""Exercise deployment backup functions with synthetic commands; never deploy.""" import os from pathlib import Path import shlex @@ -85,5 +85,73 @@ class BackupConnectionTests(unittest.TestCase): self.assertNotIn('dummy-secret', result.stdout + result.stderr) +class BackupUploadTests(unittest.TestCase): + def upload(self, fail_operation=''): + source = (ROOT / 'deploy/deploy.sh').read_text(encoding='utf-8') + start = source.index('upload_current_backup_set() {') + function = source[start:source.index('\n}\n', start) + 3] + gate_start = source.index('if ! apply_backup_retention || ! upload_current_backup_set; then') + gate = source[gate_start:source.index('\nfi\n', gate_start) + 4] + with tempfile.TemporaryDirectory() as temporary: + folder = Path(temporary) + for filename in ['jobtracker-db-synthetic.sql.gz', 'jobtracker-files-synthetic.tar.gz', + 'jobtracker-db-older.sql.gz']: + (folder / filename).touch() + (folder / 'monthly' / '2099-01').mkdir(parents=True) + script = '\n'.join([ + 'set -euo pipefail', + 'BACKUP_DIR=' + shlex.quote(folder.as_posix()), + 'BACKUP_STAMP=synthetic', + 'BACKUP_RCLONE_REMOTE=fixture:encrypted/', + 'FAIL_OPERATION=' + shlex.quote(fail_operation), + 'date() { echo 2099-01; }', + 'apply_backup_retention() { return 0; }', + 'rclone() {', + ' printf "UPLOAD %s %s %s\\n" "$1" "$3" "$4";', + ' if [ "$1" = "$FAIL_OPERATION" ]; then return 23; fi;', + '}', + function, + gate, + 'echo "DEPLOY WOULD CONTINUE"', + ]) + result = subprocess.run([BASH, '--noprofile', '--norc'], input=script, + text=True, encoding='utf-8', capture_output=True, + cwd=ROOT, timeout=10, + env={key: value for key, value in os.environ.items() + if key.upper() in {'PATH', 'SYSTEMROOT', 'WINDIR', 'TEMP', 'TMP'}}) + calls = [line for line in result.stdout.splitlines() if line.startswith('UPLOAD ')] + return result, calls + + def test_uploads_only_current_daily_files_then_monthly_snapshot(self): + result, calls = self.upload() + self.assertEqual(0, result.returncode, result.stdout + result.stderr) + self.assertEqual([ + 'UPLOAD copyto fixture:encrypted/daily/jobtracker-db-synthetic.sql.gz --immutable', + 'UPLOAD copyto fixture:encrypted/daily/jobtracker-files-synthetic.tar.gz --immutable', + 'UPLOAD copy fixture:encrypted/monthly/2099-01 --immutable', + ], calls) + self.assertIn('DEPLOY WOULD CONTINUE', result.stdout) + self.assertIn('Current recovery set copied to configured off-host remote.', result.stdout) + + def test_daily_copy_failure_stops_before_remaining_files_and_monthly_copy(self): + result, calls = self.upload('copyto') + self.assertEqual(1, result.returncode, result.stdout + result.stderr) + self.assertEqual([ + 'UPLOAD copyto fixture:encrypted/daily/jobtracker-db-synthetic.sql.gz --immutable', + ], calls) + self.assertIn('Daily off-host backup upload failed', result.stdout) + self.assertNotIn('DEPLOY WOULD CONTINUE', result.stdout) + self.assertNotIn('Current recovery set copied', result.stdout) + + def test_monthly_copy_failure_blocks_deploy(self): + result, calls = self.upload('copy') + self.assertEqual(1, result.returncode, result.stdout + result.stderr) + self.assertEqual(3, len(calls)) + self.assertEqual('UPLOAD copy fixture:encrypted/monthly/2099-01 --immutable', calls[-1]) + self.assertIn('Monthly off-host backup upload failed', result.stdout) + self.assertNotIn('DEPLOY WOULD CONTINUE', result.stdout) + self.assertNotIn('Current recovery set copied', result.stdout) + + if __name__ == '__main__': unittest.main()