fix(release): stop deployment when off-host backup transfer fails

This commit is contained in:
cesnimda
2026-11-10 13:24:30 +01:00
parent e73681008a
commit cedf33a358
5 changed files with 90 additions and 5 deletions
@@ -52,3 +52,12 @@ Both auth-config temporary-file cleanup calls occur before the rollback decision
- `python scripts/test-deploy-backup.py -v` **PASSED 5/5** and `python scripts/test-mariadb-workflow.py -v` **PASSED 4/4**. Git Bash `bash -n deploy/deploy.sh` passed syntax validation.
RR-12 remains open for an authorised isolated rehearsal. After the prior images are recreated, verify exact image identity, `/health`, `/ready`, and authenticated reads and writes against representative migrated and post-write data. A successful Compose call does not establish service health or recover schema, data, or the AI image. RR-13 still needs authorised off-host retrieval/decryption and recovery-set consistency verification. No Docker, network service, database, remote storage, deployment or restore was invoked by these checks.
## RR-13 off-host upload failure handling
Reproduced two transfer failures in the deployment gate using the real `upload_current_backup_set` function, synthetic files and a stub `rclone`. The caller uses `! upload_current_backup_set`, so Bash suppresses errexit inside that function. When either daily `copyto` or monthly `copy` returned 23, the old code still printed success and the gate returned 0. The healthy control selected only current-stamp daily files, then the monthly directory.
The bounded fix explicitly returns failure on either copy error. A daily error stops before subsequent files and the monthly copy. A monthly error stops before the deployment continues. The successful path still uses immutable transfers and no longer claims the configured remote is encrypted. A failed upload may leave some remote files already transferred.
- Red run: **2 failures and 1 passing control**. Green run: **8/8 backup tests** including the three upload cases. The independent rollback handler and hosted quarantine guard checks passed **5/5** and **4/4**. Bash syntax validation passed.
- Encrypted off-host retrieval, decryption, completeness and consistent restore of database/files/keys/tombstones remain required. No actual transfer, restore, production configuration, keys or backup files were accessed.
@@ -16,6 +16,6 @@ Discovery only, 2026-11-02. These are qualification gaps unless a concrete defec
| RR-10 | AI quality/activation | Deterministic Docling fixtures are distinct from real-model semantic accuracy. Optional unqualified adapter remains disabled; no AI invocation authorised here |
| RR-11 | Backup host parsing rejected supported `Host=` alias and uppercase spelling | FIXED AND OFFLINE-TESTED. See [bounded remediation](RR-11_IMPLEMENTATION.md). Real dump/restore qualification remains open |
| RR-12 | Rollback recreates core images without post-rollback health qualification; no schema/data/AI reversal | OFFLINE HANDLER CHECKS PASSED 5/5. See [failure-injection evidence](11_BACKUP_RESTORE_ROLLBACK.md#rr-12-offline-failure-injection). Actual image recovery and health/readiness after migration/writes still require isolated rehearsal |
| RR-13 | Off-host encryption and recovery-set consistency not established by script success messages | NOT TESTED operational qualification. Verify remote encryption, retrieval/decryption, database/files/key/tombstone consistency in approved isolation |
| RR-13 | Off-host encryption and recovery-set consistency need independent proof. A failed transfer previously let the deployment continue | Upload failure gate FIXED AND OFFLINE-TESTED. Actual encryption, retrieval/decryption and database/files/key/tombstone consistency require approved isolation |
Continue evidence work against the [source endpoint and environment census](03_ENDPOINT_ENVIRONMENT_CENSUS.md). Actual qualification requires an accepted candidate and isolated environment. Every fix needs minimal scope, targeted verification and the applicable stage approval.
+2
View File
@@ -2,6 +2,8 @@
RR-12 offline failure injection passed 5/5 against the unchanged deploy exit handler. The evidence covers synthetic prior-image recreation, failed rollback diagnostics, original exit status and conditions that skip rollback. [Recovery report](11_BACKUP_RESTORE_ROLLBACK.md#rr-12-offline-failure-injection) distinguishes this from a real post-migration/post-write rehearsal. RR-13 off-host retrieval/decryption and consistent restore remain open. No production qualification.
RR-13 transfer guard [reproduced and fixed offline](11_BACKUP_RESTORE_ROLLBACK.md#rr-13-off-host-upload-failure-handling). Daily and monthly copy errors stop deployment, and the success log makes no encryption claim. Actual remote encryption and restore remain open.
10 November: [endpoint and environment census](03_ENDPOINT_ENVIRONMENT_CENSUS.md) maps 41 source route families, anonymous exceptions and failure cases to qualification owners on the backup-parser branch. No runtime qualification. Carry the map into Stage 2 only after the release candidate and environment are accepted.
Branch `planning/release-readiness-discovery`, baseline `5cb903b989742b7a96cb22ddf4c0e24993b18980`; documentation only. Stage 1 IN_PROGRESS, not complete. Full authoritative release prompt read. No final GO/NO-GO completion report or deployable release claimed.