docs(release): record offline rollback health verification

This commit is contained in:
cesnimda
2026-11-11 07:20:46 +01:00
parent a39ac84729
commit 55431b5ad0
7 changed files with 96 additions and 2 deletions
@@ -11,7 +11,7 @@
| `deploy/deploy.sh:backup_database` | Requires explicit provider; SQL dump for MariaDB, volume archive for SQLite; rejects missing/empty/malformed/trailerless dumps | Format checks do not restore or compare records; emergency skip exists and is not approved here |
| `backup_recovery_volumes` | MariaDB additionally archives required application data volume; independently archives tombstones when present | Consistency between dump/files/tombstones and current key availability still needs isolated rehearsal |
| Retention/off-host copy | Daily/monthly retention and optional rclone copy | Source message calls remote encrypted but source does not establish actual remote encryption configuration; off-host configuration and recovery remain unverified |
| `finish_deploy` | On failure after replacement, restarts previous backend/frontend image references when both exist; retains failed exit | No database reversal, no previous AI image restoration, and no post-rollback health check in this handler |
| `finish_deploy` | On failure after replacement, restarts previous backend/frontend references, checks public frontend/liveness/readiness with bounded retries and retains failed exit | Offline stubs verify control flow. Actual recovery, image identity, database reversal and previous AI image restoration remain unqualified |
| Tombstone store and replay | Validates ledger records; Program invokes `StageRestoredAccountsAsync` at startup; deleted identities are staged again | Test restored data with latest independent tombstones; corrupt ledger must fail closed. Existing AccountDeletionTests include restore replay, not a complete operational restore |
Build retry paths prune Docker build/system cache and deployment may warm an Ollama model. Therefore even a proposed "local deploy smoke" must not run this script against the shared workstation. No such action was taken.
@@ -71,3 +71,11 @@ The fix explicitly checks preparation and copy. It writes the stamp to a tempora
- Red run: **2 failures and 1 passing control** against the pre-fix function. The partial-copy case directly reproduced false completion; the marker-publication case required an atomic rename the old function did not perform.
- Green run: **12/12 backup tests** including four monthly cases (successful copy and stamp, partial copy, failed marker rename, existing completed month). Rollback handler and hosted quarantine checks passed **5/5** and **4/4**. Bash syntax validation passed.
- Only temporary synthetic files were created or copied. Pruning and upload were stubbed. Approved rehearsal still needs independent checks of artifacts for the marker stamp, encryption, retrieval and actual restore before any release decision.
## RR-12 post-rollback core health checks
11 November 2026. Source `a39ac847298e1fbe7667cf42671e7ec265500fcd` on `codex/release-rollback-health`, based on `24a62997`. After a successful previous-image restart, the handler now checks the configured public frontend, `/health` and `/ready`. Each curl request has connection/request timeouts and bounded retries for startup failures. A failed check reports that manual recovery is required and collects status/logs. The deployment's original nonzero status is retained even if diagnostics fail. The success message is limited to restarted core images and passing public core checks.
The red test run executed six methods and reported four assertion failures because the old handler performed no probes. The green run passed six methods, including failed-probe subcases for all three endpoints, Compose failure, incomplete references, pre-replacement failure and successful deployment. Probe failures and Compose failure also use failing diagnostic stubs to verify original-status preservation. Twelve backup tests and shell syntax validation passed separately. Earlier five-test receipts remain historical and overlap this run.
Only the extracted exit handler/trap ran with shell functions substituting Compose, Docker, curl and cleanup. The full deployment script was never sourced or executed, and no network/containers/models were used. Raw red/green/backup/syntax receipts are in `evidence/rollback-health/`. Actual startup timing, proxy routing, exact image identity, post-migration writes and database/files/keys/tombstone recovery still need the authorised isolated rehearsal. Passing public probes alone does not establish recovery correctness.