Recovery implementation review and rehearsal plan

2026-11-02; source 5cb903b989742b7a96cb22ddf4c0e24993b18980. Read-only discovery. The entire 765-line deploy/deploy.sh was inspected, never executed or sourced. No operational keys, credentials, environment files, databases, containers or remote storage were accessed.

Distinct recovery mechanisms

Owner	Verified source behavior	Qualification boundary
`SqliteDatabaseBackupRunner`	SQLite-only VACUUM snapshot and count retention; explicitly unsupported for MariaDB	Does not prove MariaDB backups exist
`BackupController`	Authenticated, encrypted user-data export using application Data Protection; includes structured records and attachment metadata	Not a server-wide database/files/keys recovery set; do not use it as disaster recovery proof
`deploy/deploy.sh:backup_database`	Requires explicit provider; SQL dump for MariaDB, volume archive for SQLite; rejects missing/empty/malformed/trailerless dumps	Format checks do not restore or compare records; emergency skip exists and is not approved here
`backup_recovery_volumes`	MariaDB additionally archives required application data volume; independently archives tombstones when present	Consistency between dump/files/tombstones and current key availability still needs isolated rehearsal
Retention/off-host copy	Daily/monthly retention and optional rclone copy	Source message calls remote encrypted but source does not establish actual remote encryption configuration; off-host configuration and recovery remain unverified
`finish_deploy`	On failure after replacement, restarts previous backend/frontend references, checks public frontend/liveness/readiness with bounded retries and retains failed exit	Offline stubs verify control flow. Actual recovery, image identity, database reversal and previous AI image restoration remain unqualified
Tombstone store and replay	Validates ledger records; Program invokes `StageRestoredAccountsAsync` at startup; deleted identities are staged again	Test restored data with latest independent tombstones; corrupt ledger must fail closed. Existing AccountDeletionTests include restore replay, not a complete operational restore

Build retry paths prune Docker build/system cache and deployment may warm an Ollama model. Therefore even a proposed "local deploy smoke" must not run this script against the shared workstation. No such action was taken.

RR-11: backup connection parser mismatch - reproduced

The script's host sed expression accepts Server= but returns no host for SERVER= and Host=. The latter is a documented MySqlConnector alias (official connection options, checked 2026-11-02). The script then fails its required-host check before dump execution. Impact: a valid application connection spelling can prevent deployment backup/deployment; no observed data loss or actual production impact. Engineering priority proposed P2; not a security vulnerability rating.

Three synthetic cases used only loopback and dummy identifiers. Exact source expression was extracted from the db_host assignment, then executed through Git Bash with echo <shell-quoted synthetic input> | sed -n <shell-quoted source expression>. Results: canonical Server -> 127.0.0.1; uppercase SERVER -> empty; alias Host -> empty. Assertions confirmed all three results. No connection was opened. Initial direct Windows-to-sed invocations failed the positive-control assertion and were discarded as invalid harness evidence; the shell-native rerun passed its positive control.

Proposed future remediation: parse with the same supported connection-string semantics as the API, or explicitly validate/document a narrower deploy contract before any operational step. Cover aliases/case, quoted separators, duplicate keys, missing required values and secret-redaction behavior. Do not print real input. Any implementation belongs to an authorised release remediation scope, not P10 cleanup.

Isolated rehearsal specification - NOT RUN

Bind accepted candidate, image digests, migration history and disposable host ownership. Use synthetic accounts and a separate temporary MariaDB/files/key/tombstone set; never copy production data here without separate authority.
Seed two owners with canonical profiles, CV overrides, letter history, submitted packages, attachments, encrypted provider placeholders and completed deletion tombstones. Record independent counts and artifact hashes, without recording plaintext secrets.
Produce the actual supported dump/files/tombstone recovery set. Demonstrate a failed dump or missing required volume prevents promotion. Verify off-host retrieval/decryption separately if that integration is approved.
Restore into a second isolated environment. Independently compare rows/ownership/revisions/artifact bytes; exercise application decryption and document reopening. Restore an older database with newer tombstones and prove erased accounts cannot regain access. Inject a malformed ledger and assert fail-closed startup.
Rehearse prior-image rollback after failed replacement, then after a representative additive migration and writes. Check /health, /ready, exact version and authenticated read/write preservation. Document when image rollback is insufficient and database restore would lose newer writes.
Measure recovery duration and data-loss window against owner-approved RTO/RPO. No target or measurement exists in this checkpoint. Preserve cleanup and evidence identities; no production execution or release approval implied.

Fresh checks: Git Bash bash -n deploy/deploy.sh PASSED (syntax only); python scripts/test-mariadb-workflow.py PASSED 4/4 (source quarantine guards only). These are separate from the earlier 5/5 offline runner tests. Restore, failure injection, hosted isolation, live encryption and rollback health remain NOT TESTED.

RR-12 offline failure injection

Tested the unchanged finish_deploy function and EXIT trap from baseline 47c6834cee192a5f1f054ddcd500b4d45c48c985, deploy/deploy.sh blob 8341110de5a827698efc6eeb6951f296770562da. The test extracts only the handler and trap and supplies synthetic image refs and mock Compose and cleanup functions. The full deploy script was never sourced or executed.

Case	Observed behavior
Replacement started, both previous core image references exist	Recreates backend and frontend with those refs only. The failed deployment status is preserved. No health probe follows the successful Compose call.
Compose rejects rollback	Attempts status and backend/frontend log collection. The original failure status is preserved, even when the rollback command returns a different status.
Either or both previous references are missing	Skips automatic recreation and reports that no complete prior core release is available.
Deployment fails before replacement, or succeeds	Skips automatic recreation.

Both auth-config temporary-file cleanup calls occur before the rollback decision. The test stubs deletion and checks the exact call sequence.

python scripts/test-deploy-rollback.py -v PASSED 5/5, including three missing-reference combinations.
python scripts/test-deploy-backup.py -v PASSED 5/5 and python scripts/test-mariadb-workflow.py -v PASSED 4/4. Git Bash bash -n deploy/deploy.sh passed syntax validation.

RR-12 remains open for an authorised isolated rehearsal. After the prior images are recreated, verify exact image identity, /health, /ready, and authenticated reads and writes against representative migrated and post-write data. A successful Compose call does not establish service health or recover schema, data, or the AI image. RR-13 still needs authorised off-host retrieval/decryption and recovery-set consistency verification. No Docker, network service, database, remote storage, deployment or restore was invoked by these checks.

RR-13 off-host upload failure handling

Reproduced two transfer failures in the deployment gate using the real upload_current_backup_set function, synthetic files and a stub rclone. The caller uses ! upload_current_backup_set, so Bash suppresses errexit inside that function. When either daily copyto or monthly copy returned 23, the old code still printed success and the gate returned 0. The healthy control selected only current-stamp daily files, then the monthly directory.

The bounded fix explicitly returns failure on either copy error. A daily error stops before subsequent files and the monthly copy. A monthly error stops before the deployment continues. The successful path still uses immutable transfers and no longer claims the configured remote is encrypted. A failed upload may leave some remote files already transferred.

Red run: 2 failures and 1 passing control. Green run: 8/8 backup tests including the three upload cases. The independent rollback handler and hosted quarantine guard checks passed 5/5 and 4/4. Bash syntax validation passed.
Encrypted off-host retrieval, decryption, completeness and consistent restore of database/files/keys/tombstones remain required. No actual transfer, restore, production configuration, keys or backup files were accessed.

RR-13 monthly completion marker

The same conditional Bash behavior applies to apply_backup_retention. With a stubbed partial file copy returning 23, the old function wrote .complete, reported the month retained and continued through the mocked pruning, upload and deployment boundary. The successful control retained byte-identical synthetic files and the expected stamp.

The fix explicitly checks preparation and copy. It writes the stamp to a temporary file in the monthly directory and renames it to .complete only after the copy and stamp write succeed. A failed rename blocks later steps and attempts to remove the temporary marker. Existing completed snapshots are preserved. A failed copy can leave unmarked files, and previously written markers still require verification. A marker records the selected stamp and successful local copy; it does not attest to recovery-set integrity.

Red run: 2 failures and 1 passing control against the pre-fix function. The partial-copy case directly reproduced false completion; the marker-publication case required an atomic rename the old function did not perform.
Green run: 12/12 backup tests including four monthly cases (successful copy and stamp, partial copy, failed marker rename, existing completed month). Rollback handler and hosted quarantine checks passed 5/5 and 4/4. Bash syntax validation passed.
Only temporary synthetic files were created or copied. Pruning and upload were stubbed. Approved rehearsal still needs independent checks of artifacts for the marker stamp, encryption, retrieval and actual restore before any release decision.

RR-12 post-rollback core health checks

11 November 2026. Source a39ac847298e1fbe7667cf42671e7ec265500fcd on codex/release-rollback-health, based on 24a62997. After a successful previous-image restart, the handler now checks the configured public frontend, /health and /ready. Each curl request has connection/request timeouts and bounded retries for startup failures. A failed check reports that manual recovery is required and collects status/logs. The deployment's original nonzero status is retained even if diagnostics fail. The success message is limited to restarted core images and passing public core checks.

The red test run executed six methods and reported four assertion failures because the old handler performed no probes. The green run passed six methods, including failed-probe subcases for all three endpoints, Compose failure, incomplete references, pre-replacement failure and successful deployment. Probe failures and Compose failure also use failing diagnostic stubs to verify original-status preservation. Twelve backup tests and shell syntax validation passed separately. Earlier five-test receipts remain historical and overlap this run.

Only the extracted exit handler/trap ran with shell functions substituting Compose, Docker, curl and cleanup. The full deployment script was never sourced or executed, and no network/containers/models were used. Raw red/green/backup/syntax receipts are in evidence/rollback-health/. Actual startup timing, proxy routing, exact image identity, post-migration writes and database/files/keys/tombstone recovery still need the authorised isolated rehearsal. Passing public probes alone does not establish recovery correctness.

RR-13 retention failure propagation

11 November 2026. Based on 3cca441874abfab6ad581c5a07913a6d66622423 on codex/release-rollback-health. Bounded offline remediation under prompt 11, sections 13 and 16. Engineering priority P2: retention failures could falsely report success and permit upload/deployment, leaving retention incomplete; no actual data loss or production impact was observed. No security severity or release-risk acceptance is implied.

The real conditional deployment caller suppresses Bash errexit within apply_backup_retention. Previously a failed daily find -delete or monthly rm was masked by the final success message. Monthly find | sort failures were also hidden by process substitution feeding mapfile. Seven retention test methods reproduced four failing assertions: daily prune, monthly enumeration, sorting and monthly prune all returned success and reached the mocked upload/deployment boundary.

Daily pruning and each monthly removal now explicitly return failure. Monthly enumeration is captured in a checked command substitution under the script's existing pipefail, then split into the same ordered array; an empty listing produces an empty array. Failure prevents later monthly removals, upload and deployment, and suppresses the final retention-success message. Existing daily age/count defaults, monthly path guard, copy/atomic-marker behavior, archive/dump/upload functions and backend error handling remain unchanged. Pruning can already have removed some eligible old files before a command fails; this change does not undo prior removals or prove recoverability.

Fresh evidence, Windows / Python 3.12.10 / Git Bash:

python scripts/test-deploy-backup.py BackupRetentionTests -v: red run 7 methods, 4 assertion failures (monthly enumeration has two failing subcases). Four existing methods passed. An additional empty-list control was added afterward.
python scripts/test-deploy-backup.py -v: final 16/16 methods PASSED, including 8 retention, 5 connection-parser and 3 upload methods. Healthy retention requests removal of the two oldest synthetic months in order, preserves current copied bytes/stamp and an existing completed snapshot, and reaches upload/deployment. Daily failure prevents enumeration; either pipeline failure prevents monthly pruning; first monthly removal failure prevents further removals. Empty-list control makes no removal request. The red and earlier green totals overlap; do not add them.
python scripts/test-deploy-rollback.py -v: 6/6 PASSED independently; original failure statuses, rollback diagnostics and health-check gates retained.
Git Bash bash -n deploy/deploy.sh: PASSED, syntax only. Raw receipts and SHA-256 source identities: retention-failures evidence, alongside red.txt, green-backup.txt, rollback.txt, and syntax.txt.
graphify update .: AST-only update completed, 18,539 nodes / 45,993 edges. Optional SQL extractor missing and unsupported/no-symbol files reported; no dependency installed or semantic/model extraction run. Generated graph remains untracked local navigation material.

Tests execute only extracted functions/caller with synthetic temporary files. Every pruning find, rm and upload operation is a shell stub; sorting uses only mocked listing text. The full deployment script was never executed or sourced. No real backup, deletion, network transfer, containers, database, migration, providers, model, infrastructure repair or production operation was performed. Product/browser/full MariaDB/hosted/security suites were not rerun for this shell-only package. Actual Linux-host pruning, encrypted retrieval/decryption, consistent restore, post-write rollback and agreed RTO/RPO remain unqualified. RR-09/RR-12/RR-13 recovery gates and production NO-GO remain open. Reverting this bounded commit restores the prior failure-masking behavior; no schema or user-data reversal is needed.

16 KiB Raw Blame History