docs(release): document recovery gaps and reproduced parser mismatch

2026-11-02 11:37:38 +01:00
parent edd2aa9ae4
commit 653f8d04d4
3 changed files with 42 additions and 1 deletions
@@ -0,0 +1,36 @@
+# Recovery implementation review and rehearsal plan
+
+2026-11-02; source `5cb903b989742b7a96cb22ddf4c0e24993b18980`. Read-only discovery. The entire 765-line `deploy/deploy.sh` was inspected, never executed or sourced. No operational keys, credentials, environment files, databases, containers or remote storage were accessed.
+
+## Distinct recovery mechanisms
+
+| Owner | Verified source behavior | Qualification boundary |
+| --- | --- | --- |
+| `SqliteDatabaseBackupRunner` | SQLite-only VACUUM snapshot and count retention; explicitly unsupported for MariaDB | Does not prove MariaDB backups exist |
+| `BackupController` | Authenticated, encrypted user-data export using application Data Protection; includes structured records and attachment metadata | Not a server-wide database/files/keys recovery set; do not use it as disaster recovery proof |
+| `deploy/deploy.sh:backup_database` | Requires explicit provider; SQL dump for MariaDB, volume archive for SQLite; rejects missing/empty/malformed/trailerless dumps | Format checks do not restore or compare records; emergency skip exists and is not approved here |
+| `backup_recovery_volumes` | MariaDB additionally archives required application data volume; independently archives tombstones when present | Consistency between dump/files/tombstones and current key availability still needs isolated rehearsal |
+| Retention/off-host copy | Daily/monthly retention and optional rclone copy | Source message calls remote encrypted but source does not establish actual remote encryption configuration; off-host configuration and recovery remain unverified |
+| `finish_deploy` | On failure after replacement, restarts previous backend/frontend image references when both exist; retains failed exit | No database reversal, no previous AI image restoration, and no post-rollback health check in this handler |
+| Tombstone store and replay | Validates ledger records; Program invokes `StageRestoredAccountsAsync` at startup; deleted identities are staged again | Test restored data with latest independent tombstones; corrupt ledger must fail closed. Existing AccountDeletionTests include restore replay, not a complete operational restore |
+
+Build retry paths prune Docker build/system cache and deployment may warm an Ollama model. Therefore even a proposed "local deploy smoke" must not run this script against the shared workstation. No such action was taken.
+
+## RR-11: backup connection parser mismatch - reproduced
+
+The script's host `sed` expression accepts `Server=` but returns no host for `SERVER=` and `Host=`. The latter is a documented MySqlConnector alias ([official connection options](https://mysqlconnector.net/connection-options/), checked 2026-11-02). The script then fails its required-host check before dump execution. Impact: a valid application connection spelling can prevent deployment backup/deployment; no observed data loss or actual production impact. Engineering priority proposed P2; not a security vulnerability rating.
+
+Three synthetic cases used only loopback and dummy identifiers. Exact source expression was extracted from the `db_host` assignment, then executed through Git Bash with `echo <shell-quoted synthetic input> | sed -n <shell-quoted source expression>`. Results: canonical `Server` -> `127.0.0.1`; uppercase `SERVER` -> empty; alias `Host` -> empty. Assertions confirmed all three results. No connection was opened. Initial direct Windows-to-sed invocations failed the positive-control assertion and were discarded as invalid harness evidence; the shell-native rerun passed its positive control.
+
+Proposed future remediation: parse with the same supported connection-string semantics as the API, or explicitly validate/document a narrower deploy contract before any operational step. Cover aliases/case, quoted separators, duplicate keys, missing required values and secret-redaction behavior. Do not print real input. Any implementation belongs to an authorised release remediation scope, not P10 cleanup.
+
+## Isolated rehearsal specification - NOT RUN
+
+1. Bind accepted candidate, image digests, migration history and disposable host ownership. Use synthetic accounts and a separate temporary MariaDB/files/key/tombstone set; never copy production data here without separate authority.
+2. Seed two owners with canonical profiles, CV overrides, letter history, submitted packages, attachments, encrypted provider placeholders and completed deletion tombstones. Record independent counts and artifact hashes, without recording plaintext secrets.
+3. Produce the actual supported dump/files/tombstone recovery set. Demonstrate a failed dump or missing required volume prevents promotion. Verify off-host retrieval/decryption separately if that integration is approved.
+4. Restore into a second isolated environment. Independently compare rows/ownership/revisions/artifact bytes; exercise application decryption and document reopening. Restore an older database with newer tombstones and prove erased accounts cannot regain access. Inject a malformed ledger and assert fail-closed startup.
+5. Rehearse prior-image rollback after failed replacement, then after a representative additive migration and writes. Check `/health`, `/ready`, exact version and authenticated read/write preservation. Document when image rollback is insufficient and database restore would lose newer writes.
+6. Measure recovery duration and data-loss window against owner-approved RTO/RPO. No target or measurement exists in this checkpoint. Preserve cleanup and evidence identities; no production execution or release approval implied.
+
+Fresh checks: Git Bash `bash -n deploy/deploy.sh` PASSED (syntax only); `python scripts/test-mariadb-workflow.py` PASSED 4/4 (source quarantine guards only). These are separate from the earlier 5/5 offline runner tests. Restore, failure injection, hosted isolation, live encryption and rollback health remain NOT TESTED.
@@ -14,5 +14,8 @@ Discovery only, 2026-11-02. These are qualification gaps unless a concrete defec
 | RR-08 | MySQL TLS / dependency / production identity | Latest handoff retains MYSQL-TLS-01 and OPS-01. No live configuration read; current advisory disposition and deployed equivalence unverified |
 | RR-09 | Backup/restore and rollback | Existing procedures and historical seeded rehearsal are not a current restore. Need isolated recovery-set completeness, decryption, tombstone behavior, post-write rollback compatibility and agreed RPO/RTO |
 | RR-10 | AI quality/activation | Deterministic Docling fixtures are distinct from real-model semantic accuracy. Optional unqualified adapter remains disabled; no AI invocation authorised here |
+| RR-11 | Backup host parsing rejects supported `Host=` alias; uppercase spelling also unrecognised | REPRODUCED with synthetic input and shell-native positive control. Deployment aborts before backup; proposed P2 compatibility defect. Fix scope awaits authorised release remediation |
+| RR-12 | Rollback restores core images without post-rollback health qualification; no schema/data/AI reversal | VERIFIED source limitation, runtime impact NOT TESTED. Requires isolated post-migration/post-write recovery rehearsal; no rollback guarantee |
+| RR-13 | Off-host encryption and recovery-set consistency not established by script success messages | NOT TESTED operational qualification. Verify remote encryption, retrieval/decryption, database/files/key/tombstone consistency in approved isolation |

-Next remediation is evidence work: complete the source/endpoint/environment census and resolve historical records against their exact latest evidence. No runtime fixes are proposed from this preliminary inventory. Any later concrete defect needs reproduction, minimal scope, targeted verification and the applicable stage approval.
+Next remediation is evidence work: complete the source/endpoint/environment census and resolve historical records against their exact latest evidence. RR-11 now has a bounded future fix proposal in 11_BACKUP_RESTORE_ROLLBACK.md; no runtime implementation is authorised by this discovery checkpoint. Every fix needs minimal scope, targeted verification and the applicable stage approval.
@@ -6,4 +6,6 @@ Completed: initial source inventory, historical blocker reconciliation, prelimin

 Next authorised work: complete endpoint and environment census, map each failure boundary to existing test/runbook owners, and inspect remaining backup/restore/deployment implementation without executing it. Then prepare a concrete isolated-rehearsal plan. Final baseline/qualification depends on accepted roadmap source; do not run thousands of unchanged tests to fill this report.

+Continuation: recovery implementation inspection and isolated rehearsal specification are now recorded in 11_BACKUP_RESTORE_ROLLBACK.md. RR-11 parser mismatch reproduced using three synthetic cases; RR-12/13 retain rollback/encryption qualification limits. Four source quarantine guard tests and shell syntax check passed. No deployment script execution. Next: endpoint/environment census, then remaining independent I18N/homepage audit work. Stage 1 remains IN_PROGRESS.
+
 No production credentials/environment files were opened; no infrastructure, models, main branch or autonomous orchestrator changed. The earlier delegated researcher was interrupted by a service limit; parent supplied this partial checkpoint. No background agent remains active.