cesnimda
37 lines
5.8 KiB
Markdown
37 lines
5.8 KiB
Markdown
# Recovery implementation review and rehearsal plan
|
|
|
|
2026-11-02; source `5cb903b989742b7a96cb22ddf4c0e24993b18980`. Read-only discovery. The entire 765-line `deploy/deploy.sh` was inspected, never executed or sourced. No operational keys, credentials, environment files, databases, containers or remote storage were accessed.
|
|
|
|
## Distinct recovery mechanisms
|
|
|
|
| Owner | Verified source behavior | Qualification boundary |
|
|
| --- | --- | --- |
|
|
| `SqliteDatabaseBackupRunner` | SQLite-only VACUUM snapshot and count retention; explicitly unsupported for MariaDB | Does not prove MariaDB backups exist |
|
|
| `BackupController` | Authenticated, encrypted user-data export using application Data Protection; includes structured records and attachment metadata | Not a server-wide database/files/keys recovery set; do not use it as disaster recovery proof |
|
|
| `deploy/deploy.sh:backup_database` | Requires explicit provider; SQL dump for MariaDB, volume archive for SQLite; rejects missing/empty/malformed/trailerless dumps | Format checks do not restore or compare records; emergency skip exists and is not approved here |
|
|
| `backup_recovery_volumes` | MariaDB additionally archives required application data volume; independently archives tombstones when present | Consistency between dump/files/tombstones and current key availability still needs isolated rehearsal |
|
|
| Retention/off-host copy | Daily/monthly retention and optional rclone copy | Source message calls remote encrypted but source does not establish actual remote encryption configuration; off-host configuration and recovery remain unverified |
|
|
| `finish_deploy` | On failure after replacement, restarts previous backend/frontend image references when both exist; retains failed exit | No database reversal, no previous AI image restoration, and no post-rollback health check in this handler |
|
|
| Tombstone store and replay | Validates ledger records; Program invokes `StageRestoredAccountsAsync` at startup; deleted identities are staged again | Test restored data with latest independent tombstones; corrupt ledger must fail closed. Existing AccountDeletionTests include restore replay, not a complete operational restore |
|
|
|
|
Build retry paths prune Docker build/system cache and deployment may warm an Ollama model. Therefore even a proposed "local deploy smoke" must not run this script against the shared workstation. No such action was taken.
|
|
|
|
## RR-11: backup connection parser mismatch - reproduced
|
|
|
|
The script's host `sed` expression accepts `Server=` but returns no host for `SERVER=` and `Host=`. The latter is a documented MySqlConnector alias ([official connection options](https://mysqlconnector.net/connection-options/), checked 2026-11-02). The script then fails its required-host check before dump execution. Impact: a valid application connection spelling can prevent deployment backup/deployment; no observed data loss or actual production impact. Engineering priority proposed P2; not a security vulnerability rating.
|
|
|
|
Three synthetic cases used only loopback and dummy identifiers. Exact source expression was extracted from the `db_host` assignment, then executed through Git Bash with `echo <shell-quoted synthetic input> | sed -n <shell-quoted source expression>`. Results: canonical `Server` -> `127.0.0.1`; uppercase `SERVER` -> empty; alias `Host` -> empty. Assertions confirmed all three results. No connection was opened. Initial direct Windows-to-sed invocations failed the positive-control assertion and were discarded as invalid harness evidence; the shell-native rerun passed its positive control.
|
|
|
|
Proposed future remediation: parse with the same supported connection-string semantics as the API, or explicitly validate/document a narrower deploy contract before any operational step. Cover aliases/case, quoted separators, duplicate keys, missing required values and secret-redaction behavior. Do not print real input. Any implementation belongs to an authorised release remediation scope, not P10 cleanup.
|
|
|
|
## Isolated rehearsal specification - NOT RUN
|
|
|
|
1. Bind accepted candidate, image digests, migration history and disposable host ownership. Use synthetic accounts and a separate temporary MariaDB/files/key/tombstone set; never copy production data here without separate authority.
|
|
2. Seed two owners with canonical profiles, CV overrides, letter history, submitted packages, attachments, encrypted provider placeholders and completed deletion tombstones. Record independent counts and artifact hashes, without recording plaintext secrets.
|
|
3. Produce the actual supported dump/files/tombstone recovery set. Demonstrate a failed dump or missing required volume prevents promotion. Verify off-host retrieval/decryption separately if that integration is approved.
|
|
4. Restore into a second isolated environment. Independently compare rows/ownership/revisions/artifact bytes; exercise application decryption and document reopening. Restore an older database with newer tombstones and prove erased accounts cannot regain access. Inject a malformed ledger and assert fail-closed startup.
|
|
5. Rehearse prior-image rollback after failed replacement, then after a representative additive migration and writes. Check `/health`, `/ready`, exact version and authenticated read/write preservation. Document when image rollback is insufficient and database restore would lose newer writes.
|
|
6. Measure recovery duration and data-loss window against owner-approved RTO/RPO. No target or measurement exists in this checkpoint. Preserve cleanup and evidence identities; no production execution or release approval implied.
|
|
|
|
Fresh checks: Git Bash `bash -n deploy/deploy.sh` PASSED (syntax only); `python scripts/test-mariadb-workflow.py` PASSED 4/4 (source quarantine guards only). These are separate from the earlier 5/5 offline runner tests. Restore, failure injection, hosted isolation, live encryption and rollback health remain NOT TESTED.
|