feat(ai): add safe benchmark harness

docs/production/production-ai-rollout-and-rollback.md

@@ -0,0 +1,106 @@
+# Production AI rollout and rollback
+
+Updated: 2026-08-15
+
+Status: plan only. The 2026-08-15 activity was read-only. No backup, model pull, service restart, firewall/network change, deployment, database mutation, or restore was performed.
+
+## Current rollback anchors
+
+- Deployed application commit: `de937d25dc5e`; configured version `157`.
+- The production checkout is dirty because `deploy/deploy.sh` has a mode-only change. Preserve and review that state before any checkout/reset.
+- JobTracker Ollama image digest: `sha256:f1a705f2bd113fb8d15f85f7c217f0dc5f6bebda6b0cc42b82c3ad165ffcb9dc`.
+- Existing model: `qwen2.5:7b`, ID prefix `845dbda0ea48`.
+- Existing model must not be removed during benchmark or rollout.
+- All four current application containers are healthy, restart `unless-stopped`, and had zero restart count at capture.
+
+Environment secret values were deliberately not copied. Recovery still depends on the protected production environment file and data-protection material; their custody has not been verified.
+
+## Hard stop conditions before mutation
+
+1. PR 28 must pass current CI and be approved for the normal deployment path.
+2. Review/preserve the production-only mode change to `deploy/deploy.sh`; never erase it with a blind reset.
+3. Remove JobTracker's host-published 11434 listener and broader network membership, and decide whether the separate shared Ollama listener is an approved private dependency. Verify no Internet/Traefik route and no unintended LAN clients.
+4. Produce a current MariaDB dump, a recoverable owner-file volume snapshot, data-protection-key recovery, protected tombstone storage, and non-secret configuration inventory. Keep tombstones outside any restored application backup.
+5. Restore the complete set into an isolated scratch environment and verify user/application/Career row counts, owned file access, non-ASCII content, key decryption, and tombstone replay. Never restore over live production for rehearsal.
+6. Define RPO, RTO, backup expiry, tombstone retention, legal hold, and account-deletion provider/cache semantics.
+7. Keep account deletion disabled and all previously inert workers off.
+8. Resolve resource limits for the AI sidecar/Ollama or document measured stop thresholds. Current containers have no CPU, memory, or PID caps.
+9. Confirm at least 25% free on the filesystem receiving image/model layers and enough root headroom for build/temp/log growth. Root is currently 83% used; Docker/model data is on the spacious secondary filesystem.
+10. Use only synthetic benchmark inputs. Do not use production CVs, correspondence, job descriptions, or prompts.
+
+If any item fails, stop. Do not work around it by deleting models, pruning unknown Docker data, weakening health checks, exposing ports, or enabling external processing.
+
+## Intended release-branch topology
+
+- Frontend: reachable only through the approved reverse proxy network; no direct host 3000 publication.
+- Backend: internal application/proxy networks; no host port.
+- AI sidecar: only backend plus private AI network; no host port; service token required for non-health endpoints.
+- Ollama: private AI network only when bundled, or a separately approved private endpoint. No all-interface/public host publication.
+- Routing: local-first with external processing globally disabled by default and additionally requiring per-user opt-in when later approved.
+- Workers: durable AI worker remains disabled until queue/privacy/entitlement/notification production checks pass.
+
+The deployed runtime does not yet have this topology or routing contract.
+
+## Backup gate
+
+The observed 21 MariaDB dumps are gzip-valid but old and database-only. Before an approved deployment:
+
+1. create a new out-of-band MariaDB dump through the supported script;
+2. record its UTC time, size, duration, client/server versions, and SHA-256 without recording credentials;
+3. snapshot/copy the complete JobTracker data volume, including attachments, CV artifacts, generated exports, and account exports;
+4. separately secure data-protection keys and the account-deletion tombstone ledger;
+5. record the exact non-secret Compose/image/model digests;
+6. restore all components to an isolated scratch stack and complete the documented verification matrix;
+7. retain the pre-change set until the observation window and rollback decision expire.
+
+Database-only success is not a complete backup.
+
+## Benchmark sequence after approval
+
+1. Capture idle host/container/GPU metrics and `ollama ps`.
+2. Keep `qwen2.5:7b` as the baseline; do not pull multiple candidates together.
+3. Run one synthetic request, then verify loaded model, GPU/CPU split, VRAM, RAM/swap, temperature, power, latency, tokens/second, JSON validity, and logs free of content.
+4. Stop if swap grows materially, memory pressure remains elevated, root free space drops below the agreed threshold, the GPU overheats/throttles, containers become unhealthy, or latency breaches the package threshold.
+5. Run the checked-in synthetic evaluation set at 4K and 8K context. Try 16K only after measured headroom.
+6. Repeat enough times to distinguish warm/cold load and variance. One successful response is not a model decision.
+7. Test queue congestion, timeout, restart, local outage, privacy opt-out, Free/Pro denial, and two-tenant isolation before any rollout.
+8. Record exact model tag, digest, license, quantization, context, options, resource use, quality and failure categories.
+
+Do not change production's selected model merely because it fits in VRAM.
+
+## Staged rollout
+
+1. Deploy configuration/topology/security changes with AI workers and account deletion still disabled.
+2. Verify admin version badge equals the deployed commit and confirm direct ports are closed.
+3. Re-run anonymous/authenticated health and core non-AI smoke checks.
+4. Run one admin-only synthetic local inference while watching CPU/RAM/swap/GPU/container health.
+5. Enable one controlled synthetic queue canary; verify durable operation, notification, cancellation, restart recovery, entitlement and tenant boundaries.
+6. Observe for the agreed period before increasing concurrency above one.
+7. External fallback remains off until explicit privacy/provider approval and separate synthetic verification.
+8. Account deletion remains off until its retention/restore/provider/cache rehearsal is complete.
+
+## Rollback plan
+
+Rollback must use the supported deploy/runbook path, not ad-hoc container deletion.
+
+1. Stop admission of new AI work; leave durable queued records intact.
+2. Request cancellation/drain and wait for the bounded deadline. Do not kill the database.
+3. Set AI worker and external processing gates back to false.
+4. Redeploy the recorded pre-change commit/images through the approved script, preserving the reviewed production script state.
+5. Keep the pre-existing `qwen2.5:7b`; changing application selection does not require deleting candidate model files.
+6. Restore database/files/keys only if application rollback is insufficient and only from the verified complete pre-change set. Apply deletion tombstones before readiness.
+7. Verify container health, direct-port closure, login, application counts, Career/CV, attachments, non-AI behavior, operation reconciliation, and admin version.
+8. Record incident times and sanitized failure categories. Do not copy prompts or user content into the report.
+
+Expected interruption is one controlled application-container recreation plus model cold-load time. Exact duration remains unmeasured and must be captured during the approved rehearsal.
+
+## Outstanding approvals
+
+- network/firewall/container mutation to close Ollama and frontend host ports;
+- complete backup and scratch restore authority;
+- retention/tombstone/legal decisions;
+- model pull and synthetic production benchmark;
+- deployment and worker activation;
+- external-provider fallback and any real-provider checks.
+
+Until those approvals and gates are complete, the correct state is the current read-only evidence plus disabled release-branch features—not a partial production rollout.