feat(ai): add safe benchmark harness

docs/work-programmes/decisions.md

@@ -749,3 +749,23 @@
 - **Consequences:** users can download JSON and owned files with independently verifiable SHA-256 checksums. Missing/legacy/external/backup categories are disclosed truthfully. The service becomes the authoritative inventory seam for deletion without coupling export to deletion activation.
 - **User approval required:** No; this is the requested repository-side data lifecycle, using synthetic tests and no production data.
 - **Reversible:** Remove the endpoint/UI and service. Existing downloaded ZIPs remain user-owned files; no stored schema or data changed.
+
+## DEC-076 — Dark-launch account deletion as a durable staged lifecycle
+
+- **Date:** 2026-08-15
+- **Decision:** Replace admin Identity-row deletion with one disabled-by-default coordinator for self-service and admin deletion. Lock the identity immediately, quarantine verified owned files before a transactional explicit row purge, retain retry ledgers, and write a minimal pseudonymous tombstone outside the restored database before completion.
+- **Reason/evidence:** Identity-only deletion leaves owned rows, files, provider credentials, sessions and queued work behind. Cascades cannot coordinate filesystem failures or prevent an older backup from resurrecting the user. Real-SQLite tests prove two-owner isolation, safe quarantine failure, idempotence and restored-account replay.
+- **Alternatives considered:** broad cascade foreign keys; best-effort controller deletes; delete database rows before files; edit old backups in place; enable immediately after local tests. These lose retry/audit boundaries, risk stranded private files or partial erasure, and overclaim backup/provider guarantees.
+- **Consequences:** `AccountLifecycle:DeletionEnabled` remains false until retention, protected tombstone custody, cache/provider handling and a disposable restore rehearsal are approved. Existing sessions are rejected as soon as status becomes pending. Deletion request/file records intentionally survive the user row and the background worker resumes durable requests even while the feature flag blocks new ones.
+- **User approval required:** Production activation and retention decisions only. Repository implementation uses synthetic data and remains inert by default.
+- **Reversible:** Disable requests (the default), allow in-flight reconciliation to finish, then downgrade the additive migration only after no request remains. Never remove the separate tombstone ledger while backups capable of restoring deleted identities still exist.
+
+## DEC-077 — Make production model benchmarking plan-only by default
+
+- **Date:** 2026-08-15
+- **Decision:** Use one standard-library harness over the checked-in synthetic evaluation set. It plans without network access by default, requires `--execute` plus an explicit output for inference, never pulls/deletes models, permits only loopback or an explicitly opted-in literal private IP, and persists hashes/metrics/scores rather than raw inputs/prompts/outputs.
+- **Reason/evidence:** The measured production host can run bounded tests, but its network and backup stop conditions are open and model execution is not authorized. A ready harness removes future ad-hoc prompt/data/report handling without silently widening current authority.
+- **Alternatives considered:** benchmark immediately over SSH; use production CV/email content; install a benchmark dependency/framework; persist raw responses for later scoring; auto-pull every candidate. These violate current authority/privacy boundaries or add avoidable supply-chain/resource risk.
+- **Consequences:** candidate selection remains blocked and no model decision is claimed. Approved runs can compare 4K/8K timing, token rate, JSON/constraint quality and Ollama VRAM metadata reproducibly while host GPU/RAM metrics are captured separately.
+- **User approval required:** Yes before any model pull, production inference, or production report execution. No approval is required for plan-only fixture validation.
+- **Reversible:** Remove the script/tests/template; no dependency, model, application, production, schema or configuration state changed.

docs/work-programmes/master-progress.md

@@ -2,21 +2,21 @@
 
 Updated: 2026-08-15
 
-- **Overall programme status:** Active. Eight packages are locally verified; twenty-four are implemented with verification incomplete; SEC-009 is in progress. The prioritized admin-only version indicator and every immediate repository/browser item are implemented on the release branch; remote and production verification remain.
-- **Current work package:** `SEC-009` — complete readable export and account deletion lifecycle (`IN PROGRESS`). Owner-scoped generated paths and the real-SQLite-verified readable ZIP are complete; proceed with the disabled deletion lifecycle while retention/restore policy blocks production activation.
+- **Overall programme status:** Active but externally blocked. Eight packages are locally verified and twenty-five are implemented with verification incomplete. The prioritized admin-only version indicator, every immediate repository/browser item, SEC-009, the PROD-001 read-only inventory, and the PROD-003 safe benchmark harness are complete on the release branch.
+- **Current work package:** None. Every remaining package now requires a user/operator decision, authorized production mutation/restore/provider action, or explicit package-index access.
 - **Completed work packages:** None are `DONE`; all repository security/AI packages still have applicable browser, provider and/or production gates.
 - **Locally verified work:** SEC-001, SEC-002, SEC-003, SEC-005A, CORE-001, PROD-002, DEP-001 and VER-001 (`VERIFIED LOCALLY`).
-- **Implemented, verification incomplete:** SEC-004, SEC-005B, SEC-008, CORE-002, BG-001, OPS-001A/B/C, POL-001/002, AI-001/002/003/004, UX-001/002/003, QA-001, CAREER-001/002, MAIL-001, JOBS-001/002 and PRODUCT-001 (`IMPLEMENTED — NOT VERIFIED`). Their safe repository/browser scope is implemented; production/native-device/provider gates remain where recorded.
+- **Implemented, verification incomplete:** SEC-004, SEC-005B, SEC-008, SEC-009, CORE-002, BG-001, OPS-001A/B/C, POL-001/002, AI-001/002/003/004, UX-001/002/003, QA-001, CAREER-001/002, MAIL-001, JOBS-001/002 and PRODUCT-001 (`IMPLEMENTED — NOT VERIFIED`). Their safe repository/browser scope is implemented; production/native-device/provider/retention gates remain where recorded.
 - **Production-verified work:** None.
-- **Blocked work:** SEC-006 parser upgrades remain outside the scoped frontend advisory permission; PROD-001/003/004 and REL-001 require documented production access and unfinished dependencies. Real provider, SMTP/MariaDB and production environments are unavailable; DEP-001 awaits approved merge/live verification. The in-app browser is available for local UI checks.
+- **Blocked work:** SEC-006 parser upgrades remain outside the scoped frontend advisory permission; PROD-001/003/004 and REL-001 require network/backup/model/deployment authority and unfinished dependencies. Real provider and live deletion/restore checks remain gated; DEP-001 awaits approved merge/live verification.
 - **Deferred work:** None. Conditional multi-replica coordination, model deletion, realtime operation delivery and unrelated production changes remain outside current packages.
-- **Immediate order:** SEC-009 owner inventory/export first, then its disabled deletion lifecycle. The eight-item immediate queue is complete locally: admin version (`a6cffe0`), Career persistence (`f0b9b22`), CV contrast (`3b86ea2`), JOBS-002 (`deed948`), accessibility (`a7c2549`), PRODUCT-001 (`a25c31b`), VER-001 and tracking reconciliation. External-only work remains skipped, not allowed to stall repository progress.
-- **Status counts:** 8 `VERIFIED LOCALLY`; 24 `IMPLEMENTED — NOT VERIFIED`; 1 `IN PROGRESS`; 1 `NOT STARTED`; 5 `BLOCKED`; 0 `DONE`; 0 `DEFERRED`.
-- **Test status:** backend 650/650; frontend 58/58 suites and 234/234 tests; AI sidecar 22/22; optimized production build/TypeScript; Docker Compose config; safe-failure deployment preflight; and the preceding Playwright 9/9 pass. A fresh Chromium Free account now also downloads the real readable ZIP response. npm audit 0 evidence remains current because the lockfile did not change. Historical JT-019 and Jest force-exit/open-handle behavior remain recorded.
+- **Immediate order:** the eight-item immediate queue is complete locally: admin version (`a6cffe0`), Career persistence (`f0b9b22`), CV contrast (`3b86ea2`), JOBS-002 (`deed948`), accessibility (`a7c2549`), PRODUCT-001 (`a25c31b`), VER-001 and tracking reconciliation. SEC-009 is complete at `842e793`; PROD-001 read-only evidence and the PROD-003 plan-only harness are complete pending this documentation commit. No further independent implementation remains.
+- **Status counts:** 8 `VERIFIED LOCALLY`; 25 `IMPLEMENTED — NOT VERIFIED`; 0 `IN PROGRESS`; 1 `NOT STARTED`; 5 `BLOCKED`; 0 `DONE`; 0 `DEFERRED`.
+- **Test status:** backend 657/657; frontend 58/58 suites and 237/237 tests; AI sidecar 22/22; Ollama benchmark harness 4/4 plus safe dry-run; optimized production build/TypeScript; EF model parity and MariaDB migration-script generation; Docker Compose config; safe-failure deployment preflight; and Playwright 9/9. Five real-SQLite deletion tests cover lockout, isolation, quarantine failure and restored-backup replay. npm audit 0 evidence remains current because the lockfile did not change. Historical JT-019 and Jest force-exit/open-handle behavior remain recorded.
 - **Deployment status:** Gitea pull-request run 609 passed the complete CI job in 4m20s. Deploy was intentionally skipped because the workflow deploys only a `push` to `main`; live remains unchanged. No merge/deployment was performed directly, no production migrations were run and the AI operation worker remains disabled by default.
-- **Production status:** Unchanged and unverified. No provider/model call, model pull, external request or paid API occurred.
+- **Production status:** State unchanged. Sanitized read-only SSH inventory was performed; no logs, prompts, private rows/content or secret values were read, and no provider/model call, model pull, service restart, file/config change, backup, restore, migration or deployment occurred. It confirmed all-interface Ollama/frontend listeners and stale database-only backups as rollout blockers.
 - **Known regressions:** None found by automated/local browser checks. Jest still needs `--forceExit` and reports its existing open-handle notice. Email-provider/send tests are fake/local only; real delivery is not claimed. Current MAIL browser evidence is 1280×720 only because the browser surface could not resize or perform native Tab traversal. Interrupted attempts are aged after 15 minutes and notified without retry; the five-minute scan is unmeasured on a large ledger. Direct clean EF-only SQLite migration still hits the pre-existing historical blank-chain defect before later migrations; normal startup owns reconciliation. Cross-feature monthly AI usage accounting remains a rollout gap.
-- **Outstanding security findings:** JT-001 repository ownership remains High deployment risk until migration/inventory/provider checks; production portion of JT-002; JT-006/JT-009 and associated JT-011/JT-012/JT-022 prerequisites. JT-005 foundations are implemented; AI worker activation awaits controlled rollout. JT-007/JT-008/JT-010 lack browser/provider/production verification.
+- **Outstanding security findings:** JT-001 repository ownership remains High deployment risk until migration/inventory/provider checks; production portion of JT-002; JT-006 and SEC-009 production retention/restore plus JT-011/JT-012/JT-022 prerequisites. JT-005 foundations are implemented; AI worker activation awaits controlled rollout. Production still exposes ports contrary to the release-branch contract, and JT-007/JT-008/JT-010 lack provider/production verification.
 
 ## Current evidence
 
@@ -31,6 +31,7 @@ Updated: 2026-08-15
 - `docs/verification/core-001-sqlite-provider-parity.md`
 - `docs/verification/core-002-route-uniqueness.md`
 - `docs/verification/sec-008-attachment-consistency.md`
+- `docs/verification/sec-009-account-lifecycle.md`
 - `docs/verification/bg-001-tenant-workers.md`
 - `docs/verification/ops-001a-durable-operations.md`
 - `docs/verification/ops-001b-notifications.md`
@@ -52,4 +53,7 @@ Updated: 2026-08-15
 - `docs/verification/product-001-honest-plans.md`
 - `docs/verification/ver-001-complete-regression.md`
 - `docs/verification/prod-002-ai-evaluation.md`
+- `docs/production/production-ai-hardware-assessment.md`
+- `docs/production/production-ai-rollout-and-rollback.md`
+- `docs/production/ollama-model-benchmark.md`
 - `docs/work-programmes/master-work-plan.md`

docs/work-programmes/master-work-plan.md

@@ -16,7 +16,7 @@ Allowed statuses are `NOT STARTED`, `IN PROGRESS`, `IMPLEMENTED — NOT VERIFIED
 
 `DONE` requires every applicable acceptance criterion, focused and regression tests, browser/accessibility/theme/mobile checks, tenant and entitlement checks, documentation, migration/rollback evidence, and production verification. Repository-only work that still requires production is at most `VERIFIED LOCALLY`.
 
-Exactly one implementation item may be `IN PROGRESS`. As of this revision it is **SEC-009**.
+At most one implementation item may be `IN PROGRESS`. No package is currently in progress: every remaining package is blocked by an external decision, production mutation, provider interaction, or explicitly unapproved package-index access.
 
 ## Consolidated dependency order
 
@@ -268,11 +268,11 @@ This queue records the highest-value work that can proceed without production cr
 - **Required tests:** two users/every entity, manifest/checksums/redaction, fault/restart/idempotence, provider failures, disposable restore replay.
 - **Required browser verification:** disposable self/admin export/delete and confirmations.
 - **Required production verification:** backup retention/tombstone rehearsal before self-service enablement.
-- **Status:** `IN PROGRESS`.
+- **Status:** `IMPLEMENTED — NOT VERIFIED`.
 - **Blocker:** legal/operator retention and production restore decisions block activation, not the repository-side disabled/dark launch.
-- **Evidence:** audit JT-009 inventory/design; `docs/verification/sec-009-account-lifecycle.md`; V-173/V-174. Owner-scoped generated storage and complete redacted readable ZIP pass real-SQLite two-owner, focused API/UI, full backend/frontend, build and Chromium checks.
-- **Commit:** none.
-- **Remaining work:** reuse the completed inventory for the additive disabled deletion coordinator, tombstone replay, UI and failure/restart verification. Production activation remains blocked by retention/restore policy.
+- **Evidence:** audit JT-009 inventory/design; `docs/verification/sec-009-account-lifecycle.md`; V-173/V-174/V-175. Owner-scoped generated storage, complete redacted readable ZIP, immediate lockout, transactional owner-isolated row/file purge, retry, fail-closed tombstones and restored-backup replay pass real-SQLite, focused API/UI, full backend/frontend, build and Chromium checks.
+- **Commit:** `842e793`.
+- **Remaining work:** repository scope is complete. Production activation remains blocked by retention/legal decisions, protected tombstone custody, restored-backup rehearsal, provider/cache semantics and staged disposable-account rollout.
 
 ### CORE-001 — Restore default SQLite/MariaDB behavior parity
 
@@ -467,10 +467,10 @@ This queue records the highest-value work that can proceed without production cr
 - **Required browser verification:** none.
 - **Required production verification:** this item is itself production read-only verification.
 - **Status:** `BLOCKED`.
-- **Blocker:** repository docs state the local environment has no production route and CI SSH secrets are unavailable; no documented callable host/credential is present.
-- **Evidence:** `docs/deployment/backup-restore.md` and `docs/operations/production-backup-verification.md` explicitly record the access gap.
+- **Blocker:** read-only inventory is complete, but Ollama/frontend host ports are published on all interfaces, complete current backups/restores are unproved, the production checkout has an unreviewed mode-only deploy-script change, and closing those gaps requires approved production mutations.
+- **Evidence:** `docs/production/production-ai-hardware-assessment.md`; `docs/production/production-ai-rollout-and-rollback.md`; V-176. Sanitized read-only SSH measured hardware, GPU, storage, Docker, networks, selected provider/model, health, limits and backup presence without reading secrets/logs/content or changing state.
 - **Commit:** none.
-- **Remaining work:** create sanitized report template locally; operator/documented access required for measured completion.
+- **Remaining work:** approve and close the network/port, complete-backup/scratch-restore, dirty-checkout and retention gates; then repeat the sanitized inventory to verify the safe target state.
 
 ### PROD-002 — AI workload inventory and synthetic evaluation set
 
@@ -503,10 +503,10 @@ This queue records the highest-value work that can proceed without production cr
 - **Required browser verification:** none.
 - **Required production verification:** measured on actual machine; local workstation results are labeled separately.
 - **Status:** `BLOCKED`.
-- **Blocker:** PROD-001 production access/hardware inventory.
-- **Evidence:** none yet.
+- **Blocker:** production model pull/inference is not authorized, and PROD-001's all-interface Ollama plus incomplete backup/restore stop conditions remain open.
+- **Evidence:** `scripts/run-ollama-evaluation.py`; `scripts/test-ollama-evaluation.py`; `docs/production/ollama-model-benchmark.md`; V-177. Plan-only harness tests 4/4 and validates an eight-request 4K/8K Strategy plan without a model call.
 - **Commit:** none.
-- **Remaining work:** repository harness can be prepared after PROD-002.
+- **Remaining work:** after approvals and PROD-001 safety closure, verify candidate metadata/licenses, pull one candidate at a time, execute repeated synthetic 4K/8K benchmarks, measure active GPU/RAM/offload and complete the evidence-based model decision.
 
 ### PROD-004 — Local-model rollout, fallback, observability and operations
 

docs/work-programmes/session-handoff.md

@@ -2,17 +2,17 @@
 
 Updated: 2026-08-15
 
-- **Exact current task:** begin SEC-009 with the owner inventory/readable export, then implement the deletion lifecycle behind a disabled production gate.
-- **Last completed step:** implemented a recent-authenticated, rate-limited, redacted readable account ZIP with complete safe row/file inventory, checksums, warnings and Settings UX.
-- **Files currently modified:** account export service/API/rate policy, Backup Settings UI/translations, real-SQLite/backend/frontend/browser tests and SEC-009 evidence.
-- **Commands already run:** readable-export backend/API 11/11, backend 650/650, frontend focused 4/4 and full 234/234, backend/frontend builds and targeted Chromium ZIP response. The preceding VER-001 sidecar/Compose/preflight/full Playwright evidence remains current.
-- **Test results:** all listed local gates pass. Provider/native-AT/production cells remain explicitly partial, not run or blocked. Jest retains the documented force-exit/open-handle notice.
+- **Exact current task:** no independent implementation remains. SEC-009, PROD-001 read-only inventory/reporting and the PROD-003 plan-only benchmark harness are complete; continue only after a recorded blocker is authorized/resolved.
+- **Last completed step:** measured production hardware/runtime/backups read-only, documented rollout stops, then added a synthetic-only benchmark harness that defaults to no network execution.
+- **Files currently modified:** master progress/work-plan/handoff/decisions/blockers/evidence; SEC-009 verification; production hardware/rollout/benchmark reports; production backup checkpoint; Ollama benchmark script/tests.
+- **Commands already run:** SEC-009 focused backend 21/21, backend 657/657, frontend focused 8/8 and full 237/237, optimized build, EF parity, MariaDB script generation and Chromium 9/9; benchmark harness 4/4 plus plan-only dry run; sanitized read-only SSH inventory and gzip integrity across 21 existing dumps.
+- **Test results:** all repository gates pass. PROD-001 is PASS/PARTIAL because measured all-interface ports and incomplete/stale backup/restore evidence fail its safety acceptance. No model inference was run. Jest retains the documented open-handle notice.
 - **Services currently running:** none on task-owned ports 3000/5202. Playwright stopped its disposable API/Next servers. Pre-existing Docker services were not changed.
-- **Temporary files or processes:** no task-owned process is running and the failed disposable migration database was removed. Existing synthetic browser evidence/account and startup-created local backup remain documented. No provider account, real email, private content, paid service or production service was accessed.
-- **Production changes currently active:** none. No deployment, migration, provider connection/sync/send or production payload occurred.
-- **Rollback status:** downgrade `20260810080858_AddEmailDraftClientRequestId`, then `20260810075206_AddEmailDrafts`, before reverting draft commits; then follow the existing MAIL rollback order (`ee5ef7e`, `449faeb`, `123fc55`/`e9937ac`, ledger downgrade before `653f011`). No production migration/deploy/provider grant occurred.
-- **Uncommitted changes:** V-174 readable-export code/tests/docs; no dependency, schema or production configuration change. V-173 is pushed as `cdcc716`.
-- **Known failures:** live deployment is not verified because PR deploy is intentionally skipped and the active branch is not approved for merge. Draft export/API/UI, full thread/category actions and non-Gmail review remain; existing accounts need re-consent and IMAP stays read-only. A clean full-chain SQLite apply fails in the pre-existing JT-019 migration before the new draft migration. Browser/provider/MariaDB/production unavailable or unverified; recovery scan performance is unmeasured at large ledger scale; Jest open handles; SEC-006 parser dependency work is still separately gated; parser isolation remains SEC-007.
-- **Exact next action:** complete full Chromium after the export addition, commit/push V-174, then add the disabled additive deletion state and coordinator using the same owner inventory.
-- **Work that can continue independently:** SEC-009 repository-side owner inventory/export and disabled deletion lifecycle. UX/JOBS/PRODUCT production, MAIL provider mutations, SEC-006/007 and PROD packages retain their recorded external gates.
-- **Decisions still required from the user:** none for synthetic/code-inspected repository work. Any provider connection or send test, internet/package upgrades, private data, external/paid providers and production actions retain explicit approval/safety gates; SEC-009 retention/legal policy remains unresolved.
+- **Temporary files or processes:** no task-owned process is running. Playwright stopped its disposable API/Next servers. No raw benchmark output/evidence file was created.
+- **Production changes currently active:** none. Read-only SSH observed metadata/health/selected non-secret settings and backup integrity only. No log/private-row/content/secret read, deployment, migration, provider call, inference, model pull, restart, backup, restore or production file/config change occurred.
+- **Rollback status:** SEC-009 is additive migration `20260815164027_AddAccountDeletionLifecycle`. Keep deletion disabled, reconcile any durable request, and retain tombstones before downgrade. `842e793` is pushed. Production still runs `de937d25dc5e` / version `157`; its checkout has a pre-existing mode-only `deploy/deploy.sh` change that must be preserved/reviewed.
+- **Uncommitted changes:** documentation and the plan-only Ollama benchmark harness/tests following pushed SEC-009 commit `842e793`; no dependency, model, application runtime, schema or production state change in this checkpoint.
+- **Known failures:** PR 28's newest remote CI is not yet confirmed. Production publishes frontend 3000 and JobTracker Ollama 11434 on all interfaces; latest observed database-only backup is 2026-08-02; no complete files/keys/tombstone restore proof; root is 83% used; deployed AI sidecar is old direct-Gemini behavior; production script mode is dirty. SEC-006 internet access, SEC-007 dependency, provider/re-consent, Stripe price, signup, retention/legal, backup/restore, model execution/deployment and legacy cutover decisions remain recorded blockers. Historical JT-019 and Jest open handles remain.
+- **Exact next action:** after this documentation/harness checkpoint is committed and pushed, stop. Resume from the highest-priority blocker the user authorizes: recommended first is production network plus complete backup/scratch-restore safety, then bounded synthetic model benchmarking.
+- **Work that can continue independently:** none identified after the PROD-003 harness. Do not bypass blockers by pulling models, changing ports/firewalls, restoring data, contacting providers, or using package indexes without explicit authority.
+- **Decisions still required from the user:** production network/port mutation; complete backup and scratch restore; retention/tombstone/legal policy; model pull/synthetic production inference; deployment/worker activation; parser package-index access; provider/Stripe/signup actions; legacy cutover timing.