feat(ai): add safe benchmark harness
This commit is contained in:
@@ -2,21 +2,21 @@
|
||||
|
||||
Updated: 2026-08-15
|
||||
|
||||
- **Overall programme status:** Active. Eight packages are locally verified; twenty-four are implemented with verification incomplete; SEC-009 is in progress. The prioritized admin-only version indicator and every immediate repository/browser item are implemented on the release branch; remote and production verification remain.
|
||||
- **Current work package:** `SEC-009` — complete readable export and account deletion lifecycle (`IN PROGRESS`). Owner-scoped generated paths and the real-SQLite-verified readable ZIP are complete; proceed with the disabled deletion lifecycle while retention/restore policy blocks production activation.
|
||||
- **Overall programme status:** Active but externally blocked. Eight packages are locally verified and twenty-five are implemented with verification incomplete. The prioritized admin-only version indicator, every immediate repository/browser item, SEC-009, the PROD-001 read-only inventory, and the PROD-003 safe benchmark harness are complete on the release branch.
|
||||
- **Current work package:** None. Every remaining package now requires a user/operator decision, authorized production mutation/restore/provider action, or explicit package-index access.
|
||||
- **Completed work packages:** None are `DONE`; all repository security/AI packages still have applicable browser, provider and/or production gates.
|
||||
- **Locally verified work:** SEC-001, SEC-002, SEC-003, SEC-005A, CORE-001, PROD-002, DEP-001 and VER-001 (`VERIFIED LOCALLY`).
|
||||
- **Implemented, verification incomplete:** SEC-004, SEC-005B, SEC-008, CORE-002, BG-001, OPS-001A/B/C, POL-001/002, AI-001/002/003/004, UX-001/002/003, QA-001, CAREER-001/002, MAIL-001, JOBS-001/002 and PRODUCT-001 (`IMPLEMENTED — NOT VERIFIED`). Their safe repository/browser scope is implemented; production/native-device/provider gates remain where recorded.
|
||||
- **Implemented, verification incomplete:** SEC-004, SEC-005B, SEC-008, SEC-009, CORE-002, BG-001, OPS-001A/B/C, POL-001/002, AI-001/002/003/004, UX-001/002/003, QA-001, CAREER-001/002, MAIL-001, JOBS-001/002 and PRODUCT-001 (`IMPLEMENTED — NOT VERIFIED`). Their safe repository/browser scope is implemented; production/native-device/provider/retention gates remain where recorded.
|
||||
- **Production-verified work:** None.
|
||||
- **Blocked work:** SEC-006 parser upgrades remain outside the scoped frontend advisory permission; PROD-001/003/004 and REL-001 require documented production access and unfinished dependencies. Real provider, SMTP/MariaDB and production environments are unavailable; DEP-001 awaits approved merge/live verification. The in-app browser is available for local UI checks.
|
||||
- **Blocked work:** SEC-006 parser upgrades remain outside the scoped frontend advisory permission; PROD-001/003/004 and REL-001 require network/backup/model/deployment authority and unfinished dependencies. Real provider and live deletion/restore checks remain gated; DEP-001 awaits approved merge/live verification.
|
||||
- **Deferred work:** None. Conditional multi-replica coordination, model deletion, realtime operation delivery and unrelated production changes remain outside current packages.
|
||||
- **Immediate order:** SEC-009 owner inventory/export first, then its disabled deletion lifecycle. The eight-item immediate queue is complete locally: admin version (`a6cffe0`), Career persistence (`f0b9b22`), CV contrast (`3b86ea2`), JOBS-002 (`deed948`), accessibility (`a7c2549`), PRODUCT-001 (`a25c31b`), VER-001 and tracking reconciliation. External-only work remains skipped, not allowed to stall repository progress.
|
||||
- **Status counts:** 8 `VERIFIED LOCALLY`; 24 `IMPLEMENTED — NOT VERIFIED`; 1 `IN PROGRESS`; 1 `NOT STARTED`; 5 `BLOCKED`; 0 `DONE`; 0 `DEFERRED`.
|
||||
- **Test status:** backend 650/650; frontend 58/58 suites and 234/234 tests; AI sidecar 22/22; optimized production build/TypeScript; Docker Compose config; safe-failure deployment preflight; and the preceding Playwright 9/9 pass. A fresh Chromium Free account now also downloads the real readable ZIP response. npm audit 0 evidence remains current because the lockfile did not change. Historical JT-019 and Jest force-exit/open-handle behavior remain recorded.
|
||||
- **Immediate order:** the eight-item immediate queue is complete locally: admin version (`a6cffe0`), Career persistence (`f0b9b22`), CV contrast (`3b86ea2`), JOBS-002 (`deed948`), accessibility (`a7c2549`), PRODUCT-001 (`a25c31b`), VER-001 and tracking reconciliation. SEC-009 is complete at `842e793`; PROD-001 read-only evidence and the PROD-003 plan-only harness are complete pending this documentation commit. No further independent implementation remains.
|
||||
- **Status counts:** 8 `VERIFIED LOCALLY`; 25 `IMPLEMENTED — NOT VERIFIED`; 0 `IN PROGRESS`; 1 `NOT STARTED`; 5 `BLOCKED`; 0 `DONE`; 0 `DEFERRED`.
|
||||
- **Test status:** backend 657/657; frontend 58/58 suites and 237/237 tests; AI sidecar 22/22; Ollama benchmark harness 4/4 plus safe dry-run; optimized production build/TypeScript; EF model parity and MariaDB migration-script generation; Docker Compose config; safe-failure deployment preflight; and Playwright 9/9. Five real-SQLite deletion tests cover lockout, isolation, quarantine failure and restored-backup replay. npm audit 0 evidence remains current because the lockfile did not change. Historical JT-019 and Jest force-exit/open-handle behavior remain recorded.
|
||||
- **Deployment status:** Gitea pull-request run 609 passed the complete CI job in 4m20s. Deploy was intentionally skipped because the workflow deploys only a `push` to `main`; live remains unchanged. No merge/deployment was performed directly, no production migrations were run and the AI operation worker remains disabled by default.
|
||||
- **Production status:** Unchanged and unverified. No provider/model call, model pull, external request or paid API occurred.
|
||||
- **Production status:** State unchanged. Sanitized read-only SSH inventory was performed; no logs, prompts, private rows/content or secret values were read, and no provider/model call, model pull, service restart, file/config change, backup, restore, migration or deployment occurred. It confirmed all-interface Ollama/frontend listeners and stale database-only backups as rollout blockers.
|
||||
- **Known regressions:** None found by automated/local browser checks. Jest still needs `--forceExit` and reports its existing open-handle notice. Email-provider/send tests are fake/local only; real delivery is not claimed. Current MAIL browser evidence is 1280×720 only because the browser surface could not resize or perform native Tab traversal. Interrupted attempts are aged after 15 minutes and notified without retry; the five-minute scan is unmeasured on a large ledger. Direct clean EF-only SQLite migration still hits the pre-existing historical blank-chain defect before later migrations; normal startup owns reconciliation. Cross-feature monthly AI usage accounting remains a rollout gap.
|
||||
- **Outstanding security findings:** JT-001 repository ownership remains High deployment risk until migration/inventory/provider checks; production portion of JT-002; JT-006/JT-009 and associated JT-011/JT-012/JT-022 prerequisites. JT-005 foundations are implemented; AI worker activation awaits controlled rollout. JT-007/JT-008/JT-010 lack browser/provider/production verification.
|
||||
- **Outstanding security findings:** JT-001 repository ownership remains High deployment risk until migration/inventory/provider checks; production portion of JT-002; JT-006 and SEC-009 production retention/restore plus JT-011/JT-012/JT-022 prerequisites. JT-005 foundations are implemented; AI worker activation awaits controlled rollout. Production still exposes ports contrary to the release-branch contract, and JT-007/JT-008/JT-010 lack provider/production verification.
|
||||
|
||||
## Current evidence
|
||||
|
||||
@@ -31,6 +31,7 @@ Updated: 2026-08-15
|
||||
- `docs/verification/core-001-sqlite-provider-parity.md`
|
||||
- `docs/verification/core-002-route-uniqueness.md`
|
||||
- `docs/verification/sec-008-attachment-consistency.md`
|
||||
- `docs/verification/sec-009-account-lifecycle.md`
|
||||
- `docs/verification/bg-001-tenant-workers.md`
|
||||
- `docs/verification/ops-001a-durable-operations.md`
|
||||
- `docs/verification/ops-001b-notifications.md`
|
||||
@@ -52,4 +53,7 @@ Updated: 2026-08-15
|
||||
- `docs/verification/product-001-honest-plans.md`
|
||||
- `docs/verification/ver-001-complete-regression.md`
|
||||
- `docs/verification/prod-002-ai-evaluation.md`
|
||||
- `docs/production/production-ai-hardware-assessment.md`
|
||||
- `docs/production/production-ai-rollout-and-rollback.md`
|
||||
- `docs/production/ollama-model-benchmark.md`
|
||||
- `docs/work-programmes/master-work-plan.md`
|
||||
|
||||
Reference in New Issue
Block a user