feat(ai): add safe benchmark harness
This commit is contained in:
@@ -16,7 +16,7 @@ Allowed statuses are `NOT STARTED`, `IN PROGRESS`, `IMPLEMENTED — NOT VERIFIED
|
||||
|
||||
`DONE` requires every applicable acceptance criterion, focused and regression tests, browser/accessibility/theme/mobile checks, tenant and entitlement checks, documentation, migration/rollback evidence, and production verification. Repository-only work that still requires production is at most `VERIFIED LOCALLY`.
|
||||
|
||||
Exactly one implementation item may be `IN PROGRESS`. As of this revision it is **SEC-009**.
|
||||
At most one implementation item may be `IN PROGRESS`. No package is currently in progress: every remaining package is blocked by an external decision, production mutation, provider interaction, or explicitly unapproved package-index access.
|
||||
|
||||
## Consolidated dependency order
|
||||
|
||||
@@ -268,11 +268,11 @@ This queue records the highest-value work that can proceed without production cr
|
||||
- **Required tests:** two users/every entity, manifest/checksums/redaction, fault/restart/idempotence, provider failures, disposable restore replay.
|
||||
- **Required browser verification:** disposable self/admin export/delete and confirmations.
|
||||
- **Required production verification:** backup retention/tombstone rehearsal before self-service enablement.
|
||||
- **Status:** `IN PROGRESS`.
|
||||
- **Status:** `IMPLEMENTED — NOT VERIFIED`.
|
||||
- **Blocker:** legal/operator retention and production restore decisions block activation, not the repository-side disabled/dark launch.
|
||||
- **Evidence:** audit JT-009 inventory/design; `docs/verification/sec-009-account-lifecycle.md`; V-173/V-174. Owner-scoped generated storage and complete redacted readable ZIP pass real-SQLite two-owner, focused API/UI, full backend/frontend, build and Chromium checks.
|
||||
- **Commit:** none.
|
||||
- **Remaining work:** reuse the completed inventory for the additive disabled deletion coordinator, tombstone replay, UI and failure/restart verification. Production activation remains blocked by retention/restore policy.
|
||||
- **Evidence:** audit JT-009 inventory/design; `docs/verification/sec-009-account-lifecycle.md`; V-173/V-174/V-175. Owner-scoped generated storage, complete redacted readable ZIP, immediate lockout, transactional owner-isolated row/file purge, retry, fail-closed tombstones and restored-backup replay pass real-SQLite, focused API/UI, full backend/frontend, build and Chromium checks.
|
||||
- **Commit:** `842e793`.
|
||||
- **Remaining work:** repository scope is complete. Production activation remains blocked by retention/legal decisions, protected tombstone custody, restored-backup rehearsal, provider/cache semantics and staged disposable-account rollout.
|
||||
|
||||
### CORE-001 — Restore default SQLite/MariaDB behavior parity
|
||||
|
||||
@@ -467,10 +467,10 @@ This queue records the highest-value work that can proceed without production cr
|
||||
- **Required browser verification:** none.
|
||||
- **Required production verification:** this item is itself production read-only verification.
|
||||
- **Status:** `BLOCKED`.
|
||||
- **Blocker:** repository docs state the local environment has no production route and CI SSH secrets are unavailable; no documented callable host/credential is present.
|
||||
- **Evidence:** `docs/deployment/backup-restore.md` and `docs/operations/production-backup-verification.md` explicitly record the access gap.
|
||||
- **Blocker:** read-only inventory is complete, but Ollama/frontend host ports are published on all interfaces, complete current backups/restores are unproved, the production checkout has an unreviewed mode-only deploy-script change, and closing those gaps requires approved production mutations.
|
||||
- **Evidence:** `docs/production/production-ai-hardware-assessment.md`; `docs/production/production-ai-rollout-and-rollback.md`; V-176. Sanitized read-only SSH measured hardware, GPU, storage, Docker, networks, selected provider/model, health, limits and backup presence without reading secrets/logs/content or changing state.
|
||||
- **Commit:** none.
|
||||
- **Remaining work:** create sanitized report template locally; operator/documented access required for measured completion.
|
||||
- **Remaining work:** approve and close the network/port, complete-backup/scratch-restore, dirty-checkout and retention gates; then repeat the sanitized inventory to verify the safe target state.
|
||||
|
||||
### PROD-002 — AI workload inventory and synthetic evaluation set
|
||||
|
||||
@@ -503,10 +503,10 @@ This queue records the highest-value work that can proceed without production cr
|
||||
- **Required browser verification:** none.
|
||||
- **Required production verification:** measured on actual machine; local workstation results are labeled separately.
|
||||
- **Status:** `BLOCKED`.
|
||||
- **Blocker:** PROD-001 production access/hardware inventory.
|
||||
- **Evidence:** none yet.
|
||||
- **Blocker:** production model pull/inference is not authorized, and PROD-001's all-interface Ollama plus incomplete backup/restore stop conditions remain open.
|
||||
- **Evidence:** `scripts/run-ollama-evaluation.py`; `scripts/test-ollama-evaluation.py`; `docs/production/ollama-model-benchmark.md`; V-177. Plan-only harness tests 4/4 and validates an eight-request 4K/8K Strategy plan without a model call.
|
||||
- **Commit:** none.
|
||||
- **Remaining work:** repository harness can be prepared after PROD-002.
|
||||
- **Remaining work:** after approvals and PROD-001 safety closure, verify candidate metadata/licenses, pull one candidate at a time, execute repeated synthetic 4K/8K benchmarks, measure active GPU/RAM/offload and complete the evidence-based model decision.
|
||||
|
||||
### PROD-004 — Local-model rollout, fallback, observability and operations
|
||||
|
||||
|
||||
Reference in New Issue
Block a user