feat(ai): add safe benchmark harness
This commit is contained in:
@@ -749,3 +749,23 @@
|
||||
- **Consequences:** users can download JSON and owned files with independently verifiable SHA-256 checksums. Missing/legacy/external/backup categories are disclosed truthfully. The service becomes the authoritative inventory seam for deletion without coupling export to deletion activation.
|
||||
- **User approval required:** No; this is the requested repository-side data lifecycle, using synthetic tests and no production data.
|
||||
- **Reversible:** Remove the endpoint/UI and service. Existing downloaded ZIPs remain user-owned files; no stored schema or data changed.
|
||||
|
||||
## DEC-076 — Dark-launch account deletion as a durable staged lifecycle
|
||||
|
||||
- **Date:** 2026-08-15
|
||||
- **Decision:** Replace admin Identity-row deletion with one disabled-by-default coordinator for self-service and admin deletion. Lock the identity immediately, quarantine verified owned files before a transactional explicit row purge, retain retry ledgers, and write a minimal pseudonymous tombstone outside the restored database before completion.
|
||||
- **Reason/evidence:** Identity-only deletion leaves owned rows, files, provider credentials, sessions and queued work behind. Cascades cannot coordinate filesystem failures or prevent an older backup from resurrecting the user. Real-SQLite tests prove two-owner isolation, safe quarantine failure, idempotence and restored-account replay.
|
||||
- **Alternatives considered:** broad cascade foreign keys; best-effort controller deletes; delete database rows before files; edit old backups in place; enable immediately after local tests. These lose retry/audit boundaries, risk stranded private files or partial erasure, and overclaim backup/provider guarantees.
|
||||
- **Consequences:** `AccountLifecycle:DeletionEnabled` remains false until retention, protected tombstone custody, cache/provider handling and a disposable restore rehearsal are approved. Existing sessions are rejected as soon as status becomes pending. Deletion request/file records intentionally survive the user row and the background worker resumes durable requests even while the feature flag blocks new ones.
|
||||
- **User approval required:** Production activation and retention decisions only. Repository implementation uses synthetic data and remains inert by default.
|
||||
- **Reversible:** Disable requests (the default), allow in-flight reconciliation to finish, then downgrade the additive migration only after no request remains. Never remove the separate tombstone ledger while backups capable of restoring deleted identities still exist.
|
||||
|
||||
## DEC-077 — Make production model benchmarking plan-only by default
|
||||
|
||||
- **Date:** 2026-08-15
|
||||
- **Decision:** Use one standard-library harness over the checked-in synthetic evaluation set. It plans without network access by default, requires `--execute` plus an explicit output for inference, never pulls/deletes models, permits only loopback or an explicitly opted-in literal private IP, and persists hashes/metrics/scores rather than raw inputs/prompts/outputs.
|
||||
- **Reason/evidence:** The measured production host can run bounded tests, but its network and backup stop conditions are open and model execution is not authorized. A ready harness removes future ad-hoc prompt/data/report handling without silently widening current authority.
|
||||
- **Alternatives considered:** benchmark immediately over SSH; use production CV/email content; install a benchmark dependency/framework; persist raw responses for later scoring; auto-pull every candidate. These violate current authority/privacy boundaries or add avoidable supply-chain/resource risk.
|
||||
- **Consequences:** candidate selection remains blocked and no model decision is claimed. Approved runs can compare 4K/8K timing, token rate, JSON/constraint quality and Ollama VRAM metadata reproducibly while host GPU/RAM metrics are captured separately.
|
||||
- **User approval required:** Yes before any model pull, production inference, or production report execution. No approval is required for plan-only fixture validation.
|
||||
- **Reversible:** Remove the script/tests/template; no dependency, model, application, production, schema or configuration state changed.
|
||||
|
||||
Reference in New Issue
Block a user