feat(ai): add safe benchmark harness
This commit is contained in:
cesnimda
@@ -206,3 +206,6 @@ Output was reduced to filenames and commit counts. The token artifact appears un
|
||||
| V-172 | Action-matrix reconciliation; full backend/frontend/sidecar/build/Compose/preflight gates; expanded authenticated and anonymous Playwright | Repository root / `job-tracker-ui` / `tools/summarizer` | Complete VER-001 local release regression without promoting mocked/provider/external checks | PASS — backend 647/647, frontend 57 suites/232 tests, sidecar 22/22, production build, Compose config, API-down/wrong-base/malformed-JSON preflight and Chromium 9/9. Browser covers admin deployment identity/normal-user absence, notifications, honest Free, jobs, Career/CV, Kanban, responsive themes and public PDF | No external provider, private data, native AT or production mutation. Optional Compose variables remain unset; existing Jest/GSI/SWIG warnings remain. Windows CRLF materialization was normalized for shell execution; indexed LF policy was already correct | VER-001 verified locally; remote CI/provider/native-AT/production cells remain |
|
||||
| V-173 | Owner-path trace; focused CV/export/controller/background tests; full backend; build and diff hygiene | Repository root | Establish attributable generated-file ownership before SEC-009 export/deletion | PASS — CV PDFs use opaque owner/date/UUID storage while preserving download names; daily exports use opaque owner directories and atomic writes; legacy/new retention paths are covered. Focused 77/77 and backend 647/647 | No existing file moved or deleted. Legacy shared-date generated files are intentionally not guessed. No migration, production path or private data used | SEC-009 owner-scoped generated-output prerequisite verified locally |
|
||||
| V-174 | Real-SQLite two-owner export fixture; manifest/checksum/file/redaction assertions; recent-session/rate-limit API tests; focused/full frontend and backend; optimized build; Chromium ZIP response | Repository root / `job-tracker-ui` | Deliver a complete user-readable export without exposing secrets or another tenant | PASS — focused backend/API 11/11, backend 650/650, frontend focused 4/4 and full 58 suites/234 tests, builds pass. Every manifest checksum/size matches; owned attachment/CV/avatar/generated/daily files are included; secret and other-owner sentinels are absent; Chromium receives HTTP 200 `application/zip` with `PK` signature | Synthetic data/files only; no production/private/provider access. Export reports backups/logs/external retention instead of claiming erasure. One initial InMemory-only test missed SQLite DateTimeOffset translation; the test moved to real SQLite and the query boundary was corrected | SEC-009 readable export verified locally; deletion/retention/restore remain |
|
||||
| V-175 | Account-deletion threat-path review; real-SQLite lifecycle/controller tests; focused/full backend/frontend; optimized build; EF parity and MariaDB script generation; disposable Chromium startup/application suite | Repository root / `job-tracker-ui` | Deliver a production-inert, owner-isolated, retryable live-account deletion lifecycle without implying backup/provider erasure | PASS — focused backend 21/21, backend 657/657, frontend focused 8/8 and full 58 suites/237 tests, optimized build, no pending EF changes, bounded MariaDB script and Chromium 9/9. Tests prove disabled gate, exact/recent confirmation, immediate lockout, idempotence, owner-isolated row/file purge, quarantine failure safety, tombstone creation and restored-account replay; `842e793` pushed | Synthetic local rows/files only. No live deletion, provider revocation, sidecar restart, backup restore, production migration or retention decision occurred. Self/admin requests remain disabled by default. Full every-provider/disposable-production rehearsal remains external | SEC-009 repository scope implemented; production activation remains blocked |
|
||||
| V-176 | Authorized sanitized read-only SSH inventory of OS/CPU/RAM/GPU/storage/Docker/Ollama/app health/networks/selected non-secret configuration and backup metadata/integrity | Production host / repository report | Replace stale guessed hardware/access assumptions and define evidence-based rollout stops without changing production | PASS/PARTIAL — confirmed Ubuntu 24.04.4, i5-8600 6C, 31 GiB RAM, GTX 1060 6GB, 1.4 TiB Docker-volume headroom, Ollama 0.31.1 with `qwen2.5:7b`, four healthy app containers and exact deployed version. Also confirmed all-interface 11434/3000 listeners, unlimited container resources, direct Gemini selection in the old sidecar, dirty deploy-script mode, and 21 gzip-valid database-only backups ending 2026-08-02 | No secret values, logs, prompts, private rows/content, provider call, inference, model pull, file read beyond metadata/integrity, service restart, backup creation, restore or mutation. Internet/NAT exposure, logical restore and complete file/key recovery remain unverified | PROD-001 inventory/report complete; safety acceptance blocked by measured network/backup/deployment gaps |
|
||||
| V-177 | Python unit tests and plan-only execution against the checked-in synthetic fixture; script/content/SSRF/report review | Repository root | Prepare reproducible privacy-safe PROD-003 evaluation without authorizing a model or network call | PASS — harness tests 4/4; plan validates four Strategy cases across 4K/8K for eight future requests. Default performs no HTTP; exact model, `--execute` and explicit output are required; model pull/delete is absent; report excludes raw fixture/prompt/output content | No Ollama request, provider/internet access, model metadata query through the harness, candidate pull, inference or production change. Candidate metadata/licenses/results and model decision remain unmeasured | PROD-003 repository harness complete; execution remains blocked |
|
||||
|
||||
@@ -18,6 +18,19 @@
|
||||
The commands below are the ones to run against production. They are recorded so the owner can execute
|
||||
the same sequence with production values substituted.
|
||||
|
||||
## 2026-08-15 read-only production checkpoint
|
||||
|
||||
The earlier access statement above remains historically accurate for the 2026-07-19 rehearsal, but access is now available. A strictly read-only production inventory found:
|
||||
|
||||
- 21 MariaDB `.sql.gz` dumps (about 5.2 MB combined), all passing `gzip -t`;
|
||||
- oldest observed dump: 2026-07-19 21:41 UTC; newest: 2026-08-02 17:32 UTC;
|
||||
- no JobTracker systemd timer or current-user cron entry;
|
||||
- 1.4 TiB free on the Docker/data filesystem and 36 GiB free on the 83%-used root filesystem;
|
||||
- database-only dump files in the backup directory: no owner-file volume, data-protection-key, protected tombstone, or configuration recovery bundle;
|
||||
- no restore, row/content read, count comparison, non-ASCII check, or production mutation.
|
||||
|
||||
This upgrades production dump *presence and compressed-stream integrity* from unknown to observed, but it does not close the restore/RPO/completeness checklist. The newest observed dump was 13 days old at capture. See `docs/production/production-ai-hardware-assessment.md` and `docs/production/production-ai-rollout-and-rollback.md`.
|
||||
|
||||
## 1. Backup configuration
|
||||
|
||||
Read from `deploy/deploy.sh` and `docker-compose.yml`:
|
||||
|
||||
@@ -0,0 +1,57 @@
|
||||
# Ollama model benchmark
|
||||
|
||||
Updated: 2026-08-15
|
||||
|
||||
Status: `BLOCKED`. The synthetic benchmark harness is implemented, but no candidate was pulled and no inference was run. Production benchmarking requires the network, backup, model-pull and synthetic-execution approvals recorded in `BLOCKERS.md` and `production-ai-rollout-and-rollback.md`.
|
||||
|
||||
## Measured baseline inventory
|
||||
|
||||
| Candidate | Size | Quantisation | VRAM | RAM | GPU offload | Context | Load / first token / total / tok/s | Quality / JSON / Norwegian / failure | Licence | Recommendation |
|
||||
|---|---:|---|---:|---:|---|---:|---|---|---|---|
|
||||
| `qwen2.5:7b` (installed) | 4.7 GB | Q4_K_M; 7.6B parameters | Not measured loaded | Not measured loaded | Not measured | advertised 32K; test 4K/8K first | Not run | Not run | Apache-2.0 | Baseline only; do not select without measured JobTracker results |
|
||||
| `qwen3.5:4b` | Unknown locally | Not inspected | Not measured | Not measured | Not measured | 4K/8K first | Not run | Not run | Verify before pull/use | Leading candidate from the source plan, not an approved model |
|
||||
| `qwen3:4b` | Unknown locally | Not inspected | Not measured | Not measured | Not measured | 4K/8K first | Not run | Not run | Verify before pull/use | Candidate only |
|
||||
| `gemma3:4b` | Unknown locally | Not inspected | Not measured | Not measured | Not measured | 4K/8K first | Not run | Not run | Verify hosted-use terms before pull/use | Candidate only |
|
||||
|
||||
Optional 8B/9B candidates remain excluded until the 4B/baseline measurements prove headroom on the 6 GB GPU. Cloud-labelled Ollama models are out of scope.
|
||||
|
||||
## Harness
|
||||
|
||||
`scripts/run-ollama-evaluation.py`:
|
||||
|
||||
- reads only the checked-in `syntheticOnly=true` evaluation fixture;
|
||||
- runs one exact installed model and never pulls/deletes a model;
|
||||
- defaults to plan-only mode; network calls require `--execute` and an explicit output path;
|
||||
- accepts loopback only unless a literal private IP is explicitly opted in;
|
||||
- supports 4K/8K/conditional context comparisons, repetition, temperature, output limit and keep-alive;
|
||||
- streams responses to measure first-token and wall latency;
|
||||
- records Ollama load/prompt/generation timings, token rate and `/api/ps` loaded/VRAM sizes when available;
|
||||
- scores must-contain, forbidden-claim, strict-JSON and required-key constraints;
|
||||
- writes hashes and scores, never raw fixture input, prompts, or model output.
|
||||
|
||||
Plan-only example (safe; no network call):
|
||||
|
||||
```bash
|
||||
python scripts/run-ollama-evaluation.py \
|
||||
--task STRATEGY \
|
||||
--model qwen2.5:7b
|
||||
```
|
||||
|
||||
Approved local/private execution shape:
|
||||
|
||||
```bash
|
||||
python scripts/run-ollama-evaluation.py \
|
||||
--task STRATEGY \
|
||||
--model qwen2.5:7b \
|
||||
--context 4096 \
|
||||
--context 8192 \
|
||||
--repeat 3 \
|
||||
--execute \
|
||||
--output evidence/strategy-qwen2.5-7b.json
|
||||
```
|
||||
|
||||
The output directory should remain outside source control unless a sanitized evidence path is explicitly approved.
|
||||
|
||||
## Decision gate
|
||||
|
||||
No primary or fallback is selected. A decision requires repeated measured results for every relevant generative task, exact tag/digest/version/license/quantization, active GPU offload, peak VRAM/RAM/swap, cold/warm load, first-token/total latency, tokens/second, JSON success, English/Norwegian quality, factuality/injection/failure behavior, and stability. Deterministic tasks in `docs/ai/workload-inventory.md` remain non-model work regardless of benchmark scores.
|
||||
@@ -0,0 +1,121 @@
|
||||
# Production AI hardware and runtime assessment
|
||||
|
||||
Updated: 2026-08-15
|
||||
|
||||
Status: read-only production inventory complete. No service, file, firewall, model, database, backup, container, or configuration value was changed. Host identity, addresses, credentials, environment secrets, logs, prompts, and private application content are intentionally omitted.
|
||||
|
||||
## Executive result
|
||||
|
||||
The remembered hardware profile is accurate: the host has about 32 GiB RAM and one NVIDIA GeForce GTX 1060 6GB. The machine has ample capacity on the Docker/model filesystem for bounded sequential benchmarks, but rollout is not safe yet:
|
||||
|
||||
- the JobTracker Ollama container publishes port 11434 on every IPv4 and IPv6 host interface;
|
||||
- the deployed application is older than the release branch and still selects Gemini directly in the AI sidecar rather than the release branch's disabled-external, local-first policy;
|
||||
- no CPU, memory, PID, or read-only-root limits are applied to the four application containers;
|
||||
- database backups are small and gzip-valid, but the newest observed file is 13 days old and no JobTracker timer/cron entry was found;
|
||||
- observed backups cover MariaDB only, not the owner-file volume, data-protection keys, protected deletion tombstones, or non-secret recovery configuration.
|
||||
|
||||
These are rollout stop conditions, not permission to change production.
|
||||
|
||||
## Measured host
|
||||
|
||||
| Area | Read-only measurement |
|
||||
|---|---|
|
||||
| OS | Ubuntu 24.04.4 LTS, x86_64 |
|
||||
| Kernel | Linux 6.8.0-110-generic |
|
||||
| Time zone | Europe/Oslo |
|
||||
| Uptime | 3 weeks 6 days at capture |
|
||||
| Load | 3.25 / 2.80 / 2.29 |
|
||||
| CPU | Intel Core i5-8600 @ 3.10 GHz; 1 socket, 6 physical/logical cores, 1 thread/core |
|
||||
| RAM | 31 GiB total; 11 GiB used; 20 GiB available |
|
||||
| Swap | 8 GiB total; 2.1 GiB used |
|
||||
| Memory pressure | PSI `some` avg10 0.16%; `full` avg10 0.06% |
|
||||
| Shell open-file limit | 1,024 |
|
||||
| Root filesystem | ext4, 217 GiB total, 171 GiB used, 36 GiB available (83% used) |
|
||||
| Docker/model filesystem | ext4, 1.8 TiB total, 316 GiB used, 1.4 TiB available (19% used) |
|
||||
| Docker | Engine 29.3.0, API 1.54; data root on the large filesystem |
|
||||
|
||||
Other workloads share the host. The available-memory and pressure measurements are therefore more useful than total RAM alone; benchmarks must capture concurrent load rather than assuming an idle dedicated server.
|
||||
|
||||
## GPU
|
||||
|
||||
| Area | Read-only measurement |
|
||||
|---|---|
|
||||
| GPU | NVIDIA GeForce GTX 1060 6GB |
|
||||
| VRAM | 6,144 MiB total; 3 MiB used; 6,064 MiB reported free at capture |
|
||||
| Driver | 580.159.03 |
|
||||
| CUDA compatibility reported by driver | 13.0 |
|
||||
| Idle state | P8, 37 C, 0% utilization, about 5.5 W / 120 W |
|
||||
| GPU processes | None at capture |
|
||||
| Ollama GPU device request | All GPUs requested by the Ollama container |
|
||||
| AI sidecar GPU device request | None; health reports CPU and `gpu_available=false` |
|
||||
|
||||
Ollama was not serving a loaded model during capture, so GPU-layer offload is not yet proven. A bounded benchmark must verify the `PROCESSOR`/offload result while a request is active; idle `nvidia-smi` is not evidence of successful GPU inference.
|
||||
|
||||
## Ollama and models
|
||||
|
||||
- Deployment method: Docker Compose, `ollama/ollama:latest`.
|
||||
- Runtime version: 0.31.1.
|
||||
- Captured image digest: `sha256:f1a705f2bd113fb8d15f85f7c217f0dc5f6bebda6b0cc42b82c3ad165ffcb9dc`.
|
||||
- Installed JobTracker model: `qwen2.5:7b`, model ID prefix `845dbda0ea48`, 4.7 GB.
|
||||
- Model volume use: about 4.4 GiB.
|
||||
- Loaded models: none at capture.
|
||||
- Health: Ollama reachable from the AI sidecar; model present; container healthy with zero observed restarts since 2026-08-02.
|
||||
- GPU use: not active at capture.
|
||||
|
||||
### Exposure finding
|
||||
|
||||
`OLLAMA_HOST=0.0.0.0:11434`, Docker publishes 11434 on all IPv4/IPv6 interfaces, and the container joins the private AI network plus broader application/shared networks. A second, non-JobTracker Ollama listener also exists on host port 11435. Internet/NAT reachability was not tested, but all-interface host publication already fails the intended localhost/private-network-only contract.
|
||||
|
||||
The release branch removes JobTracker's host publication and shared/default network membership for its bundled Ollama. Deployment must still decide whether to use that private bundled instance or a separately controlled shared instance; it must not leave an orphaned published container.
|
||||
|
||||
## Application deployment
|
||||
|
||||
| Component | Runtime state | Point-in-time usage |
|
||||
|---|---|---|
|
||||
| Backend | healthy, zero restarts | 0.28% CPU; 216 MiB RAM; 25 PIDs |
|
||||
| Frontend/nginx | healthy, zero restarts | 0% CPU; 7.6 MiB RAM; 7 PIDs |
|
||||
| AI sidecar | healthy, zero restarts | 0.07% CPU; 2.24 GiB RAM; 30 PIDs |
|
||||
| JobTracker Ollama | healthy, zero restarts | 0.08% CPU; 4.56 GiB container-accounted RAM; 13 PIDs |
|
||||
|
||||
- Deployed commit: `de937d25dc5e`; configured app version: `157`.
|
||||
- The deployed repository is on `main` with a mode-only local change to `deploy/deploy.sh`; do not reset or overwrite it without operator review.
|
||||
- All four containers use `unless-stopped`, rotating `json-file` logs at 10 MiB x 3.
|
||||
- All four have unlimited CPU/memory/PIDs, writable root filesystems, and are not privileged.
|
||||
- Backend and frontend share the normal application network; backend and sidecar share the AI network. The deployed Ollama additionally joins broader networks.
|
||||
- The sidecar is internal-only on port 8001 and requires the shared service token for non-health endpoints.
|
||||
- The deployed frontend publishes host port 3000. Backend and sidecar are not published by this Compose project. The host also has an unrelated listener on 8080.
|
||||
- nginx has no explicit connect/read/send proxy timeout in the deployed file, so defaults apply.
|
||||
|
||||
## Current AI behavior
|
||||
|
||||
The deployed sidecar is older than the release branch:
|
||||
|
||||
- it loads `sshleifer/distilbart-cnn-12-6` on CPU for summarization;
|
||||
- it has Ollama configured with `qwen2.5:7b`, but no Ollama model was loaded at capture;
|
||||
- `AI_PROVIDER=gemini`, with Gemini and Groq credential variable names present (values were not read);
|
||||
- deployed code selects that provider directly and does not yet expose the release branch's `EXTERNAL_AI_ENABLED` / `AI_ROUTING_MODE` controls;
|
||||
- the backend's new durable AI worker settings are absent from the old deployment, so the release-branch defaults must be reviewed during deployment rather than inferred from this runtime.
|
||||
|
||||
Production is therefore not currently evidence for the release branch's local-first routing, durable queue, privacy gate, or entitlement behavior.
|
||||
|
||||
## Storage and backups
|
||||
|
||||
- JobTracker owner-file volume: about 14 MiB total at capture.
|
||||
- Visible categories: attachments 4 KiB, CV artifacts 1.1 MiB, generated CVs 16 KiB, daily exports 196 KiB.
|
||||
- The dark-launch account-export and deletion-tombstone directories are absent because their release has not been deployed.
|
||||
- The MariaDB data directory and Docker volumes are on the large filesystem; the SSH user cannot read their host-level sizes without privileged access.
|
||||
- `/opt/job-tracker/backups` contains 21 MariaDB `.sql.gz` files, about 5.2 MB combined. All passed `gzip -t`.
|
||||
- Observed range: 2026-07-19 through 2026-08-02. No newer dump and no JobTracker timer/current-user cron entry were observed.
|
||||
- No application-volume, attachment/CV, data-protection-key, tombstone, or configuration bundle was present in that backup directory.
|
||||
|
||||
This proves only that existing compressed dump files are structurally readable. It does not prove logical restore, row counts, non-ASCII fidelity, current RPO, or complete disaster recovery.
|
||||
|
||||
## Monitoring and evidence gaps
|
||||
|
||||
Available today: Docker health status, restart count, container stats, bounded container logs, sidecar health, `ollama list`, `ollama ps`, and host/GPU metrics. Missing: durable metrics/history, alerting tied to JobTracker SLOs, queue depth in this old deployment, inference latency/throughput history, active GPU-offload evidence, scheduled backup evidence, and current restore proof.
|
||||
|
||||
No production logs were read because they may contain prompts, paths, identifiers, or private content.
|
||||
|
||||
## Safe conclusion
|
||||
|
||||
The host can support a cautious one-model-at-a-time synthetic benchmark. Do not install or load a candidate, enable workers, enable external fallback, or deploy until the all-interface Ollama exposure, stale/incomplete backups, dirty deployment script, release-version gap, and rollback prerequisites in `production-ai-rollout-and-rollback.md` are resolved.
|
||||
@@ -0,0 +1,106 @@
|
||||
# Production AI rollout and rollback
|
||||
|
||||
Updated: 2026-08-15
|
||||
|
||||
Status: plan only. The 2026-08-15 activity was read-only. No backup, model pull, service restart, firewall/network change, deployment, database mutation, or restore was performed.
|
||||
|
||||
## Current rollback anchors
|
||||
|
||||
- Deployed application commit: `de937d25dc5e`; configured version `157`.
|
||||
- The production checkout is dirty because `deploy/deploy.sh` has a mode-only change. Preserve and review that state before any checkout/reset.
|
||||
- JobTracker Ollama image digest: `sha256:f1a705f2bd113fb8d15f85f7c217f0dc5f6bebda6b0cc42b82c3ad165ffcb9dc`.
|
||||
- Existing model: `qwen2.5:7b`, ID prefix `845dbda0ea48`.
|
||||
- Existing model must not be removed during benchmark or rollout.
|
||||
- All four current application containers are healthy, restart `unless-stopped`, and had zero restart count at capture.
|
||||
|
||||
Environment secret values were deliberately not copied. Recovery still depends on the protected production environment file and data-protection material; their custody has not been verified.
|
||||
|
||||
## Hard stop conditions before mutation
|
||||
|
||||
1. PR 28 must pass current CI and be approved for the normal deployment path.
|
||||
2. Review/preserve the production-only mode change to `deploy/deploy.sh`; never erase it with a blind reset.
|
||||
3. Remove JobTracker's host-published 11434 listener and broader network membership, and decide whether the separate shared Ollama listener is an approved private dependency. Verify no Internet/Traefik route and no unintended LAN clients.
|
||||
4. Produce a current MariaDB dump, a recoverable owner-file volume snapshot, data-protection-key recovery, protected tombstone storage, and non-secret configuration inventory. Keep tombstones outside any restored application backup.
|
||||
5. Restore the complete set into an isolated scratch environment and verify user/application/Career row counts, owned file access, non-ASCII content, key decryption, and tombstone replay. Never restore over live production for rehearsal.
|
||||
6. Define RPO, RTO, backup expiry, tombstone retention, legal hold, and account-deletion provider/cache semantics.
|
||||
7. Keep account deletion disabled and all previously inert workers off.
|
||||
8. Resolve resource limits for the AI sidecar/Ollama or document measured stop thresholds. Current containers have no CPU, memory, or PID caps.
|
||||
9. Confirm at least 25% free on the filesystem receiving image/model layers and enough root headroom for build/temp/log growth. Root is currently 83% used; Docker/model data is on the spacious secondary filesystem.
|
||||
10. Use only synthetic benchmark inputs. Do not use production CVs, correspondence, job descriptions, or prompts.
|
||||
|
||||
If any item fails, stop. Do not work around it by deleting models, pruning unknown Docker data, weakening health checks, exposing ports, or enabling external processing.
|
||||
|
||||
## Intended release-branch topology
|
||||
|
||||
- Frontend: reachable only through the approved reverse proxy network; no direct host 3000 publication.
|
||||
- Backend: internal application/proxy networks; no host port.
|
||||
- AI sidecar: only backend plus private AI network; no host port; service token required for non-health endpoints.
|
||||
- Ollama: private AI network only when bundled, or a separately approved private endpoint. No all-interface/public host publication.
|
||||
- Routing: local-first with external processing globally disabled by default and additionally requiring per-user opt-in when later approved.
|
||||
- Workers: durable AI worker remains disabled until queue/privacy/entitlement/notification production checks pass.
|
||||
|
||||
The deployed runtime does not yet have this topology or routing contract.
|
||||
|
||||
## Backup gate
|
||||
|
||||
The observed 21 MariaDB dumps are gzip-valid but old and database-only. Before an approved deployment:
|
||||
|
||||
1. create a new out-of-band MariaDB dump through the supported script;
|
||||
2. record its UTC time, size, duration, client/server versions, and SHA-256 without recording credentials;
|
||||
3. snapshot/copy the complete JobTracker data volume, including attachments, CV artifacts, generated exports, and account exports;
|
||||
4. separately secure data-protection keys and the account-deletion tombstone ledger;
|
||||
5. record the exact non-secret Compose/image/model digests;
|
||||
6. restore all components to an isolated scratch stack and complete the documented verification matrix;
|
||||
7. retain the pre-change set until the observation window and rollback decision expire.
|
||||
|
||||
Database-only success is not a complete backup.
|
||||
|
||||
## Benchmark sequence after approval
|
||||
|
||||
1. Capture idle host/container/GPU metrics and `ollama ps`.
|
||||
2. Keep `qwen2.5:7b` as the baseline; do not pull multiple candidates together.
|
||||
3. Run one synthetic request, then verify loaded model, GPU/CPU split, VRAM, RAM/swap, temperature, power, latency, tokens/second, JSON validity, and logs free of content.
|
||||
4. Stop if swap grows materially, memory pressure remains elevated, root free space drops below the agreed threshold, the GPU overheats/throttles, containers become unhealthy, or latency breaches the package threshold.
|
||||
5. Run the checked-in synthetic evaluation set at 4K and 8K context. Try 16K only after measured headroom.
|
||||
6. Repeat enough times to distinguish warm/cold load and variance. One successful response is not a model decision.
|
||||
7. Test queue congestion, timeout, restart, local outage, privacy opt-out, Free/Pro denial, and two-tenant isolation before any rollout.
|
||||
8. Record exact model tag, digest, license, quantization, context, options, resource use, quality and failure categories.
|
||||
|
||||
Do not change production's selected model merely because it fits in VRAM.
|
||||
|
||||
## Staged rollout
|
||||
|
||||
1. Deploy configuration/topology/security changes with AI workers and account deletion still disabled.
|
||||
2. Verify admin version badge equals the deployed commit and confirm direct ports are closed.
|
||||
3. Re-run anonymous/authenticated health and core non-AI smoke checks.
|
||||
4. Run one admin-only synthetic local inference while watching CPU/RAM/swap/GPU/container health.
|
||||
5. Enable one controlled synthetic queue canary; verify durable operation, notification, cancellation, restart recovery, entitlement and tenant boundaries.
|
||||
6. Observe for the agreed period before increasing concurrency above one.
|
||||
7. External fallback remains off until explicit privacy/provider approval and separate synthetic verification.
|
||||
8. Account deletion remains off until its retention/restore/provider/cache rehearsal is complete.
|
||||
|
||||
## Rollback plan
|
||||
|
||||
Rollback must use the supported deploy/runbook path, not ad-hoc container deletion.
|
||||
|
||||
1. Stop admission of new AI work; leave durable queued records intact.
|
||||
2. Request cancellation/drain and wait for the bounded deadline. Do not kill the database.
|
||||
3. Set AI worker and external processing gates back to false.
|
||||
4. Redeploy the recorded pre-change commit/images through the approved script, preserving the reviewed production script state.
|
||||
5. Keep the pre-existing `qwen2.5:7b`; changing application selection does not require deleting candidate model files.
|
||||
6. Restore database/files/keys only if application rollback is insufficient and only from the verified complete pre-change set. Apply deletion tombstones before readiness.
|
||||
7. Verify container health, direct-port closure, login, application counts, Career/CV, attachments, non-AI behavior, operation reconciliation, and admin version.
|
||||
8. Record incident times and sanitized failure categories. Do not copy prompts or user content into the report.
|
||||
|
||||
Expected interruption is one controlled application-container recreation plus model cold-load time. Exact duration remains unmeasured and must be captured during the approved rehearsal.
|
||||
|
||||
## Outstanding approvals
|
||||
|
||||
- network/firewall/container mutation to close Ollama and frontend host ports;
|
||||
- complete backup and scratch restore authority;
|
||||
- retention/tombstone/legal decisions;
|
||||
- model pull and synthetic production benchmark;
|
||||
- deployment and worker activation;
|
||||
- external-provider fallback and any real-provider checks.
|
||||
|
||||
Until those approvals and gates are complete, the correct state is the current read-only evidence plus disabled release-branch features—not a partial production rollout.
|
||||
@@ -1,8 +1,8 @@
|
||||
# Production AI validation
|
||||
|
||||
Updated: 2026-08-02
|
||||
Updated: 2026-08-15
|
||||
|
||||
Status: `BLOCKED`. No production access, deployment, Ollama installation, model pull, benchmark, provider call or configuration change has been performed by this programme.
|
||||
Status: `BLOCKED`. Sanitized read-only production inventory is complete. No deployment, Ollama installation, model pull, inference/benchmark, provider call, backup/restore, restart or configuration/network change has been performed by this programme.
|
||||
|
||||
## Required before any production change
|
||||
|
||||
@@ -16,7 +16,9 @@ Status: `BLOCKED`. No production access, deployment, Ollama installation, model
|
||||
- local-only bind/network proof for Ollama;
|
||||
- canary, monitoring and rollback procedure.
|
||||
|
||||
Repository-side evidence will be linked here as packages PROD-001 through PROD-004 advance. Until then, production state is unchanged and unverified.
|
||||
PROD-001 confirms the remembered 32 GiB / GTX 1060 6GB hardware, current Ollama/model/runtime and healthy application containers. It also confirms rollout stop conditions: all-interface JobTracker Ollama/frontend listeners, unlimited container resources, an old direct-Gemini sidecar, a dirty deploy-script mode and database-only backups ending 2026-08-02. See `production-ai-hardware-assessment.md` and `production-ai-rollout-and-rollback.md`. Production state was not changed.
|
||||
|
||||
PROD-003 now has a tested plan-only synthetic benchmark harness and an honest empty result table in `ollama-model-benchmark.md`. No candidate was pulled or called; no primary/fallback model is selected.
|
||||
|
||||
BG-001 tenant-safe owner execution is implemented locally, but job enrichment remains default-off. It must not be enabled until durable operations, Pro entitlement and AI privacy policy pass their own gates; see `docs/verification/bg-001-tenant-workers.md`.
|
||||
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
Updated: 2026-08-15
|
||||
|
||||
Status: `IN PROGRESS`. Generated-output ownership and the readable export are implemented. The disabled deletion lifecycle remains to be implemented.
|
||||
Status: `IMPLEMENTED — NOT VERIFIED`. Generated-output ownership, readable export, and the dark-launched deletion lifecycle are implemented and locally verified. Production activation remains blocked by retention and restore policy.
|
||||
|
||||
## Owner inventory boundary
|
||||
|
||||
@@ -29,22 +29,35 @@ No existing generated file is moved or guessed. Legacy shared-date outputs stay
|
||||
- The Settings Backup tab presents the readable export separately from the application-key-encrypted operational backup and explains recent sign-in without weakening the API rule.
|
||||
- Temporary ZIPs live under an opaque owner root and are opened with delete-on-close when returned by the controller.
|
||||
|
||||
## Checkpoint 3 — disabled, retryable deletion lifecycle
|
||||
|
||||
- Additive Identity status plus durable request/file-ledger tables track request, stage, retry, file checksum, row count, warnings, and sanitized failure state. SQLite is EF-generated; the MariaDB migration uses explicit bounded types and its generated script was reviewed.
|
||||
- `AccountLifecycle:DeletionEnabled` is explicitly `false` by default. Both self-service and admin requests fail safely while disabled; the old admin Identity-only delete path has been removed.
|
||||
- A valid request immediately marks the account pending, rotates its security stamp, revokes sessions and trusted devices, unpublishes public CVs, cancels queued work, and requests cancellation of running work. Pending users cannot sign in, complete 2FA, or reuse an existing local session.
|
||||
- Self-service requires an exact server-provided `DELETE <email>` phrase and a session created within 15 minutes. Last-administrator protection remains enforced. Admin deletion uses the same coordinator and exact-email confirmation header.
|
||||
- One managed-root inventory covers attachments, CV artifacts, file-backed avatars, generated CVs, daily exports, and previously generated account-export ZIPs. Files move to same-volume quarantine markers before any database delete; partial file failure restores them and leaves rows untouched.
|
||||
- Database deletion is explicit and transactional across all owned application, Career, CV, correspondence, provider-credential, queue/notification, security, and Identity rows. Request/file ledgers survive for retry and audit. Commit-acknowledgement ambiguity leaves files quarantined and replays deletion instead of risking data resurrection.
|
||||
- Purge clears backend in-memory caches, removes quarantined files, then writes a minimal pseudonymous tombstone to a separate append-only JSONL root. Invalid ledger records fail closed.
|
||||
- Startup stages restored identities matching tombstones before readiness, and the background reconciler resumes all durable non-completed requests even while new deletion requests remain disabled.
|
||||
- Settings explains the disabled production gate; when enabled it uses the reusable prompt dialog and exact phrase. Admin user deletion supplies the matching account email.
|
||||
|
||||
## Verification
|
||||
|
||||
- Owner-storage focused CV/export/controller/background tests: 77/77.
|
||||
- Readable-export focused backend/API tests: 11/11, including real SQLite, two-owner isolation, file inclusion, every checksum and secret-redaction sentinels.
|
||||
- Full backend: 650/650.
|
||||
- Frontend export/Settings tests: 4/4; full frontend 58 suites/234 tests.
|
||||
- Account lifecycle/export/auth/admin focused backend/API tests: 21/21. Five real-SQLite deletion tests cover disabled requests, exact confirmation/recent authentication, immediate lockout/idempotency, two-owner row/file isolation, quarantine failure, repeat reconciliation, tombstone creation, and restored-backup replay.
|
||||
- Full backend: 657/657.
|
||||
- Frontend export/Settings/admin tests: 8/8; full frontend 58 suites/237 tests.
|
||||
- Backend build: pass, zero warnings/errors.
|
||||
- Optimized frontend build/TypeScript: pass.
|
||||
- Chromium: fresh Free account receives a real ZIP response with a `PK` signature and readable-export success state.
|
||||
- EF model parity: no pending model changes. MariaDB migration script generation: pass with bounded Identity/lifecycle schema and indexes.
|
||||
- Chromium: full disposable startup/application suite 9/9; fresh Free account receives a real ZIP response with a `PK` signature and readable-export success state.
|
||||
- `git diff --check`: pass aside from line-ending notices.
|
||||
|
||||
## Remaining repository work
|
||||
## Remaining external/production work
|
||||
|
||||
1. Reuse the completed owner inventory in the deletion coordinator.
|
||||
2. Add the additive deletion state/request/file schema and disabled coordinator.
|
||||
3. Add pending-account authentication/mutation gates, session/queue cancellation, provider cleanup and idempotent file quarantine/database purge.
|
||||
4. Add separate tombstone storage/replay and settings/admin UX while keeping production activation disabled.
|
||||
1. Decide backup, audit/security-log, quarantine, and tombstone retention plus any legal-hold obligations.
|
||||
2. Mount/protect the tombstone root outside restored application data and rehearse a pre-deletion backup restore with tombstone replay.
|
||||
3. Rehearse sidecar cache purge/restart and remote provider-revocation semantics using a disposable synthetic account.
|
||||
4. Only then enable admin deletion, observe it, and separately approve self-service activation.
|
||||
|
||||
Production retention, legal hold and restored-backup decisions remain recorded in `BLOCKERS.md`.
|
||||
|
||||
@@ -749,3 +749,23 @@
|
||||
- **Consequences:** users can download JSON and owned files with independently verifiable SHA-256 checksums. Missing/legacy/external/backup categories are disclosed truthfully. The service becomes the authoritative inventory seam for deletion without coupling export to deletion activation.
|
||||
- **User approval required:** No; this is the requested repository-side data lifecycle, using synthetic tests and no production data.
|
||||
- **Reversible:** Remove the endpoint/UI and service. Existing downloaded ZIPs remain user-owned files; no stored schema or data changed.
|
||||
|
||||
## DEC-076 — Dark-launch account deletion as a durable staged lifecycle
|
||||
|
||||
- **Date:** 2026-08-15
|
||||
- **Decision:** Replace admin Identity-row deletion with one disabled-by-default coordinator for self-service and admin deletion. Lock the identity immediately, quarantine verified owned files before a transactional explicit row purge, retain retry ledgers, and write a minimal pseudonymous tombstone outside the restored database before completion.
|
||||
- **Reason/evidence:** Identity-only deletion leaves owned rows, files, provider credentials, sessions and queued work behind. Cascades cannot coordinate filesystem failures or prevent an older backup from resurrecting the user. Real-SQLite tests prove two-owner isolation, safe quarantine failure, idempotence and restored-account replay.
|
||||
- **Alternatives considered:** broad cascade foreign keys; best-effort controller deletes; delete database rows before files; edit old backups in place; enable immediately after local tests. These lose retry/audit boundaries, risk stranded private files or partial erasure, and overclaim backup/provider guarantees.
|
||||
- **Consequences:** `AccountLifecycle:DeletionEnabled` remains false until retention, protected tombstone custody, cache/provider handling and a disposable restore rehearsal are approved. Existing sessions are rejected as soon as status becomes pending. Deletion request/file records intentionally survive the user row and the background worker resumes durable requests even while the feature flag blocks new ones.
|
||||
- **User approval required:** Production activation and retention decisions only. Repository implementation uses synthetic data and remains inert by default.
|
||||
- **Reversible:** Disable requests (the default), allow in-flight reconciliation to finish, then downgrade the additive migration only after no request remains. Never remove the separate tombstone ledger while backups capable of restoring deleted identities still exist.
|
||||
|
||||
## DEC-077 — Make production model benchmarking plan-only by default
|
||||
|
||||
- **Date:** 2026-08-15
|
||||
- **Decision:** Use one standard-library harness over the checked-in synthetic evaluation set. It plans without network access by default, requires `--execute` plus an explicit output for inference, never pulls/deletes models, permits only loopback or an explicitly opted-in literal private IP, and persists hashes/metrics/scores rather than raw inputs/prompts/outputs.
|
||||
- **Reason/evidence:** The measured production host can run bounded tests, but its network and backup stop conditions are open and model execution is not authorized. A ready harness removes future ad-hoc prompt/data/report handling without silently widening current authority.
|
||||
- **Alternatives considered:** benchmark immediately over SSH; use production CV/email content; install a benchmark dependency/framework; persist raw responses for later scoring; auto-pull every candidate. These violate current authority/privacy boundaries or add avoidable supply-chain/resource risk.
|
||||
- **Consequences:** candidate selection remains blocked and no model decision is claimed. Approved runs can compare 4K/8K timing, token rate, JSON/constraint quality and Ollama VRAM metadata reproducibly while host GPU/RAM metrics are captured separately.
|
||||
- **User approval required:** Yes before any model pull, production inference, or production report execution. No approval is required for plan-only fixture validation.
|
||||
- **Reversible:** Remove the script/tests/template; no dependency, model, application, production, schema or configuration state changed.
|
||||
|
||||
@@ -2,21 +2,21 @@
|
||||
|
||||
Updated: 2026-08-15
|
||||
|
||||
- **Overall programme status:** Active. Eight packages are locally verified; twenty-four are implemented with verification incomplete; SEC-009 is in progress. The prioritized admin-only version indicator and every immediate repository/browser item are implemented on the release branch; remote and production verification remain.
|
||||
- **Current work package:** `SEC-009` — complete readable export and account deletion lifecycle (`IN PROGRESS`). Owner-scoped generated paths and the real-SQLite-verified readable ZIP are complete; proceed with the disabled deletion lifecycle while retention/restore policy blocks production activation.
|
||||
- **Overall programme status:** Active but externally blocked. Eight packages are locally verified and twenty-five are implemented with verification incomplete. The prioritized admin-only version indicator, every immediate repository/browser item, SEC-009, the PROD-001 read-only inventory, and the PROD-003 safe benchmark harness are complete on the release branch.
|
||||
- **Current work package:** None. Every remaining package now requires a user/operator decision, authorized production mutation/restore/provider action, or explicit package-index access.
|
||||
- **Completed work packages:** None are `DONE`; all repository security/AI packages still have applicable browser, provider and/or production gates.
|
||||
- **Locally verified work:** SEC-001, SEC-002, SEC-003, SEC-005A, CORE-001, PROD-002, DEP-001 and VER-001 (`VERIFIED LOCALLY`).
|
||||
- **Implemented, verification incomplete:** SEC-004, SEC-005B, SEC-008, CORE-002, BG-001, OPS-001A/B/C, POL-001/002, AI-001/002/003/004, UX-001/002/003, QA-001, CAREER-001/002, MAIL-001, JOBS-001/002 and PRODUCT-001 (`IMPLEMENTED — NOT VERIFIED`). Their safe repository/browser scope is implemented; production/native-device/provider gates remain where recorded.
|
||||
- **Implemented, verification incomplete:** SEC-004, SEC-005B, SEC-008, SEC-009, CORE-002, BG-001, OPS-001A/B/C, POL-001/002, AI-001/002/003/004, UX-001/002/003, QA-001, CAREER-001/002, MAIL-001, JOBS-001/002 and PRODUCT-001 (`IMPLEMENTED — NOT VERIFIED`). Their safe repository/browser scope is implemented; production/native-device/provider/retention gates remain where recorded.
|
||||
- **Production-verified work:** None.
|
||||
- **Blocked work:** SEC-006 parser upgrades remain outside the scoped frontend advisory permission; PROD-001/003/004 and REL-001 require documented production access and unfinished dependencies. Real provider, SMTP/MariaDB and production environments are unavailable; DEP-001 awaits approved merge/live verification. The in-app browser is available for local UI checks.
|
||||
- **Blocked work:** SEC-006 parser upgrades remain outside the scoped frontend advisory permission; PROD-001/003/004 and REL-001 require network/backup/model/deployment authority and unfinished dependencies. Real provider and live deletion/restore checks remain gated; DEP-001 awaits approved merge/live verification.
|
||||
- **Deferred work:** None. Conditional multi-replica coordination, model deletion, realtime operation delivery and unrelated production changes remain outside current packages.
|
||||
- **Immediate order:** SEC-009 owner inventory/export first, then its disabled deletion lifecycle. The eight-item immediate queue is complete locally: admin version (`a6cffe0`), Career persistence (`f0b9b22`), CV contrast (`3b86ea2`), JOBS-002 (`deed948`), accessibility (`a7c2549`), PRODUCT-001 (`a25c31b`), VER-001 and tracking reconciliation. External-only work remains skipped, not allowed to stall repository progress.
|
||||
- **Status counts:** 8 `VERIFIED LOCALLY`; 24 `IMPLEMENTED — NOT VERIFIED`; 1 `IN PROGRESS`; 1 `NOT STARTED`; 5 `BLOCKED`; 0 `DONE`; 0 `DEFERRED`.
|
||||
- **Test status:** backend 650/650; frontend 58/58 suites and 234/234 tests; AI sidecar 22/22; optimized production build/TypeScript; Docker Compose config; safe-failure deployment preflight; and the preceding Playwright 9/9 pass. A fresh Chromium Free account now also downloads the real readable ZIP response. npm audit 0 evidence remains current because the lockfile did not change. Historical JT-019 and Jest force-exit/open-handle behavior remain recorded.
|
||||
- **Immediate order:** the eight-item immediate queue is complete locally: admin version (`a6cffe0`), Career persistence (`f0b9b22`), CV contrast (`3b86ea2`), JOBS-002 (`deed948`), accessibility (`a7c2549`), PRODUCT-001 (`a25c31b`), VER-001 and tracking reconciliation. SEC-009 is complete at `842e793`; PROD-001 read-only evidence and the PROD-003 plan-only harness are complete pending this documentation commit. No further independent implementation remains.
|
||||
- **Status counts:** 8 `VERIFIED LOCALLY`; 25 `IMPLEMENTED — NOT VERIFIED`; 0 `IN PROGRESS`; 1 `NOT STARTED`; 5 `BLOCKED`; 0 `DONE`; 0 `DEFERRED`.
|
||||
- **Test status:** backend 657/657; frontend 58/58 suites and 237/237 tests; AI sidecar 22/22; Ollama benchmark harness 4/4 plus safe dry-run; optimized production build/TypeScript; EF model parity and MariaDB migration-script generation; Docker Compose config; safe-failure deployment preflight; and Playwright 9/9. Five real-SQLite deletion tests cover lockout, isolation, quarantine failure and restored-backup replay. npm audit 0 evidence remains current because the lockfile did not change. Historical JT-019 and Jest force-exit/open-handle behavior remain recorded.
|
||||
- **Deployment status:** Gitea pull-request run 609 passed the complete CI job in 4m20s. Deploy was intentionally skipped because the workflow deploys only a `push` to `main`; live remains unchanged. No merge/deployment was performed directly, no production migrations were run and the AI operation worker remains disabled by default.
|
||||
- **Production status:** Unchanged and unverified. No provider/model call, model pull, external request or paid API occurred.
|
||||
- **Production status:** State unchanged. Sanitized read-only SSH inventory was performed; no logs, prompts, private rows/content or secret values were read, and no provider/model call, model pull, service restart, file/config change, backup, restore, migration or deployment occurred. It confirmed all-interface Ollama/frontend listeners and stale database-only backups as rollout blockers.
|
||||
- **Known regressions:** None found by automated/local browser checks. Jest still needs `--forceExit` and reports its existing open-handle notice. Email-provider/send tests are fake/local only; real delivery is not claimed. Current MAIL browser evidence is 1280×720 only because the browser surface could not resize or perform native Tab traversal. Interrupted attempts are aged after 15 minutes and notified without retry; the five-minute scan is unmeasured on a large ledger. Direct clean EF-only SQLite migration still hits the pre-existing historical blank-chain defect before later migrations; normal startup owns reconciliation. Cross-feature monthly AI usage accounting remains a rollout gap.
|
||||
- **Outstanding security findings:** JT-001 repository ownership remains High deployment risk until migration/inventory/provider checks; production portion of JT-002; JT-006/JT-009 and associated JT-011/JT-012/JT-022 prerequisites. JT-005 foundations are implemented; AI worker activation awaits controlled rollout. JT-007/JT-008/JT-010 lack browser/provider/production verification.
|
||||
- **Outstanding security findings:** JT-001 repository ownership remains High deployment risk until migration/inventory/provider checks; production portion of JT-002; JT-006 and SEC-009 production retention/restore plus JT-011/JT-012/JT-022 prerequisites. JT-005 foundations are implemented; AI worker activation awaits controlled rollout. Production still exposes ports contrary to the release-branch contract, and JT-007/JT-008/JT-010 lack provider/production verification.
|
||||
|
||||
## Current evidence
|
||||
|
||||
@@ -31,6 +31,7 @@ Updated: 2026-08-15
|
||||
- `docs/verification/core-001-sqlite-provider-parity.md`
|
||||
- `docs/verification/core-002-route-uniqueness.md`
|
||||
- `docs/verification/sec-008-attachment-consistency.md`
|
||||
- `docs/verification/sec-009-account-lifecycle.md`
|
||||
- `docs/verification/bg-001-tenant-workers.md`
|
||||
- `docs/verification/ops-001a-durable-operations.md`
|
||||
- `docs/verification/ops-001b-notifications.md`
|
||||
@@ -52,4 +53,7 @@ Updated: 2026-08-15
|
||||
- `docs/verification/product-001-honest-plans.md`
|
||||
- `docs/verification/ver-001-complete-regression.md`
|
||||
- `docs/verification/prod-002-ai-evaluation.md`
|
||||
- `docs/production/production-ai-hardware-assessment.md`
|
||||
- `docs/production/production-ai-rollout-and-rollback.md`
|
||||
- `docs/production/ollama-model-benchmark.md`
|
||||
- `docs/work-programmes/master-work-plan.md`
|
||||
|
||||
@@ -16,7 +16,7 @@ Allowed statuses are `NOT STARTED`, `IN PROGRESS`, `IMPLEMENTED — NOT VERIFIED
|
||||
|
||||
`DONE` requires every applicable acceptance criterion, focused and regression tests, browser/accessibility/theme/mobile checks, tenant and entitlement checks, documentation, migration/rollback evidence, and production verification. Repository-only work that still requires production is at most `VERIFIED LOCALLY`.
|
||||
|
||||
Exactly one implementation item may be `IN PROGRESS`. As of this revision it is **SEC-009**.
|
||||
At most one implementation item may be `IN PROGRESS`. No package is currently in progress: every remaining package is blocked by an external decision, production mutation, provider interaction, or explicitly unapproved package-index access.
|
||||
|
||||
## Consolidated dependency order
|
||||
|
||||
@@ -268,11 +268,11 @@ This queue records the highest-value work that can proceed without production cr
|
||||
- **Required tests:** two users/every entity, manifest/checksums/redaction, fault/restart/idempotence, provider failures, disposable restore replay.
|
||||
- **Required browser verification:** disposable self/admin export/delete and confirmations.
|
||||
- **Required production verification:** backup retention/tombstone rehearsal before self-service enablement.
|
||||
- **Status:** `IN PROGRESS`.
|
||||
- **Status:** `IMPLEMENTED — NOT VERIFIED`.
|
||||
- **Blocker:** legal/operator retention and production restore decisions block activation, not the repository-side disabled/dark launch.
|
||||
- **Evidence:** audit JT-009 inventory/design; `docs/verification/sec-009-account-lifecycle.md`; V-173/V-174. Owner-scoped generated storage and complete redacted readable ZIP pass real-SQLite two-owner, focused API/UI, full backend/frontend, build and Chromium checks.
|
||||
- **Commit:** none.
|
||||
- **Remaining work:** reuse the completed inventory for the additive disabled deletion coordinator, tombstone replay, UI and failure/restart verification. Production activation remains blocked by retention/restore policy.
|
||||
- **Evidence:** audit JT-009 inventory/design; `docs/verification/sec-009-account-lifecycle.md`; V-173/V-174/V-175. Owner-scoped generated storage, complete redacted readable ZIP, immediate lockout, transactional owner-isolated row/file purge, retry, fail-closed tombstones and restored-backup replay pass real-SQLite, focused API/UI, full backend/frontend, build and Chromium checks.
|
||||
- **Commit:** `842e793`.
|
||||
- **Remaining work:** repository scope is complete. Production activation remains blocked by retention/legal decisions, protected tombstone custody, restored-backup rehearsal, provider/cache semantics and staged disposable-account rollout.
|
||||
|
||||
### CORE-001 — Restore default SQLite/MariaDB behavior parity
|
||||
|
||||
@@ -467,10 +467,10 @@ This queue records the highest-value work that can proceed without production cr
|
||||
- **Required browser verification:** none.
|
||||
- **Required production verification:** this item is itself production read-only verification.
|
||||
- **Status:** `BLOCKED`.
|
||||
- **Blocker:** repository docs state the local environment has no production route and CI SSH secrets are unavailable; no documented callable host/credential is present.
|
||||
- **Evidence:** `docs/deployment/backup-restore.md` and `docs/operations/production-backup-verification.md` explicitly record the access gap.
|
||||
- **Blocker:** read-only inventory is complete, but Ollama/frontend host ports are published on all interfaces, complete current backups/restores are unproved, the production checkout has an unreviewed mode-only deploy-script change, and closing those gaps requires approved production mutations.
|
||||
- **Evidence:** `docs/production/production-ai-hardware-assessment.md`; `docs/production/production-ai-rollout-and-rollback.md`; V-176. Sanitized read-only SSH measured hardware, GPU, storage, Docker, networks, selected provider/model, health, limits and backup presence without reading secrets/logs/content or changing state.
|
||||
- **Commit:** none.
|
||||
- **Remaining work:** create sanitized report template locally; operator/documented access required for measured completion.
|
||||
- **Remaining work:** approve and close the network/port, complete-backup/scratch-restore, dirty-checkout and retention gates; then repeat the sanitized inventory to verify the safe target state.
|
||||
|
||||
### PROD-002 — AI workload inventory and synthetic evaluation set
|
||||
|
||||
@@ -503,10 +503,10 @@ This queue records the highest-value work that can proceed without production cr
|
||||
- **Required browser verification:** none.
|
||||
- **Required production verification:** measured on actual machine; local workstation results are labeled separately.
|
||||
- **Status:** `BLOCKED`.
|
||||
- **Blocker:** PROD-001 production access/hardware inventory.
|
||||
- **Evidence:** none yet.
|
||||
- **Blocker:** production model pull/inference is not authorized, and PROD-001's all-interface Ollama plus incomplete backup/restore stop conditions remain open.
|
||||
- **Evidence:** `scripts/run-ollama-evaluation.py`; `scripts/test-ollama-evaluation.py`; `docs/production/ollama-model-benchmark.md`; V-177. Plan-only harness tests 4/4 and validates an eight-request 4K/8K Strategy plan without a model call.
|
||||
- **Commit:** none.
|
||||
- **Remaining work:** repository harness can be prepared after PROD-002.
|
||||
- **Remaining work:** after approvals and PROD-001 safety closure, verify candidate metadata/licenses, pull one candidate at a time, execute repeated synthetic 4K/8K benchmarks, measure active GPU/RAM/offload and complete the evidence-based model decision.
|
||||
|
||||
### PROD-004 — Local-model rollout, fallback, observability and operations
|
||||
|
||||
|
||||
@@ -2,17 +2,17 @@
|
||||
|
||||
Updated: 2026-08-15
|
||||
|
||||
- **Exact current task:** begin SEC-009 with the owner inventory/readable export, then implement the deletion lifecycle behind a disabled production gate.
|
||||
- **Last completed step:** implemented a recent-authenticated, rate-limited, redacted readable account ZIP with complete safe row/file inventory, checksums, warnings and Settings UX.
|
||||
- **Files currently modified:** account export service/API/rate policy, Backup Settings UI/translations, real-SQLite/backend/frontend/browser tests and SEC-009 evidence.
|
||||
- **Commands already run:** readable-export backend/API 11/11, backend 650/650, frontend focused 4/4 and full 234/234, backend/frontend builds and targeted Chromium ZIP response. The preceding VER-001 sidecar/Compose/preflight/full Playwright evidence remains current.
|
||||
- **Test results:** all listed local gates pass. Provider/native-AT/production cells remain explicitly partial, not run or blocked. Jest retains the documented force-exit/open-handle notice.
|
||||
- **Exact current task:** no independent implementation remains. SEC-009, PROD-001 read-only inventory/reporting and the PROD-003 plan-only benchmark harness are complete; continue only after a recorded blocker is authorized/resolved.
|
||||
- **Last completed step:** measured production hardware/runtime/backups read-only, documented rollout stops, then added a synthetic-only benchmark harness that defaults to no network execution.
|
||||
- **Files currently modified:** master progress/work-plan/handoff/decisions/blockers/evidence; SEC-009 verification; production hardware/rollout/benchmark reports; production backup checkpoint; Ollama benchmark script/tests.
|
||||
- **Commands already run:** SEC-009 focused backend 21/21, backend 657/657, frontend focused 8/8 and full 237/237, optimized build, EF parity, MariaDB script generation and Chromium 9/9; benchmark harness 4/4 plus plan-only dry run; sanitized read-only SSH inventory and gzip integrity across 21 existing dumps.
|
||||
- **Test results:** all repository gates pass. PROD-001 is PASS/PARTIAL because measured all-interface ports and incomplete/stale backup/restore evidence fail its safety acceptance. No model inference was run. Jest retains the documented open-handle notice.
|
||||
- **Services currently running:** none on task-owned ports 3000/5202. Playwright stopped its disposable API/Next servers. Pre-existing Docker services were not changed.
|
||||
- **Temporary files or processes:** no task-owned process is running and the failed disposable migration database was removed. Existing synthetic browser evidence/account and startup-created local backup remain documented. No provider account, real email, private content, paid service or production service was accessed.
|
||||
- **Production changes currently active:** none. No deployment, migration, provider connection/sync/send or production payload occurred.
|
||||
- **Rollback status:** downgrade `20260810080858_AddEmailDraftClientRequestId`, then `20260810075206_AddEmailDrafts`, before reverting draft commits; then follow the existing MAIL rollback order (`ee5ef7e`, `449faeb`, `123fc55`/`e9937ac`, ledger downgrade before `653f011`). No production migration/deploy/provider grant occurred.
|
||||
- **Uncommitted changes:** V-174 readable-export code/tests/docs; no dependency, schema or production configuration change. V-173 is pushed as `cdcc716`.
|
||||
- **Known failures:** live deployment is not verified because PR deploy is intentionally skipped and the active branch is not approved for merge. Draft export/API/UI, full thread/category actions and non-Gmail review remain; existing accounts need re-consent and IMAP stays read-only. A clean full-chain SQLite apply fails in the pre-existing JT-019 migration before the new draft migration. Browser/provider/MariaDB/production unavailable or unverified; recovery scan performance is unmeasured at large ledger scale; Jest open handles; SEC-006 parser dependency work is still separately gated; parser isolation remains SEC-007.
|
||||
- **Exact next action:** complete full Chromium after the export addition, commit/push V-174, then add the disabled additive deletion state and coordinator using the same owner inventory.
|
||||
- **Work that can continue independently:** SEC-009 repository-side owner inventory/export and disabled deletion lifecycle. UX/JOBS/PRODUCT production, MAIL provider mutations, SEC-006/007 and PROD packages retain their recorded external gates.
|
||||
- **Decisions still required from the user:** none for synthetic/code-inspected repository work. Any provider connection or send test, internet/package upgrades, private data, external/paid providers and production actions retain explicit approval/safety gates; SEC-009 retention/legal policy remains unresolved.
|
||||
- **Temporary files or processes:** no task-owned process is running. Playwright stopped its disposable API/Next servers. No raw benchmark output/evidence file was created.
|
||||
- **Production changes currently active:** none. Read-only SSH observed metadata/health/selected non-secret settings and backup integrity only. No log/private-row/content/secret read, deployment, migration, provider call, inference, model pull, restart, backup, restore or production file/config change occurred.
|
||||
- **Rollback status:** SEC-009 is additive migration `20260815164027_AddAccountDeletionLifecycle`. Keep deletion disabled, reconcile any durable request, and retain tombstones before downgrade. `842e793` is pushed. Production still runs `de937d25dc5e` / version `157`; its checkout has a pre-existing mode-only `deploy/deploy.sh` change that must be preserved/reviewed.
|
||||
- **Uncommitted changes:** documentation and the plan-only Ollama benchmark harness/tests following pushed SEC-009 commit `842e793`; no dependency, model, application runtime, schema or production state change in this checkpoint.
|
||||
- **Known failures:** PR 28's newest remote CI is not yet confirmed. Production publishes frontend 3000 and JobTracker Ollama 11434 on all interfaces; latest observed database-only backup is 2026-08-02; no complete files/keys/tombstone restore proof; root is 83% used; deployed AI sidecar is old direct-Gemini behavior; production script mode is dirty. SEC-006 internet access, SEC-007 dependency, provider/re-consent, Stripe price, signup, retention/legal, backup/restore, model execution/deployment and legacy cutover decisions remain recorded blockers. Historical JT-019 and Jest open handles remain.
|
||||
- **Exact next action:** after this documentation/harness checkpoint is committed and pushed, stop. Resume from the highest-priority blocker the user authorizes: recommended first is production network plus complete backup/scratch-restore safety, then bounded synthetic model benchmarking.
|
||||
- **Work that can continue independently:** none identified after the PROD-003 harness. Do not bypass blockers by pulling models, changing ports/firewalls, restoring data, contacting providers, or using package indexes without explicit authority.
|
||||
- **Decisions still required from the user:** production network/port mutation; complete backup and scratch restore; retention/tombstone/legal policy; model pull/synthetic production inference; deployment/worker activation; parser package-index access; provider/Stripe/signup actions; legacy cutover timing.
|
||||
|
||||
Reference in New Issue
Block a user