feat(ai): add safe benchmark harness

docs/production/ollama-model-benchmark.md

@@ -0,0 +1,57 @@
+# Ollama model benchmark
+
+Updated: 2026-08-15
+
+Status: `BLOCKED`. The synthetic benchmark harness is implemented, but no candidate was pulled and no inference was run. Production benchmarking requires the network, backup, model-pull and synthetic-execution approvals recorded in `BLOCKERS.md` and `production-ai-rollout-and-rollback.md`.
+
+## Measured baseline inventory
+
+| Candidate | Size | Quantisation | VRAM | RAM | GPU offload | Context | Load / first token / total / tok/s | Quality / JSON / Norwegian / failure | Licence | Recommendation |
+|---|---:|---|---:|---:|---|---:|---|---|---|---|
+| `qwen2.5:7b` (installed) | 4.7 GB | Q4_K_M; 7.6B parameters | Not measured loaded | Not measured loaded | Not measured | advertised 32K; test 4K/8K first | Not run | Not run | Apache-2.0 | Baseline only; do not select without measured JobTracker results |
+| `qwen3.5:4b` | Unknown locally | Not inspected | Not measured | Not measured | Not measured | 4K/8K first | Not run | Not run | Verify before pull/use | Leading candidate from the source plan, not an approved model |
+| `qwen3:4b` | Unknown locally | Not inspected | Not measured | Not measured | Not measured | 4K/8K first | Not run | Not run | Verify before pull/use | Candidate only |
+| `gemma3:4b` | Unknown locally | Not inspected | Not measured | Not measured | Not measured | 4K/8K first | Not run | Not run | Verify hosted-use terms before pull/use | Candidate only |
+
+Optional 8B/9B candidates remain excluded until the 4B/baseline measurements prove headroom on the 6 GB GPU. Cloud-labelled Ollama models are out of scope.
+
+## Harness
+
+`scripts/run-ollama-evaluation.py`:
+
+- reads only the checked-in `syntheticOnly=true` evaluation fixture;
+- runs one exact installed model and never pulls/deletes a model;
+- defaults to plan-only mode; network calls require `--execute` and an explicit output path;
+- accepts loopback only unless a literal private IP is explicitly opted in;
+- supports 4K/8K/conditional context comparisons, repetition, temperature, output limit and keep-alive;
+- streams responses to measure first-token and wall latency;
+- records Ollama load/prompt/generation timings, token rate and `/api/ps` loaded/VRAM sizes when available;
+- scores must-contain, forbidden-claim, strict-JSON and required-key constraints;
+- writes hashes and scores, never raw fixture input, prompts, or model output.
+
+Plan-only example (safe; no network call):
+
+```bash
+python scripts/run-ollama-evaluation.py \
+  --task STRATEGY \
+  --model qwen2.5:7b
+```
+
+Approved local/private execution shape:
+
+```bash
+python scripts/run-ollama-evaluation.py \
+  --task STRATEGY \
+  --model qwen2.5:7b \
+  --context 4096 \
+  --context 8192 \
+  --repeat 3 \
+  --execute \
+  --output evidence/strategy-qwen2.5-7b.json
+```
+
+The output directory should remain outside source control unless a sanitized evidence path is explicitly approved.
+
+## Decision gate
+
+No primary or fallback is selected. A decision requires repeated measured results for every relevant generative task, exact tag/digest/version/license/quantization, active GPU offload, peak VRAM/RAM/swap, cold/warm load, first-token/total latency, tokens/second, JSON success, English/Norwegian quality, factuality/injection/failure behavior, and stability. Deterministic tasks in `docs/ai/workload-inventory.md` remain non-model work regardless of benchmark scores.

docs/production/production-ai-hardware-assessment.md

@@ -0,0 +1,121 @@
+# Production AI hardware and runtime assessment
+
+Updated: 2026-08-15
+
+Status: read-only production inventory complete. No service, file, firewall, model, database, backup, container, or configuration value was changed. Host identity, addresses, credentials, environment secrets, logs, prompts, and private application content are intentionally omitted.
+
+## Executive result
+
+The remembered hardware profile is accurate: the host has about 32 GiB RAM and one NVIDIA GeForce GTX 1060 6GB. The machine has ample capacity on the Docker/model filesystem for bounded sequential benchmarks, but rollout is not safe yet:
+
+- the JobTracker Ollama container publishes port 11434 on every IPv4 and IPv6 host interface;
+- the deployed application is older than the release branch and still selects Gemini directly in the AI sidecar rather than the release branch's disabled-external, local-first policy;
+- no CPU, memory, PID, or read-only-root limits are applied to the four application containers;
+- database backups are small and gzip-valid, but the newest observed file is 13 days old and no JobTracker timer/cron entry was found;
+- observed backups cover MariaDB only, not the owner-file volume, data-protection keys, protected deletion tombstones, or non-secret recovery configuration.
+
+These are rollout stop conditions, not permission to change production.
+
+## Measured host
+
+| Area | Read-only measurement |
+|---|---|
+| OS | Ubuntu 24.04.4 LTS, x86_64 |
+| Kernel | Linux 6.8.0-110-generic |
+| Time zone | Europe/Oslo |
+| Uptime | 3 weeks 6 days at capture |
+| Load | 3.25 / 2.80 / 2.29 |
+| CPU | Intel Core i5-8600 @ 3.10 GHz; 1 socket, 6 physical/logical cores, 1 thread/core |
+| RAM | 31 GiB total; 11 GiB used; 20 GiB available |
+| Swap | 8 GiB total; 2.1 GiB used |
+| Memory pressure | PSI `some` avg10 0.16%; `full` avg10 0.06% |
+| Shell open-file limit | 1,024 |
+| Root filesystem | ext4, 217 GiB total, 171 GiB used, 36 GiB available (83% used) |
+| Docker/model filesystem | ext4, 1.8 TiB total, 316 GiB used, 1.4 TiB available (19% used) |
+| Docker | Engine 29.3.0, API 1.54; data root on the large filesystem |
+
+Other workloads share the host. The available-memory and pressure measurements are therefore more useful than total RAM alone; benchmarks must capture concurrent load rather than assuming an idle dedicated server.
+
+## GPU
+
+| Area | Read-only measurement |
+|---|---|
+| GPU | NVIDIA GeForce GTX 1060 6GB |
+| VRAM | 6,144 MiB total; 3 MiB used; 6,064 MiB reported free at capture |
+| Driver | 580.159.03 |
+| CUDA compatibility reported by driver | 13.0 |
+| Idle state | P8, 37 C, 0% utilization, about 5.5 W / 120 W |
+| GPU processes | None at capture |
+| Ollama GPU device request | All GPUs requested by the Ollama container |
+| AI sidecar GPU device request | None; health reports CPU and `gpu_available=false` |
+
+Ollama was not serving a loaded model during capture, so GPU-layer offload is not yet proven. A bounded benchmark must verify the `PROCESSOR`/offload result while a request is active; idle `nvidia-smi` is not evidence of successful GPU inference.
+
+## Ollama and models
+
+- Deployment method: Docker Compose, `ollama/ollama:latest`.
+- Runtime version: 0.31.1.
+- Captured image digest: `sha256:f1a705f2bd113fb8d15f85f7c217f0dc5f6bebda6b0cc42b82c3ad165ffcb9dc`.
+- Installed JobTracker model: `qwen2.5:7b`, model ID prefix `845dbda0ea48`, 4.7 GB.
+- Model volume use: about 4.4 GiB.
+- Loaded models: none at capture.
+- Health: Ollama reachable from the AI sidecar; model present; container healthy with zero observed restarts since 2026-08-02.
+- GPU use: not active at capture.
+
+### Exposure finding
+
+`OLLAMA_HOST=0.0.0.0:11434`, Docker publishes 11434 on all IPv4/IPv6 interfaces, and the container joins the private AI network plus broader application/shared networks. A second, non-JobTracker Ollama listener also exists on host port 11435. Internet/NAT reachability was not tested, but all-interface host publication already fails the intended localhost/private-network-only contract.
+
+The release branch removes JobTracker's host publication and shared/default network membership for its bundled Ollama. Deployment must still decide whether to use that private bundled instance or a separately controlled shared instance; it must not leave an orphaned published container.
+
+## Application deployment
+
+| Component | Runtime state | Point-in-time usage |
+|---|---|---|
+| Backend | healthy, zero restarts | 0.28% CPU; 216 MiB RAM; 25 PIDs |
+| Frontend/nginx | healthy, zero restarts | 0% CPU; 7.6 MiB RAM; 7 PIDs |
+| AI sidecar | healthy, zero restarts | 0.07% CPU; 2.24 GiB RAM; 30 PIDs |
+| JobTracker Ollama | healthy, zero restarts | 0.08% CPU; 4.56 GiB container-accounted RAM; 13 PIDs |
+
+- Deployed commit: `de937d25dc5e`; configured app version: `157`.
+- The deployed repository is on `main` with a mode-only local change to `deploy/deploy.sh`; do not reset or overwrite it without operator review.
+- All four containers use `unless-stopped`, rotating `json-file` logs at 10 MiB x 3.
+- All four have unlimited CPU/memory/PIDs, writable root filesystems, and are not privileged.
+- Backend and frontend share the normal application network; backend and sidecar share the AI network. The deployed Ollama additionally joins broader networks.
+- The sidecar is internal-only on port 8001 and requires the shared service token for non-health endpoints.
+- The deployed frontend publishes host port 3000. Backend and sidecar are not published by this Compose project. The host also has an unrelated listener on 8080.
+- nginx has no explicit connect/read/send proxy timeout in the deployed file, so defaults apply.
+
+## Current AI behavior
+
+The deployed sidecar is older than the release branch:
+
+- it loads `sshleifer/distilbart-cnn-12-6` on CPU for summarization;
+- it has Ollama configured with `qwen2.5:7b`, but no Ollama model was loaded at capture;
+- `AI_PROVIDER=gemini`, with Gemini and Groq credential variable names present (values were not read);
+- deployed code selects that provider directly and does not yet expose the release branch's `EXTERNAL_AI_ENABLED` / `AI_ROUTING_MODE` controls;
+- the backend's new durable AI worker settings are absent from the old deployment, so the release-branch defaults must be reviewed during deployment rather than inferred from this runtime.
+
+Production is therefore not currently evidence for the release branch's local-first routing, durable queue, privacy gate, or entitlement behavior.
+
+## Storage and backups
+
+- JobTracker owner-file volume: about 14 MiB total at capture.
+- Visible categories: attachments 4 KiB, CV artifacts 1.1 MiB, generated CVs 16 KiB, daily exports 196 KiB.
+- The dark-launch account-export and deletion-tombstone directories are absent because their release has not been deployed.
+- The MariaDB data directory and Docker volumes are on the large filesystem; the SSH user cannot read their host-level sizes without privileged access.
+- `/opt/job-tracker/backups` contains 21 MariaDB `.sql.gz` files, about 5.2 MB combined. All passed `gzip -t`.
+- Observed range: 2026-07-19 through 2026-08-02. No newer dump and no JobTracker timer/current-user cron entry were observed.
+- No application-volume, attachment/CV, data-protection-key, tombstone, or configuration bundle was present in that backup directory.
+
+This proves only that existing compressed dump files are structurally readable. It does not prove logical restore, row counts, non-ASCII fidelity, current RPO, or complete disaster recovery.
+
+## Monitoring and evidence gaps
+
+Available today: Docker health status, restart count, container stats, bounded container logs, sidecar health, `ollama list`, `ollama ps`, and host/GPU metrics. Missing: durable metrics/history, alerting tied to JobTracker SLOs, queue depth in this old deployment, inference latency/throughput history, active GPU-offload evidence, scheduled backup evidence, and current restore proof.
+
+No production logs were read because they may contain prompts, paths, identifiers, or private content.
+
+## Safe conclusion
+
+The host can support a cautious one-model-at-a-time synthetic benchmark. Do not install or load a candidate, enable workers, enable external fallback, or deploy until the all-interface Ollama exposure, stale/incomplete backups, dirty deployment script, release-version gap, and rollback prerequisites in `production-ai-rollout-and-rollback.md` are resolved.

docs/production/production-ai-rollout-and-rollback.md

@@ -0,0 +1,106 @@
+# Production AI rollout and rollback
+
+Updated: 2026-08-15
+
+Status: plan only. The 2026-08-15 activity was read-only. No backup, model pull, service restart, firewall/network change, deployment, database mutation, or restore was performed.
+
+## Current rollback anchors
+
+- Deployed application commit: `de937d25dc5e`; configured version `157`.
+- The production checkout is dirty because `deploy/deploy.sh` has a mode-only change. Preserve and review that state before any checkout/reset.
+- JobTracker Ollama image digest: `sha256:f1a705f2bd113fb8d15f85f7c217f0dc5f6bebda6b0cc42b82c3ad165ffcb9dc`.
+- Existing model: `qwen2.5:7b`, ID prefix `845dbda0ea48`.
+- Existing model must not be removed during benchmark or rollout.
+- All four current application containers are healthy, restart `unless-stopped`, and had zero restart count at capture.
+
+Environment secret values were deliberately not copied. Recovery still depends on the protected production environment file and data-protection material; their custody has not been verified.
+
+## Hard stop conditions before mutation
+
+1. PR 28 must pass current CI and be approved for the normal deployment path.
+2. Review/preserve the production-only mode change to `deploy/deploy.sh`; never erase it with a blind reset.
+3. Remove JobTracker's host-published 11434 listener and broader network membership, and decide whether the separate shared Ollama listener is an approved private dependency. Verify no Internet/Traefik route and no unintended LAN clients.
+4. Produce a current MariaDB dump, a recoverable owner-file volume snapshot, data-protection-key recovery, protected tombstone storage, and non-secret configuration inventory. Keep tombstones outside any restored application backup.
+5. Restore the complete set into an isolated scratch environment and verify user/application/Career row counts, owned file access, non-ASCII content, key decryption, and tombstone replay. Never restore over live production for rehearsal.
+6. Define RPO, RTO, backup expiry, tombstone retention, legal hold, and account-deletion provider/cache semantics.
+7. Keep account deletion disabled and all previously inert workers off.
+8. Resolve resource limits for the AI sidecar/Ollama or document measured stop thresholds. Current containers have no CPU, memory, or PID caps.
+9. Confirm at least 25% free on the filesystem receiving image/model layers and enough root headroom for build/temp/log growth. Root is currently 83% used; Docker/model data is on the spacious secondary filesystem.
+10. Use only synthetic benchmark inputs. Do not use production CVs, correspondence, job descriptions, or prompts.
+
+If any item fails, stop. Do not work around it by deleting models, pruning unknown Docker data, weakening health checks, exposing ports, or enabling external processing.
+
+## Intended release-branch topology
+
+- Frontend: reachable only through the approved reverse proxy network; no direct host 3000 publication.
+- Backend: internal application/proxy networks; no host port.
+- AI sidecar: only backend plus private AI network; no host port; service token required for non-health endpoints.
+- Ollama: private AI network only when bundled, or a separately approved private endpoint. No all-interface/public host publication.
+- Routing: local-first with external processing globally disabled by default and additionally requiring per-user opt-in when later approved.
+- Workers: durable AI worker remains disabled until queue/privacy/entitlement/notification production checks pass.
+
+The deployed runtime does not yet have this topology or routing contract.
+
+## Backup gate
+
+The observed 21 MariaDB dumps are gzip-valid but old and database-only. Before an approved deployment:
+
+1. create a new out-of-band MariaDB dump through the supported script;
+2. record its UTC time, size, duration, client/server versions, and SHA-256 without recording credentials;
+3. snapshot/copy the complete JobTracker data volume, including attachments, CV artifacts, generated exports, and account exports;
+4. separately secure data-protection keys and the account-deletion tombstone ledger;
+5. record the exact non-secret Compose/image/model digests;
+6. restore all components to an isolated scratch stack and complete the documented verification matrix;
+7. retain the pre-change set until the observation window and rollback decision expire.
+
+Database-only success is not a complete backup.
+
+## Benchmark sequence after approval
+
+1. Capture idle host/container/GPU metrics and `ollama ps`.
+2. Keep `qwen2.5:7b` as the baseline; do not pull multiple candidates together.
+3. Run one synthetic request, then verify loaded model, GPU/CPU split, VRAM, RAM/swap, temperature, power, latency, tokens/second, JSON validity, and logs free of content.
+4. Stop if swap grows materially, memory pressure remains elevated, root free space drops below the agreed threshold, the GPU overheats/throttles, containers become unhealthy, or latency breaches the package threshold.
+5. Run the checked-in synthetic evaluation set at 4K and 8K context. Try 16K only after measured headroom.
+6. Repeat enough times to distinguish warm/cold load and variance. One successful response is not a model decision.
+7. Test queue congestion, timeout, restart, local outage, privacy opt-out, Free/Pro denial, and two-tenant isolation before any rollout.
+8. Record exact model tag, digest, license, quantization, context, options, resource use, quality and failure categories.
+
+Do not change production's selected model merely because it fits in VRAM.
+
+## Staged rollout
+
+1. Deploy configuration/topology/security changes with AI workers and account deletion still disabled.
+2. Verify admin version badge equals the deployed commit and confirm direct ports are closed.
+3. Re-run anonymous/authenticated health and core non-AI smoke checks.
+4. Run one admin-only synthetic local inference while watching CPU/RAM/swap/GPU/container health.
+5. Enable one controlled synthetic queue canary; verify durable operation, notification, cancellation, restart recovery, entitlement and tenant boundaries.
+6. Observe for the agreed period before increasing concurrency above one.
+7. External fallback remains off until explicit privacy/provider approval and separate synthetic verification.
+8. Account deletion remains off until its retention/restore/provider/cache rehearsal is complete.
+
+## Rollback plan
+
+Rollback must use the supported deploy/runbook path, not ad-hoc container deletion.
+
+1. Stop admission of new AI work; leave durable queued records intact.
+2. Request cancellation/drain and wait for the bounded deadline. Do not kill the database.
+3. Set AI worker and external processing gates back to false.
+4. Redeploy the recorded pre-change commit/images through the approved script, preserving the reviewed production script state.
+5. Keep the pre-existing `qwen2.5:7b`; changing application selection does not require deleting candidate model files.
+6. Restore database/files/keys only if application rollback is insufficient and only from the verified complete pre-change set. Apply deletion tombstones before readiness.
+7. Verify container health, direct-port closure, login, application counts, Career/CV, attachments, non-AI behavior, operation reconciliation, and admin version.
+8. Record incident times and sanitized failure categories. Do not copy prompts or user content into the report.
+
+Expected interruption is one controlled application-container recreation plus model cold-load time. Exact duration remains unmeasured and must be captured during the approved rehearsal.
+
+## Outstanding approvals
+
+- network/firewall/container mutation to close Ollama and frontend host ports;
+- complete backup and scratch restore authority;
+- retention/tombstone/legal decisions;
+- model pull and synthetic production benchmark;
+- deployment and worker activation;
+- external-provider fallback and any real-provider checks.
+
+Until those approvals and gates are complete, the correct state is the current read-only evidence plus disabled release-branch features—not a partial production rollout.

docs/production/production-ai-validation.md

@@ -1,8 +1,8 @@
 # Production AI validation
 
-Updated: 2026-08-02
+Updated: 2026-08-15
 
-Status: `BLOCKED`. No production access, deployment, Ollama installation, model pull, benchmark, provider call or configuration change has been performed by this programme.
+Status: `BLOCKED`. Sanitized read-only production inventory is complete. No deployment, Ollama installation, model pull, inference/benchmark, provider call, backup/restore, restart or configuration/network change has been performed by this programme.
 
 ## Required before any production change
 
@@ -16,7 +16,9 @@ Status: `BLOCKED`. No production access, deployment, Ollama installation, model
 - local-only bind/network proof for Ollama;
 - canary, monitoring and rollback procedure.
 
-Repository-side evidence will be linked here as packages PROD-001 through PROD-004 advance. Until then, production state is unchanged and unverified.
+PROD-001 confirms the remembered 32 GiB / GTX 1060 6GB hardware, current Ollama/model/runtime and healthy application containers. It also confirms rollout stop conditions: all-interface JobTracker Ollama/frontend listeners, unlimited container resources, an old direct-Gemini sidecar, a dirty deploy-script mode and database-only backups ending 2026-08-02. See `production-ai-hardware-assessment.md` and `production-ai-rollout-and-rollback.md`. Production state was not changed.
+
+PROD-003 now has a tested plan-only synthetic benchmark harness and an honest empty result table in `ollama-model-benchmark.md`. No candidate was pulled or called; no primary/fallback model is selected.
 
 BG-001 tenant-safe owner execution is implemented locally, but job enrichment remains default-off. It must not be enabled until durable operations, Pro entitlement and AI privacy policy pass their own gates; see `docs/verification/bg-001-tenant-workers.md`.