310 lines
40 KiB
Markdown
310 lines
40 KiB
Markdown
# P5-13 — Configured-model semantic qualification
|
|
|
|
**Current decision: BLOCKED — configured target remains unavailable.** The recovery checkpoint below identified the runtime entry as an AF_UNIX socket and reproduced the Docker failure. The supported Model Runner disable command also failed; neither Docker nor the actual deployed model identity was recovered. No generation/evaluation model run or human output review was completed. Zero observed failures from zero runs is not a factuality pass. P5-13 UI remains NO-GO.
|
|
|
|
Parent: `2398ff354ed680c750500d1b417eee23e18f181d`, branch `codex/ui-redesign-phase4`. This is a documentation-only qualification checkpoint. The application, providers, prompts, proposal enforcement, revisions, rendering, dependencies and CI are unchanged. P5-13 UI, P5-14 and B07 have not begun.
|
|
|
|
The original corpus declaration was committed as `14e9e60ce5f37bbf4449087612e345d7186c9f81`. The first environment checkpoint is preserved in [environment-checkpoint.json](environment-checkpoint.json). This bounded recovery checkpoint starts from `d37b97ce390ba31ec3b7bcea6b371aff1057992c`; sanitized observations are in [environment-recovery-checkpoint.json](environment-recovery-checkpoint.json). Neither checkpoint changes the original [qualification results](qualification-results.json), corpus or rubric. The historical CONDITIONAL GO below is not the current environment acceptance decision.
|
|
|
|
## Qualification-environment recovery — 2026-10-08
|
|
|
|
### Read-only filesystem diagnosis
|
|
|
|
The exact inspected entry was the absolute current-account path expanded from `%LOCALAPPDATA%\Docker\run\dockerInference`, conventionally `C:\Users\<Windows-account>\AppData\Local\Docker\run\dockerInference`. The live diagnostic retained the expanded path; the committed evidence omits the account name. No new rename, deletion, permission or reparse-point mutation was attempted.
|
|
|
|
| Property / inspection | Observed result |
|
|
|---|---|
|
|
| Native `FindFirstFileW` enumeration | File entry, zero bytes, attributes `0x420` (`Archive`, `ReparsePoint`); not a directory |
|
|
| Reparse tag, from `WIN32_FIND_DATA.dwReserved0` | **`0x80000023`, `IO_REPARSE_TAG_AF_UNIX`**: an AF_UNIX socket endpoint, not a symlink/junction |
|
|
| Filesystem target / target existence | Not applicable: this tag represents a socket endpoint, not a filesystem link to a target path |
|
|
| Created / last written (UTC) | **2026-08-19 16:12:50.512702 / 2026-09-14 07:56:57.680523**; unchanged after this attempt |
|
|
| Open the entry itself for attributes/security | `CreateFileW` with `FILE_FLAG_OPEN_REPARSE_POINT`, `FILE_FLAG_BACKUP_SEMANTICS`, `READ_CONTROL` and `FILE_READ_ATTRIBUTES` returned **1920**, “The file cannot be accessed by the system” |
|
|
| Filesystem ownership | **Unverified**. The entry cannot be opened for its security descriptor; independent `icacls <exact-path> /L` also failed with the same access error. A PowerShell `Get-Acl` module-load error was a separate tooling limitation, not filesystem evidence |
|
|
| Process/lock ownership | No Desktop/backend/Ollama processes before inspection or after cleanup. Read-only Restart Manager resource registration/listing returned **6 (`ERROR_INVALID_HANDLE`)**, so it did **not** prove that there are no other holders |
|
|
| Docker component association | Backend startup logs explicitly name the inference manager and this exact Unix listener path. This establishes the failing component, independently of the filename |
|
|
| Original creator | **Unverified**: no creation audit was available. Component association and timestamps do not prove which process originally created the entry |
|
|
|
|
The native inspection used Python standard-library `ctypes` against Windows APIs; the scratch helper/output remain ignored under `.artifacts/p5-13-environment-recovery/`. Handles and Restart Manager sessions were closed; neither `RmShutdown` nor `RmRestart` was called. It enumerated the reparse entry without following it and did not read a socket payload. `DeviceIoControl(FSCTL_GET_REPARSE_POINT)`/security inspection could not proceed because the metadata/security open failed.
|
|
|
|
**Diagnosed failure boundary:** Docker Desktop cannot remove an existing AF_UNIX socket entry while starting its inference-manager listener. The deeper Windows/filesystem cause of error 1920 remains unknown; this is not reported as confirmed ACL corruption, a junction target problem, a proven file lock or an Ollama/model failure.
|
|
|
|
The previous checkpoint's **one failed `Move-Item -LiteralPath` attempt remains recorded below**. It created no backup and did not move the entry. This checkpoint did not repeat that operation or attempt a low-level reparse deletion.
|
|
|
|
### Required application path and supported recovery attempt
|
|
|
|
JobTracker's existing path is **API → private FastAPI `ai-service:8001` → ordinary Ollama HTTP `/api/generate` on port 11434**. [Compose](../../../../docker-compose.yml) supports an opt-in `bundled-ollama` container with persistent `ollama_data`, or an existing configured HTTP origin through `OLLAMA_BASE_URL`; the [provider implementation](../../../../tools/summarizer/app.py) does not use Docker Desktop Model Runner. A host/shared service is usable only if its actual identity/settings match the verified target. No host Ollama process or established equivalent endpoint was found.
|
|
|
|
Docker Desktop's inference feature is therefore **not an application requirement**. Its failing manager nevertheless blocks this local Desktop engine before the normal containers can run. Disabling the nonessential feature was tested through the installed public CLI, without touching its socket or storage:
|
|
|
|
| Bounded command / check | Result |
|
|
|---|---|
|
|
| Installed binaries/help | Desktop backend **4.86.0.236216**, Docker CLI **29.7.2**, Desktop plugin **v0.4.3**; `docker desktop disable model-runner --help` explicitly supports “Disable Docker Model Runner” |
|
|
| `rtk proxy docker desktop disable model-runner` before startup | Failed to update settings: `dockerBackendApiServer` named pipe unavailable |
|
|
| `rtk proxy docker desktop start --detach --timeout 45` | Startup accepted; **2026-10-08T15:04:06Z** backend log reproduced the same inference listener/socket-removal failure |
|
|
| Same supported disable command after startup | Also failed with the unavailable backend settings named pipe; no successful disable/re-enable claimed |
|
|
| `rtk proxy docker ps --format "{{.Names}}\|{{.Status}}\|{{.Ports}}"` | Linux-engine API unavailable; `dockerDesktopLinuxEngine` named pipe missing |
|
|
| `rtk proxy docker desktop stop --timeout 15` | Timed out; only the eight Desktop/backend PIDs observed starting during this attempt were stopped, with process-name/start-time identity checks |
|
|
| Final verification | Zero Desktop/backend processes, original socket still present, previous proposed backup absent; native tag/size/timestamps unchanged |
|
|
|
|
The table's escaped separators display the literal `|` characters passed to `--format`. Selected non-secret settings `EnableDockerAI=true` and `InferenceCanUseGPUVariant=true` were unchanged; no explicit Model Runner enabled flag was present in the inspected keys. This limited inspection is not a complete settings-file equality claim. No guessed setting was edited. No factory reset, reinstall, WSL unregister, volume/image/container deletion, application/database modification or production operation occurred.
|
|
|
|
### Deployed identity and alternative transport
|
|
|
|
The local current-code candidate remains **Ollama `qwen2.5:7b`**, with the existing generation/evaluation settings; it is **not verified deployed identity**. Independent read-only deployment inspection still finds `/opt/job-tracker/shared/.env` plus process overrides and CI references to `PROD_HOST`, `PROD_USER`, `PROD_SSH_KEY`, without a usable public connection value. Local SSH configuration remains absent. No secret was retrieved and no production/staging endpoint was contacted.
|
|
|
|
An existing Docker-independent **partial** diagnostic route was identified: authenticated `GET /api/jobapplications/ai-metrics` (alias `summarizer-metrics`), or authorised admin `GET /api/admin/system`. These obtain live private-sidecar health through [SummarizerService](../../../../JobTrackerApi/Services/SummarizerService.cs). Use **`ollamaModel`**, not the separate DistilBART `model` value. They can establish configured model/reachability/available model names; admin diagnostics also identify the backend revision. They require an established deployed application origin and authorised session, unavailable here. They omit full provider/routing, model digest and sidecar image/prompt identity, so even a successful response would only supply part of the gate.
|
|
|
|
The existing private sidecar `/health`, deployed non-secret settings, runtime image/source hashes and the actual service's Ollama inventory are still needed to complete identity. An ordinary HTTP transport may avoid local Desktop, but **no same-target alternative is established**. Neither Gemini nor another local model was substituted, no model was downloaded, and no corpus call was made against a guess.
|
|
|
|
### Recovery result, validation and next operator action
|
|
|
|
**BLOCKED — configured target remains unavailable. Docker was not safely recovered.** Actual deployed provider/model/digest/settings remain unverified, target reachability remains unestablished, smoke test **not run**, qualification **0/366**, human reviews **0**, factuality failures **unmeasured**. The frozen corpus/rubric and deterministic application protections are unchanged. P5-13 UI remains **NO-GO**.
|
|
|
|
Further socket/filesystem repair is uncertain because filesystem ownership and original creation are unproved and ordinary metadata/security operations fail. Obtain a supported Docker/Windows diagnosis for this exact **AF_UNIX tag + error 1920 + inference startup log**, preserving Docker/WSL/application data. Do not delete this entry or reset storage on the basis of its name alone. No destructive recovery proposal was executed or represented as necessary.
|
|
|
|
Independently, the operator must provide the existing authorised deployed/staging connection or sanitized runtime diagnostics: provider/routing/model settings, model digest, actual sidecar/API image revisions and prompt-source hashes. This can remove the identity blocker without repairing local Desktop. Once a semantically identical isolated target is verified/reachable, run one uncached synthetic smoke request through the application routing before the unchanged 366-call qualification and full human-review packet. A local default or API health response alone cannot clear the whole identity gate.
|
|
|
|
Fresh checks for this **documentation-only** recovery checkpoint: configuration/Compose **3/3**, frozen LF-normalized hashes, JSON integrity/decision/count checks, current-document local links, staged secret scan and Git whitespace/scope checks. Application suites were **not rerun**. Inherited counts remain **1,021 backend / 455 frontend / 49 sidecar / 106 focused AI / 112 browser / 1 real HTTP / 5 revision**, **16/16 fidelity (36 = 36 pages)**, **8/8 settings exports (24 = 24 pages)** and **9/6/1/3 security/configuration**; no test-count changes. No application, environment template, Compose, dependency or CI file changed. All unrelated debt, including F04, remains open. Work stops before P5-13 UI, P5-14 and B07.
|
|
|
|
## Configured-model environment checkpoint — 2026-10-08
|
|
|
|
### Active configuration and target identity
|
|
|
|
Only allowlisted non-secret AI settings were inspected in the regular local `.env`; no credential lines were exposed, copied or committed. The active local entries select:
|
|
|
|
| Key | Local value / effective current-code behavior |
|
|
|---|---|
|
|
| `OLLAMA_BASE_URL` | HTTP Compose-service origin `ollama`, port `11434`, without URL credentials or non-root path |
|
|
| `OLLAMA_MODEL` | `qwen2.5:7b` |
|
|
| `AI_PROVIDER` | `gemini` |
|
|
| `GEMINI_MODEL` | `gemini-2.0-flash` |
|
|
| `GROQ_MODEL` | `llama-3.3-70b-versatile` |
|
|
| `AI_ROUTING_MODE` | Absent locally; current Compose defaults to `local_first` |
|
|
| `EXTERNAL_AI_ENABLED` | Absent locally; current Compose defaults to `false` |
|
|
|
|
With the current code and previously inspected unset process overrides, this local configuration intends **Ollama `qwen2.5:7b` as primary**, with Gemini selected but gated off for external processing. It does not authorize a direct Gemini call. An older sidecar could implement different routing, so checkout settings alone cannot establish the running behavior. Temperatures, prompts and inference options remain as documented below, with no changes.
|
|
|
|
Production deployment uses `/opt/job-tracker/app` and `/opt/job-tracker/shared/.env`, preserving process overrides. The local file is not that deployment symlink. Public CI exposes only secret-reference names for the production host/user/key; the accessible local SSH configuration is absent. No established production SSH alias or approved staging endpoint was found. No CI secrets, SSH keys or credentials were retrieved. The actual deployed provider, model/digest, endpoint and sidecar image remain **unverified**.
|
|
|
|
No equivalent target was asserted from the local file, historical documentation, model defaults or an alternate provider. No substitute model was selected.
|
|
|
|
### Docker restoration attempt and exact failure
|
|
|
|
The authorized minimal recovery used the installed Desktop CLI. It did not perform a factory reset, reinstall, database reset, WSL unregister, container deletion or Compose shutdown.
|
|
|
|
| Step | Observation |
|
|
|---|---|
|
|
| `rtk proxy docker desktop status` | Could not retrieve status; engine initially unavailable |
|
|
| `rtk proxy docker desktop start --detach --timeout 45` | CLI accepted startup; Desktop/backend processes started |
|
|
| Formatted `docker ps` | Linux-engine API remained unavailable; missing `dockerDesktopLinuxEngine` named pipe |
|
|
| Current backend startup diagnostic | At `2026-10-08T14:43:14Z`, backend cancelled while initializing the inference manager; it could not remove the `dockerInference` runtime path |
|
|
| Literal runtime-path metadata | Zero bytes, `Archive, ReparsePoint`; created 2026-08-19 and last modified 2026-09-14 |
|
|
| Native reparse-point query | `Error 1920: The file cannot be accessed by the system.` |
|
|
| `rtk proxy docker desktop stop --timeout 30` | Graceful stop timed out; only the Desktop/backend process IDs started by this task were then stopped |
|
|
| Bounded reversible repair | After all those processes exited, the exact source and backup paths were validated within the named Docker runtime directory; one `Move-Item -LiteralPath` attempt failed with the same access error |
|
|
| Final local state | Original runtime path still exists; proposed backup does not exist; zero Desktop/backend processes; no model/sidecar/container service started |
|
|
|
|
The confirmed failure boundary is the inaccessible inference-manager runtime reparse point, not an Ollama response or CV application error. The underlying filesystem/OS reason for that entry's inaccessibility is not established; it is not claimed to be fixed. No socket contents were read. The failed rename changed neither runtime path. Desktop startup generated its own diagnostic logs; no Desktop settings were edited by this task.
|
|
|
|
The formatted listener/container check used exactly:
|
|
|
|
```text
|
|
rtk proxy docker ps --format "{{.Names}}|{{.Status}}|{{.Ports}}"
|
|
```
|
|
|
|
Sanitized diagnostic:
|
|
|
|
```text
|
|
starting services: initializing Inference manager:
|
|
listening on unix://<LOCALAPPDATA>/Docker/run/dockerInference:
|
|
remove <LOCALAPPDATA>/Docker/run/dockerInference:
|
|
The file cannot be accessed by the system.
|
|
(listener: The filename, directory name, or volume label syntax is incorrect.)
|
|
```
|
|
|
|
The investigation stopped here. Low-level reparse deletion, broad runtime-directory removal, permissions changes, WSL/container-data recovery and Docker reset were not attempted. Those operations cannot identify the actual deployed target and are not a safe substitute for the failed narrow repair. No production/staging target was contacted or restarted.
|
|
|
|
### Qualification, scoring and human-review status
|
|
|
|
The frozen corpus and rubric hashes remain identical to the declaration checkpoint. Planned calls are still **366**; **attempted inference calls 0, completed 0, provider failures 0, retries 0**. The zero provider-failure count means no inference was attempted, not a healthy provider. Desktop startup failure is recorded separately from provider errors.
|
|
|
|
There is no uncached-run evidence because there were no runs. Automated factuality, scope, unsupported-claim, overstatement, missing-skill, schema and language scoring are **not run**. Critical factuality failures are **unmeasured**, not zero qualified failures. No prompt or grounding change was justified by absent output, and no selective or full-corpus rerun occurred.
|
|
|
|
Human reviews completed: **0**. No generated-output review packet can be produced without model outputs; the existing full per-output worksheet in [the frozen rubric](RUBRIC.md#per-run-human-review-template) remains ready and unfilled. Every generated output/repeat still requires human review; the sample has not been reduced and an AI reviewer is not counted as a human.
|
|
|
|
The application, environment templates, Compose, prompts, provider routing, packages, CI and renderer are unchanged. The inherited deterministic baseline remains **1,021 backend / 455 frontend / 49 sidecar / 106 focused AI / 112 browser / 1 real HTTP / 5 revision**, with **16/16 fidelity, 36 = 36 pages**, **8/8 settings exports, 24 = 24 pages**, and **9/6/1/3 rate-limit/ingress/PDF/configuration**. This environment/documentation-only checkpoint does not claim those suites were rerun. There are no test-count changes; the last actual executions remain identified in the original results and parent record.
|
|
|
|
### Exact blockers and next operator evidence
|
|
|
|
1. **Docker Linux engine cannot start.** Resolve the exact local inference runtime-entry failure with a supported Docker/Windows repair. Do not reset container/WSL/database storage merely to run this corpus. After recovery, provide successful `rtk proxy docker desktop status` and `rtk proxy docker ps` output, and only then start the necessary configured model/sidecar services.
|
|
2. **Actual production-equivalent identity is unverified.** Provide the already established deployment/staging connection or sanitized runtime evidence using the read-only identity commands below: actual sidecar/API image revision, selected non-secret model/routing settings and model identity/digest. Do not send keys, tokens, full `.env`, cookies or credential-bearing URLs. The local `qwen2.5:7b` setting is a known configuration candidate, not deployed-model proof.
|
|
3. **Required inference and human acceptance are incomplete.** Once both prerequisites are satisfied, execute the unchanged 366-call procedure, retain every output/error and complete every required human worksheet. A safe rejection cannot convert fabricated generation into a semantic pass.
|
|
|
|
**Environment-checkpoint decision: BLOCKED / NO-GO for P5-13 UI.** Neither the actual configured target nor the required model runs/human acceptance is available. The prior deterministic readiness acceptance remains intact, but it does not clear this critical prerequisite. Work stops before P5-13 UI, P5-14 and B07; F04 and all unrelated debt remain open.
|
|
|
|
Sources: [grounding readiness](../p5-13-readiness/README.md), [Phase 5 plan](../PHASE5_PLAN.md), [AI interaction model](../../phase-3/AI_INTERACTION_MODEL.md), [CV workspace](../../phase-3/CV_WORKSPACE.md), [preservation impact](../PRESERVATION_IMPACT.md), [AI architecture](../../../ai/architecture.md), and the actual [suggestion service](../../../../JobTrackerApi/Services/CvAiSuggestionService.cs), [provider transport](../../../../JobTrackerApi/Services/SummarizerService.cs) and [sidecar](../../../../tools/summarizer/app.py).
|
|
|
|
## Qualification scope and predeclared acceptance
|
|
|
|
The version-controlled [corpus](corpus.json) and [rubric](RUBRIC.md) define **32 synthetic semantic cases** before any live run, each with generation, generated-output assessment and a fixed assessment probe. They supplement the existing [26 assessment cases](../../../../JobTrackerApi.Tests/Fixtures/AiEvaluation/cv-rewrite-grounding.json); they do not replace or weaken them. All model cases require three independent executions at the unchanged configured settings: **96 generations + 96 generated-output assessments + 96 fixed assessments + 78 existing assessments = 366 planned provider calls**. None was attempted here. The single stale integration reference is a persistence check, excluded from model counts.
|
|
|
|
Qualification must separately inspect:
|
|
|
|
1. Raw generation output, including output subsequently rejected by the evaluator. Unsupported facts in rejected output still fail the generation-quality gate; rejection proves enforcement, not truthful generation.
|
|
2. The model's factual assessment, including false approvals and false rejections against fixed supported/unsupported proposals.
|
|
3. The authoritative API decision, including schema, quote membership, entry scope, numeric guards and protected proposal issuance.
|
|
4. Human claim-by-claim semantic review, scope, preservation, missing-skill honesty, usefulness, language and overstatement.
|
|
|
|
Every candidate-specific claim must be supported by the allowed selected source. A job requirement, instruction or plausible responsibility is not candidate evidence. Exact quotes prove source membership, not entailment. Any fabricated candidate fact is a critical failure even when usefulness or language scores are high. Unresolved ambiguous support cannot receive acceptance. Mixed supported/unsupported output is rejected wholesale; no partial-merge semantics are added.
|
|
|
|
No keyword blacklist, relaxed threshold, substitute model or changed temperature can turn this corpus into production-equivalent evidence. `scripts/run-ollama-evaluation.py` remains useful for its original tasks but uses different prompts, options and substring scoring; it cannot qualify this runtime rewrite/evaluation route.
|
|
|
|
## Configured target: what is known
|
|
|
|
| Setting | Checked-in behavior at the parent | Actual deployed setting |
|
|
|---|---|---|
|
|
| Local model | Compose defaults `OLLAMA_MODEL=qwen2.5:7b`; direct sidecar execution defaults to an empty model | Unverified |
|
|
| Ollama origin | Compose defaults `http://ollama:11434`; may use an existing shared origin via `OLLAMA_BASE_URL` | Unverified |
|
|
| Routing | `AI_PROVIDER=ollama`, `AI_ROUTING_MODE=local_first`, `EXTERNAL_AI_ENABLED=false` | Unverified |
|
|
| Optional external models | Gemini `gemini-2.0-flash`; Groq `llama-3.3-70b-versatile` | Unverified; neither selected or called here |
|
|
| Generation | `/cv/rewrite`, temperature **0.2**, provider timeout **180s**, including typed block-array generation | Code verified; running image unverified |
|
|
| Evaluation | `/cv/evaluate-rewrite`, JSON mode, temperature **0.1**, provider timeout **120s** | Code verified; running image unverified |
|
|
| Additional generation options | No explicit seed, `top_p`, `num_predict` or `max_tokens` in this provider path | Model/runtime defaults unverified |
|
|
| API transport timeout | Existing `ai-service` HTTP client **30s** | Runtime/image unverified |
|
|
| Prompt identity | Inline controller instruction plus sidecar rewrite/review prompts, identified by parent commit and file hashes | Deployed prompt/image revision unverified |
|
|
|
|
The two operations share the provider router but route independently. A local failure/circuit change can select another allowed provider on one operation; record generation and assessment provenance independently. Existing response metadata exposes `X-Ai-Provider`, `X-Ai-Model`, `X-Ai-Route-Reason` and optional `X-Ai-Fallback-Reason`. No independent evaluator model is configured.
|
|
|
|
The private sidecar maps both paths to the existing `cv-rewrite` task. External fallback retains administrator policy, task allowlist, account consent/entitlement and input-size checks. This checkpoint authorizes neither a provider change nor additional external-processing consent.
|
|
|
|
Deployment loads operator environment/shared `.env` values that can override public defaults. A historical production report dated 2026-08-15 describes older routing and cannot identify today's model, digest, image or evaluator. No production SSH credentials, secret files, CI secrets or provider keys were inspected. No remote production endpoint was guessed.
|
|
|
|
The bundled Ollama Compose service is opt-in (`bundled-ollama`); an ordinary Compose startup does not guarantee a model exists. The warm-up script can pull a model and restart services, so it was not run merely to substitute a guessed environment.
|
|
|
|
## Availability observed in this environment
|
|
|
|
Read-only checks on 2026-10-08:
|
|
|
|
| Check | Exact observed result | Interpretation |
|
|
|---|---|---|
|
|
| `Get-Command ollama,docker` | Docker executable exists; no `ollama` command found on PATH | No CLI target established; not proof that no model files exist elsewhere |
|
|
| Selected TCP listeners | No listeners on `11434`, `8088`, `8000`, `8001`, `8080`, `5000` or `5302` at inspection time | No local known model/sidecar endpoint observed |
|
|
| `docker ps` | Docker API connection failed: `dockerDesktopLinuxEngine` named pipe missing | Container/model availability cannot be inspected with the engine stopped |
|
|
| `http://127.0.0.1:11434/api/version` | Request failed with `WebException` | Default local Ollama endpoint unavailable; no generation call sent |
|
|
| Selected process overrides | `OLLAMA_HOST`, `OLLAMA_BASE_URL`, `OLLAMA_MODEL`, `AI_PROVIDER`, `EXTERNAL_AI_ENABLED`, `AI_ROUTING_MODE` and inspected local-resource settings unset | These process values do not establish deployed/container settings |
|
|
|
|
No model was pulled, installed, started, substituted or called. No production service was restarted. The configured target and model digest remain unavailable, rather than qualified by the test provider.
|
|
|
|
## Corpus boundaries
|
|
|
|
The corpus covers strong/partial/weak matches; related but unequal technologies; unsupported responsibilities, quantities and education upgrades; job-text injection; English/Bokmål; truthful shortening/grammar/tailoring; entry-specific evidence; effective CV overrides; captured historical source; and stale-apply integration.
|
|
|
|
Only existing actions/scopes are exercised. A new “suggest skills” workflow is not added. Tailoring may emphasize supported facts; absent requirements must not become CV claims. Recommendation text is assessed as a recommendation only when it is clearly conditional and appropriate to the requested operation; analysis inserted into a CV-only rewrite can still fail scope fidelity.
|
|
|
|
Generation source and assessment evidence reflect the actual contract: section generation receives selected items; review receives that selected block including title/subtitle. Document generation/review receives included resolved visible blocks. The service's separate `EditableBlocks` permits an explicitly selected omitted section to be edited; it does not supply hidden content to document generation. Unrelated profile fields, hidden whole-document content, current facts outside a captured historical fixture and job claims are not added as candidate evidence.
|
|
|
|
The actual CV revision history stores a settings lens, not a full immutable canonical-profile snapshot. The historical case qualifies a supplied captured source and must not be misreported as a public historical-generation feature. Submitted/frozen capture remains the historical output truth. The override case uses effective CV content, with canonical differences recorded for review but excluded from the provider's selected evidence. No profile mutation or new history behavior is introduced.
|
|
|
|
## Run and human-review status
|
|
|
|
| Evidence category | This checkpoint |
|
|
|---|---|
|
|
| Live configured-model generation runs | **0 — unavailable** |
|
|
| Live configured-model factual-assessment runs | **0 — unavailable** |
|
|
| Automated live semantic checks | **Not run** |
|
|
| Critical factuality failures | **Not measurable**, not zero accepted failures |
|
|
| Structured human output reviews | **0 — pending** |
|
|
| Assistant review of fixture/rubric structure | Performed; not human semantic acceptance |
|
|
| Deterministic API/SQLite/source/history checks | Passing; separate from model accuracy |
|
|
|
|
No generated model output is stored because there was none. Expected rubric outcomes are requirements, not observed outputs or reviewer judgments. The synthetic provider used in regression validation cannot contribute to the semantic run count.
|
|
|
|
## Procedure when the configured target becomes available
|
|
|
|
Use an isolated test deployment with the actual configured image, model identity/digest, prompts and router settings. Do not perform corpus writes or restart an API on the production user database. Existing processing consent and service authentication must remain in force. Credentials remain in the operator's established mechanism, never in fixture files, reports or pasted commands.
|
|
|
|
### Establish identity before execution
|
|
|
|
The operator can run these read-only commands in the existing deployment directory. Do not publish the private sidecar or dump the full Compose environment:
|
|
|
|
```sh
|
|
rtk git rev-parse HEAD
|
|
rtk proxy docker compose images backend ai-service
|
|
rtk proxy docker compose exec -T ai-service python -c 'import json,os; keys=("OLLAMA_MODEL","AI_PROVIDER","AI_ROUTING_MODE","EXTERNAL_AI_ENABLED","EXTERNAL_AI_ALLOWED_TASKS","GEMINI_MODEL","GROQ_MODEL"); print(json.dumps({k:os.getenv(k) for k in keys},sort_keys=True))'
|
|
```
|
|
|
|
If the opt-in bundled Ollama service is the configured target, `rtk proxy docker compose exec -T ollama ollama list` records installed tags/digests. For a shared Ollama service, use its already established operator connection and obtain equivalent model identity. Do not invent a hostname, create credentials or select a different model. Record the actual sidecar/API image revision and compute actual runtime source-file hashes, comparing them with the expected checkpoint hashes in `qualification-results.json` (UTF-8 file bytes with CRLF normalized to LF, never prompt text). Copying expected hashes does not establish deployed identity. Missing/stale `/cv/evaluate-rewrite` or typed document support blocks the run.
|
|
|
|
### Generation and application path
|
|
|
|
Use the existing [real-HTTP fixture](../../../../job-tracker-ui/ai-tests/cv-ai-readiness.spec.ts) as the account/CSRF/source/snapshot reference, with a **separate** isolated live-provider setup. The committed `playwright.cv-ai.config.ts` deliberately launches a synthetic provider; running it unchanged never qualifies a model.
|
|
|
|
- Create a purpose-built synthetic owner with existing AI entitlement and appropriate processing consent. Authenticate with existing session cookies, `X-CSRF-TOKEN` and `X-Session-Context`; never include those values in evidence.
|
|
- Persist only the case's canonical synthetic profile with `PUT /api/career/profile`, then create its variant via `POST /api/cv/variants`. Apply the case's supported CV override/settings through the ordinary versioned save. Compare the effective selected source with the corpus before sending an AI request. Do not treat `candidateSource` JSON as a canonical-profile PUT payload.
|
|
- Invoke `POST /api/cv/ai/assist` with `variantId`, current `expectedVersion`, existing `action`, `language`, optional `role`/`context`, and the section `targetId`/`text`; document actions omit `targetId`/`text`. Let the existing controller construct the instruction. Do not replace it with the generic offline evaluation prompt.
|
|
- Record raw synthetic output and assessment at the isolated provider transport as well as the final API status. A `422` alone conceals whether generation or evaluation was correct; no production personal-data logging may be enabled. Record actual provider/model/route separately for generation and assessment, exact source/operation hashes, run number, latency and error category. Exclude protected proposal tokens, cookies, service tokens and keys.
|
|
- Capture variant, variant history, canonical profile and canonical history before/after generation; generation must not change them. In a separate accepted safe case, explicit `POST /api/cv/variants/{id}/ai/apply` must use ordinary save/history and keep canonical data unchanged.
|
|
- Execute the stale sequence with an actually issued safe proposal: update the CV or canonical profile normally, then Apply the old proposal. Require **409 `cv_ai_source_changed`**, unchanged latest source/history after rejection, and no stale AI revision.
|
|
|
|
**Captured-source exceptions:** S13/S15 deliberately qualify supplied historical job/source captures, rather than a current-owned historical-generation feature. Submit their exact `generationSource` to the existing private `/cv/rewrite` with the actual controller-composed instruction and existing `output_mode` (`text` for a section, `cv_blocks_json` for a document); assess the result through `/cv/evaluate-rewrite` with their exact `candidateSource`/`contextOnly` and the strict C# guard. Label these private supplied-source results separately. Do not load newer canonical data or imply the public assist API accepts an old revision snapshot. The fixed assessment lane likewise remains a private route judgment, distinct from authenticated Apply evidence.
|
|
|
|
The API caches identical generation input for six hours. A new identical variant/account does not defeat that content-keyed cache. Three API responses are not automatically three model runs. Restart **only the isolated API process** between repetition batches, preserving its disposable synthetic database; this clears the generation cache without modifying prompts/temperature. Confirm actual rewrite transport calls. Evaluation is uncached. Do not restart production or change input/seed just to produce variation. A cached response is explicitly excluded from model-run counts.
|
|
|
|
### Fixed adversarial assessment path
|
|
|
|
Submit each fixed probe and the existing 26 cases to the same private `POST /cv/evaluate-rewrite` route as:
|
|
|
|
```json
|
|
{
|
|
"candidate_source": "the exact selected synthetic evidence",
|
|
"proposal": "the fixed proposal being judged",
|
|
"context_only": "requirements/instructions, never candidate evidence"
|
|
}
|
|
```
|
|
|
|
Preserve existing service-token/task/privacy headers using the established internal client. Evaluate the returned `assessment` through the existing `CvAiGrounding.IsSupported` boundary as well as the human rubric. Record complete coverage, actual source-quote membership, correct entry provenance and false approvals/rejections. Do not manually label the provider assessment `supported` or accept its top-level verdict without the API guard.
|
|
|
|
Run every semantic case/probe and all 26 existing assessments three times. Retain failures/timeouts, blocked output and run ordering. The existing 30s API transport timeout is not widened to obtain a semantic pass; if it prevents end-to-end qualification, record availability separately from any direct sidecar semantic result. No timeout defect is repaired in this unavailable-runtime checkpoint.
|
|
|
|
### Evidence to return
|
|
|
|
Return sanitized model/digest/runtime/image/prompt identity, settings and routing metadata; corpus hash; actual uncached generation/evaluation run counts; every case/run's synthetic output and assessment; automated validator decision; and a signed/datable human review worksheet using [the rubric](RUBRIC.md). The reviewer must classify every candidate claim as supported, unsupported, ambiguous or a conditional recommendation against the named source. No assistant-generated reviewer sign-off can fill that human gate.
|
|
|
|
Zero fabricated facts across all required runs, usable grounded rewrites, acceptable scope and completed human review are required. Any unsafe generation, false approval or unresolved ambiguity is **NO-GO**. Classify the cause as source selection, prompt construction, model behavior, ambiguity, parsing or evaluator error before proposing a bounded correction. Preserve failed results; rerun the frozen full corpus after a justified fix. Do not tune expected outcomes or mask unsafe output with high writing-quality scores.
|
|
|
|
## Preservation and validation
|
|
|
|
| Contract | Qualification requirement / retained evidence |
|
|
|---|---|
|
|
| PRES-C11 | Truthful selected/whole CV rewriting, review, no unaccepted mutation; configured semantics still blocked |
|
|
| PRES-C01 / PRES-C07 | Canonical/profile isolation and effective CV override/entry evidence; no extra source context permitted |
|
|
| PRES-C10 | Ordinary version/history acceptance; stale source rejects without replacing latest state |
|
|
| PRES-C06 | Existing assistant/editor protocol and saving owner retained; no UI migration |
|
|
| PRES-S04 / PRES-E09 | Existing provider privacy, entitlement and account identity boundaries retained |
|
|
| PRES-C08 / PRES-C09 / PRES-C12 / PRES-W05 | Appearance, renderer, PDF and frozen submitted content remain unchanged; inherited proofs retained |
|
|
|
|
No protected capability or route changed. This package does not imply exhaustive coverage of all 57 capability groups.
|
|
|
|
Fresh checks and inherited gates are recorded separately in [qualification-results.json](qualification-results.json). The full backend suite, frontend suite, sidecar tests, focused AI checks, revision checks and isolated real-HTTP contract were rerun. The unchanged 112 Chromium cases, 16/16 standard fidelity (36 = 36 pages), 8/8 settings exports (24 = 24 pages), TypeScript/lint/build and security/configuration baseline remain the parent checkpoint's evidence. They are not represented as fresh executions in this documentation-only checkpoint. No new automated test case was added or removed.
|
|
|
|
Reproduction of fresh checks:
|
|
|
|
```text
|
|
rtk proxy C:/Users/Connor/.dotnet/dotnet.exe test JobTrackerApi.Tests/JobTrackerApi.Tests.csproj --configuration Release --no-build --logger "trx;LogFileName=qualification-baseline.trx" --results-directory .artifacts/p5-13-semantic-qualification/backend
|
|
rtk proxy C:/Users/Connor/.dotnet/dotnet.exe test JobTrackerApi.Tests/JobTrackerApi.Tests.csproj --configuration Release --no-build --filter "FullyQualifiedName~CvAi|FullyQualifiedName~AiWorkspace|FullyQualifiedName~AiPrivacyPolicy|FullyQualifiedName~MeteredSummarizerService"
|
|
rtk proxy C:/Users/Connor/.dotnet/dotnet.exe test JobTrackerApi.Tests/JobTrackerApi.Tests.csproj --configuration Release --no-build --filter FullyQualifiedName~CvRevisionReadinessTests
|
|
rtk proxy tools/summarizer/.venv/Scripts/python.exe -m pytest tools/summarizer/tests -q
|
|
```
|
|
|
|
From `job-tracker-ui`, run `rtk proxy npm.cmd test -- --runInBand --silent`; for the isolated HTTP case set `DOTNET_ROOT=C:/Users/Connor/.dotnet`, `DOTNET_HOST_PATH=C:/Users/Connor/.dotnet/dotnet.exe` and run `rtk proxy npm.cmd exec playwright test -- --config=playwright.cv-ai.config.ts --reporter=line`. This last case uses the synthetic provider, not the configured model. Raw validation output remains ignored under `.artifacts/p5-13-semantic-qualification/`; no authentication state is committed.
|
|
|
|
## Original declaration decision and stopping point — 14e9e60
|
|
|
|
The following preserves the declaration checkpoint's decision. The newer environment decision above is **BLOCKED / NO-GO**; no configured-model or human semantic acceptance has occurred since declaration.
|
|
|
|
**Configured-model semantic gate: BLOCKED — configured-model qualification unavailable. P5-13 readiness: CONDITIONAL GO unchanged.** An unconditional GO cannot be issued: actual model/settings/digest are unverified, required repeated model runs are zero, and structured human output review is pending. Existing deterministic enforcement, stale conflict, canonical isolation and history protection remain passing independently.
|
|
|
|
The next bounded task is to make the already configured target available in an isolated environment, establish its identity, execute this frozen corpus without cache substitutions, and obtain human semantic review. No P5-13 UI implementation may claim this prerequisite passed.
|
|
|
|
F04 production evidence, human screen-reader/incomplete accessibility checks, production font/photo and pointer/reflow qualification, Dashboard preference-save 500, Board scaling/native-touch debt, deferred `followMode` UI and other [carried debt](../p5-13-readiness/README.md#remaining-scope-and-open-debt) remain open. P2-D01 remains an ordinary passing case. No unrelated issue was closed, no production deployment occurred, and work stops before P5-13 UI, P5-14 and B07.
|