feat(dev): enable isolated local workers and verified Ollama

dev/README.md

@@ -0,0 +1,37 @@
+# Local workers and models
+
+This overlay targets the separately configured `jobjakt-dev` stack in the primary checkout. It does not replace that checkout's standalone compose file. Production services, databases and model caches are excluded.
+
+The local runtime uses Ollama 0.40.11 and `qwen2.5:7b` (Q4_K_M, Apache-2.0), matching the model name in the repository's production configuration. Exact deployed digest parity has not been established. The frozen official registry manifest is in `model-manifests/qwen2.5-7b.json`; its SHA256 is `845dbda0ea48ed749caafd9e6037047aa19acfcfd82e704d7ca97d631a0b697e`. The image digest and resource limits are pinned in `ollama.compose.yml`. CPU inference is limited to four CPUs and 8GB; cloud processing is disabled. Model files use a dedicated development volume; the API binds only `127.0.0.1:11434`.
+
+Initial local setup, explicitly authorised by the owner on 27 October 2026:
+
+```powershell
+rtk proxy docker compose -f dev/ollama.compose.yml up -d --wait
+rtk proxy docker compose -f dev/ollama.compose.yml exec ollama ollama pull qwen2.5:7b
+rtk proxy python dev/verify_ollama.py --manifest dev/model-manifests/qwen2.5-7b.json --output .artifacts/pre-phase10-ux/local-ai/verification.json
+```
+
+Verification checks the actual image, dedicated cache and every model/config/template/license blob against the frozen manifest before inference. A future changed upstream tag must fail verification; review the artifact before updating the pin. The download network remains available for explicit model maintenance; no cloud provider is enabled.
+
+The existing approved DistilBART CPU checkpoint is reused through a read-only mount after all files are checked against `docs/redesign/phase-9/p9-sec-02/full-model/ARTIFACTS.json`. Transformers is offline, so this operation cannot fetch replacement weights. The parser-capable sidecar is also checked against its previously qualified local image ID.
+
+Preflight and apply (run from this worktree; replace paths only with the verified development checkout and approved artifact):
+
+```powershell
+rtk proxy python dev/enable_local_workers.py --checkout "E:/Storage/Projects/Active/Job tracker" --bart-artifact "E:/Storage/Projects/Active/Job tracker/.artifacts/p9-sec-02/full-model/artifact" --output .artifacts/pre-phase10-ux/local-ai/worker-preflight.json
+rtk proxy python dev/enable_local_workers.py --checkout "E:/Storage/Projects/Active/Job tracker" --bart-artifact "E:/Storage/Projects/Active/Job tracker/.artifacts/p9-sec-02/full-model/artifact" --output .artifacts/pre-phase10-ux/local-ai/workers-applied.json --apply
+rtk proxy python dev/check_local_ai.py --output .artifacts/pre-phase10-ux/local-ai/sidecar-smoke.json
+```
+
+The apply helper validates project, Development environment, loopback backend ports, development database, current backend image, offline model mount and Mailpit settings. It recreates only the development backend/sidecar without a build or pull, preserving the database and volumes. It enables rules, reminders, saved-search alerts, daily exports, job enrichment and queued AI processing. These features still respect account entitlements, AI consent, configured schedules and available input. Email goes only to Mailpit at `http://localhost:8026`; it is not delivered externally.
+
+This configuration operation does not publish source changes. In particular, the running older backend image cannot use the new `Ai:RequestTimeoutSeconds=240` setting until built from the validated feature source. Its old 30-second limit can interrupt CPU generation. Source builds and application checks have separate evidence.
+
+OAuth/mail synchronisation, billing and external data providers need their own approved credentials and configuration. Enabling workers cannot provide them. CV document import uses deterministic Docling; generative CV-import assistance stays disabled pending its separate source-grounding/accuracy qualification.
+
+## Evidence, 27 October 2026
+
+The local setup and worker apply passed. Independent model-blob verification passed. A real constrained JSON CPU response passed in 20.546 seconds including model loading. Authenticated sidecar checks passed: trained local job summarisation in 2.518 seconds and an Ollama-backed CV rewrite in 40.997 seconds. These checks assert actual nonempty content, not semantic correctness. No production digest parity or CV extraction accuracy is claimed.
+
+Official sources: [Ollama model](https://ollama.com/library/qwen2.5:7b), [frozen registry source](https://registry.ollama.ai/v2/library/qwen2.5/manifests/7b), [Ollama Docker guidance](https://docs.ollama.com/docker), [local/cloud configuration](https://docs.ollama.com/faq).

dev/check_local_ai.py

@@ -0,0 +1,55 @@
+"""Exercise the isolated development sidecar with fictional data only."""
+import argparse
+import json
+from pathlib import Path
+import subprocess
+
+
+def main():
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--output", type=Path, required=True)
+    args = parser.parse_args()
+    # Authentication stays inside the container; neither secrets nor model text are logged.
+    program = r'''
+import json
+import os
+import time
+import urllib.request
+
+records = []
+def request(path, payload, field):
+    started = time.monotonic()
+    req = urllib.request.Request("http://127.0.0.1:8001" + path,
+        data=json.dumps(payload).encode(),
+        headers={"Content-Type": "application/json", "X-AI-Service-Token": os.environ["AI_SERVICE_TOKEN"]})
+    with urllib.request.urlopen(req, timeout=240) as response:
+        result = json.load(response)
+    value = result.get(field)
+    assert isinstance(value, str) and value.strip(), "Missing generated content: " + field
+    records.append({"path": path, "status": "PASS", "seconds": round(time.monotonic() - started, 3), "characters": len(value)})
+
+assert os.environ["OLLAMA_BASE_URL"] == "http://jobjakt-dev-ollama:11434"
+assert os.environ["AI_ROUTING_MODE"] == "local_only"
+assert os.environ["EXTERNAL_AI_ENABLED"].lower() == "false"
+request("/summarize", {"text": "Fictional Systems seeks a software engineer. Responsibilities include developing C# APIs, writing SQL queries and maintaining Docker services. Candidates should communicate clearly and work with a small team. This fictional advertisement is solely a development check.", "max_length": 60, "min_length": 10}, "summary")
+request("/cv/rewrite", {"instruction": "Correct grammar only. Return exactly one sentence. Preserve the facts and add no headings.", "text": "A fictional engineer build reliable software.", "max_length": 48, "min_length": 8}, "rewritten_text")
+print(json.dumps({"status": "PASS", "model": os.environ["OLLAMA_MODEL"], "routing": "local_only", "checks": records,
+    "limits": "Synthetic sidecar connectivity and nonempty response only; no factual extraction accuracy or production parity claim."}))
+'''
+    identity = json.loads(subprocess.check_output(["rtk", "proxy", "docker", "inspect", "jobjakt-dev-ai-service-1"], text=True))[0]
+    assert identity["Config"]["Labels"]["com.docker.compose.project"] == "jobjakt-dev"
+    assert identity["Image"] == "sha256:8a29cb90fcc2817b0d784826d55a172c943f5ad7e1b6105e690a2d79ee15fe87"
+    result = subprocess.run(["rtk", "proxy", "docker", "exec", "-i", "jobjakt-dev-ai-service-1", "python", "-"],
+                            input=program, text=True, capture_output=True, timeout=500)
+    if result.returncode:
+        # Avoid forwarding exception bodies which might contain request data.
+        raise SystemExit("Local synthetic sidecar check failed; exit " + str(result.returncode))
+    record = json.loads(result.stdout)
+    assert record["status"] == "PASS" and len(record["checks"]) == 2
+    args.output.parent.mkdir(parents=True, exist_ok=True)
+    args.output.write_text(json.dumps(record, indent=2) + "\n", encoding="utf-8")
+    print(json.dumps(record))
+
+
+if __name__ == "__main__":
+    main()

dev/enable_local_workers.py

@@ -0,0 +1,74 @@
+"""Guard and apply worker settings to the existing isolated local development stack."""
+import argparse
+import hashlib
+import json
+import os
+from pathlib import Path
+import subprocess
+
+ROOT = Path(__file__).resolve().parent.parent
+SIDECAR = "sha256:8a29cb90fcc2817b0d784826d55a172c943f5ad7e1b6105e690a2d79ee15fe87"
+
+
+def capture(arguments, environment):
+    return subprocess.check_output(["rtk", "proxy", *arguments], env=environment, text=True)
+
+
+def main():
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--checkout", type=Path, required=True)
+    parser.add_argument("--bart-artifact", type=Path, required=True)
+    parser.add_argument("--output", type=Path, required=True)
+    parser.add_argument("--apply", action="store_true")
+    args = parser.parse_args()
+    checkout = args.checkout.resolve()
+    artifact = args.bart_artifact.resolve()
+    environment = {**os.environ, "JOBTRACKER_DEV_BART_ARTIFACT": str(artifact)}
+    manifest = json.loads((ROOT / "docs/redesign/phase-9/p9-sec-02/full-model/ARTIFACTS.json").read_text())["weights-only"]
+    for row in manifest["files"]:
+        path = artifact / row["file"]
+        assert path.is_file() and path.stat().st_size == row["bytes"], "Local model file missing or wrong size"
+        with path.open("rb") as stream:
+            assert hashlib.file_digest(stream, "sha256").hexdigest() == row["sha256"], "Local model integrity mismatch"
+    assert json.loads((ROOT / ".artifacts/pre-phase10-ux/local-ai/verification.json").read_text())["status"] == "PASS"
+    compose = ["docker", "compose", "--project-directory", str(checkout), "--env-file", str(checkout / ".env.dev"),
+               "-f", str(checkout / "docker-compose.dev.yml"), "-f", str(ROOT / "dev/local-workers.override.yml")]
+    config = json.loads(capture([*compose, "config", "--format", "json"], environment))
+    assert config["name"] == "jobjakt-dev", "Only the local development project is permitted"
+    backend = config["services"]["backend"]
+    sidecar = config["services"]["ai-service"]
+    settings = backend["environment"]
+    assert settings["ASPNETCORE_ENVIRONMENT"] == "Development"
+    connection = settings["ConnectionStrings__JobTracker"].lower()
+    assert "server=mariadb;" in connection and "database=jobtracker_dev;" in connection
+    assert settings["Ai__ExternalProcessingEnabled"].lower() == "false"
+    assert settings["Email__SmtpHost"] == "mailpit" and settings["Email__SmtpPort"] == "1025"
+    assert not settings["Email__SmtpUser"] and not settings["Email__SmtpPassword"]
+    assert all(port.get("host_ip") == "127.0.0.1" for port in backend.get("ports", []))
+    assert sidecar["environment"]["HF_HUB_OFFLINE"] == "1"
+    assert sidecar["environment"]["TRANSFORMERS_OFFLINE"] == "1"
+    assert sidecar["environment"]["EXTERNAL_AI_ENABLED"].lower() == "false"
+    assert sidecar["environment"]["OLLAMA_BASE_URL"] == "http://jobjakt-dev-ollama:11434"
+    assert any(volume.get("source") == str(artifact) and volume.get("read_only") for volume in sidecar["volumes"])
+    assert json.loads(capture(["docker", "image", "inspect", sidecar["image"]], environment))[0]["Id"] == SIDECAR
+    current = json.loads(capture(["docker", "inspect", "jobjakt-dev-backend-1"], environment))[0]
+    assert current["Config"]["Labels"]["com.docker.compose.project"] == "jobjakt-dev"
+    assert json.loads(capture(["docker", "image", "inspect", backend["image"]], environment))[0]["Id"] == current["Image"], "This operation changes worker configuration, not backend code"
+    record = {"project": config["name"], "verifiedBartRevision": manifest["sourceRevision"], "sidecarImage": SIDECAR,
+              "enabledWorkers": {key: value for key, value in settings.items() if key.startswith("Workers__")},
+              "emailSink": "Mailpit", "externalAI": False, "applied": False}
+    args.output.parent.mkdir(parents=True, exist_ok=True)
+    if args.apply:
+        with args.output.with_suffix(".log").open("w", encoding="utf-8") as log:
+            result = subprocess.run(["rtk", "proxy", *compose, "up", "-d", "--no-deps", "--no-build", "--pull", "never",
+                                     "--wait", "--wait-timeout", "120", "backend", "ai-service"], env=environment, stdout=log, stderr=subprocess.STDOUT)
+        record["exitCode"] = result.returncode
+        record["applied"] = result.returncode == 0
+    args.output.write_text(json.dumps(record, indent=2) + "\n", encoding="utf-8")
+    print(json.dumps(record))
+    if args.apply:
+        raise SystemExit(record["exitCode"])
+
+
+if __name__ == "__main__":
+    main()

dev/local-workers.override.yml

@@ -0,0 +1,47 @@
+# Apply only to the separately verified standalone jobjakt-dev stack.
+services:
+  backend:
+    environment:
+      Workers__RulesEnabled: "true"
+      Workers__FollowUpRemindersEnabled: "true"
+      Workers__SavedJobSearchAlertsEnabled: "true"
+      Workers__DailyExportEnabled: "true"
+      Workers__JobEnrichmentEnabled: "true"
+      Workers__AiOperationsEnabled: "true"
+      Workers__CvDocumentImportsEnabled: "true"
+      Exports__DailyEnabled: "true"
+      Email__FollowUpReminders__Enabled: "true"
+      Email__Enabled: "true"
+      Email__SmtpHost: mailpit
+      Email__SmtpPort: "1025"
+      Email__SmtpEnableSsl: "false"
+      Email__SmtpUser: ""
+      Email__SmtpPassword: ""
+      Ai__ExternalProcessingEnabled: "false"
+      Ai__RequestTimeoutSeconds: "240"
+  ai-service:
+    # Previously qualified local image contains both the legacy and isolated Docling runtimes.
+    image: jobtracker-cv-import-sidecar:dev-04
+    environment:
+      OLLAMA_BASE_URL: http://jobjakt-dev-ollama:11434
+      OLLAMA_MODEL: qwen2.5:7b
+      AI_PROVIDER: ollama
+      AI_ROUTING_MODE: local_only
+      EXTERNAL_AI_ENABLED: "false"
+      CV_IMPORT_AI_ENABLED: "false"
+      AI_SERVICE_SKIP_MODEL_LOAD: "0"
+      SKIP_MODEL_LOAD: "0"
+      HF_HOME: /models
+      HF_HUB_OFFLINE: "1"
+      TRANSFORMERS_OFFLINE: "1"
+      HF_HUB_DISABLE_TELEMETRY: "1"
+      OMP_NUM_THREADS: "2"
+      MKL_NUM_THREADS: "2"
+    volumes:
+      - type: bind
+        source: ${JOBTRACKER_DEV_BART_ARTIFACT:?Verify the isolated trained checkpoint first}
+        target: /models/hub/models--sshleifer--distilbart-cnn-12-6/snapshots/a4f8f3ea906ed274767e9906dbaede7531d660ff
+        read_only: true
+    mem_limit: 4g
+    cpus: 2
+    pids_limit: 256

dev/model-manifests/qwen2.5-7b.json

@@ -0,0 +1 @@
+{"schemaVersion":2,"mediaType":"application/vnd.docker.distribution.manifest.v2+json","config":{"mediaType":"application/vnd.docker.container.image.v1+json","digest":"sha256:2f15b3218f0552c60647ce60ada83632d2c09755b16259b13e3e4458e9ae419d","size":487},"layers":[{"mediaType":"application/vnd.ollama.image.model","digest":"sha256:2bada8a7450677000f678be90653b85d364de7db25eb5ea54136ada5f3933730","size":4683073952},{"mediaType":"application/vnd.ollama.image.system","digest":"sha256:66b9ea09bd5b7099cbb4fc820f31b575c0366fa439b08245566692c6784e281e","size":68},{"mediaType":"application/vnd.ollama.image.template","digest":"sha256:eb4402837c7829a690fa845de4d7f3fd842c2adee476d5341da8a46ea9255175","size":1482},{"mediaType":"application/vnd.ollama.image.license","digest":"sha256:832dd9e00a68dd83b3c3fb9f5588dad7dcf337a0db50f7d9483f310cd292e92e","size":11343}]}

dev/ollama.compose.yml

@@ -0,0 +1,47 @@
+# Local development only. Uses the existing isolated jobjakt-dev AI network.
+name: jobjakt-dev-ollama
+services:
+  ollama:
+    image: ollama/ollama:0.40.11@sha256:459985308d740b99071719baec2740a60f5b2b2d9b835c171991ae25690203da
+    environment:
+      OLLAMA_HOST: 0.0.0.0:11434
+      OLLAMA_NO_CLOUD: "1"
+      OLLAMA_NUM_PARALLEL: "1"
+      OLLAMA_MAX_LOADED_MODELS: "1"
+      OLLAMA_CONTEXT_LENGTH: "4096"
+      OLLAMA_KEEP_ALIVE: 5m
+    ports:
+      - "127.0.0.1:11434:11434"
+    volumes:
+      - models:/root/.ollama
+    networks:
+      ai_internal:
+        aliases: [jobjakt-dev-ollama]
+      model_download: {}
+    mem_limit: 8g
+    cpus: 4
+    pids_limit: 256
+    read_only: true
+    tmpfs:
+      - /tmp:size=256m,mode=1777
+    cap_drop: [ALL]
+    security_opt: [no-new-privileges:true]
+    restart: unless-stopped
+    healthcheck:
+      test: [CMD, ollama, list]
+      interval: 15s
+      timeout: 5s
+      retries: 5
+      start_period: 15s
+    logging:
+      options:
+        max-size: 10m
+        max-file: "3"
+networks:
+  ai_internal:
+    external: true
+    name: jobjakt-dev_ai_internal
+  model_download: {}
+volumes:
+  models:
+    name: jobjakt-dev-ollama-models

dev/verify_ollama.py

@@ -0,0 +1,56 @@
+"""Verify the dedicated development model cache before local inference."""
+import argparse
+import hashlib
+import json
+import re
+import subprocess
+import urllib.request
+from pathlib import Path
+
+MODEL = "qwen2.5:7b"
+MANIFEST = "845dbda0ea48ed749caafd9e6037047aa19acfcfd82e704d7ca97d631a0b697e"
+IMAGE = "ollama/ollama:0.40.11@sha256:459985308d740b99071719baec2740a60f5b2b2d9b835c171991ae25690203da"
+CONTAINER = "jobjakt-dev-ollama-ollama-1"
+
+
+def run(*arguments):
+    return subprocess.check_output(["rtk", "proxy", *arguments], text=True).strip()
+
+
+def main():
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--manifest", required=True, type=Path)
+    parser.add_argument("--output", required=True, type=Path)
+    args = parser.parse_args()
+    source = args.manifest.read_bytes()
+    assert hashlib.sha256(source).hexdigest() == MANIFEST, "Registry manifest changed"
+    manifest = json.loads(source)
+    container = json.loads(run("docker", "inspect", CONTAINER))[0]
+    assert container["Config"]["Labels"]["com.docker.compose.project"] == "jobjakt-dev-ollama"
+    assert container["Image"] == json.loads(run("docker", "image", "inspect", IMAGE))[0]["Id"]
+    assert "OLLAMA_NO_CLOUD=1" in container["Config"]["Env"]
+    assert container["HostConfig"]["ReadonlyRootfs"]
+    assert container["HostConfig"]["Memory"] == 8 * 1024**3
+    assert all(mount.get("Name") == "jobjakt-dev-ollama-models" for mount in container["Mounts"] if mount["Type"] == "volume")
+    descriptors = [manifest["config"], *manifest["layers"], {"digest": "sha256:" + MANIFEST}]
+    checked = []
+    for descriptor in descriptors:
+        digest = descriptor["digest"]
+        assert re.fullmatch(r"sha256:[a-f0-9]{64}", digest)
+        path = "/root/.ollama/models/blobs/" + digest.replace(":", "-")
+        actual = run("docker", "exec", CONTAINER, "sha256sum", path).split()[0]
+        assert actual == digest[7:], "Downloaded model blob changed"
+        checked.append(digest)
+    with urllib.request.urlopen("http://127.0.0.1:11434/api/tags", timeout=5) as response:
+        models = json.load(response)["models"]
+    actual = next(model for model in models if model["name"] == MODEL)
+    assert actual["digest"] == MANIFEST
+    args.output.parent.mkdir(parents=True, exist_ok=True)
+    record = {"status": "PASS", "model": MODEL, "manifestSha256": MANIFEST, "checkedBlobs": checked,
+              "image": IMAGE, "details": actual["details"], "limits": "Local artifact verification only; no inference, production digest parity or extraction accuracy claim."}
+    args.output.write_text(json.dumps(record, indent=2) + "\n", encoding="utf-8")
+    print(json.dumps(record))
+
+
+if __name__ == "__main__":
+    main()