docs(ci): report runner-only backend test failure investigation

Bisected the failure across four CI runs (job logs are not readable via the
Gitea API) down to the AiWorkspace test classes — 10 tests that pass on
Windows, in a clean Linux container, under a 1GB memory cap, in CI's exact
step order, with a custom-dir SDK and no DOTNET_ROOT, serially, and under a
hostile locale/timezone.

Ruled out: Linux behaviour, case sensitivity, path separators, locale/culture,
time zone, environment variables, parallel execution, test ordering, shared
state, memory. Not testable remotely: host permissions/limits.

Assessment is environmental: the workflow already documents three failure
modes on this same runner with an identical signature (processes dying with no
error output — SDK cache corruption, npm ci SIGSEGV, CRA build OOM/SIGSEGV).
Report includes evidence table and recommended infrastructure fix.

Removes the temporary bisection scaffolding; keeps the restore/build/test split
and the host smoke. No test was weakened, skipped, or filtered.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

.gitea/workflows/ci-deploy.yml

@@ -62,47 +62,14 @@ jobs:
       - name: Test backend (host smoke)
         run: dotnet test JobTrackerApi.Tests/JobTrackerApi.Tests.csproj --configuration Release --no-build --filter "FullyQualifiedName~CvBuilderTests.Every_catalog_theme_renders_valid_html"
 
-      # The host smoke above passes, so the test host starts fine; the full suite still dies ~3s in
+      # Runner-only failure, localised to the AiWorkspace tests by bisecting across steps (the job
-      # even with parallelism disabled, i.e. one specific test takes the process down on this runner
+      # log is not readable via the Gitea API). Those 10 tests pass on Windows, in a clean Linux
-      # only. Job logs are not readable via the API, so the suite is bisected across steps: the first
+      # container, under a 1GB memory cap, with a custom-dir SDK and no DOTNET_ROOT, serially, and
-      # failing step localises the offending class. Slices are alphabetical because execution order is
+      # under a hostile locale/timezone -- so the trigger is specific to this runner host.
-      # alphabetical once collection parallelism is off. Temporary diagnostic — every test still runs.
+      # See docs/ci-runner-investigation.md. Collection parallelism stays off for determinism, the
-      # Slice 1 (A-C) is the failing slice; split per class to name the offender in one run.
+      # same reason the frontend runs --runInBand. Every test still runs; nothing is filtered.
-      - name: T AiWorkspace
+      - name: Test backend
-        run: dotnet test JobTrackerApi.Tests/JobTrackerApi.Tests.csproj --configuration Release --no-build -- xUnit.parallelizeTestCollections=false xUnit.maxParallelThreads=1 --filter "FullyQualifiedName~AiWorkspace"
+        run: dotnet test JobTrackerApi.Tests/JobTrackerApi.Tests.csproj --configuration Release --no-build -- xUnit.parallelizeTestCollections=false xUnit.maxParallelThreads=1
-
-      - name: T Attachment
-        run: dotnet test JobTrackerApi.Tests/JobTrackerApi.Tests.csproj --configuration Release --no-build -- xUnit.parallelizeTestCollections=false xUnit.maxParallelThreads=1 --filter "FullyQualifiedName~Attachment"
-
-      - name: T AuthAndSystem
-        run: dotnet test JobTrackerApi.Tests/JobTrackerApi.Tests.csproj --configuration Release --no-build -- xUnit.parallelizeTestCollections=false xUnit.maxParallelThreads=1 --filter "FullyQualifiedName~AuthAndSystem"
-
-      - name: T BackupController
-        run: dotnet test JobTrackerApi.Tests/JobTrackerApi.Tests.csproj --configuration Release --no-build -- xUnit.parallelizeTestCollections=false xUnit.maxParallelThreads=1 --filter "FullyQualifiedName~BackupController"
-
-      - name: T CareerProfile
-        run: dotnet test JobTrackerApi.Tests/JobTrackerApi.Tests.csproj --configuration Release --no-build -- xUnit.parallelizeTestCollections=false xUnit.maxParallelThreads=1 --filter "FullyQualifiedName~CareerProfile"
-
-      - name: T ClientErrors
-        run: dotnet test JobTrackerApi.Tests/JobTrackerApi.Tests.csproj --configuration Release --no-build -- xUnit.parallelizeTestCollections=false xUnit.maxParallelThreads=1 --filter "FullyQualifiedName~ClientErrors"
-
-      - name: T Correspondence
-        run: dotnet test JobTrackerApi.Tests/JobTrackerApi.Tests.csproj --configuration Release --no-build -- xUnit.parallelizeTestCollections=false xUnit.maxParallelThreads=1 --filter "FullyQualifiedName~Correspondence"
-
-      - name: T CvBuilder
-        run: dotnet test JobTrackerApi.Tests/JobTrackerApi.Tests.csproj --configuration Release --no-build -- xUnit.parallelizeTestCollections=false xUnit.maxParallelThreads=1 --filter "FullyQualifiedName~CvBuilder"
-
-      - name: T CvCorpus
-        run: dotnet test JobTrackerApi.Tests/JobTrackerApi.Tests.csproj --configuration Release --no-build -- xUnit.parallelizeTestCollections=false xUnit.maxParallelThreads=1 --filter "FullyQualifiedName~CvCorpus"
-
-      - name: Test backend (slice 2 D-J)
-        run: dotnet test JobTrackerApi.Tests/JobTrackerApi.Tests.csproj --configuration Release --no-build -- xUnit.parallelizeTestCollections=false xUnit.maxParallelThreads=1 --filter "FullyQualifiedName~DatabaseBackup|FullyQualifiedName~EmailStatus|FullyQualifiedName~GmailController|FullyQualifiedName~GoogleToken|FullyQualifiedName~Imap|FullyQualifiedName~InterviewPrep|FullyQualifiedName~JobApplications"
-
-      - name: Test backend (slice 3 J-P)
-        run: dotnet test JobTrackerApi.Tests/JobTrackerApi.Tests.csproj --configuration Release --no-build -- xUnit.parallelizeTestCollections=false xUnit.maxParallelThreads=1 --filter "FullyQualifiedName~JobCvMatch|FullyQualifiedName~JobImport|FullyQualifiedName~JobPipeline|FullyQualifiedName~LocalAuth|FullyQualifiedName~MicrosoftGraph|FullyQualifiedName~MicrosoftToken|FullyQualifiedName~Ownership|FullyQualifiedName~ProductionConfig|FullyQualifiedName~ProfileCv"
-
-      - name: Test backend (slice 4 R-T)
-        run: dotnet test JobTrackerApi.Tests/JobTrackerApi.Tests.csproj --configuration Release --no-build -- xUnit.parallelizeTestCollections=false xUnit.maxParallelThreads=1 --filter "FullyQualifiedName~RulesEngine|FullyQualifiedName~Sessions|FullyQualifiedName~SqliteMigration|FullyQualifiedName~StageAnalytics|FullyQualifiedName~Summarizer|FullyQualifiedName~TwoFactor"
 
       - name: Install frontend deps
         working-directory: job-tracker-ui

docs/ci-runner-investigation.md

@@ -0,0 +1,100 @@
+# CI Runner Investigation — backend suite fails only on `Live-Runner`
+
+> 2026-07-18. Status: **root cause narrowed to the runner host; not reproducible in any other
+> environment.** Final confirmation needs the job log, which the Gitea API will not serve without a
+> token. Recommended infrastructure fix at the bottom.
+
+## Summary
+
+The backend test suite (306 tests) passes everywhere it has been run except the self-hosted Gitea
+runner, where the test step dies after ~3 s. Restore and build succeed; the test host starts and can
+run a test. The failure is localised to the **`AiWorkspace` test classes** (10 tests).
+
+Important context: **this suite had never actually executed in CI.** The workflow built only
+`JobTrackerApi` and then ran `dotnet test --no-build`, so the test project was never compiled and the
+step was a ~1 s no-op (fixed in `cfba7fb`). The failure is therefore *newly surfaced*, not a
+regression — it may have been present for a long time.
+
+## How it was narrowed
+
+Job logs return `401 token is required` from the Gitea API, so step boundaries were the only readable
+telemetry. The suite was split and bisected across CI steps over four runs.
+
+| Run | Observation |
+|---|---|
+| 524 | `Test backend` fails at 8 s (restore+build+test in one step) |
+| 525 | Split: restore 4 s ✓, build 4 s ✓, **test 3 s ✗** → not a compile/restore error |
+| 526 | One-test host smoke **1 s ✓**, full suite (parallelism off) **3 s ✗** → host starts fine; not parallelism |
+| 527 | Alphabetical quarters: **slice 1 (A–C) ✗ at 3 s**, later slices never ran |
+| 528 | Per class: **`T AiWorkspace` ✗ at 3 s**; all other A–C classes never reached |
+
+The `FullyQualifiedName~AiWorkspace` filter matches 10 tests across `AiWorkspaceTests` (Phase 5) and
+`AiWorkspaceNotePersistenceTests` (pre-existing). Locally that filter completes in **1 s**.
+
+## What was ruled out (evidence)
+
+Each was reproduced against the same commit, all 306 tests passing unless noted.
+
+| Hypothesis | Test performed | Result |
+|---|---|---|
+| Linux-specific behaviour | clean `mcr.microsoft.com/dotnet/sdk:9.0` container | 306 pass |
+| File system case sensitivity | same (Linux is case-sensitive) | 306 pass |
+| Path separators | same | 306 pass |
+| CI step ordering | reproduced CI's exact build-then-test order | 306 pass |
+| Memory pressure | `--memory=1g --memory-swap=1g` | 306 pass |
+| Environment variables / SDK layout | bare `ubuntu:22.04`, SDK via `dotnet-install.sh` to `$HOME/.dotnet`, `PATH` only, **`DOTNET_ROOT` unset** (mirrors the runner) | 306 pass |
+| Parallel execution | `xUnit.parallelizeTestCollections=false xUnit.maxParallelThreads=1` | passes locally; **still fails on runner** |
+| Test ordering / shared state | `TestHostFactory.CreateInMemoryDb` uses `Guid.NewGuid()` per test — no shared store | isolated |
+| Locale / culture | `LANG=LC_ALL=tr_TR.UTF-8` (Turkish-I trap) | 10/10 pass in 1 s |
+| Time zone | `TZ=Pacific/Kiritimati` (UTC+14) | 10/10 pass in 1 s |
+| Docker availability | these tests use EF InMemory + Moq; no Docker, network, or filesystem use | n/a |
+| Permissions | not testable remotely | **unresolved** |
+
+## Assessment: environmental, not code
+
+The `AiWorkspace` tests use EF InMemory with a per-test GUID database, `Moq`, and a fake summarizer.
+They touch no clock-sensitive assertion, no file, no socket, and no external process. They pass under
+every adverse condition that could be simulated. That points at the runner host itself.
+
+The workflow already documents three separate failure modes on this exact runner, all with the same
+signature — **a process dying with no usable error output**:
+
+- `actions/setup-dotnet` "intermittently leaves a partial extraction in the shared tool-cache … or
+  corrupts the SDK download" (hence the hand-rolled `dotnet-install.sh` + retry).
+- `npm ci` "occasionally segfaults on the runner (SIGSEGV/139, a memory/native flake)".
+- The frontend build "has repeatedly died silently on this runner with no error output
+  (OOM/SIGSEGV signature — same resource-starved-runner class)".
+
+A .NET test host exiting ~3 s into a 10-test run, on a host with a documented history of silently
+killed processes, is consistent with the same resource starvation — most plausibly the OOM killer or
+a cgroup limit, triggered while the runner shares the box with the production Docker stack. A 1 GB
+container cap was not enough to reproduce it, so the runner's *available* memory at that moment is
+likely lower, or the limit is on process/thread count rather than memory.
+
+## Recommended infrastructure fix
+
+In priority order:
+
+1. **Read the job log** for step `T AiWorkspace` (run 528) — 20 lines settles this immediately. Either
+   grant a read-scoped Gitea token so CI failures can be diagnosed without a human relay, or paste the
+   step output. Everything below is contingent on that.
+2. **Check the host for OOM kills** around the run: `dmesg -T | grep -i -E 'oom|killed process'` and
+   `journalctl -u <gitea-runner> --since '1 hour ago'`. A killed `testhost`/`dotnet` confirms it.
+3. **Give the runner headroom / isolation.** It currently appears to share the host with the prod
+   stack. Either cap the prod containers' memory, give the runner its own cgroup allocation, or move
+   it off the production host. This would also fix the pre-existing npm/CRA/SDK flakes the workflow
+   works around with retries.
+4. **Check `ulimit`/cgroup pids** for the runner user (`ulimit -a`, `cat /sys/fs/cgroup/pids.max`).
+   The .NET test host spawns more threads than `npm ci`, so a low pids limit would hit it first.
+
+## Repository-side changes made during the investigation
+
+Kept, because they are correct independent of the outcome:
+
+- `cfba7fb` — CI actually runs the backend suite (dropped the no-op `--no-build`).
+- `45725ac` — restore / build / test split into separate steps, restore retries once.
+- `2bdc4a9` — one-test host smoke step; collection parallelism disabled for determinism.
+- Bisection scaffolding was removed once it had served its purpose (`7fa3080`, `0f62dc4`).
+
+No test was weakened, skipped, or filtered at any point. CI remains red on purpose — the failure is
+real and should stay visible until the runner is fixed.