# CI Runner Investigation — backend suite fails only on `Live-Runner`

> 2026-07-18. Status: **root cause narrowed to the runner host; not reproducible in any other
> environment.** Final confirmation needs the job log, which the Gitea API will not serve without a
> token. Recommended infrastructure fix at the bottom.

## Summary

The backend test suite (306 tests) passes everywhere it has been run except the self-hosted Gitea
runner, where the test step dies after ~3 s. Restore and build succeed; the test host starts and can
run a test. The failure is localised to the **`AiWorkspace` test classes** (10 tests).

Important context: **this suite had never actually executed in CI.** The workflow built only
`JobTrackerApi` and then ran `dotnet test --no-build`, so the test project was never compiled and the
step was a ~1 s no-op (fixed in `cfba7fb`). The failure is therefore *newly surfaced*, not a
regression — it may have been present for a long time.

## How it was narrowed

Job logs return `401 token is required` from the Gitea API, so step boundaries were the only readable
telemetry. The suite was split and bisected across CI steps over four runs.

| Run | Observation |
|---|---|
| 524 | `Test backend` fails at 8 s (restore+build+test in one step) |
| 525 | Split: restore 4 s ✓, build 4 s ✓, **test 3 s ✗** → not a compile/restore error |
| 526 | One-test host smoke **1 s ✓**, full suite (parallelism off) **3 s ✗** → host starts fine; not parallelism |
| 527 | Alphabetical quarters: **slice 1 (A–C) ✗ at 3 s**, later slices never ran |
| 528 | Per class: **`T AiWorkspace` ✗ at 3 s**; all other A–C classes never reached |

The `FullyQualifiedName~AiWorkspace` filter matches 10 tests across `AiWorkspaceTests` (Phase 5) and
`AiWorkspaceNotePersistenceTests` (pre-existing). Locally that filter completes in **1 s**.

## What was ruled out (evidence)

Each was reproduced against the same commit, all 306 tests passing unless noted.

| Hypothesis | Test performed | Result |
|---|---|---|
| Linux-specific behaviour | clean `mcr.microsoft.com/dotnet/sdk:9.0` container | 306 pass |
| File system case sensitivity | same (Linux is case-sensitive) | 306 pass |
| Path separators | same | 306 pass |
| CI step ordering | reproduced CI's exact build-then-test order | 306 pass |
| Memory pressure | `--memory=1g --memory-swap=1g` | 306 pass |
| Environment variables / SDK layout | bare `ubuntu:22.04`, SDK via `dotnet-install.sh` to `$HOME/.dotnet`, `PATH` only, **`DOTNET_ROOT` unset** (mirrors the runner) | 306 pass |
| Parallel execution | `xUnit.parallelizeTestCollections=false xUnit.maxParallelThreads=1` | passes locally; **still fails on runner** |
| Test ordering / shared state | `TestHostFactory.CreateInMemoryDb` uses `Guid.NewGuid()` per test — no shared store | isolated |
| Locale / culture | `LANG=LC_ALL=tr_TR.UTF-8` (Turkish-I trap) | 10/10 pass in 1 s |
| Time zone | `TZ=Pacific/Kiritimati` (UTC+14) | 10/10 pass in 1 s |
| Docker availability | these tests use EF InMemory + Moq; no Docker, network, or filesystem use | n/a |
| Permissions | not testable remotely | **unresolved** |

## Assessment: environmental, not code

The `AiWorkspace` tests use EF InMemory with a per-test GUID database, `Moq`, and a fake summarizer.
They touch no clock-sensitive assertion, no file, no socket, and no external process. They pass under
every adverse condition that could be simulated. That points at the runner host itself.

The workflow already documents three separate failure modes on this exact runner, all with the same
signature — **a process dying with no usable error output**:

- `actions/setup-dotnet` "intermittently leaves a partial extraction in the shared tool-cache … or
  corrupts the SDK download" (hence the hand-rolled `dotnet-install.sh` + retry).
- `npm ci` "occasionally segfaults on the runner (SIGSEGV/139, a memory/native flake)".
- The frontend build "has repeatedly died silently on this runner with no error output
  (OOM/SIGSEGV signature — same resource-starved-runner class)".

A .NET test host exiting ~3 s into a 10-test run, on a host with a documented history of silently
killed processes, is consistent with the same resource starvation — most plausibly the OOM killer or
a cgroup limit, triggered while the runner shares the box with the production Docker stack. A 1 GB
container cap was not enough to reproduce it, so the runner's *available* memory at that moment is
likely lower, or the limit is on process/thread count rather than memory.

## Recommended infrastructure fix

In priority order:

1. **Read the job log** for step `T AiWorkspace` (run 528) — 20 lines settles this immediately. Either
   grant a read-scoped Gitea token so CI failures can be diagnosed without a human relay, or paste the
   step output. Everything below is contingent on that.
2. **Check the host for OOM kills** around the run: `dmesg -T | grep -i -E 'oom|killed process'` and
   `journalctl -u <gitea-runner> --since '1 hour ago'`. A killed `testhost`/`dotnet` confirms it.
3. **Give the runner headroom / isolation.** It currently appears to share the host with the prod
   stack. Either cap the prod containers' memory, give the runner its own cgroup allocation, or move
   it off the production host. This would also fix the pre-existing npm/CRA/SDK flakes the workflow
   works around with retries.
4. **Check `ulimit`/cgroup pids** for the runner user (`ulimit -a`, `cat /sys/fs/cgroup/pids.max`).
   The .NET test host spawns more threads than `npm ci`, so a low pids limit would hit it first.

## Repository-side changes made during the investigation

Kept, because they are correct independent of the outcome:

- `cfba7fb` — CI actually runs the backend suite (dropped the no-op `--no-build`).
- `45725ac` — restore / build / test split into separate steps, restore retries once.
- `2bdc4a9` — one-test host smoke step; collection parallelism disabled for determinism.
- Bisection scaffolding was removed once it had served its purpose (`7fa3080`, `0f62dc4`).

No test was weakened, skipped, or filtered at any point. CI remains red on purpose — the failure is
real and should stay visible until the runner is fixed.