jobtrackingapp/docs/ci-runner-investigation.md

CI Runner Investigation — backend suite fails only on Live-Runner

2026-07-18. Status: root cause narrowed to the runner host; not reproducible in any other environment. Final confirmation needs the job log, which the Gitea API will not serve without a token. Recommended infrastructure fix at the bottom.

Summary

The backend test suite (306 tests) passes everywhere it has been run except the self-hosted Gitea runner, where the test step dies after ~3 s. Restore and build succeed; the test host starts and can run a test. The failure is localised to the AiWorkspace test classes (10 tests).

Important context: this suite had never actually executed in CI. The workflow built only JobTrackerApi and then ran dotnet test --no-build, so the test project was never compiled and the step was a ~1 s no-op (fixed in cfba7fb). The failure is therefore newly surfaced, not a regression — it may have been present for a long time.

How it was narrowed

Job logs return 401 token is required from the Gitea API, so step boundaries were the only readable telemetry. The suite was split and bisected across CI steps over four runs.

Run	Observation
524	Test backend fails at 8 s (restore+build+test in one step)
525	Split: restore 4 s ✓, build 4 s ✓, test 3 s ✗ → not a compile/restore error
526	One-test host smoke 1 s ✓, full suite (parallelism off) 3 s ✗ → host starts fine; not parallelism
527	Alphabetical quarters: slice 1 (A–C) ✗ at 3 s, later slices never ran
528	Per class: T AiWorkspace ✗ at 3 s; all other A–C classes never reached

The FullyQualifiedName~AiWorkspace filter matches 10 tests across AiWorkspaceTests (Phase 5) and AiWorkspaceNotePersistenceTests (pre-existing). Locally that filter completes in 1 s.

What was ruled out (evidence)

Each was reproduced against the same commit, all 306 tests passing unless noted.

Hypothesis	Test performed	Result
Linux-specific behaviour	clean mcr.microsoft.com/dotnet/sdk:9.0 container	306 pass
File system case sensitivity	same (Linux is case-sensitive)	306 pass
Path separators	same	306 pass
CI step ordering	reproduced CI's exact build-then-test order	306 pass
Memory pressure	--memory=1g --memory-swap=1g	306 pass
Environment variables / SDK layout	bare ubuntu:22.04, SDK via dotnet-install.sh to $HOME/.dotnet, PATH only, DOTNET_ROOT unset (mirrors the runner)	306 pass
Parallel execution	xUnit.parallelizeTestCollections=false xUnit.maxParallelThreads=1	passes locally; still fails on runner
Test ordering / shared state	TestHostFactory.CreateInMemoryDb uses Guid.NewGuid() per test — no shared store	isolated
Locale / culture	LANG=LC_ALL=tr_TR.UTF-8 (Turkish-I trap)	10/10 pass in 1 s
Time zone	TZ=Pacific/Kiritimati (UTC+14)	10/10 pass in 1 s
Docker availability	these tests use EF InMemory + Moq; no Docker, network, or filesystem use	n/a
Permissions	not testable remotely	unresolved

Assessment: environmental, not code

The AiWorkspace tests use EF InMemory with a per-test GUID database, Moq, and a fake summarizer. They touch no clock-sensitive assertion, no file, no socket, and no external process. They pass under every adverse condition that could be simulated. That points at the runner host itself.

The workflow already documents three separate failure modes on this exact runner, all with the same signature — a process dying with no usable error output:

• actions/setup-dotnet "intermittently leaves a partial extraction in the shared tool-cache … or corrupts the SDK download" (hence the hand-rolled dotnet-install.sh + retry).
• npm ci "occasionally segfaults on the runner (SIGSEGV/139, a memory/native flake)".
• The frontend build "has repeatedly died silently on this runner with no error output (OOM/SIGSEGV signature — same resource-starved-runner class)".

A .NET test host exiting ~3 s into a 10-test run, on a host with a documented history of silently killed processes, is consistent with the same resource starvation — most plausibly the OOM killer or a cgroup limit, triggered while the runner shares the box with the production Docker stack. A 1 GB container cap was not enough to reproduce it, so the runner's available memory at that moment is likely lower, or the limit is on process/thread count rather than memory.

Recommended infrastructure fix

In priority order:

1. Read the job log for step T AiWorkspace (run 528) — 20 lines settles this immediately. Either grant a read-scoped Gitea token so CI failures can be diagnosed without a human relay, or paste the step output. Everything below is contingent on that.
2. Check the host for OOM kills around the run: dmesg -T | grep -i -E 'oom|killed process' and journalctl -u <gitea-runner> --since '1 hour ago'. A killed testhost/dotnet confirms it.
3. Give the runner headroom / isolation. It currently appears to share the host with the prod stack. Either cap the prod containers' memory, give the runner its own cgroup allocation, or move it off the production host. This would also fix the pre-existing npm/CRA/SDK flakes the workflow works around with retries.
4. Check ulimit/cgroup pids for the runner user (ulimit -a, cat /sys/fs/cgroup/pids.max). The .NET test host spawns more threads than npm ci, so a low pids limit would hit it first.

Repository-side changes made during the investigation

Kept, because they are correct independent of the outcome:

• cfba7fb — CI actually runs the backend suite (dropped the no-op --no-build).
• 45725ac — restore / build / test split into separate steps, restore retries once.
• 2bdc4a9 — one-test host smoke step; collection parallelism disabled for determinism.
• Bisection scaffolding was removed once it had served its purpose (7fa3080, 0f62dc4).

No test was weakened, skipped, or filtered at any point. CI remains red on purpose — the failure is real and should stay visible until the runner is fixed.