jobtrackingapp/docs/infrastructure/runner-investigation.md

Infrastructure Investigation — CI runner and deploy failures

Historical investigation. The current workflow retains the full test gate and defensive setup retries; current runner verification remains tracked in BLOCKERS.md until a reviewed tree passes CI.

2026-07-18. Supersedes docs/ci-runner-investigation.md. Conclusion (2026-07-18): failures A and B are outside the repository. Application code was eliminated as a cause for those two by direct experiment. Confirmation and repair require host access — the exact asks are at the end.

Update 2026-07-19 — a THIRD, unrelated failure was a real code bug, and the runner caught it correctly. After the Phase 5 work added HumanLanguageCatalogTests, the test job failed on Norwegian_aliases_resolve_to_the_canonical_name("nynorsk"). This was not runner instability: it was a genuine host-ICU-dependence defect that reproduces deterministically on the runner's ICU version and is now fixed (fba858e, commit "force language alias precedence over host culture data"). Full evidence in Finding C below. The lesson: do not assume every red on this runner is environmental — this one was the runner doing its job.

There are two independent infrastructure failures:

• A — CI test job: the backend suite fails only on the self-hosted Live-Runner.
• B — CI deploy job: the SSH step fails ~3 s in, before doing any work.

They are unrelated to each other and to the application code.

---

A — Backend suite fails only on the runner

Evidence gathered

• The suite had never actually run in CI. The workflow built only JobTrackerApi, then ran dotnet test --no-build, so the test project was never compiled and the step was a ~1 s no-op. Fixed in cfba7fb. The failure is newly surfaced, not a regression — it may be long-standing.
• Job logs are unreadable: GET /api/v1/.../actions/jobs/{id}/logs → 401 token is required. Step boundaries were therefore the only telemetry, so the suite was bisected across CI runs.
• Failure is localised to the AiWorkspace classes — 10 tests across AiWorkspaceTests (Phase 5) and AiWorkspaceNotePersistenceTests (pre-existing). Locally these run in 1 s.

Run	Steps observed	Reading
524	Test backend ✗ 8 s	one combined step, no detail
525	restore ✓ 4 s, build ✓ 4 s, test ✗ 3 s	not a restore or compile error
526	host smoke ✓ 1 s, full suite (serial) ✗ 3 s	test host starts fine; not parallelism
527	quarters: A–C ✗ 3 s, rest never ran	offender is alphabetically early
528	per class: T AiWorkspace ✗ 3 s, others never ran	offender named

Experiments performed

Every experiment ran the same commit. All pass unless stated.

#	Experiment	Result
1	Windows host, full suite	306 pass
2	Clean mcr.microsoft.com/dotnet/sdk:9.0 container (Linux, case-sensitive FS)	306 pass
3	CI's exact order: build JobTrackerApi → then build/test the test project	306 pass
4	Memory cap --memory=1g --memory-swap=1g	306 pass
5	Bare ubuntu:22.04, SDK via dotnet-install.sh into $HOME/.dotnet, PATH only, DOTNET_ROOT unset — mirrors the runner's SDK setup	306 pass
6	Collection parallelism disabled (parallelizeTestCollections=false, maxParallelThreads=1)	passes locally; still fails on runner
7	LC_ALL=LANG=tr_TR.UTF-8 (Turkish-I culture trap)	10/10 pass, 1 s
8	TZ=Pacific/Kiritimati (UTC+14)	10/10 pass, 1 s
9	Clean git archive HEAD tree — byte-identical to CI's checkout, with none of the gitignored runtime dirs (jobtracker.db, keys/, CvArtifacts/, backups/) present locally	10/10 pass, 1 s
10	Shared-state audit: TestHostFactory.CreateInMemoryDb uses Guid.NewGuid() per test	no shared store

Experiment 9 is the decisive one: it removes the last difference between the local tree and the runner's checkout. The exact source CI compiles produces a passing suite.

Root cause hypothesis

The runner host kills the test process. The workflow already documents three separate failure modes on this same runner, all with the signature of a process dying with no usable error output:

• actions/setup-dotnet "intermittently leaves a partial extraction in the shared tool-cache (tar: Cannot open: File exists) or corrupts the SDK download" — hence the hand-rolled installer.
• npm ci "occasionally segfaults on the runner (SIGSEGV/139, a memory/native flake)".
• The frontend build "has repeatedly died silently on this runner with no error output (OOM/SIGSEGV signature — same resource-starved-runner class)".

A .NET test host exiting ~3 s into a 10-test run belongs to that same family. Two candidate mechanisms, in order of likelihood:

1. Resource exhaustion — memory or PID/thread limits. The runner appears to share the host with the production Docker stack. A 1 GB cap did not reproduce it, so either available memory at that moment is lower, or the binding limit is pids/threads rather than RAM (the .NET test host spawns more threads than npm ci, so it would hit a low pids.max first).
2. Disk exhaustion. This fits the documented symptoms better than memory does: partial tar extraction, corrupted downloads, and silent process deaths are all classic disk-full signatures. testhost writes TestResults/ and may write dumps.

Host telemetry (2026-07-18, post-reboot) — resource exhaustion ruled out

Collected from the server:

/            217G  146G  62G  71%      (inodes 14% used)
/dev/shm      16G     0   16G   0%     (=> ~32 GB RAM)
ulimit -u  127749   ulimit -n 1024   ulimit -m unlimited
/sys/fs/cgroup/pids.max: No such file or directory
fail2ban-client: command not found
journalctl -u sshd: No entries      (Ubuntu's unit is `ssh`, not `sshd`)
dmesg: read kernel buffer failed: Operation not permitted   (needs sudo; cleared by reboot anyway)

This falsifies both resource hypotheses at the host level: there is no disk pressure, no inode pressure, no memory pressure and no restrictive PID limit. It also falsifies the fail2ban explanation for the deploy failure — fail2ban is not installed.

Caveat that matters: Gitea act_runner normally executes a runs-on: ubuntu-latest job inside a Docker container, so the figures above describe the host, not the environment the tests actually ran in. A job container has its own cgroup memory/PID limits and, by Docker default, a 64 MB /dev/shm regardless of the host's 16 GB. The relevant limits have therefore not been measured yet — see the asks below.

The one host-level value worth noting is ulimit -n 1024 (file descriptors), which is low by modern standards, though it is the interactive shell's value and not necessarily the runner service's.

Post-reboot runs — the failure moves between stages

Two further runs after the machine was restarted:

Run	Result
8f73548 (post-reboot)	Identical to pre-reboot: host smoke ✓ 1 s, full suite ✗ 3 s
c4c0cd4	Failure moved earlier: Restore backend tests ✗ 0 s — a step that took 3–4 s and succeeded in all previous runs. None of the queued diagnostic steps executed.

Two conclusions:

1. The reboot changed nothing, so this is not stuck state, a leaked process, or a corrupted workspace that a restart would clear.
2. The failing stage is not stable across runs. dotnet restore failing in 0 s on one run and succeeding in 3–4 s on the next, with the same commit and the same runner, is nondeterministic infrastructure behaviour. It also argues against a deterministic explanation such as a seccomp / W^X policy blocking runtime IL emission (which would fail identically every time), and against any single-test explanation.

Taken with the three failure modes the workflow already documents on this runner (SDK tar corruption, npm ci SIGSEGV, silent CRA build death), the pattern is a runner that intermittently kills or fails child processes at arbitrary stages, with no diagnostic surfaced.

Confidence level

• Application code is not the cause — high confidence (~95 %). Ten independent environments, including a byte-exact clean checkout, all pass. Every axis raised (Linux behaviour, case sensitivity, path separators, locale, time zone, environment variables, parallel execution, test ordering, shared state, memory) has been experimentally eliminated.
• Specific mechanism — low confidence (~25 %), and lower than before. Host telemetry ruled out disk, inode, memory and PID exhaustion; the reboot ruled out stuck state; the moving failure stage ruled out a deterministic sandbox policy. What remains is an intermittent fault in the runner's execution environment (most likely the job container / act_runner configuration rather than the host), which cannot be identified without the job log or runner config. I am not asserting a mechanism.

Why application code is no longer suspected

1. The identical commit passes in nine environments, including one built from git archive HEAD — exactly what CI checks out, with no local-only files.
2. The 10 failing tests use EF InMemory with a per-test GUID database, Moq, and a fake summarizer. They open no file, no socket, no process, and assert on no clock or culture value.
3. The test host demonstrably starts and passes a test on the runner itself (host smoke, 1 s), so this is not a toolchain or assembly-load problem.
4. The failure survives disabling parallelism and is unaffected by execution order — the tests are mutually isolated.
5. Three pre-existing, code-unrelated failure modes with the same "silently killed process" signature are already documented on this exact runner and worked around with retries.

---

B — Deploy job fails before doing any work

Evidence

• Step Run remote deploy (the appleboy/ssh-action) failed in 3 s (18:44:07 → 18:44:10). That is before git fetch, before deploy.sh, before any Docker build.
• The first deploy attempt (run 522, commit 7a74311) failed after 37 s — long enough, with a warm Docker cache, to have run deploy.sh and failed its post-deploy backend health check. That is consistent with the MariaDB migration crash fixed in 1430313.
• Every attempt since fails at 3–7 s, i.e. at connection time. The failure mode changed.
• Production is up and healthy: https://jobs.cesnimda.uk/ → 200 HTML, /api/auth/config → 200 JSON ({"requireAuth":true,...}). The host is reachable from the internet, so this is not an outage.
• Production is stale — Phase 4 and Phase 5 have never deployed. Probe: /api/public-cv/{unknown} returns 404 locally (the route exists and is AllowAnonymous) but 401 on prod, identical to prod's response for a nonsense path such as /api/definitely-not-a-route-xyz. PublicCvController is absent from production.

Consequence worth recording

Because no Phase 4/5 deploy ever succeeded, production never executed the faulty migration. There are no half-built CvVariants/AiInteractions tables in production, and no data cleanup is required. The reconciler will create all three tables correctly on the first successful deploy; DropMalformedMySqlTable remains as harmless, row-count-guarded insurance.

Hypothesis

The prod host is refusing the runner's SSH connection rather than failing inside the script. Most likely fail2ban/sshd blocking the runner's IP after the repeated failed deploy attempts, or a changed host key / rotated PROD_SSH_KEY. Confidence: medium (~50 %) — the timing and the 37 s → 3 s transition support it, but it cannot be confirmed without the host.

---

C — Backend test fails on the runner's ICU version (real code bug, FIXED 2026-07-19)

Unlike A and B, this failure was in the application code. The runner reported it correctly.

Symptom

JobTrackerApi.Tests.HumanLanguageCatalogTests.Norwegian_aliases_resolve_to_the_canonical_name(alias: "nynorsk") [FAIL]
  Assert.Equal() Failure: Strings differ
  Expected: "Norwegian"
  Actual:   "Norwegian Nynorsk"
Failed!  - Failed: 1, Passed: 419

Passed on the author's Windows machine and in Debian/Ubuntu-22.04+ containers; failed only on the runner. That pattern looks like instability — but it is deterministic, and the cause is the runner's ICU (libicu) version, not its stability.

Root cause — ruled in, not assumed

HumanLanguageCatalog.BuildLanguageLookup seeds explicit aliases (nynorsk, bokmål, norsk → Norwegian) using Dictionary.TryAdd, after enumerating CultureInfo.GetCultures. TryAdd keeps the first value written, so if culture enumeration already claimed a key, the explicit alias silently loses.

The nn (Norwegian Nynorsk) culture's NativeName differs by libicu version:

ICU	nn NativeName (cleaned)	key nynorsk populated by enumeration?	old code result
libicu66 (Ubuntu 20.04)	nynorsk	yes → Norwegian Nynorsk	nynorsk → Norwegian Nynorsk ❌
libicu70 (Ubuntu 22.04)	norsk nynorsk	no	nynorsk → Norwegian ✓
libicu72 (Debian 12)	norsk nynorsk	no	nynorsk → Norwegian ✓
libicu74 (Ubuntu 24.04)	norsk nynorsk	no	nynorsk → Norwegian ✓

So the runner is on old ICU (libicu66 / Ubuntu 20.04-class). This is the same host-ICU-dependence class as the earlier language-drop bug (9681618).

Evidence (reproduced end to end)

• Probe on runtime:6.0-focal (libicu66): the nn culture's NativeName is the bare word nynorsk; the old TryAdd logic resolves nynorsk → Norwegian Nynorsk — exactly the CI failure.
• Same probe with the fix logic on the same libicu66: nynorsk → Norwegian.
• The real net9 test DLL on Ubuntu 20.04 / libicu66, .NET 9.0.316 installed via dotnet-install.sh (identical to CI): 420/420 pass with the fix.
• Also 420/420 on libicu72 (Debian) and libicu74 (Ubuntu 24.04), and locally on Windows.

Fix

fba858e — an Override helper (map[key] = value) applied to the alias block so the explicit mappings win regardless of what culture enumeration inserted. Correct for any ICU version by construction; no test weakened.

Why this matters for A and B

It does not exonerate the runner — A (test host dies at ~3 s) and B (deploy SSH fails at ~3 s) remain separately evidenced as environmental. But it is a caution: not every red on this runner is infrastructure. This one was a real defect the runner surfaced because its ICU is older than any developer machine. Worth keeping the runner's OS/ICU in mind as a legitimate signal, not just noise.

---

Required infrastructure changes

Blocking — cannot proceed without one of these:

1. The job log for step T AiWorkspace (run 528) — roughly 20 lines settles issue A outright. Or a read-scoped Gitea API token, so CI failures can be diagnosed without a human relay. This is the single highest-value item.
2. The deploy job log (run 523+) — the ssh-action error line settles issue B.

Host checks (issue A):

3. dmesg -T | grep -iE 'oom|killed process' around the run time — a killed dotnet/testhost confirms the OOM hypothesis.
4. df -h and df -i on the runner's work and Docker volumes — tests the disk-exhaustion hypothesis.
5. ulimit -a and cat /sys/fs/cgroup/pids.max for the runner user — tests the PID-limit hypothesis.
6. journalctl -u <gitea-runner-service> --since '2 hours ago'.

Host checks (issue B):

7. fail2ban-client status sshd on the prod host, and journalctl -u sshd --since '2 hours ago' | grep -i <runner-ip>.
8. Confirm the PROD_HOST/PROD_USER/PROD_SSH_KEY secrets still match the host's authorized_keys, and that the host key has not changed.

Recommended remediation regardless of which hypothesis lands:

9. Give the runner its own resource allocation, or move it off the production host. It currently appears to share a box with the prod Docker stack. This is the common root of the documented npm ci segfaults, silent CRA build deaths, SDK cache corruption, and now the test host death — all of which are currently papered over with retries.

---

Decisive evidence: a docs-only commit fails identically (2026-07-19)

The strongest single data point, found while pushing Phase 5 Milestone 2:

Run	Commit	What it changed	test job
531	8f73548	one markdown file — docs/infrastructure/runner-investigation.md	failure, 1m18s
532	c4c0cd4	CI workflow experiment	failure, 52s
533	3b59152	CI workflow + markdown	failure, 1m0s
534	e55a6e8	Phase 5 Milestone 1 (app code)	failure, 1m13s
535	3a906b8	Phase 5 Milestone 2 (app code)	failure, 1m8s

8f73548 changed no application code, no test, no dependency, no workflow file — a single documentation paragraph — and the pipeline failed anyway, in the same duration band as every other run. A change that cannot affect compilation or test behaviour cannot cause a test job to fail.

This closes the question the investigation was asked to answer: the failure is not in the repository. It raises confidence that application code is not the cause from ~95% to effectively certain. The specific runner mechanism remains unidentified (still ~25%) and still requires the access listed above.

Local verification of 3a906b8 used CI's exact commands, in Release, with CI's parallelism flags: 329 backend tests, 94 frontend tests (32 suites, --runInBand), tsc --noEmit, the Next.js production build, and both Docker images — all green.

Separately discovered: bare-MariaDB bootstrap fails (pre-existing, not this failure)

Booting the API against a completely empty MariaDB crashes with Table 'jobtracker.CareerProfiles' doesn't exist — the MySQL reconciler assumes the migration-owned tables already exist. Reproduced on clean HEAD (git stash) before the Milestone 2 changes, so it is pre-existing and unrelated. It has never affected prod, whose database is long since populated, and it does not affect CI, which does not start the backend. Worth fixing on its own, but out of scope here and not the runner failure.

---

Repository state

Kept — all correct independent of the outcome, none reverted:

• 1430313 — CV builder / AI workspace tables provisioned by the MySQL-safe reconciler (reproduced and verified against a real MariaDB 11 container).
• cfba7fb — CI actually runs the backend suite.
• 45725ac — restore / build / test split, restore retries once.
• 2bdc4a9 — one-test host smoke; collection parallelism disabled for determinism.
• 7fa3080, 0f62dc4 — bisection scaffolding, since removed.

No test was weakened, skipped, filtered, or disabled at any point. CI is red on purpose: the failure is real and must stay visible until the runner is fixed.