jobtrackingapp/docs/infrastructure/runner-investigation.md

Infrastructure Investigation — CI runner and deploy failures

2026-07-18. Supersedes docs/ci-runner-investigation.md. Conclusion: both failures are outside the repository. Application code has been eliminated as a cause by direct experiment. Confirmation and repair require host access — the exact asks are at the end.

There are two independent infrastructure failures:

• A — CI test job: the backend suite fails only on the self-hosted Live-Runner.
• B — CI deploy job: the SSH step fails ~3 s in, before doing any work.

They are unrelated to each other and to the application code.

---

A — Backend suite fails only on the runner

Evidence gathered

• The suite had never actually run in CI. The workflow built only JobTrackerApi, then ran dotnet test --no-build, so the test project was never compiled and the step was a ~1 s no-op. Fixed in cfba7fb. The failure is newly surfaced, not a regression — it may be long-standing.
• Job logs are unreadable: GET /api/v1/.../actions/jobs/{id}/logs → 401 token is required. Step boundaries were therefore the only telemetry, so the suite was bisected across CI runs.
• Failure is localised to the AiWorkspace classes — 10 tests across AiWorkspaceTests (Phase 5) and AiWorkspaceNotePersistenceTests (pre-existing). Locally these run in 1 s.

Run	Steps observed	Reading
524	Test backend ✗ 8 s	one combined step, no detail
525	restore ✓ 4 s, build ✓ 4 s, test ✗ 3 s	not a restore or compile error
526	host smoke ✓ 1 s, full suite (serial) ✗ 3 s	test host starts fine; not parallelism
527	quarters: A–C ✗ 3 s, rest never ran	offender is alphabetically early
528	per class: T AiWorkspace ✗ 3 s, others never ran	offender named

Experiments performed

Every experiment ran the same commit. All pass unless stated.

#	Experiment	Result
1	Windows host, full suite	306 pass
2	Clean mcr.microsoft.com/dotnet/sdk:9.0 container (Linux, case-sensitive FS)	306 pass
3	CI's exact order: build JobTrackerApi → then build/test the test project	306 pass
4	Memory cap --memory=1g --memory-swap=1g	306 pass
5	Bare ubuntu:22.04, SDK via dotnet-install.sh into $HOME/.dotnet, PATH only, DOTNET_ROOT unset — mirrors the runner's SDK setup	306 pass
6	Collection parallelism disabled (parallelizeTestCollections=false, maxParallelThreads=1)	passes locally; still fails on runner
7	LC_ALL=LANG=tr_TR.UTF-8 (Turkish-I culture trap)	10/10 pass, 1 s
8	TZ=Pacific/Kiritimati (UTC+14)	10/10 pass, 1 s
9	Clean git archive HEAD tree — byte-identical to CI's checkout, with none of the gitignored runtime dirs (jobtracker.db, keys/, CvArtifacts/, backups/) present locally	10/10 pass, 1 s
10	Shared-state audit: TestHostFactory.CreateInMemoryDb uses Guid.NewGuid() per test	no shared store

Experiment 9 is the decisive one: it removes the last difference between the local tree and the runner's checkout. The exact source CI compiles produces a passing suite.

Root cause hypothesis

The runner host kills the test process. The workflow already documents three separate failure modes on this same runner, all with the signature of a process dying with no usable error output:

• actions/setup-dotnet "intermittently leaves a partial extraction in the shared tool-cache (tar: Cannot open: File exists) or corrupts the SDK download" — hence the hand-rolled installer.
• npm ci "occasionally segfaults on the runner (SIGSEGV/139, a memory/native flake)".
• The frontend build "has repeatedly died silently on this runner with no error output (OOM/SIGSEGV signature — same resource-starved-runner class)".

A .NET test host exiting ~3 s into a 10-test run belongs to that same family. Two candidate mechanisms, in order of likelihood:

1. Resource exhaustion — memory or PID/thread limits. The runner appears to share the host with the production Docker stack. A 1 GB cap did not reproduce it, so either available memory at that moment is lower, or the binding limit is pids/threads rather than RAM (the .NET test host spawns more threads than npm ci, so it would hit a low pids.max first).
2. Disk exhaustion. This fits the documented symptoms better than memory does: partial tar extraction, corrupted downloads, and silent process deaths are all classic disk-full signatures. testhost writes TestResults/ and may write dumps.

Host telemetry (2026-07-18, post-reboot) — resource exhaustion ruled out

Collected from the server:

/            217G  146G  62G  71%      (inodes 14% used)
/dev/shm      16G     0   16G   0%     (=> ~32 GB RAM)
ulimit -u  127749   ulimit -n 1024   ulimit -m unlimited
/sys/fs/cgroup/pids.max: No such file or directory
fail2ban-client: command not found
journalctl -u sshd: No entries      (Ubuntu's unit is `ssh`, not `sshd`)
dmesg: read kernel buffer failed: Operation not permitted   (needs sudo; cleared by reboot anyway)

This falsifies both resource hypotheses at the host level: there is no disk pressure, no inode pressure, no memory pressure and no restrictive PID limit. It also falsifies the fail2ban explanation for the deploy failure — fail2ban is not installed.

Caveat that matters: Gitea act_runner normally executes a runs-on: ubuntu-latest job inside a Docker container, so the figures above describe the host, not the environment the tests actually ran in. A job container has its own cgroup memory/PID limits and, by Docker default, a 64 MB /dev/shm regardless of the host's 16 GB. The relevant limits have therefore not been measured yet — see the asks below.

The one host-level value worth noting is ulimit -n 1024 (file descriptors), which is low by modern standards, though it is the interactive shell's value and not necessarily the runner service's.

Post-reboot runs — the failure moves between stages

Two further runs after the machine was restarted:

Run	Result
8f73548 (post-reboot)	Identical to pre-reboot: host smoke ✓ 1 s, full suite ✗ 3 s
c4c0cd4	Failure moved earlier: Restore backend tests ✗ 0 s — a step that took 3–4 s and succeeded in all previous runs. None of the queued diagnostic steps executed.

Two conclusions:

1. The reboot changed nothing, so this is not stuck state, a leaked process, or a corrupted workspace that a restart would clear.
2. The failing stage is not stable across runs. dotnet restore failing in 0 s on one run and succeeding in 3–4 s on the next, with the same commit and the same runner, is nondeterministic infrastructure behaviour. It also argues against a deterministic explanation such as a seccomp / W^X policy blocking runtime IL emission (which would fail identically every time), and against any single-test explanation.

Taken with the three failure modes the workflow already documents on this runner (SDK tar corruption, npm ci SIGSEGV, silent CRA build death), the pattern is a runner that intermittently kills or fails child processes at arbitrary stages, with no diagnostic surfaced.

Confidence level

• Application code is not the cause — high confidence (~95 %). Ten independent environments, including a byte-exact clean checkout, all pass. Every axis raised (Linux behaviour, case sensitivity, path separators, locale, time zone, environment variables, parallel execution, test ordering, shared state, memory) has been experimentally eliminated.
• Specific mechanism — low confidence (~25 %), and lower than before. Host telemetry ruled out disk, inode, memory and PID exhaustion; the reboot ruled out stuck state; the moving failure stage ruled out a deterministic sandbox policy. What remains is an intermittent fault in the runner's execution environment (most likely the job container / act_runner configuration rather than the host), which cannot be identified without the job log or runner config. I am not asserting a mechanism.

Why application code is no longer suspected

1. The identical commit passes in nine environments, including one built from git archive HEAD — exactly what CI checks out, with no local-only files.
2. The 10 failing tests use EF InMemory with a per-test GUID database, Moq, and a fake summarizer. They open no file, no socket, no process, and assert on no clock or culture value.
3. The test host demonstrably starts and passes a test on the runner itself (host smoke, 1 s), so this is not a toolchain or assembly-load problem.
4. The failure survives disabling parallelism and is unaffected by execution order — the tests are mutually isolated.
5. Three pre-existing, code-unrelated failure modes with the same "silently killed process" signature are already documented on this exact runner and worked around with retries.

---

B — Deploy job fails before doing any work

Evidence

• Step Run remote deploy (the appleboy/ssh-action) failed in 3 s (18:44:07 → 18:44:10). That is before git fetch, before deploy.sh, before any Docker build.
• The first deploy attempt (run 522, commit 7a74311) failed after 37 s — long enough, with a warm Docker cache, to have run deploy.sh and failed its post-deploy backend health check. That is consistent with the MariaDB migration crash fixed in 1430313.
• Every attempt since fails at 3–7 s, i.e. at connection time. The failure mode changed.
• Production is up and healthy: https://jobs.cesnimda.uk/ → 200 HTML, /api/auth/config → 200 JSON ({"requireAuth":true,...}). The host is reachable from the internet, so this is not an outage.
• Production is stale — Phase 4 and Phase 5 have never deployed. Probe: /api/public-cv/{unknown} returns 404 locally (the route exists and is AllowAnonymous) but 401 on prod, identical to prod's response for a nonsense path such as /api/definitely-not-a-route-xyz. PublicCvController is absent from production.

Consequence worth recording

Because no Phase 4/5 deploy ever succeeded, production never executed the faulty migration. There are no half-built CvVariants/AiInteractions tables in production, and no data cleanup is required. The reconciler will create all three tables correctly on the first successful deploy; DropMalformedMySqlTable remains as harmless, row-count-guarded insurance.

Hypothesis

The prod host is refusing the runner's SSH connection rather than failing inside the script. Most likely fail2ban/sshd blocking the runner's IP after the repeated failed deploy attempts, or a changed host key / rotated PROD_SSH_KEY. Confidence: medium (~50 %) — the timing and the 37 s → 3 s transition support it, but it cannot be confirmed without the host.

---

Required infrastructure changes

Blocking — cannot proceed without one of these:

1. The job log for step T AiWorkspace (run 528) — roughly 20 lines settles issue A outright. Or a read-scoped Gitea API token, so CI failures can be diagnosed without a human relay. This is the single highest-value item.
2. The deploy job log (run 523+) — the ssh-action error line settles issue B.

Host checks (issue A):

3. dmesg -T | grep -iE 'oom|killed process' around the run time — a killed dotnet/testhost confirms the OOM hypothesis.
4. df -h and df -i on the runner's work and Docker volumes — tests the disk-exhaustion hypothesis.
5. ulimit -a and cat /sys/fs/cgroup/pids.max for the runner user — tests the PID-limit hypothesis.
6. journalctl -u <gitea-runner-service> --since '2 hours ago'.

Host checks (issue B):

7. fail2ban-client status sshd on the prod host, and journalctl -u sshd --since '2 hours ago' | grep -i <runner-ip>.
8. Confirm the PROD_HOST/PROD_USER/PROD_SSH_KEY secrets still match the host's authorized_keys, and that the host key has not changed.

Recommended remediation regardless of which hypothesis lands:

9. Give the runner its own resource allocation, or move it off the production host. It currently appears to share a box with the prod Docker stack. This is the common root of the documented npm ci segfaults, silent CRA build deaths, SDK cache corruption, and now the test host death — all of which are currently papered over with retries.

---

Repository state

Kept — all correct independent of the outcome, none reverted:

• 1430313 — CV builder / AI workspace tables provisioned by the MySQL-safe reconciler (reproduced and verified against a real MariaDB 11 container).
• cfba7fb — CI actually runs the backend suite.
• 45725ac — restore / build / test split, restore retries once.
• 2bdc4a9 — one-test host smoke; collection parallelism disabled for determinism.
• 7fa3080, 0f62dc4 — bisection scaffolding, since removed.

No test was weakened, skipped, filtered, or disabled at any point. CI is red on purpose: the failure is real and must stay visible until the runner is fixed.