# Infrastructure Investigation — CI runner and deploy failures > 2026-07-18. Supersedes `docs/ci-runner-investigation.md`. > **Conclusion: both failures are outside the repository.** Application code has been eliminated as a > cause by direct experiment. Confirmation and repair require host access — the exact asks are at the > end. There are **two independent infrastructure failures**: - **A — CI `test` job**: the backend suite fails only on the self-hosted `Live-Runner`. - **B — CI `deploy` job**: the SSH step fails ~3 s in, before doing any work. They are unrelated to each other and to the application code. --- ## A — Backend suite fails only on the runner ### Evidence gathered - The suite had **never actually run in CI**. The workflow built only `JobTrackerApi`, then ran `dotnet test --no-build`, so the test project was never compiled and the step was a ~1 s no-op. Fixed in `cfba7fb`. **The failure is newly surfaced, not a regression** — it may be long-standing. - Job logs are unreadable: `GET /api/v1/.../actions/jobs/{id}/logs` → `401 token is required`. Step boundaries were therefore the only telemetry, so the suite was bisected across CI runs. - Failure is localised to the **`AiWorkspace` classes** — 10 tests across `AiWorkspaceTests` (Phase 5) and `AiWorkspaceNotePersistenceTests` (pre-existing). Locally these run in **1 s**. | Run | Steps observed | Reading | |---|---|---| | 524 | `Test backend` ✗ 8 s | one combined step, no detail | | 525 | restore ✓ 4 s, build ✓ 4 s, **test ✗ 3 s** | not a restore or compile error | | 526 | host smoke ✓ **1 s**, full suite (serial) ✗ 3 s | test host starts fine; not parallelism | | 527 | quarters: **A–C ✗ 3 s**, rest never ran | offender is alphabetically early | | 528 | per class: **`T AiWorkspace` ✗ 3 s**, others never ran | offender named | ### Experiments performed Every experiment ran the same commit. All pass unless stated. | # | Experiment | Result | |---|---|---| | 1 | Windows host, full suite | 306 pass | | 2 | Clean `mcr.microsoft.com/dotnet/sdk:9.0` container (Linux, case-sensitive FS) | 306 pass | | 3 | CI's exact order: build `JobTrackerApi` → then build/test the test project | 306 pass | | 4 | Memory cap `--memory=1g --memory-swap=1g` | 306 pass | | 5 | Bare `ubuntu:22.04`, SDK via `dotnet-install.sh` into `$HOME/.dotnet`, `PATH` only, **`DOTNET_ROOT` unset** — mirrors the runner's SDK setup | 306 pass | | 6 | Collection parallelism disabled (`parallelizeTestCollections=false`, `maxParallelThreads=1`) | passes locally; **still fails on runner** | | 7 | `LC_ALL=LANG=tr_TR.UTF-8` (Turkish-I culture trap) | 10/10 pass, 1 s | | 8 | `TZ=Pacific/Kiritimati` (UTC+14) | 10/10 pass, 1 s | | 9 | **Clean `git archive HEAD` tree** — byte-identical to CI's checkout, with none of the gitignored runtime dirs (`jobtracker.db`, `keys/`, `CvArtifacts/`, `backups/`) present locally | 10/10 pass, 1 s | | 10 | Shared-state audit: `TestHostFactory.CreateInMemoryDb` uses `Guid.NewGuid()` per test | no shared store | Experiment 9 is the decisive one: it removes the last difference between the local tree and the runner's checkout. The exact source CI compiles produces a passing suite. ### Root cause hypothesis The runner host kills the test process. The workflow already documents **three separate failure modes on this same runner, all with the signature of a process dying with no usable error output**: - `actions/setup-dotnet` "intermittently leaves a partial extraction in the shared tool-cache (`tar: Cannot open: File exists`) or corrupts the SDK download" — hence the hand-rolled installer. - `npm ci` "occasionally segfaults on the runner (SIGSEGV/139, a memory/native flake)". - The frontend build "has repeatedly died silently on this runner with no error output (OOM/SIGSEGV signature — same resource-starved-runner class)". A .NET test host exiting ~3 s into a 10-test run belongs to that same family. Two candidate mechanisms, in order of likelihood: 1. **Resource exhaustion — memory or PID/thread limits.** The runner appears to share the host with the production Docker stack. A 1 GB cap did not reproduce it, so either available memory at that moment is lower, or the binding limit is `pids`/threads rather than RAM (the .NET test host spawns more threads than `npm ci`, so it would hit a low `pids.max` first). 2. **Disk exhaustion.** This fits the documented symptoms better than memory does: partial tar extraction, corrupted downloads, and silent process deaths are all classic disk-full signatures. `testhost` writes `TestResults/` and may write dumps. ### Confidence level - **Application code is not the cause — high confidence (~95 %).** Ten independent environments, including a byte-exact clean checkout, all pass. Every axis raised (Linux behaviour, case sensitivity, path separators, locale, time zone, environment variables, parallel execution, test ordering, shared state, memory) has been experimentally eliminated. - **Specific mechanism (OOM vs pids vs disk) — low/medium confidence (~40 %).** Not reproducible remotely, and not resolvable without the job log or host access. I am deliberately not asserting which one it is. ### Why application code is no longer suspected 1. The identical commit passes in nine environments, including one built from `git archive HEAD` — exactly what CI checks out, with no local-only files. 2. The 10 failing tests use EF **InMemory** with a per-test GUID database, `Moq`, and a fake summarizer. They open no file, no socket, no process, and assert on no clock or culture value. 3. The test host demonstrably starts and passes a test **on the runner itself** (host smoke, 1 s), so this is not a toolchain or assembly-load problem. 4. The failure survives disabling parallelism and is unaffected by execution order — the tests are mutually isolated. 5. Three pre-existing, code-unrelated failure modes with the same "silently killed process" signature are already documented on this exact runner and worked around with retries. --- ## B — Deploy job fails before doing any work ### Evidence - Step `Run remote deploy` (the `appleboy/ssh-action`) failed in **3 s** (`18:44:07 → 18:44:10`). That is before `git fetch`, before `deploy.sh`, before any Docker build. - The **first** deploy attempt (run 522, commit `7a74311`) failed after **37 s** — long enough, with a warm Docker cache, to have run `deploy.sh` and failed its post-deploy backend health check. That is consistent with the MariaDB migration crash fixed in `1430313`. - Every attempt since fails at **3–7 s**, i.e. at connection time. The failure mode changed. - **Production is up and healthy**: `https://jobs.cesnimda.uk/` → 200 HTML, `/api/auth/config` → 200 JSON (`{"requireAuth":true,...}`). The host is reachable from the internet, so this is not an outage. - **Production is stale — Phase 4 and Phase 5 have never deployed.** Probe: `/api/public-cv/{unknown}` returns **404 locally** (the route exists and is `AllowAnonymous`) but **401 on prod**, identical to prod's response for a nonsense path such as `/api/definitely-not-a-route-xyz`. `PublicCvController` is absent from production. ### Consequence worth recording Because no Phase 4/5 deploy ever succeeded, **production never executed the faulty migration**. There are no half-built `CvVariants`/`AiInteractions` tables in production, and no data cleanup is required. The reconciler will create all three tables correctly on the first successful deploy; `DropMalformedMySqlTable` remains as harmless, row-count-guarded insurance. ### Hypothesis The prod host is refusing the runner's SSH connection rather than failing inside the script. Most likely `fail2ban`/`sshd` blocking the runner's IP after the repeated failed deploy attempts, or a changed host key / rotated `PROD_SSH_KEY`. Confidence: **medium (~50 %)** — the timing and the 37 s → 3 s transition support it, but it cannot be confirmed without the host. --- ## Required infrastructure changes **Blocking — cannot proceed without one of these:** 1. **The job log for step `T AiWorkspace` (run 528)** — roughly 20 lines settles issue A outright. Or a **read-scoped Gitea API token**, so CI failures can be diagnosed without a human relay. This is the single highest-value item. 2. **The deploy job log (run 523+)** — the `ssh-action` error line settles issue B. **Host checks (issue A):** 3. `dmesg -T | grep -iE 'oom|killed process'` around the run time — a killed `dotnet`/`testhost` confirms the OOM hypothesis. 4. `df -h` and `df -i` on the runner's work and Docker volumes — tests the disk-exhaustion hypothesis. 5. `ulimit -a` and `cat /sys/fs/cgroup/pids.max` for the runner user — tests the PID-limit hypothesis. 6. `journalctl -u --since '2 hours ago'`. **Host checks (issue B):** 7. `fail2ban-client status sshd` on the prod host, and `journalctl -u sshd --since '2 hours ago' | grep -i `. 8. Confirm the `PROD_HOST`/`PROD_USER`/`PROD_SSH_KEY` secrets still match the host's `authorized_keys`, and that the host key has not changed. **Recommended remediation regardless of which hypothesis lands:** 9. **Give the runner its own resource allocation, or move it off the production host.** It currently appears to share a box with the prod Docker stack. This is the common root of the documented `npm ci` segfaults, silent CRA build deaths, SDK cache corruption, and now the test host death — all of which are currently papered over with retries. --- ## Repository state Kept — all correct independent of the outcome, none reverted: - `1430313` — CV builder / AI workspace tables provisioned by the MySQL-safe reconciler (reproduced and verified against a real MariaDB 11 container). - `cfba7fb` — CI actually runs the backend suite. - `45725ac` — restore / build / test split, restore retries once. - `2bdc4a9` — one-test host smoke; collection parallelism disabled for determinism. - `7fa3080`, `0f62dc4` — bisection scaffolding, since removed. **No test was weakened, skipped, filtered, or disabled at any point.** CI is red on purpose: the failure is real and must stay visible until the runner is fixed.