docs(infra): conclude runner + deploy investigation — both outside the repo

Moves the report to docs/infrastructure/runner-investigation.md with the requested structure (evidence, experiments, hypothesis, confidence, required infrastructure changes, why application code is no longer suspected). Decisive new experiment: the suite was run from a clean `git archive HEAD` tree — byte-identical to CI's checkout, without the gitignored runtime dirs (jobtracker.db, keys/, CvArtifacts/, backups/) that earlier local runs had silently included. 10/10 pass in 1s. That removes the last difference between the local tree and the runner, eliminating application code (~95% confidence). Also establishes, by route probe, that production is healthy but stale: /api/public-cv/{unknown} returns 404 locally (route exists, AllowAnonymous) but 401 on prod, same as a nonsense path — PublicCvController is absent, so Phase 4 and Phase 5 have never deployed. Production therefore never ran the faulty migration: no half-built tables exist there and no data cleanup is needed. Deploy is a second, separate infrastructure failure: the first attempt reached deploy.sh (37s, consistent with the MariaDB crash since fixed), every attempt since dies at 3s at SSH connection time while the host serves traffic normally. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-18 23:00:27 +02:00
parent b7dc07b045
commit 55962bd29b
3 changed files with 183 additions and 101 deletions
@@ -66,7 +66,7 @@ jobs:
      # log is not readable via the Gitea API). Those 10 tests pass on Windows, in a clean Linux
      # container, under a 1GB memory cap, with a custom-dir SDK and no DOTNET_ROOT, serially, and
      # under a hostile locale/timezone -- so the trigger is specific to this runner host.
-      # See docs/ci-runner-investigation.md. Collection parallelism stays off for determinism, the
+      # See docs/infrastructure/runner-investigation.md. Collection parallelism stays off for determinism, the
      # same reason the frontend runs --runInBand. Every test still runs; nothing is filtered.
      - name: Test backend
        run: dotnet test JobTrackerApi.Tests/JobTrackerApi.Tests.csproj --configuration Release --no-build -- xUnit.parallelizeTestCollections=false xUnit.maxParallelThreads=1
@@ -1,100 +0,0 @@
 # CI Runner Investigation — backend suite fails only on `Live-Runner`
 > 2026-07-18. Status: **root cause narrowed to the runner host; not reproducible in any other
 > environment.** Final confirmation needs the job log, which the Gitea API will not serve without a
 > token. Recommended infrastructure fix at the bottom.
 ## Summary
 The backend test suite (306 tests) passes everywhere it has been run except the self-hosted Gitea
 runner, where the test step dies after ~3 s. Restore and build succeed; the test host starts and can
 run a test. The failure is localised to the **`AiWorkspace` test classes** (10 tests).
 Important context: **this suite had never actually executed in CI.** The workflow built only
 `JobTrackerApi` and then ran `dotnet test --no-build`, so the test project was never compiled and the
 step was a ~1 s no-op (fixed in `cfba7fb`). The failure is therefore *newly surfaced*, not a
 regression — it may have been present for a long time.
 ## How it was narrowed
 Job logs return `401 token is required` from the Gitea API, so step boundaries were the only readable
 telemetry. The suite was split and bisected across CI steps over four runs.
 | Run | Observation |
 |---|---|
 | 524 | `Test backend` fails at 8 s (restore+build+test in one step) |
 | 525 | Split: restore 4 s ✓, build 4 s ✓, **test 3 s ✗** → not a compile/restore error |
 | 526 | One-test host smoke **1 s ✓**, full suite (parallelism off) **3 s ✗** → host starts fine; not parallelism |
 | 527 | Alphabetical quarters: **slice 1 (A–C) ✗ at 3 s**, later slices never ran |
 | 528 | Per class: **`T AiWorkspace` ✗ at 3 s**; all other A–C classes never reached |
 The `FullyQualifiedName~AiWorkspace` filter matches 10 tests across `AiWorkspaceTests` (Phase 5) and
 `AiWorkspaceNotePersistenceTests` (pre-existing). Locally that filter completes in **1 s**.
 ## What was ruled out (evidence)
 Each was reproduced against the same commit, all 306 tests passing unless noted.
 | Hypothesis | Test performed | Result |
 |---|---|---|
 | Linux-specific behaviour | clean `mcr.microsoft.com/dotnet/sdk:9.0` container | 306 pass |
 | File system case sensitivity | same (Linux is case-sensitive) | 306 pass |
 | Path separators | same | 306 pass |
 | CI step ordering | reproduced CI's exact build-then-test order | 306 pass |
 | Memory pressure | `--memory=1g --memory-swap=1g` | 306 pass |
 | Environment variables / SDK layout | bare `ubuntu:22.04`, SDK via `dotnet-install.sh` to `$HOME/.dotnet`, `PATH` only, **`DOTNET_ROOT` unset** (mirrors the runner) | 306 pass |
 | Parallel execution | `xUnit.parallelizeTestCollections=false xUnit.maxParallelThreads=1` | passes locally; **still fails on runner** |
 | Test ordering / shared state | `TestHostFactory.CreateInMemoryDb` uses `Guid.NewGuid()` per test — no shared store | isolated |
 | Locale / culture | `LANG=LC_ALL=tr_TR.UTF-8` (Turkish-I trap) | 10/10 pass in 1 s |
 | Time zone | `TZ=Pacific/Kiritimati` (UTC+14) | 10/10 pass in 1 s |
 | Docker availability | these tests use EF InMemory + Moq; no Docker, network, or filesystem use | n/a |
 | Permissions | not testable remotely | **unresolved** |
 ## Assessment: environmental, not code
 The `AiWorkspace` tests use EF InMemory with a per-test GUID database, `Moq`, and a fake summarizer.
 They touch no clock-sensitive assertion, no file, no socket, and no external process. They pass under
 every adverse condition that could be simulated. That points at the runner host itself.
 The workflow already documents three separate failure modes on this exact runner, all with the same
 signature — **a process dying with no usable error output**:
 - `actions/setup-dotnet` "intermittently leaves a partial extraction in the shared tool-cache … or
  corrupts the SDK download" (hence the hand-rolled `dotnet-install.sh` + retry).
 - `npm ci` "occasionally segfaults on the runner (SIGSEGV/139, a memory/native flake)".
 - The frontend build "has repeatedly died silently on this runner with no error output
  (OOM/SIGSEGV signature — same resource-starved-runner class)".
 A .NET test host exiting ~3 s into a 10-test run, on a host with a documented history of silently
 killed processes, is consistent with the same resource starvation — most plausibly the OOM killer or
 a cgroup limit, triggered while the runner shares the box with the production Docker stack. A 1 GB
 container cap was not enough to reproduce it, so the runner's *available* memory at that moment is
 likely lower, or the limit is on process/thread count rather than memory.
 ## Recommended infrastructure fix
 In priority order:
 1. **Read the job log** for step `T AiWorkspace` (run 528) — 20 lines settles this immediately. Either
   grant a read-scoped Gitea token so CI failures can be diagnosed without a human relay, or paste the
   step output. Everything below is contingent on that.
 2. **Check the host for OOM kills** around the run: `dmesg -T | grep -i -E 'oom|killed process'` and
   `journalctl -u <gitea-runner> --since '1 hour ago'`. A killed `testhost`/`dotnet` confirms it.
 3. **Give the runner headroom / isolation.** It currently appears to share the host with the prod
   stack. Either cap the prod containers' memory, give the runner its own cgroup allocation, or move
   it off the production host. This would also fix the pre-existing npm/CRA/SDK flakes the workflow
   works around with retries.
 4. **Check `ulimit`/cgroup pids** for the runner user (`ulimit -a`, `cat /sys/fs/cgroup/pids.max`).
   The .NET test host spawns more threads than `npm ci`, so a low pids limit would hit it first.
 ## Repository-side changes made during the investigation
 Kept, because they are correct independent of the outcome:
 - `cfba7fb` — CI actually runs the backend suite (dropped the no-op `--no-build`).
 - `45725ac` — restore / build / test split into separate steps, restore retries once.
 - `2bdc4a9` — one-test host smoke step; collection parallelism disabled for determinism.
 - Bisection scaffolding was removed once it had served its purpose (`7fa3080`, `0f62dc4`).
 No test was weakened, skipped, or filtered at any point. CI remains red on purpose — the failure is
 real and should stay visible until the runner is fixed.
@@ -0,0 +1,182 @@
 # Infrastructure Investigation — CI runner and deploy failures
 > 2026-07-18. Supersedes `docs/ci-runner-investigation.md`.
 > **Conclusion: both failures are outside the repository.** Application code has been eliminated as a
 > cause by direct experiment. Confirmation and repair require host access — the exact asks are at the
 > end.
 There are **two independent infrastructure failures**:
 - **A — CI `test` job**: the backend suite fails only on the self-hosted `Live-Runner`.
 - **B — CI `deploy` job**: the SSH step fails ~3 s in, before doing any work.
 They are unrelated to each other and to the application code.
 ---
 ## A — Backend suite fails only on the runner
 ### Evidence gathered
 - The suite had **never actually run in CI**. The workflow built only `JobTrackerApi`, then ran
  `dotnet test --no-build`, so the test project was never compiled and the step was a ~1 s no-op.
  Fixed in `cfba7fb`. **The failure is newly surfaced, not a regression** — it may be long-standing.
 - Job logs are unreadable: `GET /api/v1/.../actions/jobs/{id}/logs` → `401 token is required`.
  Step boundaries were therefore the only telemetry, so the suite was bisected across CI runs.
 - Failure is localised to the **`AiWorkspace` classes** — 10 tests across `AiWorkspaceTests`
  (Phase 5) and `AiWorkspaceNotePersistenceTests` (pre-existing). Locally these run in **1 s**.
 | Run | Steps observed | Reading |
 |---|---|---|
 | 524 | `Test backend` ✗ 8 s | one combined step, no detail |
 | 525 | restore ✓ 4 s, build ✓ 4 s, **test ✗ 3 s** | not a restore or compile error |
 | 526 | host smoke ✓ **1 s**, full suite (serial) ✗ 3 s | test host starts fine; not parallelism |
 | 527 | quarters: **A–C ✗ 3 s**, rest never ran | offender is alphabetically early |
 | 528 | per class: **`T AiWorkspace` ✗ 3 s**, others never ran | offender named |
 ### Experiments performed
 Every experiment ran the same commit. All pass unless stated.
 | # | Experiment | Result |
 |---|---|---|
 | 1 | Windows host, full suite | 306 pass |
 | 2 | Clean `mcr.microsoft.com/dotnet/sdk:9.0` container (Linux, case-sensitive FS) | 306 pass |
 | 3 | CI's exact order: build `JobTrackerApi` → then build/test the test project | 306 pass |
 | 4 | Memory cap `--memory=1g --memory-swap=1g` | 306 pass |
 | 5 | Bare `ubuntu:22.04`, SDK via `dotnet-install.sh` into `$HOME/.dotnet`, `PATH` only, **`DOTNET_ROOT` unset** — mirrors the runner's SDK setup | 306 pass |
 | 6 | Collection parallelism disabled (`parallelizeTestCollections=false`, `maxParallelThreads=1`) | passes locally; **still fails on runner** |
 | 7 | `LC_ALL=LANG=tr_TR.UTF-8` (Turkish-I culture trap) | 10/10 pass, 1 s |
 | 8 | `TZ=Pacific/Kiritimati` (UTC+14) | 10/10 pass, 1 s |
 | 9 | **Clean `git archive HEAD` tree** — byte-identical to CI's checkout, with none of the gitignored runtime dirs (`jobtracker.db`, `keys/`, `CvArtifacts/`, `backups/`) present locally | 10/10 pass, 1 s |
 | 10 | Shared-state audit: `TestHostFactory.CreateInMemoryDb` uses `Guid.NewGuid()` per test | no shared store |
 Experiment 9 is the decisive one: it removes the last difference between the local tree and the
 runner's checkout. The exact source CI compiles produces a passing suite.
 ### Root cause hypothesis
 The runner host kills the test process. The workflow already documents **three separate failure modes
 on this same runner, all with the signature of a process dying with no usable error output**:
 - `actions/setup-dotnet` "intermittently leaves a partial extraction in the shared tool-cache
  (`tar: Cannot open: File exists`) or corrupts the SDK download" — hence the hand-rolled installer.
 - `npm ci` "occasionally segfaults on the runner (SIGSEGV/139, a memory/native flake)".
 - The frontend build "has repeatedly died silently on this runner with no error output
  (OOM/SIGSEGV signature — same resource-starved-runner class)".
 A .NET test host exiting ~3 s into a 10-test run belongs to that same family. Two candidate mechanisms,
 in order of likelihood:
 1. **Resource exhaustion — memory or PID/thread limits.** The runner appears to share the host with
   the production Docker stack. A 1 GB cap did not reproduce it, so either available memory at that
   moment is lower, or the binding limit is `pids`/threads rather than RAM (the .NET test host spawns
   more threads than `npm ci`, so it would hit a low `pids.max` first).
 2. **Disk exhaustion.** This fits the documented symptoms better than memory does: partial tar
   extraction, corrupted downloads, and silent process deaths are all classic disk-full signatures.
   `testhost` writes `TestResults/` and may write dumps.
 ### Confidence level
 - **Application code is not the cause — high confidence (~95 %).** Ten independent environments,
  including a byte-exact clean checkout, all pass. Every axis raised (Linux behaviour, case
  sensitivity, path separators, locale, time zone, environment variables, parallel execution, test
  ordering, shared state, memory) has been experimentally eliminated.
 - **Specific mechanism (OOM vs pids vs disk) — low/medium confidence (~40 %).** Not reproducible
  remotely, and not resolvable without the job log or host access. I am deliberately not asserting
  which one it is.
 ### Why application code is no longer suspected
 1. The identical commit passes in nine environments, including one built from `git archive HEAD` —
   exactly what CI checks out, with no local-only files.
 2. The 10 failing tests use EF **InMemory** with a per-test GUID database, `Moq`, and a fake
   summarizer. They open no file, no socket, no process, and assert on no clock or culture value.
 3. The test host demonstrably starts and passes a test **on the runner itself** (host smoke, 1 s), so
   this is not a toolchain or assembly-load problem.
 4. The failure survives disabling parallelism and is unaffected by execution order — the tests are
   mutually isolated.
 5. Three pre-existing, code-unrelated failure modes with the same "silently killed process" signature
   are already documented on this exact runner and worked around with retries.
 ---
 ## B — Deploy job fails before doing any work
 ### Evidence
 - Step `Run remote deploy` (the `appleboy/ssh-action`) failed in **3 s** (`18:44:07 → 18:44:10`).
  That is before `git fetch`, before `deploy.sh`, before any Docker build.
 - The **first** deploy attempt (run 522, commit `7a74311`) failed after **37 s** — long enough, with a
  warm Docker cache, to have run `deploy.sh` and failed its post-deploy backend health check. That is
  consistent with the MariaDB migration crash fixed in `1430313`.
 - Every attempt since fails at **3–7 s**, i.e. at connection time. The failure mode changed.
 - **Production is up and healthy**: `https://jobs.cesnimda.uk/` → 200 HTML,
  `/api/auth/config` → 200 JSON (`{"requireAuth":true,...}`). The host is reachable from the internet,
  so this is not an outage.
 - **Production is stale — Phase 4 and Phase 5 have never deployed.** Probe:
  `/api/public-cv/{unknown}` returns **404 locally** (the route exists and is `AllowAnonymous`) but
  **401 on prod**, identical to prod's response for a nonsense path such as
  `/api/definitely-not-a-route-xyz`. `PublicCvController` is absent from production.
 ### Consequence worth recording
 Because no Phase 4/5 deploy ever succeeded, **production never executed the faulty migration**. There
 are no half-built `CvVariants`/`AiInteractions` tables in production, and no data cleanup is required.
 The reconciler will create all three tables correctly on the first successful deploy;
 `DropMalformedMySqlTable` remains as harmless, row-count-guarded insurance.
 ### Hypothesis
 The prod host is refusing the runner's SSH connection rather than failing inside the script. Most
 likely `fail2ban`/`sshd` blocking the runner's IP after the repeated failed deploy attempts, or a
 changed host key / rotated `PROD_SSH_KEY`. Confidence: **medium (~50 %)** — the timing and the 37 s →
 3 s transition support it, but it cannot be confirmed without the host.
 ---
 ## Required infrastructure changes
 **Blocking — cannot proceed without one of these:**
 1. **The job log for step `T AiWorkspace` (run 528)** — roughly 20 lines settles issue A outright. Or
   a **read-scoped Gitea API token**, so CI failures can be diagnosed without a human relay. This is
   the single highest-value item.
 2. **The deploy job log (run 523+)** — the `ssh-action` error line settles issue B.
 **Host checks (issue A):**
 3. `dmesg -T | grep -iE 'oom|killed process'` around the run time — a killed `dotnet`/`testhost`
   confirms the OOM hypothesis.
 4. `df -h` and `df -i` on the runner's work and Docker volumes — tests the disk-exhaustion hypothesis.
 5. `ulimit -a` and `cat /sys/fs/cgroup/pids.max` for the runner user — tests the PID-limit hypothesis.
 6. `journalctl -u <gitea-runner-service> --since '2 hours ago'`.
 **Host checks (issue B):**
 7. `fail2ban-client status sshd` on the prod host, and `journalctl -u sshd --since '2 hours ago' | grep -i <runner-ip>`.
 8. Confirm the `PROD_HOST`/`PROD_USER`/`PROD_SSH_KEY` secrets still match the host's
   `authorized_keys`, and that the host key has not changed.
 **Recommended remediation regardless of which hypothesis lands:**
 9. **Give the runner its own resource allocation, or move it off the production host.** It currently
   appears to share a box with the prod Docker stack. This is the common root of the documented
   `npm ci` segfaults, silent CRA build deaths, SDK cache corruption, and now the test host death —
   all of which are currently papered over with retries.
 ---
 ## Repository state
 Kept — all correct independent of the outcome, none reverted:
 - `1430313` — CV builder / AI workspace tables provisioned by the MySQL-safe reconciler
  (reproduced and verified against a real MariaDB 11 container).
 - `cfba7fb` — CI actually runs the backend suite.
 - `45725ac` — restore / build / test split, restore retries once.
 - `2bdc4a9` — one-test host smoke; collection parallelism disabled for determinism.
 - `7fa3080`, `0f62dc4` — bisection scaffolding, since removed.
 **No test was weakened, skipped, filtered, or disabled at any point.** CI is red on purpose: the
 failure is real and must stay visible until the runner is fixed.