docs(infra): conclude runner + deploy investigation — both outside the repo

Moves the report to docs/infrastructure/runner-investigation.md with the
requested structure (evidence, experiments, hypothesis, confidence, required
infrastructure changes, why application code is no longer suspected).

Decisive new experiment: the suite was run from a clean `git archive HEAD`
tree — byte-identical to CI's checkout, without the gitignored runtime dirs
(jobtracker.db, keys/, CvArtifacts/, backups/) that earlier local runs had
silently included. 10/10 pass in 1s. That removes the last difference between
the local tree and the runner, eliminating application code (~95% confidence).

Also establishes, by route probe, that production is healthy but stale:
/api/public-cv/{unknown} returns 404 locally (route exists, AllowAnonymous) but
401 on prod, same as a nonsense path — PublicCvController is absent, so Phase 4
and Phase 5 have never deployed. Production therefore never ran the faulty
migration: no half-built tables exist there and no data cleanup is needed.

Deploy is a second, separate infrastructure failure: the first attempt reached
deploy.sh (37s, consistent with the MariaDB crash since fixed), every attempt
since dies at 3s at SSH connection time while the host serves traffic normally.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

.gitea/workflows/ci-deploy.yml

@@ -66,7 +66,7 @@ jobs:
       # log is not readable via the Gitea API). Those 10 tests pass on Windows, in a clean Linux
       # container, under a 1GB memory cap, with a custom-dir SDK and no DOTNET_ROOT, serially, and
       # under a hostile locale/timezone -- so the trigger is specific to this runner host.
-      # See docs/ci-runner-investigation.md. Collection parallelism stays off for determinism, the
+      # See docs/infrastructure/runner-investigation.md. Collection parallelism stays off for determinism, the
       # same reason the frontend runs --runInBand. Every test still runs; nothing is filtered.
       - name: Test backend
         run: dotnet test JobTrackerApi.Tests/JobTrackerApi.Tests.csproj --configuration Release --no-build -- xUnit.parallelizeTestCollections=false xUnit.maxParallelThreads=1

docs/ci-runner-investigation.md

@@ -1,100 +0,0 @@
-# CI Runner Investigation — backend suite fails only on `Live-Runner`
-
-> 2026-07-18. Status: **root cause narrowed to the runner host; not reproducible in any other
-> environment.** Final confirmation needs the job log, which the Gitea API will not serve without a
-> token. Recommended infrastructure fix at the bottom.
-
-## Summary
-
-The backend test suite (306 tests) passes everywhere it has been run except the self-hosted Gitea
-runner, where the test step dies after ~3 s. Restore and build succeed; the test host starts and can
-run a test. The failure is localised to the **`AiWorkspace` test classes** (10 tests).
-
-Important context: **this suite had never actually executed in CI.** The workflow built only
-`JobTrackerApi` and then ran `dotnet test --no-build`, so the test project was never compiled and the
-step was a ~1 s no-op (fixed in `cfba7fb`). The failure is therefore *newly surfaced*, not a
-regression — it may have been present for a long time.
-
-## How it was narrowed
-
-Job logs return `401 token is required` from the Gitea API, so step boundaries were the only readable
-telemetry. The suite was split and bisected across CI steps over four runs.
-
-| Run | Observation |
-|---|---|
-| 524 | `Test backend` fails at 8 s (restore+build+test in one step) |
-| 525 | Split: restore 4 s ✓, build 4 s ✓, **test 3 s ✗** → not a compile/restore error |
-| 526 | One-test host smoke **1 s ✓**, full suite (parallelism off) **3 s ✗** → host starts fine; not parallelism |
-| 527 | Alphabetical quarters: **slice 1 (A–C) ✗ at 3 s**, later slices never ran |
-| 528 | Per class: **`T AiWorkspace` ✗ at 3 s**; all other A–C classes never reached |
-
-The `FullyQualifiedName~AiWorkspace` filter matches 10 tests across `AiWorkspaceTests` (Phase 5) and
-`AiWorkspaceNotePersistenceTests` (pre-existing). Locally that filter completes in **1 s**.
-
-## What was ruled out (evidence)
-
-Each was reproduced against the same commit, all 306 tests passing unless noted.
-
-| Hypothesis | Test performed | Result |
-|---|---|---|
-| Linux-specific behaviour | clean `mcr.microsoft.com/dotnet/sdk:9.0` container | 306 pass |
-| File system case sensitivity | same (Linux is case-sensitive) | 306 pass |
-| Path separators | same | 306 pass |
-| CI step ordering | reproduced CI's exact build-then-test order | 306 pass |
-| Memory pressure | `--memory=1g --memory-swap=1g` | 306 pass |
-| Environment variables / SDK layout | bare `ubuntu:22.04`, SDK via `dotnet-install.sh` to `$HOME/.dotnet`, `PATH` only, **`DOTNET_ROOT` unset** (mirrors the runner) | 306 pass |
-| Parallel execution | `xUnit.parallelizeTestCollections=false xUnit.maxParallelThreads=1` | passes locally; **still fails on runner** |
-| Test ordering / shared state | `TestHostFactory.CreateInMemoryDb` uses `Guid.NewGuid()` per test — no shared store | isolated |
-| Locale / culture | `LANG=LC_ALL=tr_TR.UTF-8` (Turkish-I trap) | 10/10 pass in 1 s |
-| Time zone | `TZ=Pacific/Kiritimati` (UTC+14) | 10/10 pass in 1 s |
-| Docker availability | these tests use EF InMemory + Moq; no Docker, network, or filesystem use | n/a |
-| Permissions | not testable remotely | **unresolved** |
-
-## Assessment: environmental, not code
-
-The `AiWorkspace` tests use EF InMemory with a per-test GUID database, `Moq`, and a fake summarizer.
-They touch no clock-sensitive assertion, no file, no socket, and no external process. They pass under
-every adverse condition that could be simulated. That points at the runner host itself.
-
-The workflow already documents three separate failure modes on this exact runner, all with the same
-signature — **a process dying with no usable error output**:
-
-- `actions/setup-dotnet` "intermittently leaves a partial extraction in the shared tool-cache … or
-  corrupts the SDK download" (hence the hand-rolled `dotnet-install.sh` + retry).
-- `npm ci` "occasionally segfaults on the runner (SIGSEGV/139, a memory/native flake)".
-- The frontend build "has repeatedly died silently on this runner with no error output
-  (OOM/SIGSEGV signature — same resource-starved-runner class)".
-
-A .NET test host exiting ~3 s into a 10-test run, on a host with a documented history of silently
-killed processes, is consistent with the same resource starvation — most plausibly the OOM killer or
-a cgroup limit, triggered while the runner shares the box with the production Docker stack. A 1 GB
-container cap was not enough to reproduce it, so the runner's *available* memory at that moment is
-likely lower, or the limit is on process/thread count rather than memory.
-
-## Recommended infrastructure fix
-
-In priority order:
-
-1. **Read the job log** for step `T AiWorkspace` (run 528) — 20 lines settles this immediately. Either
-   grant a read-scoped Gitea token so CI failures can be diagnosed without a human relay, or paste the
-   step output. Everything below is contingent on that.
-2. **Check the host for OOM kills** around the run: `dmesg -T | grep -i -E 'oom|killed process'` and
-   `journalctl -u <gitea-runner> --since '1 hour ago'`. A killed `testhost`/`dotnet` confirms it.
-3. **Give the runner headroom / isolation.** It currently appears to share the host with the prod
-   stack. Either cap the prod containers' memory, give the runner its own cgroup allocation, or move
-   it off the production host. This would also fix the pre-existing npm/CRA/SDK flakes the workflow
-   works around with retries.
-4. **Check `ulimit`/cgroup pids** for the runner user (`ulimit -a`, `cat /sys/fs/cgroup/pids.max`).
-   The .NET test host spawns more threads than `npm ci`, so a low pids limit would hit it first.
-
-## Repository-side changes made during the investigation
-
-Kept, because they are correct independent of the outcome:
-
-- `cfba7fb` — CI actually runs the backend suite (dropped the no-op `--no-build`).
-- `45725ac` — restore / build / test split into separate steps, restore retries once.
-- `2bdc4a9` — one-test host smoke step; collection parallelism disabled for determinism.
-- Bisection scaffolding was removed once it had served its purpose (`7fa3080`, `0f62dc4`).
-
-No test was weakened, skipped, or filtered at any point. CI remains red on purpose — the failure is
-real and should stay visible until the runner is fixed.

docs/infrastructure/runner-investigation.md

@@ -0,0 +1,182 @@
+# Infrastructure Investigation — CI runner and deploy failures
+
+> 2026-07-18. Supersedes `docs/ci-runner-investigation.md`.
+> **Conclusion: both failures are outside the repository.** Application code has been eliminated as a
+> cause by direct experiment. Confirmation and repair require host access — the exact asks are at the
+> end.
+
+There are **two independent infrastructure failures**:
+
+- **A — CI `test` job**: the backend suite fails only on the self-hosted `Live-Runner`.
+- **B — CI `deploy` job**: the SSH step fails ~3 s in, before doing any work.
+
+They are unrelated to each other and to the application code.
+
+---
+
+## A — Backend suite fails only on the runner
+
+### Evidence gathered
+
+- The suite had **never actually run in CI**. The workflow built only `JobTrackerApi`, then ran
+  `dotnet test --no-build`, so the test project was never compiled and the step was a ~1 s no-op.
+  Fixed in `cfba7fb`. **The failure is newly surfaced, not a regression** — it may be long-standing.
+- Job logs are unreadable: `GET /api/v1/.../actions/jobs/{id}/logs` → `401 token is required`.
+  Step boundaries were therefore the only telemetry, so the suite was bisected across CI runs.
+- Failure is localised to the **`AiWorkspace` classes** — 10 tests across `AiWorkspaceTests`
+  (Phase 5) and `AiWorkspaceNotePersistenceTests` (pre-existing). Locally these run in **1 s**.
+
+| Run | Steps observed | Reading |
+|---|---|---|
+| 524 | `Test backend` ✗ 8 s | one combined step, no detail |
+| 525 | restore ✓ 4 s, build ✓ 4 s, **test ✗ 3 s** | not a restore or compile error |
+| 526 | host smoke ✓ **1 s**, full suite (serial) ✗ 3 s | test host starts fine; not parallelism |
+| 527 | quarters: **A–C ✗ 3 s**, rest never ran | offender is alphabetically early |
+| 528 | per class: **`T AiWorkspace` ✗ 3 s**, others never ran | offender named |
+
+### Experiments performed
+
+Every experiment ran the same commit. All pass unless stated.
+
+| # | Experiment | Result |
+|---|---|---|
+| 1 | Windows host, full suite | 306 pass |
+| 2 | Clean `mcr.microsoft.com/dotnet/sdk:9.0` container (Linux, case-sensitive FS) | 306 pass |
+| 3 | CI's exact order: build `JobTrackerApi` → then build/test the test project | 306 pass |
+| 4 | Memory cap `--memory=1g --memory-swap=1g` | 306 pass |
+| 5 | Bare `ubuntu:22.04`, SDK via `dotnet-install.sh` into `$HOME/.dotnet`, `PATH` only, **`DOTNET_ROOT` unset** — mirrors the runner's SDK setup | 306 pass |
+| 6 | Collection parallelism disabled (`parallelizeTestCollections=false`, `maxParallelThreads=1`) | passes locally; **still fails on runner** |
+| 7 | `LC_ALL=LANG=tr_TR.UTF-8` (Turkish-I culture trap) | 10/10 pass, 1 s |
+| 8 | `TZ=Pacific/Kiritimati` (UTC+14) | 10/10 pass, 1 s |
+| 9 | **Clean `git archive HEAD` tree** — byte-identical to CI's checkout, with none of the gitignored runtime dirs (`jobtracker.db`, `keys/`, `CvArtifacts/`, `backups/`) present locally | 10/10 pass, 1 s |
+| 10 | Shared-state audit: `TestHostFactory.CreateInMemoryDb` uses `Guid.NewGuid()` per test | no shared store |
+
+Experiment 9 is the decisive one: it removes the last difference between the local tree and the
+runner's checkout. The exact source CI compiles produces a passing suite.
+
+### Root cause hypothesis
+
+The runner host kills the test process. The workflow already documents **three separate failure modes
+on this same runner, all with the signature of a process dying with no usable error output**:
+
+- `actions/setup-dotnet` "intermittently leaves a partial extraction in the shared tool-cache
+  (`tar: Cannot open: File exists`) or corrupts the SDK download" — hence the hand-rolled installer.
+- `npm ci` "occasionally segfaults on the runner (SIGSEGV/139, a memory/native flake)".
+- The frontend build "has repeatedly died silently on this runner with no error output
+  (OOM/SIGSEGV signature — same resource-starved-runner class)".
+
+A .NET test host exiting ~3 s into a 10-test run belongs to that same family. Two candidate mechanisms,
+in order of likelihood:
+
+1. **Resource exhaustion — memory or PID/thread limits.** The runner appears to share the host with
+   the production Docker stack. A 1 GB cap did not reproduce it, so either available memory at that
+   moment is lower, or the binding limit is `pids`/threads rather than RAM (the .NET test host spawns
+   more threads than `npm ci`, so it would hit a low `pids.max` first).
+2. **Disk exhaustion.** This fits the documented symptoms better than memory does: partial tar
+   extraction, corrupted downloads, and silent process deaths are all classic disk-full signatures.
+   `testhost` writes `TestResults/` and may write dumps.
+
+### Confidence level
+
+- **Application code is not the cause — high confidence (~95 %).** Ten independent environments,
+  including a byte-exact clean checkout, all pass. Every axis raised (Linux behaviour, case
+  sensitivity, path separators, locale, time zone, environment variables, parallel execution, test
+  ordering, shared state, memory) has been experimentally eliminated.
+- **Specific mechanism (OOM vs pids vs disk) — low/medium confidence (~40 %).** Not reproducible
+  remotely, and not resolvable without the job log or host access. I am deliberately not asserting
+  which one it is.
+
+### Why application code is no longer suspected
+
+1. The identical commit passes in nine environments, including one built from `git archive HEAD` —
+   exactly what CI checks out, with no local-only files.
+2. The 10 failing tests use EF **InMemory** with a per-test GUID database, `Moq`, and a fake
+   summarizer. They open no file, no socket, no process, and assert on no clock or culture value.
+3. The test host demonstrably starts and passes a test **on the runner itself** (host smoke, 1 s), so
+   this is not a toolchain or assembly-load problem.
+4. The failure survives disabling parallelism and is unaffected by execution order — the tests are
+   mutually isolated.
+5. Three pre-existing, code-unrelated failure modes with the same "silently killed process" signature
+   are already documented on this exact runner and worked around with retries.
+
+---
+
+## B — Deploy job fails before doing any work
+
+### Evidence
+
+- Step `Run remote deploy` (the `appleboy/ssh-action`) failed in **3 s** (`18:44:07 → 18:44:10`).
+  That is before `git fetch`, before `deploy.sh`, before any Docker build.
+- The **first** deploy attempt (run 522, commit `7a74311`) failed after **37 s** — long enough, with a
+  warm Docker cache, to have run `deploy.sh` and failed its post-deploy backend health check. That is
+  consistent with the MariaDB migration crash fixed in `1430313`.
+- Every attempt since fails at **3–7 s**, i.e. at connection time. The failure mode changed.
+- **Production is up and healthy**: `https://jobs.cesnimda.uk/` → 200 HTML,
+  `/api/auth/config` → 200 JSON (`{"requireAuth":true,...}`). The host is reachable from the internet,
+  so this is not an outage.
+- **Production is stale — Phase 4 and Phase 5 have never deployed.** Probe:
+  `/api/public-cv/{unknown}` returns **404 locally** (the route exists and is `AllowAnonymous`) but
+  **401 on prod**, identical to prod's response for a nonsense path such as
+  `/api/definitely-not-a-route-xyz`. `PublicCvController` is absent from production.
+
+### Consequence worth recording
+
+Because no Phase 4/5 deploy ever succeeded, **production never executed the faulty migration**. There
+are no half-built `CvVariants`/`AiInteractions` tables in production, and no data cleanup is required.
+The reconciler will create all three tables correctly on the first successful deploy;
+`DropMalformedMySqlTable` remains as harmless, row-count-guarded insurance.
+
+### Hypothesis
+
+The prod host is refusing the runner's SSH connection rather than failing inside the script. Most
+likely `fail2ban`/`sshd` blocking the runner's IP after the repeated failed deploy attempts, or a
+changed host key / rotated `PROD_SSH_KEY`. Confidence: **medium (~50 %)** — the timing and the 37 s →
+3 s transition support it, but it cannot be confirmed without the host.
+
+---
+
+## Required infrastructure changes
+
+**Blocking — cannot proceed without one of these:**
+
+1. **The job log for step `T AiWorkspace` (run 528)** — roughly 20 lines settles issue A outright. Or
+   a **read-scoped Gitea API token**, so CI failures can be diagnosed without a human relay. This is
+   the single highest-value item.
+2. **The deploy job log (run 523+)** — the `ssh-action` error line settles issue B.
+
+**Host checks (issue A):**
+
+3. `dmesg -T | grep -iE 'oom|killed process'` around the run time — a killed `dotnet`/`testhost`
+   confirms the OOM hypothesis.
+4. `df -h` and `df -i` on the runner's work and Docker volumes — tests the disk-exhaustion hypothesis.
+5. `ulimit -a` and `cat /sys/fs/cgroup/pids.max` for the runner user — tests the PID-limit hypothesis.
+6. `journalctl -u <gitea-runner-service> --since '2 hours ago'`.
+
+**Host checks (issue B):**
+
+7. `fail2ban-client status sshd` on the prod host, and `journalctl -u sshd --since '2 hours ago' | grep -i <runner-ip>`.
+8. Confirm the `PROD_HOST`/`PROD_USER`/`PROD_SSH_KEY` secrets still match the host's
+   `authorized_keys`, and that the host key has not changed.
+
+**Recommended remediation regardless of which hypothesis lands:**
+
+9. **Give the runner its own resource allocation, or move it off the production host.** It currently
+   appears to share a box with the prod Docker stack. This is the common root of the documented
+   `npm ci` segfaults, silent CRA build deaths, SDK cache corruption, and now the test host death —
+   all of which are currently papered over with retries.
+
+---
+
+## Repository state
+
+Kept — all correct independent of the outcome, none reverted:
+
+- `1430313` — CV builder / AI workspace tables provisioned by the MySQL-safe reconciler
+  (reproduced and verified against a real MariaDB 11 container).
+- `cfba7fb` — CI actually runs the backend suite.
+- `45725ac` — restore / build / test split, restore retries once.
+- `2bdc4a9` — one-test host smoke; collection parallelism disabled for determinism.
+- `7fa3080`, `0f62dc4` — bisection scaffolding, since removed.
+
+**No test was weakened, skipped, filtered, or disabled at any point.** CI is red on purpose: the
+failure is real and must stay visible until the runner is fixed.