docs(infra): conclude runner + deploy investigation — both outside the repo

Moves the report to docs/infrastructure/runner-investigation.md with the
requested structure (evidence, experiments, hypothesis, confidence, required
infrastructure changes, why application code is no longer suspected).

Decisive new experiment: the suite was run from a clean `git archive HEAD`
tree — byte-identical to CI's checkout, without the gitignored runtime dirs
(jobtracker.db, keys/, CvArtifacts/, backups/) that earlier local runs had
silently included. 10/10 pass in 1s. That removes the last difference between
the local tree and the runner, eliminating application code (~95% confidence).

Also establishes, by route probe, that production is healthy but stale:
/api/public-cv/{unknown} returns 404 locally (route exists, AllowAnonymous) but
401 on prod, same as a nonsense path — PublicCvController is absent, so Phase 4
and Phase 5 have never deployed. Production therefore never ran the faulty
migration: no half-built tables exist there and no data cleanup is needed.

Deploy is a second, separate infrastructure failure: the first attempt reached
deploy.sh (37s, consistent with the MariaDB crash since fixed), every attempt
since dies at 3s at SSH connection time while the host serves traffic normally.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

docs/infrastructure/runner-investigation.md

@@ -0,0 +1,182 @@
+# Infrastructure Investigation — CI runner and deploy failures
+
+> 2026-07-18. Supersedes `docs/ci-runner-investigation.md`.
+> **Conclusion: both failures are outside the repository.** Application code has been eliminated as a
+> cause by direct experiment. Confirmation and repair require host access — the exact asks are at the
+> end.
+
+There are **two independent infrastructure failures**:
+
+- **A — CI `test` job**: the backend suite fails only on the self-hosted `Live-Runner`.
+- **B — CI `deploy` job**: the SSH step fails ~3 s in, before doing any work.
+
+They are unrelated to each other and to the application code.
+
+---
+
+## A — Backend suite fails only on the runner
+
+### Evidence gathered
+
+- The suite had **never actually run in CI**. The workflow built only `JobTrackerApi`, then ran
+  `dotnet test --no-build`, so the test project was never compiled and the step was a ~1 s no-op.
+  Fixed in `cfba7fb`. **The failure is newly surfaced, not a regression** — it may be long-standing.
+- Job logs are unreadable: `GET /api/v1/.../actions/jobs/{id}/logs` → `401 token is required`.
+  Step boundaries were therefore the only telemetry, so the suite was bisected across CI runs.
+- Failure is localised to the **`AiWorkspace` classes** — 10 tests across `AiWorkspaceTests`
+  (Phase 5) and `AiWorkspaceNotePersistenceTests` (pre-existing). Locally these run in **1 s**.
+
+| Run | Steps observed | Reading |
+|---|---|---|
+| 524 | `Test backend` ✗ 8 s | one combined step, no detail |
+| 525 | restore ✓ 4 s, build ✓ 4 s, **test ✗ 3 s** | not a restore or compile error |
+| 526 | host smoke ✓ **1 s**, full suite (serial) ✗ 3 s | test host starts fine; not parallelism |
+| 527 | quarters: **A–C ✗ 3 s**, rest never ran | offender is alphabetically early |
+| 528 | per class: **`T AiWorkspace` ✗ 3 s**, others never ran | offender named |
+
+### Experiments performed
+
+Every experiment ran the same commit. All pass unless stated.
+
+| # | Experiment | Result |
+|---|---|---|
+| 1 | Windows host, full suite | 306 pass |
+| 2 | Clean `mcr.microsoft.com/dotnet/sdk:9.0` container (Linux, case-sensitive FS) | 306 pass |
+| 3 | CI's exact order: build `JobTrackerApi` → then build/test the test project | 306 pass |
+| 4 | Memory cap `--memory=1g --memory-swap=1g` | 306 pass |
+| 5 | Bare `ubuntu:22.04`, SDK via `dotnet-install.sh` into `$HOME/.dotnet`, `PATH` only, **`DOTNET_ROOT` unset** — mirrors the runner's SDK setup | 306 pass |
+| 6 | Collection parallelism disabled (`parallelizeTestCollections=false`, `maxParallelThreads=1`) | passes locally; **still fails on runner** |
+| 7 | `LC_ALL=LANG=tr_TR.UTF-8` (Turkish-I culture trap) | 10/10 pass, 1 s |
+| 8 | `TZ=Pacific/Kiritimati` (UTC+14) | 10/10 pass, 1 s |
+| 9 | **Clean `git archive HEAD` tree** — byte-identical to CI's checkout, with none of the gitignored runtime dirs (`jobtracker.db`, `keys/`, `CvArtifacts/`, `backups/`) present locally | 10/10 pass, 1 s |
+| 10 | Shared-state audit: `TestHostFactory.CreateInMemoryDb` uses `Guid.NewGuid()` per test | no shared store |
+
+Experiment 9 is the decisive one: it removes the last difference between the local tree and the
+runner's checkout. The exact source CI compiles produces a passing suite.
+
+### Root cause hypothesis
+
+The runner host kills the test process. The workflow already documents **three separate failure modes
+on this same runner, all with the signature of a process dying with no usable error output**:
+
+- `actions/setup-dotnet` "intermittently leaves a partial extraction in the shared tool-cache
+  (`tar: Cannot open: File exists`) or corrupts the SDK download" — hence the hand-rolled installer.
+- `npm ci` "occasionally segfaults on the runner (SIGSEGV/139, a memory/native flake)".
+- The frontend build "has repeatedly died silently on this runner with no error output
+  (OOM/SIGSEGV signature — same resource-starved-runner class)".
+
+A .NET test host exiting ~3 s into a 10-test run belongs to that same family. Two candidate mechanisms,
+in order of likelihood:
+
+1. **Resource exhaustion — memory or PID/thread limits.** The runner appears to share the host with
+   the production Docker stack. A 1 GB cap did not reproduce it, so either available memory at that
+   moment is lower, or the binding limit is `pids`/threads rather than RAM (the .NET test host spawns
+   more threads than `npm ci`, so it would hit a low `pids.max` first).
+2. **Disk exhaustion.** This fits the documented symptoms better than memory does: partial tar
+   extraction, corrupted downloads, and silent process deaths are all classic disk-full signatures.
+   `testhost` writes `TestResults/` and may write dumps.
+
+### Confidence level
+
+- **Application code is not the cause — high confidence (~95 %).** Ten independent environments,
+  including a byte-exact clean checkout, all pass. Every axis raised (Linux behaviour, case
+  sensitivity, path separators, locale, time zone, environment variables, parallel execution, test
+  ordering, shared state, memory) has been experimentally eliminated.
+- **Specific mechanism (OOM vs pids vs disk) — low/medium confidence (~40 %).** Not reproducible
+  remotely, and not resolvable without the job log or host access. I am deliberately not asserting
+  which one it is.
+
+### Why application code is no longer suspected
+
+1. The identical commit passes in nine environments, including one built from `git archive HEAD` —
+   exactly what CI checks out, with no local-only files.
+2. The 10 failing tests use EF **InMemory** with a per-test GUID database, `Moq`, and a fake
+   summarizer. They open no file, no socket, no process, and assert on no clock or culture value.
+3. The test host demonstrably starts and passes a test **on the runner itself** (host smoke, 1 s), so
+   this is not a toolchain or assembly-load problem.
+4. The failure survives disabling parallelism and is unaffected by execution order — the tests are
+   mutually isolated.
+5. Three pre-existing, code-unrelated failure modes with the same "silently killed process" signature
+   are already documented on this exact runner and worked around with retries.
+
+---
+
+## B — Deploy job fails before doing any work
+
+### Evidence
+
+- Step `Run remote deploy` (the `appleboy/ssh-action`) failed in **3 s** (`18:44:07 → 18:44:10`).
+  That is before `git fetch`, before `deploy.sh`, before any Docker build.
+- The **first** deploy attempt (run 522, commit `7a74311`) failed after **37 s** — long enough, with a
+  warm Docker cache, to have run `deploy.sh` and failed its post-deploy backend health check. That is
+  consistent with the MariaDB migration crash fixed in `1430313`.
+- Every attempt since fails at **3–7 s**, i.e. at connection time. The failure mode changed.
+- **Production is up and healthy**: `https://jobs.cesnimda.uk/` → 200 HTML,
+  `/api/auth/config` → 200 JSON (`{"requireAuth":true,...}`). The host is reachable from the internet,
+  so this is not an outage.
+- **Production is stale — Phase 4 and Phase 5 have never deployed.** Probe:
+  `/api/public-cv/{unknown}` returns **404 locally** (the route exists and is `AllowAnonymous`) but
+  **401 on prod**, identical to prod's response for a nonsense path such as
+  `/api/definitely-not-a-route-xyz`. `PublicCvController` is absent from production.
+
+### Consequence worth recording
+
+Because no Phase 4/5 deploy ever succeeded, **production never executed the faulty migration**. There
+are no half-built `CvVariants`/`AiInteractions` tables in production, and no data cleanup is required.
+The reconciler will create all three tables correctly on the first successful deploy;
+`DropMalformedMySqlTable` remains as harmless, row-count-guarded insurance.
+
+### Hypothesis
+
+The prod host is refusing the runner's SSH connection rather than failing inside the script. Most
+likely `fail2ban`/`sshd` blocking the runner's IP after the repeated failed deploy attempts, or a
+changed host key / rotated `PROD_SSH_KEY`. Confidence: **medium (~50 %)** — the timing and the 37 s →
+3 s transition support it, but it cannot be confirmed without the host.
+
+---
+
+## Required infrastructure changes
+
+**Blocking — cannot proceed without one of these:**
+
+1. **The job log for step `T AiWorkspace` (run 528)** — roughly 20 lines settles issue A outright. Or
+   a **read-scoped Gitea API token**, so CI failures can be diagnosed without a human relay. This is
+   the single highest-value item.
+2. **The deploy job log (run 523+)** — the `ssh-action` error line settles issue B.
+
+**Host checks (issue A):**
+
+3. `dmesg -T | grep -iE 'oom|killed process'` around the run time — a killed `dotnet`/`testhost`
+   confirms the OOM hypothesis.
+4. `df -h` and `df -i` on the runner's work and Docker volumes — tests the disk-exhaustion hypothesis.
+5. `ulimit -a` and `cat /sys/fs/cgroup/pids.max` for the runner user — tests the PID-limit hypothesis.
+6. `journalctl -u <gitea-runner-service> --since '2 hours ago'`.
+
+**Host checks (issue B):**
+
+7. `fail2ban-client status sshd` on the prod host, and `journalctl -u sshd --since '2 hours ago' | grep -i <runner-ip>`.
+8. Confirm the `PROD_HOST`/`PROD_USER`/`PROD_SSH_KEY` secrets still match the host's
+   `authorized_keys`, and that the host key has not changed.
+
+**Recommended remediation regardless of which hypothesis lands:**
+
+9. **Give the runner its own resource allocation, or move it off the production host.** It currently
+   appears to share a box with the prod Docker stack. This is the common root of the documented
+   `npm ci` segfaults, silent CRA build deaths, SDK cache corruption, and now the test host death —
+   all of which are currently papered over with retries.
+
+---
+
+## Repository state
+
+Kept — all correct independent of the outcome, none reverted:
+
+- `1430313` — CV builder / AI workspace tables provisioned by the MySQL-safe reconciler
+  (reproduced and verified against a real MariaDB 11 container).
+- `cfba7fb` — CI actually runs the backend suite.
+- `45725ac` — restore / build / test split, restore retries once.
+- `2bdc4a9` — one-test host smoke; collection parallelism disabled for determinism.
+- `7fa3080`, `0f62dc4` — bisection scaffolding, since removed.
+
+**No test was weakened, skipped, filtered, or disabled at any point.** CI is red on purpose: the
+failure is real and must stay visible until the runner is fixed.