diff --git a/.gitea/workflows/ci-deploy.yml b/.gitea/workflows/ci-deploy.yml index feed99c..bbe1edc 100644 --- a/.gitea/workflows/ci-deploy.yml +++ b/.gitea/workflows/ci-deploy.yml @@ -66,7 +66,7 @@ jobs: # log is not readable via the Gitea API). Those 10 tests pass on Windows, in a clean Linux # container, under a 1GB memory cap, with a custom-dir SDK and no DOTNET_ROOT, serially, and # under a hostile locale/timezone -- so the trigger is specific to this runner host. - # See docs/ci-runner-investigation.md. Collection parallelism stays off for determinism, the + # See docs/infrastructure/runner-investigation.md. Collection parallelism stays off for determinism, the # same reason the frontend runs --runInBand. Every test still runs; nothing is filtered. - name: Test backend run: dotnet test JobTrackerApi.Tests/JobTrackerApi.Tests.csproj --configuration Release --no-build -- xUnit.parallelizeTestCollections=false xUnit.maxParallelThreads=1 diff --git a/docs/ci-runner-investigation.md b/docs/ci-runner-investigation.md deleted file mode 100644 index 154fe54..0000000 --- a/docs/ci-runner-investigation.md +++ /dev/null @@ -1,100 +0,0 @@ -# CI Runner Investigation — backend suite fails only on `Live-Runner` - -> 2026-07-18. Status: **root cause narrowed to the runner host; not reproducible in any other -> environment.** Final confirmation needs the job log, which the Gitea API will not serve without a -> token. Recommended infrastructure fix at the bottom. - -## Summary - -The backend test suite (306 tests) passes everywhere it has been run except the self-hosted Gitea -runner, where the test step dies after ~3 s. Restore and build succeed; the test host starts and can -run a test. The failure is localised to the **`AiWorkspace` test classes** (10 tests). - -Important context: **this suite had never actually executed in CI.** The workflow built only -`JobTrackerApi` and then ran `dotnet test --no-build`, so the test project was never compiled and the -step was a ~1 s no-op (fixed in `cfba7fb`). The failure is therefore *newly surfaced*, not a -regression — it may have been present for a long time. - -## How it was narrowed - -Job logs return `401 token is required` from the Gitea API, so step boundaries were the only readable -telemetry. The suite was split and bisected across CI steps over four runs. - -| Run | Observation | -|---|---| -| 524 | `Test backend` fails at 8 s (restore+build+test in one step) | -| 525 | Split: restore 4 s ✓, build 4 s ✓, **test 3 s ✗** → not a compile/restore error | -| 526 | One-test host smoke **1 s ✓**, full suite (parallelism off) **3 s ✗** → host starts fine; not parallelism | -| 527 | Alphabetical quarters: **slice 1 (A–C) ✗ at 3 s**, later slices never ran | -| 528 | Per class: **`T AiWorkspace` ✗ at 3 s**; all other A–C classes never reached | - -The `FullyQualifiedName~AiWorkspace` filter matches 10 tests across `AiWorkspaceTests` (Phase 5) and -`AiWorkspaceNotePersistenceTests` (pre-existing). Locally that filter completes in **1 s**. - -## What was ruled out (evidence) - -Each was reproduced against the same commit, all 306 tests passing unless noted. - -| Hypothesis | Test performed | Result | -|---|---|---| -| Linux-specific behaviour | clean `mcr.microsoft.com/dotnet/sdk:9.0` container | 306 pass | -| File system case sensitivity | same (Linux is case-sensitive) | 306 pass | -| Path separators | same | 306 pass | -| CI step ordering | reproduced CI's exact build-then-test order | 306 pass | -| Memory pressure | `--memory=1g --memory-swap=1g` | 306 pass | -| Environment variables / SDK layout | bare `ubuntu:22.04`, SDK via `dotnet-install.sh` to `$HOME/.dotnet`, `PATH` only, **`DOTNET_ROOT` unset** (mirrors the runner) | 306 pass | -| Parallel execution | `xUnit.parallelizeTestCollections=false xUnit.maxParallelThreads=1` | passes locally; **still fails on runner** | -| Test ordering / shared state | `TestHostFactory.CreateInMemoryDb` uses `Guid.NewGuid()` per test — no shared store | isolated | -| Locale / culture | `LANG=LC_ALL=tr_TR.UTF-8` (Turkish-I trap) | 10/10 pass in 1 s | -| Time zone | `TZ=Pacific/Kiritimati` (UTC+14) | 10/10 pass in 1 s | -| Docker availability | these tests use EF InMemory + Moq; no Docker, network, or filesystem use | n/a | -| Permissions | not testable remotely | **unresolved** | - -## Assessment: environmental, not code - -The `AiWorkspace` tests use EF InMemory with a per-test GUID database, `Moq`, and a fake summarizer. -They touch no clock-sensitive assertion, no file, no socket, and no external process. They pass under -every adverse condition that could be simulated. That points at the runner host itself. - -The workflow already documents three separate failure modes on this exact runner, all with the same -signature — **a process dying with no usable error output**: - -- `actions/setup-dotnet` "intermittently leaves a partial extraction in the shared tool-cache … or - corrupts the SDK download" (hence the hand-rolled `dotnet-install.sh` + retry). -- `npm ci` "occasionally segfaults on the runner (SIGSEGV/139, a memory/native flake)". -- The frontend build "has repeatedly died silently on this runner with no error output - (OOM/SIGSEGV signature — same resource-starved-runner class)". - -A .NET test host exiting ~3 s into a 10-test run, on a host with a documented history of silently -killed processes, is consistent with the same resource starvation — most plausibly the OOM killer or -a cgroup limit, triggered while the runner shares the box with the production Docker stack. A 1 GB -container cap was not enough to reproduce it, so the runner's *available* memory at that moment is -likely lower, or the limit is on process/thread count rather than memory. - -## Recommended infrastructure fix - -In priority order: - -1. **Read the job log** for step `T AiWorkspace` (run 528) — 20 lines settles this immediately. Either - grant a read-scoped Gitea token so CI failures can be diagnosed without a human relay, or paste the - step output. Everything below is contingent on that. -2. **Check the host for OOM kills** around the run: `dmesg -T | grep -i -E 'oom|killed process'` and - `journalctl -u --since '1 hour ago'`. A killed `testhost`/`dotnet` confirms it. -3. **Give the runner headroom / isolation.** It currently appears to share the host with the prod - stack. Either cap the prod containers' memory, give the runner its own cgroup allocation, or move - it off the production host. This would also fix the pre-existing npm/CRA/SDK flakes the workflow - works around with retries. -4. **Check `ulimit`/cgroup pids** for the runner user (`ulimit -a`, `cat /sys/fs/cgroup/pids.max`). - The .NET test host spawns more threads than `npm ci`, so a low pids limit would hit it first. - -## Repository-side changes made during the investigation - -Kept, because they are correct independent of the outcome: - -- `cfba7fb` — CI actually runs the backend suite (dropped the no-op `--no-build`). -- `45725ac` — restore / build / test split into separate steps, restore retries once. -- `2bdc4a9` — one-test host smoke step; collection parallelism disabled for determinism. -- Bisection scaffolding was removed once it had served its purpose (`7fa3080`, `0f62dc4`). - -No test was weakened, skipped, or filtered at any point. CI remains red on purpose — the failure is -real and should stay visible until the runner is fixed. diff --git a/docs/infrastructure/runner-investigation.md b/docs/infrastructure/runner-investigation.md new file mode 100644 index 0000000..4258c69 --- /dev/null +++ b/docs/infrastructure/runner-investigation.md @@ -0,0 +1,182 @@ +# Infrastructure Investigation — CI runner and deploy failures + +> 2026-07-18. Supersedes `docs/ci-runner-investigation.md`. +> **Conclusion: both failures are outside the repository.** Application code has been eliminated as a +> cause by direct experiment. Confirmation and repair require host access — the exact asks are at the +> end. + +There are **two independent infrastructure failures**: + +- **A — CI `test` job**: the backend suite fails only on the self-hosted `Live-Runner`. +- **B — CI `deploy` job**: the SSH step fails ~3 s in, before doing any work. + +They are unrelated to each other and to the application code. + +--- + +## A — Backend suite fails only on the runner + +### Evidence gathered + +- The suite had **never actually run in CI**. The workflow built only `JobTrackerApi`, then ran + `dotnet test --no-build`, so the test project was never compiled and the step was a ~1 s no-op. + Fixed in `cfba7fb`. **The failure is newly surfaced, not a regression** — it may be long-standing. +- Job logs are unreadable: `GET /api/v1/.../actions/jobs/{id}/logs` → `401 token is required`. + Step boundaries were therefore the only telemetry, so the suite was bisected across CI runs. +- Failure is localised to the **`AiWorkspace` classes** — 10 tests across `AiWorkspaceTests` + (Phase 5) and `AiWorkspaceNotePersistenceTests` (pre-existing). Locally these run in **1 s**. + +| Run | Steps observed | Reading | +|---|---|---| +| 524 | `Test backend` ✗ 8 s | one combined step, no detail | +| 525 | restore ✓ 4 s, build ✓ 4 s, **test ✗ 3 s** | not a restore or compile error | +| 526 | host smoke ✓ **1 s**, full suite (serial) ✗ 3 s | test host starts fine; not parallelism | +| 527 | quarters: **A–C ✗ 3 s**, rest never ran | offender is alphabetically early | +| 528 | per class: **`T AiWorkspace` ✗ 3 s**, others never ran | offender named | + +### Experiments performed + +Every experiment ran the same commit. All pass unless stated. + +| # | Experiment | Result | +|---|---|---| +| 1 | Windows host, full suite | 306 pass | +| 2 | Clean `mcr.microsoft.com/dotnet/sdk:9.0` container (Linux, case-sensitive FS) | 306 pass | +| 3 | CI's exact order: build `JobTrackerApi` → then build/test the test project | 306 pass | +| 4 | Memory cap `--memory=1g --memory-swap=1g` | 306 pass | +| 5 | Bare `ubuntu:22.04`, SDK via `dotnet-install.sh` into `$HOME/.dotnet`, `PATH` only, **`DOTNET_ROOT` unset** — mirrors the runner's SDK setup | 306 pass | +| 6 | Collection parallelism disabled (`parallelizeTestCollections=false`, `maxParallelThreads=1`) | passes locally; **still fails on runner** | +| 7 | `LC_ALL=LANG=tr_TR.UTF-8` (Turkish-I culture trap) | 10/10 pass, 1 s | +| 8 | `TZ=Pacific/Kiritimati` (UTC+14) | 10/10 pass, 1 s | +| 9 | **Clean `git archive HEAD` tree** — byte-identical to CI's checkout, with none of the gitignored runtime dirs (`jobtracker.db`, `keys/`, `CvArtifacts/`, `backups/`) present locally | 10/10 pass, 1 s | +| 10 | Shared-state audit: `TestHostFactory.CreateInMemoryDb` uses `Guid.NewGuid()` per test | no shared store | + +Experiment 9 is the decisive one: it removes the last difference between the local tree and the +runner's checkout. The exact source CI compiles produces a passing suite. + +### Root cause hypothesis + +The runner host kills the test process. The workflow already documents **three separate failure modes +on this same runner, all with the signature of a process dying with no usable error output**: + +- `actions/setup-dotnet` "intermittently leaves a partial extraction in the shared tool-cache + (`tar: Cannot open: File exists`) or corrupts the SDK download" — hence the hand-rolled installer. +- `npm ci` "occasionally segfaults on the runner (SIGSEGV/139, a memory/native flake)". +- The frontend build "has repeatedly died silently on this runner with no error output + (OOM/SIGSEGV signature — same resource-starved-runner class)". + +A .NET test host exiting ~3 s into a 10-test run belongs to that same family. Two candidate mechanisms, +in order of likelihood: + +1. **Resource exhaustion — memory or PID/thread limits.** The runner appears to share the host with + the production Docker stack. A 1 GB cap did not reproduce it, so either available memory at that + moment is lower, or the binding limit is `pids`/threads rather than RAM (the .NET test host spawns + more threads than `npm ci`, so it would hit a low `pids.max` first). +2. **Disk exhaustion.** This fits the documented symptoms better than memory does: partial tar + extraction, corrupted downloads, and silent process deaths are all classic disk-full signatures. + `testhost` writes `TestResults/` and may write dumps. + +### Confidence level + +- **Application code is not the cause — high confidence (~95 %).** Ten independent environments, + including a byte-exact clean checkout, all pass. Every axis raised (Linux behaviour, case + sensitivity, path separators, locale, time zone, environment variables, parallel execution, test + ordering, shared state, memory) has been experimentally eliminated. +- **Specific mechanism (OOM vs pids vs disk) — low/medium confidence (~40 %).** Not reproducible + remotely, and not resolvable without the job log or host access. I am deliberately not asserting + which one it is. + +### Why application code is no longer suspected + +1. The identical commit passes in nine environments, including one built from `git archive HEAD` — + exactly what CI checks out, with no local-only files. +2. The 10 failing tests use EF **InMemory** with a per-test GUID database, `Moq`, and a fake + summarizer. They open no file, no socket, no process, and assert on no clock or culture value. +3. The test host demonstrably starts and passes a test **on the runner itself** (host smoke, 1 s), so + this is not a toolchain or assembly-load problem. +4. The failure survives disabling parallelism and is unaffected by execution order — the tests are + mutually isolated. +5. Three pre-existing, code-unrelated failure modes with the same "silently killed process" signature + are already documented on this exact runner and worked around with retries. + +--- + +## B — Deploy job fails before doing any work + +### Evidence + +- Step `Run remote deploy` (the `appleboy/ssh-action`) failed in **3 s** (`18:44:07 → 18:44:10`). + That is before `git fetch`, before `deploy.sh`, before any Docker build. +- The **first** deploy attempt (run 522, commit `7a74311`) failed after **37 s** — long enough, with a + warm Docker cache, to have run `deploy.sh` and failed its post-deploy backend health check. That is + consistent with the MariaDB migration crash fixed in `1430313`. +- Every attempt since fails at **3–7 s**, i.e. at connection time. The failure mode changed. +- **Production is up and healthy**: `https://jobs.cesnimda.uk/` → 200 HTML, + `/api/auth/config` → 200 JSON (`{"requireAuth":true,...}`). The host is reachable from the internet, + so this is not an outage. +- **Production is stale — Phase 4 and Phase 5 have never deployed.** Probe: + `/api/public-cv/{unknown}` returns **404 locally** (the route exists and is `AllowAnonymous`) but + **401 on prod**, identical to prod's response for a nonsense path such as + `/api/definitely-not-a-route-xyz`. `PublicCvController` is absent from production. + +### Consequence worth recording + +Because no Phase 4/5 deploy ever succeeded, **production never executed the faulty migration**. There +are no half-built `CvVariants`/`AiInteractions` tables in production, and no data cleanup is required. +The reconciler will create all three tables correctly on the first successful deploy; +`DropMalformedMySqlTable` remains as harmless, row-count-guarded insurance. + +### Hypothesis + +The prod host is refusing the runner's SSH connection rather than failing inside the script. Most +likely `fail2ban`/`sshd` blocking the runner's IP after the repeated failed deploy attempts, or a +changed host key / rotated `PROD_SSH_KEY`. Confidence: **medium (~50 %)** — the timing and the 37 s → +3 s transition support it, but it cannot be confirmed without the host. + +--- + +## Required infrastructure changes + +**Blocking — cannot proceed without one of these:** + +1. **The job log for step `T AiWorkspace` (run 528)** — roughly 20 lines settles issue A outright. Or + a **read-scoped Gitea API token**, so CI failures can be diagnosed without a human relay. This is + the single highest-value item. +2. **The deploy job log (run 523+)** — the `ssh-action` error line settles issue B. + +**Host checks (issue A):** + +3. `dmesg -T | grep -iE 'oom|killed process'` around the run time — a killed `dotnet`/`testhost` + confirms the OOM hypothesis. +4. `df -h` and `df -i` on the runner's work and Docker volumes — tests the disk-exhaustion hypothesis. +5. `ulimit -a` and `cat /sys/fs/cgroup/pids.max` for the runner user — tests the PID-limit hypothesis. +6. `journalctl -u --since '2 hours ago'`. + +**Host checks (issue B):** + +7. `fail2ban-client status sshd` on the prod host, and `journalctl -u sshd --since '2 hours ago' | grep -i `. +8. Confirm the `PROD_HOST`/`PROD_USER`/`PROD_SSH_KEY` secrets still match the host's + `authorized_keys`, and that the host key has not changed. + +**Recommended remediation regardless of which hypothesis lands:** + +9. **Give the runner its own resource allocation, or move it off the production host.** It currently + appears to share a box with the prod Docker stack. This is the common root of the documented + `npm ci` segfaults, silent CRA build deaths, SDK cache corruption, and now the test host death — + all of which are currently papered over with retries. + +--- + +## Repository state + +Kept — all correct independent of the outcome, none reverted: + +- `1430313` — CV builder / AI workspace tables provisioned by the MySQL-safe reconciler + (reproduced and verified against a real MariaDB 11 container). +- `cfba7fb` — CI actually runs the backend suite. +- `45725ac` — restore / build / test split, restore retries once. +- `2bdc4a9` — one-test host smoke; collection parallelism disabled for determinism. +- `7fa3080`, `0f62dc4` — bisection scaffolding, since removed. + +**No test was weakened, skipped, filtered, or disabled at any point.** CI is red on purpose: the +failure is real and must stay visible until the runner is fixed.