From b7dc07b045ab704b9d4707ccf32018979cd57377 Mon Sep 17 00:00:00 2001 From: cesnimda Date: Sat, 18 Jul 2026 22:12:00 +0200 Subject: [PATCH] docs(ci): report runner-only backend test failure investigation MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Bisected the failure across four CI runs (job logs are not readable via the Gitea API) down to the AiWorkspace test classes — 10 tests that pass on Windows, in a clean Linux container, under a 1GB memory cap, in CI's exact step order, with a custom-dir SDK and no DOTNET_ROOT, serially, and under a hostile locale/timezone. Ruled out: Linux behaviour, case sensitivity, path separators, locale/culture, time zone, environment variables, parallel execution, test ordering, shared state, memory. Not testable remotely: host permissions/limits. Assessment is environmental: the workflow already documents three failure modes on this same runner with an identical signature (processes dying with no error output — SDK cache corruption, npm ci SIGSEGV, CRA build OOM/SIGSEGV). Report includes evidence table and recommended infrastructure fix. Removes the temporary bisection scaffolding; keeps the restore/build/test split and the host smoke. No test was weakened, skipped, or filtered. Co-Authored-By: Claude Opus 4.8 --- .gitea/workflows/ci-deploy.yml | 49 +++------------- docs/ci-runner-investigation.md | 100 ++++++++++++++++++++++++++++++++ 2 files changed, 108 insertions(+), 41 deletions(-) create mode 100644 docs/ci-runner-investigation.md diff --git a/.gitea/workflows/ci-deploy.yml b/.gitea/workflows/ci-deploy.yml index bdcafe4..feed99c 100644 --- a/.gitea/workflows/ci-deploy.yml +++ b/.gitea/workflows/ci-deploy.yml @@ -62,47 +62,14 @@ jobs: - name: Test backend (host smoke) run: dotnet test JobTrackerApi.Tests/JobTrackerApi.Tests.csproj --configuration Release --no-build --filter "FullyQualifiedName~CvBuilderTests.Every_catalog_theme_renders_valid_html" - # The host smoke above passes, so the test host starts fine; the full suite still dies ~3s in - # even with parallelism disabled, i.e. one specific test takes the process down on this runner - # only. Job logs are not readable via the API, so the suite is bisected across steps: the first - # failing step localises the offending class. Slices are alphabetical because execution order is - # alphabetical once collection parallelism is off. Temporary diagnostic — every test still runs. - # Slice 1 (A-C) is the failing slice; split per class to name the offender in one run. - - name: T AiWorkspace - run: dotnet test JobTrackerApi.Tests/JobTrackerApi.Tests.csproj --configuration Release --no-build -- xUnit.parallelizeTestCollections=false xUnit.maxParallelThreads=1 --filter "FullyQualifiedName~AiWorkspace" - - - name: T Attachment - run: dotnet test JobTrackerApi.Tests/JobTrackerApi.Tests.csproj --configuration Release --no-build -- xUnit.parallelizeTestCollections=false xUnit.maxParallelThreads=1 --filter "FullyQualifiedName~Attachment" - - - name: T AuthAndSystem - run: dotnet test JobTrackerApi.Tests/JobTrackerApi.Tests.csproj --configuration Release --no-build -- xUnit.parallelizeTestCollections=false xUnit.maxParallelThreads=1 --filter "FullyQualifiedName~AuthAndSystem" - - - name: T BackupController - run: dotnet test JobTrackerApi.Tests/JobTrackerApi.Tests.csproj --configuration Release --no-build -- xUnit.parallelizeTestCollections=false xUnit.maxParallelThreads=1 --filter "FullyQualifiedName~BackupController" - - - name: T CareerProfile - run: dotnet test JobTrackerApi.Tests/JobTrackerApi.Tests.csproj --configuration Release --no-build -- xUnit.parallelizeTestCollections=false xUnit.maxParallelThreads=1 --filter "FullyQualifiedName~CareerProfile" - - - name: T ClientErrors - run: dotnet test JobTrackerApi.Tests/JobTrackerApi.Tests.csproj --configuration Release --no-build -- xUnit.parallelizeTestCollections=false xUnit.maxParallelThreads=1 --filter "FullyQualifiedName~ClientErrors" - - - name: T Correspondence - run: dotnet test JobTrackerApi.Tests/JobTrackerApi.Tests.csproj --configuration Release --no-build -- xUnit.parallelizeTestCollections=false xUnit.maxParallelThreads=1 --filter "FullyQualifiedName~Correspondence" - - - name: T CvBuilder - run: dotnet test JobTrackerApi.Tests/JobTrackerApi.Tests.csproj --configuration Release --no-build -- xUnit.parallelizeTestCollections=false xUnit.maxParallelThreads=1 --filter "FullyQualifiedName~CvBuilder" - - - name: T CvCorpus - run: dotnet test JobTrackerApi.Tests/JobTrackerApi.Tests.csproj --configuration Release --no-build -- xUnit.parallelizeTestCollections=false xUnit.maxParallelThreads=1 --filter "FullyQualifiedName~CvCorpus" - - - name: Test backend (slice 2 D-J) - run: dotnet test JobTrackerApi.Tests/JobTrackerApi.Tests.csproj --configuration Release --no-build -- xUnit.parallelizeTestCollections=false xUnit.maxParallelThreads=1 --filter "FullyQualifiedName~DatabaseBackup|FullyQualifiedName~EmailStatus|FullyQualifiedName~GmailController|FullyQualifiedName~GoogleToken|FullyQualifiedName~Imap|FullyQualifiedName~InterviewPrep|FullyQualifiedName~JobApplications" - - - name: Test backend (slice 3 J-P) - run: dotnet test JobTrackerApi.Tests/JobTrackerApi.Tests.csproj --configuration Release --no-build -- xUnit.parallelizeTestCollections=false xUnit.maxParallelThreads=1 --filter "FullyQualifiedName~JobCvMatch|FullyQualifiedName~JobImport|FullyQualifiedName~JobPipeline|FullyQualifiedName~LocalAuth|FullyQualifiedName~MicrosoftGraph|FullyQualifiedName~MicrosoftToken|FullyQualifiedName~Ownership|FullyQualifiedName~ProductionConfig|FullyQualifiedName~ProfileCv" - - - name: Test backend (slice 4 R-T) - run: dotnet test JobTrackerApi.Tests/JobTrackerApi.Tests.csproj --configuration Release --no-build -- xUnit.parallelizeTestCollections=false xUnit.maxParallelThreads=1 --filter "FullyQualifiedName~RulesEngine|FullyQualifiedName~Sessions|FullyQualifiedName~SqliteMigration|FullyQualifiedName~StageAnalytics|FullyQualifiedName~Summarizer|FullyQualifiedName~TwoFactor" + # Runner-only failure, localised to the AiWorkspace tests by bisecting across steps (the job + # log is not readable via the Gitea API). Those 10 tests pass on Windows, in a clean Linux + # container, under a 1GB memory cap, with a custom-dir SDK and no DOTNET_ROOT, serially, and + # under a hostile locale/timezone -- so the trigger is specific to this runner host. + # See docs/ci-runner-investigation.md. Collection parallelism stays off for determinism, the + # same reason the frontend runs --runInBand. Every test still runs; nothing is filtered. + - name: Test backend + run: dotnet test JobTrackerApi.Tests/JobTrackerApi.Tests.csproj --configuration Release --no-build -- xUnit.parallelizeTestCollections=false xUnit.maxParallelThreads=1 - name: Install frontend deps working-directory: job-tracker-ui diff --git a/docs/ci-runner-investigation.md b/docs/ci-runner-investigation.md new file mode 100644 index 0000000..154fe54 --- /dev/null +++ b/docs/ci-runner-investigation.md @@ -0,0 +1,100 @@ +# CI Runner Investigation — backend suite fails only on `Live-Runner` + +> 2026-07-18. Status: **root cause narrowed to the runner host; not reproducible in any other +> environment.** Final confirmation needs the job log, which the Gitea API will not serve without a +> token. Recommended infrastructure fix at the bottom. + +## Summary + +The backend test suite (306 tests) passes everywhere it has been run except the self-hosted Gitea +runner, where the test step dies after ~3 s. Restore and build succeed; the test host starts and can +run a test. The failure is localised to the **`AiWorkspace` test classes** (10 tests). + +Important context: **this suite had never actually executed in CI.** The workflow built only +`JobTrackerApi` and then ran `dotnet test --no-build`, so the test project was never compiled and the +step was a ~1 s no-op (fixed in `cfba7fb`). The failure is therefore *newly surfaced*, not a +regression — it may have been present for a long time. + +## How it was narrowed + +Job logs return `401 token is required` from the Gitea API, so step boundaries were the only readable +telemetry. The suite was split and bisected across CI steps over four runs. + +| Run | Observation | +|---|---| +| 524 | `Test backend` fails at 8 s (restore+build+test in one step) | +| 525 | Split: restore 4 s ✓, build 4 s ✓, **test 3 s ✗** → not a compile/restore error | +| 526 | One-test host smoke **1 s ✓**, full suite (parallelism off) **3 s ✗** → host starts fine; not parallelism | +| 527 | Alphabetical quarters: **slice 1 (A–C) ✗ at 3 s**, later slices never ran | +| 528 | Per class: **`T AiWorkspace` ✗ at 3 s**; all other A–C classes never reached | + +The `FullyQualifiedName~AiWorkspace` filter matches 10 tests across `AiWorkspaceTests` (Phase 5) and +`AiWorkspaceNotePersistenceTests` (pre-existing). Locally that filter completes in **1 s**. + +## What was ruled out (evidence) + +Each was reproduced against the same commit, all 306 tests passing unless noted. + +| Hypothesis | Test performed | Result | +|---|---|---| +| Linux-specific behaviour | clean `mcr.microsoft.com/dotnet/sdk:9.0` container | 306 pass | +| File system case sensitivity | same (Linux is case-sensitive) | 306 pass | +| Path separators | same | 306 pass | +| CI step ordering | reproduced CI's exact build-then-test order | 306 pass | +| Memory pressure | `--memory=1g --memory-swap=1g` | 306 pass | +| Environment variables / SDK layout | bare `ubuntu:22.04`, SDK via `dotnet-install.sh` to `$HOME/.dotnet`, `PATH` only, **`DOTNET_ROOT` unset** (mirrors the runner) | 306 pass | +| Parallel execution | `xUnit.parallelizeTestCollections=false xUnit.maxParallelThreads=1` | passes locally; **still fails on runner** | +| Test ordering / shared state | `TestHostFactory.CreateInMemoryDb` uses `Guid.NewGuid()` per test — no shared store | isolated | +| Locale / culture | `LANG=LC_ALL=tr_TR.UTF-8` (Turkish-I trap) | 10/10 pass in 1 s | +| Time zone | `TZ=Pacific/Kiritimati` (UTC+14) | 10/10 pass in 1 s | +| Docker availability | these tests use EF InMemory + Moq; no Docker, network, or filesystem use | n/a | +| Permissions | not testable remotely | **unresolved** | + +## Assessment: environmental, not code + +The `AiWorkspace` tests use EF InMemory with a per-test GUID database, `Moq`, and a fake summarizer. +They touch no clock-sensitive assertion, no file, no socket, and no external process. They pass under +every adverse condition that could be simulated. That points at the runner host itself. + +The workflow already documents three separate failure modes on this exact runner, all with the same +signature — **a process dying with no usable error output**: + +- `actions/setup-dotnet` "intermittently leaves a partial extraction in the shared tool-cache … or + corrupts the SDK download" (hence the hand-rolled `dotnet-install.sh` + retry). +- `npm ci` "occasionally segfaults on the runner (SIGSEGV/139, a memory/native flake)". +- The frontend build "has repeatedly died silently on this runner with no error output + (OOM/SIGSEGV signature — same resource-starved-runner class)". + +A .NET test host exiting ~3 s into a 10-test run, on a host with a documented history of silently +killed processes, is consistent with the same resource starvation — most plausibly the OOM killer or +a cgroup limit, triggered while the runner shares the box with the production Docker stack. A 1 GB +container cap was not enough to reproduce it, so the runner's *available* memory at that moment is +likely lower, or the limit is on process/thread count rather than memory. + +## Recommended infrastructure fix + +In priority order: + +1. **Read the job log** for step `T AiWorkspace` (run 528) — 20 lines settles this immediately. Either + grant a read-scoped Gitea token so CI failures can be diagnosed without a human relay, or paste the + step output. Everything below is contingent on that. +2. **Check the host for OOM kills** around the run: `dmesg -T | grep -i -E 'oom|killed process'` and + `journalctl -u --since '1 hour ago'`. A killed `testhost`/`dotnet` confirms it. +3. **Give the runner headroom / isolation.** It currently appears to share the host with the prod + stack. Either cap the prod containers' memory, give the runner its own cgroup allocation, or move + it off the production host. This would also fix the pre-existing npm/CRA/SDK flakes the workflow + works around with retries. +4. **Check `ulimit`/cgroup pids** for the runner user (`ulimit -a`, `cat /sys/fs/cgroup/pids.max`). + The .NET test host spawns more threads than `npm ci`, so a low pids limit would hit it first. + +## Repository-side changes made during the investigation + +Kept, because they are correct independent of the outcome: + +- `cfba7fb` — CI actually runs the backend suite (dropped the no-op `--no-build`). +- `45725ac` — restore / build / test split into separate steps, restore retries once. +- `2bdc4a9` — one-test host smoke step; collection parallelism disabled for determinism. +- Bisection scaffolding was removed once it had served its purpose (`7fa3080`, `0f62dc4`). + +No test was weakened, skipped, or filtered at any point. CI remains red on purpose — the failure is +real and should stay visible until the runner is fixed.