- consolidate API ownership and remove dead vendor code
- add Stripe billing, learning paths, and public CV hardening
- add migration, recovery, security, audit, and browser gates
Two post-reboot runs: the first reproduced the failure identically (smoke 1s
pass, suite 3s fail), the second failed earlier at `dotnet restore` in 0s — a
step that succeeded in 3-4s on every previous run, same commit, same runner.
That rules out stuck state (reboot changed nothing) and rules out a
deterministic sandbox policy such as seccomp/W^X blocking runtime IL emission,
which was the leading remaining hypothesis. Combined with host telemetry
showing no disk/memory/PID pressure, confidence in any specific mechanism drops
to ~25%; confidence that application code is not the cause stays high.
Removes the pure-vs-Moq diagnostic scaffolding (it never executed). No test
skipped or weakened.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Moves the report to docs/infrastructure/runner-investigation.md with the
requested structure (evidence, experiments, hypothesis, confidence, required
infrastructure changes, why application code is no longer suspected).
Decisive new experiment: the suite was run from a clean `git archive HEAD`
tree — byte-identical to CI's checkout, without the gitignored runtime dirs
(jobtracker.db, keys/, CvArtifacts/, backups/) that earlier local runs had
silently included. 10/10 pass in 1s. That removes the last difference between
the local tree and the runner, eliminating application code (~95% confidence).
Also establishes, by route probe, that production is healthy but stale:
/api/public-cv/{unknown} returns 404 locally (route exists, AllowAnonymous) but
401 on prod, same as a nonsense path — PublicCvController is absent, so Phase 4
and Phase 5 have never deployed. Production therefore never ran the faulty
migration: no half-built tables exist there and no data cleanup is needed.
Deploy is a second, separate infrastructure failure: the first attempt reached
deploy.sh (37s, consistent with the MariaDB crash since fixed), every attempt
since dies at 3s at SSH connection time while the host serves traffic normally.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Bisected the failure across four CI runs (job logs are not readable via the
Gitea API) down to the AiWorkspace test classes — 10 tests that pass on
Windows, in a clean Linux container, under a 1GB memory cap, in CI's exact
step order, with a custom-dir SDK and no DOTNET_ROOT, serially, and under a
hostile locale/timezone.
Ruled out: Linux behaviour, case sensitivity, path separators, locale/culture,
time zone, environment variables, parallel execution, test ordering, shared
state, memory. Not testable remotely: host permissions/limits.
Assessment is environmental: the workflow already documents three failure
modes on this same runner with an identical signature (processes dying with no
error output — SDK cache corruption, npm ci SIGSEGV, CRA build OOM/SIGSEGV).
Report includes evidence table and recommended infrastructure fix.
Removes the temporary bisection scaffolding; keeps the restore/build/test split
and the host smoke. No test was weakened, skipped, or filtered.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The host smoke passes, so the test host starts; the full suite still dies ~3s
in with parallelism disabled, so one specific test takes the process down on
this runner only. Job logs are unreadable via the API, so the suite is sliced
across four steps — the first failing step identifies the class. Temporary
diagnostic scaffolding; every test still runs, nothing is skipped.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Restore and build pass on the self-hosted runner but the test run dies after
~3s — too fast to have executed 306 tests. The suite passes on Windows, in a
clean Linux container, under a 1GB memory cap, in CI's exact step order, and
with the SDK installed to a custom dir without DOTNET_ROOT, so the trigger is
specific to this runner rather than the code.
Adds a one-test host smoke step (separates "host cannot start" from "the suite
takes the host down" using step boundaries, since job logs are not readable via
the API) and disables xUnit collection parallelism for the full run — the same
remedy the frontend already needs (--runInBand) on this resource-flaky runner.
All 306 tests still run; only concurrency changes.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The backend test step fails on the self-hosted runner after 8s while passing
on Windows, in a clean Linux container, and in CI's exact build-then-test
order. The job log is not readable via the Gitea API (401), so step boundaries
are the only available telemetry: splitting restore / build / test makes the
failing phase identifiable from step timings alone.
Restore retries once, mirroring the npm ci and dotnet SDK retries already in
this workflow for the same runner's known flakiness. The suite itself is
unchanged — still the whole suite, nothing filtered or skipped.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The build step only builds JobTrackerApi, so the test project was never
compiled — and `dotnet test --no-build` then made the step a ~1s no-op
(locally it errors "test source file not found"; on the persistent
self-hosted runner it can silently run a stale binary). The 306 backend
tests have not been gating CI.
Drop --no-build so the test project is compiled and the suite runs.
Verified locally: 306 passed in 14s instead of "succeeding" in 1s.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
npm run build (Terser minify + fork-ts-checker workers) has now died three
distinct ways on this runner in this session: a printed Terser minify error,
an explicit SIGSEGV, and a fully silent kill with zero output between
'Creating an optimized production build...' and the failure line (OOM/SIGSEGV
signature — process killed before it could flush an error). All three are the
same resource-starved-runner class as the npm ci and dotnet-install flakes
already retried elsewhere in this workflow. Retry once, matching that pattern.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The single self-hosted act_runner intermittently fails actions/setup-dotnet:
a partial extraction sticks in the shared tool-cache (tar: Cannot open: File
exists) or the SDK tarball download corrupts. Install into a clean private
$HOME/.dotnet via dotnet-install.sh with a rm -rf + retry-once, matching the
npm ci and NuGet publish retries already in this pipeline.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The frontend deps step occasionally crashes with "Segmentation fault (core
dumped)" (exit 139) during `npm ci` — a memory/native flake on the act_runner,
unrelated to the change under test (it failed the deploy-fix PR whose only change
is the Dockerfile). Retry once with a clean node_modules before failing.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The whitelist silently skipped new suites; two regressions in
non-whitelisted suites reached main unnoticed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
cesnimda
cesnimda