Two post-reboot runs: the first reproduced the failure identically (smoke 1s
pass, suite 3s fail), the second failed earlier at `dotnet restore` in 0s — a
step that succeeded in 3-4s on every previous run, same commit, same runner.
That rules out stuck state (reboot changed nothing) and rules out a
deterministic sandbox policy such as seccomp/W^X blocking runtime IL emission,
which was the leading remaining hypothesis. Combined with host telemetry
showing no disk/memory/PID pressure, confidence in any specific mechanism drops
to ~25%; confidence that application code is not the cause stays high.
Removes the pure-vs-Moq diagnostic scaffolding (it never executed). No test
skipped or weakened.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Server shows 62G free (71% used), inodes 14%, 16G /dev/shm (~32G RAM), ulimit
-u 127749, no cgroup pids.max, and no fail2ban installed. That falsifies both
resource-exhaustion hypotheses at the host level and the fail2ban explanation
for the deploy failure.
Notes the caveat that Gitea act_runner usually runs jobs inside a Docker
container, so host figures do not describe the environment the tests ran in
(separate cgroup limits, and a 64MB /dev/shm by Docker default).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Moves the report to docs/infrastructure/runner-investigation.md with the
requested structure (evidence, experiments, hypothesis, confidence, required
infrastructure changes, why application code is no longer suspected).
Decisive new experiment: the suite was run from a clean `git archive HEAD`
tree — byte-identical to CI's checkout, without the gitignored runtime dirs
(jobtracker.db, keys/, CvArtifacts/, backups/) that earlier local runs had
silently included. 10/10 pass in 1s. That removes the last difference between
the local tree and the runner, eliminating application code (~95% confidence).
Also establishes, by route probe, that production is healthy but stale:
/api/public-cv/{unknown} returns 404 locally (route exists, AllowAnonymous) but
401 on prod, same as a nonsense path — PublicCvController is absent, so Phase 4
and Phase 5 have never deployed. Production therefore never ran the faulty
migration: no half-built tables exist there and no data cleanup is needed.
Deploy is a second, separate infrastructure failure: the first attempt reached
deploy.sh (37s, consistent with the MariaDB crash since fixed), every attempt
since dies at 3s at SSH connection time while the host serves traffic normally.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
cesnimda