docs(infra): runner fails at different stages across runs — nondeterministic
Two post-reboot runs: the first reproduced the failure identically (smoke 1s pass, suite 3s fail), the second failed earlier at `dotnet restore` in 0s — a step that succeeded in 3-4s on every previous run, same commit, same runner. That rules out stuck state (reboot changed nothing) and rules out a deterministic sandbox policy such as seccomp/W^X blocking runtime IL emission, which was the leading remaining hypothesis. Combined with host telemetry showing no disk/memory/PID pressure, confidence in any specific mechanism drops to ~25%; confidence that application code is not the cause stays high. Removes the pure-vs-Moq diagnostic scaffolding (it never executed). No test skipped or weakened. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -103,15 +103,41 @@ measured yet — see the asks below.
|
||||
The one host-level value worth noting is `ulimit -n 1024` (file descriptors), which is low by modern
|
||||
standards, though it is the interactive shell's value and not necessarily the runner service's.
|
||||
|
||||
### Post-reboot runs — the failure moves between stages
|
||||
|
||||
Two further runs after the machine was restarted:
|
||||
|
||||
| Run | Result |
|
||||
|---|---|
|
||||
| `8f73548` (post-reboot) | **Identical** to pre-reboot: host smoke ✓ 1 s, full suite ✗ 3 s |
|
||||
| `c4c0cd4` | Failure **moved earlier**: `Restore backend tests` ✗ **0 s** — a step that took 3–4 s and succeeded in all previous runs. None of the queued diagnostic steps executed. |
|
||||
|
||||
Two conclusions:
|
||||
|
||||
1. **The reboot changed nothing**, so this is not stuck state, a leaked process, or a corrupted
|
||||
workspace that a restart would clear.
|
||||
2. **The failing stage is not stable across runs.** `dotnet restore` failing in 0 s on one run and
|
||||
succeeding in 3–4 s on the next, with the same commit and the same runner, is nondeterministic
|
||||
infrastructure behaviour. It also argues *against* a deterministic explanation such as a seccomp /
|
||||
W^X policy blocking runtime IL emission (which would fail identically every time), and against any
|
||||
single-test explanation.
|
||||
|
||||
Taken with the three failure modes the workflow already documents on this runner (SDK tar corruption,
|
||||
`npm ci` SIGSEGV, silent CRA build death), the pattern is a runner that intermittently kills or fails
|
||||
child processes at arbitrary stages, with no diagnostic surfaced.
|
||||
|
||||
### Confidence level
|
||||
|
||||
- **Application code is not the cause — high confidence (~95 %).** Ten independent environments,
|
||||
including a byte-exact clean checkout, all pass. Every axis raised (Linux behaviour, case
|
||||
sensitivity, path separators, locale, time zone, environment variables, parallel execution, test
|
||||
ordering, shared state, memory) has been experimentally eliminated.
|
||||
- **Specific mechanism (OOM vs pids vs disk) — low/medium confidence (~40 %).** Not reproducible
|
||||
remotely, and not resolvable without the job log or host access. I am deliberately not asserting
|
||||
which one it is.
|
||||
- **Specific mechanism — low confidence (~25 %), and lower than before.** Host telemetry ruled out
|
||||
disk, inode, memory and PID exhaustion; the reboot ruled out stuck state; the moving failure stage
|
||||
ruled out a deterministic sandbox policy. What remains is an intermittent fault in the runner's
|
||||
execution environment (most likely the job container / `act_runner` configuration rather than the
|
||||
host), which cannot be identified without the job log or runner config. I am not asserting a
|
||||
mechanism.
|
||||
|
||||
### Why application code is no longer suspected
|
||||
|
||||
|
||||
Reference in New Issue
Block a user