cesnimda/jobtrackingapp
1
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity

docs(infra): host telemetry rules out disk/memory/PID exhaustion and fail2ban
CI and Deploy / test (push) Failing after 1m18s
Details
CI and Deploy / deploy (push) Has been skipped
Details

Browse Source 
Server shows 62G free (71% used), inodes 14%, 16G /dev/shm (~32G RAM), ulimit
-u 127749, no cgroup pids.max, and no fail2ban installed. That falsifies both
resource-exhaustion hypotheses at the host level and the fail2ban explanation
for the deploy failure.

Notes the caveat that Gitea act_runner usually runs jobs inside a Docker
container, so host figures do not describe the environment the tests ran in
(separate cgroup limits, and a 64MB /dev/shm by Docker default).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
cesnimda
2026-07-18 23:14:28 +02:00
parent 55962bd29b
commit 8f73548e33
1 changed files with 27 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/infrastructure/runner-investigation.md
+27
View File
@@ -76,6 +76,33 @@ in order of likelihood:
extraction, corrupted downloads, and silent process deaths are all classic disk-full signatures.
`testhost` writes `TestResults/` and may write dumps.
### Host telemetry (2026-07-18, post-reboot) — resource exhaustion ruled out
Collected from the server:
```
/ 217G 146G 62G 71% (inodes 14% used)
/dev/shm 16G 0 16G 0% (=> ~32 GB RAM)
ulimit -u 127749 ulimit -n 1024 ulimit -m unlimited
/sys/fs/cgroup/pids.max: No such file or directory
fail2ban-client: command not found
journalctl -u sshd: No entries (Ubuntu's unit is `ssh`, not `sshd`)
dmesg: read kernel buffer failed: Operation not permitted (needs sudo; cleared by reboot anyway)
```
**This falsifies both resource hypotheses at the host level**: there is no disk pressure, no inode
pressure, no memory pressure and no restrictive PID limit. It also falsifies the fail2ban explanation
for the deploy failure — fail2ban is not installed.
**Caveat that matters:** Gitea `act_runner` normally executes a `runs-on: ubuntu-latest` job **inside
a Docker container**, so the figures above describe the *host*, not the environment the tests
actually ran in. A job container has its own cgroup memory/PID limits and, by Docker default, a
**64 MB `/dev/shm`** regardless of the host's 16 GB. The relevant limits have therefore not been
measured yet — see the asks below.
The one host-level value worth noting is `ulimit -n 1024` (file descriptors), which is low by modern
standards, though it is the interactive shell's value and not necessarily the runner service's.
### Confidence level
- **Application code is not the cause — high confidence (~95 %).** Ten independent environments,