docs(infra): host telemetry rules out disk/memory/PID exhaustion and fail2ban
Server shows 62G free (71% used), inodes 14%, 16G /dev/shm (~32G RAM), ulimit -u 127749, no cgroup pids.max, and no fail2ban installed. That falsifies both resource-exhaustion hypotheses at the host level and the fail2ban explanation for the deploy failure. Notes the caveat that Gitea act_runner usually runs jobs inside a Docker container, so host figures do not describe the environment the tests ran in (separate cgroup limits, and a 64MB /dev/shm by Docker default). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -76,6 +76,33 @@ in order of likelihood:
|
|||||||
extraction, corrupted downloads, and silent process deaths are all classic disk-full signatures.
|
extraction, corrupted downloads, and silent process deaths are all classic disk-full signatures.
|
||||||
`testhost` writes `TestResults/` and may write dumps.
|
`testhost` writes `TestResults/` and may write dumps.
|
||||||
|
|
||||||
|
### Host telemetry (2026-07-18, post-reboot) — resource exhaustion ruled out
|
||||||
|
|
||||||
|
Collected from the server:
|
||||||
|
|
||||||
|
```
|
||||||
|
/ 217G 146G 62G 71% (inodes 14% used)
|
||||||
|
/dev/shm 16G 0 16G 0% (=> ~32 GB RAM)
|
||||||
|
ulimit -u 127749 ulimit -n 1024 ulimit -m unlimited
|
||||||
|
/sys/fs/cgroup/pids.max: No such file or directory
|
||||||
|
fail2ban-client: command not found
|
||||||
|
journalctl -u sshd: No entries (Ubuntu's unit is `ssh`, not `sshd`)
|
||||||
|
dmesg: read kernel buffer failed: Operation not permitted (needs sudo; cleared by reboot anyway)
|
||||||
|
```
|
||||||
|
|
||||||
|
**This falsifies both resource hypotheses at the host level**: there is no disk pressure, no inode
|
||||||
|
pressure, no memory pressure and no restrictive PID limit. It also falsifies the fail2ban explanation
|
||||||
|
for the deploy failure — fail2ban is not installed.
|
||||||
|
|
||||||
|
**Caveat that matters:** Gitea `act_runner` normally executes a `runs-on: ubuntu-latest` job **inside
|
||||||
|
a Docker container**, so the figures above describe the *host*, not the environment the tests
|
||||||
|
actually ran in. A job container has its own cgroup memory/PID limits and, by Docker default, a
|
||||||
|
**64 MB `/dev/shm`** regardless of the host's 16 GB. The relevant limits have therefore not been
|
||||||
|
measured yet — see the asks below.
|
||||||
|
|
||||||
|
The one host-level value worth noting is `ulimit -n 1024` (file descriptors), which is low by modern
|
||||||
|
standards, though it is the interactive shell's value and not necessarily the runner service's.
|
||||||
|
|
||||||
### Confidence level
|
### Confidence level
|
||||||
|
|
||||||
- **Application code is not the cause — high confidence (~95 %).** Ten independent environments,
|
- **Application code is not the cause — high confidence (~95 %).** Ten independent environments,
|
||||||
|
|||||||
Reference in New Issue
Block a user