From 8f73548e33631c7873481736293d875a177a7cd5 Mon Sep 17 00:00:00 2001 From: cesnimda Date: Sat, 18 Jul 2026 23:14:28 +0200 Subject: [PATCH] docs(infra): host telemetry rules out disk/memory/PID exhaustion and fail2ban Server shows 62G free (71% used), inodes 14%, 16G /dev/shm (~32G RAM), ulimit -u 127749, no cgroup pids.max, and no fail2ban installed. That falsifies both resource-exhaustion hypotheses at the host level and the fail2ban explanation for the deploy failure. Notes the caveat that Gitea act_runner usually runs jobs inside a Docker container, so host figures do not describe the environment the tests ran in (separate cgroup limits, and a 64MB /dev/shm by Docker default). Co-Authored-By: Claude Opus 4.8 --- docs/infrastructure/runner-investigation.md | 27 +++++++++++++++++++++ 1 file changed, 27 insertions(+) diff --git a/docs/infrastructure/runner-investigation.md b/docs/infrastructure/runner-investigation.md index 4258c69..0e9cca4 100644 --- a/docs/infrastructure/runner-investigation.md +++ b/docs/infrastructure/runner-investigation.md @@ -76,6 +76,33 @@ in order of likelihood: extraction, corrupted downloads, and silent process deaths are all classic disk-full signatures. `testhost` writes `TestResults/` and may write dumps. +### Host telemetry (2026-07-18, post-reboot) — resource exhaustion ruled out + +Collected from the server: + +``` +/ 217G 146G 62G 71% (inodes 14% used) +/dev/shm 16G 0 16G 0% (=> ~32 GB RAM) +ulimit -u 127749 ulimit -n 1024 ulimit -m unlimited +/sys/fs/cgroup/pids.max: No such file or directory +fail2ban-client: command not found +journalctl -u sshd: No entries (Ubuntu's unit is `ssh`, not `sshd`) +dmesg: read kernel buffer failed: Operation not permitted (needs sudo; cleared by reboot anyway) +``` + +**This falsifies both resource hypotheses at the host level**: there is no disk pressure, no inode +pressure, no memory pressure and no restrictive PID limit. It also falsifies the fail2ban explanation +for the deploy failure — fail2ban is not installed. + +**Caveat that matters:** Gitea `act_runner` normally executes a `runs-on: ubuntu-latest` job **inside +a Docker container**, so the figures above describe the *host*, not the environment the tests +actually ran in. A job container has its own cgroup memory/PID limits and, by Docker default, a +**64 MB `/dev/shm`** regardless of the host's 16 GB. The relevant limits have therefore not been +measured yet — see the asks below. + +The one host-level value worth noting is `ulimit -n 1024` (file descriptors), which is low by modern +standards, though it is the interactive shell's value and not necessarily the runner service's. + ### Confidence level - **Application code is not the cause — high confidence (~95 %).** Ten independent environments,