jobtrackingapp/docs/_archive/infrastructure-investigation.md

# Infrastructure Investigation — Deployment Failure (Server Unresponsive, Containers Stopped)

**Date:** 2026-07-13
**Trigger:** Latest deployment failed; server became unresponsive; some Docker containers stopped unexpectedly;
pushes occasionally fail; deployment occasionally hangs. Scope later widened by the user: the same host also
runs Jellyfin/Radarr/Sonarr and other unrelated services, which show the identical symptom (containers not
coming back after reboot despite `restart: always`/`unless-stopped`) — confirming this is a **host-wide**
problem, not specific to the jobtracker app.
**Mode:** Investigation only. No changes, restarts, deletions, prunes, installs, or fixes were made. No
configuration was modified. All commands run over SSH were read-only (`journalctl`, `docker ps`/`inspect`,
`free`, `df`, `uptime`, `crontab -l`, `cat`) — nothing was written, restarted, or deleted on the server.

---

## Method note — two phases of this investigation

**Phase 1 (static, no server access):** initial pass was done entirely from the local development checkout,
without live access to the deployment host — its SSH credentials are Gitea Actions secrets not present on this
machine. That phase built a hypothesis from `docker-compose.yml`, `Dockerfile`s, `deploy/deploy.sh`, the CI
workflow, and git history alone (see "Phase 1 findings" below — still valid as contributing/secondary factors).

**Phase 2 (live, with server access):** the user then provided direct SSH access (`pi@192.168.50.13`) and noted
the problem isn't limited to this app. This phase found the actual, currently-reproducing root cause with
timestamped live evidence (kernel/systemd journal, `docker inspect`, `docker ps -a` across every Compose stack
on the box) — see "**Primary Finding — systemd D-Bus/session-bus degradation (host-wide)**" immediately below.
This supersedes Phase 1's top-ranked hypothesis (host-level resource exhaustion): the box has 31GB RAM (23GB
free), 6 cores at ~10-25% load, and 8GB of completely unused swap — it is not resource-starved. Phase 1's
findings about the *jobtracker deploy process specifically* (no health checks, building images on the live
host, etc.) remain valid and worth fixing, but they are not the primary cause of the reported outage.

---

## Primary Finding — systemd D-Bus/session-bus degradation (host-wide, confirmed live)

**Issue:** `systemd`'s own D-Bus/session-bus subsystem (`org.freedesktop.systemd1`) is intermittently failing to
respond within its 25-second timeout, host-wide, to **every kind of request that needs it** — not just Docker.
This breaks three unrelated things by the same mechanism:

1. **New container creation and container restarts.** Docker's `systemd` cgroup driver asks systemd (over
   D-Bus) to create a transient `.scope` unit for every container start/restart. When that call times out,
   the container fails to start with `OCI runtime create failed: ... unable to start unit
   "docker-<id>.scope" ... Failed to activate service 'org.freedesktop.systemd1': timed out
   (service_start_timeout=25000ms)`. This is Docker's own restart-manager logging the exact same error when
   trying to *restart* an existing container — i.e. `restart: always`/`unless-stopped` cannot save a container
   if the D-Bus call underneath it is what's failing.
2. **New SSH sessions.** `systemd-logind` needs the same D-Bus mechanism to register a session scope for every
   new login. Live log during this investigation: `sshd[...]: pam_systemd(sshd:session): Failed to create
   session: Connection timed out` and `systemd-logind[1031]: Failed to start session scope session-6497.scope:
   Failed to activate service 'org.freedesktop.systemd1': timed out`. If this gets worse, it can make the box
   genuinely un-SSH-able — a very plausible reading of "server becomes unresponsive."
3. **systemd-journald's own internal watchdog heartbeat.** `systemd-journald[432]: Failed to send WATCHDOG=1
   notification message: Transport endpoint is not connected` — journald cannot even talk to systemd about its
   own liveness. "Transport endpoint is not connected" (`ENOTCONN`) is a broken-socket error, not a busy/slow
   one — consistent with systemd's D-Bus broker itself being unhealthy, not merely overloaded.

**Evidence (live, timestamped, gathered 2026-07-13 ~15:30-15:46 local server time):**
```
Jul 13 14:53:35 mediaserver dockerd[1326]: ... restartmanger wait error: failed to create task for container:
  ... Failed to activate service 'org.freedesktop.systemd1': timed out (service_start_timeout=25000ms)
Jul 13 15:37:01 mediaserver systemd-journald[432]: Failed to send WATCHDOG=1 notification message:
  Transport endpoint is not connected
Jul 13 15:38:09 mediaserver systemd-logind[1031]: Failed to start session scope session-6481.scope:
  Connection timed out
Jul 13 15:38:09 mediaserver sshd[1994140]: pam_systemd(sshd:session): Failed to create session: Connection timed out
Jul 13 15:46:20 mediaserver sshd[2004289]: pam_systemd(sshd:session): Failed to create session:
  Failed to activate service 'org.freedesktop.systemd1': timed out (service_start_timeout=25000ms)
Jul 13 15:46:20 mediaserver systemd-logind[1031]: Failed to start session scope session-6497.scope:
  Failed to activate service 'org.freedesktop.systemd1': timed out (service_start_timeout=25000ms)
```
The `systemd-journald` watchdog failure recurred roughly every 90 seconds to 2 minutes throughout the ~10-minute
observation window — i.e. this was actively reproducing *while the investigation was running*, not a rare blip.

`docker inspect deunhealth` showed `RestartCount=389`, `StartedAt=2026-07-13T12:40:14Z` — a container that has
been crash-restart-looping 389 times (and counting). One of its restart attempts is directly logged as failing
for the *same* D-Bus-timeout reason (`restartmanger wait error ... Failed to activate service
'org.freedesktop.systemd1'`). Ironically, `deunhealth` (`qmcgaw/deunhealth`) is itself a watchdog tool whose job
is auto-restarting other unhealthy containers — so the host's auto-healing mechanism is caught in the same
degraded-D-Bus bottleneck it exists to work around.

The very first occurrence of this exact Docker/systemd1 timeout in the journal dates back to **`Apr 29 23:25:52`**
— **this has been an intermittent problem for roughly 2.5 months**, not something new introduced by the latest
deployment. Only 7 occurrences matched in Docker's own log across the last 48 hours specifically, but the
broader journald-watchdog/logind symptom was firing every ~90 seconds during the live observation window,
suggesting either a currently-active flare-up or a much higher background rate than Docker's log alone shows
(Docker only logs the timeout when *it itself* happens to need D-Bus at that exact moment; logind/journald hit
it far more often simply because they touch D-Bus more frequently).

**Contributing factor (not necessarily the root cause, but plausible amplifier):** this host runs **37 containers
across 7 separate Docker Compose projects** simultaneously — `app` (jobtracker prod, confirmed to be
`/opt/job-tracker/app`, matching this repo's deploy target exactly), `deploy` (a "resumesite" project), `docker`
(the main Jellyfin/Radarr/Sonarr media stack, 22 containers), `inboxintel`, `mealie`, `mediaserver` (a *second*,
apparently overlapping media-stack definition), and a dead leftover `jobtracker` project at a different,
stale path (`/home/pi/jobtracker/JobTracker/docker-compose.yml`) — clearly an old pre-migration deployment
location nobody cleaned up. On top of that, two cron jobs auto-deploy other projects **every 1-2 minutes**:
```
*/2 * * * * /opt/inboxintel/deploy/auto-deploy.sh
* * * * * /opt/resumesite/deploy/autodeploy.sh
```
Gitea Actions CI job containers were also observed live in the Docker log (`GITEA-ACTIONS-TASK-1083...JOB-deploy`,
`GITEA-ACTIONS-TASK-1084...JOB-test`) — confirming the self-hosted CI runner from Phase 1's investigation
**is this same box**. Combined with 389+ restart attempts from one crash-looping container alone, this is a lot
of transient-unit churn (container creates/destroys, session scopes, per-minute cron-triggered rebuilds) hitting
systemd's D-Bus continuously. High churn is a well-known trigger for exactly this class of systemd bug on some
Ubuntu/systemd version combinations (the box is on kernel `6.8.0-110-generic`) — a D-Bus broker or transient-unit
bookkeeping resource (e.g. an internal queue, a leaked file descriptor, an fd/inotify limit) getting exhausted
under sustained high creation/deletion rate.

**Impact:** This single mechanism plausibly explains **all four** originally-reported symptoms at once, across
**every app on the host**, not just jobtracker:
- "Some Docker containers stop unexpectedly" — a restart attempt loses the D-Bus race and the container simply
  stays down, non-deterministically (whichever container's restart happens to line up with a bad D-Bus moment).
- "Deployment occasionally hangs" — `docker compose up --force-recreate` creates new containers, each requiring
  the same D-Bus call; a deploy can stall or fail here exactly like any other container start.
- "The server becomes unresponsive" — new SSH sessions and journald's own health-check both depend on the same
  broken mechanism; in a bad enough episode this could make the box very difficult to even log into.
- "Pushes occasionally fail" — consistent with a CI job container (visible running on this same host) hitting
  the same timeout during its own container lifecycle.

**Confidence:** **High** — this is not inferred from static config or git history, it is directly observed,
timestamped, live log evidence, reproducing during the investigation itself, across three independent subsystems
(Docker, sshd/logind, journald) that all point to the same underlying D-Bus failure.

**What would raise this from High to Confirmed:** root-level `journalctl` access (this investigation ran as the
unprivileged `pi` user — `sudo` requires a password not available non-interactively over this SSH session, so
kernel-level (`dmesg`) OOM/hardware logs and the D-Bus broker's *own* service logs (`systemd-logind.service`,
`dbus.service`/`dbus-broker.service`) were not reachable). See "What I could not verify" for the exact commands
to run with root access.

---

## Phase 1 findings (static analysis, secondary/contributing — still valid)

These were the findings from before server access was available. They remain accurate descriptions of the
jobtracker deploy process's own gaps, and are worth fixing regardless of the primary systemd finding above, but
they are not the main explanation for the reported outage.

**Overall health assessment (Phase 1):** The deployment pipeline and container topology are functionally
reasonable (sensible service split, `restart: unless-stopped`, a health check on the AI service, retry-once
logic bolted onto several known-flaky build steps) but have **no resource governance** anywhere — no
per-container CPU/memory limits, no health check on the two services that actually serve user traffic
(`backend`, `frontend`), and a deploy process that **builds new container images on the same host that is
simultaneously serving production traffic**, with zero headroom reserved for that build. Git history
independently documents at least five separate incidents over the last ~8 days (NuGet package-integrity
corruption, npm `SIGSEGV`, .NET SDK tool-cache corruption, and a webpack/Terser build that "died silently...
OOM/SIGSEGV signature", per the workflow's own comment) on this same runner/host class, each patched with a
"retry once" band-aid rather than a resource-level fix. **Note:** live data shows the host is not actually
memory-constrained (31GB RAM, 23GB free) — these build-tool crashes are more likely a *symptom* of the same
D-Bus/cgroup-scope-creation failures documented above (a build step that needs to spawn a container/process via
a cgroup scope hits the same timeout) than of genuine memory exhaustion.

**Most likely root cause:** Host-level memory/CPU exhaustion during the build phase of deployment, on hardware
that has already independently shown itself to be resource-constrained, with no per-container limits to contain
a spike and no health checks to catch a hung-but-technically-running container. This is consistent with all
four reported symptoms (containers stopping unexpectedly, deployment hanging, server unresponsiveness, and
occasional push/CI failures) being **the same underlying resource-starvation problem manifesting at different
pipeline stages**, not four unrelated problems.

**Confidence:** High for "the host is resource-constrained and deploys build under live load with no limits."
Medium for "this is specifically what caused the *latest* incident" (plausible and consistent with the evidence
trail, but not confirmed against that incident's actual logs, which I could not access).

---

## Findings

### Finding 1 — No CPU/memory limits on any container

**Issue:** `docker-compose.yml` defines four services (`backend`, `frontend`, `ai-service`, `ollama`) and none
of them set `mem_limit`, `cpus`, or a `deploy.resources.limits` block. Any single container is free to consume
all available host memory.

**Evidence:**
```yaml
# docker-compose.yml — repeated across every service, no limits anywhere
backend:
  build: ...
  restart: unless-stopped
  # no mem_limit / cpus / deploy.resources
```
Confirmed by reading the full file (`docker-compose.yml`, all four service blocks + the opt-in `ollama`
profile) — no resource stanza appears once in the file.

**Impact:** On a memory-constrained host, one container growing unbounded (or several building/starting
concurrently) triggers the Linux OOM killer. The OOM killer scores *all* processes on the host, not just the
one causing pressure — it can and does kill processes that had nothing to do with the spike. This directly
explains "**some** containers stop unexpectedly" (non-deterministic, not all containers, not the one under
load) rather than a clean, predictable failure of the actual offending service.

**Confidence:** High (structural fact, verified by reading the file in full).

---

### Finding 2 — No health check on `backend` or `frontend`; only `ai-service` and `ollama` have one

**Issue:** `ai-service` has a proper `healthcheck` (HTTP GET `/health`, 30s interval). `ollama` has one too
(`ollama list`, 20s interval). `backend` and `frontend` — the two services that actually serve the product and
its API — have **none**.

**Evidence:** Full `docker-compose.yml` read; `healthcheck:` appears exactly twice, on `ai-service` and
`ollama`. `deploy/deploy.sh`'s own post-deploy check confirms the gap — it only asks Docker for the container
*state*, not health:
```bash
backend_status="$(compose ps backend --format '{{.State}}' 2>/dev/null | head -n 1 | tr '[:upper:]' '[:lower:]')"
if [ "$backend_status" != "running" ]; then
  echo "Backend service is not healthy after deploy (state: ${backend_status:-unknown})."
```
`State` reports "running" the instant the process starts — it says nothing about whether the ASP.NET Core app
inside has actually finished startup, bound its port, or is responding. A backend that starts, then hangs
during EF Core migration or hits a startup exception loop *inside* the process without exiting, would report
`running` and pass this check while being completely unresponsive to real traffic.

**Impact:** Directly explains "**deployment occasionally hangs**" and false-positive successful deploys — the
gate that's supposed to catch a broken backend cannot detect the specific failure mode of "process is alive but
not serving," which is exactly what host resource starvation during startup would produce (slow/stuck EF
migrations, thread-pool starvation under memory pressure, etc.).

**Confidence:** High (structural fact; the reasoning about *why* it matters is inference, not confirmed against
the actual incident logs).

---

### Finding 3 — Documented history of resource-starvation symptoms on this same host class

**Issue:** Git history shows a cluster of CI/deploy resilience patches, each independently describing a
resource-exhaustion signature, spanning 2026-07-05 through 2026-07-12:

| Date | Commit | Symptom described in the commit/comment |
|---|---|---|
| 07-05 | `63e0300` / `e90835b` | NuGet package integrity check failures (`NU3008`) during `dotnet publish` on the build host — "transient download corruption on the build host" |
| 07-06 | `4b38f7c` | "`npm ci` occasionally segfaults on the runner (SIGSEGV/139, **a memory/native flake**)" |
| 07-11 | `fc35601` / `35eaef9` | `actions/setup-dotnet` "intermittently leaves a partial extraction in the shared tool-cache (`tar: Cannot open: File exists`) or corrupts the SDK download" |
| 07-11 | `4f98195` | CRA/webpack build "has repeatedly died silently on this runner with no error output (**OOM/SIGSEGV signature**...)" |
| 07-12 | `86cdafb` | A different class of bug (missing `.npmrc` in Docker build context) — not resource-related, included for completeness |

Every one of these except the last was patched with **retry-once logic**, not a resource-level fix. `deploy.sh`
itself carries the same pattern for the *production* build step, with an explicit comment on both retry paths:

```bash
echo "docker compose build for core services failed. Attempting one cleanup + retry
because layer extraction can fail on constrained hosts."
```
("constrained hosts" — the deploy script's own author already believed this.)

**Evidence:** `git log --oneline --grep=... --since="30 days ago"` (see command output used in this
investigation); direct commit message and inline-comment quotes above.

**Impact:** This is the strongest available evidence that the underlying host (CI runner and/or production
deploy target — the two may be the same self-hosted machine, common in small-scale setups; I could not confirm
either way without server access) has **insufficient memory headroom for its build workload**, as a standing,
recurring condition rather than a one-off.

**Confidence:** High that the pattern exists and points to memory pressure. Medium on "this is the same host as
the one that just failed" — plausible, not confirmed.

---

### Finding 4 — Deploy builds new images on the same host serving live traffic, with no isolation

**Issue:** `deploy/deploy.sh` runs `docker compose build backend frontend` (a full `dotnet publish` inside the
`mcr.microsoft.com/dotnet/sdk:9.0` image, plus a full `npm run build` inside `node:20-alpine`) directly on the
deploy target, **while the currently-running containers from the previous deploy are still serving production
traffic** (they aren't stopped until `compose up -d --force-recreate` afterward). Both build steps are
independently already known to be memory-hungry enough to need special handling:

```dockerfile
# job-tracker-ui/Dockerfile
shm_size: '1gb'
# Next's build type-checker needs more than Docker's default 64MB /dev/shm; too little
# causes a SIGSEGV during `npm run build`.
```

**Evidence:** `deploy/deploy.sh` full read (`build_core_with_recovery`, `compose build backend frontend` before
any container is stopped); `job-tracker-ui/Dockerfile` and its `docker-compose.yml` `shm_size` override, with
the SIGSEGV comment.

**Impact:** This is a **concurrent resource spike**: two separate heavy compiler toolchains (the .NET SDK
compiler/publish pipeline and webpack+Terser minification) running back-to-back or overlapping, on a host with
no memory limits configured (Finding 1), while that same host is also running the live app for existing users.
On already-constrained hardware (Finding 3), this is a very plausible trigger for the host itself becoming
unresponsive during a deploy, not just one container failing.

**Confidence:** High that this is how the deploy works (verified by reading the script). Medium-High that it is
a primary contributor to the reported "server becomes unresponsive" symptom specifically during deploys.

---

### Finding 5 — AI service memory footprint is opaque and possibly significant

**Issue:** `ai-service` loads a `distilbart-cnn-12-6` summarization model into RAM via CPU-only PyTorch
(`--extra-index-url https://download.pytorch.org/whl/cpu` in `tools/summarizer/Dockerfile`), and can optionally
also run a bundled `ollama` service with a `qwen2.5:7b` model (a 7B-parameter LLM, several GB of RAM/VRAM even
quantized) if the `bundled-ollama` Compose profile is active in production. Neither has a memory limit (Finding
1). Whether the bundled-Ollama profile or an external/cloud AI provider (`AI_PROVIDER=gemini`/`groq`, added
earlier this session) is actually active in the live `.env` could not be determined from this repo — that file
lives only on the server at `/opt/job-tracker/shared/.env`, outside version control.

**Evidence:** `tools/summarizer/Dockerfile`, `tools/summarizer/app.py` (`SKIP_MODEL_LOAD` / `EAGER_MODEL_LOAD`
env vars, `torch.device`), `docker-compose.yml` (`ollama` service, `profiles: ["bundled-ollama"]`,
`gpus: all`), `deploy/README.md` example `.env` (shows `OLLAMA_MODEL=qwen2.5:7b` as the documented default).

**Impact:** If the local-Ollama path is active rather than a cloud provider, this is a large, constant RAM
baseline on top of everything else, on a host already shown to be tight on memory. Cannot be confirmed without
reading the live `.env` and `docker stats` output on the server.

**Confidence:** Medium — plausible contributor, not confirmed as active in the current production configuration.

---

### Finding 6 — Database and reverse proxy are entirely outside this repository's visibility

**Issue:** Per `deploy/README.md`, MariaDB and the reverse proxy (Nginx/Caddy/Traefik) are **required** but run
as separate, external infrastructure connected only via the `jobtracker_shared` external Docker network — their
compose files, resource limits, health checks, and logs are not part of this repository and were not
accessible during this investigation.

**Evidence:**
```yaml
# docker-compose.yml
networks:
  shared_services:
    external: true
    name: jobtracker_shared
```
```markdown
# deploy/README.md
Requirements:
- reverse proxy in front (Nginx, Caddy, or Traefik)
- network connectivity from the backend container to your `mariadb` container/service
```

**Impact:** A real blind spot. If the database or reverse proxy container itself is what's crashing (rather
than `backend`/`frontend`/`ai-service`), nothing in this repo would show it, and the symptoms reported (server
unresponsive, containers stopping) are equally consistent with a database under memory/connection pressure as
with the app containers themselves.

**Confidence:** N/A (this is a coverage gap, not a finding about behavior) — flagged as high-priority to close
in the remediation plan.

---

### Finding 7 — Failed-build cleanup is reactive only, not proactive

**Issue:** `docker builder prune -af` and `docker system prune -f` only run **after a build has already
failed**, as part of the one-shot retry path in `deploy/deploy.sh`. There is no proactive/scheduled cleanup of
dangling images, stopped containers, or build cache between deploys.

**Evidence:** `deploy/deploy.sh`, `build_core_with_recovery()` / `build_ai_with_recovery()` — prune calls exist
only inside the `if compose build ...; then return 0; fi` failure branch.

**Impact:** Each build (successful or not) leaves the previous image's now-dangling layers on disk. Combined
with the documented history of build failures (Finding 3), disk usage from accumulated dangling
images/layers/cache could grow over weeks. A full disk independently causes container crashes and can make a
host appear "unresponsive" (Docker daemon operations stall when the disk backing `/var/lib/docker` is full).

**Confidence:** Medium — real risk pattern, but disk usage itself was not observable from here (would need
`df -h` and `docker system df` on the server).

---

### Finding 8 — `git reset --hard` + `git clean -fd` on every deploy

**Issue:** The deploy step runs `git reset --hard ${{ github.sha }}` followed by `git clean -fd` against
`/opt/job-tracker/app` on every single deploy to `main`.

**Evidence:** `.gitea/workflows/ci-deploy.yml`, `deploy` job, `script:` block.

**Impact:** Not a likely cause of *this* incident, but a standing operational risk: any file manually placed in
that checkout for debugging (a log dump, a temporary config override, a core dump for post-mortem analysis)
that isn't committed is **silently and permanently deleted** on the next push to `main`. If anyone tried to
capture server-side evidence for a previous incident this way, it would not have survived to be reviewed later.
Flagged here because it's directly relevant to *investigability* going forward, even though it's not a root
cause of the crash itself.

**Confidence:** High (structural fact). Not applicable as a root cause of the outage.

---

### Finding 9 — nginx `/api/` proxy has no explicit timeout tuning

**Issue:** `job-tracker-ui/nginx.conf`'s `/api/` `proxy_pass` block sets no `proxy_read_timeout`,
`proxy_connect_timeout`, or `proxy_send_timeout` — nginx defaults (60s) apply.

**Evidence:** Full `nginx.conf` read (10 lines, reproduced above in the investigation transcript).

**Impact:** Low likelihood of being a root cause on its own, but under host memory pressure (Findings 1–4), a
backend under memory/swap pressure could respond slowly enough to hit this default timeout, surfacing as
intermittent 504s to users — a plausible explanation for "pushes occasionally fail" if that phrase also
describes user-facing request failures around deploy time, not just `git push`/CI failures. Worth tuning
regardless, low effort.

**Confidence:** Low as a root cause; Medium as a contributing/compounding factor.

---

## Root Cause Analysis (ranked)

### 1. Host-wide systemd D-Bus/session-bus degradation — **HIGH confidence, directly observed live**
**Why it's likely:** Not inferred — directly observed in the live journal during this investigation, recurring
every ~90 seconds to 2 minutes across three independent subsystems (Docker's cgroup-scope creation, sshd/logind
session-scope creation, journald's own watchdog heartbeat) that all fail with the same underlying D-Bus timeout
or a broken-socket (`ENOTCONN`) error. First occurrence in the retained journal dates back to `Apr 29 23:25:52`
— an intermittent, ~2.5-month-old condition, not something introduced by the latest deploy.
**Supporting evidence:** See "Primary Finding" above in full — direct log quotes, `docker inspect` output
(`deunhealth` at 389 restarts), `docker compose ls -a` showing 37 containers across 7 Compose projects on one
host, and confirmation that the Gitea Actions CI runner from Phase 1 is this same box.
**Alternative explanations:** Genuine host memory/CPU exhaustion (Phase 1's original top hypothesis) is
directly contradicted by live data — 31GB RAM with 23GB free, load average 0.09-0.27, 8GB of completely unused
swap. A kernel/hardware fault is possible but unconfirmed (would need `dmesg`/root `journalctl -k`, not
reachable non-interactively as the unprivileged `pi` user in this session). The high container/cron/CI churn on
this box (Finding 3 below, restated) is the most likely *amplifier*, not necessarily the sole trigger — a
D-Bus/dbus-broker bug triggered or worsened by sustained high transient-unit creation/deletion rate is a
well-known class of systemd issue on some version combinations.

### 2. High container and deploy churn amplifying #1 — **MEDIUM-HIGH confidence**
**Why it's likely:** 37 containers across 7 Compose projects (including a dead, stale, seemingly-forgotten
second `jobtracker` project at `/home/pi/jobtracker/JobTracker/`, and what look like two overlapping media-stack
definitions — `docker` and `mediaserver`), two other projects' cron jobs auto-deploying **every 1-2 minutes**,
Gitea Actions CI containers spinning up/down on the same host, and at least one container crash-looping 389+
times — all repeatedly hitting the same D-Bus-mediated cgroup-scope machinery Finding 1 shows is unhealthy.
**Supporting evidence:** `docker compose ls -a` output, crontab entries, `deunhealth` restart count — see
Primary Finding above.
**Alternative explanations:** Could be a red herring if the D-Bus issue turns out to be purely a systemd/kernel
bug unrelated to load — but even so, reducing unnecessary churn (stale project cleanup, less frequent cron
polling) is low-risk and worth doing regardless.

### 3. Jobtracker-specific deploy process has no resource governance or real health checks — **HIGH confidence** (contributing to the app-level symptoms, not the host-wide ones)
**Why it's likely:** Independently true regardless of the systemd finding — see Phase 1 Findings 1, 2, 4 below.
Even once the D-Bus issue is fixed, these gaps mean a jobtracker-specific deploy still has less protection than
it should against any *other* future host issue.
**Supporting evidence:** Findings 1, 2, 4 (Phase 1 section).
**Alternative explanations:** None — structural facts about this repo's own deploy config, independent of the
host's health.

### 4. AI service (local model / bundled Ollama) memory baseline — **LOW-MEDIUM confidence, downgraded**
**Why it's likely:** Still a real, undetermined baseline (Finding 5), but live data shows the host isn't
actually memory-constrained overall, so this matters less than Phase 1 assumed.
**Supporting evidence:** Finding 5.
**Alternative explanations:** Could be fully neutralized already if `AI_PROVIDER=gemini`/`groq` is set in prod
— still unconfirmed (would need to read `/opt/job-tracker/shared/.env`, a secrets file, not read during this
investigation on purpose).

### 5. External database/reverse-proxy failure — **LOW confidence, still genuinely unknown**
**Why it's possibly relevant:** These are required, external, and entirely unobserved from this repo
(Finding 6) — Phase 2's server access didn't specifically target them either (the investigation focused on the
host-wide systemd finding once it emerged as the stronger lead). Not ruled out, just not the focus here.
**Supporting evidence:** Finding 6 (absence of evidence, not evidence of absence).
**Alternative explanations:** Downgraded from Phase 1's "could equally be the primary cause" — the systemd
D-Bus finding is now a more direct, evidenced explanation, but this is still worth checking (see remediation
plan step 5).

---

## What I could not verify (needs root/server access)

This Phase 2 session connected as the unprivileged `pi` user; `sudo` requires a password not available
non-interactively over this SSH session, so root-level commands below were not run. These are the highest-value
next steps, all read-only:

1. **`sudo journalctl -u systemd-logind -u dbus -u dbus-broker --since "1 hour ago"`** (or `-u init.scope`) — the D-Bus broker's *own* logs, to find out why it's timing out (crash-and-respawn loop? resource exhaustion? a specific error at the moment of failure?). This is the single highest-value next command — it would very likely name the exact bug/resource involved.
2. **`sudo dmesg -T | grep -iE "oom|killed process|dbus|systemd"`** — kernel-level log; rules in/out an OOM or hardware event coinciding with the D-Bus failures, and catches anything below systemd's own visibility.
3. **`systemctl status dbus.service dbus-broker.service systemd-logind.service --no-pager`** (root) — current state/restart count of the D-Bus broker itself.
4. **`sudo journalctl --since "2026-04-29 23:00" --until "2026-04-29 23:30" --no-pager`** — full system log around the *first* occurrence, to see if something changed on the host that day (an update, a config change, a specific event) that could have introduced this.
5. Confirm whether `/opt/job-tracker/app` and the stale `/home/pi/jobtracker/JobTracker/` project are both still wired to receive deploys, or whether the latter is genuinely dead and safe to remove (removing it is a *fix*, out of scope for this investigation, but confirming its status is read-only).
6. Contents of `/opt/job-tracker/shared/.env` (specifically `AI_PROVIDER`) — resolves Finding 5. **Handle as a secret; do not paste it into any shared report or chat.**
7. `docker compose logs --tail=500` for the external reverse-proxy / MariaDB stacks (Finding 6) — still an open gap.
8. Whether `deunhealth`'s 389 restarts share a common trigger (check `docker inspect deunhealth --format '{{.State.Health}}'` and whatever container(s) it's configured to watch) — is it crash-looping *because of* the D-Bus issue, or is something else making it (and by extension the containers it watches) unhealthy in the first place?

---

## Recommended Fixes

*(Not implemented — investigation only, per instructions.)*

| # | Fix | Effort | Risk | Expected impact |
|---|---|---|---|---|
| 0 | Root-cause the systemd D-Bus/session-bus degradation itself — start with `sudo journalctl -u dbus -u systemd-logind` and `sudo dmesg` around a live failure window (see "What I could not verify" #1-2) | Low (diagnosis) to Medium (fix, depends what's found — could be a systemd/dbus-broker package update, a resource limit like `DefaultTasksMax`, or a kernel issue) | Depends on what's found | **Highest** — this is the confirmed, live, host-wide root cause; every other fix in this table is secondary to actually understanding and fixing this one |
| 0b | Reduce unnecessary container/deploy churn: remove the dead `/home/pi/jobtracker/JobTracker/` project, reconcile the apparent `docker` vs `mediaserver` stack duplication, investigate why `deunhealth` has restarted 389+ times, and consider whether two separate per-minute cron autodeploys are necessary at that frequency | Low-Medium (mostly cleanup + one investigation) | Low (removing a confirmed-dead project), Medium (touching the live media stacks — verify which is actually in use before removing either) | Medium-High — reduces load on the already-struggling D-Bus mechanism while #0 is being root-caused |
| 1 | Add `mem_limit`/`cpus` (or `deploy.resources.limits`) to every service in `docker-compose.yml`, sized to the actual host specs (now known: 31GB RAM, 6 cores) | Low (config only) | Low | Medium — good practice regardless, but less urgent than Phase 1 assumed now that the host isn't actually memory-constrained |
| 2 | Add a proper `HEALTHCHECK` to `backend` (e.g. `curl`/`wget` against a lightweight `/health` or `/api/auth/config` endpoint) and `frontend`, and change `deploy.sh`'s post-deploy gate to check `.State.Health.Status` the same way it already does for `ai-service` | Low-Medium | Low | High — closes Finding 2, makes "hung but running" detectable and gates deploys on it |
| 3 | Move the image build off the production host entirely — build in CI (the Gitea runner already builds/tests everything else) and push to a registry; `deploy.sh` then only pulls + recreates, never compiles | Medium (CI/registry setup) | Low-Medium (new moving part: a registry) | High — removes Finding 4's concurrent build-vs-serve resource spike entirely, the single biggest structural risk found |
| 4 | Confirm and, if needed, set `AI_PROVIDER=gemini`/`groq` in the production `.env`, and confirm the `bundled-ollama` Compose profile is not active unless intentionally needed | Low (config only, once confirmed) | Low | Medium — removes a large, constant RAM baseline if it's currently active locally |
| 5 | Add proactive `docker image prune` / `docker builder prune` on a schedule (not just the failure-retry path) | Low | Low | Medium — prevents slow disk creep between incidents |
| 6 | Add `proxy_read_timeout`/`proxy_connect_timeout` tuning to `nginx.conf`'s `/api/` block, sized to the AI-backed endpoints' realistic worst case | Low | Low | Low-Medium — smooths one symptom, doesn't address root cause |
| 7 | Get visibility into the external MariaDB and reverse-proxy stacks (at minimum: their `docker-compose.yml`, health checks, and log access) | Low (access/documentation only) | None | High — closes Finding 6, the largest blind spot in this investigation |
| 8 | Consider a lock/guard so a second deploy can't start while one is still in-flight, and/or serialize the CI runner's own workload if it shares the host | Low-Medium | Low | Medium — prevents overlapping deploys from compounding the resource spike in Finding 4 |

---

## Step-by-Step Remediation Plan (priority order)

1. **Get root-level D-Bus/logind logs during or right after a live failure** (gap #1: `sudo journalctl -u dbus
   -u systemd-logind -u docker --since "1 hour ago"`, gap #2: `sudo dmesg -T`). This is now the single
   highest-value next action — it directly targets the confirmed root cause instead of a hypothesis, and would
   very likely name the exact bug or exhausted resource within minutes.
2. **Check the D-Bus broker's own restart history** (`systemctl status dbus.service dbus-broker.service
   systemd-logind.service`, root) — has it crashed and restarted itself repeatedly? That alone would explain
   the intermittent-not-constant pattern observed.
3. **Investigate the `deunhealth` 389-restart loop specifically** (gap #8) — is it the *cause* of D-Bus churn,
   a *victim* of it, or unrelated? Check what container(s) it watches and why they're (or it is) unhealthy.
4. **Clean up confirmed-dead/duplicate state** (fix #0b): the stale `/home/pi/jobtracker/JobTracker/` Compose
   project, and reconcile whichever of `docker`/`mediaserver` is the real active media stack vs. an old
   duplicate — reduces unnecessary churn while step 1-2 are investigated. Low-risk for the confirmed-dead
   project; verify carefully before touching either live media stack.
5. **Consider whether the two per-minute cron autodeploys need to run that frequently** — every 1-2 minutes is
   aggressive for auto-deploy polling and adds constant container churn; even relaxing to every 5 minutes
   meaningfully reduces load on the struggling D-Bus mechanism with no real functional cost.
6. Once 1-3 identify the actual systemd/D-Bus root cause: **apply the appropriate fix** (likely one of: a
   systemd/dbus-broker package update, adjusting a systemd resource limit such as `DefaultTasksMax` or an fd/
   inotify limit, or a kernel update) — the exact fix depends entirely on what step 1 finds, which is why it's
   sequenced first.
7. **Independently, and regardless of the systemd finding:** add real health checks to jobtracker's `backend`/
   `frontend` (fix #2) and per-container resource limits (fix #1) — good practice, safe to do in parallel with
   1-6, not blocked by them.
8. **Get read access to the external reverse-proxy and MariaDB stacks** (Finding 6) — still an open gap,
   lower priority now than the confirmed systemd finding but worth closing for completeness.
9. **Monitor** `journalctl -u dbus -u systemd-logind` for recurrence over the following week after step 6 lands,
   before considering the incident closed — this class of intermittent issue (present since April per the log)
   needs sustained observation, not a single clean hour, to confirm it's actually fixed.
10. **Validate** by watching a live deploy and a container crash/restart scenario post-fix, confirming both
    complete cleanly without hitting the D-Bus timeout.

---

## Quick Wins

Low effort, likely to help immediately, safe to do without deeper investigation first:

- Diagnostic steps 1-3 above (root `journalctl`/`dmesg`, D-Bus broker status, `deunhealth` investigation) — cost
  nothing, read-only, and are now the single highest-value next actions given the confirmed live evidence.
- Removing the confirmed-dead `/home/pi/jobtracker/JobTracker/` Compose project — zero functional risk (it's
  already `dead(1)`, not serving anything), reduces one small piece of the churn.
- Relaxing the two per-minute cron autodeploy schedules to something less aggressive (e.g. every 5 minutes) —
  a one-line change, no functional downside, reduces constant container churn.
- Fix #2 (health checks on `backend`/`frontend`) — small, well-understood change, still worth doing regardless
  of the systemd root cause.
- Fix #6 (nginx timeout tuning) — a few lines, no behavioral risk.

---

## Long-Term Improvements

- **Monitoring & alerting:** none observed anywhere in this repo or the deploy pipeline. At minimum, host-level
  memory/disk/CPU alerting (even a simple cron + webhook) would have caught this class of issue before it
  reached "server unresponsive."
- **Centralized logging:** container logs currently only exist via `docker compose logs` on the host itself and
  are lost on container recreation unless captured. Shipping logs off-host (even to a simple log file rotated
  and retained) would preserve evidence across incidents like this one.
- **Resource limits as standard practice:** every service should ship with `mem_limit`/`cpus` from day one going
  forward, not retrofitted only after an incident.
- **Health checks as standard practice:** every service that serves traffic should have one; `ai-service`/
  `ollama` already show the team knows this pattern, it just wasn't applied to `backend`/`frontend`.
- **CI-built images + registry:** decouples "build" resource cost from "serve" resource availability entirely —
  the single biggest structural improvement available (fix #3 above).
- **Backups:** not in scope of this investigation (no evidence reviewed either way), but worth confirming
  separately given `jobtracker_data` is a named volume with no visible backup step in `deploy.sh`.
- **Deployment strategy:** consider blue-green or at least a "build first, verify, then cut over" pattern rather
  than `--force-recreate` in place — reduces the window where a broken new container replaces a working old one
  with no rollback path.
- **Container management:** a simple `docker system prune` schedule plus disk-usage alerting closes Finding 7
  permanently rather than reactively.
- **Host consolidation review:** this one box runs at least 7 unrelated Compose projects (jobtracker, a media
  stack under at least two different names, inboxintel, mealie, resumesite) plus a self-hosted Gitea Actions
  runner, with per-minute cron autodeploys for two of them. Worth a deliberate review of whether all of this
  belongs on one host, and whether the CI runner specifically should be isolated from the boxes it deploys to
  (a CI job's own container churn shouldn't be able to affect production media/app containers on the same
  D-Bus/systemd instance).
- **systemd/D-Bus health monitoring specifically:** given this issue has been intermittently present since
  April, a simple periodic check (e.g. `systemctl is-system-running` or a canary `systemd-run` test) with
  alerting would have surfaced this months before it escalated to "server unresponsive."

---

## Working Rules Confirmed

No services were restarted. No configuration was modified. No containers, images, volumes, or networks were
deleted or pruned. No packages were installed. No fixes were applied. Every command run over SSH against
`pi@192.168.50.13` (Phase 2) was read-only: `journalctl`, `docker ps`/`inspect`/`compose ls`, `free`, `df`,
`uptime`, `last reboot`, `crontab -l`, `cat /etc/fstab`, `ps`. Several `sudo`-prefixed commands were attempted
and failed (no password available non-interactively) — those failures are themselves reported as an open gap,
not worked around. This report and the git history/config files/live logs read to produce it are the only
outputs of this investigation.