jobtrackingapp/docs/_archive/infrastructure-investigation.md

Infrastructure Investigation — Deployment Failure (Server Unresponsive, Containers Stopped)

Date: 2026-07-13 Trigger: Latest deployment failed; server became unresponsive; some Docker containers stopped unexpectedly; pushes occasionally fail; deployment occasionally hangs. Scope later widened by the user: the same host also runs Jellyfin/Radarr/Sonarr and other unrelated services, which show the identical symptom (containers not coming back after reboot despite restart: always/unless-stopped) — confirming this is a host-wide problem, not specific to the jobtracker app. Mode: Investigation only. No changes, restarts, deletions, prunes, installs, or fixes were made. No configuration was modified. All commands run over SSH were read-only (journalctl, docker ps/inspect, free, df, uptime, crontab -l, cat) — nothing was written, restarted, or deleted on the server.

---

Method note — two phases of this investigation

Phase 1 (static, no server access): initial pass was done entirely from the local development checkout, without live access to the deployment host — its SSH credentials are Gitea Actions secrets not present on this machine. That phase built a hypothesis from docker-compose.yml, Dockerfiles, deploy/deploy.sh, the CI workflow, and git history alone (see "Phase 1 findings" below — still valid as contributing/secondary factors).

Phase 2 (live, with server access): the user then provided direct SSH access (pi@192.168.50.13) and noted the problem isn't limited to this app. This phase found the actual, currently-reproducing root cause with timestamped live evidence (kernel/systemd journal, docker inspect, docker ps -a across every Compose stack on the box) — see "Primary Finding — systemd D-Bus/session-bus degradation (host-wide)" immediately below. This supersedes Phase 1's top-ranked hypothesis (host-level resource exhaustion): the box has 31GB RAM (23GB free), 6 cores at ~10-25% load, and 8GB of completely unused swap — it is not resource-starved. Phase 1's findings about the jobtracker deploy process specifically (no health checks, building images on the live host, etc.) remain valid and worth fixing, but they are not the primary cause of the reported outage.

---

Primary Finding — systemd D-Bus/session-bus degradation (host-wide, confirmed live)

Issue: systemd's own D-Bus/session-bus subsystem (org.freedesktop.systemd1) is intermittently failing to respond within its 25-second timeout, host-wide, to every kind of request that needs it — not just Docker. This breaks three unrelated things by the same mechanism:

1. New container creation and container restarts. Docker's systemd cgroup driver asks systemd (over D-Bus) to create a transient .scope unit for every container start/restart. When that call times out, the container fails to start with OCI runtime create failed: ... unable to start unit "docker-<id>.scope" ... Failed to activate service 'org.freedesktop.systemd1': timed out (service_start_timeout=25000ms). This is Docker's own restart-manager logging the exact same error when trying to restart an existing container — i.e. restart: always/unless-stopped cannot save a container if the D-Bus call underneath it is what's failing.
2. New SSH sessions. systemd-logind needs the same D-Bus mechanism to register a session scope for every new login. Live log during this investigation: sshd[...]: pam_systemd(sshd:session): Failed to create session: Connection timed out and systemd-logind[1031]: Failed to start session scope session-6497.scope: Failed to activate service 'org.freedesktop.systemd1': timed out. If this gets worse, it can make the box genuinely un-SSH-able — a very plausible reading of "server becomes unresponsive."
3. systemd-journald's own internal watchdog heartbeat. systemd-journald[432]: Failed to send WATCHDOG=1 notification message: Transport endpoint is not connected — journald cannot even talk to systemd about its own liveness. "Transport endpoint is not connected" (ENOTCONN) is a broken-socket error, not a busy/slow one — consistent with systemd's D-Bus broker itself being unhealthy, not merely overloaded.

Evidence (live, timestamped, gathered 2026-07-13 ~15:30-15:46 local server time):

Jul 13 14:53:35 mediaserver dockerd[1326]: ... restartmanger wait error: failed to create task for container:
  ... Failed to activate service 'org.freedesktop.systemd1': timed out (service_start_timeout=25000ms)
Jul 13 15:37:01 mediaserver systemd-journald[432]: Failed to send WATCHDOG=1 notification message:
  Transport endpoint is not connected
Jul 13 15:38:09 mediaserver systemd-logind[1031]: Failed to start session scope session-6481.scope:
  Connection timed out
Jul 13 15:38:09 mediaserver sshd[1994140]: pam_systemd(sshd:session): Failed to create session: Connection timed out
Jul 13 15:46:20 mediaserver sshd[2004289]: pam_systemd(sshd:session): Failed to create session:
  Failed to activate service 'org.freedesktop.systemd1': timed out (service_start_timeout=25000ms)
Jul 13 15:46:20 mediaserver systemd-logind[1031]: Failed to start session scope session-6497.scope:
  Failed to activate service 'org.freedesktop.systemd1': timed out (service_start_timeout=25000ms)

The systemd-journald watchdog failure recurred roughly every 90 seconds to 2 minutes throughout the ~10-minute observation window — i.e. this was actively reproducing while the investigation was running, not a rare blip.

docker inspect deunhealth showed RestartCount=389, StartedAt=2026-07-13T12:40:14Z — a container that has been crash-restart-looping 389 times (and counting). One of its restart attempts is directly logged as failing for the same D-Bus-timeout reason (restartmanger wait error ... Failed to activate service 'org.freedesktop.systemd1'). Ironically, deunhealth (qmcgaw/deunhealth) is itself a watchdog tool whose job is auto-restarting other unhealthy containers — so the host's auto-healing mechanism is caught in the same degraded-D-Bus bottleneck it exists to work around.

The very first occurrence of this exact Docker/systemd1 timeout in the journal dates back to Apr 29 23:25:52 — this has been an intermittent problem for roughly 2.5 months, not something new introduced by the latest deployment. Only 7 occurrences matched in Docker's own log across the last 48 hours specifically, but the broader journald-watchdog/logind symptom was firing every ~90 seconds during the live observation window, suggesting either a currently-active flare-up or a much higher background rate than Docker's log alone shows (Docker only logs the timeout when it itself happens to need D-Bus at that exact moment; logind/journald hit it far more often simply because they touch D-Bus more frequently).

Contributing factor (not necessarily the root cause, but plausible amplifier): this host runs 37 containers across 7 separate Docker Compose projects simultaneously — app (jobtracker prod, confirmed to be /opt/job-tracker/app, matching this repo's deploy target exactly), deploy (a "resumesite" project), docker (the main Jellyfin/Radarr/Sonarr media stack, 22 containers), inboxintel, mealie, mediaserver (a second, apparently overlapping media-stack definition), and a dead leftover jobtracker project at a different, stale path (/home/pi/jobtracker/JobTracker/docker-compose.yml) — clearly an old pre-migration deployment location nobody cleaned up. On top of that, two cron jobs auto-deploy other projects every 1-2 minutes:

*/2 * * * * /opt/inboxintel/deploy/auto-deploy.sh
* * * * * /opt/resumesite/deploy/autodeploy.sh

Gitea Actions CI job containers were also observed live in the Docker log (GITEA-ACTIONS-TASK-1083...JOB-deploy, GITEA-ACTIONS-TASK-1084...JOB-test) — confirming the self-hosted CI runner from Phase 1's investigation is this same box. Combined with 389+ restart attempts from one crash-looping container alone, this is a lot of transient-unit churn (container creates/destroys, session scopes, per-minute cron-triggered rebuilds) hitting systemd's D-Bus continuously. High churn is a well-known trigger for exactly this class of systemd bug on some Ubuntu/systemd version combinations (the box is on kernel 6.8.0-110-generic) — a D-Bus broker or transient-unit bookkeeping resource (e.g. an internal queue, a leaked file descriptor, an fd/inotify limit) getting exhausted under sustained high creation/deletion rate.

Impact: This single mechanism plausibly explains all four originally-reported symptoms at once, across every app on the host, not just jobtracker:

• "Some Docker containers stop unexpectedly" — a restart attempt loses the D-Bus race and the container simply stays down, non-deterministically (whichever container's restart happens to line up with a bad D-Bus moment).
• "Deployment occasionally hangs" — docker compose up --force-recreate creates new containers, each requiring the same D-Bus call; a deploy can stall or fail here exactly like any other container start.
• "The server becomes unresponsive" — new SSH sessions and journald's own health-check both depend on the same broken mechanism; in a bad enough episode this could make the box very difficult to even log into.
• "Pushes occasionally fail" — consistent with a CI job container (visible running on this same host) hitting the same timeout during its own container lifecycle.

Confidence: High — this is not inferred from static config or git history, it is directly observed, timestamped, live log evidence, reproducing during the investigation itself, across three independent subsystems (Docker, sshd/logind, journald) that all point to the same underlying D-Bus failure.

What would raise this from High to Confirmed: root-level journalctl access (this investigation ran as the unprivileged pi user — sudo requires a password not available non-interactively over this SSH session, so kernel-level (dmesg) OOM/hardware logs and the D-Bus broker's own service logs (systemd-logind.service, dbus.service/dbus-broker.service) were not reachable). See "What I could not verify" for the exact commands to run with root access.

---

Phase 1 findings (static analysis, secondary/contributing — still valid)

These were the findings from before server access was available. They remain accurate descriptions of the jobtracker deploy process's own gaps, and are worth fixing regardless of the primary systemd finding above, but they are not the main explanation for the reported outage.

Overall health assessment (Phase 1): The deployment pipeline and container topology are functionally reasonable (sensible service split, restart: unless-stopped, a health check on the AI service, retry-once logic bolted onto several known-flaky build steps) but have no resource governance anywhere — no per-container CPU/memory limits, no health check on the two services that actually serve user traffic (backend, frontend), and a deploy process that builds new container images on the same host that is simultaneously serving production traffic, with zero headroom reserved for that build. Git history independently documents at least five separate incidents over the last ~8 days (NuGet package-integrity corruption, npm SIGSEGV, .NET SDK tool-cache corruption, and a webpack/Terser build that "died silently... OOM/SIGSEGV signature", per the workflow's own comment) on this same runner/host class, each patched with a "retry once" band-aid rather than a resource-level fix. Note: live data shows the host is not actually memory-constrained (31GB RAM, 23GB free) — these build-tool crashes are more likely a symptom of the same D-Bus/cgroup-scope-creation failures documented above (a build step that needs to spawn a container/process via a cgroup scope hits the same timeout) than of genuine memory exhaustion.

Most likely root cause: Host-level memory/CPU exhaustion during the build phase of deployment, on hardware that has already independently shown itself to be resource-constrained, with no per-container limits to contain a spike and no health checks to catch a hung-but-technically-running container. This is consistent with all four reported symptoms (containers stopping unexpectedly, deployment hanging, server unresponsiveness, and occasional push/CI failures) being the same underlying resource-starvation problem manifesting at different pipeline stages, not four unrelated problems.

Confidence: High for "the host is resource-constrained and deploys build under live load with no limits." Medium for "this is specifically what caused the latest incident" (plausible and consistent with the evidence trail, but not confirmed against that incident's actual logs, which I could not access).

---

Findings

Finding 1 — No CPU/memory limits on any container

Issue: docker-compose.yml defines four services (backend, frontend, ai-service, ollama) and none of them set mem_limit, cpus, or a deploy.resources.limits block. Any single container is free to consume all available host memory.

Evidence:

# docker-compose.yml — repeated across every service, no limits anywhere
backend:
  build: ...
  restart: unless-stopped
  # no mem_limit / cpus / deploy.resources

Confirmed by reading the full file (docker-compose.yml, all four service blocks + the opt-in ollama profile) — no resource stanza appears once in the file.

Impact: On a memory-constrained host, one container growing unbounded (or several building/starting concurrently) triggers the Linux OOM killer. The OOM killer scores all processes on the host, not just the one causing pressure — it can and does kill processes that had nothing to do with the spike. This directly explains "some containers stop unexpectedly" (non-deterministic, not all containers, not the one under load) rather than a clean, predictable failure of the actual offending service.

Confidence: High (structural fact, verified by reading the file in full).

---

Finding 2 — No health check on backend or frontend; only ai-service and ollama have one

Issue: ai-service has a proper healthcheck (HTTP GET /health, 30s interval). ollama has one too (ollama list, 20s interval). backend and frontend — the two services that actually serve the product and its API — have none.

Evidence: Full docker-compose.yml read; healthcheck: appears exactly twice, on ai-service and ollama. deploy/deploy.sh's own post-deploy check confirms the gap — it only asks Docker for the container state, not health:

backend_status="$(compose ps backend --format '{{.State}}' 2>/dev/null | head -n 1 | tr '[:upper:]' '[:lower:]')"
if [ "$backend_status" != "running" ]; then
  echo "Backend service is not healthy after deploy (state: ${backend_status:-unknown})."

State reports "running" the instant the process starts — it says nothing about whether the ASP.NET Core app inside has actually finished startup, bound its port, or is responding. A backend that starts, then hangs during EF Core migration or hits a startup exception loop inside the process without exiting, would report running and pass this check while being completely unresponsive to real traffic.

Impact: Directly explains "deployment occasionally hangs" and false-positive successful deploys — the gate that's supposed to catch a broken backend cannot detect the specific failure mode of "process is alive but not serving," which is exactly what host resource starvation during startup would produce (slow/stuck EF migrations, thread-pool starvation under memory pressure, etc.).

Confidence: High (structural fact; the reasoning about why it matters is inference, not confirmed against the actual incident logs).

---

Finding 3 — Documented history of resource-starvation symptoms on this same host class

Issue: Git history shows a cluster of CI/deploy resilience patches, each independently describing a resource-exhaustion signature, spanning 2026-07-05 through 2026-07-12:

Date	Commit	Symptom described in the commit/comment
07-05	63e0300 / e90835b	NuGet package integrity check failures (NU3008) during dotnet publish on the build host — "transient download corruption on the build host"
07-06	4b38f7c	"npm ci occasionally segfaults on the runner (SIGSEGV/139, a memory/native flake)"
07-11	fc35601 / 35eaef9	actions/setup-dotnet "intermittently leaves a partial extraction in the shared tool-cache (tar: Cannot open: File exists) or corrupts the SDK download"
07-11	4f98195	CRA/webpack build "has repeatedly died silently on this runner with no error output (OOM/SIGSEGV signature...)"
07-12	86cdafb	A different class of bug (missing .npmrc in Docker build context) — not resource-related, included for completeness

Every one of these except the last was patched with retry-once logic, not a resource-level fix. deploy.sh itself carries the same pattern for the production build step, with an explicit comment on both retry paths:

echo "docker compose build for core services failed. Attempting one cleanup + retry
because layer extraction can fail on constrained hosts."

("constrained hosts" — the deploy script's own author already believed this.)

Evidence: git log --oneline --grep=... --since="30 days ago" (see command output used in this investigation); direct commit message and inline-comment quotes above.

Impact: This is the strongest available evidence that the underlying host (CI runner and/or production deploy target — the two may be the same self-hosted machine, common in small-scale setups; I could not confirm either way without server access) has insufficient memory headroom for its build workload, as a standing, recurring condition rather than a one-off.

Confidence: High that the pattern exists and points to memory pressure. Medium on "this is the same host as the one that just failed" — plausible, not confirmed.

---

Finding 4 — Deploy builds new images on the same host serving live traffic, with no isolation

Issue: deploy/deploy.sh runs docker compose build backend frontend (a full dotnet publish inside the mcr.microsoft.com/dotnet/sdk:9.0 image, plus a full npm run build inside node:20-alpine) directly on the deploy target, while the currently-running containers from the previous deploy are still serving production traffic (they aren't stopped until compose up -d --force-recreate afterward). Both build steps are independently already known to be memory-hungry enough to need special handling:

# job-tracker-ui/Dockerfile
shm_size: '1gb'
# Next's build type-checker needs more than Docker's default 64MB /dev/shm; too little
# causes a SIGSEGV during `npm run build`.

Evidence: deploy/deploy.sh full read (build_core_with_recovery, compose build backend frontend before any container is stopped); job-tracker-ui/Dockerfile and its docker-compose.yml shm_size override, with the SIGSEGV comment.

Impact: This is a concurrent resource spike: two separate heavy compiler toolchains (the .NET SDK compiler/publish pipeline and webpack+Terser minification) running back-to-back or overlapping, on a host with no memory limits configured (Finding 1), while that same host is also running the live app for existing users. On already-constrained hardware (Finding 3), this is a very plausible trigger for the host itself becoming unresponsive during a deploy, not just one container failing.

Confidence: High that this is how the deploy works (verified by reading the script). Medium-High that it is a primary contributor to the reported "server becomes unresponsive" symptom specifically during deploys.

---

Finding 5 — AI service memory footprint is opaque and possibly significant

Issue: ai-service loads a distilbart-cnn-12-6 summarization model into RAM via CPU-only PyTorch (--extra-index-url https://download.pytorch.org/whl/cpu in tools/summarizer/Dockerfile), and can optionally also run a bundled ollama service with a qwen2.5:7b model (a 7B-parameter LLM, several GB of RAM/VRAM even quantized) if the bundled-ollama Compose profile is active in production. Neither has a memory limit (Finding 1). Whether the bundled-Ollama profile or an external/cloud AI provider (AI_PROVIDER=gemini/groq, added earlier this session) is actually active in the live .env could not be determined from this repo — that file lives only on the server at /opt/job-tracker/shared/.env, outside version control.

Evidence: tools/summarizer/Dockerfile, tools/summarizer/app.py (SKIP_MODEL_LOAD / EAGER_MODEL_LOAD env vars, torch.device), docker-compose.yml (ollama service, profiles: ["bundled-ollama"], gpus: all), deploy/README.md example .env (shows OLLAMA_MODEL=qwen2.5:7b as the documented default).

Impact: If the local-Ollama path is active rather than a cloud provider, this is a large, constant RAM baseline on top of everything else, on a host already shown to be tight on memory. Cannot be confirmed without reading the live .env and docker stats output on the server.

Confidence: Medium — plausible contributor, not confirmed as active in the current production configuration.

---

Finding 6 — Database and reverse proxy are entirely outside this repository's visibility

Issue: Per deploy/README.md, MariaDB and the reverse proxy (Nginx/Caddy/Traefik) are required but run as separate, external infrastructure connected only via the jobtracker_shared external Docker network — their compose files, resource limits, health checks, and logs are not part of this repository and were not accessible during this investigation.

Evidence:

# docker-compose.yml
networks:
  shared_services:
    external: true
    name: jobtracker_shared

# deploy/README.md
Requirements:
- reverse proxy in front (Nginx, Caddy, or Traefik)
- network connectivity from the backend container to your `mariadb` container/service

Impact: A real blind spot. If the database or reverse proxy container itself is what's crashing (rather than backend/frontend/ai-service), nothing in this repo would show it, and the symptoms reported (server unresponsive, containers stopping) are equally consistent with a database under memory/connection pressure as with the app containers themselves.

Confidence: N/A (this is a coverage gap, not a finding about behavior) — flagged as high-priority to close in the remediation plan.

---

Finding 7 — Failed-build cleanup is reactive only, not proactive

Issue: docker builder prune -af and docker system prune -f only run after a build has already failed, as part of the one-shot retry path in deploy/deploy.sh. There is no proactive/scheduled cleanup of dangling images, stopped containers, or build cache between deploys.

Evidence: deploy/deploy.sh, build_core_with_recovery() / build_ai_with_recovery() — prune calls exist only inside the if compose build ...; then return 0; fi failure branch.

Impact: Each build (successful or not) leaves the previous image's now-dangling layers on disk. Combined with the documented history of build failures (Finding 3), disk usage from accumulated dangling images/layers/cache could grow over weeks. A full disk independently causes container crashes and can make a host appear "unresponsive" (Docker daemon operations stall when the disk backing /var/lib/docker is full).

Confidence: Medium — real risk pattern, but disk usage itself was not observable from here (would need df -h and docker system df on the server).

---

Finding 8 — git reset --hard + git clean -fd on every deploy

Issue: The deploy step runs git reset --hard ${{ github.sha }} followed by git clean -fd against /opt/job-tracker/app on every single deploy to main.

Evidence: .gitea/workflows/ci-deploy.yml, deploy job, script: block.

Impact: Not a likely cause of this incident, but a standing operational risk: any file manually placed in that checkout for debugging (a log dump, a temporary config override, a core dump for post-mortem analysis) that isn't committed is silently and permanently deleted on the next push to main. If anyone tried to capture server-side evidence for a previous incident this way, it would not have survived to be reviewed later. Flagged here because it's directly relevant to investigability going forward, even though it's not a root cause of the crash itself.

Confidence: High (structural fact). Not applicable as a root cause of the outage.

---

Finding 9 — nginx /api/ proxy has no explicit timeout tuning

Issue: job-tracker-ui/nginx.conf's /api/ proxy_pass block sets no proxy_read_timeout, proxy_connect_timeout, or proxy_send_timeout — nginx defaults (60s) apply.

Evidence: Full nginx.conf read (10 lines, reproduced above in the investigation transcript).

Impact: Low likelihood of being a root cause on its own, but under host memory pressure (Findings 1–4), a backend under memory/swap pressure could respond slowly enough to hit this default timeout, surfacing as intermittent 504s to users — a plausible explanation for "pushes occasionally fail" if that phrase also describes user-facing request failures around deploy time, not just git push/CI failures. Worth tuning regardless, low effort.

Confidence: Low as a root cause; Medium as a contributing/compounding factor.

---

Root Cause Analysis (ranked)

1. Host-wide systemd D-Bus/session-bus degradation — HIGH confidence, directly observed live

Why it's likely: Not inferred — directly observed in the live journal during this investigation, recurring every ~90 seconds to 2 minutes across three independent subsystems (Docker's cgroup-scope creation, sshd/logind session-scope creation, journald's own watchdog heartbeat) that all fail with the same underlying D-Bus timeout or a broken-socket (ENOTCONN) error. First occurrence in the retained journal dates back to Apr 29 23:25:52 — an intermittent, ~2.5-month-old condition, not something introduced by the latest deploy. Supporting evidence: See "Primary Finding" above in full — direct log quotes, docker inspect output (deunhealth at 389 restarts), docker compose ls -a showing 37 containers across 7 Compose projects on one host, and confirmation that the Gitea Actions CI runner from Phase 1 is this same box. Alternative explanations: Genuine host memory/CPU exhaustion (Phase 1's original top hypothesis) is directly contradicted by live data — 31GB RAM with 23GB free, load average 0.09-0.27, 8GB of completely unused swap. A kernel/hardware fault is possible but unconfirmed (would need dmesg/root journalctl -k, not reachable non-interactively as the unprivileged pi user in this session). The high container/cron/CI churn on this box (Finding 3 below, restated) is the most likely amplifier, not necessarily the sole trigger — a D-Bus/dbus-broker bug triggered or worsened by sustained high transient-unit creation/deletion rate is a well-known class of systemd issue on some version combinations.

2. High container and deploy churn amplifying #1 — MEDIUM-HIGH confidence

Why it's likely: 37 containers across 7 Compose projects (including a dead, stale, seemingly-forgotten second jobtracker project at /home/pi/jobtracker/JobTracker/, and what look like two overlapping media-stack definitions — docker and mediaserver), two other projects' cron jobs auto-deploying every 1-2 minutes, Gitea Actions CI containers spinning up/down on the same host, and at least one container crash-looping 389+ times — all repeatedly hitting the same D-Bus-mediated cgroup-scope machinery Finding 1 shows is unhealthy. Supporting evidence: docker compose ls -a output, crontab entries, deunhealth restart count — see Primary Finding above. Alternative explanations: Could be a red herring if the D-Bus issue turns out to be purely a systemd/kernel bug unrelated to load — but even so, reducing unnecessary churn (stale project cleanup, less frequent cron polling) is low-risk and worth doing regardless.

3. Jobtracker-specific deploy process has no resource governance or real health checks — HIGH confidence (contributing to the app-level symptoms, not the host-wide ones)

Why it's likely: Independently true regardless of the systemd finding — see Phase 1 Findings 1, 2, 4 below. Even once the D-Bus issue is fixed, these gaps mean a jobtracker-specific deploy still has less protection than it should against any other future host issue. Supporting evidence: Findings 1, 2, 4 (Phase 1 section). Alternative explanations: None — structural facts about this repo's own deploy config, independent of the host's health.

4. AI service (local model / bundled Ollama) memory baseline — LOW-MEDIUM confidence, downgraded

Why it's likely: Still a real, undetermined baseline (Finding 5), but live data shows the host isn't actually memory-constrained overall, so this matters less than Phase 1 assumed. Supporting evidence: Finding 5. Alternative explanations: Could be fully neutralized already if AI_PROVIDER=gemini/groq is set in prod — still unconfirmed (would need to read /opt/job-tracker/shared/.env, a secrets file, not read during this investigation on purpose).

5. External database/reverse-proxy failure — LOW confidence, still genuinely unknown

Why it's possibly relevant: These are required, external, and entirely unobserved from this repo (Finding 6) — Phase 2's server access didn't specifically target them either (the investigation focused on the host-wide systemd finding once it emerged as the stronger lead). Not ruled out, just not the focus here. Supporting evidence: Finding 6 (absence of evidence, not evidence of absence). Alternative explanations: Downgraded from Phase 1's "could equally be the primary cause" — the systemd D-Bus finding is now a more direct, evidenced explanation, but this is still worth checking (see remediation plan step 5).

---

What I could not verify (needs root/server access)

This Phase 2 session connected as the unprivileged pi user; sudo requires a password not available non-interactively over this SSH session, so root-level commands below were not run. These are the highest-value next steps, all read-only:

1. sudo journalctl -u systemd-logind -u dbus -u dbus-broker --since "1 hour ago" (or -u init.scope) — the D-Bus broker's own logs, to find out why it's timing out (crash-and-respawn loop? resource exhaustion? a specific error at the moment of failure?). This is the single highest-value next command — it would very likely name the exact bug/resource involved.
2. sudo dmesg -T | grep -iE "oom|killed process|dbus|systemd" — kernel-level log; rules in/out an OOM or hardware event coinciding with the D-Bus failures, and catches anything below systemd's own visibility.
3. systemctl status dbus.service dbus-broker.service systemd-logind.service --no-pager (root) — current state/restart count of the D-Bus broker itself.
4. sudo journalctl --since "2026-04-29 23:00" --until "2026-04-29 23:30" --no-pager — full system log around the first occurrence, to see if something changed on the host that day (an update, a config change, a specific event) that could have introduced this.
5. Confirm whether /opt/job-tracker/app and the stale /home/pi/jobtracker/JobTracker/ project are both still wired to receive deploys, or whether the latter is genuinely dead and safe to remove (removing it is a fix, out of scope for this investigation, but confirming its status is read-only).
6. Contents of /opt/job-tracker/shared/.env (specifically AI_PROVIDER) — resolves Finding 5. Handle as a secret; do not paste it into any shared report or chat.
7. docker compose logs --tail=500 for the external reverse-proxy / MariaDB stacks (Finding 6) — still an open gap.
8. Whether deunhealth's 389 restarts share a common trigger (check docker inspect deunhealth --format '{{.State.Health}}' and whatever container(s) it's configured to watch) — is it crash-looping because of the D-Bus issue, or is something else making it (and by extension the containers it watches) unhealthy in the first place?

---

Recommended Fixes

(Not implemented — investigation only, per instructions.)

#	Fix	Effort	Risk	Expected impact
0	Root-cause the systemd D-Bus/session-bus degradation itself — start with sudo journalctl -u dbus -u systemd-logind and sudo dmesg around a live failure window (see "What I could not verify" #1-2)	Low (diagnosis) to Medium (fix, depends what's found — could be a systemd/dbus-broker package update, a resource limit like DefaultTasksMax, or a kernel issue)	Depends on what's found	Highest — this is the confirmed, live, host-wide root cause; every other fix in this table is secondary to actually understanding and fixing this one
0b	Reduce unnecessary container/deploy churn: remove the dead /home/pi/jobtracker/JobTracker/ project, reconcile the apparent docker vs mediaserver stack duplication, investigate why deunhealth has restarted 389+ times, and consider whether two separate per-minute cron autodeploys are necessary at that frequency	Low-Medium (mostly cleanup + one investigation)	Low (removing a confirmed-dead project), Medium (touching the live media stacks — verify which is actually in use before removing either)	Medium-High — reduces load on the already-struggling D-Bus mechanism while #0 is being root-caused
1	Add mem_limit/cpus (or deploy.resources.limits) to every service in docker-compose.yml, sized to the actual host specs (now known: 31GB RAM, 6 cores)	Low (config only)	Low	Medium — good practice regardless, but less urgent than Phase 1 assumed now that the host isn't actually memory-constrained
2	Add a proper HEALTHCHECK to backend (e.g. curl/wget against a lightweight /health or /api/auth/config endpoint) and frontend, and change deploy.sh's post-deploy gate to check .State.Health.Status the same way it already does for ai-service	Low-Medium	Low	High — closes Finding 2, makes "hung but running" detectable and gates deploys on it
3	Move the image build off the production host entirely — build in CI (the Gitea runner already builds/tests everything else) and push to a registry; deploy.sh then only pulls + recreates, never compiles	Medium (CI/registry setup)	Low-Medium (new moving part: a registry)	High — removes Finding 4's concurrent build-vs-serve resource spike entirely, the single biggest structural risk found
4	Confirm and, if needed, set AI_PROVIDER=gemini/groq in the production .env, and confirm the bundled-ollama Compose profile is not active unless intentionally needed	Low (config only, once confirmed)	Low	Medium — removes a large, constant RAM baseline if it's currently active locally
5	Add proactive docker image prune / docker builder prune on a schedule (not just the failure-retry path)	Low	Low	Medium — prevents slow disk creep between incidents
6	Add proxy_read_timeout/proxy_connect_timeout tuning to nginx.conf's /api/ block, sized to the AI-backed endpoints' realistic worst case	Low	Low	Low-Medium — smooths one symptom, doesn't address root cause
7	Get visibility into the external MariaDB and reverse-proxy stacks (at minimum: their docker-compose.yml, health checks, and log access)	Low (access/documentation only)	None	High — closes Finding 6, the largest blind spot in this investigation
8	Consider a lock/guard so a second deploy can't start while one is still in-flight, and/or serialize the CI runner's own workload if it shares the host	Low-Medium	Low	Medium — prevents overlapping deploys from compounding the resource spike in Finding 4

---

Step-by-Step Remediation Plan (priority order)

1. Get root-level D-Bus/logind logs during or right after a live failure (gap #1: sudo journalctl -u dbus -u systemd-logind -u docker --since "1 hour ago", gap #2: sudo dmesg -T). This is now the single highest-value next action — it directly targets the confirmed root cause instead of a hypothesis, and would very likely name the exact bug or exhausted resource within minutes.
2. Check the D-Bus broker's own restart history (systemctl status dbus.service dbus-broker.service systemd-logind.service, root) — has it crashed and restarted itself repeatedly? That alone would explain the intermittent-not-constant pattern observed.
3. Investigate the deunhealth 389-restart loop specifically (gap #8) — is it the cause of D-Bus churn, a victim of it, or unrelated? Check what container(s) it watches and why they're (or it is) unhealthy.
4. Clean up confirmed-dead/duplicate state (fix #0b): the stale /home/pi/jobtracker/JobTracker/ Compose project, and reconcile whichever of docker/mediaserver is the real active media stack vs. an old duplicate — reduces unnecessary churn while step 1-2 are investigated. Low-risk for the confirmed-dead project; verify carefully before touching either live media stack.
5. Consider whether the two per-minute cron autodeploys need to run that frequently — every 1-2 minutes is aggressive for auto-deploy polling and adds constant container churn; even relaxing to every 5 minutes meaningfully reduces load on the struggling D-Bus mechanism with no real functional cost.
6. Once 1-3 identify the actual systemd/D-Bus root cause: apply the appropriate fix (likely one of: a systemd/dbus-broker package update, adjusting a systemd resource limit such as DefaultTasksMax or an fd/ inotify limit, or a kernel update) — the exact fix depends entirely on what step 1 finds, which is why it's sequenced first.
7. Independently, and regardless of the systemd finding: add real health checks to jobtracker's backend/ frontend (fix #2) and per-container resource limits (fix #1) — good practice, safe to do in parallel with 1-6, not blocked by them.
8. Get read access to the external reverse-proxy and MariaDB stacks (Finding 6) — still an open gap, lower priority now than the confirmed systemd finding but worth closing for completeness.
9. Monitor journalctl -u dbus -u systemd-logind for recurrence over the following week after step 6 lands, before considering the incident closed — this class of intermittent issue (present since April per the log) needs sustained observation, not a single clean hour, to confirm it's actually fixed.
10. Validate by watching a live deploy and a container crash/restart scenario post-fix, confirming both complete cleanly without hitting the D-Bus timeout.

---

Quick Wins

Low effort, likely to help immediately, safe to do without deeper investigation first:

• Diagnostic steps 1-3 above (root journalctl/dmesg, D-Bus broker status, deunhealth investigation) — cost nothing, read-only, and are now the single highest-value next actions given the confirmed live evidence.
• Removing the confirmed-dead /home/pi/jobtracker/JobTracker/ Compose project — zero functional risk (it's already dead(1), not serving anything), reduces one small piece of the churn.
• Relaxing the two per-minute cron autodeploy schedules to something less aggressive (e.g. every 5 minutes) — a one-line change, no functional downside, reduces constant container churn.
• Fix #2 (health checks on backend/frontend) — small, well-understood change, still worth doing regardless of the systemd root cause.
• Fix #6 (nginx timeout tuning) — a few lines, no behavioral risk.

---

Long-Term Improvements

• Monitoring & alerting: none observed anywhere in this repo or the deploy pipeline. At minimum, host-level memory/disk/CPU alerting (even a simple cron + webhook) would have caught this class of issue before it reached "server unresponsive."
• Centralized logging: container logs currently only exist via docker compose logs on the host itself and are lost on container recreation unless captured. Shipping logs off-host (even to a simple log file rotated and retained) would preserve evidence across incidents like this one.
• Resource limits as standard practice: every service should ship with mem_limit/cpus from day one going forward, not retrofitted only after an incident.
• Health checks as standard practice: every service that serves traffic should have one; ai-service/ ollama already show the team knows this pattern, it just wasn't applied to backend/frontend.
• CI-built images + registry: decouples "build" resource cost from "serve" resource availability entirely — the single biggest structural improvement available (fix #3 above).
• Backups: not in scope of this investigation (no evidence reviewed either way), but worth confirming separately given jobtracker_data is a named volume with no visible backup step in deploy.sh.
• Deployment strategy: consider blue-green or at least a "build first, verify, then cut over" pattern rather than --force-recreate in place — reduces the window where a broken new container replaces a working old one with no rollback path.
• Container management: a simple docker system prune schedule plus disk-usage alerting closes Finding 7 permanently rather than reactively.
• Host consolidation review: this one box runs at least 7 unrelated Compose projects (jobtracker, a media stack under at least two different names, inboxintel, mealie, resumesite) plus a self-hosted Gitea Actions runner, with per-minute cron autodeploys for two of them. Worth a deliberate review of whether all of this belongs on one host, and whether the CI runner specifically should be isolated from the boxes it deploys to (a CI job's own container churn shouldn't be able to affect production media/app containers on the same D-Bus/systemd instance).
• systemd/D-Bus health monitoring specifically: given this issue has been intermittently present since April, a simple periodic check (e.g. systemctl is-system-running or a canary systemd-run test) with alerting would have surfaced this months before it escalated to "server unresponsive."

---

Working Rules Confirmed

No services were restarted. No configuration was modified. No containers, images, volumes, or networks were deleted or pruned. No packages were installed. No fixes were applied. Every command run over SSH against pi@192.168.50.13 (Phase 2) was read-only: journalctl, docker ps/inspect/compose ls, free, df, uptime, last reboot, crontab -l, cat /etc/fstab, ps. Several sudo-prefixed commands were attempted and failed (no password available non-interactively) — those failures are themselves reported as an open gap, not worked around. This report and the git history/config files/live logs read to produce it are the only outputs of this investigation.