jobtrackingapp/docs/production/production-ai-hardware-assessment.md

# Production AI hardware and runtime assessment

Updated: 2026-08-15

Status: read-only production inventory complete. No service, file, firewall, model, database, backup, container, or configuration value was changed. Host identity, addresses, credentials, environment secrets, logs, prompts, and private application content are intentionally omitted.

## Executive result

The remembered hardware profile is accurate: the host has about 32 GiB RAM and one NVIDIA GeForce GTX 1060 6GB. The machine has ample capacity on the Docker/model filesystem for bounded sequential benchmarks, but rollout is not safe yet:

- the JobTracker Ollama container publishes port 11434 on every IPv4 and IPv6 host interface;
- the deployed application is older than the release branch and still selects Gemini directly in the AI sidecar rather than the release branch's disabled-external, local-first policy;
- no CPU, memory, PID, or read-only-root limits are applied to the four application containers;
- database backups are small and gzip-valid, but the newest observed file is 13 days old and no JobTracker timer/cron entry was found;
- observed backups cover MariaDB only, not the owner-file volume, data-protection keys, protected deletion tombstones, or non-secret recovery configuration.

These are rollout stop conditions, not permission to change production.

## Measured host

| Area | Read-only measurement |
|---|---|
| OS | Ubuntu 24.04.4 LTS, x86_64 |
| Kernel | Linux 6.8.0-110-generic |
| Time zone | Europe/Oslo |
| Uptime | 3 weeks 6 days at capture |
| Load | 3.25 / 2.80 / 2.29 |
| CPU | Intel Core i5-8600 @ 3.10 GHz; 1 socket, 6 physical/logical cores, 1 thread/core |
| RAM | 31 GiB total; 11 GiB used; 20 GiB available |
| Swap | 8 GiB total; 2.1 GiB used |
| Memory pressure | PSI `some` avg10 0.16%; `full` avg10 0.06% |
| Shell open-file limit | 1,024 |
| Root filesystem | ext4, 217 GiB total, 171 GiB used, 36 GiB available (83% used) |
| Docker/model filesystem | ext4, 1.8 TiB total, 316 GiB used, 1.4 TiB available (19% used) |
| Docker | Engine 29.3.0, API 1.54; data root on the large filesystem |

Other workloads share the host. The available-memory and pressure measurements are therefore more useful than total RAM alone; benchmarks must capture concurrent load rather than assuming an idle dedicated server.

## GPU

| Area | Read-only measurement |
|---|---|
| GPU | NVIDIA GeForce GTX 1060 6GB |
| VRAM | 6,144 MiB total; 3 MiB used; 6,064 MiB reported free at capture |
| Driver | 580.159.03 |
| CUDA compatibility reported by driver | 13.0 |
| Idle state | P8, 37 C, 0% utilization, about 5.5 W / 120 W |
| GPU processes | None at capture |
| Ollama GPU device request | All GPUs requested by the Ollama container |
| AI sidecar GPU device request | None; health reports CPU and `gpu_available=false` |

Ollama was not serving a loaded model during capture, so GPU-layer offload is not yet proven. A bounded benchmark must verify the `PROCESSOR`/offload result while a request is active; idle `nvidia-smi` is not evidence of successful GPU inference.

## Ollama and models

- Deployment method: Docker Compose, `ollama/ollama:latest`.
- Runtime version: 0.31.1.
- Captured image digest: `sha256:f1a705f2bd113fb8d15f85f7c217f0dc5f6bebda6b0cc42b82c3ad165ffcb9dc`.
- Installed JobTracker model: `qwen2.5:7b`, model ID prefix `845dbda0ea48`, 4.7 GB.
- Model volume use: about 4.4 GiB.
- Loaded models: none at capture.
- Health: Ollama reachable from the AI sidecar; model present; container healthy with zero observed restarts since 2026-08-02.
- GPU use: not active at capture.

### Exposure finding

`OLLAMA_HOST=0.0.0.0:11434`, Docker publishes 11434 on all IPv4/IPv6 interfaces, and the container joins the private AI network plus broader application/shared networks. A second, non-JobTracker Ollama listener also exists on host port 11435. Internet/NAT reachability was not tested, but all-interface host publication already fails the intended localhost/private-network-only contract.

The release branch removes JobTracker's host publication and shared/default network membership for its bundled Ollama. Deployment must still decide whether to use that private bundled instance or a separately controlled shared instance; it must not leave an orphaned published container.

## Application deployment

| Component | Runtime state | Point-in-time usage |
|---|---|---|
| Backend | healthy, zero restarts | 0.28% CPU; 216 MiB RAM; 25 PIDs |
| Frontend/nginx | healthy, zero restarts | 0% CPU; 7.6 MiB RAM; 7 PIDs |
| AI sidecar | healthy, zero restarts | 0.07% CPU; 2.24 GiB RAM; 30 PIDs |
| JobTracker Ollama | healthy, zero restarts | 0.08% CPU; 4.56 GiB container-accounted RAM; 13 PIDs |

- Deployed commit: `de937d25dc5e`; configured app version: `157`.
- The deployed repository is on `main` with a mode-only local change to `deploy/deploy.sh`; do not reset or overwrite it without operator review.
- All four containers use `unless-stopped`, rotating `json-file` logs at 10 MiB x 3.
- All four have unlimited CPU/memory/PIDs, writable root filesystems, and are not privileged.
- Backend and frontend share the normal application network; backend and sidecar share the AI network. The deployed Ollama additionally joins broader networks.
- The sidecar is internal-only on port 8001 and requires the shared service token for non-health endpoints.
- The deployed frontend publishes host port 3000. Backend and sidecar are not published by this Compose project. The host also has an unrelated listener on 8080.
- nginx has no explicit connect/read/send proxy timeout in the deployed file, so defaults apply.

## Current AI behavior

The deployed sidecar is older than the release branch:

- it loads `sshleifer/distilbart-cnn-12-6` on CPU for summarization;
- it has Ollama configured with `qwen2.5:7b`, but no Ollama model was loaded at capture;
- `AI_PROVIDER=gemini`, with Gemini and Groq credential variable names present (values were not read);
- deployed code selects that provider directly and does not yet expose the release branch's `EXTERNAL_AI_ENABLED` / `AI_ROUTING_MODE` controls;
- the backend's new durable AI worker settings are absent from the old deployment, so the release-branch defaults must be reviewed during deployment rather than inferred from this runtime.

Production is therefore not currently evidence for the release branch's local-first routing, durable queue, privacy gate, or entitlement behavior.

## Storage and backups

- JobTracker owner-file volume: about 14 MiB total at capture.
- Visible categories: attachments 4 KiB, CV artifacts 1.1 MiB, generated CVs 16 KiB, daily exports 196 KiB.
- The dark-launch account-export and deletion-tombstone directories are absent because their release has not been deployed.
- The MariaDB data directory and Docker volumes are on the large filesystem; the SSH user cannot read their host-level sizes without privileged access.
- `/opt/job-tracker/backups` contains 21 MariaDB `.sql.gz` files, about 5.2 MB combined. All passed `gzip -t`.
- Observed range: 2026-07-19 through 2026-08-02. No newer dump and no JobTracker timer/current-user cron entry were observed.
- No application-volume, attachment/CV, data-protection-key, tombstone, or configuration bundle was present in that backup directory.

This proves only that existing compressed dump files are structurally readable. It does not prove logical restore, row counts, non-ASCII fidelity, current RPO, or complete disaster recovery.

## Monitoring and evidence gaps

Available today: Docker health status, restart count, container stats, bounded container logs, sidecar health, `ollama list`, `ollama ps`, and host/GPU metrics. Missing: durable metrics/history, alerting tied to JobTracker SLOs, queue depth in this old deployment, inference latency/throughput history, active GPU-offload evidence, scheduled backup evidence, and current restore proof.

No production logs were read because they may contain prompts, paths, identifiers, or private content.

## Safe conclusion

The host can support a cautious one-model-at-a-time synthetic benchmark. Do not install or load a candidate, enable workers, enable external fallback, or deploy until the all-interface Ollama exposure, stale/incomplete backups, dirty deployment script, release-version gap, and rollback prerequisites in `production-ai-rollout-and-rollback.md` are resolved.