Files
jobtrackingapp/docs/audits/2026-09-07/validation/b06-completion.md
T
cesnimda 115ecadd4d
CI and Deploy / test (push) Failing after 4m12s
CI and Deploy / deploy (push) Has been skipped
docs(deploy): record release checks and runner failure
2026-09-09 19:29:41 +02:00

329 lines
48 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# B06 — Ingress boundary implementation and verification record
**Latest runtime follow-up:** [2026-09-09 observations and remaining gates](#runtime-verification-follow-up-2026-09-09). Public DNS/TLS and bounded direct-access observations are now available. Docker still cannot start; real container/Traefik verification remains blocked. **B06 implementation complete, but F04 remains open.** The initial implementation-pass record below is preserved chronologically; statements about no public requests/DNS/TLS apply to that earlier pass.
Date: **2026-09-09**. **Repository/application changes implemented; full local acceptance and production verification are NOT complete.** B01–B05 regression coverage passes. B07 was not started. No deployment, host firewall change, schema change, external scan, image pull or Docker reset was performed.
Two gates remain: (1) execute the real nginx/container topology tests with a working Docker engine and the intended image; (2) establish the deployed Traefik socket peers, forwarding configuration and host/provider network controls. **F04 remains open, conditional P1.** This file is a completion record for work performed, not a claim that the remaining gates passed.
## Finding and independently established boundary
The original [F04 validation](README.md#f04--forwarded-headers-and-direct-ingress) and [B04 deferral](b04-completion.md) remain historical evidence. The vulnerability is **not a production API host-port mapping**: rendered base Compose has no backend published port. It publishes frontend/nginx **host 3000 → container 80**, without a host-IP restriction. Nginx accepts the canonical Host and previously passed arbitrary incoming `X-Forwarded-For` / `X-Forwarded-Proto` to the API. The API legitimately trusts nginx's dedicated network, so bypassing the external sanitizing proxy could launder attacker-supplied headers through a trusted peer. A canonical Host is not proof of a trusted caller.
Before changing application/configuration code, `docker compose --env-file .artifacts/b06/compose.env -f docker-compose.yml --profile bundled-ollama config --format json` confirmed this exposure using synthetic values. The new permanent effective-config test failed because production had no upstream-peer enforcement setting. Source tracing confirmed verbatim forwarding in all three nginx proxy locations. **No pre-fix live nginx exploit was reproduced in this environment**, because the Docker engine was unavailable. The optional `--baseline` topology probe can test HEAD's original nginx sources before these edits are committed. It is not a substitute for an immutable baseline after committing; original source excerpts/observations remain in the audit.
Deployment evidence conflicts: `docs/architecture/deployment.md` asserted a firewall blocked port 3000; `deploy/README.md` claimed no frontend host mapping and container-direct Traefik routing; `sec-002-ingress-compose.md` retained a host-port compatibility dependency while also listing a contradictory “production ports absent” check. Current rendered configuration and deployment scripts take precedence. These claims are now corrected in active documentation. The old verification record is retained as historical evidence, not current proof.
## Production-intended topology and actual configuration evidence
```text
Internet → DNS/provider network/host firewall [unverified]
→ operator-owned Traefik [TLS termination/routing/config unverified]
→ Docker host TCP 3000 [current compatibility mapping]
→ frontend nginx TCP 80 [new explicit socket-peer allowlist]
→ dedicated internal web_proxy network
→ ASP.NET API TCP 8080 [unpublished]
```
The repository establishes the expected contract, **not the observed current production topology**. No Traefik configuration, live `.env`, DNS, firewall rules or infrastructure credentials were available/used. A request for the real Traefik placement/observed peer addresses remains unanswered; “continue” was not treated as a network value or production approval.
| Surface | Rendered production configuration | After B06 | Evidence level |
|---|---|---|---|
| Frontend nginx | TCP 3000 → 80, host IP unspecified; Docker normally publishes broadly, platform IPv6 behavior must be checked | Mapping preserved; application requests require configured Traefik socket peers, single client IP and HTTPS metadata | Mapping proven by Compose; nginx enforcement source/generation checked, runtime pending |
| API | `expose: 8080`, no `ports`; listens on container interfaces | Same exposure; explicit `ASPNETCORE_ENVIRONMENT=Production` | Proven in rendered configuration |
| `web_proxy` | Dedicated bridge, `internal: true`, configured `WEB_PROXY_SUBNET` (synthetic/default /29) | Only frontend/backend members permitted by preflight; API known network must match exactly | Proven in configuration; live membership pending |
| `jobtracker_shared` | External network; frontend and backend both join | Preserved for existing operator-managed service dependencies | Membership configured; actual other members/access need host inspection |
| Backend `default` / `ai_internal` | Backend also joins both; ai-service only `ai_internal` | Unchanged | Normal bridge networks are not air gaps or protection from a Docker administrator |
| AI 8001 / optional Ollama 11434 | No host mappings, including `bundled-ollama` profile | Unchanged; new preflight rejects publications | Proven in rendered configuration |
| Database | External MariaDB/MySQL or volume-backed SQLite; no DB service/port in this Compose | Unchanged | External DB listener/network requires operator verification |
| PDF renderer | API child process, no separate listening service | B03 admission unchanged | Source and regression tests |
| Development | Explicit `-f docker-compose.dev.yml`, frontend 3000, API 5202, optional Ollama 11434 | Explicit direct nginx mode, HTTP localhost, API forwarding trust disabled | Rendered and tested separately; cannot pass production preflight |
Frontend and API are intended to share the canonical HTTPS origin; UI API base is `/api`. Preflight rejects a different production build-time API URL. Public 80/443 handling belongs to operator-owned Traefik and cannot be proved here. No cache/debug/control service is published by base Compose. `expose` is metadata, not a firewall. Host processes, attached containers, Docker administrators and routed/direct-container-IP configurations remain separate trust considerations.
## Implementation and trust rules
1. **Nginx original socket peer:** `INGRESS_MODE=proxy` is explicit in production. `INGRESS_PROXY_CIDRS` must list observed Traefik peers; there is no permissive default. The generated `geo $realip_remote_addr` gate checks the original socket peer, before allowing application locations. Untrusted peers receive **403**, including callers that know the canonical Host and forge every forwarding header.
2. **Normalized forwarding:** nginx `set_real_ip_from` uses that same explicit list; `real_ip_header X-Forwarded-For`, `real_ip_recursive off`. It sends `$remote_addr` as one normalized XFF value, canonical Host/port and canonical HTTPS scheme. It strips `X-Forwarded-Host` and RFC `Forwarded`; raw headers do not select an identity. All `/api/`, canonical `/health` and `/ready` proxy locations use this rule. Authentication/B02 headers pass through normally and retain their existing server checks.
3. **Malformed/extra metadata:** missing XFF, missing/nonexact `https`, or comma-separated XFF chains return **400** at nginx in proxy mode. An invalid single IP makes nginx realip retain the actual peer; only that peer's address is emitted, not the invalid supplied text. This safely charges the proxy peer rather than manufacturing a client partition; it also signals a broken upstream contract. This nginx fallback behavior still needs the runtime probe. ASP.NET's independent malformed/rightmost-hop behavior is tested and proven. Traefik must overwrite attacker headers, not append arbitrary chains.
4. **Development:** `INGRESS_MODE=direct` is permitted only with an HTTP canonical origin. It does not enable realip or trust incoming forwarding metadata; nginx forwards the actual socket IP and HTTP scheme. Production preflight rejects direct mode/Development environment. Local API-port convenience remains unchanged.
5. **Fail-safe configuration:** nginx rejects empty, broad/default-route or directive-shaped peer inputs and calls `nginx -t` before serving. The standard-library Python preflight validates the actual Compose JSON, requires narrow peer networks (private /24 or narrower IPv4; /64 or narrower IPv6; individual public proxy addresses), exact dedicated backend trust, same-origin HTTPS, Production environment, no host networking and no additional service publications. It rejects `DOTNET_ENVIRONMENT=Development` and automatic framework-forwarding enablement. Shell startup enforces CIDR characters/prefix bounds; nginx itself validates address syntax. Prefer individual /32 or /128 peers rather than trusting other containers in a subnet.
6. **API defence:** `ForwardedProxyConfiguration` still clears implicit known proxies/networks, trusts one configured hop, processes XFF/XFP only, and now rejects `/0`. Untrusted sockets—including mapped IPv4—cannot change effective IP/scheme. Existing `ExternalOrigin` still rejects noncanonical application Hosts in Production, allows only internal liveness exceptions, generates reset/verification/OAuth links from configuration and controls secure cookies. Forwarded Host is never enabled. `AllowedHosts=*` remains as a framework setting; the existing explicit canonical-host middleware supplies the actual Production host restriction.
7. **CORS correction:** Production previously defaulted to `http://localhost:3000` when no allowlist existed. It now defaults to the configured canonical origin; conflicting overrides/wildcards fail before listening. Credentialed requests remain allowed only for that origin. Development's configured allowlists remain usable. This is not a network-access control.
B04 middleware ordering is unchanged: canonical host/proxy handling → existing CSRF → authentication → endpoint authorization → rate limiter → B02 session context → controller. Owner quotas remain principal-based; anonymous/security quotas use the effective socket/proxy client IP. B03 admission is still inside the shared exporter, independent of ingress or endpoint. B01 outbound transport and B05 CSV behavior were not changed.
The host-port mapping is deliberately retained because the repository records a live dependency and does not reveal whether Traefik is host-local, containerized or remote. Blind loopback binding/removal could cause downtime. The new nginx peer gate is the repository's application-level closure for that compatibility listener. **If Docker/NAT makes trusted and untrusted traffic appear as the same peer, this gate is insufficient: do not allowlist that shared gateway.** Establish a verified binding/firewall or dedicated proxy route before rollout. This is a production stop condition, not a claim that a guessed CIDR fixes the host.
## Headers, limits and health
Nginx explicitly supplies `X-Content-Type-Options: nosniff`, `X-Frame-Options: DENY`, `Referrer-Policy: strict-origin-when-cross-origin` and restrictive camera/microphone/geolocation permissions on canonical responses. No new duplicate headers were added. Base API HTTPS redirection/HSTS are disabled because the external proxy owns TLS/redirects; actual Traefik HSTS/redirect/header policy is unverified. No proxy request-body or timeout tuning is configured in this nginx template: nginx defaults apply. B06 does not claim a special rate/WAF/body policy at Traefik. Existing application size/time limits and B03 admission remain; tuning broader limits is separate work.
Container health requires no host API publication: backend curls `127.0.0.1:8080/health`; frontend checks `127.0.0.1:80/health`. The nginx default-host `/health` returns only static liveness and remains accessible even to untrusted peers. Canonical `/health` still proxies backend liveness; `/ready` remains canonical/proxy-gated. Default-host application paths are rejected. Health checks do not prove that the new trusted-proxy route is configured correctly, so the route smoke below is mandatory.
## Permanent tests and local results
| Check | Result / exact scope |
|---|---|
| `scripts/test-ingress.py` | **3 tests pass**, including real production/dev Compose rendering and 17 unsafe-config mutations; no daemon needed. Before config edits, safe-production test failed as expected. |
| `ForwardedIngressTests` plus existing origin/rate-key tests | **30 focused pass**; 13 new backend cases cover trusted/untrusted/mapped/IPv6 peers, scheme, host, malformed/rightmost chains and trust-all rejection. |
| `ingress-tests/production.spec.ts` | **6 real HTTP scenarios pass**: Production CORS/default/unsafe startup, Host/XFH, secure-cookie scheme independence, untrusted IP-limit forgery, trusted clients and excess-chain defence. |
| Broader backend | **915 pass**, no selected skips; includes B01–B05. One pre-existing external-DNS IMAP test excluded because internet access was not authorized. |
| Frontend | **298 tests / 67 suites pass**, including nginx-health, B02/B03 and B05 download regressions. |
| Browser | **18 pass**, no failures/flaky outcomes/skips; includes B02 switching and B05 actual download. |
| B04 HTTP / B03 admission probes | **9 / 1 pass** unchanged. |
| Build/static checks | API/test-project Release warnings-as-errors: **0 warnings/errors**. Next production build and targeted lint pass. Shell syntax checks, Python compilation and `git diff --check` pass. |
| Standalone TypeScript | Existing unrelated `src/interview-prep.test.tsx(120,10): error TS2571: Object is of type 'unknown'.` remains; exit **1**. No suppression/fix added. |
| Entrypoint generation | **9 local guard/render cases pass**, seven rejected configurations plus production and development generation. Temporary probe substitutes only fixture paths and disables `nginx -t`; explicitly **not nginx runtime proof**. |
| Real nginx/container topology | **BLOCKED before resource creation**. Docker Desktop engine pipe absent; attempted startup crashed on inaccessible `dockerInference` socket. No factory reset/repair, image pull or production request attempted. |
The first native Production fixture run passed 2/4 but inherited the project's Development launch profile despite its environment setting. The fixture now uses `dotnet run --no-launch-profile`; all six final tests prove Production behavior. This was a fixture correction, not a relaxation of host/CORS assertions. Final results/hashes: [manifest](evidence/b06-test-results.json). Safe effective topology/guard observations: [observations](evidence/b06-observations.json). Raw local files remain in `.artifacts/b06`.
The permanent `scripts/test-ingress-topology.py` uses only an already-installed nginx image, synthetic data, unique containers/networks and loopback-only ephemeral host publication. It exercises this repository's real nginx template/entrypoint and a header-echo backend, plus a simulated trusted proxy peer. It does **not** claim to run production Traefik or a containerized full API. API behavior is covered separately by the real native HTTP tests. The topology test asserts trusted IPv4/IPv6 forwarding, two clients, host/header normalization, unchanged B02 header transport, untrusted direct nginx denial, missing/bad scheme and chain rejection, safe malformed-IP fallback, no backend publication and denied edge-to-backend routing. It cleans up only resources it created. Its runtime result is still pending.
### Commands run
Repository root (PowerShell):
```powershell
python scripts/test-ingress.py
docker compose --env-file .artifacts/b06/compose.env -f docker-compose.yml --profile bundled-ollama config --format json > .artifacts/b06/compose-after.json
Get-Content .artifacts/b06/compose-after.json -Raw | python scripts/validate-ingress.py
& C:/Users/Connor/.dotnet/dotnet.exe test JobTrackerApi.Tests/JobTrackerApi.Tests.csproj --configuration Release --no-restore --filter 'FullyQualifiedName~ForwardedIngressTests|FullyQualifiedName~ExternalOriginTests|FullyQualifiedName~RateLimitPartitionKeysTests' --logger 'trx;LogFileName=focused.trx' --results-directory .artifacts/b06 -- xUnit.parallelizeTestCollections=false xUnit.maxParallelThreads=1
& C:/Users/Connor/.dotnet/dotnet.exe test JobTrackerApi.Tests/JobTrackerApi.Tests.csproj --configuration Release --no-restore --no-build --filter 'FullyQualifiedName!~ImapServiceSsrfGuardTests.ConnectAsync_rejects_unresolvable_host_without_leaking_dns_detail' --logger 'trx;LogFileName=backend.trx' --results-directory .artifacts/b06 -- xUnit.parallelizeTestCollections=false xUnit.maxParallelThreads=1
& C:/Users/Connor/.dotnet/dotnet.exe build JobTrackerApi.Tests/JobTrackerApi.Tests.csproj --configuration Release --no-restore -warnaserror
& C:/Users/Connor/.cache/codex-runtimes/codex-primary-runtime/dependencies/native/git/usr/bin/sh.exe -n deploy/deploy.sh
& C:/Users/Connor/.cache/codex-runtimes/codex-primary-runtime/dependencies/native/git/usr/bin/sh.exe -n job-tracker-ui/configure-nginx-origin.sh
python -m py_compile scripts/validate-ingress.py scripts/test-ingress.py scripts/test-ingress-topology.py
python .artifacts/b06/render-probe.py
python scripts/test-ingress-topology.py # blocked: Docker engine unavailable
git diff --check
```
UI directory; API suites ran serially because all use port 5302:
```powershell
$env:DOTNET_HOST_PATH='C:/Users/Connor/.dotnet/dotnet.exe'
$env:NEXT_TELEMETRY_DISABLED='1'
node node_modules/@playwright/test/cli.js test --config=playwright.ingress.config.ts --reporter=json > ../.artifacts/b06/ingress-api.json
node node_modules/@playwright/test/cli.js test --config=playwright.rate-limits.config.ts --reporter=json > ../.artifacts/b06/rate-limits.json
node node_modules/@playwright/test/cli.js test --config=playwright.pdf.config.ts --reporter=json > ../.artifacts/b06/pdf-admission.json
node node_modules/jest/bin/jest.js --runInBand --watchAll=false --json --outputFile=../.artifacts/b06/frontend.json
$env:PLAYWRIGHT_STATIC_EXPORT='true'
$env:PLAYWRIGHT_PREBUILT_EXPORT='false'
node node_modules/@playwright/test/cli.js test --config=../.artifacts/b06/playwright.offline.config.ts --reporter=json > ../.artifacts/b06/browser.json
$env:NEXT_PUBLIC_API_BASE_URL='/api'
node --max-old-space-size=4096 node_modules/next/dist/bin/next build
node node_modules/eslint/bin/eslint.js playwright.ingress.config.ts ingress-tests/production.spec.ts --max-warnings=0
node node_modules/typescript/bin/tsc --noEmit --pretty false
```
The offline browser configuration was copied from the existing B02 fixture; external providers/workers were disabled. Its fresh test-targeted static build was subsequently replaced by the normal `/api` build. Neither was deployed.
## Required deployment verification — exact safe procedure
Run the container probe first on a **safe development/test Docker engine**, not the production host:
```bash
python3 scripts/test-ingress.py
python3 scripts/test-ingress-topology.py --image nginx:1.29.8-alpine
# Image must already exist. The script never pulls it. Arrange any image acquisition separately.
```
On the documented Linux deployment host, use the existing checkout `/opt/job-tracker/app` and shared configuration `/opt/job-tracker/shared/.env`. The following commands are **read-only**; do not run `deploy.sh`, `up`, firewall mutation or restore commands as part of this inspection. Review outputs locally; never paste complete rendered environment variables or container environment/secrets.
```bash
cd /opt/job-tracker/app
docker compose --env-file /opt/job-tracker/shared/.env -f docker-compose.yml --profile bundled-ollama config --format json | python3 scripts/validate-ingress.py
docker compose --env-file /opt/job-tracker/shared/.env -f docker-compose.yml ps
docker ps --format 'table {{.Names}}\t{{.Image}}\t{{.Ports}}'
sudo ss -lntp
if command -v nft >/dev/null; then sudo nft list ruleset; fi
if command -v iptables-save >/dev/null; then sudo iptables-save; fi
if command -v ip6tables-save >/dev/null; then sudo ip6tables-save; fi
if command -v ufw >/dev/null; then sudo ufw status verbose; fi
frontend_id=$(docker compose --env-file /opt/job-tracker/shared/.env -f docker-compose.yml ps -q frontend)
backend_id=$(docker compose --env-file /opt/job-tracker/shared/.env -f docker-compose.yml ps -q backend)
test -n "$frontend_id" && test -n "$backend_id"
docker inspect -f '{{json .NetworkSettings.Ports}} {{json .NetworkSettings.Networks}}' "$frontend_id" "$backend_id"
docker network inspect jobtracker_shared --format '{{json .IPAM.Config}} {{json .Containers}}'
docker network ls --filter label=com.docker.compose.network=web_proxy
docker exec "$frontend_id" nginx -T
docker exec "$backend_id" curl -fsS http://127.0.0.1:8080/health
docker exec "$frontend_id" wget -qO- http://127.0.0.1/health
```
Identify the actual Traefik service/process from `docker ps` / `ss`. If containerized, inspect its **mount paths and networks**, then inspect the identified configuration files locally; do not assume a filename or container name. If host-managed, inspect the identified systemd/service configuration. Confirm exact Host rule, TLS certificate/entrypoint, upstream host:3000 target, redirect behavior and whether `forwardedHeaders.insecure`/trusted upstream settings permit client forgery. Determine any real CDN/LB hop separately; it is not present in this repository. Inspect actual provider firewall/security-group settings as well as Docker's NAT/FORWARD rules; a UFW INPUT rule alone is not proof for Docker-published ports.
Obtain the **actual socket source seen at nginx**, not a guessed Docker subnet. Where host tools permit, a bounded SYN-only capture in nginx's network namespace avoids capturing HTTP bodies/credentials:
```bash
frontend_pid=$(docker inspect -f '{{.State.Pid}}' "$frontend_id")
test "$frontend_pid" -gt 0
sudo nsenter -t "$frontend_pid" -n tcpdump -nn -i any -c 10 'tcp dst port 80 and (tcp[tcpflags] & tcp-syn != 0)'
```
Use that only while issuing explicitly controlled health/config requests through Traefik and from an untrusted test client. If `nsenter`/`tcpdump` is unavailable, use the operator's equivalent packet/flow evidence; do not install tools or change production networking implicitly. If both routes share the same observed NAT source, **stop**. Choose a topology-specific binding/firewall or dedicated proxy route with an explicit rollback before allowing that peer. Configure `INGRESS_PROXY_CIDRS` only after this evidence; a wrong list will deny legitimate ingress. Normal deployment preflight fails before service replacement if the setting is absent.
After the change is separately approved and deployed, from an independently controlled **external test machine**, set values from verified DNS/host inventory, not guesses:
```bash
public_host='the-verified-canonical-host'
public_ip='the-verified-host-address'
curl --max-time 10 -sS -D - -o /dev/null "https://$public_host/api/auth/config"
curl --max-time 10 -sS -D - -o /dev/null "https://$public_host/api/auth/config" \
-H 'X-Forwarded-For: 198.51.100.10' -H 'X-Forwarded-Proto: http' \
-H 'X-Forwarded-Host: attacker.example.test'
curl --max-time 5 -sS -D - -o /dev/null "http://$public_ip:3000/api/auth/config" \
-H "Host: $public_host" -H 'X-Forwarded-For: 198.51.100.10' -H 'X-Forwarded-Proto: https'
curl --max-time 5 -sS -o /dev/null "http://$public_ip:5202/health"
curl --max-time 5 -sS -o /dev/null "http://$public_ip:8080/health"
curl --max-time 10 -sS -D - -o /dev/null "http://$public_host/api/auth/config"
```
Expected: canonical TLS path succeeds with appropriate headers; spoofing at the external edge is overwritten or rejected, not trusted; direct port-3000 application request is firewall-denied or nginx **403**, never successful; development/API listeners are unavailable unless a separately documented protected service owns that port; public HTTP redirects according to the verified Traefik policy. Repeat for every published A/AAAA address, with IPv6 bracket syntax and `curl -6`; do not infer IPv6 rules from IPv4 results. Verify certificate hostname/chain without `-k`. Match DNS to the inspected ingress. Do not probe unrelated services/ports or production login quotas. A config endpoint's 200 alone cannot prove client-IP integrity; inspect trusted forwarding/flow evidence or repeat the synthetic limiter checks in staging. Capture intended login/session/CSV/PDF smoke through the canonical route with a dedicated test account after normal deployment authorization.
## Files, residual risk and stop conditions
Application/config: `docker-compose.yml`, `docker-compose.dev.yml`, nginx template/entrypoint, `ForwardedProxyConfiguration.cs`, `Program.cs` CORS block, `.env.example`; preflight in `deploy/deploy.sh`, new `scripts/validate-ingress.py`, CI safety-script invocation. Tests: `ForwardedIngressTests.cs`, `scripts/test-ingress.py`, `scripts/test-ingress-topology.py`, `playwright.ingress.config.ts`, `ingress-tests/production.spec.ts`. Documentation: active deployment docs, this record, validation README/queue/evidence, TODO and B06 evidence files. Earlier uncommitted B01–B05 changes and unrelated `docs/redesign` remain untouched; the entire working-tree diff is not B06.
No schema or data migration. Backup-only automation does not require the new ingress gate. A bad peer list or unexpected forwarding chain can break the intended route; preflight/syntax/liveness checks alone cannot validate the peer value. Preserve existing images/env and verify both routes before rollout. Rolling back nginx to the old behavior reopens the header-laundering assumption unless network isolation is independently enforced; rollback therefore requires verified firewall/binding protection. Do not bypass the gate with trust-all networks.
Limits remain process-local after B04/B03; additional API instances multiply budgets as documented there. B06 introduces no distributed system or new hosting platform. Host/container compromise and shared-network lateral access are not eliminated by a peer allowlist. Privileged Docker users can attach containers regardless of service names. No claim is made about external DB/Ollama listeners, external TLS, firewall correctness, raw backend network reachability from other trusted containers, or an uninspected additional ingress route.
**Local acceptance:** criteria 2/4/6/7/8/9 have repository/API/test evidence; backend publication checks for criterion 1 pass, but the nginx closure supporting criteria 1/3/5 still requires real runtime/topology verification. **Do not mark B06 locally complete or F04 resolved yet. Production verification is also incomplete.** The next action is to complete these B06 gates and supply the observed production proxy peers. **Next queued implementation ticket: B07, only after the required B06 decision/verification and a further instruction; not started.**
## Runtime verification follow-up 2026-09-09
**Outcome: B06 implementation complete, but F04 remains open.** Implementation complete describes the existing application/configuration change; it does **not** mean full local-runtime acceptance or production verification passed. This follow-up changed evidence/documentation only. No application, Compose, nginx, B01–B05 behavior, credentials, DNS, certificate or production service was changed. B07 was not started.
The user explicitly authorized targeted public DNS/TLS/direct-access checks in this follow-up. These were bounded unauthenticated GETs to the documented hostname and its observed A address, not production load tests. No login budgets, user data or credentials were used. Response bodies and cookies were discarded; retained headers are allowlisted. [Runtime evidence](evidence/b06-runtime-observations.json), [test/evidence manifest](evidence/b06-runtime-results.json) and [reproducible public probe](evidence/b06-runtime-public-probe.py.txt) preserve commands and observations. Raw local files are under `.artifacts/b06-runtime/`.
### Actual execution and listener evidence
The real production Compose file was rendered again, including the optional Ollama profile, with the existing synthetic `.artifacts/b06/compose.env`. The preflight passed. [Sanitized effective configuration](evidence/b06-runtime-compose.json) confirms frontend TCP **3000 → 80**, host address unspecified, and **zero published backend/AI/Ollama ports**. The internal `web_proxy` network contains frontend/backend in configuration; backend also joins `shared_services`, `default` and `ai_internal`. The synthetic `192.0.2.10/32` ingress value is a test input, **not an observed Traefik peer or a recommended production value**.
The actual topology could not be started:
- `docker desktop status` returned exit 1, unable to retrieve status.
- `docker desktop start --detach` returned exit 0 (startup dispatched), then Docker Desktop **4.86.0** reported the same fatal inference-manager socket failure supplied by Connor. This exit code is not engine readiness.
- A server-version request timed out at 20 seconds; a final bounded request timed out at 10 seconds. Earlier missing-pipe evidence remains preserved.
- The failing `C:/Users/Connor/AppData/Local/Docker/run/dockerInference` entry is zero bytes with `Archive, ReparsePoint` attributes, last modified 2026-08-24. Read-only `fsutil reparsepoint query` returned **Error 1920: The file cannot be accessed by the system**. This does not establish the underlying Docker/Windows defect or prove the entry is unused.
- A graceful `docker desktop stop` attempt did not complete; the waiting command was interrupted. Failed Desktop/backend processes remained observable. No socket, Docker data, WSL distribution or configuration was deleted/renamed/reset, and no process was force-killed to improvise a repair.
- The inspected **Windows workstation**, not the Ubuntu deployment host, had no listeners on 80, 443, 3000 or 5202. Its `127.0.0.1:8080` listener belonged to **steamwebhelper**, not Jobjakt. Windows firewall profiles were enabled; their default actions were `NotConfigured`. Neither observation establishes production firewall/NAT behavior.
Consequently there is **no new running-container evidence** for `nginx -t`, actual host bindings, network membership, trusted-peer matching, direct nginx rejection, internal API reachability, container restarts or missing/invalid-variable startup. No actual peer CIDR was guessed. The existing echo-backend/simulated-peer nginx probe would be supplementary only; it cannot satisfy this request for **real Traefik + real application + actual Compose** and was not represented as such.
### Observed public DNS, HTTPS and direct access
Measurements were taken around **2026-09-09 10:43–10:47 UTC** from Connor's Windows workstation (`192.168.50.2`). Requests used `curl --noproxy '*'`, TLS verification enabled, connect timeout 4 seconds, total timeout 8 seconds and at most three simultaneous harmless GETs. Independence from the production LAN/NAT is **not established**; these observations must be repeated from a known external network.
| Check | Observed result | What this proves / does not prove |
|---|---|---|
| Resolver plus both authoritative servers, `oaklyn.ns.cloudflare.com` and `thaddeus.ns.cloudflare.com` | `jobs.cesnimda.uk` CNAME → `cesnimda.uk`; A answer **109.189.21.248**; no AAAA answer (SOA/NODATA) | Consistent current answers for this hostname. No additional A/AAAA address was observed. Intended address ownership, other hostnames and complete provider-zone inventory remain operator checks. |
| `https://jobs.cesnimda.uk/api/auth/config` | **200**, valid hostname/chain; TLS 1.3 independently verified with Python's default trust context | Public HTTPS application path works. It does not identify Traefik, nginx's peer, deployed image or backend client IP. |
| Certificate | SAN/CN `jobs.cesnimda.uk`, Let's Encrypt YR2; valid **2026-08-16 20:35:58 UTC to 2026-11-14 20:35:57 UTC** | Hostname/chain validation succeeded without bypass flags. No separate revocation audit is claimed. |
| HTTP port 80 | **301** to `https://jobs.cesnimda.uk:444/api/auth/config` | An actual redirect differs from the documented standard HTTPS origin. |
| Observed redirect target, HTTPS port 444 | **Connection failure**, curl exit **7** | HTTP-to-HTTPS navigation is broken from this vantage point. Operator must inspect the real redirect entrypoint/NAT configuration and establish the intended reachable target. No certificate or production proxy change was attempted. |
| HTTPS with forged XFF/XFP/XFH; HTTPS with malformed XFF chain | **200** in both cases | Availability only. A 200 cannot distinguish sanitized forwarding from a vulnerable acceptance path; backend effective identity is still unobserved. |
| HTTPS with `Host: attacker.example.test`, original canonical TLS SNI | **421 Misdirected Request** | This forged Host request was rejected at the observed public boundary. It does not prove every internal/generated-URL path. |
| Direct A-address TCP 3000, canonical Host, six cases: no forwarding, forged XFF, forged HTTPS, forged Host, combined headers, malformed chain | **6/6 connection failures**, curl exit **7**, approximately 2.0 seconds each | No HTTP request reached a listener from this vantage point. This is **not** evidence of nginx 403 or attribution to any specific firewall. |
| Direct A-address TCP 5202 and 8080 `/health` | **2/2 connection failures**, curl exit **7** | Neither tested port was reachable here. No claim about other interfaces, IPv6 firewall policy, host-local/container access or unknown alternate ports. |
Public responses include HSTS, nosniff, `X-Frame-Options: SAMEORIGIN`, `Referrer-Policy: same-origin` and a Permissions Policy. These differ in part from the checked-in nginx template (`DENY`, `strict-origin-when-cross-origin`). An upstream override or a different deployed revision could explain this; the source cannot be attributed without operator configuration/image evidence. Public success therefore cannot certify that the current B06 working tree is deployed.
### Fresh automated checks and preserved broader results
| Check in this follow-up | Result | Evidence classification |
|---|---|---|
| Effective production/development Compose tests, including 17 hostile mutations | **3 passed**; preflight of newly rendered production JSON also exit 0 | Proven by automated test |
| Forwarded ingress, canonical origin, B01 HTTP transport, B02 session middleware, B03 admission, B04 rate keys and B05 CSV | **166 passed**, 0 failed/skipped; Release compilation succeeded | Proven by automated test; API/native boundary, not live proxy chain |
| Native Production ingress HTTP suite | **6 passed**, 0 unexpected/flaky/skipped | Proven by automated test, real native API |
| Native B04 HTTP suite | **9 passed**, 0 unexpected/flaky/skipped | Proven by automated test, real native API |
| B04 synthetic shared-IP evidence | In each exhaustion order, A **2 admitted / 1 rejected**, B **2 admitted / 1 rejected**; concurrent two-session case admits **4 total**, then each account rejects **2** attempts using other IPs; cookie/bearer share owner budgets | Does not prove real Traefik/nginx IP propagation. No production quotas were consumed. |
| Existing broader verification, unchanged application source | Prior **915 backend / 298 frontend / 18 browser** passes, Release builds and targeted lint retained above; broader suites were not relabeled as rerun | Preserved prior automated evidence |
| Standalone TypeScript | Prior unrelated `src/interview-prep.test.tsx(120,10): TS2571` remains the documented result; not rerun in this evidence-only follow-up | Existing limitation; no claimed fix |
| Final evidence checks | `git diff --check` **exit 0** (Git line-ending notices only); probe Python syntax and all new evidence JSON parse checks pass | Evidence/documentation verification |
Exact fresh commands (repository root unless stated otherwise):
```powershell
python scripts/test-ingress.py
docker compose --env-file .artifacts/b06/compose.env -f docker-compose.yml --profile bundled-ollama config --format json
# Output saved to .artifacts/b06-runtime/compose-effective.json; synthetic inputs only.
Get-Content .artifacts/b06-runtime/compose-effective.json -Raw | python scripts/validate-ingress.py
& C:/Users/Connor/.dotnet/dotnet.exe test JobTrackerApi.Tests/JobTrackerApi.Tests.csproj --configuration Release --no-restore --filter 'FullyQualifiedName~ForwardedIngressTests|FullyQualifiedName~ExternalOriginTests|FullyQualifiedName~JobImportHttpTransportTests|FullyQualifiedName~SessionContextMiddlewareTests|FullyQualifiedName~PdfRenderAdmissionTests|FullyQualifiedName~RateLimitPartitionKeysTests|FullyQualifiedName~CsvExportTests' --logger 'trx;LogFileName=regressions.trx' --results-directory .artifacts/b06-runtime -- xUnit.parallelizeTestCollections=false xUnit.maxParallelThreads=1
# In job-tracker-ui, serially:
$env:DOTNET_HOST_PATH='C:/Users/Connor/.dotnet/dotnet.exe'
$env:NEXT_TELEMETRY_DISABLED='1'
node node_modules/@playwright/test/cli.js test --config=playwright.ingress.config.ts --reporter=json
node node_modules/@playwright/test/cli.js test --config=playwright.rate-limits.config.ts --reporter=json
# Repository root:
python .artifacts/b06-runtime/probe.py
Resolve-DnsName jobs.cesnimda.uk -Type A -DnsOnly
Resolve-DnsName jobs.cesnimda.uk -Type AAAA -DnsOnly
Resolve-DnsName jobs.cesnimda.uk -Type CNAME -DnsOnly
Resolve-DnsName cesnimda.uk -Type NS -DnsOnly -NoHostsFile
# Repeat A/AAAA/CNAME using -Server for each observed authoritative nameserver.
Get-NetTCPConnection -State Listen
Get-NetFirewallProfile
git diff --check
```
### Operator handoff and evidence required for closure
Permitted deployment docs/scripts/CI configuration refer only to `PROD_HOST`, `PROD_USER` and `PROD_SSH_KEY` secret names. `C:/Users/Connor/.ssh/config` does **not exist**. No unambiguous SSH alias/username was found; no SSH connection was guessed, and no CI secrets, keys/passwords or live `.env` were retrieved. Production read-only inspection therefore remains an operator handoff under Connor's explicit instruction.
Run the earlier [exact host procedure](#required-deployment-verification--exact-safe-procedure) on the actual Ubuntu host. It uses the documented `/opt/job-tracker/app` checkout and `/opt/job-tracker/shared/.env`, prints validation results rather than secret environment data, inventories listeners/published ports/networks, checks real nginx configuration/internal health and obtains SYN-only peer evidence. Add these read-only checks:
```bash
cd /opt/job-tracker/app
git rev-parse HEAD
git status --short
docker inspect --format '{{.Name}} image={{.Image}} ports={{json .NetworkSettings.Ports}}' "$frontend_id" "$backend_id"
# Use frontend_id/backend_id established by the earlier procedure.
# Observe the marked request in the existing nginx access log without sharing unrelated requests:
docker logs --since 2m "$frontend_id" 2>&1 | grep -F '/api/auth/config?b06_probe=runtime-check'
```
While capturing the SYN peer, issue `curl --noproxy '*' --max-time 10 -sS -o /dev/null 'https://jobs.cesnimda.uk/api/auth/config?b06_probe=runtime-check'` from the controlled client. Compare the original socket source with nginx's generated `geo`/`set_real_ip_from` entries, and the access log's effective client address with the known test source. The access-log format must be inspected before interpreting fields. These alone do not prove the IP finally used by backend rate limiting; that requires the real-chain synthetic A/B scenarios on an isolated staging/test topology.
| Evidence to return, with secrets/cookies/user content removed | Safe result | Unsafe or unresolved result |
|---|---|---|
| Host/provider identity, deployed revision/image IDs, Compose preflight result and frontend/backend `Ports`/network subsets | Current intended build/configuration, only intended frontend host publication; API unpublished | Old/unknown build, dev overlay, host networking or extra API/public internal-service binding |
| Relevant `ss`, Docker NAT/FORWARD, IPv4/IPv6 firewall and provider rule excerpts | Only documented public routes; direct app path blocked or explicitly peer-restricted | Rule comments alone, unknown IPv6 handling or externally reachable API |
| Traefik entrypoint/router/upstream and forwarded-header directives only; generated nginx trust entries; marked SYN peer observation | Actual Traefik source matches narrow allowed peer; untrusted source differs; client forwarding sanitized | Trust-all, guessed subnet or shared NAT identity for trusted and untrusted paths — stop before permitting that peer |
| Internal nginx/API health and controlled canonical-route result | Healthy internal API with working canonical ingress | Public API port required for health, internal 502 or legitimate caller rejected |
| Direct nginx attempts from an actually untrusted reachable test network, plus spoof/malformed variants | 403/444 or independently explained network denial; no forged identity reaches backend | Successful application response from forbidden peer or caller-selected IP/scheme/host |
| Fresh A/AAAA/provider inventory, certificate validation and HTTP redirect result from a known external network | Intended ingress address(es), valid TLS, HTTP target reachable at canonical HTTPS origin | Stale/alternate bypass record, invalid chain or the observed unreachable `:444` redirect |
| Isolated real-chain A/B budgets, actual backend client-IP observations and B02 context checks | Each owner isolated; sessions share owner; spoofing cannot select IP budget; stale context rejected | Backend sees only proxy IP or client-forged IP; header/session selects another owner |
Return only selected sanitized excerpts and the timestamp/test vantage point. **Do not send complete `.env`, full `docker inspect`, authentication headers, tokens, passwords, private keys/certificates or unrelated access logs.** Do not run restart or missing/invalid-configuration tests against production as part of this read-only handoff. Those remain mandatory in the isolated real-container test environment after Docker is operational; record startup failure, restart health, peer/network stability and repeat direct-access rejection.
### Closure checklist by evidence level
1. **Proven by automated test:** effective production port/trust constraints, fail-closed preflight, native trusted/untrusted forwarding behavior, B04 principal/IP separation, B02 session guard, B01/B03/B05 focused regressions.
2. **Proven in running local/container topology:** **none in this follow-up**. Docker failure prevents actual Compose startup, real nginx syntax/enforcement, actual Traefik peer capture, internal routing, restart/config-failure tests and real-chain A/B verification.
3. **Proven against the deployed public endpoint from this workstation:** DNS observations, valid HTTPS/200, forged Host 421, direct-port connection failures from this vantage, HTTP redirect to unreachable 444. This is partial public-endpoint evidence, not certification of the internal deployed chain.
4. **Still requires operator verification:** deployed build/topology/trust CIDRs, actual socket peers, backend client IP/scheme, real-chain session/account/IP behavior, host/container listeners and other network memberships, host/provider IPv4/IPv6 firewall/NAT, independent external reachability, intended DNS-address inventory and correct HTTP redirect target.
The blocking items are therefore the unavailable local Docker engine, absent verified production host access/evidence, missing real-chain peer/identity tests, and the observed HTTP redirect mismatch. **B06 implementation complete, but F04 remains open.** No change to F04's conditional severity or closure status is justified by connection failures alone.
## Authorized release attempt and CI prerequisite correction
Connor subsequently authorized commit/push to live. B01–B06 were committed as `0954876`, and the redesign brief as `1bbbd05`; both were pushed to `main`. [Gitea run 736](https://git.cesnimda.uk/cesnimda/jobtrackingapp/actions/runs/736/jobs/1607) failed before backend/frontend testing: `scripts/test-ingress.py` raised `FileNotFoundError: 'docker'` in the runner. Deployment was skipped; the push was not a successful live rollout.
The first correction (`f485f72`) attempted to install pinned Compose v5.3.1 through the [official Docker setup action](https://github.com/docker/setup-compose-action), pinned to commit `2fe291b7677a45ee1269ec56a42604c143505e7e`. Gitea tokens were deliberately not passed to GitHub. The config test accepts standalone Compose, retaining `docker compose` on existing Docker installations. Nothing is skipped and the production ingress gate remains unchanged.
Validation before the corrective push: **3/3 tests through `docker compose` and 3/3 through the real standalone binary** with Docker discovery disabled for the latter fixture; tracked-secret scan and `git diff --check` pass. This corrects a CI dependency assumption; it neither repairs local Docker Desktop nor supplies any missing F04 deployment evidence.
[Run 737](https://git.cesnimda.uk/cesnimda/jobtrackingapp/actions/runs/737/jobs/1609) then failed in the action itself: `Parameter token or opts.auth is required`; deployment again skipped. The final setup uses the [official standalone release](https://github.com/docker/compose/releases/tag/v5.3.1) directly, in a runner-local temporary directory. It pins Linux x86_64 and SHA-256 `f9ebc6ebdb19d769b793c245a736caaeb198c62587f13b25c660c13b4987f959`, verifies before execution, and needs neither a daemon nor an authentication token. A fresh local download matches that official release digest; 3/3 ingress config tests and diff validation pass. Runner execution and deployment are tracked separately from these local checks.
[Run 738](https://git.cesnimda.uk/cesnimda/jobtrackingapp/actions/runs/738/jobs/1611), for `97fce89`, passed Compose installation, all repository safety scripts, browser-extension tests, secret/SBOM checks, parser tests and the Release API build. Test-project build then failed with MSB3021/MSB3024: the Linux `PdfBrowserStub` apphost was copied to an output path already used as the fixture directory. Windows' `.exe` suffix had hidden the collision. The project reference now sets `Private="false"`; it still builds the fixture, while the existing explicit subdirectory copy is its sole output owner. SDK target inspection confirms this disables transitive content copying. No renderer or admission behavior changes.
Local verification: `C:/Users/Connor/.dotnet/dotnet.exe test JobTrackerApi.Tests/JobTrackerApi.Tests.csproj --configuration Release --no-restore --filter FullyQualifiedName~PdfRenderAdmissionTests --logger "console;verbosity=minimal"` built successfully and passed **22/22** tests, including real exporter/fixture failure recovery. A preliminary unqualified `dotnet` invocation could not find the pinned SDK because PATH selected the system runtime; the explicit installed SDK resolved it. Diff checks pass. Linux runner confirmation remains required after push; a build/package fix does not close F04.
[Run 739](https://git.cesnimda.uk/cesnimda/jobtrackingapp/actions/runs/739/jobs/1613), for `23cc250`, failed earlier in the unchanged API compiler: `csc.dll` exited 139 on both the ordinary and existing single-process retry. This is the previously documented runner crash signature, not a compile diagnostic or test assertion. It prevented Linux confirmation of the packaging correction. No retries/gates were weakened. Fresh local broader verification with the explicit SDK passed **915/915** selected backend tests, no selected skips, with only the same documented external-DNS-dependent IMAP exclusion; `.artifacts/redesign-audit/backend.trx` records the run.
Pushing this evidence provides one further complete CI attempt with unchanged application/test logic. If the existing bounded retry is exhausted again, stop release attempts and obtain runner-host diagnostics under the [runner investigation](../../../infrastructure/runner-investigation.md); do not increase retry loops indefinitely or deploy past a red test gate. Phase 1 redesign audit is separately committed on `codex/ui-redesign-audit`, with no redesign implementation. F04 remains open regardless of CI outcome until its runtime/production checklist is proven.