fix(cv): isolate document parsing

Run untrusted document decoders in a secret-free, resource-bounded child process and terminate its process tree on deadline. Harden the production container and enforce parser and lint gates in CI.
This commit is contained in:
cesnimda
2026-08-30 11:12:25 +02:00
parent a8bf505ce5
commit 19c5251612
13 changed files with 677 additions and 176 deletions
+1
View File
@@ -222,3 +222,4 @@ Output was reduced to filenames and commit counts. The token artifact appears un
| V-188 | 75-user query-count fixture, 205-message/two-tenant pagination fixture, focused/full backend and frontend, optimized build | Repository root / `job-tracker-ui` | Remove JT-021's confirmed N+1 and silent 200-message ceiling without breaking old clients | PASS — admin list performs two reads independent of 75 users; page 3 returns the final 5 of 205 owned messages and excludes another tenant. Focused backend 9/9, correspondence UI 15/15, backend 683/683, frontend 58 suites/239 tests and build pass. UI page navigation and filter reset are covered; the legacy inbox endpoint remains unchanged | Synthetic SQLite/InMemory/JSDOM only; no provider mailbox, production dataset or p95 load test. Current page linked/inbound chips intentionally describe the visible page | JT-021 repository defects closed; provider/production capacity remains operational evidence |
| V-189 | Fresh Python 3.12 virtual environments; `pytest -q`; `npm run lint`; full frontend Jest; `npm run build`; `npm audit --audit-level=moderate` | Repository root / `tools/summarizer` / `job-tracker-ui` | Restore the local Python test environment and establish a reproducible zero-warning frontend lint gate | PASS — Python 3.12.10; sidecar 26/26; ESLint zero findings; frontend 64 suites/272 tests; optimized build/TypeScript; npm audit zero vulnerabilities | Five existing SWIG deprecation warnings; Jest retains its documented force-exit/open-handle notice. ESLint 9 is intentionally pinned because the Next 16 React plugin is not ESLint 10 compatible | Local Python/frontend quality gates restored; parser dependency remediation proceeds separately under SEC-006 |
| V-190 | Compatible parser dependency resolution; Linux CPU hash lock; clean hash install; `pip-audit`; generated parser boundary tests; focused/full backend and sidecar suites; Docker daemon probe | Repository root / `tools/summarizer` | Remove reachable upload-parser advisories and close unsafe fallback/resource-boundary paths without parsing hostile fixtures | PASS/PARTIAL — FastAPI 0.141.1/Starlette 1.6.0, Pillow 12.3.0, pypdf 6.16.2 and python-multipart 0.0.32 resolve and install from hashes; upload-facing packages audit clear; parser 32/32, focused backend 51/51 and backend 719/719. Signature/container/page/pixel/decompression/output limits pass; backend binary fallback is removed and unexpected failures are sanitized | Docker daemon unavailable, so production image smoke/container assertions did not run. `pip-audit` reports 45 Torch/Transformers model-stack advisories tracked separately under JT-017. One Starlette/httpx test-client warning and five SWIG warnings remain | SEC-006 implemented locally; SEC-007 boundary work in progress, with child-process and container isolation still required |
| V-191 | Real parser child extraction; minimal-environment probe; timeout/process-tree/capacity/stale-cleanup tests; `py_compile`; parser suite; Compose interpolation/control inspection | Repository root / `tools/summarizer` | Ensure untrusted document decode cannot consume the AI web process or inherit provider secrets and receives explicit runtime/container budgets | PASS/PARTIAL — parser 37/37 without warnings; TXT extraction executes in a child; provider/service secrets are absent; timeout kills parent and descendant; capacity recovers; stale cleanup preserves recent/unrelated paths. Compose resolves read-only root, `cap_drop: ALL`, no-new-privileges, 96 PIDs, 2 CPUs, 2 GiB memory, 768 MiB tmpfs and one model-cache volume | Windows proves deadline/process-tree behavior but cannot execute Linux `setrlimit`; Docker Desktop daemon is offline, so image build, non-root identity, inside-container rlimits and benign PDF/DOCX/image sizing remain unverified | SEC-007 repository boundary implemented; Linux/container/browser/production verification remains |
@@ -0,0 +1,46 @@
# SEC-007 parser isolation verification
## Implemented boundary
- The FastAPI web process reads at most 5 MiB and performs cheap extension, signature and DOCX-container preflight checks.
- TXT, Markdown, PDF, DOCX and image decoding runs in `parser_child.py`, never in the web process.
- Only one parser child is admitted at a time; excess work receives a stable busy response instead of building an in-memory parser queue.
- The child receives a minimal environment without AI provider keys or the service token.
- Linux applies CPU, address-space, output-file, open-file and core-dump limits before importing parser libraries.
- The parent enforces a 25-second deadline and terminates the process group/tree, including OCR descendants.
- Per-request work uses an owner-private temporary directory. Old `work-*` directories are reconciled without following symlinks or removing unrelated paths.
- The backend has no PDF/DOCX/image fallback. Stable messages replace unexpected internal exceptions and paths.
## Container boundary
The production Compose definition resolves the AI service with:
- non-root Dockerfile user;
- read-only root filesystem;
- all Linux capabilities dropped;
- `no-new-privileges`;
- 96 PID, 2 CPU and 2 GiB memory limits;
- 768 MiB `noexec,nosuid,nodev` `/tmp` tmpfs;
- one writable named volume only for the Hugging Face model cache.
## Local proof
- Parser tests: 37/37 passed.
- Focused backend extraction/operation tests: 51/51 passed.
- Full backend: 719/719 passed.
- Python compilation: `app.py` and `parser_child.py` passed.
- Compose config: valid with all intended controls present.
- Timeout regression spawns a harmless sleeping descendant and proves the process tree is gone after the deadline.
- Environment regression proves `GEMINI_API_KEY`, `GROQ_API_KEY` and `AI_SERVICE_TOKEN` are absent from the child.
## Remaining external proof
Docker Desktop's Linux daemon is offline, so the image could not be built or run. Before activation:
1. Build the hash-locked image and record its digest.
2. Confirm the runtime UID is non-root and the root filesystem rejects writes.
3. Inspect the applied CPU/memory/PID/capability/tmpfs settings.
4. Exercise benign TXT, Markdown, PDF, DOCX and image extraction within the configured budgets.
5. Measure peak parent/child memory and OCR duration; lower or evidence any limit increase.
6. Run the authenticated browser upload/review journey with synthetic files.
7. Canary in production only after explicit authorization; disable CV import rather than removing the boundary if the budget is insufficient.
+6 -6
View File
@@ -44,7 +44,7 @@ Updated: 2026-08-30
### In progress
- SEC-007 bounded document processing. Upload-facing parser dependencies are fixed and hash-locked; the sidecar now rejects signature mismatches, decompression/page/pixel/output excesses, the backend no longer retries PDF/DOCX/image parsing in-process, and unexpected parser details are sanitized. Child-process deadline/resource isolation and container controls remain.
- No repository implementation package is currently in progress. SEC-006/SEC-007 await a running Linux Docker daemon for image/runtime proof; production activation remains separately gated.
### Remaining
@@ -108,17 +108,17 @@ Updated: 2026-08-30
- Playwright: 8/10 passed on the first complete run; both failures were ambiguous selectors in the newly responsive Career selector, not product failures. Both corrected focused regressions now pass (2/2); final full rerun remains in the end-of-batch gate.
- Manual desktop browser review: webpack development server rendered the new Career navigation and Overview correctly in dark mode; API-dependent profile status remained unavailable because the backend was not running for that isolated UI review.
- **Overall programme status:** Active but externally blocked. Eight packages are locally verified, twenty-six are implemented with verification incomplete, and SEC-007 is in progress. The prioritized admin-only version indicator, every immediate repository/browser item, SEC-009, the PROD-001 read-only inventory, and the PROD-003 safe benchmark harness are complete on the release branch.
- **Current work package:** SEC-007 bounded isolated document processing; SEC-006 is implemented locally and awaits Docker image smoke.
- **Overall programme status:** Active but externally blocked. Eight packages are locally verified and twenty-seven are implemented with verification incomplete. The prioritized admin-only version indicator, every immediate repository/browser item, SEC-006/SEC-007 repository boundaries, SEC-009, the PROD-001 read-only inventory, and the PROD-003 safe benchmark harness are complete on the feature branch.
- **Current work package:** Reconcile the next safe repository item while SEC-006/SEC-007 await Linux image/runtime proof.
- **Completed work packages:** None are `DONE`; all repository security/AI packages still have applicable browser, provider and/or production gates.
- **Locally verified work:** SEC-001, SEC-002, SEC-003, SEC-005A, CORE-001, PROD-002, DEP-001 and VER-001 (`VERIFIED LOCALLY`).
- **Implemented, verification incomplete:** SEC-004, SEC-005B, SEC-006, SEC-008, SEC-009, CORE-002, BG-001, OPS-001A/B/C, POL-001/002, AI-001/002/003/004, UX-001/002/003, QA-001, CAREER-001/002, MAIL-001, JOBS-001/002 and PRODUCT-001 (`IMPLEMENTED — NOT VERIFIED`). Their safe repository/browser scope is implemented; production/native-device/provider/retention gates remain where recorded.
- **Implemented, verification incomplete:** SEC-004, SEC-005B, SEC-006, SEC-007, SEC-008, SEC-009, CORE-002, BG-001, OPS-001A/B/C, POL-001/002, AI-001/002/003/004, UX-001/002/003, QA-001, CAREER-001/002, MAIL-001, JOBS-001/002 and PRODUCT-001 (`IMPLEMENTED — NOT VERIFIED`). Their safe repository/browser scope is implemented; production/native-device/provider/retention gates remain where recorded.
- **Production-verified work:** None.
- **Blocked work:** PROD-001/003/004 and REL-001 require network/backup/model/deployment authority and unfinished dependencies. Real provider and live deletion/restore checks remain gated; DEP-001 awaits approved merge/live verification.
- **Deferred work:** None. Conditional multi-replica coordination, model deletion, realtime operation delivery and unrelated production changes remain outside current packages.
- **Immediate order:** all sixteen immediate repository items are complete locally, including the original UI/release queue plus SEC-009 cache/tombstone safety, worker restart clocks, universal AI accounting, email-token/Stripe lifecycle tests, exhaustive Job email selectors, the repaired migration chain, CV/public-edge hardening and measured admin/mail scaling. PROD-001 read-only evidence and the PROD-003 plan-only harness are also complete. The final audit is checking tooling/documentation before declaring only external blockers remain.
- **Status counts:** 8 `VERIFIED LOCALLY`; 26 `IMPLEMENTED — NOT VERIFIED`; 1 `IN PROGRESS`; 0 `NOT STARTED`; 4 `BLOCKED`; 0 `DONE`; 0 `DEFERRED`.
- **Test status:** backend 719/719; frontend 64/64 suites and 272/272 tests; ESLint zero findings; AI sidecar 32/32; Ollama benchmark harness 5/5 plus safe dry-run; optimized production build/TypeScript; EF model parity; SQLite/MariaDB migration scripts; blank/idempotent/populated SQLite migration-chain tests; disposable fresh/restarted MariaDB 11.8 application startup; Docker Compose config; safe-failure deployment preflight; and Playwright 9/9. npm audit remains at zero. The parser image build is pending because Docker Desktop's daemon is offline. Jest's slow/open-handle behavior and six Python deprecation warnings remain recorded.
- **Status counts:** 8 `VERIFIED LOCALLY`; 27 `IMPLEMENTED — NOT VERIFIED`; 0 `IN PROGRESS`; 0 `NOT STARTED`; 4 `BLOCKED`; 0 `DONE`; 0 `DEFERRED`.
- **Test status:** backend 719/719; frontend 64/64 suites and 272/272 tests; ESLint zero findings; AI sidecar 37/37 without warnings; Ollama benchmark harness 5/5 plus safe dry-run; optimized production build/TypeScript; EF model parity; SQLite/MariaDB migration scripts; blank/idempotent/populated SQLite migration-chain tests; disposable fresh/restarted MariaDB 11.8 application startup; Docker Compose config; safe-failure deployment preflight; and Playwright 9/9. npm audit remains at zero. The parser image build is pending because Docker Desktop's daemon is offline. Jest's slow/open-handle behavior remains recorded.
- **Deployment status:** Gitea pull-request run 609 passed the complete CI job in 4m20s. Deploy was intentionally skipped because the workflow deploys only a `push` to `main`; live remains unchanged. No merge/deployment was performed directly, no production migrations were run and the AI operation worker remains disabled by default.
- **Production status:** State unchanged. Sanitized read-only SSH inventory was performed; no logs, prompts, private rows/content or secret values were read, and no provider/model call, model pull, service restart, file/config change, backup, restore, migration or deployment occurred. It confirmed all-interface Ollama/frontend listeners and stale database-only backups as rollout blockers.
- **Known regressions:** None found by automated/local browser checks. Jest still needs `--forceExit` and reports its existing open-handle notice. Email-provider/send tests are fake/local only; real delivery is not claimed. Current MAIL browser evidence is 1280×720 only because the browser surface could not resize or perform native Tab traversal. Interrupted attempts are aged after 15 minutes and notified without retry; the five-minute scan is unmeasured on a large ledger. The direct clean EF-only SQLite defect and synchronous AI accounting gap are closed; migration/reconciler dual ownership remains architectural debt.
+4 -4
View File
@@ -240,11 +240,11 @@ This queue records the highest-value work that can proceed without production cr
- **Required tests:** generated boundary/corrupt fixtures, harmless sleeping child, cancellation/restart cleanup, outage/no-fallback, container assertions.
- **Required browser verification:** synthetic CV upload status/failure; authorized private CV local-only only after safeguards.
- **Required production verification:** measured memory/CPU limits and canary synthetic extraction.
- **Status:** `IN PROGRESS`.
- **Blocker:** production sizing requires access; local Docker container assertions require a running daemon.
- **Evidence:** signature/container validation, page/pixel/decompression/output ceilings and stable parser errors have boundary tests; backend binary fallback is removed and raw unexpected failures are sanitized. Focused backend 51/51, full backend 719/719 and parser 32/32 pass.
- **Status:** `IMPLEMENTED — NOT VERIFIED`.
- **Blocker:** production sizing and canary require access; an actual Linux image build/runtime smoke requires a running Docker daemon.
- **Evidence:** `docs/verification/sec-007-parser-isolation.md`; child process/minimal environment/deadline/process-tree termination/stale cleanup tests pass; Compose resolves non-root-image/read-only/capability/PID/CPU/memory/tmpfs controls. Parser 37/37, focused backend 51/51 and full backend 719/719 pass.
- **Commit:** none.
- **Remaining work:** move decode into a deadline-bounded child process/process group, add descendant termination and cleanup tests, bound scheduling/backpressure, then apply and verify non-root/read-only/PID/CPU/memory/tmpfs container controls.
- **Remaining work:** build and run the Linux image, prove rlimits and container controls from inside it, measure benign PDF/DOCX/image memory/CPU, perform the synthetic browser upload failure/success journey, then canary only with explicit production authorization.
### SEC-008 — Recoverable attachment mutations