fix(cv): harden document parsing

Upgrade and hash-lock upload-facing parser dependencies, reject resource-heavy or mismatched inputs, remove unsafe backend binary fallbacks, and prevent internal parser failures from leaking to users.
This commit is contained in:
cesnimda
2026-08-30 11:00:52 +02:00
parent a74daa7aa4
commit a8bf505ce5
14 changed files with 1869 additions and 110 deletions
+6 -6
View File
@@ -44,7 +44,7 @@ Updated: 2026-08-30
### In progress
- SEC-006 parser dependency remediation is unblocked now that package-index access has been explicitly authorized. Resolve a compatible fixed parser stack, audit it, and rerun the benign extraction corpus before starting SEC-007 isolation.
- SEC-007 bounded document processing. Upload-facing parser dependencies are fixed and hash-locked; the sidecar now rejects signature mismatches, decompression/page/pixel/output excesses, the backend no longer retries PDF/DOCX/image parsing in-process, and unexpected parser details are sanitized. Child-process deadline/resource isolation and container controls remain.
### Remaining
@@ -108,17 +108,17 @@ Updated: 2026-08-30
- Playwright: 8/10 passed on the first complete run; both failures were ambiguous selectors in the newly responsive Career selector, not product failures. Both corrected focused regressions now pass (2/2); final full rerun remains in the end-of-batch gate.
- Manual desktop browser review: webpack development server rendered the new Career navigation and Overview correctly in dark mode; API-dependent profile status remained unavailable because the backend was not running for that isolated UI review.
- **Overall programme status:** Active but externally blocked. Eight packages are locally verified and twenty-five are implemented with verification incomplete. The prioritized admin-only version indicator, every immediate repository/browser item, SEC-009, the PROD-001 read-only inventory, and the PROD-003 safe benchmark harness are complete on the release branch.
- **Current work package:** SEC-006 parser dependency remediation; package-index access is now authorized.
- **Overall programme status:** Active but externally blocked. Eight packages are locally verified, twenty-six are implemented with verification incomplete, and SEC-007 is in progress. The prioritized admin-only version indicator, every immediate repository/browser item, SEC-009, the PROD-001 read-only inventory, and the PROD-003 safe benchmark harness are complete on the release branch.
- **Current work package:** SEC-007 bounded isolated document processing; SEC-006 is implemented locally and awaits Docker image smoke.
- **Completed work packages:** None are `DONE`; all repository security/AI packages still have applicable browser, provider and/or production gates.
- **Locally verified work:** SEC-001, SEC-002, SEC-003, SEC-005A, CORE-001, PROD-002, DEP-001 and VER-001 (`VERIFIED LOCALLY`).
- **Implemented, verification incomplete:** SEC-004, SEC-005B, SEC-008, SEC-009, CORE-002, BG-001, OPS-001A/B/C, POL-001/002, AI-001/002/003/004, UX-001/002/003, QA-001, CAREER-001/002, MAIL-001, JOBS-001/002 and PRODUCT-001 (`IMPLEMENTED — NOT VERIFIED`). Their safe repository/browser scope is implemented; production/native-device/provider/retention gates remain where recorded.
- **Implemented, verification incomplete:** SEC-004, SEC-005B, SEC-006, SEC-008, SEC-009, CORE-002, BG-001, OPS-001A/B/C, POL-001/002, AI-001/002/003/004, UX-001/002/003, QA-001, CAREER-001/002, MAIL-001, JOBS-001/002 and PRODUCT-001 (`IMPLEMENTED — NOT VERIFIED`). Their safe repository/browser scope is implemented; production/native-device/provider/retention gates remain where recorded.
- **Production-verified work:** None.
- **Blocked work:** PROD-001/003/004 and REL-001 require network/backup/model/deployment authority and unfinished dependencies. Real provider and live deletion/restore checks remain gated; DEP-001 awaits approved merge/live verification.
- **Deferred work:** None. Conditional multi-replica coordination, model deletion, realtime operation delivery and unrelated production changes remain outside current packages.
- **Immediate order:** all sixteen immediate repository items are complete locally, including the original UI/release queue plus SEC-009 cache/tombstone safety, worker restart clocks, universal AI accounting, email-token/Stripe lifecycle tests, exhaustive Job email selectors, the repaired migration chain, CV/public-edge hardening and measured admin/mail scaling. PROD-001 read-only evidence and the PROD-003 plan-only harness are also complete. The final audit is checking tooling/documentation before declaring only external blockers remain.
- **Status counts:** 8 `VERIFIED LOCALLY`; 25 `IMPLEMENTED — NOT VERIFIED`; 0 `IN PROGRESS`; 1 `NOT STARTED`; 5 `BLOCKED`; 0 `DONE`; 0 `DEFERRED`.
- **Test status:** backend 683/683; frontend 58/58 suites and 239/239 tests; AI sidecar 23/23; Ollama benchmark harness 5/5 plus safe dry-run; optimized production build/TypeScript; EF model parity; SQLite/MariaDB migration scripts; blank/idempotent/populated SQLite migration-chain tests; disposable fresh/restarted MariaDB 11.8 application startup; Docker Compose config; safe-failure deployment preflight; and Playwright 9/9. npm audit 0 evidence remains current because the lockfile did not change. Jest's slow/open-handle behavior remains recorded.
- **Status counts:** 8 `VERIFIED LOCALLY`; 26 `IMPLEMENTED — NOT VERIFIED`; 1 `IN PROGRESS`; 0 `NOT STARTED`; 4 `BLOCKED`; 0 `DONE`; 0 `DEFERRED`.
- **Test status:** backend 719/719; frontend 64/64 suites and 272/272 tests; ESLint zero findings; AI sidecar 32/32; Ollama benchmark harness 5/5 plus safe dry-run; optimized production build/TypeScript; EF model parity; SQLite/MariaDB migration scripts; blank/idempotent/populated SQLite migration-chain tests; disposable fresh/restarted MariaDB 11.8 application startup; Docker Compose config; safe-failure deployment preflight; and Playwright 9/9. npm audit remains at zero. The parser image build is pending because Docker Desktop's daemon is offline. Jest's slow/open-handle behavior and six Python deprecation warnings remain recorded.
- **Deployment status:** Gitea pull-request run 609 passed the complete CI job in 4m20s. Deploy was intentionally skipped because the workflow deploys only a `push` to `main`; live remains unchanged. No merge/deployment was performed directly, no production migrations were run and the AI operation worker remains disabled by default.
- **Production status:** State unchanged. Sanitized read-only SSH inventory was performed; no logs, prompts, private rows/content or secret values were read, and no provider/model call, model pull, service restart, file/config change, backup, restore, migration or deployment occurred. It confirmed all-interface Ollama/frontend listeners and stale database-only backups as rollout blockers.
- **Known regressions:** None found by automated/local browser checks. Jest still needs `--forceExit` and reports its existing open-handle notice. Email-provider/send tests are fake/local only; real delivery is not claimed. Current MAIL browser evidence is 1280×720 only because the browser surface could not resize or perform native Tab traversal. Interrupted attempts are aged after 15 minutes and notified without retry; the five-minute scan is unmeasured on a large ledger. The direct clean EF-only SQLite defect and synchronous AI accounting gap are closed; migration/reconciler dual ownership remains architectural debt.
+8 -8
View File
@@ -222,11 +222,11 @@ This queue records the highest-value work that can proceed without production cr
- **Required tests:** dependency resolution/audit, Python suite, generated benign corpus.
- **Required browser verification:** none for dependency-only package.
- **Required production verification:** image digest and smoke before activation.
- **Status:** `BLOCKED`.
- **Blocker:** repository instructions prohibit internet/package resolution without explicit permission; fixed-version compatibility cannot be resolved or verified offline.
- **Evidence:** audit lists reachable Pillow/pypdf/multipart/Starlette advisories and compatibility conflict.
- **Status:** `IMPLEMENTED — NOT VERIFIED`.
- **Blocker:** Docker Desktop is installed but its Linux daemon is offline, so the hash-locked production image has not been built locally.
- **Evidence:** `docs/verification/sec-006-parser-dependencies.md`; upload-facing dependency audit is clear; clean hash install succeeds; parser suite 32/32; backend 719/719.
- **Commit:** none.
- **Remaining work:** explicit package-index permission, compatible fixed-version resolution, lock/hash refresh, audit, benign corpus parity and image smoke; do not execute malicious files.
- **Remaining work:** build the production image from `requirements-linux.lock` and run its health/benign extraction smoke when a Docker daemon is available. Torch/Transformers advisories remain separately tracked under JT-017 and are not accepted as upload-parser findings.
### SEC-007 — Bounded isolated document processing
@@ -240,11 +240,11 @@ This queue records the highest-value work that can proceed without production cr
- **Required tests:** generated boundary/corrupt fixtures, harmless sleeping child, cancellation/restart cleanup, outage/no-fallback, container assertions.
- **Required browser verification:** synthetic CV upload status/failure; authorized private CV local-only only after safeguards.
- **Required production verification:** measured memory/CPU limits and canary synthetic extraction.
- **Status:** `NOT STARTED`.
- **Blocker:** follows dependency update; production sizing requires access.
- **Evidence:** audit parser call path and limits design.
- **Status:** `IN PROGRESS`.
- **Blocker:** production sizing requires access; local Docker container assertions require a running daemon.
- **Evidence:** signature/container validation, page/pixel/decompression/output ceilings and stable parser errors have boundary tests; backend binary fallback is removed and raw unexpected failures are sanitized. Focused backend 51/51, full backend 719/719 and parser 32/32 pass.
- **Commit:** none.
- **Remaining work:** split behavioral and container commits if needed.
- **Remaining work:** move decode into a deadline-bounded child process/process group, add descendant termination and cleanup tests, bound scheduling/backpressure, then apply and verify non-root/read-only/PID/CPU/memory/tmpfs container controls.
### SEC-008 — Recoverable attachment mutations