fix(cv): harden document parsing
Upgrade and hash-lock upload-facing parser dependencies, reject resource-heavy or mismatched inputs, remove unsafe backend binary fallbacks, and prevent internal parser failures from leaking to users.
This commit is contained in:
@@ -221,3 +221,4 @@ Output was reduced to filenames and commit counts. The token artifact appears un
|
||||
| V-187 | Focused public-CV/rate-key tests, focused authenticated-preview Jest, full backend, production frontend build and tracked-tree secret-pattern scan | Repository root / `job-tracker-ui` | Close the remaining low-risk public PDF, preview sandbox and tracked JWT hardening findings | PASS — backend focused 4/4 and full 681/681; preview Jest 15/15; production build passes. Two clients receive independent same-slug PDF partitions, both authenticated preview iframes disable scripts with a sandbox, and the values-suppressed tracked-tree scan finds no JWT/private-key pattern outside audit/operations records | No stress test, production request, history rewrite or Data Protection inspection. A coordinated history rewrite remains outside this change | JT-023/JT-025 current-tree gaps closed; expired JT-020 artifact removed from current tree |
|
||||
| V-188 | 75-user query-count fixture, 205-message/two-tenant pagination fixture, focused/full backend and frontend, optimized build | Repository root / `job-tracker-ui` | Remove JT-021's confirmed N+1 and silent 200-message ceiling without breaking old clients | PASS — admin list performs two reads independent of 75 users; page 3 returns the final 5 of 205 owned messages and excludes another tenant. Focused backend 9/9, correspondence UI 15/15, backend 683/683, frontend 58 suites/239 tests and build pass. UI page navigation and filter reset are covered; the legacy inbox endpoint remains unchanged | Synthetic SQLite/InMemory/JSDOM only; no provider mailbox, production dataset or p95 load test. Current page linked/inbound chips intentionally describe the visible page | JT-021 repository defects closed; provider/production capacity remains operational evidence |
|
||||
| V-189 | Fresh Python 3.12 virtual environments; `pytest -q`; `npm run lint`; full frontend Jest; `npm run build`; `npm audit --audit-level=moderate` | Repository root / `tools/summarizer` / `job-tracker-ui` | Restore the local Python test environment and establish a reproducible zero-warning frontend lint gate | PASS — Python 3.12.10; sidecar 26/26; ESLint zero findings; frontend 64 suites/272 tests; optimized build/TypeScript; npm audit zero vulnerabilities | Five existing SWIG deprecation warnings; Jest retains its documented force-exit/open-handle notice. ESLint 9 is intentionally pinned because the Next 16 React plugin is not ESLint 10 compatible | Local Python/frontend quality gates restored; parser dependency remediation proceeds separately under SEC-006 |
|
||||
| V-190 | Compatible parser dependency resolution; Linux CPU hash lock; clean hash install; `pip-audit`; generated parser boundary tests; focused/full backend and sidecar suites; Docker daemon probe | Repository root / `tools/summarizer` | Remove reachable upload-parser advisories and close unsafe fallback/resource-boundary paths without parsing hostile fixtures | PASS/PARTIAL — FastAPI 0.141.1/Starlette 1.6.0, Pillow 12.3.0, pypdf 6.16.2 and python-multipart 0.0.32 resolve and install from hashes; upload-facing packages audit clear; parser 32/32, focused backend 51/51 and backend 719/719. Signature/container/page/pixel/decompression/output limits pass; backend binary fallback is removed and unexpected failures are sanitized | Docker daemon unavailable, so production image smoke/container assertions did not run. `pip-audit` reports 45 Torch/Transformers model-stack advisories tracked separately under JT-017. One Starlette/httpx test-client warning and five SWIG warnings remain | SEC-006 implemented locally; SEC-007 boundary work in progress, with child-process and container isolation still required |
|
||||
|
||||
@@ -0,0 +1,40 @@
|
||||
# SEC-006 parser dependency verification
|
||||
|
||||
## Result
|
||||
|
||||
The libraries reached by untrusted CV upload parsing were upgraded to compatible fixed releases and locked with hashes for the Linux CPU image. The dependency-only package is implemented locally but is not production-verified because the local Docker Desktop daemon is unavailable.
|
||||
|
||||
## Fixed parser stack
|
||||
|
||||
- Python base: `3.12.10-slim-bookworm`
|
||||
- FastAPI: `0.141.1`
|
||||
- Starlette: `1.6.0`
|
||||
- Pillow: `12.3.0`
|
||||
- pypdf: `6.16.2`
|
||||
- python-multipart: `0.0.32`
|
||||
- Reproducible Linux CPU dependency set: `tools/summarizer/requirements-linux.lock`
|
||||
|
||||
The lock was generated for Python 3.12 on `x86_64-unknown-linux-gnu` with the PyTorch CPU index and contains hashes for every resolved package. A clean local environment installed the generated set with `--require-hashes` and imported the upgraded parser packages successfully.
|
||||
|
||||
## Advisory review
|
||||
|
||||
`pip-audit -r requirements.txt` reports no advisory for FastAPI, Starlette, Pillow, pypdf, PyMuPDF, python-docx, pytesseract or python-multipart. It reports 45 advisories across `torch==2.6.0` and `transformers==4.48.3`.
|
||||
|
||||
Those remaining findings belong to the model-loading stack rather than the CV file decoder paths remediated by SEC-006. They remain open under JT-017; this record does not accept or close them.
|
||||
|
||||
## Regression proof
|
||||
|
||||
- Python parser suite: 32/32 passed.
|
||||
- Focused CV extraction/backend suite: 51/51 passed.
|
||||
- Full backend suite: 719/719 passed.
|
||||
- Generated boundary cases cover mismatched signatures, invalid text bytes, excessive PDF pages, excessive image pixels, excessive DOCX entries and sanitized unexpected parser failures.
|
||||
- The API no longer performs local PDF, DOCX or image parsing when the sidecar returns no text. A bounded UTF-8 text/Markdown fallback remains.
|
||||
|
||||
## Remaining verification
|
||||
|
||||
`docker info` fails because `dockerDesktopLinuxEngine` is not running. When a daemon is available:
|
||||
|
||||
1. Build `tools/summarizer/Dockerfile` without changing the lock.
|
||||
2. Start the image with the intended SEC-007 runtime restrictions.
|
||||
3. Run health plus benign TXT, Markdown, PDF, DOCX and image extraction smoke tests.
|
||||
4. Record the image digest and measured memory/CPU use before any production activation.
|
||||
@@ -44,7 +44,7 @@ Updated: 2026-08-30
|
||||
|
||||
### In progress
|
||||
|
||||
- SEC-006 parser dependency remediation is unblocked now that package-index access has been explicitly authorized. Resolve a compatible fixed parser stack, audit it, and rerun the benign extraction corpus before starting SEC-007 isolation.
|
||||
- SEC-007 bounded document processing. Upload-facing parser dependencies are fixed and hash-locked; the sidecar now rejects signature mismatches, decompression/page/pixel/output excesses, the backend no longer retries PDF/DOCX/image parsing in-process, and unexpected parser details are sanitized. Child-process deadline/resource isolation and container controls remain.
|
||||
|
||||
### Remaining
|
||||
|
||||
@@ -108,17 +108,17 @@ Updated: 2026-08-30
|
||||
- Playwright: 8/10 passed on the first complete run; both failures were ambiguous selectors in the newly responsive Career selector, not product failures. Both corrected focused regressions now pass (2/2); final full rerun remains in the end-of-batch gate.
|
||||
- Manual desktop browser review: webpack development server rendered the new Career navigation and Overview correctly in dark mode; API-dependent profile status remained unavailable because the backend was not running for that isolated UI review.
|
||||
|
||||
- **Overall programme status:** Active but externally blocked. Eight packages are locally verified and twenty-five are implemented with verification incomplete. The prioritized admin-only version indicator, every immediate repository/browser item, SEC-009, the PROD-001 read-only inventory, and the PROD-003 safe benchmark harness are complete on the release branch.
|
||||
- **Current work package:** SEC-006 parser dependency remediation; package-index access is now authorized.
|
||||
- **Overall programme status:** Active but externally blocked. Eight packages are locally verified, twenty-six are implemented with verification incomplete, and SEC-007 is in progress. The prioritized admin-only version indicator, every immediate repository/browser item, SEC-009, the PROD-001 read-only inventory, and the PROD-003 safe benchmark harness are complete on the release branch.
|
||||
- **Current work package:** SEC-007 bounded isolated document processing; SEC-006 is implemented locally and awaits Docker image smoke.
|
||||
- **Completed work packages:** None are `DONE`; all repository security/AI packages still have applicable browser, provider and/or production gates.
|
||||
- **Locally verified work:** SEC-001, SEC-002, SEC-003, SEC-005A, CORE-001, PROD-002, DEP-001 and VER-001 (`VERIFIED LOCALLY`).
|
||||
- **Implemented, verification incomplete:** SEC-004, SEC-005B, SEC-008, SEC-009, CORE-002, BG-001, OPS-001A/B/C, POL-001/002, AI-001/002/003/004, UX-001/002/003, QA-001, CAREER-001/002, MAIL-001, JOBS-001/002 and PRODUCT-001 (`IMPLEMENTED — NOT VERIFIED`). Their safe repository/browser scope is implemented; production/native-device/provider/retention gates remain where recorded.
|
||||
- **Implemented, verification incomplete:** SEC-004, SEC-005B, SEC-006, SEC-008, SEC-009, CORE-002, BG-001, OPS-001A/B/C, POL-001/002, AI-001/002/003/004, UX-001/002/003, QA-001, CAREER-001/002, MAIL-001, JOBS-001/002 and PRODUCT-001 (`IMPLEMENTED — NOT VERIFIED`). Their safe repository/browser scope is implemented; production/native-device/provider/retention gates remain where recorded.
|
||||
- **Production-verified work:** None.
|
||||
- **Blocked work:** PROD-001/003/004 and REL-001 require network/backup/model/deployment authority and unfinished dependencies. Real provider and live deletion/restore checks remain gated; DEP-001 awaits approved merge/live verification.
|
||||
- **Deferred work:** None. Conditional multi-replica coordination, model deletion, realtime operation delivery and unrelated production changes remain outside current packages.
|
||||
- **Immediate order:** all sixteen immediate repository items are complete locally, including the original UI/release queue plus SEC-009 cache/tombstone safety, worker restart clocks, universal AI accounting, email-token/Stripe lifecycle tests, exhaustive Job email selectors, the repaired migration chain, CV/public-edge hardening and measured admin/mail scaling. PROD-001 read-only evidence and the PROD-003 plan-only harness are also complete. The final audit is checking tooling/documentation before declaring only external blockers remain.
|
||||
- **Status counts:** 8 `VERIFIED LOCALLY`; 25 `IMPLEMENTED — NOT VERIFIED`; 0 `IN PROGRESS`; 1 `NOT STARTED`; 5 `BLOCKED`; 0 `DONE`; 0 `DEFERRED`.
|
||||
- **Test status:** backend 683/683; frontend 58/58 suites and 239/239 tests; AI sidecar 23/23; Ollama benchmark harness 5/5 plus safe dry-run; optimized production build/TypeScript; EF model parity; SQLite/MariaDB migration scripts; blank/idempotent/populated SQLite migration-chain tests; disposable fresh/restarted MariaDB 11.8 application startup; Docker Compose config; safe-failure deployment preflight; and Playwright 9/9. npm audit 0 evidence remains current because the lockfile did not change. Jest's slow/open-handle behavior remains recorded.
|
||||
- **Status counts:** 8 `VERIFIED LOCALLY`; 26 `IMPLEMENTED — NOT VERIFIED`; 1 `IN PROGRESS`; 0 `NOT STARTED`; 4 `BLOCKED`; 0 `DONE`; 0 `DEFERRED`.
|
||||
- **Test status:** backend 719/719; frontend 64/64 suites and 272/272 tests; ESLint zero findings; AI sidecar 32/32; Ollama benchmark harness 5/5 plus safe dry-run; optimized production build/TypeScript; EF model parity; SQLite/MariaDB migration scripts; blank/idempotent/populated SQLite migration-chain tests; disposable fresh/restarted MariaDB 11.8 application startup; Docker Compose config; safe-failure deployment preflight; and Playwright 9/9. npm audit remains at zero. The parser image build is pending because Docker Desktop's daemon is offline. Jest's slow/open-handle behavior and six Python deprecation warnings remain recorded.
|
||||
- **Deployment status:** Gitea pull-request run 609 passed the complete CI job in 4m20s. Deploy was intentionally skipped because the workflow deploys only a `push` to `main`; live remains unchanged. No merge/deployment was performed directly, no production migrations were run and the AI operation worker remains disabled by default.
|
||||
- **Production status:** State unchanged. Sanitized read-only SSH inventory was performed; no logs, prompts, private rows/content or secret values were read, and no provider/model call, model pull, service restart, file/config change, backup, restore, migration or deployment occurred. It confirmed all-interface Ollama/frontend listeners and stale database-only backups as rollout blockers.
|
||||
- **Known regressions:** None found by automated/local browser checks. Jest still needs `--forceExit` and reports its existing open-handle notice. Email-provider/send tests are fake/local only; real delivery is not claimed. Current MAIL browser evidence is 1280×720 only because the browser surface could not resize or perform native Tab traversal. Interrupted attempts are aged after 15 minutes and notified without retry; the five-minute scan is unmeasured on a large ledger. The direct clean EF-only SQLite defect and synchronous AI accounting gap are closed; migration/reconciler dual ownership remains architectural debt.
|
||||
|
||||
@@ -222,11 +222,11 @@ This queue records the highest-value work that can proceed without production cr
|
||||
- **Required tests:** dependency resolution/audit, Python suite, generated benign corpus.
|
||||
- **Required browser verification:** none for dependency-only package.
|
||||
- **Required production verification:** image digest and smoke before activation.
|
||||
- **Status:** `BLOCKED`.
|
||||
- **Blocker:** repository instructions prohibit internet/package resolution without explicit permission; fixed-version compatibility cannot be resolved or verified offline.
|
||||
- **Evidence:** audit lists reachable Pillow/pypdf/multipart/Starlette advisories and compatibility conflict.
|
||||
- **Status:** `IMPLEMENTED — NOT VERIFIED`.
|
||||
- **Blocker:** Docker Desktop is installed but its Linux daemon is offline, so the hash-locked production image has not been built locally.
|
||||
- **Evidence:** `docs/verification/sec-006-parser-dependencies.md`; upload-facing dependency audit is clear; clean hash install succeeds; parser suite 32/32; backend 719/719.
|
||||
- **Commit:** none.
|
||||
- **Remaining work:** explicit package-index permission, compatible fixed-version resolution, lock/hash refresh, audit, benign corpus parity and image smoke; do not execute malicious files.
|
||||
- **Remaining work:** build the production image from `requirements-linux.lock` and run its health/benign extraction smoke when a Docker daemon is available. Torch/Transformers advisories remain separately tracked under JT-017 and are not accepted as upload-parser findings.
|
||||
|
||||
### SEC-007 — Bounded isolated document processing
|
||||
|
||||
@@ -240,11 +240,11 @@ This queue records the highest-value work that can proceed without production cr
|
||||
- **Required tests:** generated boundary/corrupt fixtures, harmless sleeping child, cancellation/restart cleanup, outage/no-fallback, container assertions.
|
||||
- **Required browser verification:** synthetic CV upload status/failure; authorized private CV local-only only after safeguards.
|
||||
- **Required production verification:** measured memory/CPU limits and canary synthetic extraction.
|
||||
- **Status:** `NOT STARTED`.
|
||||
- **Blocker:** follows dependency update; production sizing requires access.
|
||||
- **Evidence:** audit parser call path and limits design.
|
||||
- **Status:** `IN PROGRESS`.
|
||||
- **Blocker:** production sizing requires access; local Docker container assertions require a running daemon.
|
||||
- **Evidence:** signature/container validation, page/pixel/decompression/output ceilings and stable parser errors have boundary tests; backend binary fallback is removed and raw unexpected failures are sanitized. Focused backend 51/51, full backend 719/719 and parser 32/32 pass.
|
||||
- **Commit:** none.
|
||||
- **Remaining work:** split behavioral and container commits if needed.
|
||||
- **Remaining work:** move decode into a deadline-bounded child process/process group, add descendant termination and cleanup tests, bound scheduling/backpressure, then apply and verify non-root/read-only/PID/CPU/memory/tmpfs container controls.
|
||||
|
||||
### SEC-008 — Recoverable attachment mutations
|
||||
|
||||
|
||||
Reference in New Issue
Block a user