cesnimda
36 lines
5.6 KiB
Markdown
36 lines
5.6 KiB
Markdown
# Docling extraction and local-model boundary
|
||
|
||
26 October 2026. Development implementation; production remains NO-GO.
|
||
|
||
The existing FastAPI sidecar owns `POST /cv-import/extract`. It accepts PDF/DOCX bytes and returns text, MIME/name metadata, page count where available, ordered `blocks[{id,text,kind,page}]`, explicit warnings and converter method. Existing `/extract-text` remains available for legacy text/image/history compatibility. The API transports and persists the structured evidence; it does not run an in-process binary parser.
|
||
|
||
## Reproducible dependencies
|
||
|
||
`tools/summarizer/requirements-cv-import.txt` and its universal Python 3.12 hash lock pin Docling slim 2.142.1, core 2.104.0, parse 7.23.0, python-docx 1.2.0, Pydantic 2.14.1 and Pydantic settings 2.16.1. Verify against the lock rather than assuming the old `.venv` is current. This runtime is separate because the legacy model dependency set pins incompatible python-docx/Pydantic versions. No inherited model pins or caches were replaced.
|
||
|
||
Release/API verification used actual PyPI wheel metadata and installed source, including `DocumentConverter`, `WordFormatOption`, `NativePdfFormatOption`, `NativePdfPipelineOptions`, `DocumentStream`, `ConversionStatus`, `iterate_items` and `TableItem.export_to_markdown`. Sources: [PyPI Docling](https://pypi.org/project/docling/), [Docling pipeline options](https://docling-project.github.io/docling/reference/pipeline_options/), [local/remote configuration](https://github.com/docling-project/docling/blob/main/docs/usage/advanced_options.md). Search indexing was stale at 2.129.0; the actual package index and installed source identified 2.142.1.
|
||
|
||
Create a dedicated interpreter and install with `pip install --require-hashes -r tools/summarizer/requirements-cv-import.lock`. Set `CV_IMPORT_PYTHON` to its absolute executable in development. The main sidecar can retain its existing interpreter. No Docker/production service was changed.
|
||
|
||
## Conversion and limits
|
||
|
||
DOCX uses model-free Word conversion with ordered headings, lists, paragraphs and tables. It cannot establish physical page numbers; `page=null` is deliberate. PDF uses the supported **NativePdfPipeline**, without layout/table/OCR models. Its parser order and text-cell grouping are not a qualified reconstruction of complex multi-column/table layouts. Every result declares `pdf_native_order_no_layout_or_tables` or `docx_page_numbers_unavailable`, and `ocr_disabled`.
|
||
|
||
Remote services, external plugins, picture enrichment, chart rendering, page/bitmap rendering and OCR are disabled. The child uses offline model variables and a Python audit hook denying sockets and subprocesses. It accepts only validated local byte streams. There were no model downloads, generative inference calls or production/cache modifications.
|
||
|
||
The existing subprocess runner bounds concurrency to one, input to 5 MiB, PDF to 40 pages, extracted text to 200,000 characters, blocks to 5,000 and JSON to 1 MiB. DOCX validation retains the existing entry/expanded-size/ratio limits and adds safe XML/external-content checks. Parent deadlines, process-tree termination, private work directories, secret-stripped environment, stale-work cleanup and sanitized failures remain. New settings are bounded: `CV_IMPORT_TIMEOUT_SECONDS` default 40 (5–120), `CV_IMPORT_CPU_SECONDS` default 30, `CV_IMPORT_MEMORY_MIB` default 1024 (512–2048). POSIX adds CPU/address-space limits; Windows currently has wall-clock/process termination rather than a kernel-enforced memory ceiling. This platform difference is an open resource qualification limitation.
|
||
|
||
Scans and image-only PDFs fail cleanly with OCR disabled. Encrypted documents, active/embedded content, unsafe external relationships, malformed/bomb archives and excessive inputs are rejected. PDF table-cell relationships may require manual review. No arbitrary confidence percentage is exposed.
|
||
|
||
## AI adapter
|
||
|
||
`cv_import_ai.py` provides a configurable local Ollama transport, versioned source-as-data instruction, constrained output schema and strict field/quote/block validation. It defaults disabled and has no automatic model selection. Synthetic `generate` injection tests malformed output, invented facts, date precision, unknown blocks/paths and unsafe provider addresses. Suggestions are labelled `model-assisted` / `needs_review` and never directly write a profile.
|
||
|
||
The adapter is a qualification seam; the active import path remains deterministic and does not call it. Actual provider/model compatibility, factual relationship validation, EN/Bokmål language quality, adversarial semantics, GPU behavior and model accuracy remain **BLOCKED**. Enabling production inference requires a separately approved immutable artifact and isolated qualification; environment configuration is not acceptance.
|
||
|
||
## Current focused evidence
|
||
|
||
- `.artifacts/cv-import/sidecar-focused-02.log`: 23 passed, including real native PDF/DOCX conversion and synthetic model-adapter tests; no generative inference.
|
||
- `.artifacts/cv-import/python-offline-01.log`: 73 passed in immutable existing Python runtime `sha256:2fe77e47a6c5ecea66764dbd6264049724f24849141e16af5491c2de34af53c1`, network disabled/read-only source/model loading disabled. This covers app/parser-security/grounding/adapter behavior, not new Docling dependency compatibility in that legacy image. One pytest cache-option warning retained.
|
||
- `.artifacts/cv-import/extraction-benchmark-01`: 26 text-bearing generated documents converted; the separate synthetic scan was rejected. Actual source hashes, per-process wall/CPU/peak-working-set/input/output measurements are recorded. Later benchmark attempts supersede generator defects explicitly; no historical result is overwritten.
|