feat(cv-import): add bounded model-free Docling extraction

This commit is contained in:
cesnimda
2026-10-26 14:17:43 +01:00
parent 731536db26
commit 3ea8ff9d62
7 changed files with 1593 additions and 11 deletions
+35
View File
@@ -0,0 +1,35 @@
# Docling extraction and local-model boundary
26 October 2026. Development implementation; production remains NO-GO.
The existing FastAPI sidecar owns `POST /cv-import/extract`. It accepts PDF/DOCX bytes and returns text, MIME/name metadata, page count where available, ordered `blocks[{id,text,kind,page}]`, explicit warnings and converter method. Existing `/extract-text` remains available for legacy text/image/history compatibility. The API transports and persists the structured evidence; it does not run an in-process binary parser.
## Reproducible dependencies
`tools/summarizer/requirements-cv-import.txt` and its universal Python 3.12 hash lock pin Docling slim 2.142.1, core 2.104.0, parse 7.23.0, python-docx 1.2.0, Pydantic 2.14.1 and Pydantic settings 2.16.1. Verify against the lock rather than assuming the old `.venv` is current. This runtime is separate because the legacy model dependency set pins incompatible python-docx/Pydantic versions. No inherited model pins or caches were replaced.
Release/API verification used actual PyPI wheel metadata and installed source, including `DocumentConverter`, `WordFormatOption`, `NativePdfFormatOption`, `NativePdfPipelineOptions`, `DocumentStream`, `ConversionStatus`, `iterate_items` and `TableItem.export_to_markdown`. Sources: [PyPI Docling](https://pypi.org/project/docling/), [Docling pipeline options](https://docling-project.github.io/docling/reference/pipeline_options/), [local/remote configuration](https://github.com/docling-project/docling/blob/main/docs/usage/advanced_options.md). Search indexing was stale at 2.129.0; the actual package index and installed source identified 2.142.1.
Create a dedicated interpreter and install with `pip install --require-hashes -r tools/summarizer/requirements-cv-import.lock`. Set `CV_IMPORT_PYTHON` to its absolute executable in development. The main sidecar can retain its existing interpreter. No Docker/production service was changed.
## Conversion and limits
DOCX uses model-free Word conversion with ordered headings, lists, paragraphs and tables. It cannot establish physical page numbers; `page=null` is deliberate. PDF uses the supported **NativePdfPipeline**, without layout/table/OCR models. Its parser order and text-cell grouping are not a qualified reconstruction of complex multi-column/table layouts. Every result declares `pdf_native_order_no_layout_or_tables` or `docx_page_numbers_unavailable`, and `ocr_disabled`.
Remote services, external plugins, picture enrichment, chart rendering, page/bitmap rendering and OCR are disabled. The child uses offline model variables and a Python audit hook denying sockets and subprocesses. It accepts only validated local byte streams. There were no model downloads, generative inference calls or production/cache modifications.
The existing subprocess runner bounds concurrency to one, input to 5 MiB, PDF to 40 pages, extracted text to 200,000 characters, blocks to 5,000 and JSON to 1 MiB. DOCX validation retains the existing entry/expanded-size/ratio limits and adds safe XML/external-content checks. Parent deadlines, process-tree termination, private work directories, secret-stripped environment, stale-work cleanup and sanitized failures remain. New settings are bounded: `CV_IMPORT_TIMEOUT_SECONDS` default 40 (5–120), `CV_IMPORT_CPU_SECONDS` default 30, `CV_IMPORT_MEMORY_MIB` default 1024 (512–2048). POSIX adds CPU/address-space limits; Windows currently has wall-clock/process termination rather than a kernel-enforced memory ceiling. This platform difference is an open resource qualification limitation.
Scans and image-only PDFs fail cleanly with OCR disabled. Encrypted documents, active/embedded content, unsafe external relationships, malformed/bomb archives and excessive inputs are rejected. PDF table-cell relationships may require manual review. No arbitrary confidence percentage is exposed.
## AI adapter
`cv_import_ai.py` provides a configurable local Ollama transport, versioned source-as-data instruction, constrained output schema and strict field/quote/block validation. It defaults disabled and has no automatic model selection. Synthetic `generate` injection tests malformed output, invented facts, date precision, unknown blocks/paths and unsafe provider addresses. Suggestions are labelled `model-assisted` / `needs_review` and never directly write a profile.
The adapter is a qualification seam; the active import path remains deterministic and does not call it. Actual provider/model compatibility, factual relationship validation, EN/Bokmål language quality, adversarial semantics, GPU behavior and model accuracy remain **BLOCKED**. Enabling production inference requires a separately approved immutable artifact and isolated qualification; environment configuration is not acceptance.
## Current focused evidence
- `.artifacts/cv-import/sidecar-focused-02.log`: 23 passed, including real native PDF/DOCX conversion and synthetic model-adapter tests; no generative inference.
- `.artifacts/cv-import/python-offline-01.log`: 73 passed in immutable existing Python runtime `sha256:2fe77e47a6c5ecea66764dbd6264049724f24849141e16af5491c2de34af53c1`, network disabled/read-only source/model loading disabled. This covers app/parser-security/grounding/adapter behavior, not new Docling dependency compatibility in that legacy image. One pytest cache-option warning retained.
- `.artifacts/cv-import/extraction-benchmark-01`: 26 text-bearing generated documents converted; the separate synthetic scan was rejected. Actual source hashes, per-process wall/CPU/peak-working-set/input/output measurements are recorded. Later benchmark attempts supersede generator defects explicitly; no historical result is overwritten.
+65 -10
View File
@@ -110,6 +110,10 @@ PARSER_CHILD_PATH = Path(__file__).with_name("parser_child.py")
PARSER_TIMEOUT_SECONDS = max(5, min(int(os.getenv("PARSER_TIMEOUT_SECONDS", "25")), 120)) PARSER_TIMEOUT_SECONDS = max(5, min(int(os.getenv("PARSER_TIMEOUT_SECONDS", "25")), 120))
PARSER_CPU_SECONDS = max(2, min(int(os.getenv("PARSER_CPU_SECONDS", "20")), PARSER_TIMEOUT_SECONDS)) PARSER_CPU_SECONDS = max(2, min(int(os.getenv("PARSER_CPU_SECONDS", "20")), PARSER_TIMEOUT_SECONDS))
PARSER_MEMORY_BYTES = max(256, min(int(os.getenv("PARSER_MEMORY_MIB", "512")), 2048)) * 1024 * 1024 PARSER_MEMORY_BYTES = max(256, min(int(os.getenv("PARSER_MEMORY_MIB", "512")), 2048)) * 1024 * 1024
CV_IMPORT_PYTHON = os.getenv("CV_IMPORT_PYTHON", sys.executable)
CV_IMPORT_TIMEOUT_SECONDS = max(5, min(int(os.getenv("CV_IMPORT_TIMEOUT_SECONDS", "40")), 120))
CV_IMPORT_CPU_SECONDS = max(2, min(int(os.getenv("CV_IMPORT_CPU_SECONDS", "30")), CV_IMPORT_TIMEOUT_SECONDS))
CV_IMPORT_MEMORY_BYTES = max(512, min(int(os.getenv("CV_IMPORT_MEMORY_MIB", "1024")), 2048)) * 1024 * 1024
PARSER_MAX_RESULT_BYTES = 1024 * 1024 PARSER_MAX_RESULT_BYTES = 1024 * 1024
_parser_capacity = threading.BoundedSemaphore(value=1) _parser_capacity = threading.BoundedSemaphore(value=1)
PARSER_TEMP_ROOT = Path(os.getenv("PARSER_TEMP_ROOT", tempfile.gettempdir()), "jobtracker-cv-parser").resolve() PARSER_TEMP_ROOT = Path(os.getenv("PARSER_TEMP_ROOT", tempfile.gettempdir()), "jobtracker-cv-parser").resolve()
@@ -1184,6 +1188,15 @@ _PARSER_ERRORS = {
"docx_entry_size": (422, "The DOCX contains an oversized file."), "docx_entry_size": (422, "The DOCX contains an oversized file."),
"docx_ratio": (422, "The DOCX compression ratio is not supported."), "docx_ratio": (422, "The DOCX compression ratio is not supported."),
"docx_expanded_size": (422, "The DOCX expands beyond the supported size."), "docx_expanded_size": (422, "The DOCX expands beyond the supported size."),
"docx_unsafe_archive": (422, "The DOCX archive structure is not supported."),
"pdf_signature": (400, "The uploaded file does not match its PDF extension."),
"pdf_invalid": (422, "The PDF is malformed or unsupported."),
"pdf_encrypted": (422, "Encrypted PDFs are not supported."),
"pdf_object_limit": (422, "The PDF structure exceeds the supported limits."),
"external_content": (422, "Active or external document content is not supported."),
"block_limit": (422, "The document contains too many text blocks."),
"file_size_limit": (400, "The uploaded file size is not supported."),
"docling_unavailable": (503, "Document conversion runtime is unavailable."),
"pdf_page_limit": (422, "The PDF contains too many pages."), "pdf_page_limit": (422, "The PDF contains too many pages."),
"pdf_ocr_limit": (422, "The PDF requires too much OCR processing."), "pdf_ocr_limit": (422, "The PDF requires too much OCR processing."),
"image_dimensions": (422, "The document image dimensions are not supported."), "image_dimensions": (422, "The document image dimensions are not supported."),
@@ -1203,6 +1216,9 @@ def _parser_environment(temp_root: str) -> dict[str, str]:
"PYTHONDONTWRITEBYTECODE": "1", "PYTHONDONTWRITEBYTECODE": "1",
"OMP_NUM_THREADS": "1", "OMP_NUM_THREADS": "1",
"OPENBLAS_NUM_THREADS": "1", "OPENBLAS_NUM_THREADS": "1",
"HF_HUB_OFFLINE": "1",
"TRANSFORMERS_OFFLINE": "1",
"HF_HUB_DISABLE_TELEMETRY": "1",
"TMPDIR": temp_root, "TMPDIR": temp_root,
"TEMP": temp_root, "TEMP": temp_root,
"TMP": temp_root, "TMP": temp_root,
@@ -1259,7 +1275,7 @@ def _raise_parser_error(code: str) -> None:
raise HTTPException(status_code=status_code, detail=detail) raise HTTPException(status_code=status_code, detail=detail)
def _run_parser_child(extension: str, data: bytes) -> dict[str, object]: def _run_parser_child(extension: str, data: bytes, *, document: bool = False) -> dict[str, object]:
if not _parser_capacity.acquire(blocking=False): if not _parser_capacity.acquire(blocking=False):
raise HTTPException(status_code=503, detail="Document extraction is busy. Try again shortly.") raise HTTPException(status_code=503, detail="Document extraction is busy. Try again shortly.")
try: try:
@@ -1268,14 +1284,16 @@ def _run_parser_child(extension: str, data: bytes) -> dict[str, object]:
output_path = Path(temp_root, "result.json") output_path = Path(temp_root, "result.json")
input_path.write_bytes(data) input_path.write_bytes(data)
command = [ command = [
sys.executable, CV_IMPORT_PYTHON if document else sys.executable,
str(PARSER_CHILD_PATH), str(PARSER_CHILD_PATH),
"--input", str(input_path), "--input", str(input_path),
"--output", str(output_path), "--output", str(output_path),
"--extension", extension, "--extension", extension,
"--cpu-seconds", str(PARSER_CPU_SECONDS), "--cpu-seconds", str(CV_IMPORT_CPU_SECONDS if document else PARSER_CPU_SECONDS),
"--memory-bytes", str(PARSER_MEMORY_BYTES), "--memory-bytes", str(CV_IMPORT_MEMORY_BYTES if document else PARSER_MEMORY_BYTES),
] ]
if document:
command.extend(["--mode", "document"])
creation_flags = subprocess.CREATE_NEW_PROCESS_GROUP if os.name == "nt" else 0 creation_flags = subprocess.CREATE_NEW_PROCESS_GROUP if os.name == "nt" else 0
try: try:
process = subprocess.Popen( process = subprocess.Popen(
@@ -1291,7 +1309,7 @@ def _run_parser_child(extension: str, data: bytes) -> dict[str, object]:
except OSError: except OSError:
_raise_parser_error("parser_failed") _raise_parser_error("parser_failed")
try: try:
process.wait(timeout=PARSER_TIMEOUT_SECONDS) process.wait(timeout=CV_IMPORT_TIMEOUT_SECONDS if document else PARSER_TIMEOUT_SECONDS)
except subprocess.TimeoutExpired as exc: except subprocess.TimeoutExpired as exc:
_terminate_parser_process(process) _terminate_parser_process(process)
try: try:
@@ -1311,13 +1329,35 @@ def _run_parser_child(extension: str, data: bytes) -> dict[str, object]:
text = result.get("text") text = result.get("text")
if not isinstance(text, str) or not text or len(text) > MAX_EXTRACT_CHARS: if not isinstance(text, str) or not text or len(text) > MAX_EXTRACT_CHARS:
_raise_parser_error("parser_failed") _raise_parser_error("parser_failed")
if document:
_validate_document_result(result)
return result return result
finally: finally:
_parser_capacity.release() _parser_capacity.release()
@app.post("/extract-text") def _validate_document_result(result: dict[str, object]) -> None:
async def extract_text(file: UploadFile = File(...)): blocks = result.get("blocks")
warnings = result.get("warnings")
page_count = result.get("page_count")
if (result.get("method") not in {"docling-docx-v1", "docling-native-pdf-v1"}
or result.get("ocr_used") is not False
or not isinstance(blocks, list) or not 1 <= len(blocks) <= 5000
or not isinstance(warnings, list) or len(warnings) > 16
or any(not isinstance(warning, str) or not re.fullmatch(r"[a-z0-9_]{1,100}", warning) for warning in warnings)
or (page_count is not None and (type(page_count) is not int or not 1 <= page_count <= 40))):
_raise_parser_error("parser_failed")
for index, block in enumerate(blocks):
if (not isinstance(block, dict) or block.get("id") != f"b{index + 1:04d}"
or not isinstance(block.get("text"), str) or not block["text"].strip()
or block.get("kind") not in {"paragraph", "section_header", "title", "list_item", "table"}
or (block.get("page") is not None and (type(block["page"]) is not int or page_count is None or not 1 <= block["page"] <= page_count))):
_raise_parser_error("parser_failed")
if "\n".join(block["text"] for block in blocks) != result.get("text"):
_raise_parser_error("parser_failed")
async def _extract_upload(file: UploadFile, *, document: bool = False):
filename = file.filename or "document" filename = file.filename or "document"
extension = "." + filename.rsplit(".", 1)[1].lower() if "." in filename else "" extension = "." + filename.rsplit(".", 1)[1].lower() if "." in filename else ""
data = bytearray() data = bytearray()
@@ -1331,14 +1371,15 @@ async def extract_text(file: UploadFile = File(...)):
raise HTTPException(status_code=400, detail="The uploaded file was empty.") raise HTTPException(status_code=400, detail="The uploaded file was empty.")
if len(data) > MAX_EXTRACT_FILE_BYTES: if len(data) > MAX_EXTRACT_FILE_BYTES:
raise HTTPException(status_code=400, detail="The uploaded file is too large for AI extraction.") raise HTTPException(status_code=400, detail="The uploaded file is too large for AI extraction.")
if extension not in {".txt", ".md", ".docx", ".pdf", *IMAGE_EXTENSIONS}: supported = {".docx", ".pdf"} if document else {".txt", ".md", ".docx", ".pdf", *IMAGE_EXTENSIONS}
if extension not in supported:
raise HTTPException(status_code=400, detail="This file type is not supported for AI extraction.") raise HTTPException(status_code=400, detail="This file type is not supported for AI extraction.")
_validate_file_signature(extension, data) _validate_file_signature(extension, data)
result = await asyncio.to_thread(_run_parser_child, extension, data) result = await asyncio.to_thread(_run_parser_child, extension, data, document=True) if document else await asyncio.to_thread(_run_parser_child, extension, data)
text = result["text"] text = result["text"]
return { response = {
"text": text, "text": text,
"ocr_used": result.get("ocr_used", False), "ocr_used": result.get("ocr_used", False),
"content_type": file.content_type, "content_type": file.content_type,
@@ -1346,3 +1387,17 @@ async def extract_text(file: UploadFile = File(...)):
"characters": len(text), "characters": len(text),
"file_name": filename, "file_name": filename,
} }
if document:
_validate_document_result(result)
response.update({key: result[key] for key in ("blocks", "warnings", "method")})
return response
@app.post("/extract-text")
async def extract_text(file: UploadFile = File(...)):
return await _extract_upload(file)
@app.post("/cv-import/extract")
async def extract_cv_document(file: UploadFile = File(...)):
return await _extract_upload(file, document=True)
+148 -1
View File
@@ -10,6 +10,7 @@ from pathlib import Path
import re import re
import sys import sys
import zipfile import zipfile
from pathlib import PurePosixPath
MAX_EXTRACT_CHARS = 200_000 MAX_EXTRACT_CHARS = 200_000
@@ -220,6 +221,143 @@ def extract(extension: str, data: bytes) -> dict[str, object]:
raise ParserError("unsupported_type") raise ParserError("unsupported_type")
def _validate_import_docx(data: bytes) -> None:
validate_docx_container(data)
from defusedxml import ElementTree
try:
with zipfile.ZipFile(io.BytesIO(data)) as archive:
names: set[str] = set()
for entry in archive.infolist():
name = entry.filename
path = PurePosixPath(name)
if name in names or path.is_absolute() or ".." in path.parts or "\\" in name or ":" in name or entry.flag_bits & 1:
raise ParserError("docx_unsafe_archive")
names.add(name)
if name.lower().endswith((".bin", ".emf", ".wmf")) or name.startswith("word/embeddings/"):
raise ParserError("external_content")
if name.endswith((".xml", ".rels")):
root = ElementTree.fromstring(archive.read(entry))
for node in root.iter():
if node.tag.rsplit("}", 1)[-1] in {"altChunk", "OLEObject", "object"}:
raise ParserError("external_content")
if node.attrib.get("TargetMode", "").lower() == "external":
raise ParserError("external_content")
target = node.attrib.get("Target", "")
if ":" in target or "\\" in target or target.startswith("/"):
raise ParserError("external_content")
except ParserError:
raise
except Exception as exc:
raise ParserError("docx_signature") from exc
def _validate_import_pdf(data: bytes) -> int:
if not data.startswith(b"%PDF-"):
raise ParserError("pdf_signature")
from pypdf import PdfReader
from pypdf.generic import ArrayObject, DictionaryObject, IndirectObject
try:
reader = PdfReader(io.BytesIO(data), strict=True)
if reader.is_encrypted:
raise ParserError("pdf_encrypted")
count = len(reader.pages)
if count > MAX_PDF_PAGES:
raise ParserError("pdf_page_limit")
forbidden = {"/OpenAction", "/AA", "/JS", "/JavaScript", "/Launch", "/GoToR", "/URI", "/SubmitForm", "/ImportData", "/EmbeddedFiles", "/Filespec", "/RichMedia", "/XFA"}
pending = [(reader.trailer, 0)]
seen: set[int] = set()
while pending:
value, depth = pending.pop()
if isinstance(value, IndirectObject):
value = value.get_object()
if not isinstance(value, (DictionaryObject, ArrayObject)) or id(value) in seen:
continue
seen.add(id(value))
if depth > 40 or len(seen) > 10_000:
raise ParserError("pdf_object_limit")
if isinstance(value, DictionaryObject):
if forbidden.intersection(value) or str(value.get("/S", "")) in forbidden or str(value.get("/Type", "")) in forbidden:
raise ParserError("external_content")
pending.extend((item, depth + 1) for item in value.values())
else:
pending.extend((item, depth + 1) for item in value)
return count
except ParserError:
raise
except Exception as exc:
raise ParserError("pdf_invalid") from exc
def extract_document(extension: str, data: bytes) -> dict[str, object]:
"""Model-free Docling conversion of validated bytes, never URLs or local paths."""
if not data or len(data) > 5 * 1024 * 1024:
raise ParserError("file_size_limit")
if extension == ".docx":
_validate_import_docx(data)
page_count = None
warnings = ["docx_page_numbers_unavailable", "ocr_disabled"]
elif extension == ".pdf":
page_count = _validate_import_pdf(data)
warnings = ["pdf_native_order_no_layout_or_tables", "ocr_disabled"]
else:
raise ParserError("unsupported_type")
from docling.datamodel.base_models import DocumentStream, InputFormat, ConversionStatus
from docling.datamodel.backend_options import MsWordBackendOptions, ThreadedDoclingParseBackendOptions
from docling.datamodel.pipeline_options import ConvertPipelineOptions, NativePdfPipelineOptions
from docling.document_converter import DocumentConverter, WordFormatOption, NativePdfFormatOption
from docling_core.types.doc import TextItem, TableItem
common = dict(enable_remote_services=False, allow_external_plugins=False, do_picture_classification=False, do_picture_description=False, document_timeout=20)
if extension == ".pdf":
source_format = InputFormat.PDF
option = NativePdfFormatOption(
pipeline_options=NativePdfPipelineOptions(**common, generate_page_images=False, generate_picture_images=False, parser_threads=1),
backend_options=ThreadedDoclingParseBackendOptions(parser_threads=1, max_concurrent_results=1, render_pages=False, include_bitmap_images=False),
)
else:
source_format = InputFormat.DOCX
option = WordFormatOption(
pipeline_options=ConvertPipelineOptions(**common),
backend_options=MsWordBackendOptions(render_chart_images=False, max_total_bytes=MAX_DOCX_UNCOMPRESSED_BYTES, max_file_bytes=MAX_DOCX_ENTRY_BYTES, max_member_count=MAX_DOCX_ENTRIES),
)
converter = DocumentConverter(allowed_formats=[source_format], format_options={source_format: option})
conversion = converter.convert(DocumentStream(name="document" + extension, stream=io.BytesIO(data)), max_num_pages=MAX_PDF_PAGES, max_file_size=5 * 1024 * 1024)
if conversion.status != ConversionStatus.SUCCESS:
raise ParserError("parser_failed")
blocks: list[dict[str, object]] = []
characters = 0
for item, _ in conversion.document.iterate_items():
if isinstance(item, TableItem):
text = normalize_text(item.export_to_markdown(doc=conversion.document))
kind = "table"
elif isinstance(item, TextItem):
text = normalize_text(item.text)
kind = {"section_header": "section_header", "title": "title", "list_item": "list_item"}.get(item.label.value, "paragraph")
else:
continue
if not text:
continue
characters += len(text) + (1 if blocks else 0)
if characters > MAX_EXTRACT_CHARS:
raise ParserError("extracted_text_limit")
if len(blocks) >= 5000:
raise ParserError("block_limit")
page = item.prov[0].page_no if extension == ".pdf" and item.prov else None
blocks.append({"id": f"b{len(blocks) + 1:04d}", "text": text, "kind": kind, "page": page})
if not blocks:
raise ParserError("no_text")
return {"text": "\n".join(block["text"] for block in blocks), "ocr_used": False, "page_count": page_count, "blocks": blocks, "warnings": warnings, "method": "docling-native-pdf-v1" if extension == ".pdf" else "docling-docx-v1"}
def _deny_import_side_effects(event: str, args: tuple) -> None:
# Model-free conversion has no reason to open a socket or invoke an external tool.
if event in {"socket.connect", "socket.connect_ex", "socket.getaddrinfo", "subprocess.Popen", "os.system"}:
raise ParserError("external_content")
def _write_result(path: Path, result: dict[str, object]) -> None: def _write_result(path: Path, result: dict[str, object]) -> None:
path.write_text(json.dumps(result, ensure_ascii=False, separators=(",", ":")), encoding="utf-8") path.write_text(json.dumps(result, ensure_ascii=False, separators=(",", ":")), encoding="utf-8")
@@ -231,12 +369,18 @@ def main() -> int:
parser.add_argument("--extension", required=True) parser.add_argument("--extension", required=True)
parser.add_argument("--cpu-seconds", type=int, default=20) parser.add_argument("--cpu-seconds", type=int, default=20)
parser.add_argument("--memory-bytes", type=int, default=512 * 1024 * 1024) parser.add_argument("--memory-bytes", type=int, default=512 * 1024 * 1024)
parser.add_argument("--mode", choices=("legacy", "document"), default="legacy")
args = parser.parse_args() args = parser.parse_args()
output_path = Path(args.output) output_path = Path(args.output)
try: try:
_apply_resource_limits(args.cpu_seconds, args.memory_bytes) _apply_resource_limits(args.cpu_seconds, args.memory_bytes)
data = Path(args.input).read_bytes() data = Path(args.input).read_bytes()
result = extract(args.extension.lower(), data) if args.mode == "document":
os.environ.update({"HF_HUB_OFFLINE": "1", "HF_HUB_DISABLE_TELEMETRY": "1", "TRANSFORMERS_OFFLINE": "1"})
sys.addaudithook(_deny_import_side_effects)
result = extract_document(args.extension.lower(), data)
else:
result = extract(args.extension.lower(), data)
if not result.get("text"): if not result.get("text"):
raise ParserError("no_text") raise ParserError("no_text")
_write_result(output_path, {"ok": True, **result}) _write_result(output_path, {"ok": True, **result})
@@ -244,6 +388,9 @@ def main() -> int:
except ParserError as exc: except ParserError as exc:
_write_result(output_path, {"ok": False, "code": exc.code}) _write_result(output_path, {"ok": False, "code": exc.code})
return 2 return 2
except ImportError:
_write_result(output_path, {"ok": False, "code": "docling_unavailable" if args.mode == "document" else "parser_failed"})
return 3
except BaseException: except BaseException:
_write_result(output_path, {"ok": False, "code": "parser_failed"}) _write_result(output_path, {"ok": False, "code": "parser_failed"})
return 3 return 3
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,13 @@
# Separate interpreter: intentionally do not merge with the legacy model requirements.
docling-slim[format-docx,format-pdf-docling]==2.142.1
docling-core==2.104.0
docling-parse==7.23.0
python-docx==1.2.0
pydantic==2.14.1
pydantic-settings==2.16.1
pypdf==6.21.2
defusedxml==0.7.1
pillow==13.0.0
# Keep parser transport/build constraints aligned with the reviewed service floor.
certifi==2026.10.23
urllib3==2.8.0
+26
View File
@@ -616,6 +616,32 @@ def test_extract_text_rejects_oversized_upload_before_parsing(monkeypatch):
assert "too large" in response.json()["detail"].lower() assert "too large" in response.json()["detail"].lower()
def test_document_extraction_returns_structural_evidence_without_changing_legacy(monkeypatch):
module = load_app_module(monkeypatch)
def child(extension, data, *, document=False):
assert extension == ".pdf" and data == b"%PDF-synthetic"
assert document is True
return {"text": "Engineer", "ocr_used": False, "page_count": 1, "method": "docling-native-pdf-v1", "warnings": ["ocr_disabled"], "blocks": [{"id": "b0001", "text": "Engineer", "kind": "paragraph", "page": 1}]}
monkeypatch.setattr(module, "_run_parser_child", child)
response = TestClient(module.app).post("/cv-import/extract", files={"file": ("resume.pdf", b"%PDF-synthetic", "application/pdf")})
assert response.status_code == 200
assert response.json() == {"text": "Engineer", "ocr_used": False, "content_type": "application/pdf", "page_count": 1, "characters": 8, "file_name": "resume.pdf", "method": "docling-native-pdf-v1", "warnings": ["ocr_disabled"], "blocks": [{"id": "b0001", "text": "Engineer", "kind": "paragraph", "page": 1}]}
def test_document_extraction_rejects_text_and_signature_spoofing(monkeypatch):
module = load_app_module(monkeypatch)
monkeypatch.setattr(module, "_run_parser_child", lambda *args, **kwargs: pytest.fail("Invalid upload reached parser"))
client = TestClient(module.app)
for filename, data in [("resume.txt", b"text"), ("resume.pdf", b"not pdf")]:
assert client.post("/cv-import/extract", files={"file": (filename, data)}).status_code == 400
def test_document_extraction_requires_service_token(monkeypatch):
module = load_app_module(monkeypatch, service_token="secret")
response = TestClient(module.app).post("/cv-import/extract", files={"file": ("resume.pdf", b"%PDF-synthetic")})
assert response.status_code == 401
def test_extract_text_rejects_extension_signature_mismatch(monkeypatch): def test_extract_text_rejects_extension_signature_mismatch(monkeypatch):
module = load_app_module(monkeypatch) module = load_app_module(monkeypatch)
client = TestClient(module.app) client = TestClient(module.app)
@@ -0,0 +1,137 @@
"""Real, synthetic documents; no model artifacts or inference."""
import io
import socket
import sys
import zipfile
import subprocess
import json
from pathlib import Path
import pytest
from docx import Document
from pypdf import PdfWriter
from pypdf.generic import DecodedStreamObject, DictionaryObject, NameObject, TextStringObject
sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
import parser_child
def docx_bytes():
document = Document()
document.add_heading("Erfaring", level=1)
document.add_paragraph("Utvikler | Eksempel AS | 2021–2023")
document.add_paragraph("Bygget sikre API-er", style="List Bullet")
table = document.add_table(rows=1, cols=2)
table.cell(0, 0).text = "Språk"
table.cell(0, 1).text = "Norsk, English"
output = io.BytesIO()
document.save(output)
return output.getvalue()
def pdf_bytes(*, pages=2, external=False):
writer = PdfWriter()
for number in range(pages):
page = writer.add_blank_page(width=612, height=792)
font = DictionaryObject({NameObject("/Type"): NameObject("/Font"), NameObject("/Subtype"): NameObject("/Type1"), NameObject("/BaseFont"): NameObject("/Helvetica")})
page[NameObject("/Resources")] = DictionaryObject({NameObject("/Font"): DictionaryObject({NameObject("/F1"): writer._add_object(font)})})
stream = DecodedStreamObject()
stream.set_data(f"BT /F1 12 Tf 40 740 Td (Experience page {number + 1}) Tj 0 -20 Td (Engineer 2020-2022) Tj ET".encode())
page[NameObject("/Contents")] = writer._add_object(stream)
if external:
writer._root_object[NameObject("/OpenAction")] = DictionaryObject({NameObject("/S"): NameObject("/URI"), NameObject("/URI"): TextStringObject("http://127.0.0.1/private")})
output = io.BytesIO()
writer.write(output)
return output.getvalue()
@pytest.fixture(autouse=True)
def no_network(monkeypatch):
def denied(*args, **kwargs):
raise AssertionError("Document extraction attempted network access")
monkeypatch.setattr(socket.socket, "connect", denied)
monkeypatch.setattr(socket, "getaddrinfo", denied)
def test_docx_preserves_ordered_sections_lists_tables_and_unknown_pages():
result = parser_child.extract_document(".docx", docx_bytes())
assert result["method"] == "docling-docx-v1"
assert result["ocr_used"] is False
assert result["page_count"] is None
assert [block["kind"] for block in result["blocks"]] == ["section_header", "paragraph", "list_item", "table"]
assert result["blocks"][0] == {"id": "b0001", "text": "Erfaring", "kind": "section_header", "page": None}
assert "2021–2023" in result["blocks"][1]["text"]
assert "Norsk, English" in result["blocks"][3]["text"]
assert "docx_page_numbers_unavailable" in result["warnings"]
def test_native_pdf_keeps_page_evidence_and_declares_layout_limits():
result = parser_child.extract_document(".pdf", pdf_bytes())
assert result["method"] == "docling-native-pdf-v1"
assert result["page_count"] == 2
assert [block["page"] for block in result["blocks"]] == [1, 1, 2, 2]
assert [block["id"] for block in result["blocks"]] == ["b0001", "b0002", "b0003", "b0004"]
assert "Experience page 1" in result["blocks"][0]["text"]
assert "pdf_native_order_no_layout_or_tables" in result["warnings"]
assert "ocr_disabled" in result["warnings"]
@pytest.mark.parametrize("extension,data,code", [
(".pdf", b"not pdf", "pdf_signature"),
(".pdf", b"%PDF-broken", "pdf_invalid"),
(".pdf", pdf_bytes(pages=41), "pdf_page_limit"),
(".pdf", pdf_bytes(external=True), "external_content"),
(".txt", b"plain text", "unsupported_type"),
], ids=["signature", "malformed", "pages", "external", "type"])
def test_rejects_invalid_or_active_documents_before_conversion(extension, data, code):
with pytest.raises(parser_child.ParserError) as failure:
parser_child.extract_document(extension, data)
assert failure.value.code == code
def test_docx_rejects_external_relationships():
payload = io.BytesIO()
with zipfile.ZipFile(io.BytesIO(docx_bytes())) as source, zipfile.ZipFile(payload, "w", zipfile.ZIP_DEFLATED) as output:
for entry in source.infolist():
body = source.read(entry)
if entry.filename == "word/_rels/document.xml.rels":
body = body.replace(b"</Relationships>", b'<Relationship Id="external" Type="image" TargetMode="External" Target="file:///etc/passwd"/></Relationships>')
output.writestr(entry.filename, body)
with pytest.raises(parser_child.ParserError) as failure:
parser_child.extract_document(".docx", payload.getvalue())
assert failure.value.code == "external_content"
def test_docx_rejects_archive_traversal():
payload = io.BytesIO()
with zipfile.ZipFile(io.BytesIO(docx_bytes())) as source, zipfile.ZipFile(payload, "w", zipfile.ZIP_DEFLATED) as output:
for entry in source.infolist():
output.writestr(entry.filename, source.read(entry))
output.writestr("../secret", "inert")
with pytest.raises(parser_child.ParserError) as failure:
parser_child.extract_document(".docx", payload.getvalue())
assert failure.value.code == "docx_unsafe_archive"
def test_extraction_refuses_truncated_output_instead_of_silently_losing_text(monkeypatch):
monkeypatch.setattr(parser_child, "MAX_EXTRACT_CHARS", 10)
with pytest.raises(parser_child.ParserError) as failure:
parser_child.extract_document(".pdf", pdf_bytes())
assert failure.value.code == "extracted_text_limit"
def test_document_child_uses_real_converter_and_cleans_no_source_files(tmp_path):
source, output = tmp_path / "input.bin", tmp_path / "result.json"
source.write_bytes(docx_bytes())
result = subprocess.run([sys.executable, str(Path(parser_child.__file__)), "--input", str(source), "--output", str(output), "--extension", ".docx", "--mode", "document", "--memory-bytes", str(1024 ** 3)], capture_output=True, timeout=20)
assert result.returncode == 0, (result.stderr, output.read_text() if output.exists() else "missing output")
assert json.loads(output.read_text(encoding="utf-8"))["method"] == "docling-docx-v1"
assert source.read_bytes().startswith(b"PK")
def test_document_child_forbids_socket_and_external_process_side_effects():
for event in ["socket.connect", "socket.connect_ex", "socket.getaddrinfo", "subprocess.Popen", "os.system"]:
with pytest.raises(parser_child.ParserError) as failure:
parser_child._deny_import_side_effects(event, ())
assert failure.value.code == "external_content"