feat(cv-import): add bounded model-free Docling extraction
This commit is contained in:
cesnimda
@@ -0,0 +1,35 @@
|
||||
# Docling extraction and local-model boundary
|
||||
|
||||
26 October 2026. Development implementation; production remains NO-GO.
|
||||
|
||||
The existing FastAPI sidecar owns `POST /cv-import/extract`. It accepts PDF/DOCX bytes and returns text, MIME/name metadata, page count where available, ordered `blocks[{id,text,kind,page}]`, explicit warnings and converter method. Existing `/extract-text` remains available for legacy text/image/history compatibility. The API transports and persists the structured evidence; it does not run an in-process binary parser.
|
||||
|
||||
## Reproducible dependencies
|
||||
|
||||
`tools/summarizer/requirements-cv-import.txt` and its universal Python 3.12 hash lock pin Docling slim 2.142.1, core 2.104.0, parse 7.23.0, python-docx 1.2.0, Pydantic 2.14.1 and Pydantic settings 2.16.1. Verify against the lock rather than assuming the old `.venv` is current. This runtime is separate because the legacy model dependency set pins incompatible python-docx/Pydantic versions. No inherited model pins or caches were replaced.
|
||||
|
||||
Release/API verification used actual PyPI wheel metadata and installed source, including `DocumentConverter`, `WordFormatOption`, `NativePdfFormatOption`, `NativePdfPipelineOptions`, `DocumentStream`, `ConversionStatus`, `iterate_items` and `TableItem.export_to_markdown`. Sources: [PyPI Docling](https://pypi.org/project/docling/), [Docling pipeline options](https://docling-project.github.io/docling/reference/pipeline_options/), [local/remote configuration](https://github.com/docling-project/docling/blob/main/docs/usage/advanced_options.md). Search indexing was stale at 2.129.0; the actual package index and installed source identified 2.142.1.
|
||||
|
||||
Create a dedicated interpreter and install with `pip install --require-hashes -r tools/summarizer/requirements-cv-import.lock`. Set `CV_IMPORT_PYTHON` to its absolute executable in development. The main sidecar can retain its existing interpreter. No Docker/production service was changed.
|
||||
|
||||
## Conversion and limits
|
||||
|
||||
DOCX uses model-free Word conversion with ordered headings, lists, paragraphs and tables. It cannot establish physical page numbers; `page=null` is deliberate. PDF uses the supported **NativePdfPipeline**, without layout/table/OCR models. Its parser order and text-cell grouping are not a qualified reconstruction of complex multi-column/table layouts. Every result declares `pdf_native_order_no_layout_or_tables` or `docx_page_numbers_unavailable`, and `ocr_disabled`.
|
||||
|
||||
Remote services, external plugins, picture enrichment, chart rendering, page/bitmap rendering and OCR are disabled. The child uses offline model variables and a Python audit hook denying sockets and subprocesses. It accepts only validated local byte streams. There were no model downloads, generative inference calls or production/cache modifications.
|
||||
|
||||
The existing subprocess runner bounds concurrency to one, input to 5 MiB, PDF to 40 pages, extracted text to 200,000 characters, blocks to 5,000 and JSON to 1 MiB. DOCX validation retains the existing entry/expanded-size/ratio limits and adds safe XML/external-content checks. Parent deadlines, process-tree termination, private work directories, secret-stripped environment, stale-work cleanup and sanitized failures remain. New settings are bounded: `CV_IMPORT_TIMEOUT_SECONDS` default 40 (5–120), `CV_IMPORT_CPU_SECONDS` default 30, `CV_IMPORT_MEMORY_MIB` default 1024 (512–2048). POSIX adds CPU/address-space limits; Windows currently has wall-clock/process termination rather than a kernel-enforced memory ceiling. This platform difference is an open resource qualification limitation.
|
||||
|
||||
Scans and image-only PDFs fail cleanly with OCR disabled. Encrypted documents, active/embedded content, unsafe external relationships, malformed/bomb archives and excessive inputs are rejected. PDF table-cell relationships may require manual review. No arbitrary confidence percentage is exposed.
|
||||
|
||||
## AI adapter
|
||||
|
||||
`cv_import_ai.py` provides a configurable local Ollama transport, versioned source-as-data instruction, constrained output schema and strict field/quote/block validation. It defaults disabled and has no automatic model selection. Synthetic `generate` injection tests malformed output, invented facts, date precision, unknown blocks/paths and unsafe provider addresses. Suggestions are labelled `model-assisted` / `needs_review` and never directly write a profile.
|
||||
|
||||
The adapter is a qualification seam; the active import path remains deterministic and does not call it. Actual provider/model compatibility, factual relationship validation, EN/Bokmål language quality, adversarial semantics, GPU behavior and model accuracy remain **BLOCKED**. Enabling production inference requires a separately approved immutable artifact and isolated qualification; environment configuration is not acceptance.
|
||||
|
||||
## Current focused evidence
|
||||
|
||||
- `.artifacts/cv-import/sidecar-focused-02.log`: 23 passed, including real native PDF/DOCX conversion and synthetic model-adapter tests; no generative inference.
|
||||
- `.artifacts/cv-import/python-offline-01.log`: 73 passed in immutable existing Python runtime `sha256:2fe77e47a6c5ecea66764dbd6264049724f24849141e16af5491c2de34af53c1`, network disabled/read-only source/model loading disabled. This covers app/parser-security/grounding/adapter behavior, not new Docling dependency compatibility in that legacy image. One pytest cache-option warning retained.
|
||||
- `.artifacts/cv-import/extraction-benchmark-01`: 26 text-bearing generated documents converted; the separate synthetic scan was rejected. Actual source hashes, per-process wall/CPU/peak-working-set/input/output measurements are recorded. Later benchmark attempts supersede generator defects explicitly; no historical result is overwritten.
|
||||
+65
-10
@@ -110,6 +110,10 @@ PARSER_CHILD_PATH = Path(__file__).with_name("parser_child.py")
|
||||
PARSER_TIMEOUT_SECONDS = max(5, min(int(os.getenv("PARSER_TIMEOUT_SECONDS", "25")), 120))
|
||||
PARSER_CPU_SECONDS = max(2, min(int(os.getenv("PARSER_CPU_SECONDS", "20")), PARSER_TIMEOUT_SECONDS))
|
||||
PARSER_MEMORY_BYTES = max(256, min(int(os.getenv("PARSER_MEMORY_MIB", "512")), 2048)) * 1024 * 1024
|
||||
CV_IMPORT_PYTHON = os.getenv("CV_IMPORT_PYTHON", sys.executable)
|
||||
CV_IMPORT_TIMEOUT_SECONDS = max(5, min(int(os.getenv("CV_IMPORT_TIMEOUT_SECONDS", "40")), 120))
|
||||
CV_IMPORT_CPU_SECONDS = max(2, min(int(os.getenv("CV_IMPORT_CPU_SECONDS", "30")), CV_IMPORT_TIMEOUT_SECONDS))
|
||||
CV_IMPORT_MEMORY_BYTES = max(512, min(int(os.getenv("CV_IMPORT_MEMORY_MIB", "1024")), 2048)) * 1024 * 1024
|
||||
PARSER_MAX_RESULT_BYTES = 1024 * 1024
|
||||
_parser_capacity = threading.BoundedSemaphore(value=1)
|
||||
PARSER_TEMP_ROOT = Path(os.getenv("PARSER_TEMP_ROOT", tempfile.gettempdir()), "jobtracker-cv-parser").resolve()
|
||||
@@ -1184,6 +1188,15 @@ _PARSER_ERRORS = {
|
||||
"docx_entry_size": (422, "The DOCX contains an oversized file."),
|
||||
"docx_ratio": (422, "The DOCX compression ratio is not supported."),
|
||||
"docx_expanded_size": (422, "The DOCX expands beyond the supported size."),
|
||||
"docx_unsafe_archive": (422, "The DOCX archive structure is not supported."),
|
||||
"pdf_signature": (400, "The uploaded file does not match its PDF extension."),
|
||||
"pdf_invalid": (422, "The PDF is malformed or unsupported."),
|
||||
"pdf_encrypted": (422, "Encrypted PDFs are not supported."),
|
||||
"pdf_object_limit": (422, "The PDF structure exceeds the supported limits."),
|
||||
"external_content": (422, "Active or external document content is not supported."),
|
||||
"block_limit": (422, "The document contains too many text blocks."),
|
||||
"file_size_limit": (400, "The uploaded file size is not supported."),
|
||||
"docling_unavailable": (503, "Document conversion runtime is unavailable."),
|
||||
"pdf_page_limit": (422, "The PDF contains too many pages."),
|
||||
"pdf_ocr_limit": (422, "The PDF requires too much OCR processing."),
|
||||
"image_dimensions": (422, "The document image dimensions are not supported."),
|
||||
@@ -1203,6 +1216,9 @@ def _parser_environment(temp_root: str) -> dict[str, str]:
|
||||
"PYTHONDONTWRITEBYTECODE": "1",
|
||||
"OMP_NUM_THREADS": "1",
|
||||
"OPENBLAS_NUM_THREADS": "1",
|
||||
"HF_HUB_OFFLINE": "1",
|
||||
"TRANSFORMERS_OFFLINE": "1",
|
||||
"HF_HUB_DISABLE_TELEMETRY": "1",
|
||||
"TMPDIR": temp_root,
|
||||
"TEMP": temp_root,
|
||||
"TMP": temp_root,
|
||||
@@ -1259,7 +1275,7 @@ def _raise_parser_error(code: str) -> None:
|
||||
raise HTTPException(status_code=status_code, detail=detail)
|
||||
|
||||
|
||||
def _run_parser_child(extension: str, data: bytes) -> dict[str, object]:
|
||||
def _run_parser_child(extension: str, data: bytes, *, document: bool = False) -> dict[str, object]:
|
||||
if not _parser_capacity.acquire(blocking=False):
|
||||
raise HTTPException(status_code=503, detail="Document extraction is busy. Try again shortly.")
|
||||
try:
|
||||
@@ -1268,14 +1284,16 @@ def _run_parser_child(extension: str, data: bytes) -> dict[str, object]:
|
||||
output_path = Path(temp_root, "result.json")
|
||||
input_path.write_bytes(data)
|
||||
command = [
|
||||
sys.executable,
|
||||
CV_IMPORT_PYTHON if document else sys.executable,
|
||||
str(PARSER_CHILD_PATH),
|
||||
"--input", str(input_path),
|
||||
"--output", str(output_path),
|
||||
"--extension", extension,
|
||||
"--cpu-seconds", str(PARSER_CPU_SECONDS),
|
||||
"--memory-bytes", str(PARSER_MEMORY_BYTES),
|
||||
"--cpu-seconds", str(CV_IMPORT_CPU_SECONDS if document else PARSER_CPU_SECONDS),
|
||||
"--memory-bytes", str(CV_IMPORT_MEMORY_BYTES if document else PARSER_MEMORY_BYTES),
|
||||
]
|
||||
if document:
|
||||
command.extend(["--mode", "document"])
|
||||
creation_flags = subprocess.CREATE_NEW_PROCESS_GROUP if os.name == "nt" else 0
|
||||
try:
|
||||
process = subprocess.Popen(
|
||||
@@ -1291,7 +1309,7 @@ def _run_parser_child(extension: str, data: bytes) -> dict[str, object]:
|
||||
except OSError:
|
||||
_raise_parser_error("parser_failed")
|
||||
try:
|
||||
process.wait(timeout=PARSER_TIMEOUT_SECONDS)
|
||||
process.wait(timeout=CV_IMPORT_TIMEOUT_SECONDS if document else PARSER_TIMEOUT_SECONDS)
|
||||
except subprocess.TimeoutExpired as exc:
|
||||
_terminate_parser_process(process)
|
||||
try:
|
||||
@@ -1311,13 +1329,35 @@ def _run_parser_child(extension: str, data: bytes) -> dict[str, object]:
|
||||
text = result.get("text")
|
||||
if not isinstance(text, str) or not text or len(text) > MAX_EXTRACT_CHARS:
|
||||
_raise_parser_error("parser_failed")
|
||||
if document:
|
||||
_validate_document_result(result)
|
||||
return result
|
||||
finally:
|
||||
_parser_capacity.release()
|
||||
|
||||
|
||||
@app.post("/extract-text")
|
||||
async def extract_text(file: UploadFile = File(...)):
|
||||
def _validate_document_result(result: dict[str, object]) -> None:
|
||||
blocks = result.get("blocks")
|
||||
warnings = result.get("warnings")
|
||||
page_count = result.get("page_count")
|
||||
if (result.get("method") not in {"docling-docx-v1", "docling-native-pdf-v1"}
|
||||
or result.get("ocr_used") is not False
|
||||
or not isinstance(blocks, list) or not 1 <= len(blocks) <= 5000
|
||||
or not isinstance(warnings, list) or len(warnings) > 16
|
||||
or any(not isinstance(warning, str) or not re.fullmatch(r"[a-z0-9_]{1,100}", warning) for warning in warnings)
|
||||
or (page_count is not None and (type(page_count) is not int or not 1 <= page_count <= 40))):
|
||||
_raise_parser_error("parser_failed")
|
||||
for index, block in enumerate(blocks):
|
||||
if (not isinstance(block, dict) or block.get("id") != f"b{index + 1:04d}"
|
||||
or not isinstance(block.get("text"), str) or not block["text"].strip()
|
||||
or block.get("kind") not in {"paragraph", "section_header", "title", "list_item", "table"}
|
||||
or (block.get("page") is not None and (type(block["page"]) is not int or page_count is None or not 1 <= block["page"] <= page_count))):
|
||||
_raise_parser_error("parser_failed")
|
||||
if "\n".join(block["text"] for block in blocks) != result.get("text"):
|
||||
_raise_parser_error("parser_failed")
|
||||
|
||||
|
||||
async def _extract_upload(file: UploadFile, *, document: bool = False):
|
||||
filename = file.filename or "document"
|
||||
extension = "." + filename.rsplit(".", 1)[1].lower() if "." in filename else ""
|
||||
data = bytearray()
|
||||
@@ -1331,14 +1371,15 @@ async def extract_text(file: UploadFile = File(...)):
|
||||
raise HTTPException(status_code=400, detail="The uploaded file was empty.")
|
||||
if len(data) > MAX_EXTRACT_FILE_BYTES:
|
||||
raise HTTPException(status_code=400, detail="The uploaded file is too large for AI extraction.")
|
||||
if extension not in {".txt", ".md", ".docx", ".pdf", *IMAGE_EXTENSIONS}:
|
||||
supported = {".docx", ".pdf"} if document else {".txt", ".md", ".docx", ".pdf", *IMAGE_EXTENSIONS}
|
||||
if extension not in supported:
|
||||
raise HTTPException(status_code=400, detail="This file type is not supported for AI extraction.")
|
||||
_validate_file_signature(extension, data)
|
||||
|
||||
result = await asyncio.to_thread(_run_parser_child, extension, data)
|
||||
result = await asyncio.to_thread(_run_parser_child, extension, data, document=True) if document else await asyncio.to_thread(_run_parser_child, extension, data)
|
||||
text = result["text"]
|
||||
|
||||
return {
|
||||
response = {
|
||||
"text": text,
|
||||
"ocr_used": result.get("ocr_used", False),
|
||||
"content_type": file.content_type,
|
||||
@@ -1346,3 +1387,17 @@ async def extract_text(file: UploadFile = File(...)):
|
||||
"characters": len(text),
|
||||
"file_name": filename,
|
||||
}
|
||||
if document:
|
||||
_validate_document_result(result)
|
||||
response.update({key: result[key] for key in ("blocks", "warnings", "method")})
|
||||
return response
|
||||
|
||||
|
||||
@app.post("/extract-text")
|
||||
async def extract_text(file: UploadFile = File(...)):
|
||||
return await _extract_upload(file)
|
||||
|
||||
|
||||
@app.post("/cv-import/extract")
|
||||
async def extract_cv_document(file: UploadFile = File(...)):
|
||||
return await _extract_upload(file, document=True)
|
||||
|
||||
@@ -10,6 +10,7 @@ from pathlib import Path
|
||||
import re
|
||||
import sys
|
||||
import zipfile
|
||||
from pathlib import PurePosixPath
|
||||
|
||||
|
||||
MAX_EXTRACT_CHARS = 200_000
|
||||
@@ -220,6 +221,143 @@ def extract(extension: str, data: bytes) -> dict[str, object]:
|
||||
raise ParserError("unsupported_type")
|
||||
|
||||
|
||||
def _validate_import_docx(data: bytes) -> None:
|
||||
validate_docx_container(data)
|
||||
from defusedxml import ElementTree
|
||||
|
||||
try:
|
||||
with zipfile.ZipFile(io.BytesIO(data)) as archive:
|
||||
names: set[str] = set()
|
||||
for entry in archive.infolist():
|
||||
name = entry.filename
|
||||
path = PurePosixPath(name)
|
||||
if name in names or path.is_absolute() or ".." in path.parts or "\\" in name or ":" in name or entry.flag_bits & 1:
|
||||
raise ParserError("docx_unsafe_archive")
|
||||
names.add(name)
|
||||
if name.lower().endswith((".bin", ".emf", ".wmf")) or name.startswith("word/embeddings/"):
|
||||
raise ParserError("external_content")
|
||||
if name.endswith((".xml", ".rels")):
|
||||
root = ElementTree.fromstring(archive.read(entry))
|
||||
for node in root.iter():
|
||||
if node.tag.rsplit("}", 1)[-1] in {"altChunk", "OLEObject", "object"}:
|
||||
raise ParserError("external_content")
|
||||
if node.attrib.get("TargetMode", "").lower() == "external":
|
||||
raise ParserError("external_content")
|
||||
target = node.attrib.get("Target", "")
|
||||
if ":" in target or "\\" in target or target.startswith("/"):
|
||||
raise ParserError("external_content")
|
||||
except ParserError:
|
||||
raise
|
||||
except Exception as exc:
|
||||
raise ParserError("docx_signature") from exc
|
||||
|
||||
|
||||
def _validate_import_pdf(data: bytes) -> int:
|
||||
if not data.startswith(b"%PDF-"):
|
||||
raise ParserError("pdf_signature")
|
||||
from pypdf import PdfReader
|
||||
from pypdf.generic import ArrayObject, DictionaryObject, IndirectObject
|
||||
|
||||
try:
|
||||
reader = PdfReader(io.BytesIO(data), strict=True)
|
||||
if reader.is_encrypted:
|
||||
raise ParserError("pdf_encrypted")
|
||||
count = len(reader.pages)
|
||||
if count > MAX_PDF_PAGES:
|
||||
raise ParserError("pdf_page_limit")
|
||||
forbidden = {"/OpenAction", "/AA", "/JS", "/JavaScript", "/Launch", "/GoToR", "/URI", "/SubmitForm", "/ImportData", "/EmbeddedFiles", "/Filespec", "/RichMedia", "/XFA"}
|
||||
pending = [(reader.trailer, 0)]
|
||||
seen: set[int] = set()
|
||||
while pending:
|
||||
value, depth = pending.pop()
|
||||
if isinstance(value, IndirectObject):
|
||||
value = value.get_object()
|
||||
if not isinstance(value, (DictionaryObject, ArrayObject)) or id(value) in seen:
|
||||
continue
|
||||
seen.add(id(value))
|
||||
if depth > 40 or len(seen) > 10_000:
|
||||
raise ParserError("pdf_object_limit")
|
||||
if isinstance(value, DictionaryObject):
|
||||
if forbidden.intersection(value) or str(value.get("/S", "")) in forbidden or str(value.get("/Type", "")) in forbidden:
|
||||
raise ParserError("external_content")
|
||||
pending.extend((item, depth + 1) for item in value.values())
|
||||
else:
|
||||
pending.extend((item, depth + 1) for item in value)
|
||||
return count
|
||||
except ParserError:
|
||||
raise
|
||||
except Exception as exc:
|
||||
raise ParserError("pdf_invalid") from exc
|
||||
|
||||
|
||||
def extract_document(extension: str, data: bytes) -> dict[str, object]:
|
||||
"""Model-free Docling conversion of validated bytes, never URLs or local paths."""
|
||||
if not data or len(data) > 5 * 1024 * 1024:
|
||||
raise ParserError("file_size_limit")
|
||||
if extension == ".docx":
|
||||
_validate_import_docx(data)
|
||||
page_count = None
|
||||
warnings = ["docx_page_numbers_unavailable", "ocr_disabled"]
|
||||
elif extension == ".pdf":
|
||||
page_count = _validate_import_pdf(data)
|
||||
warnings = ["pdf_native_order_no_layout_or_tables", "ocr_disabled"]
|
||||
else:
|
||||
raise ParserError("unsupported_type")
|
||||
|
||||
from docling.datamodel.base_models import DocumentStream, InputFormat, ConversionStatus
|
||||
from docling.datamodel.backend_options import MsWordBackendOptions, ThreadedDoclingParseBackendOptions
|
||||
from docling.datamodel.pipeline_options import ConvertPipelineOptions, NativePdfPipelineOptions
|
||||
from docling.document_converter import DocumentConverter, WordFormatOption, NativePdfFormatOption
|
||||
from docling_core.types.doc import TextItem, TableItem
|
||||
|
||||
common = dict(enable_remote_services=False, allow_external_plugins=False, do_picture_classification=False, do_picture_description=False, document_timeout=20)
|
||||
if extension == ".pdf":
|
||||
source_format = InputFormat.PDF
|
||||
option = NativePdfFormatOption(
|
||||
pipeline_options=NativePdfPipelineOptions(**common, generate_page_images=False, generate_picture_images=False, parser_threads=1),
|
||||
backend_options=ThreadedDoclingParseBackendOptions(parser_threads=1, max_concurrent_results=1, render_pages=False, include_bitmap_images=False),
|
||||
)
|
||||
else:
|
||||
source_format = InputFormat.DOCX
|
||||
option = WordFormatOption(
|
||||
pipeline_options=ConvertPipelineOptions(**common),
|
||||
backend_options=MsWordBackendOptions(render_chart_images=False, max_total_bytes=MAX_DOCX_UNCOMPRESSED_BYTES, max_file_bytes=MAX_DOCX_ENTRY_BYTES, max_member_count=MAX_DOCX_ENTRIES),
|
||||
)
|
||||
converter = DocumentConverter(allowed_formats=[source_format], format_options={source_format: option})
|
||||
conversion = converter.convert(DocumentStream(name="document" + extension, stream=io.BytesIO(data)), max_num_pages=MAX_PDF_PAGES, max_file_size=5 * 1024 * 1024)
|
||||
if conversion.status != ConversionStatus.SUCCESS:
|
||||
raise ParserError("parser_failed")
|
||||
blocks: list[dict[str, object]] = []
|
||||
characters = 0
|
||||
for item, _ in conversion.document.iterate_items():
|
||||
if isinstance(item, TableItem):
|
||||
text = normalize_text(item.export_to_markdown(doc=conversion.document))
|
||||
kind = "table"
|
||||
elif isinstance(item, TextItem):
|
||||
text = normalize_text(item.text)
|
||||
kind = {"section_header": "section_header", "title": "title", "list_item": "list_item"}.get(item.label.value, "paragraph")
|
||||
else:
|
||||
continue
|
||||
if not text:
|
||||
continue
|
||||
characters += len(text) + (1 if blocks else 0)
|
||||
if characters > MAX_EXTRACT_CHARS:
|
||||
raise ParserError("extracted_text_limit")
|
||||
if len(blocks) >= 5000:
|
||||
raise ParserError("block_limit")
|
||||
page = item.prov[0].page_no if extension == ".pdf" and item.prov else None
|
||||
blocks.append({"id": f"b{len(blocks) + 1:04d}", "text": text, "kind": kind, "page": page})
|
||||
if not blocks:
|
||||
raise ParserError("no_text")
|
||||
return {"text": "\n".join(block["text"] for block in blocks), "ocr_used": False, "page_count": page_count, "blocks": blocks, "warnings": warnings, "method": "docling-native-pdf-v1" if extension == ".pdf" else "docling-docx-v1"}
|
||||
|
||||
|
||||
def _deny_import_side_effects(event: str, args: tuple) -> None:
|
||||
# Model-free conversion has no reason to open a socket or invoke an external tool.
|
||||
if event in {"socket.connect", "socket.connect_ex", "socket.getaddrinfo", "subprocess.Popen", "os.system"}:
|
||||
raise ParserError("external_content")
|
||||
|
||||
|
||||
def _write_result(path: Path, result: dict[str, object]) -> None:
|
||||
path.write_text(json.dumps(result, ensure_ascii=False, separators=(",", ":")), encoding="utf-8")
|
||||
|
||||
@@ -231,11 +369,17 @@ def main() -> int:
|
||||
parser.add_argument("--extension", required=True)
|
||||
parser.add_argument("--cpu-seconds", type=int, default=20)
|
||||
parser.add_argument("--memory-bytes", type=int, default=512 * 1024 * 1024)
|
||||
parser.add_argument("--mode", choices=("legacy", "document"), default="legacy")
|
||||
args = parser.parse_args()
|
||||
output_path = Path(args.output)
|
||||
try:
|
||||
_apply_resource_limits(args.cpu_seconds, args.memory_bytes)
|
||||
data = Path(args.input).read_bytes()
|
||||
if args.mode == "document":
|
||||
os.environ.update({"HF_HUB_OFFLINE": "1", "HF_HUB_DISABLE_TELEMETRY": "1", "TRANSFORMERS_OFFLINE": "1"})
|
||||
sys.addaudithook(_deny_import_side_effects)
|
||||
result = extract_document(args.extension.lower(), data)
|
||||
else:
|
||||
result = extract(args.extension.lower(), data)
|
||||
if not result.get("text"):
|
||||
raise ParserError("no_text")
|
||||
@@ -244,6 +388,9 @@ def main() -> int:
|
||||
except ParserError as exc:
|
||||
_write_result(output_path, {"ok": False, "code": exc.code})
|
||||
return 2
|
||||
except ImportError:
|
||||
_write_result(output_path, {"ok": False, "code": "docling_unavailable" if args.mode == "document" else "parser_failed"})
|
||||
return 3
|
||||
except BaseException:
|
||||
_write_result(output_path, {"ok": False, "code": "parser_failed"})
|
||||
return 3
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,13 @@
|
||||
# Separate interpreter: intentionally do not merge with the legacy model requirements.
|
||||
docling-slim[format-docx,format-pdf-docling]==2.142.1
|
||||
docling-core==2.104.0
|
||||
docling-parse==7.23.0
|
||||
python-docx==1.2.0
|
||||
pydantic==2.14.1
|
||||
pydantic-settings==2.16.1
|
||||
pypdf==6.21.2
|
||||
defusedxml==0.7.1
|
||||
pillow==13.0.0
|
||||
# Keep parser transport/build constraints aligned with the reviewed service floor.
|
||||
certifi==2026.10.23
|
||||
urllib3==2.8.0
|
||||
@@ -616,6 +616,32 @@ def test_extract_text_rejects_oversized_upload_before_parsing(monkeypatch):
|
||||
assert "too large" in response.json()["detail"].lower()
|
||||
|
||||
|
||||
def test_document_extraction_returns_structural_evidence_without_changing_legacy(monkeypatch):
|
||||
module = load_app_module(monkeypatch)
|
||||
def child(extension, data, *, document=False):
|
||||
assert extension == ".pdf" and data == b"%PDF-synthetic"
|
||||
assert document is True
|
||||
return {"text": "Engineer", "ocr_used": False, "page_count": 1, "method": "docling-native-pdf-v1", "warnings": ["ocr_disabled"], "blocks": [{"id": "b0001", "text": "Engineer", "kind": "paragraph", "page": 1}]}
|
||||
monkeypatch.setattr(module, "_run_parser_child", child)
|
||||
response = TestClient(module.app).post("/cv-import/extract", files={"file": ("resume.pdf", b"%PDF-synthetic", "application/pdf")})
|
||||
assert response.status_code == 200
|
||||
assert response.json() == {"text": "Engineer", "ocr_used": False, "content_type": "application/pdf", "page_count": 1, "characters": 8, "file_name": "resume.pdf", "method": "docling-native-pdf-v1", "warnings": ["ocr_disabled"], "blocks": [{"id": "b0001", "text": "Engineer", "kind": "paragraph", "page": 1}]}
|
||||
|
||||
|
||||
def test_document_extraction_rejects_text_and_signature_spoofing(monkeypatch):
|
||||
module = load_app_module(monkeypatch)
|
||||
monkeypatch.setattr(module, "_run_parser_child", lambda *args, **kwargs: pytest.fail("Invalid upload reached parser"))
|
||||
client = TestClient(module.app)
|
||||
for filename, data in [("resume.txt", b"text"), ("resume.pdf", b"not pdf")]:
|
||||
assert client.post("/cv-import/extract", files={"file": (filename, data)}).status_code == 400
|
||||
|
||||
|
||||
def test_document_extraction_requires_service_token(monkeypatch):
|
||||
module = load_app_module(monkeypatch, service_token="secret")
|
||||
response = TestClient(module.app).post("/cv-import/extract", files={"file": ("resume.pdf", b"%PDF-synthetic")})
|
||||
assert response.status_code == 401
|
||||
|
||||
|
||||
def test_extract_text_rejects_extension_signature_mismatch(monkeypatch):
|
||||
module = load_app_module(monkeypatch)
|
||||
client = TestClient(module.app)
|
||||
|
||||
@@ -0,0 +1,137 @@
|
||||
"""Real, synthetic documents; no model artifacts or inference."""
|
||||
|
||||
import io
|
||||
import socket
|
||||
import sys
|
||||
import zipfile
|
||||
import subprocess
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
from docx import Document
|
||||
from pypdf import PdfWriter
|
||||
from pypdf.generic import DecodedStreamObject, DictionaryObject, NameObject, TextStringObject
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
|
||||
import parser_child
|
||||
|
||||
|
||||
def docx_bytes():
|
||||
document = Document()
|
||||
document.add_heading("Erfaring", level=1)
|
||||
document.add_paragraph("Utvikler | Eksempel AS | 2021–2023")
|
||||
document.add_paragraph("Bygget sikre API-er", style="List Bullet")
|
||||
table = document.add_table(rows=1, cols=2)
|
||||
table.cell(0, 0).text = "Språk"
|
||||
table.cell(0, 1).text = "Norsk, English"
|
||||
output = io.BytesIO()
|
||||
document.save(output)
|
||||
return output.getvalue()
|
||||
|
||||
|
||||
def pdf_bytes(*, pages=2, external=False):
|
||||
writer = PdfWriter()
|
||||
for number in range(pages):
|
||||
page = writer.add_blank_page(width=612, height=792)
|
||||
font = DictionaryObject({NameObject("/Type"): NameObject("/Font"), NameObject("/Subtype"): NameObject("/Type1"), NameObject("/BaseFont"): NameObject("/Helvetica")})
|
||||
page[NameObject("/Resources")] = DictionaryObject({NameObject("/Font"): DictionaryObject({NameObject("/F1"): writer._add_object(font)})})
|
||||
stream = DecodedStreamObject()
|
||||
stream.set_data(f"BT /F1 12 Tf 40 740 Td (Experience page {number + 1}) Tj 0 -20 Td (Engineer 2020-2022) Tj ET".encode())
|
||||
page[NameObject("/Contents")] = writer._add_object(stream)
|
||||
if external:
|
||||
writer._root_object[NameObject("/OpenAction")] = DictionaryObject({NameObject("/S"): NameObject("/URI"), NameObject("/URI"): TextStringObject("http://127.0.0.1/private")})
|
||||
output = io.BytesIO()
|
||||
writer.write(output)
|
||||
return output.getvalue()
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def no_network(monkeypatch):
|
||||
def denied(*args, **kwargs):
|
||||
raise AssertionError("Document extraction attempted network access")
|
||||
monkeypatch.setattr(socket.socket, "connect", denied)
|
||||
monkeypatch.setattr(socket, "getaddrinfo", denied)
|
||||
|
||||
|
||||
def test_docx_preserves_ordered_sections_lists_tables_and_unknown_pages():
|
||||
result = parser_child.extract_document(".docx", docx_bytes())
|
||||
assert result["method"] == "docling-docx-v1"
|
||||
assert result["ocr_used"] is False
|
||||
assert result["page_count"] is None
|
||||
assert [block["kind"] for block in result["blocks"]] == ["section_header", "paragraph", "list_item", "table"]
|
||||
assert result["blocks"][0] == {"id": "b0001", "text": "Erfaring", "kind": "section_header", "page": None}
|
||||
assert "2021–2023" in result["blocks"][1]["text"]
|
||||
assert "Norsk, English" in result["blocks"][3]["text"]
|
||||
assert "docx_page_numbers_unavailable" in result["warnings"]
|
||||
|
||||
|
||||
def test_native_pdf_keeps_page_evidence_and_declares_layout_limits():
|
||||
result = parser_child.extract_document(".pdf", pdf_bytes())
|
||||
assert result["method"] == "docling-native-pdf-v1"
|
||||
assert result["page_count"] == 2
|
||||
assert [block["page"] for block in result["blocks"]] == [1, 1, 2, 2]
|
||||
assert [block["id"] for block in result["blocks"]] == ["b0001", "b0002", "b0003", "b0004"]
|
||||
assert "Experience page 1" in result["blocks"][0]["text"]
|
||||
assert "pdf_native_order_no_layout_or_tables" in result["warnings"]
|
||||
assert "ocr_disabled" in result["warnings"]
|
||||
|
||||
|
||||
@pytest.mark.parametrize("extension,data,code", [
|
||||
(".pdf", b"not pdf", "pdf_signature"),
|
||||
(".pdf", b"%PDF-broken", "pdf_invalid"),
|
||||
(".pdf", pdf_bytes(pages=41), "pdf_page_limit"),
|
||||
(".pdf", pdf_bytes(external=True), "external_content"),
|
||||
(".txt", b"plain text", "unsupported_type"),
|
||||
], ids=["signature", "malformed", "pages", "external", "type"])
|
||||
def test_rejects_invalid_or_active_documents_before_conversion(extension, data, code):
|
||||
with pytest.raises(parser_child.ParserError) as failure:
|
||||
parser_child.extract_document(extension, data)
|
||||
assert failure.value.code == code
|
||||
|
||||
|
||||
def test_docx_rejects_external_relationships():
|
||||
payload = io.BytesIO()
|
||||
with zipfile.ZipFile(io.BytesIO(docx_bytes())) as source, zipfile.ZipFile(payload, "w", zipfile.ZIP_DEFLATED) as output:
|
||||
for entry in source.infolist():
|
||||
body = source.read(entry)
|
||||
if entry.filename == "word/_rels/document.xml.rels":
|
||||
body = body.replace(b"</Relationships>", b'<Relationship Id="external" Type="image" TargetMode="External" Target="file:///etc/passwd"/></Relationships>')
|
||||
output.writestr(entry.filename, body)
|
||||
with pytest.raises(parser_child.ParserError) as failure:
|
||||
parser_child.extract_document(".docx", payload.getvalue())
|
||||
assert failure.value.code == "external_content"
|
||||
|
||||
|
||||
def test_docx_rejects_archive_traversal():
|
||||
payload = io.BytesIO()
|
||||
with zipfile.ZipFile(io.BytesIO(docx_bytes())) as source, zipfile.ZipFile(payload, "w", zipfile.ZIP_DEFLATED) as output:
|
||||
for entry in source.infolist():
|
||||
output.writestr(entry.filename, source.read(entry))
|
||||
output.writestr("../secret", "inert")
|
||||
with pytest.raises(parser_child.ParserError) as failure:
|
||||
parser_child.extract_document(".docx", payload.getvalue())
|
||||
assert failure.value.code == "docx_unsafe_archive"
|
||||
|
||||
|
||||
def test_extraction_refuses_truncated_output_instead_of_silently_losing_text(monkeypatch):
|
||||
monkeypatch.setattr(parser_child, "MAX_EXTRACT_CHARS", 10)
|
||||
with pytest.raises(parser_child.ParserError) as failure:
|
||||
parser_child.extract_document(".pdf", pdf_bytes())
|
||||
assert failure.value.code == "extracted_text_limit"
|
||||
|
||||
|
||||
def test_document_child_uses_real_converter_and_cleans_no_source_files(tmp_path):
|
||||
source, output = tmp_path / "input.bin", tmp_path / "result.json"
|
||||
source.write_bytes(docx_bytes())
|
||||
result = subprocess.run([sys.executable, str(Path(parser_child.__file__)), "--input", str(source), "--output", str(output), "--extension", ".docx", "--mode", "document", "--memory-bytes", str(1024 ** 3)], capture_output=True, timeout=20)
|
||||
assert result.returncode == 0, (result.stderr, output.read_text() if output.exists() else "missing output")
|
||||
assert json.loads(output.read_text(encoding="utf-8"))["method"] == "docling-docx-v1"
|
||||
assert source.read_bytes().startswith(b"PK")
|
||||
|
||||
|
||||
def test_document_child_forbids_socket_and_external_process_side_effects():
|
||||
for event in ["socket.connect", "socket.connect_ex", "socket.getaddrinfo", "subprocess.Popen", "os.system"]:
|
||||
with pytest.raises(parser_child.ParserError) as failure:
|
||||
parser_child._deny_import_side_effects(event, ())
|
||||
assert failure.value.code == "external_content"
|
||||
Reference in New Issue
Block a user