# CV-QA structural benchmark

This separate synthetic corpus has **18 logical cases: 8 development and 10 frozen**, each producing a PDF and DOCX (36 documents). It leaves `docs/cv-import/benchmarks` and its original annotations unchanged. Cases cover explicit employer/title and institution/qualification order, no headings, actual DOCX lists, explicit PDF list markers, multirow tables with actual header cells, current/year/month/day/missing dates, repeated employers and roles, overlaps, multiple qualifications, unknown headings, valid records followed by ambiguity, English/Bokmål/mixed text and long wrapped source.

The `expected` profiles are factual annotations written independently from interpreter output. Genuinely unlabeled employer/title and institution/qualification relationships have **no expected typed record**; their literal source belongs in `unresolved`. No company/title dictionaries or inferred order establish their field roles. Frozen cases are a fixed regression split, not evidence of unseen population accuracy; do not tune their expectations to observed output.

## Generate and self-check

Use an existing Python with ReportLab, python-docx and pypdf. No downloads, new dependencies or models are needed. From the repository root, replace `python` with the approved local executable:

```powershell
rtk proxy python -m unittest discover -s docs/cv-integration-qa/benchmarks -p test_benchmark.py -v
rtk proxy python docs/cv-integration-qa/benchmarks/generate.py --output .artifacts/cv-integration-qa/structural-corpus-01
```

Generation refuses existing fixture files/directories. Freeze the resulting documents with their manifest: the manifest records the corpus hash, generator hash and each file hash/size. DOCX archive timestamps mean regenerated files are not automatically byte-identical. The PDF table is a real ruled table without inserted pipe characters; model-free native PDF extraction has no qualified table inference. Its interpretation may fail while text recovery passes. Keep those results visible.

## Real extraction and interpretation

Run each manifest document **sequentially** in the existing separate, hash-locked Docling environment with its bounded child, offline/private limits and model loading disabled. This example describes one invocation; do not substitute a synthetic extraction:

```powershell
rtk proxy python tools/summarizer/parser_child.py --input FIXTURES/CASE.pdf --output EXTRACTIONS/CASE.pdf.extracted.json --extension .pdf --mode document --cpu-seconds 30 --memory-bytes 1073741824
```

Repeat for DOCX using `--extension .docx` and `CASE.docx.extracted.json`. Use the original unchanged C# harness against the new extraction directory, serialized with other .NET work:

```powershell
rtk proxy dotnet run --project docs/cv-import/benchmarks/harness --configuration Release -- EXTRACTIONS pdf FRESH-PDF-PROFILES.json
rtk proxy dotnet run --project docs/cv-import/benchmarks/harness --configuration Release -- EXTRACTIONS docx FRESH-DOCX-PROFILES.json
rtk proxy python docs/cv-integration-qa/benchmarks/evaluate.py --actual FRESH-PDF-PROFILES.json --extractions EXTRACTIONS --format pdf --output FRESH-PDF-METRICS.json
rtk proxy python docs/cv-integration-qa/benchmarks/evaluate.py --actual FRESH-DOCX-PROFILES.json --extractions EXTRACTIONS --format docx --split frozen --output FRESH-DOCX-FROZEN-METRICS.json
```

Use `--split development`/`--split frozen` for disjoint reports; a focused report is not added to its overlapping full report. An absent output still contributes its annotated facts and text fragments to the missing denominators. Extra output IDs are rejected. Metric outputs refuse overwrite.

## What each layer establishes

| Layer | Metric and denominator | Meaning |
|---|---|---|
| Docling text recovery | `doclingRecovery.literalFragmentRecall`: recovered literal source fragments / expected source fragments, with multiplicity | Whitespace/case/NFC-normalized substring recovery of paragraph/list/header/cell text; does not prove reading order, geometry, table semantics or field associations. |
| Deterministic interpretation | Existing original fact oracle, supplemented with micro/per-field precision, recall and F1; explicit expected/predicted/TP/FP/FN | Facts remain attached to employer/title/institution/qualification/date identity and multiplicity. Years do not equal invented months. Unknown-source text is scored separately. |
| Record reconstruction | Per category `complete` / expected records, `exact` / predicted records; counts supplied | Each expected record must match one output record with its associated facts. Exact also rejects extra facts. Empty categories have denominator zero and are not successful reconstructed records. |
| Source occurrence | `provenance.correct` / annotated source references | Expected literal source text and occurrence ordinal resolve to an actual extracted block. Both metadata block ID and page must match. Missing source/metadata fails; factual correctness cannot substitute for provenance. |
| Ambiguity retention | Unresolved-source precision/recall/F1 and `abstention.correctCases` / required cases | A correct abstention preserves all expected unresolved items under their literal heading (or null) without adding/removing typed facts. PDF/DOCX list markers are ignored only when comparing unresolved items. |
| Reviewed mappings/persistence | `reviewedMapping: NOT RUN` | This harness has no user review or canonical writes. Use independent HTTP/browser/provider evidence. |
| Real model accuracy | `realAiAccuracy: BLOCKED` | No inference occurs; deterministic tests cannot establish AI semantic quality. |

Factual `unsupportedClaims` can mean wrong association/category rather than invented text. `perField` keeps each field's denominators; record identity errors can affect multiple fields. A record missing any identity scalar fails its associated facts rather than receiving cross-record credit. Interpretation timings cover the existing harness call, not Docling extraction or user correction time.

## Explicit metadata regressions

`provenance` annotations use canonical source-order field paths plus `{text, occurrence}`; occurrence defaults to 1. The oracle resolves those expectations against the actual extraction, never assumed `bNNNN` identifiers. PDF pages are checked against actual extracted page provenance; DOCX page is null.

| Case | Expected metadata association |
|---|---|
| `frozen-repeated-overlapping-roles` | `jobs.0.company`, `.1.company`, `.2.company` reference the first, second and third `Company: Repeat AS` blocks; `jobs.1.title` references the second `Title: Engineer`, and its bullet references `Built second service.` |
| `dev-education-qualification-first` | `education.1.institution` references the second `Institution: Demo University`, despite identical institution text. |
| `frozen-date-precision` | `jobs.2.start` references the day-precision range; `jobs.3.title` references the fourth `Title: Tester`, without borrowing a previous date. |
| `frozen-repeated-heading-boundary` | Only the first occurrence achievement belongs to `jobs.0`; the later orphan list item remains unresolved. |
| `dev-valid-then-ambiguous` | The first explicit achievement remains on the resolved job; the later company/title/date/detail remains unresolved and cannot become that job's bullets. |

Only block identity/page is exposed by current product metadata. Multiple equal anchors inside one extracted block cannot be independently distinguished at line-span level; the report's `sourceOccurrenceRecovered` makes missing distinct occurrences visible. No claim of transported geometry, table-cell coordinates or character-span provenance is made.