test(cv-import): add frozen corpus and bounded accuracy qualification

2026-10-26 16:00:55 +01:00
parent cd67c0df21
commit b62962c68a
12 changed files with 541 additions and 1 deletions
@@ -0,0 +1,29 @@
+# Synthetic CV benchmark
+
+This corpus contains invented people/organisations and exhaustive canonical annotations. Six development and eight frozen logical cases produce 14 PDFs and 13 DOCX files (the scan has only PDF). Keep actual frozen documents and `manifest.json` together when comparing models; DOCX ZIP timestamps mean regeneration is not a claim of byte identity. Never tune on the frozen set and then describe it as unseen.
+
+Generate into a new directory with a Python environment containing ReportLab, Pillow, python-docx and pypdf:
+
+```powershell
+python docs/cv-import/benchmarks/generate.py --output .artifacts/cv-import/corpus-new
+python -m unittest discover -s docs/cv-import/benchmarks -p test_*.py -v
+```
+
+For each manifest document invoke the real bounded parser using the separately installed, hash-locked Docling interpreter:
+
+```powershell
+python tools/summarizer/parser_child.py --input CASE.pdf --output CASE.pdf.extracted.json --extension .pdf --mode document --cpu-seconds 30 --memory-bytes 1073741824
+```
+
+Exit 2 with `no_text` is the expected scan rejection. Execute sequentially in the same offline/private environment as the service. Do not replace these results with synthetic model responses. Build and run the C# harness against the resulting directory:
+
+```powershell
+dotnet run --project docs/cv-import/benchmarks/harness --configuration Release -- EXTRACTIONS pdf FRESH-PDF-RESULTS.json
+python docs/cv-import/benchmarks/evaluate.py --actual FRESH-PDF-RESULTS.json --output FRESH-PDF-METRICS.json
+```
+
+Repeat for `docx`, supplying a UTF-8 annotation subset containing only IDs with actual DOCX fixtures via `--corpus`. Use `--split development` or `--split frozen` for separate summaries. The evaluator keeps employer/title/institution/qualification/date identity attached to record facts, rejects extra output IDs and counts missing cases against completion. Dates retain original precision; whitespace/case normalization does not equate years with invented months.
+
+Precision/recall count annotated field values with multiplicity and record association. Exact employment/education/skills compare complete categories; a case with no expected records can also match exactly, so these totals are not counts of successfully reconstructed jobs. `unsupportedClaims` includes wrong categories/associations as well as text absent from annotations; report those separately from invented source words. Review burden is the number of proposed field decisions, not observed human correction time. All significant facts require user approval regardless of benchmark score.
+
+Future model runs must use the same frozen manifest, prompt/schema version and oracle, record immutable model/runtime IDs, raw suggestions and rejected claims, and report deterministic/model-assisted stages separately. Real AI accuracy is currently BLOCKED. See ../QUALITY.md and ../CV-QA-01.md.