cesnimda
32 lines
7.5 KiB
Markdown
32 lines
7.5 KiB
Markdown
# Model/cache qualification architecture and fixture identity
|
|
|
|
Source baselinef17dba2,25October2026. Actual Python consumers are `AutoTokenizer` and `AutoModelForSeq2SeqLM` for fixed BART summarisation in `tools/summarizer/app.py:96`; they initialise lazily, or eagerly only under the existing environment flag. Document parsing uses pypdf/PyMuPDF/python-docx/Pillow and resource-bounded child processes. OCR uses the native Tesseract executable and its English traineddata, not a Transformers OCR model. No Python embedding/reranking/Docling model is implemented in this source. Ollama is an HTTP client boundary with its own separately managed model storage; no Ollama model/cache/server was accessed.
|
|
|
|
The selected full repository `sshleifer/distilbart-cnn-12-6` advertises Apache-2.0 and immutable revision `a4f8f3ea906ed274767e9906dbaede7531d660ff`. Its1,222,317,369-byte PyTorch checkpoint has authoritative LFS SHA256 `3bac65d18c99463302d12ca75c2220ea714f9c81ce235f205fa818efe71df6ea`. It was not available in the checked local user cache or existing local sidecar image. Local Docker context is desktop-linux; the only volume is the distinct MariaDB development volume. No production filesystem/cache was inspected. Old image contents are not approved current-runtime artifacts.
|
|
|
|
## Small task-authorised fixtures
|
|
|
|
[ARTIFACTS.json](model/ARTIFACTS.json) records two independently hashed, isolated fixtures. Their tokenizer/vocabulary/merges/model-card and reference configuration come from the immutable upstream revision. Every downloaded ordinary Git file is checked against the published Git blob identity, then pinned with SHA256. The initial small-fixture stage downloaded1,358,673 bytes of upstream metadata/tokenizer content. It used no pretrained checkpoint or remote Python code. The separate trained-checkpoint follow-up below is additional evidence, not a relabelling of those small fixtures.
|
|
|
|
Synthetic BART weights are generated locally with fixed seed73421 using the Apache-2.0 Transformers BART implementation: vocabulary50265, dimension16, one encoder/decoder layer, two heads,32-wide feed-forward layers and1024 positions. The842704 parameters are saved once as Safetensors and once as a known state dictionary. These files are approximately3.58MB each. The second format exercises the same PyTorch serialization family as the selected full checkpoint, with explicit `weights_only=True`. No third-party pickle or trained-weight licence is assumed. The actual tokenizer's Apache-2.0 declaration is preserved from its pinned card; no model weights are redistributed in Git.
|
|
|
|
These are real native BART/tokenizer/generation artifacts, with deliberately untrained/random outputs. They establish the tested library/API/format/CPU compatibility scope. They do not establish full DistilBART output quality, production memory/latency, GPU compatibility, intended model semantics or C11/P5-13 acceptance. No owner approval of a production model or formal risk exception is implied by task-authorised test preparation.
|
|
|
|
The committed manifest is the integrity authority, not an arbitrary cache-local JSON file. The verifier rejects missing/corrupt/linked/duplicate/traversal/unlisted artifacts before loading. The fixture mount is read-only; runtime networking is none, with explicit Hub/Transformers offline flags, private tmpfs cache and twoCPU/2GiB/128PID limits. Each container has an owned label, bounded parent deadline and removal verification. Replacing source/checkpoint identities requires a new manifest and qualification.
|
|
|
|
The production model choice is unchanged. A minimal loader repair now pins tokenizer and weights to the same reviewed revision, denies remote code and explicitly requests restricted weights loading. Its new unit test failed for the missing revision before the repair; the complete native Windows suite passed59 cases afterward. That suite alone is not real-model evidence; Linux fixture inference is separate.
|
|
|
|
## Cache ownership and supported transition
|
|
|
|
filelock4 changes shared-cache lock protocols. All qualification caches are new isolated disposable directories with a single library generation. No production or developer cache was reset, rewritten or silently mixed with old lock participants. A later production rollout must stop old participants and explicitly select a separately prepared/verified cache or establish documented protocol migration compatibility. That operator action is separate from development fixture compatibility and remains withheld.
|
|
|
|
Actual compatibility, negative/offline/cache contention/OCR results and any limitations are recorded separately as they execute. The separate trained-checkpoint follow-up uses the actual configured Apache-2.0 repository and pinned1,222,317,369-byte checkpoint, with sufficient nonproduction storage/CPU/memory verified before preparation. Its actual download, integrity, offline Hub-cache loading and generation outcome is reported in validation.json; a failed or unavailable follow-up is not replaced by the small fixture. No full-model semantic, GPU or production claim follows.
|
|
|
|
The full checkpoint is necessary because the tiny fixture does not reproduce the saved trained configuration: the selected model has1024-wide embeddings,12 encoder/6 decoder layers and50264 model vocabulary entries, while its tokenizer has50265 entries. The small fixture deliberately uses16-wide embeddings and50265 entries. Successful tiny-model inference cannot prove compatibility of those original tensors/configuration with Transformers5/Torch2.13. The follow-up keeps the existing model choice, uses synthetic text and constrains the nonproduction process to4CPU/8GiB/128PIDs/300seconds. It exercises the pinned repository revision through a private offline Hub snapshot cache, with blobs pointing to the already verified read-only artifact files. No production or Ollama cache participates.
|
|
|
|
For replay, [full-model/prepare.py](full-model/prepare.py) accepts only a new nonproduction directory, the committed immutable source revision and HTTPS repository paths. It checks licence declaration, available space, exact sizes and SHA256 before promoting each partial file. Existing directories are rejected without replacement. [full-model/ARTIFACTS.json](full-model/ARTIFACTS.json) is generated only after successful original preparation; its absence means the follow-up has not supplied a validated artifact. Use the exact qualified image and offline command retained with the actual result; a new runtime requires requalification.
|
|
|
|
The native English OCR artifact is independently identified by [OCR_ARTIFACT.json](model/OCR_ARTIFACT.json): Debian `tesseract-ocr-eng`1:4.1.0-2,4113088bytes, SHA256`7d4322bd2a7749724879683fc3912cb542f19906c83bcc1a52132556427170b2`. Its installed [copyright notice](model/OCR_COPYRIGHT.txt) identifies `tessdata_fast` and Apache-2.0. This is a file identity inside the exact measured local image, not a claim that a future floating apt install is immutable. Actual PNG/image-PDF OCR is part of the parser qualification.
|
|
|
|
The two exact small fixtures are retained in a local9,895,179-byte replay capsule, [fixture-capsule.json](model/fixture-capsule.json), SHA256`a1aff347e5f08f82e8ea7176d71d00e7f83ea5735acbdc4e3bf02c73deacb643`. Archive CRC and every artifact hash were checked. Verify that archive identity, then restore only the manifest-listed members into new isolated `artifact`/`artifact-bin` directories and run `verify_fixture` before loading. No weights were committed or uploaded. The original generator used Torch2.6/Transformers4.48; regenerating under different libraries is not promised to produce byte-identical serialization and requires a new manifest/qualification.
|