cesnimda
105 lines
7.2 KiB
Markdown
105 lines
7.2 KiB
Markdown
# Reproducing the qualification lanes
|
|
|
|
Run from the repository root on the dedicated development branch. Use the pinned .NET9 SDK
|
|
and the existing frontend lock. Build Release API/tests and the production static export before
|
|
browser execution. Preserve each prior run directory; a successor is separate evidence, never a
|
|
hidden retry. The accepted result ledger records actual commands, timestamps, process outcomes,
|
|
native case identities and report hashes. Commands below describe the mechanism, not passing results.
|
|
|
|
All shell commands in this environment use `rtk proxy`. Windows examples use the existing native
|
|
MariaDB client under `.artifacts/mariadb/runtime/mariadb-12.2.2-winx64/bin/`; obtain and verify it
|
|
using the [database setup instructions](../../../database/mariadb-standardisation/README.md).
|
|
No command connects to production. Development credentials in fixtures are intentionally local.
|
|
|
|
## Existing full suites
|
|
|
|
- Backend: `dotnet test JobTrackerApi.Tests/JobTrackerApi.Tests.csproj --configuration Release --no-restore --logger "trx;LogFileName=backend.trx" --results-directory <new-output>/backend`.
|
|
- Real provider: `python scripts/mariadb-integration.py --output <new-output>/mariadb`.
|
|
- Frontend, from `job-tracker-ui`: `node node_modules/jest/bin/jest.js --runInBand --json --outputFile=<new-output>/frontend.json`.
|
|
- Types/lint/static build, from that directory: `node node_modules/typescript/bin/tsc --noEmit --incremental false`, `node node_modules/eslint/bin/eslint.js . --max-warnings=0`, `node --max-old-space-size=4096 node_modules/next/dist/bin/next build`.
|
|
- Sidecar, from `tools/summarizer`: set `SKIP_MODEL_LOAD=1` and `AI_SERVICE_SKIP_MODEL_LOAD=1`, then `.venv/Scripts/python.exe -m pytest -q tests --junitxml=<new-output>/sidecar.xml`. This does not qualify a model.
|
|
|
|
For ordinary browser discovery and execution, use the production static export with
|
|
`PLAYWRIGHT_STATIC_EXPORT=true`, `PLAYWRIGHT_PREBUILT_EXPORT=true`,
|
|
`PLAYWRIGHT_REUSE_SERVERS=false`, `NEXT_PUBLIC_API_BASE_URL=http://localhost:5302/api` and
|
|
`DOTNET_HOST_PATH=<pinned dotnet.exe>`. Direct fidelity outputs to fresh
|
|
`CV_FIDELITY_OUTPUT` and `CV_SETTINGS_FIDELITY_OUTPUT` directories. From `job-tracker-ui`, run
|
|
`node node_modules/@playwright/test/cli.js test --list --reporter=json`, then four **serial**
|
|
executions with `--shard=1/4` through `--shard=4/4`, each with `--retries=0 --reporter=list,json`,
|
|
a unique `--output` directory and `PLAYWRIGHT_JSON_OUTPUT_NAME`. Discovery-only empty results
|
|
are not runtime skips. Reconciliation requires exact discovery membership across all four shards.
|
|
|
|
Run each separate config serially with the same zero-retry/report settings:
|
|
|
|
| Lane | Config relative to `job-tracker-ui` |
|
|
| --- | --- |
|
|
| AI contract | `playwright.cv-ai.config.ts` |
|
|
| Auth recovery | `playwright.auth-recovery.config.ts` |
|
|
| Rate limits | `playwright.rate-limits.config.ts` |
|
|
| Ingress | `playwright.ingress.config.ts` |
|
|
| PDF admission | `playwright.pdf.config.ts` |
|
|
| Preferences | `../docs/redesign/phase-5/p5-15/p5-15-01/diagnostics/playwright.config.ts` |
|
|
| RD21 | `../docs/redesign/phase-9/rd21-remediation/diagnostics/playwright.config.ts` |
|
|
| Corrected diagnostics | `../docs/redesign/phase-9/p9-05/diagnostics/playwright.config.ts` |
|
|
| Volume | `../docs/redesign/phase-9/p9-05/volume/playwright.config.ts` |
|
|
| RD21 volume | `../docs/redesign/phase-9/rd21-remediation/volume/playwright.config.ts` |
|
|
|
|
Ordinary and these historical contract fixtures retain transitional SQLite where their current
|
|
configuration specifies it. They are never counted as MariaDB qualification.
|
|
|
|
## Dedicated MariaDB browser and sustained workload
|
|
|
|
Use only the pinned disposable service from `docker-compose.database.yml`, project
|
|
`jobtracker-it-p9-requal`, integration profile, with `MARIADB_TEST_PORT=33308`. Start only
|
|
`mariadb-test` with `up -d --wait`. The persistent `mariadb-dev` service and its volume are separate.
|
|
The helper verifies the exact container, Compose ownership, image pin, tmpfs, loopback binding
|
|
and server version before creating its generated schema. It refuses to overwrite earlier evidence.
|
|
|
|
With ports5302/3300 free and no other qualification workload, run these separately:
|
|
|
|
```powershell
|
|
rtk proxy python docs/redesign/phase-9/p9-requal-01/mariadb-lane.py mariadb-browser
|
|
rtk proxy python docs/redesign/phase-9/p9-requal-01/mariadb-lane.py mariadb-observations
|
|
rtk proxy python docs/redesign/phase-9/p9-requal-01/mariadb-lane.py performance
|
|
```
|
|
|
|
The performance lane takes at least15minutes after setup/preflight. Do not shorten it or run
|
|
competing tests/builds. Read [PERFORMANCE.md](PERFORMANCE.md) for its bounded claims.
|
|
Each invocation drops only the exact schema it created, after stopping its own test process.
|
|
Failures retain logs and cleanup evidence; an uncertain process shutdown retains the schema.
|
|
After all lanes, remove only this owned disposable service using Compose `rm --stop --force mariadb-test`
|
|
with the same file/project/profile. Never use `down -v` or remove the development volume.
|
|
|
|
## Native and Linux PDF fidelity
|
|
|
|
Once both existing CV producer files have completed, use Python with the repository's existing
|
|
PDF tooling (`pdfplumber`, `pypdfium2`, Pillow):
|
|
|
|
```powershell
|
|
rtk proxy <pdf-python> -B docs/redesign/phase-9/p9-requal-01/pdf-proof.py --platform windows --fidelity-root .artifacts/p9-requal-01/fidelity --settings-root .artifacts/p9-requal-01/settings-fidelity --output .artifacts/p9-requal-01/windows-pdf-oracle --record .artifacts/p9-requal-01/native-pdf.json
|
|
```
|
|
|
|
The helper stages copies, merges all eight settings states, runs the **unchanged** existing oracle,
|
|
checks fixture/page membership and hashes original inputs before/after. It will not overwrite output.
|
|
|
|
The Linux lane builds the **existing** API Dockerfile locally with tag
|
|
`jobtracker-p9-requal-api:security-check`, while retaining the build log and source identity.
|
|
With all other browser/API lanes stopped and the same verified disposable MariaDB service running,
|
|
execute `rtk proxy python docs/redesign/phase-9/p9-requal-01/linux-run.py`. It starts a generated,
|
|
task-labelled API container with tmpfs data and a private network, publishes only loopback5302,
|
|
verifies API identity/readiness and actual migration history, and executes five unchanged CV
|
|
producer cases. It records renderer/runtime/image identities and removes only its own resources.
|
|
Then run `pdf-proof.py --platform linux` against `.artifacts/p9-requal-01/linux/fidelity` and
|
|
`linux/settings-fidelity`, using fresh `linux-pdf-oracle` output and `linux-pdf.json` record.
|
|
This qualifies the Linux API renderer with a Windows Chromium preview client; physical printing,
|
|
Linux desktop browser behavior and human visual acceptance remain distinct.
|
|
|
|
## Fail-closed evidence checks
|
|
|
|
`test-reconcile.py`, `test-linux-run.py`, `test-mariadb-lane.py` and `mariadb-lane.py self-check` are offline harness
|
|
checks, not provider tests. `reconcile.py` reads the fixed fresh artifact paths and writes
|
|
`validation.json`; its default process fails if required gates are missing or not passing.
|
|
`--collect-only` deliberately collects partial evidence and cannot establish acceptance.
|
|
Frozen runtime input hashes must still match. Advisory findings and unavailable human/provider/
|
|
production evidence retain their own dispositions even when functional tests pass.
|