jobtrackingapp/docs/todo/ollama.md

Create and implement a production-safe, local-first AI architecture for JobTracker.

This is an authorised production infrastructure and application change, but proceed conservatively with inspection, benchmarking, backups, staged rollout and rollback capability.

Do not guess production connection details. Use only an existing documented SSH host, deployment mechanism or configured environment. Do not scan the network.

If production access is unavailable, complete all safe repository-side design and implementation work, document the exact access blocker, and stop before inventing credentials or connection details.

## Objectives

1. Analyse the actual production machine.
2. Identify the best current Ollama model for JobTracker’s real workloads.
3. Benchmark suitable candidates rather than selecting solely from published claims.
4. Install the selected model.
5. Make local Ollama the default production AI provider.
6. Make external AI providers secondary fallbacks.
7. Keep CVs, job descriptions and email data local whenever practical.
8. Introduce durable queuing so AI work does not block web requests.
9. Prevent Ollama congestion from hanging the application.
10. Provide progress, retry and failure states to users.
11. Preserve privacy, tenant isolation and Pro entitlement enforcement.
12. Make the deployment observable and safely reversible.

Do not remove existing models, configuration or external providers without explicit approval. Preserve them for rollback.

## Existing audit requirements

Read:

* All applicable `AGENTS.md` files
* Deployment documentation
* Docker and Compose configuration
* Environment examples
* AI-provider configuration
* Python/FastAPI AI implementation
* ASP.NET AI endpoints and hosted services
* Frontend AI workflows
* `docs/audits/full-application-audit.md`
* `docs/audits/security-threat-model.md`
* `docs/audits/audit-remediation-backlog.md`
* `docs/audits/verification-log.md`
* `docs/plans/post-audit-ux-reliability-program.md`, if present

Pay particular attention to:

* JT-005 inert tenant-scoped hosted services
* JT-006 document-parser isolation
* AI privacy controls
* Notification persistence
* Pro entitlement enforcement
* Strategy Snapshot timeouts
* Long-running CV processing
* External-provider data exposure
* Tenant context inside background jobs

Do not activate currently inert background workers until tenant resolution, persistent notification handling, AI privacy controls and entitlement enforcement are correct.

## Phase 1: Read-only production inventory

Before changing production, gather a sanitised read-only inventory.

Record:

### Operating system

* Distribution and version
* Kernel
* Uptime
* Time zone
* Current load
* Relevant system limits

### CPU and memory

* CPU model
* Physical and logical cores
* Total and available RAM
* Swap size and usage
* Memory pressure
* Other memory-intensive services

### GPU

* Exact GPU model
* VRAM
* NVIDIA driver
* Reported CUDA compatibility
* Current GPU processes
* Idle and loaded VRAM
* Temperature and power state where available
* Whether Ollama is actually using the GPU
* CPU/GPU layer offloading behaviour

### Storage

* Available capacity
* Ollama model storage location
* Filesystem
* Model sizes
* Space required for benchmark candidates
* Production database and attachment storage
* Backup capacity

Do not expose unrelated filenames or private user data.

### Ollama

* Installed version
* Installation method
* Service manager
* Service configuration
* Bound address
* Existing environment variables
* Installed models
* Currently loaded models
* Current model usage
* Existing API clients
* Health
* Logs relevant to performance or failures

Do not print secrets or complete prompt contents from production logs.

### Application deployment

* Running JobTracker services
* Docker/container versions
* Network topology
* Ollama connectivity
* Reverse-proxy timeouts
* AI service configuration
* Current external provider order
* Current model selections
* Background workers
* Queue implementation, if any
* Health checks
* Restart behaviour
* Resource limits
* Monitoring

Confirm whether the remembered specification of 32 GB RAM and GTX 1060 6 GB is accurate. Use measured production data as the source of truth.

Create:

`docs/production/production-ai-hardware-assessment.md`

Do not include hostnames, public IP addresses, credentials, tokens or other sensitive infrastructure identifiers.

## Phase 2: Production safety and backup

Before changing anything:

1. Record the current production configuration.
2. Back up affected non-secret configuration safely.
3. Record current Ollama and external-provider defaults.
4. Record currently installed model names and digests.
5. Confirm sufficient free disk space.
6. Define rollback commands.
7. Confirm how services will be restarted.
8. Identify expected interruption.
9. Verify that application and database backups are not affected.
10. Confirm Ollama is not publicly exposed.

Ollama should be reachable only through:

* Localhost
* A private container network
* An explicitly authorised private service network

It must not receive an unauthenticated public Traefik route.

Do not include secret values in the backup report.

Create:

`docs/production/production-ai-rollout-and-rollback.md`

## Phase 3: Workload inventory and evaluation set

Inventory every JobTracker AI task.

Likely tasks include, but are not limited to:

* CV text normalisation
* Career Profile extraction
* Conservative CV merge suggestions
* Job-description summarisation
* Job keyword and phrase extraction
* Skill matching
* Missing-skill identification
* Application strategy generation
* Strategy Snapshot
* CV tailoring suggestions
* Cover-letter drafting
* Follow-up drafting
* Email classification
* Recruitment-message detection
* Interview preparation
* Writing improvement
* Grammar correction
* English and Norwegian content
* Structured JSON generation

For each task record:

* Input type
* Typical input size
* Maximum expected input size
* Required output format
* Structured-output requirements
* Language
* Latency target
* Quality importance
* Privacy sensitivity
* Whether external fallback is permitted
* Whether it is interactive or background work
* Whether deterministic code should replace AI
* Current provider/model
* Pro entitlement requirement

Do not use AI for deterministic operations that are more reliably handled with code.

For example, Norwegian/English stop-word removal and basic keyword cleanup should remain deterministic even when an AI model contributes semantic phrase extraction.

Create a sanitised evaluation dataset using synthetic or redacted data.

The evaluation set must cover:

* English CV
* Norwegian CV
* Mixed-language CV
* English job advert
* Norwegian job advert
* Noisy imported job advert
* Technology-heavy role
* Sparse role
* Email classification
* Follow-up draft
* Strategy Snapshot
* Strict JSON response
* Malformed or adversarial document text
* Prompt-injection-style text inside a job advert or email
* Long input
* Empty and invalid input

Do not place real CV or email contents in the repository.

The previously authorised CV may be used locally for final local-only validation:

`F:\Documents\Work\CV and stuff\New CV\Connor Babbington - CV -English-.pdf`

Restrictions:

* Never commit it.
* Never send it to an external AI provider.
* Never expose its contents in reports.
* Remove temporary copies.
* Use synthetic fixtures for repeatable automated tests.

## Phase 4: Candidate model selection

Do not assume that the largest model is best for production.

Select candidates appropriate to the measured hardware and JobTracker workloads.

At minimum, consider the current locally available variants of:

* `qwen3.5:4b`
* `qwen3:4b`
* `gemma3:4b`
* The currently installed production model

If measured hardware and disk space permit, optionally benchmark:

* `qwen3:8b`
* `qwen3.5:9b`

Do not download very large models that clearly cannot run acceptably on the measured machine.

Do not select cloud-labelled Ollama models. The selected primary must execute locally.

For every candidate, verify:

* Exact tag
* Download size
* Quantisation
* Licence and hosted-use implications
* Required Ollama version
* Tool/structured-output compatibility
* Context requirements
* VRAM residency
* CPU offloading
* Peak system RAM
* Peak VRAM
* Load time
* Time to first token
* Tokens per second
* Total latency
* Output quality
* JSON validity
* Instruction adherence
* English quality
* Norwegian quality
* Hallucination rate in the evaluation set
* Behaviour on prompt-injection-style content
* Failure behaviour
* Stability over repeated requests

Do not rely only on synthetic benchmark scores published by model authors.

## Phase 5: Context and performance tuning

Do not use the model’s advertised maximum context automatically.

Benchmark practical context sizes such as:

* 4K
* 8K
* 16K only if measured resources permit

Select the smallest context that reliably supports JobTracker tasks.

Measure the effect of:

* Context size
* Prompt length
* Output-token limit
* Thinking/reasoning mode
* Temperature
* Structured output
* K/V cache type
* GPU offloading
* Model keep-alive
* Parallel requests

Prioritise reliable latency and bounded memory over theoretical maximum context.

For a single 6 GB GPU, start evaluation conservatively with:

* One loaded model
* One parallel inference
* Bounded Ollama queue
* Bounded application queue
* Limited context
* Model kept warm only when resource usage is acceptable

Treat these as benchmark starting points, not unquestionable final values:

```text
OLLAMA_MAX_LOADED_MODELS=1
OLLAMA_NUM_PARALLEL=1
OLLAMA_MAX_QUEUE=<measured bounded value>
OLLAMA_KEEP_ALIVE=<measured duration>
```

Evaluate `OLLAMA_KV_CACHE_TYPE=q8_0` only if supported and beneficial. Record any measured quality or memory difference.

Do not enable options unsupported by the installed GPU/runtime merely because they exist in documentation.

## Phase 6: Model decision

Create:

`docs/production/ollama-model-benchmark.md`

Include a table containing:

* Candidate
* Size
* Quantisation
* VRAM
* RAM
* GPU offload
* Context
* Load time
* First-token latency
* Total latency
* Tokens/second
* Quality score by JobTracker task
* JSON success rate
* Norwegian quality
* Failure rate
* Licence notes
* Recommendation

Select:

1. Primary local model
2. Optional lightweight fallback local model
3. Tasks that should use deterministic processing
4. Tasks that genuinely require external fallback

The selected default must be based on measured JobTracker performance.

If the remembered GTX 1060 6 GB specification is confirmed, use `qwen3.5:4b` as the initial leading candidate, but select another model if evidence shows it performs better.

Do not choose `qwen3.5:9b` merely because it is larger if GPU offloading makes it too slow or unstable.

## Phase 7: Local-first provider architecture

Refactor AI provider selection into one explicit, testable policy.

Provider priority should normally be:

1. Deterministic local processing where appropriate
2. Primary local Ollama model
3. Optional secondary local model
4. Configured external provider only when policy permits
5. Clear failure with retry option

Do not scatter provider selection throughout controllers and components.

Create a central AI routing policy that considers:

* Task type
* Required capability
* Privacy classification
* Pro entitlement
* User/admin external-AI preference
* Local health
* Queue depth
* Deadline
* Retry count
* Previous provider failure
* External-provider availability
* Cost/budget controls

### Privacy rules

Never send the following externally without an explicit server-side policy and appropriate user permission:

* Full CV
* Career Profile
* Email body
* Attachments
* Provider tokens
* Private notes
* Contact information
* Personally identifying application data

Where external fallback is allowed:

* Send only the minimum required data.
* Record which provider handled the request.
* Make fallback visible in administrative diagnostics.
* Respect a `local only` user or administrator setting.
* Never silently override an external-AI opt-out.
* Do not expose provider secrets to the browser.
* Apply Pro entitlement before work is queued.

If an external provider is unavailable or prohibited, return a useful queued/failed state rather than hanging.

## Phase 8: Durable AI queue

Do not keep long-running AI work inside the original HTTP request.

Implement a persistent queue using the project’s existing infrastructure where practical.

Before introducing a new queue library, evaluate:

* Existing database-backed job infrastructure
* Existing PostgreSQL/MariaDB capabilities
* Existing Redis deployment
* Existing hosted workers
* Existing outbox patterns

Choose the smallest reliable design that survives process restart.

### Required job states

Use explicit states such as:

* Queued
* Running
* Succeeded
* Failed
* Cancelled
* WaitingForRetry
* WaitingForExternalFallback

Persist:

* Operation ID
* Tenant/user ID
* Task type
* Entitlement decision
* Privacy/fallback policy
* Sanitised request metadata
* Provider selected
* Model
* Attempt count
* Creation/start/completion timestamps
* Progress stage
* Failure category
* Result reference
* Idempotency key

Do not store unnecessary raw CV or email contents in queue records.

### API behaviour

Long-running endpoints should:

1. Validate authorization, tenant ownership and Pro entitlement.
2. Create an idempotent operation.
3. Return `202 Accepted`.
4. Return a stable operation ID and status URL.
5. Let the frontend poll or receive safe push updates.
6. Permit safe retry where appropriate.
7. Avoid duplicate work from double clicks.
8. Survive browser refresh and application restart.

### Worker behaviour

Workers must:

* Resolve tenant context explicitly.
* Avoid the tenant-filter failure identified in JT-005.
* Re-check entitlement and cancellation before expensive work.
* Claim jobs atomically.
* Use leases/heartbeats so abandoned jobs recover.
* Enforce per-task timeout.
* Use bounded retries with jitter.
* Distinguish retryable and permanent failures.
* Avoid processing one job twice.
* Persist results transactionally.
* Generate persistent user notifications.
* Avoid logging raw private content.
* Shut down gracefully.
* Recover in-progress work after restart.

## Phase 9: Backpressure and Ollama congestion

The application queue must protect Ollama rather than forwarding unlimited parallel requests.

Implement:

* Configurable worker concurrency
* Per-model concurrency
* Global local-AI concurrency
* Bounded queue capacity
* Request prioritisation
* Load shedding
* Queue-age limit
* Operation deadline
* Circuit breaker
* Health checks
* Retry-after guidance
* Cancellation
* Graceful degradation

Suggested priority order:

1. Interactive user-requested operations
2. User-visible CV/job analysis
3. Email/follow-up drafting
4. Scheduled enrichment
5. Bulk/background regeneration

Do not let scheduled jobs starve interactive work.

For likely single-GPU deployment, begin with one Ollama inference at a time unless benchmarks demonstrate safe parallelism.

Do not allow both Ollama’s internal queue and the application queue to grow without effective bounds.

If Ollama returns 503, times out, crashes, unloads unexpectedly or becomes unhealthy:

1. Record the local attempt.
2. Apply bounded local retry where appropriate.
3. Evaluate the external-fallback policy.
4. Use external fallback only if permitted.
5. Otherwise leave a clear recoverable failure.
6. Never leave the frontend spinner running indefinitely.

## Phase 10: External fallback behaviour

External providers are secondary, not peers selected arbitrarily.

Define fallback triggers such as:

* Ollama unavailable
* Local circuit open
* Local operation exceeds its deadline
* Local model lacks a required capability
* Repeated schema-validation failure
* Explicit user request for an allowed external-quality operation

Do not trigger external fallback merely because the local queue is briefly busy unless the configured queue deadline would be exceeded and privacy policy permits it.

Before external fallback:

* Re-check user consent
* Re-check Pro entitlement
* Re-check task sensitivity
* Minimise the payload
* Apply cost limits
* Record the reason
* Avoid duplicate local and external completion

Never allow both providers to complete and charge for the same operation without an explicit race strategy and deduplication.

Make provider preference configurable by administrators:

* Local only
* Local first with approved fallback
* External only for specifically allowed tasks

Default production behaviour should be local first.

## Phase 11: Frontend queued-operation experience

Replace indefinite spinners with explicit durable operation states.

For Strategy Snapshot, CV processing, job analysis and other applicable AI actions, show:

* Queued
* Processing locally
* Waiting
* Retrying
* Using approved fallback where disclosure is appropriate
* Completed
* Failed
* Cancelled

Provide:

* Clear status
* Safe page refresh
* Navigation away without losing the operation
* Notification on completion
* Retry action
* Cancel action where supported
* No duplicate submission
* Useful error text
* No exposure of internal prompts or secrets

Do not display an unreliable queue position as an exact promise.

Free users must receive the established Pro locked state before an AI job is created.

## Phase 12: Observability

Add privacy-safe telemetry for:

* Queue depth
* Oldest queued job age
* Time spent queued
* Processing duration
* First-token latency where available
* Tokens/second
* Prompt and completion token estimates
* Model load duration
* GPU/RAM usage where safely obtainable
* Success/failure rate
* Timeout rate
* Retry rate
* Cancellation rate
* External-fallback rate
* Provider/model usage
* Schema-validation failures
* Jobs recovered after restart

Add health signals for:

* Ollama reachable
* Selected model installed
* Model warm/cold
* Queue worker running
* Queue stalled
* External provider configured
* Circuit state

Do not include CV, job-description or email contents in metrics.

Create an operational runbook:

`docs/production/local-ai-operations-runbook.md`

Include:

* Health checks
* Queue inspection
* Model status
* Safe restart
* Draining workers
* Cancelling stuck jobs
* Circuit reset
* Rollback
* External fallback disablement
* Disk management
* Model update procedure
* Incident diagnostics

## Phase 13: Installation and staged production rollout

Only after inventory and benchmarks:

1. Pull the selected model.
2. Verify its digest and size.
3. Confirm sufficient disk remains.
4. Configure conservative Ollama limits.
5. Keep the previous model installed.
6. Update the application’s production model default.
7. Configure local-first routing.
8. Deploy durable queue changes.
9. Apply required database migrations safely.
10. Start one worker.
11. Run health checks.
12. Run synthetic smoke tests.
13. Run local-only testing with the authorised CV if safe.
14. Verify no external provider received that CV.
15. Verify Free/Pro entitlement.
16. Verify tenant isolation.
17. Verify restart recovery.
18. Verify timeout and fallback behaviour.
19. Observe resource usage.
20. Expand traffic only after successful validation.

Do not delete the old model after rollout.

If production becomes unhealthy:

* Stop accepting new AI jobs.
* Drain or preserve queued work.
* Roll back application configuration.
* Restore the prior model default.
* Restart affected services safely.
* Verify the previous path.
* Preserve diagnostic evidence.

## Phase 14: Validation matrix

Create:

`docs/production/production-ai-validation.md`

Test:

* Local keyword extraction
* Norwegian job analysis
* English job analysis
* CV extraction
* Strategy Snapshot
* CV suggestion
* Email classification
* Follow-up draft
* Strict JSON output
* Prompt-injection-style input
* Concurrent submissions
* Double click
* Queue congestion
* Ollama offline
* Ollama timeout
* Ollama restart
* Application restart
* Worker restart
* External fallback allowed
* External fallback prohibited
* External provider unavailable
* User cancellation
* Free user
* Pro user
* Two different tenants
* Browser refresh
* Notification delivery
* Failed result retry

Classify each as:

* Passed locally
* Passed with approved fallback
* Failed
* Blocked
* Mock-tested
* Not applicable

## Tests

Add or update:

* AI-routing unit tests
* Entitlement tests
* Privacy-policy tests
* Provider-fallback tests
* Queue state-machine tests
* Atomic claim tests
* Lease recovery tests
* Idempotency tests
* Retry tests
* Cancellation tests
* Tenant isolation tests
* Restart recovery integration tests
* Ollama adapter tests
* External adapter tests
* Frontend queued-state tests
* End-to-end Strategy Snapshot tests
* End-to-end CV-processing tests

Do not weaken existing tests.

## Final report

Report:

1. Verified production hardware
2. Ollama version and configuration
3. Models benchmarked
4. Benchmark results
5. Selected primary local model and why
6. Selected context and tuning
7. Installed model and digest
8. Application provider order
9. Queue architecture
10. Privacy policy
11. External fallback rules
12. Production changes
13. Database migrations
14. Tests and results
15. Production smoke-test results
16. Resource usage before and after
17. Rollback procedure
18. Remaining limitations
19. Tasks still requiring external AI
20. Recommended future hardware upgrade, if evidence supports one

Do not claim the production rollout succeeded unless the deployed application, queue, Ollama model, restart recovery and local-first routing were genuinely verified.