jobtrackingapp/deploy/README.md

# Production deployment notes

## Gitea Actions
This repo includes `.gitea/workflows/ci-deploy.yml` for:
- backend build
- backend tests
- frontend tests
- frontend production build
- deployment to Ubuntu after successful tests on `main`

### Required secrets in Gitea
- `PROD_HOST`
- `PROD_USER`
- `PROD_SSH_KEY`

## Ubuntu server setup
Recommended app path:
- `/opt/job-tracker/app`

Persistent runtime secrets path:
- `/opt/job-tracker/shared/.env`

Requirements:
- Docker Engine
- Docker Compose plugin
- reverse proxy in front (Nginx, Caddy, or Traefik)
- shared env file present on server in `/opt/job-tracker/shared/.env`
- network connectivity from the backend container to your `mariadb` container/service

The deploy script will automatically create a symlink from:
- `/opt/job-tracker/shared/.env`

to:
- `/opt/job-tracker/app/.env`

This keeps secrets outside the uploaded repo checkout so they are not wiped by CI deploys.

### Frontend API base URL
The production frontend already proxies `/api` to the backend container via Nginx.

Recommended default:
- leave `REACT_APP_API_BASE_URL` unset/empty in production

Only set `REACT_APP_API_BASE_URL` if the UI must call a different external API origin on purpose.

## Example production `.env`
```env
DATABASE_PROVIDER=mariadb
JOBTRACKER_CONNECTION_STRING=server=mariadb;port=3306;database=jobtracker;user=jobtracker;password=REPLACE_ME
AUTH_JWT_KEY=replace_with_long_random_secret
AUTH_ADMIN_EMAIL=you@example.com
AUTH_ADMIN_PASSWORD=replace_with_strong_password
APP_PUBLIC_BASE_URL=https://your-domain.example
AI_SERVICE_BASE_URL=http://ai-service:8001
OLLAMA_BASE_URL=http://ollama:11434
OLLAMA_MODEL=qwen2.5:7b
EMAIL_FOLLOWUPREMINDERS_ENABLED=true
EMAIL_FOLLOWUPREMINDERS_UPCOMINGDAYS=2
# Optional backward-compatible alias if older config still references the previous name:
SUMMARIZER_BASE_URL=http://ai-service:8001
```

## Database recommendation
For production, yes — use a real database.

### Recommended direction
Short term:
- SQLite is acceptable for a single-user or very small deployment
- keep backups and volume persistence

Better production choice:
- MariaDB or PostgreSQL

### My recommendation
- **PostgreSQL** if you want the best long-term maintainability and fewer edge cases
- **MariaDB** is also fine if that is what you already know or host elsewhere

If you stay on SQLite:
- okay for small personal use
- not ideal for concurrent writes, larger scale, or operational robustness

## Practical recommendation for this project
If this app is going to be a real production service on Ubuntu:
- move to PostgreSQL first if possible
- MariaDB is still a reasonable option if preferred

## Deployment flow
1. push to `main`
2. Gitea Actions runs tests
3. if green, workflow uploads repo to server
4. `deploy/deploy.sh` links `/opt/job-tracker/shared/.env` into the repo checkout, then runs `docker compose build && docker compose up -d`
5. if `OLLAMA_MODEL` is set, the deploy script waits for Ollama, pulls the configured model if missing, then restarts `ai-service` so hybrid CV classification can use it
6. workflow checks service status after deployment

## Post-deploy verification you should also do manually the first time
- confirm reverse proxy routes to the frontend correctly
- confirm API auth/login works with production config
- confirm backend can connect to MariaDB
- confirm AI service container is reachable from backend
- confirm reminder and admin/system pages load
- verify follow-up reminder emails are enabled only when intended and that links open the correct job/tab
hat links open the correct job/tab

---

# Backups, restore and rollback

**Database restore and application rollback are two different operations.** A bad deploy usually needs
only the rollback. Restore the database only if the data itself is wrong or lost — it discards
everything written since the dump.

## Backup creation

`deploy/deploy.sh` takes a backup **before** it builds, stops or replaces anything, and **aborts the
deploy if the backup fails**. Nothing else in the deploy runs without a restore point.

- **Location:** `/opt/job-tracker/backups` — override with `BACKUP_DIR`.
- **Naming:** `jobtracker-<database>-<UTC timestamp>.sql.gz`, e.g.
  `jobtracker-jobtracker-20260719T153759Z.sql.gz`. The timestamp makes every file unique, so a deploy
  never overwrites an earlier backup.
- **SQLite deployments** (`DATABASE_PROVIDER` unset or `sqlite`) get the data volume instead:
  `jobtracker-sqlite-<UTC timestamp>.tar.gz`.
- **Credentials** come from `JOBTRACKER_CONNECTION_STRING` and are passed via `MYSQL_PWD`, never on the
  command line, so they cannot appear in the process list or the deploy log.
- **Compression:** gzip. A small database compresses to a few KB.
- **Verification:** the script rejects a dump that is empty or missing `CREATE TABLE`, because a
  truncated file that *looks* like a restore point is worse than none.

### Taking one by hand

```bash
BACKUP_DIR=/opt/job-tracker/backups
mkdir -p "$BACKUP_DIR"
MYSQL_PWD='<password>' mariadb-dump \
  --host=127.0.0.1 --port=3306 --user=<user> \
  --single-transaction --routines --events --quick \
  jobtracker | gzip -c > "$BACKUP_DIR/jobtracker-manual-$(date -u +%Y%m%dT%H%M%SZ).sql.gz"
```

### Retention

**Nothing is deleted automatically.** Backups accumulate in `BACKUP_DIR` until you remove them. Watch
disk usage and prune deliberately — a suggested policy is to keep every backup for 30 days and one per
month after that, but the script does not enforce it and will not delete your files.

## Restoring the database

Tested end-to-end against a clean MariaDB 11 container: dump taken from a seeded database, restored
into an empty one, rows verified identical.

```bash
# 1. Stop the application so nothing writes during the restore.
docker compose stop backend

# 2. Restore. This REPLACES the current contents of the named database.
gzip -dc /opt/job-tracker/backups/jobtracker-jobtracker-20260719T153759Z.sql.gz \
  | MYSQL_PWD='<password>' mariadb --host=127.0.0.1 --port=3306 --user=<user> jobtracker

# 3. Verify before starting anything.
MYSQL_PWD='<password>' mariadb --host=127.0.0.1 --port=3306 --user=<user> jobtracker \
  -e "SELECT COUNT(*) AS applications FROM JobApplications;"

# 4. Start again.
docker compose start backend
```

Restoring a **SQLite** deployment instead:

```bash
docker compose stop backend
docker run --rm -v jobtracker_data:/data -v /opt/job-tracker/backups:/backup \
  -e ARCHIVE_NAME=jobtracker-sqlite-20260719T153941Z.tar.gz \
  alpine:3 sh -c 'rm -rf /data/* && tar xzf "/backup/$ARCHIVE_NAME" -C /data'
docker compose start backend
```

## Restoring application containers (rollback)

This is the usual fix for a bad deploy, and it **does not touch the database**.

```bash
cd /opt/job-tracker/app          # the deployment checkout
git log --oneline -5             # find the last good commit
git checkout <previous-commit>
deploy/deploy.sh
```

`deploy.sh` takes a fresh backup first, so rolling back is itself protected.

**Why a code rollback is safe here:** every Phase 4/5 migration is a no-op — the startup reconciler
owns those tables — so reverting the code never leaves migration history ahead of the schema. The
reconciler is additive and never drops a table holding rows, so the older code simply ignores the
newer tables.

**What a code rollback does not undo:** rows users created in the newer tables stay. That is usually
what you want. If you additionally restore the database, those rows are lost — so restore only when
the data is the problem.

## Choosing between them

| Symptom | Action |
|---|---|
| New version starts but behaves wrong | Rollback the code. Leave the database. |
| Backend will not start; schema looks wrong | Rollback the code, then restore only if it still fails. |
| Data is missing or corrupted | Restore the database from the most recent good dump. |
| Deploy aborted before starting | Nothing to undo — the backup ran before any change. |

## Health checks

`backend` and `frontend` both report container health, so `docker compose ps` shows real state rather
than merely "running".

- **Backend:** `curl -fsS http://127.0.0.1:8080/health`. Anonymous, and deliberately **does not touch
  the database** — a health check that queried MariaDB would restart a healthy backend whenever the
  database blipped. `start_period` is 90s to cover first-boot schema reconciliation.
- **Frontend:** `wget` against nginx on port 80.
- `frontend` waits for `backend` to be *healthy*, not merely started, because nginx proxies `/api` to
  it and refuses to start if the upstream cannot be resolved.

A backend that cannot reach its database exits and is reported `unhealthy`, so a broken deploy does not
present as a running stack.