fix(release): stop deployment when off-host backup transfer fails

deploy/deploy.sh

@@ -574,14 +574,20 @@ upload_current_backup_set() {
   remote="${remote%/}"
   local current_files=("$BACKUP_DIR"/jobtracker-*"-$BACKUP_STAMP".*) file
   for file in "${current_files[@]}"; do
-    rclone copyto "$file" "$remote/daily/$(basename "$file")" --immutable
+    if ! rclone copyto "$file" "$remote/daily/$(basename "$file")" --immutable; then
+      echo "Daily off-host backup upload failed. Aborting deploy."
+      return 1
+    fi
   done
 
   local month="$(TZ="${BACKUP_TIMEZONE:-Europe/Oslo}" date +%Y-%m)"
   if [ -d "$BACKUP_DIR/monthly/$month" ]; then
-    rclone copy "$BACKUP_DIR/monthly/$month" "$remote/monthly/$month" --immutable
+    if ! rclone copy "$BACKUP_DIR/monthly/$month" "$remote/monthly/$month" --immutable; then
+      echo "Monthly off-host backup upload failed. Aborting deploy."
+      return 1
+    fi
   fi
-  echo "Current recovery set copied to configured encrypted off-host remote."
+  echo "Current recovery set copied to configured off-host remote."
 }
 
 resolve_existing_web_proxy_subnet

docs/release-readiness/11_BACKUP_RESTORE_ROLLBACK.md

@@ -52,3 +52,12 @@ Both auth-config temporary-file cleanup calls occur before the rollback decision
 - `python scripts/test-deploy-backup.py -v` **PASSED 5/5** and `python scripts/test-mariadb-workflow.py -v` **PASSED 4/4**. Git Bash `bash -n deploy/deploy.sh` passed syntax validation.
 
 RR-12 remains open for an authorised isolated rehearsal. After the prior images are recreated, verify exact image identity, `/health`, `/ready`, and authenticated reads and writes against representative migrated and post-write data. A successful Compose call does not establish service health or recover schema, data, or the AI image. RR-13 still needs authorised off-host retrieval/decryption and recovery-set consistency verification. No Docker, network service, database, remote storage, deployment or restore was invoked by these checks.
+
+## RR-13 off-host upload failure handling
+
+Reproduced two transfer failures in the deployment gate using the real `upload_current_backup_set` function, synthetic files and a stub `rclone`. The caller uses `! upload_current_backup_set`, so Bash suppresses errexit inside that function. When either daily `copyto` or monthly `copy` returned 23, the old code still printed success and the gate returned 0. The healthy control selected only current-stamp daily files, then the monthly directory.
+
+The bounded fix explicitly returns failure on either copy error. A daily error stops before subsequent files and the monthly copy. A monthly error stops before the deployment continues. The successful path still uses immutable transfers and no longer claims the configured remote is encrypted. A failed upload may leave some remote files already transferred.
+
+- Red run: **2 failures and 1 passing control**. Green run: **8/8 backup tests** including the three upload cases. The independent rollback handler and hosted quarantine guard checks passed **5/5** and **4/4**. Bash syntax validation passed.
+- Encrypted off-host retrieval, decryption, completeness and consistent restore of database/files/keys/tombstones remain required. No actual transfer, restore, production configuration, keys or backup files were accessed.

docs/release-readiness/13_REMEDIATION_REGISTER.md

@@ -16,6 +16,6 @@ Discovery only, 2026-11-02. These are qualification gaps unless a concrete defec
 | RR-10 | AI quality/activation | Deterministic Docling fixtures are distinct from real-model semantic accuracy. Optional unqualified adapter remains disabled; no AI invocation authorised here |
 | RR-11 | Backup host parsing rejected supported `Host=` alias and uppercase spelling | FIXED AND OFFLINE-TESTED. See [bounded remediation](RR-11_IMPLEMENTATION.md). Real dump/restore qualification remains open |
 | RR-12 | Rollback recreates core images without post-rollback health qualification; no schema/data/AI reversal | OFFLINE HANDLER CHECKS PASSED 5/5. See [failure-injection evidence](11_BACKUP_RESTORE_ROLLBACK.md#rr-12-offline-failure-injection). Actual image recovery and health/readiness after migration/writes still require isolated rehearsal |
-| RR-13 | Off-host encryption and recovery-set consistency not established by script success messages | NOT TESTED operational qualification. Verify remote encryption, retrieval/decryption, database/files/key/tombstone consistency in approved isolation |
+| RR-13 | Off-host encryption and recovery-set consistency need independent proof. A failed transfer previously let the deployment continue | Upload failure gate FIXED AND OFFLINE-TESTED. Actual encryption, retrieval/decryption and database/files/key/tombstone consistency require approved isolation |
 
 Continue evidence work against the [source endpoint and environment census](03_ENDPOINT_ENVIRONMENT_CENSUS.md). Actual qualification requires an accepted candidate and isolated environment. Every fix needs minimal scope, targeted verification and the applicable stage approval.

docs/release-readiness/HANDOFF.md

@@ -2,6 +2,8 @@
 
 RR-12 offline failure injection passed 5/5 against the unchanged deploy exit handler. The evidence covers synthetic prior-image recreation, failed rollback diagnostics, original exit status and conditions that skip rollback. [Recovery report](11_BACKUP_RESTORE_ROLLBACK.md#rr-12-offline-failure-injection) distinguishes this from a real post-migration/post-write rehearsal. RR-13 off-host retrieval/decryption and consistent restore remain open. No production qualification.
 
+RR-13 transfer guard [reproduced and fixed offline](11_BACKUP_RESTORE_ROLLBACK.md#rr-13-off-host-upload-failure-handling). Daily and monthly copy errors stop deployment, and the success log makes no encryption claim. Actual remote encryption and restore remain open.
+
 10 November: [endpoint and environment census](03_ENDPOINT_ENVIRONMENT_CENSUS.md) maps 41 source route families, anonymous exceptions and failure cases to qualification owners on the backup-parser branch. No runtime qualification. Carry the map into Stage 2 only after the release candidate and environment are accepted.
 
 Branch `planning/release-readiness-discovery`, baseline `5cb903b989742b7a96cb22ddf4c0e24993b18980`; documentation only. Stage 1 IN_PROGRESS, not complete. Full authoritative release prompt read. No final GO/NO-GO completion report or deployable release claimed.

scripts/test-deploy-backup.py

@@ -1,4 +1,4 @@
-"""Exercise the real backup function with a synthetic dump command; never deploy."""
+"""Exercise deployment backup functions with synthetic commands; never deploy."""
 import os
 from pathlib import Path
 import shlex
@@ -85,5 +85,73 @@ class BackupConnectionTests(unittest.TestCase):
                 self.assertNotIn('dummy-secret', result.stdout + result.stderr)
 
 
+class BackupUploadTests(unittest.TestCase):
+    def upload(self, fail_operation=''):
+        source = (ROOT / 'deploy/deploy.sh').read_text(encoding='utf-8')
+        start = source.index('upload_current_backup_set() {')
+        function = source[start:source.index('\n}\n', start) + 3]
+        gate_start = source.index('if ! apply_backup_retention || ! upload_current_backup_set; then')
+        gate = source[gate_start:source.index('\nfi\n', gate_start) + 4]
+        with tempfile.TemporaryDirectory() as temporary:
+            folder = Path(temporary)
+            for filename in ['jobtracker-db-synthetic.sql.gz', 'jobtracker-files-synthetic.tar.gz',
+                             'jobtracker-db-older.sql.gz']:
+                (folder / filename).touch()
+            (folder / 'monthly' / '2099-01').mkdir(parents=True)
+            script = '\n'.join([
+                'set -euo pipefail',
+                'BACKUP_DIR=' + shlex.quote(folder.as_posix()),
+                'BACKUP_STAMP=synthetic',
+                'BACKUP_RCLONE_REMOTE=fixture:encrypted/',
+                'FAIL_OPERATION=' + shlex.quote(fail_operation),
+                'date() { echo 2099-01; }',
+                'apply_backup_retention() { return 0; }',
+                'rclone() {',
+                '  printf "UPLOAD %s %s %s\\n" "$1" "$3" "$4";',
+                '  if [ "$1" = "$FAIL_OPERATION" ]; then return 23; fi;',
+                '}',
+                function,
+                gate,
+                'echo "DEPLOY WOULD CONTINUE"',
+            ])
+            result = subprocess.run([BASH, '--noprofile', '--norc'], input=script,
+                                    text=True, encoding='utf-8', capture_output=True,
+                                    cwd=ROOT, timeout=10,
+                                    env={key: value for key, value in os.environ.items()
+                                         if key.upper() in {'PATH', 'SYSTEMROOT', 'WINDIR', 'TEMP', 'TMP'}})
+        calls = [line for line in result.stdout.splitlines() if line.startswith('UPLOAD ')]
+        return result, calls
+
+    def test_uploads_only_current_daily_files_then_monthly_snapshot(self):
+        result, calls = self.upload()
+        self.assertEqual(0, result.returncode, result.stdout + result.stderr)
+        self.assertEqual([
+            'UPLOAD copyto fixture:encrypted/daily/jobtracker-db-synthetic.sql.gz --immutable',
+            'UPLOAD copyto fixture:encrypted/daily/jobtracker-files-synthetic.tar.gz --immutable',
+            'UPLOAD copy fixture:encrypted/monthly/2099-01 --immutable',
+        ], calls)
+        self.assertIn('DEPLOY WOULD CONTINUE', result.stdout)
+        self.assertIn('Current recovery set copied to configured off-host remote.', result.stdout)
+
+    def test_daily_copy_failure_stops_before_remaining_files_and_monthly_copy(self):
+        result, calls = self.upload('copyto')
+        self.assertEqual(1, result.returncode, result.stdout + result.stderr)
+        self.assertEqual([
+            'UPLOAD copyto fixture:encrypted/daily/jobtracker-db-synthetic.sql.gz --immutable',
+        ], calls)
+        self.assertIn('Daily off-host backup upload failed', result.stdout)
+        self.assertNotIn('DEPLOY WOULD CONTINUE', result.stdout)
+        self.assertNotIn('Current recovery set copied', result.stdout)
+
+    def test_monthly_copy_failure_blocks_deploy(self):
+        result, calls = self.upload('copy')
+        self.assertEqual(1, result.returncode, result.stdout + result.stderr)
+        self.assertEqual(3, len(calls))
+        self.assertEqual('UPLOAD copy fixture:encrypted/monthly/2099-01 --immutable', calls[-1])
+        self.assertIn('Monthly off-host backup upload failed', result.stdout)
+        self.assertNotIn('DEPLOY WOULD CONTINUE', result.stdout)
+        self.assertNotIn('Current recovery set copied', result.stdout)
+
+
 if __name__ == '__main__':
     unittest.main()