fix(release): require a final SQL backup completion comment

This commit is contained in:
cesnimda
2026-11-18 01:11:02 +01:00
parent 2fb41a96ce
commit 2fb10f7b4e
5 changed files with 148 additions and 3 deletions
+2 -3
View File
@@ -426,10 +426,9 @@ verify_sql_backup() {
local has_schema=1 has_trailer=1
set +o pipefail
if gzip -dc "$target" | grep -q "CREATE TABLE"; then has_schema=0; fi
# mariadb-dump / mysqldump write "-- Dump completed on ..." as the last line.
# Its absence means the dump died partway through.
if gzip -dc "$target" | grep -q "Dump completed"; then has_trailer=0; fi
set -o pipefail
# Require the final nonblank completion comment, not that phrase in SQL data.
if gzip -dc "$target" | awk '{ sub(/\r$/, "") } NF { last=$0 } END { exit !(last ~ /^-- Dump completed( on .*)?$/) }'; then has_trailer=0; fi
if [ "$has_schema" -ne 0 ]; then
echo "Backup ${target} contains no CREATE TABLE statement — it is not a usable dump. Aborting deploy."
@@ -0,0 +1,39 @@
# Offline SQL completion-marker selection
18 November 2026. Branch `codex/release-offline-recovery`, baseline `2fb41a96ce51847ac2ed2d1c9b2e08d8a2ff1129` recovered from `codex/release-rollback-health`. Original checkout/source was left unchanged; its existing untracked graph was preserved. The full authoritative `E:/Storage/Projects/Active/temp docs/final/prompt 11 - release readiness.txt` was read. Sections 13 and 16 permit safe offline investigation and focused remediation; they do not grant live recovery or release permission. Existing release handoff, recovery report, remediation register and acceptance matrix retain their gates.
## Reproduced defect
The real `verify_sql_backup` function described a required final dump trailer, but accepted the substring `Dump completed` anywhere in decompressed SQL. Two valid-gzip, incomplete synthetic dumps passed and reached the next mocked deployment gate:
- A `CREATE TABLE` followed by `INSERT INTO synthetic VALUES ('Dump completed');`, with no completion comment.
- A completion comment followed by another SQL statement, so that comment was not a final trailer.
Before changing the shell source, the four new test methods reported **two assertion failures and two passing controls**. Both negative cases returned exit 0 and printed `FORMAT VERIFIED` and `NEXT DEPLOY GATE WOULD START`. This is an offline false-acceptance defect, not evidence that a real deployed backup is damaged. Engineering priority proposed P2; no release-risk acceptance or security severity is implied.
## Minimal change
The verifier now checks the final nonblank line for a completion comment, using the existing host awk dependency. CRLF is normalised for this check; trailing blank/whitespace-only lines remain allowed. Both the actual timestamped `-- Dump completed on ...` form and the short `-- Dump completed` fixture form remain accepted. Existing pipefail is restored before this full-stream check, so a decompression/pipeline failure cannot be hidden by its last command. The schema predicate, permissions, retention, archive verification, transfer, tombstone handling and rollback code are unchanged.
The tests run only extracted `verify_backup_file_shape`/`verify_sql_backup` functions under the real conditional caller shape. Real local gzip creates/tests/decompresses synthetic files; real Git Bash/awk evaluate the predicate. The new fixture stubs the invalid-artifact removal and final report boundaries. It never sources or executes the complete deployment script. It creates no SQL database and performs no restore, container operation, network transfer, pruning or live deletion. Python removes only its disposable synthetic temporary directories. Temp/output were kept in the isolated E-drive worktree.
## Verification
```powershell
rtk proxy python scripts/test-deploy-backup.py BackupSqlTrailerTests -v
rtk proxy python scripts/test-deploy-backup.py -v
rtk proxy "C:/Program Files/Git/bin/bash.exe" --noprofile --norc -n deploy/deploy.sh
rtk git diff --check
```
- RED: four new methods; two expected failures, two positive/negative controls passed.
- GREEN: **26/26 backup methods passed**, including four new methods. The 22 inherited methods were rerun because the changed verifier also participates in the existing permission/backup-gate fixtures. Do not add the overlapping four-method or historical 22-method runs to this total. Native output: [green-backup.txt](green-backup.txt).
- Bash syntax: exit 0, no stdout/stderr. Source identities and native log SHA-256: [SOURCE_RECEIPTS.json](SOURCE_RECEIPTS.json).
- Whitespace check passed. No new package, dependency installation, app/.NET build, browser or server lane was used.
- Graph-first query used the existing release graph before source inspection. It was copied as an isolated update seed, then `graphify update . --no-cluster` ran AST-only here to avoid clustering CPU during the separate full-Jest lane. No semantic/model extraction or shared graph mutation was requested. Generated graph stays untracked local navigation material; optional extractor warnings are not runtime evidence.
## Remaining gates and rollback
This validates completion-marker selection only. It is not a SQL parser, usable-data proof, consistent recovery-set verification, cryptographic authenticity check, Linux-host qualification or restore rehearsal. A syntactically plausible completion comment alone cannot prove the SQL is executable or complete. Actual dump-tool/version output, encrypted off-host retrieval/decryption, permissions, database/files/keys/tombstone consistency, post-migration/post-write rollback and agreed RPO/RTO still require authorised isolated qualification. RR-09/RR-12/RR-13 and production NO-GO remain open.
The six rollback methods were not rerun because that handler is unchanged. Product, MariaDB, container, provider, hosted CI, off-host, real-model and production checks were not run. Root owns publication/integration and central records. Reverting this bounded source commit restores the previous false-acceptance behaviour; there is no database/schema/data migration to reverse.
@@ -0,0 +1,23 @@
{
"baseline": "2fb41a96ce51847ac2ed2d1c9b2e08d8a2ff1129",
"scope": "offline SQL completion-marker selection only; no recovery qualification",
"python": "3.12.10 (tags/v3.12.10:0cc8128, Apr 8 2025, 12:21:36) [MSC v.1943 64 bit (AMD64)]",
"sourceSha256": {
"deploy\\deploy.sh": "a86c532f6689e386ab68bd49322718abd4e303bd70b450e990d7ce8106cd2132",
"scripts\\test-deploy-backup.py": "a6f9e5e3523c45a898e0a340f64cbcb3d06cc8f1873e249983ad513125f4886b"
},
"bashSyntax": {
"command": [
"C:/Program Files/Git/bin/bash.exe",
"--noprofile",
"--norc",
"-n",
"deploy/deploy.sh"
],
"exitCode": 0,
"stdout": "",
"stderr": ""
},
"testReceiptSha256": "643fe0e73c222f33d98465c070d0173332558bbca244341fee20b64e8ed05934",
"productDatabaseContainerNetworkRestoreTests": "NOT RUN"
}
@@ -0,0 +1,31 @@
test_empty_password_is_preserved (__main__.BackupConnectionTests.test_empty_password_is_preserved) ... ok
test_invalid_or_ambiguous_input_never_invokes_dump_or_exposes_input (__main__.BackupConnectionTests.test_invalid_or_ambiguous_input_never_invokes_dump_or_exposes_input) ... ok
test_quoted_separators_and_shell_characters_remain_data (__main__.BackupConnectionTests.test_quoted_separators_and_shell_characters_remain_data) ... ok
test_single_quoted_unicode_password_and_database_alias_are_preserved (__main__.BackupConnectionTests.test_single_quoted_unicode_password_and_database_alias_are_preserved) ... ok
test_supported_host_alias_and_case_reach_dump_unchanged (__main__.BackupConnectionTests.test_supported_host_alias_and_case_reach_dump_unchanged) ... ok
test_exact_database_member_at_root_or_in_a_directory_is_accepted (__main__.BackupDatabaseMemberTests.test_exact_database_member_at_root_or_in_a_directory_is_accepted) ... ok
test_sidecar_or_similar_name_is_not_the_database_member (__main__.BackupDatabaseMemberTests.test_sidecar_or_similar_name_is_not_the_database_member) ... ok
test_valid_gzip_with_truncated_tar_database_member_is_rejected (__main__.BackupDatabaseMemberTests.test_valid_gzip_with_truncated_tar_database_member_is_rejected) ... ok
test_both_permission_failures_block_verification_and_next_gate (__main__.BackupPermissionTests.test_both_permission_failures_block_verification_and_next_gate) ... ok
test_fallback_permission_success_preserves_verification (__main__.BackupPermissionTests.test_fallback_permission_success_preserves_verification) ... ok
test_host_permission_success_does_not_need_fallback (__main__.BackupPermissionTests.test_host_permission_success_does_not_need_fallback) ... ok
test_completed_monthly_snapshot_contains_both_artifacts_and_stamp (__main__.BackupRetentionTests.test_completed_monthly_snapshot_contains_both_artifacts_and_stamp) ... ok
test_daily_prune_failure_blocks_monthly_pruning_upload_and_deploy (__main__.BackupRetentionTests.test_daily_prune_failure_blocks_monthly_pruning_upload_and_deploy) ... ok
test_empty_monthly_enumeration_does_not_request_removal (__main__.BackupRetentionTests.test_empty_monthly_enumeration_does_not_request_removal) ... ok
test_marker_publish_failure_does_not_expose_complete_marker_or_proceed (__main__.BackupRetentionTests.test_marker_publish_failure_does_not_expose_complete_marker_or_proceed) ... ok
test_monthly_enumeration_failure_blocks_pruning_upload_and_deploy (__main__.BackupRetentionTests.test_monthly_enumeration_failure_blocks_pruning_upload_and_deploy) ... ok
test_monthly_prune_failure_stops_before_remaining_months_upload_and_deploy (__main__.BackupRetentionTests.test_monthly_prune_failure_stops_before_remaining_months_upload_and_deploy) ... ok
test_partial_monthly_copy_cannot_mark_snapshot_complete_or_proceed (__main__.BackupRetentionTests.test_partial_monthly_copy_cannot_mark_snapshot_complete_or_proceed) ... ok
test_preserves_the_existing_completed_monthly_snapshot (__main__.BackupRetentionTests.test_preserves_the_existing_completed_monthly_snapshot) ... ok
test_completion_comment_before_more_sql_is_not_a_final_trailer (__main__.BackupSqlTrailerTests.test_completion_comment_before_more_sql_is_not_a_final_trailer) ... ok
test_completion_phrase_in_sql_value_is_not_a_dump_trailer (__main__.BackupSqlTrailerTests.test_completion_phrase_in_sql_value_is_not_a_dump_trailer) ... ok
test_final_completion_comment_is_accepted_with_supported_line_endings (__main__.BackupSqlTrailerTests.test_final_completion_comment_is_accepted_with_supported_line_endings) ... ok
test_missing_completion_comment_remains_rejected (__main__.BackupSqlTrailerTests.test_missing_completion_comment_remains_rejected) ... ok
test_daily_copy_failure_stops_before_remaining_files_and_monthly_copy (__main__.BackupUploadTests.test_daily_copy_failure_stops_before_remaining_files_and_monthly_copy) ... ok
test_monthly_copy_failure_blocks_deploy (__main__.BackupUploadTests.test_monthly_copy_failure_blocks_deploy) ... ok
test_uploads_only_current_daily_files_then_monthly_snapshot (__main__.BackupUploadTests.test_uploads_only_current_daily_files_then_monthly_snapshot) ... ok
----------------------------------------------------------------------
Ran 26 tests in 10.986s
OK
+53
View File
@@ -360,6 +360,59 @@ class BackupPermissionTests(unittest.TestCase):
self.assertNotIn('NEXT DEPLOY GATE WOULD START', result.stdout)
class BackupSqlTrailerTests(unittest.TestCase):
def verify_dump(self, text):
source = (ROOT / 'deploy/deploy.sh').read_text(encoding='utf-8')
functions = []
for name in ['verify_backup_file_shape', 'verify_sql_backup']:
start = source.index(name + '() {')
functions.append(source[start:source.index('\n}\n', start) + 3])
with tempfile.TemporaryDirectory() as temporary:
folder = Path(temporary)
target = folder / 'synthetic.sql.gz'
with gzip.open(target, 'wb') as bundle:
bundle.write(text.encode('utf-8'))
script = '\n'.join([
'set -euo pipefail',
'report_backup() { echo "FORMAT VERIFIED"; }',
'rm() { echo "INVALID ARTIFACT REJECTION REQUESTED"; }',
*functions,
'if ! verify_sql_backup ' + shlex.quote(target.as_posix()) + '; then exit 1; fi',
'echo "NEXT DEPLOY GATE WOULD START"',
])
return subprocess.run([BASH, '--noprofile', '--norc'], input=script,
text=True, encoding='utf-8', capture_output=True, cwd=ROOT,
timeout=10, env={key: value for key, value in os.environ.items()
if key.upper() in {'PATH', 'SYSTEMROOT', 'WINDIR', 'TEMP', 'TMP'}})
def test_completion_phrase_in_sql_value_is_not_a_dump_trailer(self):
result = self.verify_dump("CREATE TABLE synthetic (value text);\nINSERT INTO synthetic VALUES ('Dump completed');\n")
self.assertNotEqual(0, result.returncode, result.stdout + result.stderr)
self.assertNotIn('FORMAT VERIFIED', result.stdout)
self.assertNotIn('NEXT DEPLOY GATE WOULD START', result.stdout)
def test_completion_comment_before_more_sql_is_not_a_final_trailer(self):
result = self.verify_dump("CREATE TABLE synthetic (value text);\n-- Dump completed on synthetic-date\nINSERT INTO synthetic VALUES ('more work');\n")
self.assertNotEqual(0, result.returncode, result.stdout + result.stderr)
self.assertNotIn('FORMAT VERIFIED', result.stdout)
self.assertNotIn('NEXT DEPLOY GATE WOULD START', result.stdout)
def test_final_completion_comment_is_accepted_with_supported_line_endings(self):
for trailer, ending in [('-- Dump completed on synthetic-date', '\n'),
('-- Dump completed', '\n'),
('-- Dump completed on synthetic-date', '\r\n')]:
with self.subTest(trailer=trailer, ending=repr(ending)):
result = self.verify_dump(ending.join(['CREATE TABLE synthetic (id int);', trailer, '', ' ', '']))
self.assertEqual(0, result.returncode, result.stdout + result.stderr)
self.assertIn('FORMAT VERIFIED', result.stdout)
self.assertIn('NEXT DEPLOY GATE WOULD START', result.stdout)
def test_missing_completion_comment_remains_rejected(self):
result = self.verify_dump('CREATE TABLE synthetic (id int);\n')
self.assertNotEqual(0, result.returncode, result.stdout + result.stderr)
self.assertNotIn('NEXT DEPLOY GATE WOULD START', result.stdout)
class BackupDatabaseMemberTests(unittest.TestCase):
def verify_archive(self, member_name, truncated_tar=False):
source = (ROOT / 'deploy/deploy.sh').read_text(encoding='utf-8')