SPOC-311: Fix data loss when adding a node under write load.

[ibrarahmad]

Backport the ZODAN add_node protocol fixes from main to v5.

The root cause of the data loss was that disabled subscriptions (e.g. sub_n2_n3) created by add_node started replicating from LSN 0/0 because the replication origin was never advanced. When the apply worker received WAL from the slot's consistent point, any UPDATE on a row not yet present on the new node caused repeated "row not found" errors, which led to subscription disable and a 20-minute timeout.

Fix this by advancing both the replication slot and the replication origin to resume_lsn (the last commit LSN from spock.progress) in Phase 7 of the add_node protocol. Also add a source-node commit catchup wait in Phase 3 to close the [resume_lsn, L_slot) gap.

Additional hardening of the ZODAN protocol:

• Check wait_for_sync_event() return value instead of discarding it with PERFORM; raise on timeout.
• Pass wait_if_disabled=true to wait_for_sync_event() so the procedure tolerates not-yet-enabled subscriptions.
• Reduce all wait timeouts from 1200s to 180s and use clock_timestamp()-based bounds instead of iteration counters.
• Detect terminal subscription states (disabled/down) and fail fast with RAISE EXCEPTION instead of silently continuing.
• Replace RAISE NOTICE + CONTINUE with RAISE EXCEPTION throughout, so errors propagate instead of being swallowed.
• Add wait_for_replication_catchup_with_dblink() for bounded lag-tracker polling.
• Add bounded SQL loop replacing C-level sub_wait_for_sync() in Phase 7 to avoid rare hangs.
• Drop stale replication origins before creating disabled subscriptions to prevent stale-LSN data loss.
• Advance replication origin on new node alongside slot advancement.

Extend 011_zodan_sync_third TAP test with a remove/re-add cycle to exercise the full add_node protocol under pgbench write load, and enable 012_zodan_basics in the TAP schedule. Increase the spockbench CI TAP test timeout to 45 minutes.

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 9b2aaf0e-2d66-4609-a8c4-760f1fdf290b

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch BRANCH-5-SPOC-311-V2

📝 Coding Plan

• Generate coding plan for human review comments

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}