feat(server): add integration test for message deduplication#3099
Open
seokjin0414 wants to merge 4 commits intoapache:masterfrom
Open
feat(server): add integration test for message deduplication#3099seokjin0414 wants to merge 4 commits intoapache:masterfrom
seokjin0414 wants to merge 4 commits intoapache:masterfrom
Conversation
… calculation - Add empty batch guard after prepare_for_persistence() to prevent server panic when all messages in a batch are duplicates - Fix partition offset calculation to use actual last offset from batch instead of arithmetic that ignores gaps created by dedup removal - Create message deduplicator for lazily-initialized partitions in init_partition_inner() instead of hardcoding None Signed-off-by: shin <sars21@hanmail.net>
Add 7-step scenario testing the full deduplication pipeline: - Auto-generated IDs (id=0) all pass through with unique UUIDs - Explicit IDs are accepted on first send - Duplicate IDs are rejected, original payload preserved - All-duplicate batch does not crash server (regression for empty batch) - Mixed batch with partial duplicates only accepts new IDs - Offsets are monotonically increasing after dedup removal - TTL expiry allows previously seen IDs to be accepted again Signed-off-by: shin <sars21@hanmail.net>
Signed-off-by: shin <sars21@hanmail.net>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #3099 +/- ##
============================================
+ Coverage 72.71% 72.74% +0.02%
Complexity 943 943
============================================
Files 1117 1117
Lines 96285 96288 +3
Branches 73485 73506 +21
============================================
+ Hits 70014 70045 +31
+ Misses 23725 23669 -56
- Partials 2546 2574 +28
🚀 New features to boost your workflow:
|
1750fe5 to
6fdbeb7
Compare
…urnal offset tracking - When all messages in a batch are duplicates, advance partition offset past the assigned (but removed) offset range to prevent offset reuse in subsequent batches - Fix journal current_offset to use actual last offset from batch instead of arithmetic that ignores gaps created by dedup removal Signed-off-by: shin <sars21@hanmail.net>
6fdbeb7 to
46db1ea
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #2872
Bug fixes
Empty batch panic (
messages.rs): Afterprepare_for_persistence()removes all duplicate messages, subsequent.unwrap()calls onfirst_timestamp(),last_timestamp(),last_offset()panic. Added empty batch guard.Offset calculation (
messages.rs):last_offsetwas computed ascurrent_offset + count - 1, which doesn't account for offset gaps created by dedup removal. Changed to usesegment.end_offset(the actual last offset from batch).Deduplicator not created (
partitions.rs):init_partition_inner()hardcodedNonefor themessage_deduplicatorparameter. Addedcreate_message_deduplicator()call matching the bootstrap path.Integration test scenario
Test plan
cargo fmt --all -- --checkcargo clippy -p server -p integration --all-targets -- -D warningscargo test -p integration --test mod -- message_deduplication(CI)