fix(topk): avoid overflow panic in interleave emission for Utf8#20494
Open
aviralgarg05 wants to merge 7 commits intoapache:mainfrom
Open
fix(topk): avoid overflow panic in interleave emission for Utf8#20494aviralgarg05 wants to merge 7 commits intoapache:mainfrom
Utf8#20494aviralgarg05 wants to merge 7 commits intoapache:mainfrom
Conversation
kosiew
requested changes
Mar 17, 2026
Contributor
kosiew
left a comment
kosiew
left a comment
There was a problem hiding this comment.
@aviralgarg05
Thanks for working on this
Member
|
FYI, I encountered a similar case in sort merge: #20922, and there is a PR changing panic to error for interleavehttps://github.com/apache/arrow-rs/pull/9549 |
xudong963
reviewed
Mar 18, 2026
Contributor
|
run benchmarks |
|
🤖 Benchmark completed (GKE) | trigger Details
Resource Usagetpch — base (merge-base)
tpch — branch
|
|
🤖 Benchmark completed (GKE) | trigger Details
Resource Usagetpcds — base (merge-base)
tpcds — branch
|
|
🤖 Benchmark completed (GKE) | trigger Details
Resource Usageclickbench_partitioned — base (merge-base)
clickbench_partitioned — branch
|
alamb
requested changes
Mar 23, 2026
Contributor
alamb
left a comment
alamb
left a comment
There was a problem hiding this comment.
Thank you for this contirbution @aviralgarg05 and @xudong963 -- I am a little worried about the complexity added in this patch for two reasons:
- It doesn't appear to be adequately tested
- It may introduce a performance regression when generating output
Typically when people hit offset errors related to using Utf8 (because of offset overflows) our advice is either:
- Use Utf8View ("String View")
- Use LargeUtf8
I am not sure it is a good tradeoff to slow down / make the Utf8 path more complex
alamb
changed the title
Utf8
Member
Yes, the changes are in the hot path, for all cases, it'll do some checks. I encountered a similar overflow in the SPM path #20922, and introduced a different way to fix it: if we get the panic info, capture it, then progressively retry until no overflow, which only affects the overflow path. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Rationale for this change
While testing, I found that
ORDER BY ... LIMITcould panic in TopK when interleaving very largeUtf8values. The problem was that TopK tried to interleave all selected rows in a single call, which can overflow Arrow’s i32 string offsets in extreme cases. This PR removes that panic path so the operator behaves safely under large-string workloads.What changes are included in this PR?
TopK.batch_sizefield.test_topk_heap_emit_with_state_respects_batch_size.Are these changes tested?
Yes.
datafusion/physical-plan/src/topk/mod.rsfor chunked TopK emission behavior.cargo fmt --all --check.cargo clippy -p datafusion-physical-plan --lib --tests -- -D warnings.cargo test -p datafusion-physical-plan topk::tests:: -- --nocapturethree times after final changes.Are there any user-facing changes?
Yes. For extreme large-string cases in
ORDER BY ... LIMIT, TopK no longer panics with overflow; it now follows normal error/execution handling.