perf: Optimize split_part, support Utf8View#21119
Merged
alamb merged 7 commits intoapache:mainfrom Mar 27, 2026
Merged
Conversation
github-actions
Bot
added
sqllogictest
Contributor
Author
|
|
martin-g
reviewed
Mar 25, 2026
martin-g
approved these changes
Mar 25, 2026
alamb
approved these changes
Mar 27, 2026
Contributor
alamb
left a comment
alamb
left a comment
There was a problem hiding this comment.
Thanks @neilconway and @martin-g
| /// Finds the nth split part of `string` by `delimiter`. | ||
| #[inline] | ||
| fn split_nth<'a>(string: &'a str, delimiter: &str, n: usize) -> Option<&'a str> { | ||
| if delimiter.len() == 1 { |
Contributor
There was a problem hiding this comment.
As a follow on, we can probably hoist this check out of the loop (so call it once per batch rather than once per string) and see if that makes things any faster (sometimes it allows the compiler to optimize things better)
| match args[1].data_type() { | ||
| DataType::Utf8View => split_part_impl( | ||
| $str_arr, | ||
| &args[1].as_string_view(), |
Contributor
There was a problem hiding this comment.
As a follow on, we can probably make this even faster with a special, no copy implementation for Utf8View (reuse the same string buffers, but just adjust the views)
Rich-T-kid
pushed a commit
to Rich-T-kid/datafusion
that referenced
this pull request
Apr 21, 2026
## Which issue does this PR close? - Closes apache#21117. - Closes apache#21118 . ## Rationale for this change `split_part` currently accepts `Utf8View` but always returns `Utf8`. When given `Utf8View` input, it should instead return `Utf8View` output. While we're at it, optimize `split_part` for single-character delimiters (the common case): `str::split(&str)` is significantly slower than `str::split(char)` for single-character ASCII delimiters, because the former uses a general string matching algorithm but the latter uses `memchr::memchr`. Benchmark results (M4 Max): - `utf8_single_char/pos_first`: 142 µs → 104 µs (-26%) - `utf8_single_char/pos_middle`: 389 µs → 365 µs (-6%) - `utf8_single_char/pos_negative`: 154 µs → 109 µs (-29%) - `utf8_multi_char/pos_middle`: 356 µs → 361 µs (~0%, noise) - `utf8view_single_char/pos_first`: 143 µs → 111 µs (-22%) - `utf8_long_strings/pos_middle`: 2568 µs → 1984 µs (-23%) - `utf8view_long_parts/pos_middle`: 998 µs → 470 µs (-53%) ## What changes are included in this PR? * Revise `split_part` benchmarks to reduce redundancy and improve `Utf8View` coverage * Support `Utf8View` -> `Utf8View` in `split_part` * Refactor `split_part` to cleanup some redundant code * Optimize `split_part` for single-character delimiters * Add SLT test coverage for `split_part` with `Utf8View` input ## Are these changes tested? Yes. New tests and benchmarks added. ## Are there any user-facing changes? No.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
split_partshould preserve Utf8View input #21117.split_partfor single-character delimiters #21118 .Rationale for this change
split_partcurrently acceptsUtf8Viewbut always returnsUtf8. When givenUtf8Viewinput, it should instead returnUtf8Viewoutput.While we're at it, optimize
split_partfor single-character delimiters (the common case):str::split(&str)is significantly slower thanstr::split(char)for single-character ASCII delimiters, because the former uses a general string matching algorithm but the latter usesmemchr::memchr.Benchmark results (M4 Max):
utf8_single_char/pos_first: 142 µs → 104 µs (-26%)utf8_single_char/pos_middle: 389 µs → 365 µs (-6%)utf8_single_char/pos_negative: 154 µs → 109 µs (-29%)utf8_multi_char/pos_middle: 356 µs → 361 µs (~0%, noise)utf8view_single_char/pos_first: 143 µs → 111 µs (-22%)utf8_long_strings/pos_middle: 2568 µs → 1984 µs (-23%)utf8view_long_parts/pos_middle: 998 µs → 470 µs (-53%)What changes are included in this PR?
split_partbenchmarks to reduce redundancy and improveUtf8ViewcoverageUtf8View->Utf8Viewinsplit_partsplit_partto cleanup some redundant codesplit_partfor single-character delimiterssplit_partwithUtf8ViewinputAre these changes tested?
Yes. New tests and benchmarks added.
Are there any user-facing changes?
No.