feat: implement IsNotNull expression in vortex expression library#6969
Merged
robert3005 merged 2 commits intovortex-data:developfrom Apr 13, 2026
Merged
feat: implement IsNotNull expression in vortex expression library#6969robert3005 merged 2 commits intovortex-data:developfrom
robert3005 merged 2 commits intovortex-data:developfrom
Conversation
Merging this PR will degrade performance by 11.42%
Performance Changes
Comparing 
Footnotes
|
Contributor
|
I think if you rebase should fix the wasm-integration check |
Add a first-class IsNotNull scalar function instead of composing Not(IsNull(...)). This simplifies the expression tree, enables direct stat_falsification for zone map pruning, and updates all integration points (DataFusion, DuckDB, Python/Substrait). The stat_falsification uses is_constant && null_count > 0 as an approximation since there is no RowCount stat yet. Closes: vortex-data#6040 Signed-off-by: Xiaoxuan Li <xioxuan@amazon.com>
Contributor
|
@xiaoxuandev I have rebased your pr and added java bindings to it. I will do one more pass on the pr so we can hopefully merge it |
robert3005
approved these changes
Apr 13, 2026
Contributor
Author
|
@robert3005 Thanks for merging the PR! @joseph-isaacs @a10y Thanks for the review! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes: #6040
Add a first-class IsNotNull scalar function, replacing the previous Not(IsNull(...)) composition pattern. This simplifies the expression tree and enables direct stat_falsification for zone map pruning.
Changes:
New is_not_null.rs with ScalarFnVTable implementation, including stat_falsification using is_constant && null_count > 0 (with TODO for future RowCount stat)
Updated all integration points: DataFusion, DuckDB, Python/Substrait to use is_not_null(...) directly
Replaced the Not(IsNull(...)) fallback in erased.rs validity with IsNotNull
Registered IsNotNull in ScalarFnSession and ExprBuiltins/ArrayBuiltins
AI Assistance Disclosure
This PR was developed with AI assistance (Kiro). AI was used for code review, implementing stat_falsification, writing tests, and drafting the PR description. All output was reviewed and validated by the author.
API Changes
New public APIs:
vortex_array::expr::is_not_null(child) — creates an IsNotNull expression
Expression::is_not_null() / ArrayRef::is_not_null() via ExprBuiltins/ArrayBuiltins traits
Python: vortex._lib.expr.is_not_null(child)
Testing
9 unit tests covering: return dtype, child replacement, mixed/all-valid/all-invalid evaluation, struct field access, display formatting, null sensitivity, and stat falsification pruning expression generation.