Skip to content

Create workspace with core and util crates#1414

Open
timsaucer wants to merge 13 commits intoapache:mainfrom
timsaucer:feat/add-util-crate
Open

Create workspace with core and util crates#1414
timsaucer wants to merge 13 commits intoapache:mainfrom
timsaucer:feat/add-util-crate

Conversation

@timsaucer
Copy link
Member

@timsaucer timsaucer commented Mar 8, 2026

Which issue does this PR close?

Closes #1395.

Rationale for this change

For our downstream users who are implementing FFI based extensions, there are a handful of useful utility functions
that we are currently requiring them to copy. This is a poor way to keep code up to date. This PR creates a utility crate that our downstream users can utilize. This also helps to ensure that as our interfaces get updated that those changes propagate properly to our downstream users.

Also this change ensures our example is on the same code version as the main repository.

What changes are included in this PR?

  • Adds a cargo workspace
  • Divides the code into three projects, core, util and the FFI example.
  • Updates dependencies to all use the common workspace.

Are there any user-facing changes?

Yes, there is a slight change in that the utils module of datafusion-python has been removed. Instead users can now change their dependency to the new datafusion-python-util crate.

@timsaucer timsaucer force-pushed the feat/add-util-crate branch from e2a4ec6 to bdf3402 Compare March 8, 2026 14:52
@timsaucer
Copy link
Member Author

FYI @kevinjqliu

@timsaucer timsaucer changed the title Feat/add util crate Create workspace with core and util crates Mar 8, 2026
@timsaucer timsaucer force-pushed the feat/add-util-crate branch from 976ae54 to 0be1501 Compare March 10, 2026 07:25
Copy link
Member

@davisp davisp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

Everything looks straightforward to me. I only had the one minor question on the removed cargo config.

@@ -1,5 +0,0 @@
[target.x86_64-apple-darwin]
rustflags = ["-C", "link-arg=-undefined", "-C", "link-arg=dynamic_lookup"]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Am I missing where these got moved to, or do we just not need them any more?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed because these were serving the same functionality as the pyo3_build_config::add_extension_module_link_args() in build.rs

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

speaking of build.rs, i was testing out this integration in iceberg-rust and claude mentioned that i need to create my own build.rs file for compatibility

is this expected?

Copy link
Contributor

@kevinjqliu kevinjqliu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, i built iceberg-rust using this branch (https://github.com/kevinjqliu/iceberg-rust/pull/8/files#diff-1a7b28a196f60421838fd52ca356647a835081dc3095f1b55d526c27580cb677R23)

one nit: do you prefer the crate to be named -util or -utils?

timsaucer and others added 4 commits March 11, 2026 07:39
Co-authored-by: Kevin Liu <kevinjqliu@users.noreply.github.com>
Co-authored-by: Kevin Liu <kevinjqliu@users.noreply.github.com>
Co-authored-by: Kevin Liu <kevinjqliu@users.noreply.github.com>
Co-authored-by: Kevin Liu <kevinjqliu@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add utility crate

3 participants