Skip to content

Base implementation of Parquet file writing#2583

Draft
VeckoTheGecko wants to merge 19 commits intoParcels-code:mainfrom
VeckoTheGecko:push-zlxyoyvlpoqm
Draft

Base implementation of Parquet file writing#2583
VeckoTheGecko wants to merge 19 commits intoParcels-code:mainfrom
VeckoTheGecko:push-zlxyoyvlpoqm

Conversation

@VeckoTheGecko
Copy link
Copy Markdown
Contributor

@VeckoTheGecko VeckoTheGecko commented Apr 20, 2026

Description

This PR introduces Parquet file writing to Parcels.

I still need to work on:

  • How to work with cftime output in the Parquet (how does this work with our internal model of time in Parcels? How should it work?)
  • Reviewing the test_particlefile.py file - are there tests that are no longer needed? What would be the best testing approach here?
  • Update documentation

Posting as draft for initial feedback

Checklist

AI Disclosure

  • This PR contains AI-generated content.
    • I have tested any AI-generated content in my PR.
    • I take responsibility for any AI-generated content in my PR.
    • Describe how you used it (e.g., by pasting your prompt): Just to help with learning the PyArrow API (i.e., used it to create an example script - which I then used as an entry to exploring the docs on pyarrow)

@VeckoTheGecko VeckoTheGecko changed the title Add parquet file writing Base implementation of Parquet file writing Apr 20, 2026
@VeckoTheGecko
Copy link
Copy Markdown
Contributor Author

@erikvansebille let's table some of these questions for our meeting tomorrow (mainly around datetime serialization in the Parquet file)

Remove temporary test_cftime.py file
This function is now independent of the time_interval as time is now stored as float
Remove nested key - save on root instead
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Backlog

Development

Successfully merging this pull request may close these issues.

1 participant