Introduce morsel-driven Parquet scan

[Dandandan]

Which issue does this PR close?

• Closes Introduce morsel-driven Parquet scan #20529

TODO:

• Make sure workqueue is closed after execute stream is dropped / not shared between two concurrent instances (major design issue)
• Test changing planning again to only plan whole files instead of "early splitting" (got some slowdown on this - perhaps it removes some parallelism)
• Update test to work with morsel setting enabled
• Fix tests
• Compare different approaches
• Update docs
• Simplify / address feedback

Rationale for this change

Current parelllization of Parquet scan is bounded by the thread that has the most data / is the slowest to execute, which means in the case of data skew (driven by either larger partitions or less selective filters during pruning / filter pushdown..., variable object store latency), the parallelism will be significantly limited.

We can change the strategy by morsel-driven parallelism like described in https://db.in.tum.de/~leis/papers/morsels.pdf.

Doing so is faster for a lot of queries, when there is an amount of skew (such as clickbench) and we have enough row filters to spread out the work.
For clickbench_partitioned / clickbench_pushdown it seems up to ~2x as fast for some queries, on a 10 core machine.

It seems to have almost no regressions, perhaps some due to different file scanning order(?) - so different statistics that can be used to prune and thus some minor variation.

Morsel-Driven Execution Architecture (partly claude-generated)

This branch implements a morsel-driven execution model for Parquet scans, based on the concept
from the Morsel-Driven Parallelism paper (Leis et al.). The core idea: instead of statically
assigning files to partitions, all work is pooled in a shared queue that all partition streams pull
from dynamically.

The Problem It Solves

In the traditional model, partition 0 might get a 1 GB file while partition 1 gets nothing --
partition 1 idles while 0 is busy. Currently we already try to statically spread out work to n partitions / threads based on stats (which works very well on perfectly distributed scans on SSDs (e.g. TPCH running locally), this doesn't work well when there is any data skew caused by any of those:

• filters being more selective on part of the data
• high variation in object store response times
• overhead of filter pushdown + more serial/smaller reads and file operations

Morsel-driven execution prevents this by sharing work dynamically.

Key Types

ParquetMorsel -- datafusion/datasource-parquet/src/opener.rs:129

A morsel = one row group of a Parquet file. Stored as an extension on PartitionedFile.

  pub struct ParquetMorsel {
      pub metadata: Arc<ParquetMetaData>,   // cached, shared across morsels from same file
      pub access_plan: ParquetAccessPlan,   // which row groups to read
  }

WorkQueue -- datafusion/datasource/src/file_stream.rs:410

The shared, thread-safe queue. Each partition stream calls pull() which returns:

• Work(file) -- here's a file/morsel to process
• Wait -- queue is empty but workers are still morselizing (wait for notification)
• Done -- all work consumed

MorselState -- datafusion/datasource/src/source.rs:240

Tracks the shared queue lifecycle. A new queue is created once per execution cycle when all
partition streams have opened.

struct MorselState {
    queue: Option<Arc<WorkQueue>>,
    streams_opened: usize,
    expected_streams: usize,
}

MorselizingGuard -- datafusion/datasource/src/file_stream.rs:49

RAII wrapper that atomically decrements morselizing_count when a worker finishes -- enabling
WorkStatus::Wait vs Done decisions.

FileOpener Trait Extension -- datafusion/datasource/src/file_stream.rs:498

A new morselize() method is added to FileOpener. The default implementation is a no-op
(returns the file as-is). ParquetOpener overrides it to split files by row group.

pub trait FileOpener {
    fn open(&self, file: PartitionedFile) -> Result<FileOpenFuture>;

    // NEW: split a file into morsels (default = no-op)
    fn morselize(&self, file: PartitionedFile)
        -> BoxFuture<'static, Result<Vec<PartitionedFile>>> { ... }
}

ParquetOpener::morselize() at opener.rs:232:

1. Loads Parquet metadata once (shared via Arc across all resulting morsels)
2. Applies file-level and row-group-level statistics pruning
3. Returns one PartitionedFile per surviving row group, each carrying a ParquetMorsel extension

FileStream State Machine -- datafusion/datasource/src/file_stream.rs:141

The morsel-driven path adds two new states (Morselizing and Waiting):

  Idle
   +-- morsel_driven=true  -> queue.pull()
   |    +-- Work(file) -> Morselizing (run morselize())
   |    +-- Wait       -> Waiting (yield, wake on notify)
   |    +-- Done       -> return None
   +-- morsel_driven=false -> traditional local file_iter

  Morselizing
   +-- morsels > 1 -> push_many() back to queue -> Idle
   +-- morsel = 1  -> Open

  Waiting
   +-- notified -> Idle

Configuration

datafusion.execution.parquet.allow_morsel_driven -- datafusion/common/src/config.rs:748

Default: true. Can be disabled per-session.

FileScanConfig::morsel_driven -- datafusion/datasource/src/file_scan_config.rs:211

Automatically disabled when:

• partitioned_by_file_group = true (breaks hash-partitioning guarantees)
• preserve_order = true (breaks SortPreservingMerge guarantees)

Benchmark results

Summary: both clickbench, clickbench_partitioned

Details

🤖: Benchmark completed

Details

Comparing HEAD and parquet-morsel-driven-execution-237164415184908839
--------------------
Benchmark clickbench_extended.json
--------------------
┏━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓
┃ Query    ┃        HEAD ┃ parquet-morsel-driven-execution-237164415184908839 ┃          Change ┃
┡━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩
│ QQuery 0 │  2445.21 ms │                                         2334.07 ms │       no change │
│ QQuery 1 │   984.42 ms │                                          831.92 ms │   +1.18x faster │
│ QQuery 2 │  1847.13 ms │                                         1609.38 ms │   +1.15x faster │
│ QQuery 3 │  1078.65 ms │                                          973.60 ms │   +1.11x faster │
│ QQuery 4 │  2279.11 ms │                                         2070.83 ms │   +1.10x faster │
│ QQuery 5 │ 26832.66 ms │                                        26954.54 ms │       no change │
│ QQuery 6 │  3852.58 ms │                                            8.28 ms │ +465.50x faster │
│ QQuery 7 │  2619.05 ms │                                         2547.13 ms │       no change │
└──────────┴─────────────┴────────────────────────────────────────────────────┴─────────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary                                                 ┃            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD)                                                 │ 41938.81ms │
│ Total Time (parquet-morsel-driven-execution-237164415184908839)   │ 37329.75ms │
│ Average Time (HEAD)                                               │  5242.35ms │
│ Average Time (parquet-morsel-driven-execution-237164415184908839) │  4666.22ms │
│ Queries Faster                                                    │          5 │
│ Queries Slower                                                    │          0 │
│ Queries with No Change                                            │          3 │
│ Queries with Failure                                              │          0 │
└───────────────────────────────────────────────────────────────────┴────────────┘
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query     ┃        HEAD ┃ parquet-morsel-driven-execution-237164415184908839 ┃        Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0  │     2.59 ms │                                            2.59 ms │     no change │
│ QQuery 1  │    52.70 ms │                                           53.83 ms │     no change │
│ QQuery 2  │   170.22 ms │                                          157.23 ms │ +1.08x faster │
│ QQuery 3  │   165.42 ms │                                          167.38 ms │     no change │
│ QQuery 4  │  1090.62 ms │                                         1035.57 ms │ +1.05x faster │
│ QQuery 5  │  1240.29 ms │                                         1229.64 ms │     no change │
│ QQuery 6  │     6.51 ms │                                            7.08 ms │  1.09x slower │
│ QQuery 7  │    49.50 ms │                                           60.72 ms │  1.23x slower │
│ QQuery 8  │  1388.32 ms │                                         1441.47 ms │     no change │
│ QQuery 9  │  1743.04 ms │                                         1723.40 ms │     no change │
│ QQuery 10 │   337.39 ms │                                          335.23 ms │     no change │
│ QQuery 11 │   386.04 ms │                                          380.30 ms │     no change │
│ QQuery 12 │  1183.09 ms │                                         1140.56 ms │     no change │
│ QQuery 13 │  1871.04 ms │                                         1609.71 ms │ +1.16x faster │
│ QQuery 14 │  1231.91 ms │                                         1158.10 ms │ +1.06x faster │
│ QQuery 15 │  1172.84 ms │                                         1210.94 ms │     no change │
│ QQuery 16 │  2418.93 ms │                                         2444.96 ms │     no change │
│ QQuery 17 │  2394.23 ms │                                         2427.85 ms │     no change │
│ QQuery 18 │  4803.83 ms │                                         4575.30 ms │     no change │
│ QQuery 19 │   123.68 ms │                                          124.92 ms │     no change │
│ QQuery 20 │  1849.23 ms │                                         1689.57 ms │ +1.09x faster │
│ QQuery 21 │  2137.19 ms │                                         1979.95 ms │ +1.08x faster │
│ QQuery 22 │  3705.39 ms │                                         3533.51 ms │     no change │
│ QQuery 23 │ 16569.27 ms │                                        12289.08 ms │ +1.35x faster │
│ QQuery 24 │   212.07 ms │                                          137.98 ms │ +1.54x faster │
│ QQuery 25 │   476.64 ms │                                          406.70 ms │ +1.17x faster │
│ QQuery 26 │   205.82 ms │                                          141.86 ms │ +1.45x faster │
│ QQuery 27 │  2643.51 ms │                                         2462.51 ms │ +1.07x faster │
│ QQuery 28 │ 23612.99 ms │                                        22934.78 ms │     no change │
│ QQuery 29 │  1037.09 ms │                                         1008.12 ms │     no change │
│ QQuery 30 │  1237.34 ms │                                         1236.45 ms │     no change │
│ QQuery 31 │  1316.76 ms │                                         1286.78 ms │     no change │
│ QQuery 32 │  4511.21 ms │                                         4096.49 ms │ +1.10x faster │
│ QQuery 33 │  5462.02 ms │                                         5437.33 ms │     no change │
│ QQuery 34 │  5792.16 ms │                                         5907.89 ms │     no change │
│ QQuery 35 │  1885.60 ms │                                         1896.64 ms │     no change │
│ QQuery 36 │   197.18 ms │                                          127.31 ms │ +1.55x faster │
│ QQuery 37 │    72.42 ms │                                           55.88 ms │ +1.30x faster │
│ QQuery 38 │   112.79 ms │                                           68.44 ms │ +1.65x faster │
│ QQuery 39 │   336.37 ms │                                          224.29 ms │ +1.50x faster │
│ QQuery 40 │    41.22 ms │                                           28.57 ms │ +1.44x faster │
│ QQuery 41 │    34.57 ms │                                           27.29 ms │ +1.27x faster │
│ QQuery 42 │    30.75 ms │                                           21.99 ms │ +1.40x faster │
└───────────┴─────────────┴────────────────────────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary                                                 ┃            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD)                                                 │ 95311.78ms │
│ Total Time (parquet-morsel-driven-execution-237164415184908839)   │ 88286.18ms │
│ Average Time (HEAD)                                               │  2216.55ms │
│ Average Time (parquet-morsel-driven-execution-237164415184908839) │  2053.17ms │
│ Queries Faster                                                    │         19 │
│ Queries Slower                                                    │          2 │
│ Queries with No Change                                            │         22 │
│ Queries with Failure                                              │          0 │
└───────────────────────────────────────────────────────────────────┴────────────┘
--------------------
Benchmark tpch_mem_sf1.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query     ┃      HEAD ┃ parquet-morsel-driven-execution-237164415184908839 ┃        Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 1  │ 125.24 ms │                                          129.21 ms │     no change │
│ QQuery 2  │  33.04 ms │                                           32.48 ms │     no change │
│ QQuery 3  │  39.89 ms │                                           41.09 ms │     no change │
│ QQuery 4  │  34.35 ms │                                           35.63 ms │     no change │
│ QQuery 5  │  91.73 ms │                                           91.67 ms │     no change │
│ QQuery 6  │  23.21 ms │                                           20.64 ms │ +1.12x faster │
│ QQuery 7  │ 157.14 ms │                                          162.48 ms │     no change │
│ QQuery 8  │  40.93 ms │                                           42.22 ms │     no change │
│ QQuery 9  │ 111.58 ms │                                          115.04 ms │     no change │
│ QQuery 10 │  72.12 ms │                                           74.28 ms │     no change │
│ QQuery 11 │  18.96 ms │                                           19.11 ms │     no change │
│ QQuery 12 │  56.87 ms │                                           54.87 ms │     no change │
│ QQuery 13 │  55.66 ms │                                           54.46 ms │     no change │
│ QQuery 14 │  15.64 ms │                                           15.74 ms │     no change │
│ QQuery 15 │  34.30 ms │                                           34.93 ms │     no change │
│ QQuery 16 │  30.91 ms │                                           31.02 ms │     no change │
│ QQuery 17 │ 169.58 ms │                                          162.26 ms │     no change │
│ QQuery 18 │ 305.11 ms │                                          297.61 ms │     no change │
│ QQuery 19 │  48.79 ms │                                           53.73 ms │  1.10x slower │
│ QQuery 20 │  62.46 ms │                                           63.48 ms │     no change │
│ QQuery 21 │ 200.87 ms │                                          203.27 ms │     no change │
│ QQuery 22 │  24.72 ms │                                           23.78 ms │     no change │
└───────────┴───────────┴────────────────────────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Benchmark Summary                                                 ┃           ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ Total Time (HEAD)                                                 │ 1753.10ms │
│ Total Time (parquet-morsel-driven-execution-237164415184908839)   │ 1758.99ms │
│ Average Time (HEAD)                                               │   79.69ms │
│ Average Time (parquet-morsel-driven-execution-237164415184908839) │   79.95ms │
│ Queries Faster                                                    │         1 │
│ Queries Slower                                                    │         1 │
│ Queries with No Change                                            │        20 │
│ Queries with Failure                                              │         0 │
└───────────────────────────────────────────────────────────────────┴───────────┘

Acknowledgements

I heavily used AI (Jules / Claude) for this PR, but reviewed the code myself

[Dandandan]

run benchmarks

[adriangbot]

Benchmark job started for this request (job bench-c4053431901-196). Results will be posted here when complete.

[adriangbot]

🤖 Benchmark running (GKE) | trigger
Linux bench-c4053431901-196-bq7lc 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux
Comparing parquet-morsel-driven-execution-237164415184908839 (b02a9c0) to ed793f0 (merge-base) diff using: clickbench_extended
Results will be posted here when complete

[adriangbot]

🤖 Benchmark completed (GKE) | trigger

Details

Comparing HEAD and parquet-morsel-driven-execution-237164415184908839
--------------------
Benchmark clickbench_extended.json
--------------------
┏━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query    ┃                                      HEAD ┃ parquet-morsel-driven-execution-237164415184908839 ┃        Change ┃
┡━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0 │         819.27 / 824.23 ±5.26 / 833.83 ms │                 952.14 / 968.25 ±13.21 / 988.25 ms │  1.17x slower │
│ QQuery 1 │         225.08 / 225.74 ±0.73 / 226.68 ms │                  179.77 / 180.84 ±0.95 / 182.16 ms │ +1.25x faster │
│ QQuery 2 │         528.63 / 532.69 ±4.03 / 537.71 ms │                  416.88 / 420.78 ±2.31 / 423.43 ms │ +1.27x faster │
│ QQuery 3 │         333.08 / 334.27 ±0.94 / 335.46 ms │                  255.37 / 256.11 ±0.41 / 256.62 ms │ +1.31x faster │
│ QQuery 4 │        665.59 / 686.14 ±14.86 / 711.04 ms │                  640.72 / 648.91 ±4.71 / 655.41 ms │ +1.06x faster │
│ QQuery 5 │ 10046.59 / 10200.59 ±143.64 / 10421.45 ms │             9116.94 / 9237.59 ±153.88 / 9503.35 ms │ +1.10x faster │
│ QQuery 6 │       983.76 / 995.88 ±17.09 / 1029.36 ms │                  918.40 / 925.52 ±5.17 / 933.42 ms │ +1.08x faster │
│ QQuery 7 │        778.00 / 795.61 ±11.31 / 811.88 ms │                  769.46 / 776.09 ±4.89 / 781.16 ms │     no change │
└──────────┴───────────────────────────────────────────┴────────────────────────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary                                                 ┃            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD)                                                 │ 14595.14ms │
│ Total Time (parquet-morsel-driven-execution-237164415184908839)   │ 13414.08ms │
│ Average Time (HEAD)                                               │  1824.39ms │
│ Average Time (parquet-morsel-driven-execution-237164415184908839) │  1676.76ms │
│ Queries Faster                                                    │          6 │
│ Queries Slower                                                    │          1 │
│ Queries with No Change                                            │          1 │
│ Queries with Failure                                              │          0 │
└───────────────────────────────────────────────────────────────────┴────────────┘

Resource Usage

clickbench_extended — base (merge-base)

Metric	Value
Wall time	73.6s
Peak memory	33.2 GiB
Avg memory	28.6 GiB
CPU user	723.9s
CPU sys	23.7s
Disk read	0 B
Disk write	21.8 MiB

clickbench_extended — branch

Metric	Value
Wall time	67.6s
Peak memory	33.8 GiB
Avg memory	28.9 GiB
CPU user	728.2s
CPU sys	22.9s
Disk read	0 B
Disk write	72.0 KiB

[Dandandan]

run benchmark clickbench_pushdown

[adriangbot]

🤖 Benchmark running (GKE) | trigger
Linux bench-c4068979599-333-npw74 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux
Comparing parquet-morsel-driven-execution-237164415184908839 (b02a9c0) to ed793f0 (merge-base) diff using: clickbench_pushdown
Results will be posted here when complete

[adriangbot]

🤖 Benchmark completed (GKE) | trigger

Details

Comparing HEAD and parquet-morsel-driven-execution-237164415184908839
--------------------
Benchmark clickbench_pushdown.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query     ┃                                  HEAD ┃ parquet-morsel-driven-execution-237164415184908839 ┃        Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0  │          1.58 / 4.78 ±6.32 / 17.42 ms │                       1.62 / 4.84 ±6.36 / 17.55 ms │     no change │
│ QQuery 1  │        16.23 / 16.65 ±0.26 / 17.04 ms │                     14.62 / 15.09 ±0.39 / 15.78 ms │ +1.10x faster │
│ QQuery 2  │        58.07 / 59.30 ±1.22 / 61.58 ms │                     48.57 / 49.48 ±0.72 / 50.45 ms │ +1.20x faster │
│ QQuery 3  │        51.53 / 52.65 ±0.67 / 53.60 ms │                     42.42 / 44.32 ±1.66 / 47.10 ms │ +1.19x faster │
│ QQuery 4  │     306.49 / 310.55 ±2.75 / 315.05 ms │                  273.89 / 283.04 ±5.45 / 290.21 ms │ +1.10x faster │
│ QQuery 5  │     350.15 / 354.36 ±3.68 / 360.13 ms │                  318.56 / 325.01 ±4.35 / 332.08 ms │ +1.09x faster │
│ QQuery 6  │        11.41 / 11.87 ±0.45 / 12.54 ms │                      9.69 / 10.77 ±1.05 / 12.11 ms │ +1.10x faster │
│ QQuery 7  │        22.69 / 23.57 ±0.92 / 25.19 ms │                     19.77 / 20.22 ±0.34 / 20.82 ms │ +1.17x faster │
│ QQuery 8  │     434.81 / 440.78 ±4.10 / 445.21 ms │                 400.63 / 419.12 ±12.02 / 434.14 ms │     no change │
│ QQuery 9  │     667.83 / 679.13 ±6.51 / 685.78 ms │                  532.15 / 535.49 ±3.18 / 540.96 ms │ +1.27x faster │
│ QQuery 10 │     121.78 / 125.47 ±3.52 / 131.67 ms │                  102.99 / 106.41 ±4.33 / 114.31 ms │ +1.18x faster │
│ QQuery 11 │     135.72 / 136.35 ±0.60 / 137.48 ms │                  115.53 / 115.71 ±0.20 / 116.06 ms │ +1.18x faster │
│ QQuery 12 │     381.12 / 385.44 ±2.45 / 388.36 ms │                  345.06 / 346.97 ±1.33 / 348.62 ms │ +1.11x faster │
│ QQuery 13 │    496.26 / 511.63 ±18.20 / 545.60 ms │                 455.11 / 489.49 ±27.72 / 530.05 ms │     no change │
│ QQuery 14 │     394.37 / 397.51 ±1.66 / 399.08 ms │                  358.58 / 360.59 ±2.47 / 365.29 ms │ +1.10x faster │
│ QQuery 15 │    375.67 / 389.13 ±11.29 / 408.18 ms │                 338.54 / 353.16 ±15.73 / 383.08 ms │ +1.10x faster │
│ QQuery 16 │    749.54 / 762.51 ±16.00 / 793.46 ms │                  705.17 / 715.04 ±7.01 / 723.97 ms │ +1.07x faster │
│ QQuery 17 │    747.38 / 799.05 ±31.29 / 845.65 ms │                  701.42 / 707.28 ±4.82 / 715.36 ms │ +1.13x faster │
│ QQuery 18 │ 1478.83 / 1520.54 ±23.82 / 1550.83 ms │              1333.65 / 1450.60 ±59.56 / 1501.08 ms │     no change │
│ QQuery 19 │        37.61 / 39.24 ±1.12 / 40.57 ms │                    32.86 / 44.28 ±19.97 / 84.12 ms │  1.13x slower │
│ QQuery 20 │    706.92 / 723.81 ±16.43 / 745.82 ms │                 507.37 / 574.89 ±53.85 / 633.69 ms │ +1.26x faster │
│ QQuery 21 │     738.40 / 746.53 ±6.61 / 756.85 ms │                 586.31 / 611.26 ±18.18 / 638.22 ms │ +1.22x faster │
│ QQuery 22 │ 1180.00 / 1198.03 ±14.75 / 1216.08 ms │                971.77 / 990.53 ±17.11 / 1015.44 ms │ +1.21x faster │
│ QQuery 23 │    389.40 / 402.92 ±18.43 / 439.44 ms │                 252.89 / 294.66 ±48.45 / 388.95 ms │ +1.37x faster │
│ QQuery 24 │     116.50 / 118.25 ±1.65 / 120.95 ms │                     69.34 / 74.84 ±4.49 / 82.87 ms │ +1.58x faster │
│ QQuery 25 │     190.50 / 195.19 ±3.10 / 200.17 ms │                  156.92 / 159.83 ±2.44 / 163.97 ms │ +1.22x faster │
│ QQuery 26 │     155.50 / 161.37 ±4.52 / 167.58 ms │                  106.76 / 112.15 ±3.32 / 115.51 ms │ +1.44x faster │
│ QQuery 27 │   959.52 / 983.36 ±19.90 / 1019.90 ms │                 708.53 / 732.54 ±28.77 / 789.08 ms │ +1.34x faster │
│ QQuery 28 │ 7836.33 / 7912.82 ±47.38 / 7981.55 ms │             6689.84 / 6822.39 ±103.20 / 6978.24 ms │ +1.16x faster │
│ QQuery 29 │     331.71 / 336.14 ±6.67 / 349.38 ms │                 282.95 / 295.60 ±20.05 / 335.39 ms │ +1.14x faster │
│ QQuery 30 │     375.31 / 385.95 ±5.85 / 392.39 ms │                  342.51 / 350.79 ±7.75 / 364.77 ms │ +1.10x faster │
│ QQuery 31 │    369.28 / 386.85 ±16.44 / 414.10 ms │                  345.57 / 352.50 ±5.61 / 361.41 ms │ +1.10x faster │
│ QQuery 32 │ 1329.16 / 1366.43 ±43.61 / 1437.44 ms │              1047.60 / 1105.11 ±53.95 / 1184.53 ms │ +1.24x faster │
│ QQuery 33 │ 1610.71 / 1669.07 ±48.35 / 1742.99 ms │              1536.41 / 1553.61 ±16.93 / 1580.56 ms │ +1.07x faster │
│ QQuery 34 │ 1510.32 / 1569.52 ±36.83 / 1623.42 ms │              1555.47 / 1565.59 ±11.47 / 1585.47 ms │     no change │
│ QQuery 35 │     409.06 / 415.43 ±4.39 / 421.28 ms │                  342.24 / 346.76 ±6.65 / 359.96 ms │ +1.20x faster │
│ QQuery 36 │     102.17 / 111.12 ±6.62 / 119.99 ms │                     76.74 / 80.72 ±2.79 / 83.42 ms │ +1.38x faster │
│ QQuery 37 │        56.23 / 59.89 ±3.13 / 64.45 ms │                     44.50 / 48.87 ±3.02 / 53.46 ms │ +1.23x faster │
│ QQuery 38 │        54.39 / 58.83 ±3.37 / 62.07 ms │                     40.36 / 42.92 ±2.21 / 45.77 ms │ +1.37x faster │
│ QQuery 39 │     182.72 / 190.42 ±5.19 / 198.59 ms │                  121.77 / 129.32 ±6.70 / 141.68 ms │ +1.47x faster │
│ QQuery 40 │        34.33 / 36.94 ±2.20 / 40.86 ms │                     24.77 / 27.18 ±1.57 / 29.32 ms │ +1.36x faster │
│ QQuery 41 │        32.03 / 32.80 ±0.96 / 34.43 ms │                     22.05 / 24.20 ±1.90 / 27.30 ms │ +1.36x faster │
│ QQuery 42 │        24.75 / 25.23 ±0.30 / 25.63 ms │                     18.26 / 18.58 ±0.28 / 19.05 ms │ +1.36x faster │
└───────────┴───────────────────────────────────────┴────────────────────────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary                                                 ┃            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD)                                                 │ 26107.40ms │
│ Total Time (parquet-morsel-driven-execution-237164415184908839)   │ 22711.75ms │
│ Average Time (HEAD)                                               │   607.15ms │
│ Average Time (parquet-morsel-driven-execution-237164415184908839) │   528.18ms │
│ Queries Faster                                                    │         37 │
│ Queries Slower                                                    │          1 │
│ Queries with No Change                                            │          5 │
│ Queries with Failure                                              │          0 │
└───────────────────────────────────────────────────────────────────┴────────────┘

Resource Usage

clickbench_pushdown — base (merge-base)

Metric	Value
Wall time	131.8s
Peak memory	41.4 GiB
Avg memory	29.9 GiB
CPU user	1180.9s
CPU sys	117.8s
Disk read	0 B
Disk write	2.0 GiB

clickbench_pushdown — branch

Metric	Value
Wall time	114.8s
Peak memory	43.2 GiB
Avg memory	31.5 GiB
CPU user	1185.5s
CPU sys	101.9s
Disk read	0 B
Disk write	116.0 KiB

[Dandandan]

run benchmark clickbench_pushdown

env:
   SIMULATE_LATENCY: true

[adriangbot]

🤖 Benchmark running (GKE) | trigger
Linux bench-c4069123125-335-ndnlq 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux
Comparing parquet-morsel-driven-execution-237164415184908839 (d987e42) to c6f7145 (merge-base) diff using: clickbench_pushdown
Results will be posted here when complete

[adriangbot]

🤖 Benchmark completed (GKE) | trigger

Details

Comparing HEAD and parquet-morsel-driven-execution-237164415184908839
--------------------
Benchmark clickbench_pushdown.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query     ┃                                      HEAD ┃ parquet-morsel-driven-execution-237164415184908839 ┃        Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0  │              1.85 / 5.20 ±6.55 / 18.29 ms │                       1.78 / 5.26 ±6.65 / 18.56 ms │     no change │
│ QQuery 1  │    2051.28 / 2227.71 ±177.13 / 2488.58 ms │              1822.77 / 1889.26 ±43.42 / 1944.11 ms │ +1.18x faster │
│ QQuery 2  │    2640.98 / 2939.94 ±233.78 / 3338.32 ms │              2112.42 / 2179.45 ±56.71 / 2252.02 ms │ +1.35x faster │
│ QQuery 3  │    2553.63 / 2760.70 ±181.32 / 2982.74 ms │              2135.32 / 2181.63 ±64.71 / 2308.45 ms │ +1.27x faster │
│ QQuery 4  │     2836.68 / 2963.97 ±85.52 / 3067.33 ms │              2284.40 / 2333.57 ±54.99 / 2440.11 ms │ +1.27x faster │
│ QQuery 5  │    2645.40 / 2915.30 ±176.31 / 3151.48 ms │              2372.43 / 2434.07 ±73.40 / 2566.37 ms │ +1.20x faster │
│ QQuery 6  │       634.01 / 749.82 ±110.53 / 937.89 ms │                 474.25 / 507.62 ±25.09 / 546.09 ms │ +1.48x faster │
│ QQuery 7  │    2226.45 / 2342.47 ±133.03 / 2538.35 ms │              1851.73 / 1958.87 ±83.32 / 2078.22 ms │ +1.20x faster │
│ QQuery 8  │    2822.39 / 3060.41 ±172.76 / 3263.01 ms │             2414.89 / 2522.32 ±120.42 / 2728.31 ms │ +1.21x faster │
│ QQuery 9  │    3051.46 / 3375.14 ±192.05 / 3579.74 ms │              2643.39 / 2697.18 ±36.22 / 2756.20 ms │ +1.25x faster │
│ QQuery 10 │    5175.38 / 5486.33 ±297.08 / 5967.08 ms │             4291.72 / 4451.46 ±148.62 / 4666.86 ms │ +1.23x faster │
│ QQuery 11 │    5022.21 / 5236.48 ±172.50 / 5428.36 ms │             4246.05 / 4465.36 ±183.80 / 4765.16 ms │ +1.17x faster │
│ QQuery 12 │    2726.59 / 2964.91 ±152.01 / 3195.75 ms │              2415.08 / 2492.50 ±85.26 / 2639.72 ms │ +1.19x faster │
│ QQuery 13 │     5602.56 / 5732.70 ±90.60 / 5836.79 ms │             4619.87 / 4766.15 ±107.41 / 4950.14 ms │ +1.20x faster │
│ QQuery 14 │    5239.97 / 5543.59 ±152.54 / 5646.53 ms │             4401.33 / 4558.88 ±114.18 / 4740.10 ms │ +1.22x faster │
│ QQuery 15 │    2903.57 / 3075.31 ±156.39 / 3290.06 ms │              2336.51 / 2415.51 ±77.52 / 2558.60 ms │ +1.27x faster │
│ QQuery 16 │    3108.36 / 3230.98 ±146.02 / 3508.16 ms │              2774.88 / 2818.70 ±27.66 / 2860.69 ms │ +1.15x faster │
│ QQuery 17 │    2996.86 / 3249.99 ±145.60 / 3414.53 ms │              2809.10 / 2828.96 ±22.49 / 2872.18 ms │ +1.15x faster │
│ QQuery 18 │    3455.81 / 3670.66 ±132.61 / 3845.83 ms │              3014.70 / 3140.77 ±66.49 / 3192.49 ms │ +1.17x faster │
│ QQuery 19 │    2178.64 / 2422.30 ±239.43 / 2836.05 ms │              1680.73 / 1711.87 ±22.73 / 1746.08 ms │ +1.42x faster │
│ QQuery 20 │    2879.17 / 3144.97 ±182.39 / 3314.36 ms │             2572.29 / 2727.25 ±108.62 / 2871.43 ms │ +1.15x faster │
│ QQuery 21 │    5149.42 / 5586.17 ±329.53 / 6009.03 ms │              4877.62 / 4977.41 ±97.88 / 5105.06 ms │ +1.12x faster │
│ QQuery 22 │   9707.46 / 9937.73 ±221.73 / 10315.45 ms │             8687.22 / 8888.12 ±150.87 / 9023.85 ms │ +1.12x faster │
│ QQuery 23 │    3713.41 / 3949.50 ±168.21 / 4190.39 ms │              1871.80 / 1959.45 ±66.26 / 2026.87 ms │ +2.02x faster │
│ QQuery 24 │    2464.74 / 2666.89 ±119.90 / 2825.64 ms │              1482.20 / 1585.01 ±86.48 / 1736.16 ms │ +1.68x faster │
│ QQuery 25 │    2522.57 / 2712.88 ±140.15 / 2889.83 ms │              2253.42 / 2329.99 ±60.54 / 2436.69 ms │ +1.16x faster │
│ QQuery 26 │    2419.11 / 2680.81 ±174.85 / 2895.44 ms │             1581.70 / 1730.56 ±111.73 / 1881.88 ms │ +1.55x faster │
│ QQuery 27 │    5230.25 / 5465.87 ±254.92 / 5942.69 ms │             4955.57 / 5105.89 ±131.11 / 5339.26 ms │ +1.07x faster │
│ QQuery 28 │ 10150.98 / 10312.83 ±102.55 / 10447.80 ms │              8958.66 / 9027.47 ±41.36 / 9071.48 ms │ +1.14x faster │
│ QQuery 29 │    2492.48 / 2649.88 ±143.14 / 2889.53 ms │              2132.06 / 2214.46 ±57.24 / 2285.01 ms │ +1.20x faster │
│ QQuery 30 │    5367.93 / 5454.02 ±135.39 / 5721.43 ms │             4427.04 / 4604.45 ±158.65 / 4823.13 ms │ +1.18x faster │
│ QQuery 31 │    4979.49 / 5384.49 ±282.74 / 5719.00 ms │              4568.33 / 4612.85 ±31.07 / 4664.02 ms │ +1.17x faster │
│ QQuery 32 │    2880.71 / 3153.22 ±178.50 / 3415.00 ms │              2614.76 / 2657.18 ±30.05 / 2701.67 ms │ +1.19x faster │
│ QQuery 33 │    3708.92 / 3885.38 ±106.32 / 3978.94 ms │              3672.47 / 3735.22 ±55.44 / 3810.69 ms │     no change │
│ QQuery 34 │    3820.11 / 4068.86 ±276.42 / 4592.44 ms │              3663.35 / 3793.80 ±98.75 / 3894.39 ms │ +1.07x faster │
│ QQuery 35 │     2818.39 / 2965.07 ±93.25 / 3092.19 ms │              2364.85 / 2478.73 ±98.30 / 2619.06 ms │ +1.20x faster │
│ QQuery 36 │     1077.00 / 1158.69 ±67.86 / 1280.83 ms │                486.87 / 626.51 ±126.51 / 818.18 ms │ +1.85x faster │
│ QQuery 37 │     1032.10 / 1148.39 ±95.86 / 1290.63 ms │                  709.43 / 710.98 ±0.97 / 712.06 ms │ +1.62x faster │
│ QQuery 38 │     1238.80 / 1345.21 ±65.85 / 1399.93 ms │                526.52 / 679.91 ±121.20 / 878.66 ms │ +1.98x faster │
│ QQuery 39 │       884.73 / 980.28 ±99.41 / 1155.26 ms │                 472.67 / 535.33 ±71.03 / 668.77 ms │ +1.83x faster │
│ QQuery 40 │    1126.11 / 1310.05 ±162.67 / 1517.45 ms │                 518.06 / 623.58 ±96.91 / 794.00 ms │ +2.10x faster │
│ QQuery 41 │     1311.22 / 1344.55 ±20.43 / 1370.10 ms │               718.64 / 936.94 ±140.15 / 1147.76 ms │ +1.44x faster │
│ QQuery 42 │      946.86 / 1017.07 ±53.65 / 1097.64 ms │                  519.52 / 523.61 ±2.17 / 525.47 ms │ +1.94x faster │
└───────────┴───────────────────────────────────────────┴────────────────────────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┓
┃ Benchmark Summary                                                 ┃             ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━┩
│ Total Time (HEAD)                                                 │ 146276.69ms │
│ Total Time (parquet-morsel-driven-execution-237164415184908839)   │ 119424.10ms │
│ Average Time (HEAD)                                               │   3401.78ms │
│ Average Time (parquet-morsel-driven-execution-237164415184908839) │   2777.30ms │
│ Queries Faster                                                    │          41 │
│ Queries Slower                                                    │           0 │
│ Queries with No Change                                            │           2 │
│ Queries with Failure                                              │           0 │
└───────────────────────────────────────────────────────────────────┴─────────────┘

Resource Usage

clickbench_pushdown — base (merge-base)

Metric	Value
Wall time	732.7s
Peak memory	35.5 GiB
Avg memory	27.4 GiB
CPU user	1211.5s
CPU sys	150.6s
Disk read	0 B
Disk write	4.0 GiB

clickbench_pushdown — branch

Metric	Value
Wall time	598.4s
Peak memory	36.8 GiB
Avg memory	28.1 GiB
CPU user	1213.4s
CPU sys	145.4s
Disk read	0 B
Disk write	828.0 KiB

[adriangb]

Seems like morselization is even better w/ added latency??

[Dandandan]

Seems like morselization is even better w/ added latency??

Yeah still seems to improve things greatly!

However I think the relative improvement is smaller, because of lower utilization, perhaps also a bit due to somewhat smaller request sizes / more requests (and having no prefetching yet in place).

[Dandandan]

Oh wait - the relative improvement seems bigger still! 15% to 22% 🤯

I think it is mainly because of the added skew (from different request latencies). Nice to see this confirmed as well using this bench @alamb

[Omega359]

The results seem very promising! I'm curious at what point you think this will be ready for final review and inclusion? Right now it seems to be still in the development phase?

[alamb]

The results seem very promising! I'm curious at what point you think this will be ready for final review and inclusion? Right now it seems to be still in the development phase?

I am trying to make a "production ready" version here

• Sketch out a Morselize API #20820

I am hoping it will be ready / show the same improvements this PR does "soon" (friday hopefully)

[Dandandan]

The results seem very promising! I'm curious at what point you think this will be ready for final review and inclusion? Right now it seems to be still in the development phase?

• I am working on a proposal

[zhuqi-lucas]

Hey @Dandandan — are you still planning to continue this RG-level morsel work?

[Dandandan]

Hey @Dandandan — are you still planning to continue this RG-level morsel work?

You mean morsel splitting, or?
I plan to contribute to it.

Current state:

1. Morsel planning landed in Dynamic work scheduling in FileStream #21351
2. We want to do morsel splitting to further optimize queries with few row groups and partition skew at the "tail". We probably need some API changes in arrow-rs to make it work
3. @adriangb has some ideas to use it for better / adaptive row filters as well.

[Dandandan]

Closing this one as we have morsel-scan now 🥳 (except morsel-splitting)

[zhuqi-lucas]

Thanks @Dandandan! That makes sense.

Once morsel splitting lands, I'd like to build dynamic RG pruning for TopK on top of it — when a morsel is pulled from the queue, check the RG's min/max statistics against the current TopK dynamic filter threshold and skip it entirely if it can't contain qualifying rows. This is tracked in #21399.

Happy to collaborate when the morsel splitting API is ready!

[adriangb]

Just to be clear, by "morsel splitting" do we mean "splitting a file into smaller morsels" or do we mean "splitting a row group into smaller morsles"? Afaik neither is implemented, right?

[zhuqi-lucas]

Just to be clear, by "morsel splitting" do we mean "splitting a file into smaller morsels" or do we mean "splitting a row group into smaller morsles"? Afaik neither is implemented, right?

I am expecting the splitting a row group into smaller morsels.

[adriangb]

I think that has to happen after splitting files into many morsels, which is not currently happening.

[alamb]

FWIW I think when splitting files into morsels, we should consider using https://docs.rs/parquet/58.0.0/parquet/arrow/push_decoder/struct.ParquetPushDecoder.html#method.try_next_reader

[alamb]

I started hacking on this in

• Support Morsel output for Parquet known to be non blocking #21598

However I haven't had enough focus time to make progress really yet. I hope to have more time this Thursday / Friday