feat(ipc): Avoid quadratic dictionary delta accumulation in IPC reader

[pchintar]

Which issue does this PR close?

• Closes Avoid quadratic Dictionary Delta accumulation in IPC reader by deferring materialization #9775 .

Rationale for this change

Delta dictionaries are currently handled in the reader as:

let combined = concat::concat(&[existing, &dict_values])?;
dictionaries_by_id.insert(dict_id, combined);

This executes for every DictionaryBatch with isDelta = true, even before any RecordBatch consumes the dictionary. So, for a sequence of delta updates:

delta 1 -> copy 1
delta 2 -> copy 2
delta 3 -> copy 3
...

Total work:

O(1 + 2 + 3 + ... + N) = O(N²)

where N is the number of accumulated dictionary elements.

What changes are included in this PR?

Defer dictionary materialization until it is required for record batch decoding.
Maintain:

HashMap<i64, ArrayRef>          // materialized dictionaries
HashMap<i64, Vec<ArrayRef>>     // pending delta chunks

Behavior:

• Non-delta dictionary

  • replace the materialized dictionary
  • clear pending deltas for that id

• Delta dictionary

  • append the delta chunk to the pending list
  • do not concatenate immediately

• Before RecordBatch decode

  • concatenate:

    existing + pending deltas
    

  • update the materialized dictionary map

  • clear the pending list

---

Complexity Improvement

Before / Current Behavior

Every time a delta dictionary arrives, we rebuild the full dictionary:

// A,B,C are successive delta updates/additions to the Dictionary
delta 1 → build [base dictionary, A]
delta 2 → build [base dictionary + A, B]
delta 3 → build [base dictionary + A + B, C]
...

That means:

• we keep recreating larger and larger arrays
• earlier data gets copied again and again

Work done:
[base + A] + [base + A + B] + [base + A + B + C] + ...

---

After / New Behavior

We just store the incoming entries and delay building the full dictionary:

// A,B,C are successive delta updates/additions to the Dictionary
delta 1 → store [A]
delta 2 → store [B]
delta 3 → store [C]
...

Then only when a record batch actually needs it:

before decode → build [base dictionary, A, B, C] (once)

Work done:
[base + A + B + C + ...]   (single concat)

Are these changes tested?

Modified components:

• arrow-ipc/src/reader.rs

  • StreamReader
  • FileDecoder

• arrow-ipc/src/reader/stream.rs

  • StreamDecoder

Tests:

• Added:

  • arrow-ipc/src/tests/delta_dictionary.rs

    • test_multiple_deltas_before_record_batch

• Ensures correct behavior when multiple delta updates occur before record batch decoding

• Benchmarks and examples updated to handle the &mut self requirement introduced by FileDecoder::read_record_batch

Are there any user-facing changes?

Internal only. No public API or IPC format changes.

[alamb]

Thanks @pchintar

Do you have any benchmarks that show the performance improvement for this PR?

[pchintar]

Thanks for the response @alamb .

I haven’t run dedicated benchmarks yet. This change targets a specific accumulation pattern (repeated delta dictionary updates), which may not be directly covered by existing benchmarks and so may require a custom setup.

Due to system/hardware constraints I wasn’t able to put that together, but the change removes a quadratic copy pattern and replaces it with a linear one, so the expected improvement must follow from that. I’m happy to follow up with a microbenchmark if needed.

[alamb]

I understand it improves the code in this case; However given limited review bandwidth we are much more likely to have time to review improvements if they come with benchmarks / some other evidence to help us understand how mucht hey will help