feat(arrow-cast): fast path for Dictionary->View cast for large types and cross cast

[Abhisheklearn12]

Which issue does this PR close?

• Closes Fast path for Dictionary -> View cast for large types & cross cast #8985

Rationale for this change

unpack_dictionary handled all Dictionary→View casts correctly but incurred an unnecessary copy of the values buffer on every cast. For Dictionary arrays with many repeated values (the common use case), this copies data for every logical row rather than once.

A fast path already existed for Utf8->Utf8View and Binary->BinaryView via view_from_dict_values, which reuses the values buffer zero-copy and only writes 16-byte view structs per row. This PR extends that to the remaining cases called out in the TODO comments.

What changes are included in this PR?

• Add (LargeUtf8, Utf8View) fast path in dictionary_cast: reuses the values buffer zero-copy when i64 offsets fit in u32 (buffer < 4 GiB), falls back to unpack_dictionary when the buffer is too large
• Add (LargeBinary, BinaryView) fast path with the same offset-fit check
• Add (Utf8, BinaryView) cross cast fast path: UTF-8 strings are always valid binary so the buffer is reused unconditionally
• Add (Binary, Utf8View) cross cast via new binary_dict_to_string_view: validates UTF-8 of dictionary values and reuses the buffer zero-copy when all valid; respects CastOptions::safe , nullifies rows pointing to invalid dictionary values when safe=true, returns CastError when safe=false

Are these changes tested?

Yes. Added 6 tests in arrow-cast/src/cast/mod.rs:

• test_dict_large_utf8_to_utf8view-> LargeUtf8->Utf8View fast path, including null keys and values longer than 12 bytes (buffered views)
• test_dict_large_binary_to_binary_view -> LargeBinary->BinaryView fast path, including null keys
• test_dict_utf8_to_binary_view -> Utf8->BinaryView cross cast
• test_dict_binary_to_utf8view_valid -> Binary->Utf8View when all dictionary values are valid UTF-8 (zero-copy fast path)
• test_dict_binary_to_utf8view_invalid_utf8_strict -> Binary->Utf8view with invalid UTF-8 and safe=false returns CastError
• test_dict_binary_to_utf8view_invalid_utf8_safe -> Binary->Utf8View with invalid UTF-8 and safe=true nullifies every row whose key points to an invalid dictionary value, preserving valid rows

Are there any user-facing changes?

Yes. Casting Dictionary<_, LargeUtf8>->Utf8View, Dictionary<_, LargeBinary>->BinaryView, Dictionary<_, Utf8>->BinaryView, and Dictionary<_, Binary>->Utf8View is now significantly faster for large arrays with repeated values. The dictionary values buffer is reused without copying instead of being fully unpacked row-by-row.

[Abhisheklearn12]

Hi @Jefffrey, I’d love to get your feedback whenever you have time. Very appreciate it!

[Jefffrey]

I tried commenting out the new match arms and running the newly added tests here and hit a failure:

failures:

---- cast::tests::test_dict_binary_to_utf8view_invalid_utf8_strict stdout ----

thread 'cast::tests::test_dict_binary_to_utf8view_invalid_utf8_strict' (6965562) panicked at arrow-cast/src/cast/mod.rs:7379:9:
expected CastError, got InvalidArgumentError("Encountered non UTF-8 data: invalid utf-8 sequence of 1 bytes from index 5")


failures:
    cast::tests::test_dict_binary_to_utf8view_invalid_utf8_strict

test result: FAILED. 342 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s

We should look into this as this PR is meant to only add fast paths, so change in behaviour seems a little odd 🤔

[Jefffrey]

I feel this arm specifically should be benchmarked as it introduces new logic compared to the other arms

[Jefffrey]

This check reads a little odd to me as usually this could mean unpack_dictionary may also fail if offsets don't fit?