feat(connectors): Delta Lake Sink Connector

[kriti-sc]

Which issue does this PR close?

Closes #1852

Rationale

Delta Lake is a data analytics engine, and very popular in modern streaming analytics architectures.

What changed?

Introduces a Delta Lake Sink Connector that enables writing data from Iggy to Delta Lake.

The Delta Lake writing logic is heavily inspired by the kafka-delta-ingest project, to have a proven starting ground for writing to Delta Lake.

Local Execution

1. Produced 32632 messages with schema user_id: String, user_type: u8, email: String, source: String, state: String, created_at: DateTime<Utc>, message: String using sample data producer.
2. Consumed messages using the Delta Lake sink and created a Delta table on filesystem.
3. Verified number of rows in delta table and the schema.
4. Added unit tests and e2e tests, both passing.

Left: messages produced; Right(top): messages consumed by Delta sink; Right(bottom): Inspecting Delta table in python

AI Usage

If AI tools were used, please answer:

1. Which tools? Claude Code
2. Scope of usage? generated functions
3. How did you verify the generated code works correctly? Manual testing by producing data into Iggy and then running the sink and verifying local Delta Lake creation, unit tests and e2e tests for local Delta Lake and Delta Lake on S3.
4. Can you explain every line of the code if asked? Yes

[codecov]

Codecov Report

❌ Patch coverage is 95.57400% with 32 lines in your changes missing coverage. Please review.
✅ Project coverage is 72.85%. Comparing base (27f0f11) to head (c1006d8).

Files with missing lines	Patch %	Lines
core/connectors/sinks/delta_sink/src/sink.rs	15.38%	22 Missing ⚠️
core/connectors/sinks/delta_sink/src/coercions.rs	97.51%	6 Missing and 4 partials ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master    #2889      +/-   ##
============================================
+ Coverage     72.71%   72.85%   +0.14%     
  Complexity      943      943              
============================================
  Files          1117     1121       +4     
  Lines         96285    97008     +723     
  Branches      73485    74225     +740     
============================================
+ Hits          70014    70680     +666     
- Misses        23725    23760      +35     
- Partials       2546     2568      +22     

Components	Coverage Δ
Rust Core	73.66% <95.57%> (+0.20%)	⬆️
Java SDK	62.30% <ø> (ø)
C# SDK	69.10% <ø> (-0.31%)	⬇️
Python SDK	81.43% <ø> (ø)
Node SDK	91.53% <ø> (ø)
Go SDK	38.97% <ø> (ø)

Files with missing lines	Coverage Δ
core/connectors/sdk/src/convert.rs	100.00% <100.00%> (ø)
core/connectors/sinks/delta_sink/src/lib.rs	100.00% <100.00%> (ø)
core/connectors/sinks/delta_sink/src/storage.rs	100.00% <100.00%> (ø)
core/connectors/sinks/delta_sink/src/coercions.rs	97.51% <97.51%> (ø)
core/connectors/sinks/delta_sink/src/sink.rs	15.38% <15.38%> (ø)

... and 20 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[hubcio]

overall good direction, but needs a bit refining. thanks for contribution @kriti-sc

[hubcio]

@kriti-sc I will finish review after both #2925 and #2933 get merged and you rebase against these.

[github-actions]

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs.

If you need a review, please ensure CI is green and the PR is rebased on the latest master. Don't hesitate to ping the maintainers - either @core on Discord or by mentioning them directly here on the PR.

Thank you for your contribution!

[spetz]

Maybe instead of returning a hash map, we could have the dedicated structs (per storage provider) with optional fields - and then either an enum could be returned or a single struct with all available storages as optional fields?

[kriti-sc]

Valid point, however this change would require the config to use nested structs. Because the mechanism to convert env vars into the config (function apply_plugin_config_env_overrides in runtime) does not support nested structs.

It would be very good to support nested structs in config, but that change doesn't belong in this PR. I propose adding a TODO and following up with a PR that adds support for nested structs + changing Delta Sink config to use nested structs.

Let me know your thoughts @spetz

[spetz]

Same here (based on my previous suggestion) - we would use the dedicated option struct, and then simply serialize it.