Skip to content

[AMORO-4110] Upgrade Iceberg versions with per-engine compatibility modules#4109

Draft
xxubai wants to merge 7 commits intoapache:masterfrom
xxubai:upgrade-iceberg-version
Draft

[AMORO-4110] Upgrade Iceberg versions with per-engine compatibility modules#4109
xxubai wants to merge 7 commits intoapache:masterfrom
xxubai:upgrade-iceberg-version

Conversation

@xxubai
Copy link
Contributor

@xxubai xxubai commented Mar 5, 2026

Why are the changes needed?

Amoro is upgrading Iceberg, but different engines (Flink/Spark/Hive) have different compatibility windows for Iceberg artifacts and APIs.
To avoid a one-version-for-all conflict, this PR introduces per-module version overrides and engine-specific compatibility modules, so newer engines can use newer Iceberg while older engines keep compatible versions.

Close #4110.

Brief change log

  • Introduce per-module iceberg.version override strategy in mixed-format modules.
  • Upgrade Iceberg for newer engines while keeping compatibility for older engines.
  • Add Flink 1.17 dedicated compatibility modules:
    • amoro-mixed-flink-common-1.17
    • amoro-mixed-flink-common-iceberg-bridge-1.17
  • Adapt Flink-related Iceberg bridge code for new Iceberg APIs:
    • parquet/flink reader path adjustments
    • MiniClusterResource test API compatibility updates
  • Adapt mixed-hive code to new Iceberg parquet API changes.
  • Adapt Spark parquet readers (v3.3/v3.4/v3.5) for Iceberg API updates.
  • Update related tests (including IcebergRewriteExecutorTest) and module wiring in POMs.
  • This change set was generated with Codex assistance.

How was this patch tested?

  • Add some test cases that check the changes thoroughly including negative and positive cases if possible

  • Add screenshots for manual tests if appropriate

  • Run test locally before making a pull request

Local verification (examples):

  • mvn -pl amoro-format-mixed/amoro-mixed-hive -am -DskipTests compile
  • mvn -pl amoro-format-mixed/amoro-mixed-flink/amoro-mixed-flink-common-iceberg-bridge -am -DskipTests compile
  • mvn -pl amoro-format-mixed/amoro-mixed-flink/v1.17/amoro-mixed-flink-1.17,amoro-format-mixed/amoro-mixed-flink/v1.18/amoro-mixed-flink-1.18 -am -DskipTests compile
  • project package validation (including dist packaging) has been executed in local environment

Documentation

  • Does this pull request introduce a new feature? (no)
  • If yes, how is the feature documented? (not applicable)

@xxubai xxubai marked this pull request as ready for review March 5, 2026 16:51
@xxubai xxubai force-pushed the upgrade-iceberg-version branch from b685b0a to e43126e Compare March 5, 2026 16:54
@xxubai xxubai changed the title Upgrade Iceberg versions with per-engine compatibility modules [AMORO-4110] Upgrade Iceberg versions with per-engine compatibility modules Mar 5, 2026
@xxubai xxubai marked this pull request as draft March 5, 2026 17:04
@github-actions github-actions bot added module:mixed-flink Flink moduel for Mixed Format module:mixed-spark Spark module for Mixed Format module:mixed-hive Hive moduel for Mixed Format type:build labels Mar 5, 2026
@codecov-commenter
Copy link

Codecov Report

❌ Patch coverage is 0% with 10 lines in your changes missing coverage. Please review.
✅ Project coverage is 22.42%. Comparing base (e02295b) to head (9aba8b4).
⚠️ Report is 8 commits behind head on master.

Files with missing lines Patch % Lines
.../java/org/apache/amoro/op/MixedOverwriteFiles.java 0.00% 10 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##             master    #4109      +/-   ##
============================================
- Coverage     22.44%   22.42%   -0.02%     
  Complexity     2552     2552              
============================================
  Files           458      458              
  Lines         42022    42056      +34     
  Branches       5915     5917       +2     
============================================
  Hits           9433     9433              
- Misses        31777    31811      +34     
  Partials        812      812              
Flag Coverage Δ
trino 22.42% <0.00%> (-0.02%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

module:mixed-flink Flink moduel for Mixed Format module:mixed-hive Hive moduel for Mixed Format module:mixed-spark Spark module for Mixed Format type:build type:infra

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Improvement]: Upgrade Iceberg with per-engine compatibility modules

2 participants