[Improvement]: Upgrade Iceberg with per-engine compatibility modules #4110
Description
Search before asking
- I have searched in the issues and found no similar issues.
What would you like to be improved?
Amoro's Iceberg upgrade path currently conflicts across engines because Flink/Spark/Hive depend on different Iceberg artifact availability and API levels. A single global iceberg.version causes incompatibilities in build and runtime behavior.
How should we improve?
Adopt per-module iceberg.version overrides and engine-specific compatibility modules:
- Keep older engines on compatible Iceberg versions when upstream artifacts/APIs are limited.
- Upgrade newer engines to newer Iceberg versions.
- Maintain separate compatibility modules for Flink 1.17 paths (common + iceberg bridge).
- Adapt mixed-hive, flink bridge, and spark parquet reader code paths for newer Iceberg APIs.
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Subtasks
- Upgrade Iceberg versions with per-module override strategy.
- Add Flink 1.17 compatibility modules.
- Adapt mixed-hive/flink/spark code for Iceberg API changes.
- Final CI verification and follow-up fixes if needed.
Code of Conduct
- I agree to follow this project's Code of Conduct