During a meeting we looked at dag bundles as a possible way to achieve multiple git repositories filled with some DAGs and all present in one Airflow instance.
Current known unknowns
(this list can be incomplete)
- How do dag-bundles behave if backfill is needed?
- They mention limitations in terms of triggering, what is the exact impact of this?
- Are their limitations regarding numbers of git repositories or size?
- What happens on failure?
- How exactly does this fit into release cycles ( Ensure you pull a very specific dag on commit hash or something )
- How might cross referencing or cross dependencies look like between dag-bundles? ( if possible at all )