add support for full_refresh in template_fields#177
add support for full_refresh in template_fields#177tomasfarias merged 3 commits intotomasfarias:masterfrom
Conversation
|
@tomasfarias could you have a look at this one? :) |
|
@mvdh98 Thanks for the PR! I will do a more thorough review later, but for now I'll let CI run the tests. |
tomasfarias
left a comment
There was a problem hiding this comment.
Pretty good work! I have a few questions, but overall great, shouldn't take too much more work to get it merged.
|
CI seems a bit flaky today, but the failures do not seem to be related to this PR. Retrying seems to help. |
|
@mvdh98 One final question: May I ask for the motivation for this change? Just curious as to know in case there is more work for us to do here. Did you encounter a situation in which this feature would have been useful? Thanks! |
The |
Thanks! This is what I was looking for. I see the use case now: Have the same DAG running continuously for incremental runs, and have the option to full-refresh when required via the UI. Cool! I'll re-run the tests and if everything is green will merge & release this. |
Summary
full_refreshtotemplate_fieldsonDbtRunOperator,DbtSeedOperator,DbtCompileOperator, andDbtBuildOperator, enabling Jinja templating in DAG definitions.TableMutabilityConfig.__post_init__to handle Airflow renderingfull_refreshas a string after template resolution.Example usage
Test plan
test_full_refresh_in_template_fieldsfor each of the 4 operatorstest_full_refresh_templatedfor each of the 4 operatorstest_table_mutability_config_full_refresh_string_coercionfor string-to-bool coercionDevelopment Approach