Skip to content

add support for full_refresh in template_fields#177

Merged
tomasfarias merged 3 commits intotomasfarias:masterfrom
mvdh98:feat-add-full-refresh-to-template-fields
Mar 29, 2026
Merged

add support for full_refresh in template_fields#177
tomasfarias merged 3 commits intotomasfarias:masterfrom
mvdh98:feat-add-full-refresh-to-template-fields

Conversation

@mvdh98
Copy link
Copy Markdown

@mvdh98 mvdh98 commented Mar 26, 2026

Summary

  • Adds full_refresh to template_fields on DbtRunOperator, DbtSeedOperator, DbtCompileOperator, and DbtBuildOperator, enabling Jinja templating in DAG definitions.
  • Adds string-to-bool coercion in TableMutabilityConfig.__post_init__ to handle Airflow rendering full_refresh as a string after template resolution.

Example usage

DbtRunOperator(                        
    task_id="dbt_run",                                                                                                                                                                                  
    full_refresh="{{ params.full_refresh }}",
)

Test plan

  • Added test_full_refresh_in_template_fields for each of the 4 operators
  • Added test_full_refresh_templated for each of the 4 operators
  • Added test_table_mutability_config_full_refresh_string_coercion for string-to-bool coercion
  • Verified existing tests still pass

Development Approach

  • First added the tests and verified they failed
  • Then made the code changes
  • Then verified the tests now succeed and any already existing tests still succeed

@mvdh98
Copy link
Copy Markdown
Author

mvdh98 commented Mar 26, 2026

@tomasfarias could you have a look at this one? :)

@tomasfarias
Copy link
Copy Markdown
Owner

@mvdh98 Thanks for the PR! I will do a more thorough review later, but for now I'll let CI run the tests.

Copy link
Copy Markdown
Owner

@tomasfarias tomasfarias left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pretty good work! I have a few questions, but overall great, shouldn't take too much more work to get it merged.

@tomasfarias
Copy link
Copy Markdown
Owner

CI seems a bit flaky today, but the failures do not seem to be related to this PR. Retrying seems to help.

@tomasfarias
Copy link
Copy Markdown
Owner

@mvdh98 One final question: May I ask for the motivation for this change? Just curious as to know in case there is more work for us to do here. Did you encounter a situation in which this feature would have been useful?

Thanks!

@mvdh98
Copy link
Copy Markdown
Author

mvdh98 commented Mar 29, 2026

@mvdh98 One final question: May I ask for the motivation for this change? Just curious as to know in case there is more work for us to do here. Did you encounter a situation in which this feature would have been useful?

Thanks!

The full_refresh parameter wasn't included in template_fields, which meant it couldn't be set dynamically using Jinja templates in DAG definitions. This change allows users to control full_refresh at runtime — for example, triggering a DAG run with full_refresh=True via params without needing a separate operator or DAG for full refresh runs. I needed this functionality to be able to do ad hoc full refresh runs from the airflow UI, using the same DAG.

@tomasfarias
Copy link
Copy Markdown
Owner

I needed this functionality to be able to do ad hoc full refresh runs from the airflow UI, using the same DAG.

Thanks! This is what I was looking for. I see the use case now: Have the same DAG running continuously for incremental runs, and have the option to full-refresh when required via the UI.

Cool!

I'll re-run the tests and if everything is green will merge & release this.

@tomasfarias tomasfarias merged commit 9162b2a into tomasfarias:master Mar 29, 2026
17 of 18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants