|
| 1 | +# Managing Breaking Changes in Connectors |
| 2 | + |
| 3 | +Whenever possible, changes to connectors should be implemented in a non-breaking way. This allows users to upgrade to the latest version of a connector without additional action required on their part. Assume that _every_ breaking changes creates friction for users and should be avoided except when absolutely necessary. |
| 4 | + |
| 5 | +When it is not possible to make changes in a non-breaking manner, additional **breaking change requirements** include: |
| 6 | + |
| 7 | +1. A **Major Version** increase. (Or minor in the case of a pre-1.0.0 connector in accordance with Semantic Versioning rules) |
| 8 | +2. A [`breakingChanges` entry](https://docs.airbyte.com/connector-development/connector-metadata-file/) in the `releases` section of the `metadata.yaml` file |
| 9 | +3. A migration guide which details steps that users should take to resolve the change |
| 10 | +4. An Airbyte Engineer to complete the [Connector Breaking Change Release Playbook](https://docs.google.com/document/u/0/d/1VYQggHbL_PN0dDDu7rCyzBLGRtX-R3cpwXaY8QxEgzw/edit) (internal link) before merging the PR. |
| 11 | + |
| 12 | +## Types of Breaking Changes |
| 13 | + |
| 14 | +A breaking change is any change that requires users to take action before they can continue to sync data. The following are examples of breaking changes: |
| 15 | + |
| 16 | +- **Spec Change** - The configuration required by users of this connector have been changed and syncs will fail until users reconfigure or re-authenticate. This change is not possible via a Config Migration |
| 17 | +- **Schema Change** - The type of property previously present within a record has changed |
| 18 | +- **Stream or Property Removal** - Data that was previously being synced is no longer going to be synced. |
| 19 | +- **Destination Format / Normalization Change** - The way the destination writes the final data or how normalization cleans that data is changing in a way that requires a full-refresh. |
| 20 | +- **State Changes** - The format of the source’s state has changed, and the full dataset will need to be re-synced |
| 21 | +- **Full Downstream Rewrite** - Very rarely, a change is so sigificant that it requires a full rewrite of downstream SQL transformations or BI dashboards. In these cases, consider forking the connector as a "Gen 2" version instead of making a breaking change that would fully break users' downstream pipelines. |
| 22 | + - Example: Migration from legacy JSON-only "raw" tables to normalized typed columns in a destination tables. These historic changes were so significant that they would break all downstream SQL transformations and BI dashboards. A "gen-2" approach in these cases gives users the ability to run both "Gen 1" and "Gen 2" in parallel, migrating only after they have had a chance to adapt their code to the new data models. |
| 23 | + |
| 24 | +### Avoiding Breaking Changes with Migrations |
| 25 | + |
| 26 | +In some cases, breaking changes can be avoided through automated migrations: |
| 27 | + |
| 28 | +- **Config Migrations** - For low-code connectors, config migrations can automatically transform old configurations to new formats, avoiding the need for users to manually reconfigure. See the [low-code config migration model](https://github.com/airbytehq/airbyte-python-cdk/blob/8158f0d2a07c0480d25581359d54c8f9d30dbb29/airbyte_cdk/sources/declarative/declarative_component_schema.yaml#L4021-L4045) for technical reference. |
| 29 | + |
| 30 | +- **State Migrations** - For low-code connectors, state migrations can automatically transform old state formats to new formats, avoiding the need for a full refresh. See the [low-code state migration model](https://github.com/airbytehq/airbyte-python-cdk/blob/8158f0d2a07c0480d25581359d54c8f9d30dbb29/airbyte_cdk/sources/declarative/declarative_component_schema.yaml#L1579-L1587) for technical reference. |
| 31 | + |
| 32 | +When considering spec or state changes, evaluate whether a migration can eliminate the need for a breaking change before proceeding with a major version bump. |
| 33 | + |
| 34 | +### Schema Change Considerations: Data Type Compatibility |
| 35 | + |
| 36 | +When changing data types in schemas, consider type compatibility to determine whether the change is truly breaking: |
| 37 | + |
| 38 | +- **Widening that is breaking downstream:** Changes like `float` to `str` or `datetime` to `str` widen the type but may break downstream processes that expect numeric or date values. |
| 39 | + |
| 40 | +- **Widening that is non-breaking:** Changes like `int` to `bigint` or `int` to `double` widen the type in a compatible way that preserves data semantics. |
| 41 | + |
| 42 | +- **Non-compatible type changes:** Changes between fundamentally incompatible types (e.g., `datetime` to `float` or vice versa) are breaking and should be avoided if at all possible, as they change the semantic meaning of the data. |
| 43 | + |
| 44 | +When in doubt, treat type changes as breaking unless you can verify that all downstream use cases will handle the new type correctly. |
| 45 | + |
| 46 | +## Defining the Scope of Breaking Changes |
| 47 | + |
| 48 | +Some legitimate breaking changes may not impact all users of the connector. For example, a change to the schema of a specific stream only impacts users who are syncing that stream. |
| 49 | + |
| 50 | +The breaking change metadata allows you to specify narrowed scopes, and specifically _which streams_ are specifically affected by a breaking change. See the [`breakingChanges` entry](https://docs.airbyte.com/connector-development/connector-metadata-file/) documentation for supported scopes. If a user is not using the affected streams and therefor are not affected by a breaking change, they will not see any in-app messaging or emails about the change. |
| 51 | + |
| 52 | +## Migration Guide Documentation Requirements |
| 53 | + |
| 54 | +Your migration guide must be created as a separate file at `docs/integrations/{sources|destinations}/{connector-name}-migrations.md`. The guide should be detailed and user-focused, addressing the following for each breaking change version: |
| 55 | + |
| 56 | +- **WHAT** - What changed: Specifically, what is fixed or better for the user after this change? |
| 57 | +- **WHY** - Why did we make this change? (API improvements, upstream deprecation, bug fixes, performance improvements). |
| 58 | +- **WHO** - Which users are affected? Be specific about streams, sync modes, or configuration options that are impacted. |
| 59 | +- **STEPS** - Exact steps users must take to migrate, including when to take them (before/after upgrade, before/after first sync). |
| 60 | + |
| 61 | +Your migration guide can be as long as necessary and may include images, code snippets, SQL examples, and compatibility tables to help users understand and execute the migration. |
| 62 | + |
| 63 | +### Examples of Good Migration Guides |
| 64 | + |
| 65 | +Review these examples to understand the expected format and level of detail: |
| 66 | + |
| 67 | +- [HubSpot Migration Guide](/integrations/sources/hubspot-migrations) - Clear version sections with affected stream identification |
| 68 | +- [Google Ads Migration Guide](/integrations/sources/google-ads-migrations) - Table-based field change documentation |
| 69 | +- [Stripe Migration Guide](/integrations/sources/stripe-migrations) - Detailed sync mode and cursor field changes |
| 70 | +- [Snowflake Destination Migration Guide](/integrations/destinations/snowflake-migrations) - Use case-based migration paths |
| 71 | + |
| 72 | +## Breaking Change Metadata Requirements |
| 73 | + |
| 74 | +When adding a `breakingChanges` entry to your connector's `metadata.yaml` file, you must provide two critical fields: |
| 75 | + |
| 76 | +### Message Field |
| 77 | + |
| 78 | +The `message` field is a short summary shown in-app to all users of the connector. This message should: |
| 79 | + |
| 80 | +- Be concise but informative (users should know if they're affected and what action is needed). |
| 81 | +- Identify which users are affected. |
| 82 | + For example: "users syncing the `campaigns` stream" |
| 83 | +- Summarize the action required (detailed steps go in the migration guide). |
| 84 | +- Link to the migration guide for full details. |
| 85 | + |
| 86 | +The platform uses this message to send automated emails to affected users, so clear communication is critical. |
| 87 | + |
| 88 | +### Upgrade Deadline Field |
| 89 | + |
| 90 | +The `upgradeDeadline` field specifies the date by which users should upgrade (format: `YYYY-MM-DD`). When setting this deadline: |
| 91 | + |
| 92 | +- **Minimum timelines:** |
| 93 | + - Source connectors: At least 7 days (2 weeks recommended for simple changes) |
| 94 | + - Destination connectors: At least 1 month (destinations have broader impact on data pipelines) |
| 95 | + |
| 96 | +- **Rationale:** The deadline should provide enough time for users to review the migration guide, test in staging environments, and execute the migration steps. |
| 97 | + |
| 98 | +- **Exception: Immediate upstream breakage:** In the case of immediate upstream breaking changes, such as an already-removed upstream API endpoint, the deadline can be present-day or even in the past - with the rationale that users' connections are _already_ broken without the fix and therefor need the upgrade applied immediately. |
| 99 | + |
| 100 | +- **Automated notifications:** The platform automatically emails users when a breaking change is released and sends reminders as the deadline approaches. |
| 101 | + |
| 102 | +## Related Topics |
| 103 | + |
| 104 | +- [Semantic Versioning for Connectors](../contributing-to-airbyte/resources/pull-requests-handbook.md#semantic-versioning-for-connectors) - Guidelines for determining Major/Minor/Patch version changes |
| 105 | +- [Connector Metadata File](https://docs.airbyte.com/connector-development/connector-metadata-file/) - Technical reference for `breakingChanges` metadata format |
| 106 | +- [Pull Request Title Convention](../contributing-to-airbyte/resources/pull-requests-handbook.md#pull-request-title-convention) - How to format PR titles (use 🚨 emoji for breaking changes) |
| 107 | +- [QA Checks](../contributing-to-airbyte/resources/qa-checks.md) - Automated quality checks including breaking change requirements |
| 108 | +- [Connector Breaking Change Release Playbook](https://docs.google.com/document/u/0/d/1VYQggHbL_PN0dDDu7rCyzBLGRtX-R3cpwXaY8QxEgzw/edit) - Internal process for Airbyte Engineers (requires access) |
0 commit comments