[Feature Request] Configurable Retry Limits and Cost Control Circuit Breakers

[Saurav Ganguli (sauravganguli)]

Problem Statement

Connectors have inconsistent and uncontrollable retry behavior that can lead to runaway costs. Users currently have no way to limit retry attempts or set circuit breakers to prevent excessive credit consumption on large data volume syncs.

Real-World Impact

A recent production incident illustrates this gap:

• Connector: Klaviyo → BigQuery
• Scenario: Sync extracted >100 GB of data successfully but failed during load phase
• Issue: System automatically retried 23 times, transferring the same 100+ GB repeatedly
• Result: Massive unexpected credit consumption with no ability to stop the runaway behavior
  Different connectors retry different amounts (5x vs 23x) with no consistency or user control. For organizations managing 30+ active connections, this creates unpredictable costs and requires constant manual monitoring.

Current Limitations

❌ Retry logic is hardcoded in individual connector implementations
❌ No configuration options at connection or workspace level
❌ No safeguards against repeated large data transfers on failing syncs
❌ No way to "fail fast" for problematic connections
❌ Inconsistent behavior across different connectors

Proposed Solution

Implement three configurable control mechanisms:

1. Configurable Retry Limits Per Connection
Allow users to override connector-internal retry behavior:

• Connection-level setting: Max retry attempts (e.g., 0-5 retries)
• Workspace-level defaults: Organization-wide retry policy
• UI/API control: Easy configuration without modifying connector code

Example Use Case: Set Klaviyo → BigQuery connection to max 2 retries instead of 23

2. Data Volume Circuit Breakers
Automatic sync termination based on data transfer thresholds:

• Per-sync data limits: "Stop if > X GB transferred in single attempt"
• Cumulative limits: "Stop if total retry data > Y GB"
• Configurable at connection level: Different limits for different connections

Example Use Case: Set 150 GB limit on Klaviyo connection to prevent 23x retries of 100+ GB

3. Credit-Based Circuit Breakers
Cost-aware sync management:

• Credit limits per sync attempt: Max credits per single sync run
• Connection-level budgets: Daily/weekly credit caps per connection
• Proactive alerts: Notifications when approaching limits (e.g., 80% threshold)
• Automatic termination: Stop sync when cost limit reached

Example Use Case: Set max 100 credits per Klaviyo sync attempt

Proposed UI/UX

Connection Settings Page:

Advanced Sync Controls
━━━━━━━━━━━━━━━━━━━━━━━
Retry Behavior
  Max retry attempts: [3 ▼] (default: connector-specific)
  □ Use workspace default (5 retries)

Data Volume Limits
  □ Enable circuit breaker
    Stop sync if data volume exceeds: [100] GB per attempt
    Stop sync if cumulative retry data exceeds: [250] GB

Credit Limits
  □ Enable cost circuit breaker
    Max credits per sync attempt: [100]
    Alert threshold: [80]%

  □ Enable connection budget
    Max credits per day: [200]

Workspace Settings Page:

Workspace-wide Sync Policies
━━━━━━━━━━━━━━━━━━━━━━━━━━
Default retry limit for new connections: [3 ▼]
Apply to existing connections: [Apply to All]

Benefits

✅ Predictable costs: Prevent runaway credit consumption
✅ User control: Configure retry behavior per connection needs
✅ Fail-fast capability: Stop problematic syncs early
✅ Budget protection: Hard limits prevent overruns
✅ Consistency: Standardized behavior across all connectors
✅ Operational efficiency: Reduce manual monitoring burden

Related Issues

This feature request addresses cost control and would complement:

• Sync scheduling improvements
• Resource usage monitoring/dashboards
• Alerting and notification systems

[Aaron ("AJ") Steers (aaronsteers)]

Hi Saurav Ganguli (@sauravganguli), thank you for this detailed and well-structured feature request!

The scenario you describe — a sync retrying 23 times with 100+ GB transfers each time — is a compelling illustration of why user-facing retry and cost controls are needed. The lack of configurable retry limits and circuit breakers is a real gap in the platform today.

Your proposal covers three important areas:

1. Configurable Retry Limits — Allowing users to override hardcoded connector retry behavior at the connection and workspace level
2. Data Volume Circuit Breakers — Automatic sync termination based on data transfer thresholds
3. Credit-Based Circuit Breakers — Cost-aware sync management with budgets and alerts

The detailed UI/UX mockups are very helpful for understanding the intended user experience.

I have escalated this to the internal team for evaluation: https://github.com/airbytehq/oncall/issues/11418

A few questions that would help the team prioritize and scope this:

• Are you using Airbyte Cloud or Airbyte OSS/Self-Managed? (The credit-based controls would apply primarily to Cloud, while retry limits and data volume controls would be relevant to both.)
• For the retry behavior you observed (23 retries on Klaviyo → BigQuery), was this on the source extraction or destination loading side? You mentioned the load phase failed — understanding where the retry loop occurs would help narrow the scope.
• Would a simpler initial implementation (e.g., just configurable max retry attempts per connection) address your most urgent need, or is the full circuit breaker suite critical for your use case?

---

Need more help? Join Airbyte Community Slack for peer support, or if you're a Cloud customer, open a support ticket referencing this URL.

This message is generated from Devin.ai (internal session link).

[Saurav Ganguli (sauravganguli)]

• Are you using Airbyte Cloud or Airbyte OSS/Self-Managed? (The credit-based controls would apply primarily to Cloud, while retry limits and data volume controls would be relevant to both.)

Airbyte Cloud

• For the retry behavior you observed (23 retries on Klaviyo → BigQuery), was this on the source extraction or destination loading side? You mentioned the load phase failed — understanding where the retry loop occurs would help narrow the scope.

Issue was with the destination loading. Because Airbyte was trying to load massive amounts of data to BigQuery, the BigQuery API was failing due to rate limits.

• Would a simpler initial implementation (e.g., just configurable max retry attempts per connection) address your most urgent need, or is the full circuit breaker suite critical for your use case?

Yes max retry attempts per connection would also work fine to start with but data volume circuit-breakers would also be an essential requirement going forward to prevent a connection initial sync from ingesting massive amounts of data.

[Saurav Ganguli (sauravganguli)]

Hi Aaron, Please see the replies below: Are you using Airbyte Cloud or Airbyte OSS/Self-Managed? (The credit-based controls would apply primarily to Cloud, while retry limits and data volume controls would be relevant to both.) Airbyte Cloud For the retry behavior you observed (23 retries on Klaviyo → BigQuery), was this on the source extraction or destination loading side? You mentioned the load phase failed — understanding where the retry loop occurs would help narrow the scope. Issue was with the destination loading. Because Airbyte was trying to load massive amounts of data to BigQuery, the BigQuery API was failing due to rate limits. Would a simpler initial implementation (e.g., just configurable max retry attempts per connection) address your most urgent need, or is the full circuit breaker suite critical for your use case? Yes max retry attempts per connection would also work fine to start with but data volume circuit-breakers would also be an essential requirement going forward to prevent a connection initial sync from ingesting massive amounts of data.

…

On 24. Feb 2026, at 11:01, aaronsteers ***@***.***> wrote: Hi Saurav Ganguli ***@***.***) <https://github.com/sauravganguli>, thank you for this detailed and well-structured feature request! The scenario you describe — a sync retrying 23 times with 100+ GB transfers each time — is a compelling illustration of why user-facing retry and cost controls are needed. The lack of configurable retry limits and circuit breakers is a real gap in the platform today. Your proposal covers three important areas: Configurable Retry Limits — Allowing users to override hardcoded connector retry behavior at the connection and workspace level Data Volume Circuit Breakers — Automatic sync termination based on data transfer thresholds Credit-Based Circuit Breakers — Cost-aware sync management with budgets and alerts The detailed UI/UX mockups are very helpful for understanding the intended user experience. I have escalated this to the internal team for evaluation: airbytehq/oncall#11418 <https://github.com/airbytehq/oncall/issues/11418> A few questions that would help the team prioritize and scope this: Are you using Airbyte Cloud or Airbyte OSS/Self-Managed? (The credit-based controls would apply primarily to Cloud, while retry limits and data volume controls would be relevant to both.) For the retry behavior you observed (23 retries on Klaviyo → BigQuery), was this on the source extraction or destination loading side? You mentioned the load phase failed — understanding where the retry loop occurs would help narrow the scope. Would a simpler initial implementation (e.g., just configurable max retry attempts per connection) address your most urgent need, or is the full circuit breaker suite critical for your use case? Need more help? Join Airbyte Community Slack <https://slack.airbyte.com/> for peer support, or if you're a Cloud customer, open a support ticket <https://support.airbyte.com/> referencing this URL. This message is generated from Devin.ai <http://devin.ai/> (internal session link <https://app.devin.ai/sessions/9c8ce8b306be40ef92f350cd1bf583b1>). — Reply to this email directly, view it on GitHub <#73996 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AJHNCZBVPGRYUNC3TOEOUOL4NQOOJAVCNFSM6AAAAACV5YJ3B2VHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTKOJQHAYTIOI>. You are receiving this because you were mentioned.