Airbyte CE 2.x Helm Chart: ConnectorRollout Bean Initialization Failure with Temporal Cloud

[gregorycox]

[Bug] Airbyte CE 2.x Helm Chart: ConnectorRollout Bean Initialization Failure with Temporal Cloud

Description

When deploying Airbyte Community Edition 2.x (Helm chart v2.0.0) with Temporal Cloud integration, the application fails to start due to a ConnectorRolloutApiController bean initialization error. This occurs despite:

• All required Temporal Cloud environment variables being properly configured
• The CONNECTOR_ROLLOUT_ENABLED flag being set to false
• The ConnectorRollout worker being disabled

The issue appears to be a fundamental incompatibility between Airbyte CE 2.x and Temporal Cloud, as the ConnectorRollout feature's beans attempt to initialize during application startup before runtime configuration can prevent them.

Environment

• Helm Chart Version: 2.0.0
• App Version: 1.6.0
• Airbyte Edition: Community Edition
• Temporal: Temporal Cloud (managed service)
• Kubernetes Version: 1.30+
• Deployment Method: Helm via ArgoCD (GitOps)

Expected Behavior

When CONNECTOR_ROLLOUT_ENABLED=false and connectorRolloutWorker.enabled=false, the Airbyte server should:

1. Skip initialization of ConnectorRollout-related beans
2. Successfully start with Temporal Cloud connection
3. Function normally without ConnectorRollout features

Actual Behavior

The application crashes during startup with:

ERROR i.m.r.Micronaut(handleStartupException):349 - Error starting Micronaut server:
Bean definition [io.airbyte.server.apis.controllers.ConnectorRolloutApiController] could not be loaded:
Error instantiating bean of type [io.airbyte.connector.rollout.client.ConnectorRolloutClient]

Message: Error instantiating bean of type [io.airbyte.connector.rollout.client.ConnectorRolloutClient]
Path Taken:
ConnectorRolloutApiController.connectorRolloutHandler -->
ConnectorRolloutHandler.connectorRolloutClient -->
new ConnectorRolloutClient([ConnectorRolloutTemporalWorkflowService connectorRolloutTemporalWorkflowService])
io.micronaut.context.exceptions.BeanInstantiationException: Error instantiating bean of type [io.airbyte.connector.rollout.client.ConnectorRolloutClient]

Pod status:

NAME                                   READY   STATUS             RESTARTS
airbyte-ce-dev-server-755fdfd78-xxxxx  0/1     CrashLoopBackOff   2+

Steps to Reproduce

1. Helm Values Configuration

global:
  serviceAccountName: "airbyte-sa"
  edition: "community"
  airbyteUrl: http://airbyte.example.com

  # External PostgreSQL database
  database:
    type: external
    secretName: "airbyte-db-connection"
    host: "10.x.x.x"
    port: "5432"
    database: "airbyte"
    name: "airbyte"
    userSecretKey: "DATABASE_USER"
    passwordSecretKey: "DATABASE_PASSWORD"

  # GCS storage
  storage:
    type: gcs
    bucket:
      activityPayload: storage-bucket
      log: storage-bucket
      state: storage-bucket
      workloadOutput: storage-bucket
    secretName: "gcs-credentials"
    gcs:
      projectId: project-id

  # Temporal Cloud configuration
  temporal:
    cloud:
      enabled: true
      host: "namespace.tmprl.cloud:7233"
      namespace: "namespace-id"
      clientCert: "temporal-certs"
      clientCertSecretKey: "TEMPORAL_CLOUD_CLIENT_CERT"
      clientKey: "temporal-certs"
      clientKeySecretKey: "TEMPORAL_CLOUD_CLIENT_KEY"

    cli:
      address: "namespace.tmprl.cloud:7233"
      namespace: "namespace-id"
      tlsCert: "temporal-certs"
      tlsCertSecretKey: "TEMPORAL_CLOUD_CLIENT_CERT"
      tlsKey: "temporal-certs"
      tlsKeySecretKey: "TEMPORAL_CLOUD_CLIENT_KEY"

airbyte:
  # Disable ConnectorRollout worker
  connectorRolloutWorker:
    enabled: false

  # Disable internal PostgreSQL
  postgresql:
    enabled: false

  # Disable self-hosted Temporal
  temporal:
    enabled: false

  # Server configuration
  server:
    extraEnv:
      # Disable ConnectorRollout feature
      - name: CONNECTOR_ROLLOUT_ENABLED
        value: "false"

      # Manual Temporal Cloud configuration injection
      # (Required because chart doesn't automatically inject these)
      - name: TEMPORAL_CLOUD_ENABLED
        valueFrom:
          configMapKeyRef:
            name: airbyte-ce-dev-airbyte-env
            key: TEMPORAL_CLOUD_ENABLED

      - name: TEMPORAL_CLOUD_HOST
        valueFrom:
          configMapKeyRef:
            name: airbyte-ce-dev-airbyte-env
            key: TEMPORAL_CLOUD_HOST

      - name: TEMPORAL_CLOUD_NAMESPACE
        valueFrom:
          configMapKeyRef:
            name: airbyte-ce-dev-airbyte-env
            key: TEMPORAL_CLOUD_NAMESPACE

      # Temporal Cloud certificate credentials
      - name: TEMPORAL_CLOUD_CLIENT_CERT
        valueFrom:
          secretKeyRef:
            name: temporal-certs
            key: TEMPORAL_CLOUD_CLIENT_CERT

      - name: TEMPORAL_CLOUD_CLIENT_KEY
        valueFrom:
          secretKeyRef:
            name: temporal-certs
            key: TEMPORAL_CLOUD_CLIENT_KEY

2. Create Temporal Cloud Certificate Secret

kubectl create secret generic temporal-certs \
  --namespace=airbyte \
  --from-file=TEMPORAL_CLOUD_CLIENT_CERT=./client.pem \
  --from-file=TEMPORAL_CLOUD_CLIENT_KEY=./client-key.pem

Certificate format: PEM-encoded ECDSA P-384 certificate chain with root and client certificates.

3. Deploy Helm Chart

helm repo add airbyte https://airbytehq.github.io/charts
helm repo update
helm upgrade --install airbyte-ce airbyte/airbyte \
  --version 2.0.0 \
  --namespace airbyte \
  --create-namespace \
  --values values.yaml

4. Observe Failure

kubectl get pods -n airbyte
# Pod shows CrashLoopBackOff

kubectl logs -n airbyte <server-pod-name>
# Shows ConnectorRolloutApiController bean initialization error

Root Cause Analysis

1. Bean Initialization Timing

The ConnectorRolloutApiController is a Micronaut bean that attempts to initialize during application startup, before any runtime configuration can be evaluated. The dependency chain is:

ConnectorRolloutApiController
  → ConnectorRolloutHandler
    → ConnectorRolloutClient
      → ConnectorRolloutTemporalWorkflowServiceFactory
        → [FAILS HERE]

2. Feature Flag Limitation

The CONNECTOR_ROLLOUT_ENABLED environment variable only controls whether the feature runs, not whether the bean is created. Bean creation happens during Micronaut's initialization phase, before the flag can be evaluated.

3. Missing Conditional Bean Creation

The ConnectorRolloutApiController bean lacks @Requires annotations that would prevent its creation based on:

• Edition (Community vs Enterprise)
• Temporal configuration (self-hosted vs Cloud)
• Feature flags

4. Helm Chart Environment Variable Injection Gap

Finding: The Helm chart generates a ConfigMap (airbyte-env) with Temporal Cloud configuration but doesn't automatically inject these as environment variables:

# ConfigMap generated by chart (verified)
data:
  TEMPORAL_CLOUD_ENABLED: "true"
  TEMPORAL_CLOUD_HOST: "namespace.tmprl.cloud:7233"
  TEMPORAL_CLOUD_NAMESPACE: "namespace-id"

However, the deployment spec doesn't reference these ConfigMap keys, requiring manual injection via extraEnv.

Verification

I verified all required environment variables are present in the pod:

kubectl exec -n airbyte <server-pod> -- env | grep TEMPORAL_CLOUD

TEMPORAL_CLOUD_CLIENT_CERT=-----BEGIN CERTIFICATE-----...
TEMPORAL_CLOUD_CLIENT_KEY=-----BEGIN PRIVATE KEY-----...
TEMPORAL_CLOUD_ENABLED=true
TEMPORAL_CLOUD_HOST=namespace.tmprl.cloud:7233
TEMPORAL_CLOUD_NAMESPACE=namespace-id

Despite all configuration being correct, the application still fails at bean initialization.

Attempted Workarounds

✅ Workaround 1: Disable ConnectorRollout Worker

airbyte:
  connectorRolloutWorker:
    enabled: false

Result: Worker successfully disabled, but server pod still crashes.

✅ Workaround 2: Set Feature Flag

server:
  extraEnv:
    - name: CONNECTOR_ROLLOUT_ENABLED
      value: "false"

Result: Flag set correctly, but bean initialization occurs before flag evaluation.

✅ Workaround 3: Manual Environment Variable Injection

server:
  extraEnv:
    - name: TEMPORAL_CLOUD_ENABLED
      valueFrom:
        configMapKeyRef:
          name: airbyte-ce-dev-airbyte-env
          key: TEMPORAL_CLOUD_ENABLED
    # ... etc

Result: All environment variables confirmed present, but initialization still fails.

Proposed Solutions

Solution 1: Add Conditional Bean Creation (Recommended)

Add @Requires annotation to prevent bean creation in Community Edition:

@Controller("/v1/connector_rollout")
@Requires(property = "airbyte.edition", value = "enterprise")
// OR
@Requires(property = "connector.rollout.enabled", value = "true", defaultValue = "false")
public class ConnectorRolloutApiController {
    // ...
}

This would prevent the bean from being created at all when the feature is not supported.

Solution 2: Fix Helm Chart Environment Variable Injection

Automatically inject Temporal Cloud configuration from ConfigMap to deployment spec:

# In deployment.yaml template
env:
  {{- if .Values.global.temporal.cloud.enabled }}
  - name: TEMPORAL_CLOUD_ENABLED
    valueFrom:
      configMapKeyRef:
        name: {{ include "airbyte.fullname" . }}-env
        key: TEMPORAL_CLOUD_ENABLED
  - name: TEMPORAL_CLOUD_HOST
    valueFrom:
      configMapKeyRef:
        name: {{ include "airbyte.fullname" . }}-env
        key: TEMPORAL_CLOUD_HOST
  - name: TEMPORAL_CLOUD_NAMESPACE
    valueFrom:
      configMapKeyRef:
        name: {{ include "airbyte.fullname" . }}-env
        key: TEMPORAL_CLOUD_NAMESPACE
  {{- end }}

Currently, users must manually inject these via extraEnv.

Solution 3: Document Temporal Cloud Support in CE

If Temporal Cloud is not intended to be supported in Community Edition, please:

1. Document this limitation in the Helm chart README
2. Add validation to fail fast with a clear error message
3. Provide migration path for users wanting Temporal Cloud support

Questions for Maintainers

1. Is Temporal Cloud intended to be supported in Community Edition 2.x?

   • If yes, this is a bug that needs fixing
   • If no, should there be better documentation/validation?

2. Is ConnectorRollout feature Enterprise-only?

   • If yes, beans should use @Requires for conditional creation
   • Current behavior suggests it's trying to initialize in CE

3. What is the recommended Temporal configuration for CE 2.x?

   • Self-hosted Temporal only?
   • Temporal Cloud supported?
   • Neither (workflows disabled in CE)?

Impact

This issue prevents users from deploying Airbyte Community Edition 2.x with Temporal Cloud, requiring either:

• Switching to Enterprise Edition (may not be viable for all users)
• Running self-hosted Temporal (increased operational complexity)
• Staying on pre-2.0 versions (if available)

Additional Context

Related Issues

• Similar issue may affect other users attempting CE 2.x with Temporal Cloud
• May be related to the architectural changes in 2.x series

Environment Variables Present at Startup

All required Temporal Cloud environment variables are confirmed present:

Variable	Source	Status
TEMPORAL_CLOUD_ENABLED	ConfigMap	✅ Present
TEMPORAL_CLOUD_HOST	ConfigMap	✅ Present
TEMPORAL_CLOUD_NAMESPACE	ConfigMap	✅ Present
TEMPORAL_CLOUD_CLIENT_CERT	Secret	✅ Present
TEMPORAL_CLOUD_CLIENT_KEY	Secret	✅ Present
CONNECTOR_ROLLOUT_ENABLED	Direct value	✅ Present (false)

Full Error Stack Trace

2025-10-29 18:45:32 ERROR i.m.r.Micronaut(handleStartupException):349 - Error starting Micronaut server: Bean definition [io.airbyte.server.apis.controllers.ConnectorRolloutApiController] could not be loaded: Error instantiating bean of type [io.airbyte.connector.rollout.client.ConnectorRolloutClient]
io.micronaut.context.exceptions.BeanInstantiationException: Error instantiating bean of type [io.airbyte.connector.rollout.client.ConnectorRolloutClient]

Message: Error instantiating bean of type [io.airbyte.connector.rollout.client.ConnectorRolloutClient]
Path Taken: ConnectorRolloutApiController.connectorRolloutHandler --> ConnectorRolloutHandler.connectorRolloutClient --> new ConnectorRolloutClient([ConnectorRolloutTemporalWorkflowService connectorRolloutTemporalWorkflowService])
	at io.micronaut.context.DefaultBeanContext.initializeContext(DefaultBeanContext.java:1894)
	at io.micronaut.context.DefaultApplicationContext.initializeContext(DefaultApplicationContext.java:265)
	at io.micronaut.context.DefaultBeanContext.readAllBeanDefinitionClasses(DefaultBeanContext.java:3234)
	at io.micronaut.context.DefaultBeanContext.start(DefaultBeanContext.java:332)
	at io.micronaut.context.DefaultApplicationContext.start(DefaultApplicationContext.java:192)
	at io.micronaut.runtime.Micronaut.start(Micronaut.java:74)
	at io.micronaut.runtime.Micronaut.run(Micronaut.java:330)
	at io.micronaut.runtime.Micronaut.run(Micronaut.java:306)
	at io.airbyte.server.Application.main(Application.java:22)
Caused by: io.micronaut.context.exceptions.BeanInstantiationException: Error instantiating bean of type [io.airbyte.connector.rollout.client.ConnectorRolloutClient]

Message: Error instantiating bean of type [io.airbyte.connector.rollout.client.ConnectorRolloutClient]
Path Taken: ConnectorRolloutApiController.connectorRolloutHandler --> ConnectorRolloutHandler.connectorRolloutClient --> new ConnectorRolloutClient([ConnectorRolloutTemporalWorkflowService connectorRolloutTemporalWorkflowService])
	at io.micronaut.context.DefaultBeanContext.doCreateBean(DefaultBeanContext.java:2507)
	at io.micronaut.context.DefaultBeanContext.createAndRegisterSingleton(DefaultBeanContext.java:3324)
	at io.micronaut.context.DefaultBeanContext.getBeanForDefinition(DefaultBeanContext.java:2747)
	at io.micronaut.context.DefaultBeanContext.getBeanInternal(DefaultBeanContext.java:2718)
	at io.micronaut.context.DefaultBeanContext.getBean(DefaultBeanContext.java:1062)
	at io.micronaut.context.AbstractBeanDefinition.getBeanForConstructorArgument(AbstractBeanDefinition.java:1333)
	at io.airbyte.connector.rollout.worker.$ConnectorRolloutHandlerDefinition.build(Unknown Source)
	at io.micronaut.context.DefaultBeanContext.doCreateBean(DefaultBeanContext.java:2442)
	... (stack continues)
Caused by: java.lang.NullPointerException: Cannot invoke "io.temporal.client.WorkflowClient.newWorkflowStub(Class)" because the return value of "io.airbyte.connector.rollout.client.ConnectorRolloutTemporalWorkflowServiceFactory.createTemporalWorkflowServiceLazily()" is null
	at io.airbyte.connector.rollout.client.ConnectorRolloutClient.<init>(ConnectorRolloutClient.java:29)
	at io.airbyte.connector.rollout.client.$ConnectorRolloutClientDefinition.build(Unknown Source)
	at io.micronaut.context.DefaultBeanContext.doCreateBean(DefaultBeanContext.java:2442)
	... (stack continues)

The root cause is the ConnectorRolloutTemporalWorkflowServiceFactory.createTemporalWorkflowServiceLazily() returning null, which suggests the Temporal CLI configuration expected by ConnectorRollout is incompatible with Temporal Cloud in CE.

---

Chart Repository: https://github.com/airbytehq/helm-charts
Chart Version: 2.0.0
App Version: 1.6.0

Thank you for maintaining this project! Please let me know if you need any additional information or logs.