OOM errors after upgrading to Airbyte v2.0.1

[Ian Alton (ian-at-airbyte)]

We've received a few reports of OOM errors from people who upgraded to Airbyte 2.0.1, using both abctl and Helm. If this applies to you, please participate in this discussion. It would be helpful if you could share:

• Your values.yaml file (sanitized of PII/secret info)
• The source/destination pair that is resulting in OOM.
• Any relevant logs. This include sync logs, but also pod logs if you have them.

Having these things might allow us to isolate what's happening.

[FabrizioSpini-lastminute]

Hi, we collected a log for an OOM we just received on our airbyte 2.0.1 deployed on gke

this is our values.yaml

postgresql:
enabled: false

global:
edition: community
version: "2.0.1"
airbyteUrl: "http://airbyte-server-svc:8001"

auth:
enabled: false

network:
name: kind

database:
type: external
host: xxx
port: "5432"
name: postgres
secretName: "airbyte-database-secrets"
userSecretKey: "CloudSQLuserSecretKey"
passwordSecretKey: "CloudSQLpasswordSecretKey"

envVars:
DATABASE_URL: "jdbc:postgresql://xxx:5432/postgres?sslmode=disable"
server:
configDatabase:
hikariMaximumPoolSize: 5
hikariMinimumIdle: 1

The connection is a MySQL v3.51.5 --> BigQuery v3.0.16

Here under a log sample

2026-01-29 14:33:48,718 [pool-3-thread-1] INFO i.a.w.i.VersionedAirbyteStreamFactory(internalLog$io_airbyte_airbyte_commons_worker):248 - INFO xxx:3306 i.d.c.b.BinlogStreamingChangeEventSource(handleChange):1087 Skipping previously processed row event: Event{header=EventHeaderV4{timestamp=1769684522000, eventType=EXT_DELETE_ROWS, serverId=1234567890, headerLength=19, dataLength=8142, nextPosition=78900563, flags=0}, data=DeleteRowsEventData{tableId=1508695, includedColumns={0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36}, rows=[[B…
2026-01-29 14:33:48,964 [pool-3-thread-1] INFO i.a.w.i.VersionedAirbyteStreamFactory(internalLog$io_airbyte_airbyte_commons_worker):248 - INFO debezium-engine i.d.c.c.BaseSourceTask(logStatistics):354 1 records sent during previous 00:00:10.034, last recorded offset of {server=finance_automation} partition is {file=dbabc-slave.147516, pos=8714595, server_id=1234567890, event=9513}
[dd.trace 2026-01-29 14:34:08:457 +0000] [OkHttp http://localhost:8126/...] WARN com.datadog.profiling.uploader.ProfileUploader - Failed to upload profile to http://localhost:8126/profiling/v1/input java.net.ConnectException: Failed to connect to localhost/127.0.0.1:8126 (Will not log warnings for 5 minutes)
2026-01-29 14:34:09,005 [pool-3-thread-1] INFO i.a.w.i.VersionedAirbyteStreamFactory(internalLog$io_airbyte_airbyte_commons_worker):248 - INFO debezium-engine i.d.c.c.BaseSourceTask(logStatistics):354 2 records sent during previous 00:00:20.041, last recorded offset of {server=finance_automation} partition is {ts_sec=1769684636, file=dbabc-slave.147517, pos=2245992, server_id=1234567890, event=57976}
2026-01-29 14:34:11,215 [pool-3-thread-3] INFO i.a.w.i.VersionedAirbyteStreamFactory(internalLog$io_airbyte_airbyte_commons_worker):248 - INFO DefaultDispatcher-worker-6 i.a.c.l.s.PipelineEventBookkeepingRouter(handleStreamMessage):101 Saw first record for stream Descriptor(namespace=raw_finance_automation, name=BOOKING_FAMILY_TREE); awaiting setup complete
2026-01-29 14:34:11,216 [pool-3-thread-3] INFO i.a.w.i.VersionedAirbyteStreamFactory(internalLog$io_airbyte_airbyte_commons_worker):248 - INFO DefaultDispatcher-worker-6 i.a.c.l.s.PipelineEventBookkeepingRouter(handleStreamMessage):105 Setup complete, starting stream Descriptor(namespace=raw_finance_automation, name=BOOKING_FAMILY_TREE)
2026-01-29 14:34:11,228 [pool-3-thread-3] INFO i.a.w.i.VersionedAirbyteStreamFactory(internalLog$io_airbyte_airbyte_commons_worker):248 - INFO DefaultDispatcher-worker-9 i.a.c.l.o.d.d.DirectLoadTableDedupStreamLoader(start):91 DedupStreamLoader starting for stream: Descriptor(namespace=raw_finance_automation, name=BOOKING_FAMILY_TREE)
2026-01-29 14:34:11,236 [pool-3-thread-3] INFO i.a.w.i.VersionedAirbyteStreamFactory(internalLog$io_airbyte_airbyte_commons_worker):248 - INFO DefaultDispatcher-worker-9 i.a.c.l.o.d.d.DirectLoadTableDedupStreamLoader(start):96 Creating new temp table: airbyte_internal.raw_finatomationBOOKING_ILY_TREE26ccf60311ddc555b53b7e22f1c209b6 for stream: Descriptor(namespace=raw_finance_automation, name=BOOKING_FAMILY_TREE)
2026-01-29 14:34:13,459 [pool-3-thread-3] INFO i.a.w.i.VersionedAirbyteStreamFactory(internalLog$io_airbyte_airbyte_commons_worker):248 - INFO DefaultDispatcher-worker-6 i.a.c.l.s.PipelineEventBookkeepingRouter(handleStreamMessage):108 Initialization for stream Descriptor(namespace=raw_finance_automation, name=BOOKING_FAMILY_TREE) complete
(base) ➜ ~ kubectl -n airbyte logs replication-job-1117-attempt-0 -c orchestrator --follow
2026-01-29 14:51:52,619 [pool-3-thread-1] INFO i.a.w.i.VersionedAirbyteStreamFactory(internalLog$io_airbyte_airbyte_commons_worker):248 - INFO debezium-engine i.d.c.c.BaseSourceTask(logStatistics):354 1 records sent during previous 00:00:10.06, last recorded offset of {server=dataware_accounting} partition is {ts_sec=1769684598, file=dbabc-slave.147516, pos=8714595, server_id=1234567890, event=24589}
[dd.trace 2026-01-29 14:52:10:676 +0000] [OkHttp http://localhost:8126/...] WARN com.datadog.profiling.uploader.ProfileUploader - Failed to upload profile to http://localhost:8126/profiling/v1/input java.net.ConnectException: Failed to connect to localhost/127.0.0.1:8126 (Will not log warnings for 5 minutes)
2026-01-29 14:52:10,907 [pool-3-thread-1] INFO i.a.w.i.VersionedAirbyteStreamFactory(internalLog$io_airbyte_airbyte_commons_worker):248 - INFO blc-dbabc.bravofly.intra:3306 i.d.r.h.SchemaHistoryMetrics(onChangeApplied):140 Already applied 10987 database changes
2026-01-29 14:52:11,414 [pool-3-thread-1] INFO i.a.w.i.VersionedAirbyteStreamFactory(internalLog$io_airbyte_airbyte_commons_worker):248 - INFO blc-dbabc.bravofly.intra:3306 i.d.r.h.SchemaHistoryMetrics(onChangeApplied):140 Already applied 10995 database changes
2026-01-29 14:52:12,662 [pool-3-thread-1] INFO i.a.w.i.VersionedAirbyteStreamFactory(internalLog$io_airbyte_airbyte_commons_worker):248 - INFO debezium-engine i.d.c.c.BaseSourceTask(logStatistics):354 2 records sent during previous 00:00:20.043, last recorded offset of {server=dataware_accounting} partition is {ts_sec=1769684964, file=dbabc-slave.147518, pos=37109961, server_id=1234567890, event=2343}
2026-01-29 14:52:21,365 [pool-3-thread-1] INFO i.a.w.i.VersionedAirbyteStreamFactory(internalLog$io_airbyte_airbyte_commons_worker):248 - INFO blc-dbabc.bravofly.intra:3306 i.d.r.h.SchemaHistoryMetrics(onChangeApplied):140 Already applied 10997 database changes
2026-01-29 14:52:25,603 [pool-3-thread-1] INFO i.a.w.i.VersionedAirbyteStreamFactory(internalLog$io_airbyte_airbyte_commons_worker):248 - INFO blc-dbabc.bravofly.intra:3306 i.d.r.h.SchemaHistoryMetrics(onChangeApplied):140 Already applied 10999 database changes

java.lang.OutOfMemoryError: Java heap space

-XX:OnOutOfMemoryError="/tmp/ddprof_airbyte/pid_7/dd_oome_notifier.sh %p"

Executing /bin/sh -c "/tmp/ddprof_airbyte/pid_7/dd_oome_notifier.sh 7"...

Agent Jar: /app/dd-java-agent.jar
Tags: runtime-id:f74fe8fc-e1f6-4f33-b3d7-e7db1082038b,service:airbyte-container-orchestrator
JAVA_HOME: /usr/lib/jvm/java-21-amazon-corretto
PID: 7
[dd.trace 2026-01-29 14:52:27:746 +0000] [main] INFO datadog.crashtracking.OOMENotifier - OOME event sent
OOME Event generated successfully
Terminating due to java.lang.OutOfMemoryError: Java heap space

`

let me know if this is sufficient or you want some other evidence

[FabrizioSpini-lastminute]

orchestrator_git.log
adding another example log from our airbyte infrastructure

I add that I've tried to apply the resources override in my values.yaml using both the global.jobs.resources path (as you suggested) and the global.workloads.resources path (which seems more specific to Chart V2).
However, there seems to be a hardcoded default or a precedence issue in abctl (v0.30.0+). Even after a clean abctl local install --values values.yaml and a rollout restart of all deployments, the container-orchestrator is still spawned with: limits={memory=1024Mi, cpu=1}, requests={memory=1024Mi, cpu=1}
I can confirm that the new values are correctly "landed" into the workload-launcher and worker deployments (I see them in the env vars), but the Workload API seems to ignore them when actually generating the Pod manifest for the replication job.
I've also tried to 'Save' and 'Refresh' the connection in the UI to force a metadata update, but the 1024Mi limit persists. It really looks like the V2 architecture in abctl is not respecting these Helm overrides for the orchestrator pods.
It seems like a core issue in how abctl/Chart V2 handles job templates.