How to handle flushing buffer behavior in Airbyte #69216
Unanswered
Vladislav Bakanov (vladislav-bakanov)
asked this question in
Platform General Questions
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
Environment:
Source connector: Slack (3.1.6)
Destination: S3
Context:
I have a Slack connector, where I see that it makes flushing the buffer every 5 min, because my batches in the source are quite small (few kb).
I guess the mechanic of processing batches in Airbyte quite similar to what Kafka Connect, where you wait for duration to flush or you wait until the required volume of the buffer is reached and you can manage such behavior of buffer.
Question:
Is't possible somehow to manage buffer size / flush interval in Airbyte?
Beta Was this translation helpful? Give feedback.
All reactions