Skip to content

fix(spans): Enable better_backpressure in span buffer RunTask#113692

Merged
untitaker merged 1 commit intomasterfrom
untitaker/span-buffer-better-backpressure
Apr 22, 2026
Merged

fix(spans): Enable better_backpressure in span buffer RunTask#113692
untitaker merged 1 commit intomasterfrom
untitaker/span-buffer-better-backpressure

Conversation

@untitaker
Copy link
Copy Markdown
Member

Bumps arroyo to 2.39.0 and enables the new better_backpressure=True flag in the span buffer's RunTask.

When the flusher applies backpressure, the current RunTask implementation re-raises MessageRejected and re-runs the user function on each retry attempt. This causes increased CPU load on Redis because add-buffer.lua executes multiple times per message.

With better_backpressure=True, the function runs exactly once per message — transformed messages are stored on backpressure and retried in poll() instead.

See getsentry/arroyo#536 for the arroyo change.

Ref STREAM-881

@untitaker untitaker requested review from a team as code owners April 22, 2026 16:26
@linear-code
Copy link
Copy Markdown

linear-code Bot commented Apr 22, 2026

@github-actions github-actions Bot added the Scope: Backend Automatically applied to PRs that change backend components label Apr 22, 2026
),
next_step=flusher,
better_backpressure=True,
)
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug: The better_backpressure=True parameter is missing in a fallback path used only in contrived test scenarios where multiprocessing is disabled but multiple processes are requested.
Severity: LOW

Suggested Fix

In src/sentry/utils/arroyo.py, update the run_task_with_multiprocessing function to pass better_backpressure=True to the RunTask constructor within the fallback logic. This will ensure consistent behavior across all code paths, even those not used in production.

Prompt for AI Agent
Review the code at the location below. A potential bug has been identified by an AI
agent. Verify if this is a real issue. If it is, propose a fix; if not, explain why it's
not valid.

Location: src/sentry/spans/consumers/process/factory.py#L125

Potential issue: In a specific, non-production scenario, the `better_backpressure=True`
parameter is not applied to the stream processor. This occurs when the
`KAFKA_CONSUMER_FORCE_DISABLE_MULTIPROCESSING` setting is enabled (typically only for
tests) while simultaneously requesting more than one process (`num_processes > 1`). The
primary production path, which uses a single process, correctly receives the
`better_backpressure=True` parameter. The fallback path, handled by
`run_task_with_multiprocessing()` in `src/sentry/utils/arroyo.py`, was not updated to
include this parameter, leading to the inconsistency. However, this combination of
settings is not used in production or existing tests.

Also affects:

  • src/sentry/utils/arroyo.py

Did we get this right? 👍 / 👎 to inform future reviews.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it's legitimate but not worth fixing. i just want to see this rolled out in prod

@untitaker untitaker merged commit 9144b39 into master Apr 22, 2026
80 checks passed
@untitaker untitaker deleted the untitaker/span-buffer-better-backpressure branch April 22, 2026 18:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Scope: Backend Automatically applied to PRs that change backend components

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants