-
Notifications
You must be signed in to change notification settings - Fork 697
Description
Problem Statement
The TypeScript Strands SDK exposes hooks into model streamed chunk events, but the Python SDK has no equivalent. There is currently no way to observe or act on individual chunks during model streaming, and no mechanism to cancel an in-flight model call mid-stream.
The Python SDK's hook system provides BeforeModelCallEvent and AfterModelCallEvent, but nothing fires between them while the stream is active. Once stream_messages() starts, the only way to stop it is to cancel the entire asyncio task, which is is the approach taken to resolve #81.
Proposed Solution
Add mid-stream chunk hooks to the Python SDK (matching TypeScript SDK parity), with the ability to signal cancellation from within the hook. For example:
@dataclass
class ModelStreamChunkHookEvent(HookEvent):
chunk: StreamEvent
cancel: bool = False # Set to True to stop the stream
When cancel is set to True, the stream should:
- Stop consuming further chunks
- Clean up server-side resources (close the HTTP connection)
- Fall through to AfterModelCallEvent where hooks can set retry = True
Use Case
We need to cancel a specific in-flight model call based on an external signal (e.g., a background validation check that determines the current call is invalid) and then retry via AfterModelCallEvent.retry = True. This
requires:
- A way to observe or intercept the stream mid-flight
- A way to signal cancellation that cleanly stops the stream
- Execution continuing to AfterModelCallEvent so hooks can decide whether to retry
This is distinct from agent-level cancellation (#81, #1772), which kills the entire agent invocation. We need per-model-call granularity where the event loop continues running after cancellation.
Alternatives Solutions
A prototype implementation using AbortSignal + CancellableStream was explored but mid-stream hooks are a more general solution that also enables other use cases (logging, metrics, content filtering)
Additional Context
No response