fix(caching): store task references in LLMClientCache._remove_key

[shivaaang]

Summary

LLMClientCache._remove_key calls create_task(close_fn()) when evicting async HTTP clients, but discards the task reference. Python's GC destroys the unreferenced task before it runs, producing RuntimeWarning: coroutine 'AsyncHTTPHandler.close' was never awaited on every cache eviction.

Store task references in a class-level _background_tasks set and use add_done_callback to clean up completed tasks, following the recommended pattern from the asyncio docs.

Fixes #22128

Type

🐛 Bug Fix

Testing

Added 6 tests in tests/test_litellm/caching/test_llm_caching_handler.py:

• No unawaited coroutine warnings on async client eviction
• Async client .close() is actually called
• Sync client .close() is called
• Eviction via set_cache overflow closes async clients without warnings
• _remove_key doesn't raise when no event loop is running
• Background tasks are cleaned up after completion

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
litellm	Ready	Preview, Comment	Feb 28, 2026 2:25am

[CLAassistant]

All committers have signed the CLA.

[shivaaang]

@greptileai

[greptile-apps]

Greptile Summary

Fixes a bug where LLMClientCache._remove_key created asyncio tasks for closing async HTTP clients but discarded the task references, allowing Python's GC to destroy them before execution — producing RuntimeWarning: coroutine 'AsyncHTTPHandler.close' was never awaited on every cache eviction.

• Adds a class-level _background_tasks: Set[asyncio.Task] to hold strong references to close tasks, following the recommended asyncio pattern
• Uses task.add_done_callback(self._background_tasks.discard) for automatic cleanup of completed tasks
• Adds 6 mock-only unit tests verifying: no unawaited coroutine warnings, async/sync client close behavior, eviction-triggered close, no-event-loop safety, and background task cleanup

Confidence Score: 5/5

• This PR is safe to merge — it's a minimal, well-scoped bug fix following a well-documented asyncio pattern.
• The change is small (3 new lines of logic), follows the official asyncio documentation pattern exactly, doesn't alter any existing behavior or API surface, and is backed by 6 comprehensive mock-only unit tests. No custom rules are violated.
• No files require special attention.

Important Files Changed

Filename	Overview
litellm/caching/llm_caching_handler.py	Stores asyncio task references in a class-level _background_tasks set with add_done_callback cleanup, following the recommended asyncio pattern to prevent GC from destroying unawaited coroutines. Clean, minimal fix.
tests/test_litellm/caching/test_llm_caching_handler.py	Adds 6 mock-only tests covering async/sync client close, eviction, no-event-loop safety, and background task cleanup. No network calls — compliant with mock-only test requirement.

Sequence Diagram

sequenceDiagram
    participant Caller
    participant LLMClientCache
    participant EventLoop
    participant _background_tasks as _background_tasks Set
    participant AsyncClient

    Caller->>LLMClientCache: _remove_key(key)
    LLMClientCache->>LLMClientCache: value = cache_dict.get(key)
    LLMClientCache->>LLMClientCache: super()._remove_key(key)
    alt value has async close_fn
        LLMClientCache->>EventLoop: create_task(close_fn())
        EventLoop-->>LLMClientCache: task
        LLMClientCache->>_background_tasks: add(task)
        LLMClientCache->>EventLoop: task.add_done_callback(discard)
        EventLoop->>AsyncClient: await close()
        AsyncClient-->>EventLoop: done
        EventLoop->>_background_tasks: discard(task)
    else value has sync close_fn
        LLMClientCache->>AsyncClient: close()
    end

Loading

_{Last reviewed commit: ba849c1}

[greptile-apps]

_{2 files reviewed, 1 comment}

_{Edit Code Review Agent Settings | Greptile}

[greptile-apps]

Class-level mutable shared state

_background_tasks is a class variable, meaning it's shared across all LLMClientCache instances. While this works fine in production (where there's typically a single global instance), it could cause subtle issues if multiple instances are created — e.g., flush_cache() on one instance won't clear tasks created by another. Consider making this an instance variable initialized in __init__ instead:

Suggested change

	_background_tasks: Set[asyncio.Task] = set()
	_background_tasks: Set[asyncio.Task]
	def __init__(self, *args, **kwargs):
	super().__init__(*args, **kwargs)
	self._background_tasks: Set[asyncio.Task] = set()

[shivaaang]

Thanks for the suggestion! The shared task registry is intentional — create_task() needs a strong reference to prevent GC from triggering "coroutine was never awaited" warnings (as recommended in the asyncio docs). I added a comment to make the class-level scope explicit.

I avoided adding an __init__ override to keep this fix minimal, since LLMClientCache currently relies on the base class init path. If maintainers would prefer instance-level tracking or a module-level registry instead, I'm happy to adjust.

[shivaaang]

@greptileai

[shin-bot-litellm]

🚅 Shin's PR Review

1. Does this PR fix the issue it describes?
✅ Yes — the PR addresses the RuntimeWarning: coroutine was never awaited issue by properly storing task references in a class-level _background_tasks set with cleanup via add_done_callback. This follows the recommended asyncio pattern.

2. Has this issue already been solved elsewhere?
No — checked the codebase and this is the correct location for the fix. The LLMClientCache._remove_key method was the source of the unreferenced tasks.

3. Are there other PRs addressing the same problem?
No duplicate PRs found targeting issue #22128.

4. Other issues this could close?
Potentially related to any other reports of "coroutine was never awaited" warnings during cache eviction, though none specifically identified.

Summary: Well-structured fix with 6 comprehensive tests covering sync/async clients, eviction scenarios, and cleanup verification. The implementation follows Python's official asyncio documentation. LGTM. 🚅

[jquinter]

The CI lint failure (F401 MCPAuth imported but unused in litellm/proxy/_types.py) is not related to this PR's changes. It was already fixed on main by commit 552d9aa. Please rebase onto latest main to pick up the fix:

git rebase main
git push --force-with-lease

[jquinter]

The MCP e2e test failures (test_proxy_mcp_lists_all_servers_without_header, test_proxy_mcp_streamable_http_roundtrip) are also not related to this PR — they are pre-existing failures on main caused by stale test mocks referencing a renamed method. Fix PR: #22327

[jquinter]

The passthrough test failures (test_pass_through_request_stream_param_no_override, test_pass_through_request_stream_param_override) are also not related to this PR. The MockRequest is missing a state attribute, but this was already fixed on main in commit e7175a5 which changed _safe_get_request_headers to use safe getattr(request, "state", None). Rebasing onto main will resolve this along with the lint error.

[jquinter]

All 12 test failures (health check max_concurrency, form data parsing, spend endpoint, model output structure) are not related to this PR. They all pass on current main. Your branch is behind main and needs a rebase to pick up the fixes:

git fetch origin main
git rebase origin/main
git push --force-with-lease

This single rebase will resolve all CI failures: the lint error, MCP tests, passthrough tests, and these 12 failures.

[jquinter]

The 11 failures (guardrail endpoints, key management) also pass on current main. Same root cause — branch needs rebasing. All CI failures on this PR are due to the branch being behind main, not due to the caching changes.

[jquinter]

The UI build failure (merge conflict markers in ToolPolicies.tsx) is also not related to this PR. Already fixed on main by commit 5a123f0. Same fix: rebase onto main.