PostgreSQL connections silently dropped by RDS due to missing default idle connection lifetime config

[nikogio]

Description

When deploying LiteLLM (v1.81.9) on Kubernetes with an AWS RDS PostgreSQL instance, intermittent connection errors occur:

{"level":"ERROR","fields":{"message":"Error in PostgreSQL connection: Error { kind: Closed, cause: None }"},"target":"quaint::connector::postgres"}

This manifests as failed API requests that succeed on retry (typically after few attempts), causing visible latency spikes (P95/P99) and ~50% server error rates under moderate load.

Root Cause

RDS silently drops idle TCP connections after a period of inactivity. Quaint (Prisma's database driver) holds connections in its pool without validating them, and hands a dead connection to the next incoming request. The request fails with kind: Closed, retries on a fresh connection, and succeeds.

LiteLLM does not set a default value for max_idle_connection_lifetime in the DATABASE_URL, meaning connections can sit idle indefinitely and be silently dropped by RDS before LiteLLM recycles them.

Solution

Adding the following parameters to the DATABASE_URL resolves the issue completely:

postgresql://user:password@host:5432/dbname?max_idle_connection_lifetime=60&socket_timeout=10

• max_idle_connection_lifetime=60 — quaint proactively closes connections idle for 60 seconds, before RDS drops them
• socket_timeout=10 — if a stale connection slips through, the request fails fast rather than hanging

These are quaint-native URL parameters, confirmed supported in quaint's PostgreSQL connector.

Suggestion

LiteLLM should set a sensible default for max_idle_connection_lifetime when initialising the Prisma/quaint connection, given that managed cloud databases (RDS, Cloud SQL, Azure Database) routinely drop idle connections. Leaving it unconfigured means any LiteLLM deployment on a managed PostgreSQL instance will hit this error without a clear path to resolution.

A default of 60 seconds for max_idle_connection_lifetime would prevent this for most cloud deployments without meaningful performance impact.

Environment

• LiteLLM version: 1.81.9
• Deployment: Kubernetes (2 replicas)
• Database: AWS RDS PostgreSQL (db.t4g.small)
• Default connection pool: 10 per pod (20 total)

[nikogio]

Update: Parameters applied but error persists intermittently

After applying max_idle_connection_lifetime=60&socket_timeout=10 to the DATABASE_URL, the frequency of the error reduced significantly but did not eliminate it entirely.

Why it still occurs

max_idle_connection_lifetime controls how long quaint retains an idle connection in the pool after it's returned. However, there's still a race condition window: if a connection is checked out of the pool and a request is assigned to it at the exact moment RDS drops it, the parameter offers no protection. The error kind: Closed still fires in that window.

RDS configuration checked

TCP keepalives on RDS are set to OS defaults (not configured), so the issue is not RDS-side aggressive timeout — it's the lack of connection validation on checkout in quaint/Prisma.

Root cause clarification

The real fix would be connection validation on checkout — before handing a connection from the pool to a request, verify it's still alive and discard it if not. This is not currently implemented in quaint for the version LiteLLM uses.

Workarounds ranked by effectiveness

1. AWS RDS Proxy (most reliable) — validates connections before handing them to the application, eliminating stale connections entirely
2. max_idle_connection_lifetime=60 (partial) — reduces frequency but doesn't eliminate the race condition
3. Retry logic in LiteLLM — handle kind: Closed at the application level by automatically retrying on a fresh connection

Suggestion update

The most impactful fix on LiteLLM's side would be to implement automatic retry on kind: Closed errors at the Prisma client level, so the application transparently recovers without exposing the error to the caller. This would make the behaviour consistent regardless of cloud provider or database configuration.