document recommendations for reverse proxying prometheus

[herbetom]

It would be good if there were some recommendations documented about which settings to use when putting prometheus behind a reverse proxy like nginx.

I'm thinking of stuff like this:

proxy_set_header        Host $host;
proxy_set_header        X-Real-IP $remote_addr;
proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header        X-Forwarded-Proto $scheme;
proxy_set_header        X-Forwarded-Host $host;
proxy_set_header        X-Forwarded-Server $hostname;

(Haven't checked that this is a sensible assortment for prometheus specifically.

But then there is also the /api/v1/notifications/live endpoint which uses Server-sent events which requires special care so that is passes through nginx.

https://prometheus.io/docs/prometheus/latest/querying/api/#live-notifications

If this isn't configured correctly there will be a notification as such after a short while:

Real-time notifications interrupted.

---

With the following config it hasn't popped up yet. But i haven't checked how sensible every single one of those settings are:

upstream prometheus {
        server [::1]:9090 ;
}
server {
        listen 0.0.0.0:80 ;
        listen [::0]:80 ;
        server_name prometheus.example.org ;
        location / {
                return 301 https://$host$request_uri;
        }
}
server {
        listen 0.0.0.0:443 ssl ;
        listen [::0]:443 ssl ;
        server_name prometheus.example.org  ;
        http2 on;
        ssl_certificate /var/lib/acme/prometheus.example.org/fullchain.pem;
        ssl_certificate_key /var/lib/acme/prometheus.example.org/key.pem;
        ssl_trusted_certificate /var/lib/acme/prometheus.example.org/chain.pem;
        location / {
                proxy_pass http://prometheus;proxy_set_header
                # this is the code snippet in the top of the issue:
                include /nix/store/bjs5fy079yy18vqcfhzqw4011kc2rw04-nginx-recommended-proxy_set_header-headers.conf;
        }
        location =/api/v1/notifications/live {
                proxy_pass http://prometheus;
                proxy_http_version 1.1;
                proxy_set_header Connection "";
                proxy_buffering off;
                proxy_cache off;
                proxy_read_timeout 10m;
                proxy_send_timeout 10m;
                gzip off;
                # this is the code snippet in the top of the issue:
                include /nix/store/bjs5fy079yy18vqcfhzqw4011kc2rw04-nginx-recommended-proxy_set_header-headers.conf;
        }
}

[oindrilakha12-ui]

Hi, I’d like to work on this issue and contribute documentation for reverse proxy configuration with Prometheus. Could you please assign it to me?

[kris-guy-lab]

If additional help is needed, I’d be happy to collaborate on documenting recommended reverse‑proxy settings for Prometheus, including SSE considerations.

[krajorama]

Hello from the bug scrub (of prometheus/prometheus)!

We think think could be a blog post, since it's not strictly speaking about the configuration of Prometheus.
Blog posts are in the docs repository, so we've transferred the issue.

Please be mindful of writing such article backed up by hands on experience and avoid AI slop.

[kris-guy-lab]

Got it. Happy to proceed under the guidance from the team.

[jan--f]

Blog posts and instructions live in https://github.com/prometheus/docs/tree/main/blog.