Dropshot exposes no static metrics

[daniel-levin]

As an operator of an HTTP service/application, I need some basic, static telemetry. While the DTrace probes are useful, they are dynamic. I want a constant stream of static metrics amenable to being rotated off-box and aggregated, and wired up to alarms. While consumers of Dropshot can do this themselves, it would be useful and convenient if Dropshot was able to emit metrics.

Using the metrics crate, Dropshot could expose metrics at a negligible cost (picture tracing but for metrics). Users could decide which metrics reporter to use, if any.

In my experience, it's invaluable to have access to these metrics, which are undifferentiated and common to virtually all web services:

1. Error rate, broken down by 4xx/5xx.
2. Errors-per-API.
3. Request latency.

Dropshot could probably emit these for every user. It would be very convenient.

[davepacheco]

Indeed. For reference, in Omicron we use this helper to expose metrics for our endpoints:
https://github.com/oxidecomputer/omicron/blob/ce35f501cf52fa47dc0d2148ffc5c5482e9fa386/oximeter/instruments/src/http.rs#L161

Using it requires less-than-ideal boilerplate:
https://github.com/oxidecomputer/omicron/blob/ce35f501cf52fa47dc0d2148ffc5c5482e9fa386/nexus/src/external_api/http_entrypoints.rs#L143-L147

It would be nice if Dropshot could collect metrics and make them easily available to consumers. That said, in practice, I've found that consumers wind up wanting their own app-specific fields (e.g., the authenticated user). They then either wind up implementing their own thing or else whatever we do in Dropshot needs to somehow compose with that (e.g., have users be able to update a RequestContext with metadata that gets stuffed into the metrics) and this is where things get tricky. It hasn't been a priority for us yet.