[elastic_agent] Add TSDB dimensions and fix metric_type declarations by AndersonQ · Pull Request #18508 · elastic/integrations

AndersonQ · 2026-04-17T16:15:24Z

Proposed commit message

[elastic_agent] Add TSDB dimensions and fix metric_type declarations

Add `dimension: true` to fields across the 12 elastic_agent metrics data streams. The added dimensions are redundant with the existing dimensions (1-to-1 with `agent.id` or `component.id`) and therefore do not increase time-series cardinality.

Fix metric_type on:
- beat.stats.libbeat.pipeline.events.active: counter -> gauge
- beat.stats.libbeat.output.events.active: counter -> gauge
- filebeat_input.*.histogram.count: gauge -> counter

Add metric_type on numeric fields that were missing it: system stats, cpu ticks, memstats, handles, runtime.goroutines, uptime, cgroup stats, libbeat pipeline/config/output metrics, write-latency histogram, system.process.cgroup.{memory.mem.failures, cpuacct.percpu}, and filebeat_input.{cel_executions, system_packet_drops}.

Assisted by Claude Code

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.
I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

[ ]

How to review the PR

In order to better see what fields are being changes I suggest using git to see the diff with more context:

fetch/switch to the branch
run git show -U10 HEAD

the flag -U controls how much context (lines before and after the change) to show.

How to test this PR locally

spin up a stack. (I found an issue with the agent hostname) filter on 8.15.5, so, use the 8.19.x to have the dashboard working properly)
install agents (I installed with fleet-server, defend and packetbeat)
check the dashboard
look for errors

Related issues

Closes [Elastic Agent] Evaluate if we should increase the dimensions for the metrics data streams #17349

Screenshots

elastic-vault-github-plugin-prod · 2026-04-17T16:40:12Z

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

Add `dimension: true` to fields across the 12 elastic_agent metrics data streams. The added dimensions are redundant with the existing dimensions (1-to-1 with `agent.id` or `component.id`) and therefore do not increase time-series cardinality. Fix metric_type on: - beat.stats.libbeat.pipeline.events.active: counter -> gauge - beat.stats.libbeat.output.events.active: counter -> gauge - filebeat_input.*.histogram.count: gauge -> counter Add metric_type on numeric fields that were missing it: system stats, cpu ticks, memstats, handles, runtime.goroutines, uptime, cgroup stats, libbeat pipeline/config/output metrics, write-latency histogram, system.process.cgroup.{memory.mem.failures, cpuacct.percpu}, and filebeat_input.{cel_executions, system_packet_drops}. Assisted by Claude Code

elasticmachine · 2026-04-20T14:03:30Z

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

Copilot

Pull request overview

This PR updates the elastic_agent integration’s metrics data streams to better align with TSDB requirements by adding explicit TSDB dimensions and completing/correcting metric_type annotations for numeric fields.

Changes:

Add dimension: true to key identifier fields across the Elastic Agent metrics data streams (and related ECS fields used in those streams).
Fix incorrect metric_type values (notably “active” event counts as gauges; histogram counts as counters).
Add missing metric_type annotations for many numeric stats fields (system, runtime, cgroup, libbeat pipeline/output, etc.).

Reviewed changes

Copilot reviewed 59 out of 59 changed files in this pull request and generated 12 comments.

Show a summary per file

File	Description
packages/elastic_agent/manifest.yml	Bump package version to 2.8.0.
packages/elastic_agent/changelog.yml	Add 2.8.0 changelog entry for TSDB dimensions + metric_type fixes.
packages/elastic_agent/data_stream/packetbeat_metrics/fields/fields.yml	Add dimensions + metric_type annotations (packetbeat metrics).
packages/elastic_agent/data_stream/packetbeat_metrics/fields/ecs.yml	Mark selected ECS agent fields as TSDB dimensions.
packages/elastic_agent/data_stream/packetbeat_metrics/fields/beat-stats-fields.yml	Add dimensions + metric_type annotations for beat stats fields.
packages/elastic_agent/data_stream/packetbeat_metrics/fields/beat-fields.yml	Mark `beat.type` as a TSDB dimension.
packages/elastic_agent/data_stream/packetbeat_metrics/fields/agent.yml	Mark selected host/agent identity ECS fields as dimensions.
packages/elastic_agent/data_stream/osquerybeat_metrics/fields/fields.yml	Add dimensions + metric_type annotations (osquerybeat metrics).
packages/elastic_agent/data_stream/osquerybeat_metrics/fields/ecs.yml	Mark selected ECS agent fields as TSDB dimensions.
packages/elastic_agent/data_stream/osquerybeat_metrics/fields/beat-stats-fields.yml	Add dimensions + metric_type annotations for beat stats fields.
packages/elastic_agent/data_stream/osquerybeat_metrics/fields/beat-fields.yml	Mark `beat.type` as a TSDB dimension.
packages/elastic_agent/data_stream/osquerybeat_metrics/fields/agent.yml	Mark selected host/agent identity ECS fields as dimensions.
packages/elastic_agent/data_stream/metricbeat_metrics/fields/fields.yml	Add dimensions + metric_type annotations (metricbeat metrics).
packages/elastic_agent/data_stream/metricbeat_metrics/fields/ecs.yml	Mark selected ECS agent fields as TSDB dimensions.
packages/elastic_agent/data_stream/metricbeat_metrics/fields/beat-stats-fields.yml	Add dimensions + metric_type annotations for beat stats fields.
packages/elastic_agent/data_stream/metricbeat_metrics/fields/beat-fields.yml	Mark `beat.type` as a TSDB dimension.
packages/elastic_agent/data_stream/metricbeat_metrics/fields/agent.yml	Mark selected host/agent identity ECS fields as dimensions.
packages/elastic_agent/data_stream/heartbeat_metrics/fields/fields.yml	Add dimensions + metric_type annotations (heartbeat metrics).
packages/elastic_agent/data_stream/heartbeat_metrics/fields/ecs.yml	Mark selected ECS agent fields as TSDB dimensions.
packages/elastic_agent/data_stream/heartbeat_metrics/fields/beat-stats-fields.yml	Add dimensions + metric_type annotations for beat stats fields.
packages/elastic_agent/data_stream/heartbeat_metrics/fields/beat-fields.yml	Mark `beat.type` as a TSDB dimension.
packages/elastic_agent/data_stream/heartbeat_metrics/fields/agent.yml	Mark selected host/agent identity ECS fields as dimensions.
packages/elastic_agent/data_stream/fleet_server_metrics/fields/fields.yml	Add dimensions + metric_type annotations (fleet-server metrics).
packages/elastic_agent/data_stream/fleet_server_metrics/fields/ecs.yml	Mark selected ECS agent fields as TSDB dimensions.
packages/elastic_agent/data_stream/fleet_server_metrics/fields/beat-fields.yml	Mark `beat.type` as a TSDB dimension.
packages/elastic_agent/data_stream/fleet_server_metrics/fields/agent.yml	Mark selected host/agent identity ECS fields as dimensions.
packages/elastic_agent/data_stream/filebeat_metrics/fields/fields.yml	Add dimensions + metric_type annotations (filebeat metrics).
packages/elastic_agent/data_stream/filebeat_metrics/fields/ecs.yml	Mark selected ECS agent fields as TSDB dimensions.
packages/elastic_agent/data_stream/filebeat_metrics/fields/beat-stats-fields.yml	Add dimensions + metric_type annotations for beat stats fields.
packages/elastic_agent/data_stream/filebeat_metrics/fields/beat-fields.yml	Mark `beat.type` as a TSDB dimension.
packages/elastic_agent/data_stream/filebeat_metrics/fields/agent.yml	Mark selected host/agent identity ECS fields as dimensions.
packages/elastic_agent/data_stream/filebeat_input_metrics/fields/fields.yml	Fix histogram `.count` metric_type + add dimensions/metric_type for input fields.
packages/elastic_agent/data_stream/filebeat_input_metrics/fields/ecs.yml	Mark selected ECS agent fields as TSDB dimensions.
packages/elastic_agent/data_stream/filebeat_input_metrics/fields/beat-stats-fields.yml	Add dimensions + metric_type annotations for beat stats fields.
packages/elastic_agent/data_stream/filebeat_input_metrics/fields/agent.yml	Mark selected host/agent identity ECS fields as dimensions.
packages/elastic_agent/data_stream/endpoint_security_metrics/fields/fields.yml	Add dimensions + metric_type annotations (endpoint security metrics).
packages/elastic_agent/data_stream/endpoint_security_metrics/fields/ecs.yml	Mark selected ECS agent fields as TSDB dimensions.
packages/elastic_agent/data_stream/endpoint_security_metrics/fields/beat-fields.yml	Mark `beat.type` as a TSDB dimension.
packages/elastic_agent/data_stream/endpoint_security_metrics/fields/agent.yml	Mark selected host/agent identity ECS fields as dimensions.
packages/elastic_agent/data_stream/elastic_agent_metrics/fields/fields.yml	Add dimensions + metric_type annotations (elastic_agent metrics).
packages/elastic_agent/data_stream/elastic_agent_metrics/fields/ecs.yml	Mark selected ECS agent fields as TSDB dimensions.
packages/elastic_agent/data_stream/elastic_agent_metrics/fields/beat-stats-fields.yml	Add dimensions + metric_type annotations for beat stats fields.
packages/elastic_agent/data_stream/elastic_agent_metrics/fields/beat-fields.yml	Mark `beat.type` as a TSDB dimension.
packages/elastic_agent/data_stream/elastic_agent_metrics/fields/agent.yml	Mark selected host/agent identity ECS fields as dimensions.
packages/elastic_agent/data_stream/cloudbeat_metrics/fields/fields.yml	Add dimensions + metric_type annotations (cloudbeat metrics).
packages/elastic_agent/data_stream/cloudbeat_metrics/fields/ecs.yml	Mark selected ECS agent fields as TSDB dimensions.
packages/elastic_agent/data_stream/cloudbeat_metrics/fields/beat-stats-fields.yml	Add dimensions + metric_type annotations for beat stats fields.
packages/elastic_agent/data_stream/cloudbeat_metrics/fields/beat-fields.yml	Mark `beat.type` as a TSDB dimension.
packages/elastic_agent/data_stream/cloudbeat_metrics/fields/agent.yml	Mark selected host/agent identity ECS fields as dimensions.
packages/elastic_agent/data_stream/auditbeat_metrics/fields/fields.yml	Add dimensions + metric_type annotations (auditbeat metrics).
packages/elastic_agent/data_stream/auditbeat_metrics/fields/ecs.yml	Mark selected ECS agent fields as TSDB dimensions.
packages/elastic_agent/data_stream/auditbeat_metrics/fields/beat-stats-fields.yml	Add dimensions + metric_type annotations for beat stats fields.
packages/elastic_agent/data_stream/auditbeat_metrics/fields/beat-fields.yml	Mark `beat.type` as a TSDB dimension.
packages/elastic_agent/data_stream/auditbeat_metrics/fields/agent.yml	Mark selected host/agent identity ECS fields as dimensions.
packages/elastic_agent/data_stream/apm_server_metrics/fields/fields.yml	Add dimensions + metric_type annotations (apm-server metrics).
packages/elastic_agent/data_stream/apm_server_metrics/fields/ecs.yml	Mark selected ECS agent fields as TSDB dimensions.
packages/elastic_agent/data_stream/apm_server_metrics/fields/beat-stats-fields.yml	Add dimensions + metric_type annotations for beat stats fields.
packages/elastic_agent/data_stream/apm_server_metrics/fields/beat-fields.yml	Mark `beat.type` as a TSDB dimension.
packages/elastic_agent/data_stream/apm_server_metrics/fields/agent.yml	Mark selected host/agent identity ECS fields as dimensions.

Comments suppressed due to low confidence (1)

packages/elastic_agent/data_stream/filebeat_input_metrics/fields/fields.yml:21

The descriptions for the dynamic path_match mappings contain malformed inline code formatting (path_match:``...). This renders oddly in generated docs; consider fixing the backticks so the mapping expression is correctly shown.

    - name: '*.histogram.count'
      type: long
      metric_type: counter
      description: Map all fields with `path_match:``*.histogram.count` to `long`
    - name: '*.histogram.*'
      type: double
      metric_type: gauge
      description: Map all fields with `path_match:``*.histogram.*` to `double`

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

      type: keyword
+      dimension: true
      ignore_above: 1024
      description: The binary that exeuctes the component


      type: keyword
+      dimension: true
      ignore_above: 1024
      description: The binary that exeuctes the component


      type: keyword
+      dimension: true
      ignore_above: 1024
      description: The binary that exeuctes the component


      type: keyword
+      dimension: true
      ignore_above: 1024
      description: The binary that exeuctes the component


      type: keyword
+      dimension: true
      ignore_above: 1024
      description: The binary that exeuctes the component


      type: keyword
+      dimension: true
      ignore_above: 1024
      description: The binary that exeuctes the component


      type: keyword
+      dimension: true
      ignore_above: 1024
      description: The binary that exeuctes the component


      type: keyword
+      dimension: true
      ignore_above: 1024
      description: The binary that exeuctes the component


            - name: percpu
              type: object
              object_type: long
+              metric_type: counter


    - name: cel_executions
      type: long
+      metric_type: counter
      description: URL-ish of input resource


elasticmachine · 2026-04-20T14:26:00Z

💚 Build Succeeded

Buildkite Build
Commit: baa2f33

History

💚 Build #41592 succeeded 7cab099
💚 Build #41546 succeeded d2cf4f4

cc @AndersonQ

AndersonQ self-assigned this Apr 17, 2026

AndersonQ added Integration:elastic_agent Elastic Agent Team:Elastic-Agent-Data-Plane Agent Data Plane team [elastic/elastic-agent-data-plane] labels Apr 17, 2026

AndersonQ force-pushed the 17349-elastic-agent-tsdb-check branch from 9c202fc to d2cf4f4 Compare April 17, 2026 16:16

macroscopeapp Bot reviewed Apr 17, 2026

View reviewed changes

Comment thread packages/elastic_agent/data_stream/apm_server_metrics/fields/fields.yml

Comment thread packages/elastic_agent/data_stream/apm_server_metrics/fields/beat-stats-fields.yml Outdated

AndersonQ force-pushed the 17349-elastic-agent-tsdb-check branch from d2cf4f4 to 7cab099 Compare April 20, 2026 09:05

AndersonQ force-pushed the 17349-elastic-agent-tsdb-check branch from 7cab099 to baa2f33 Compare April 20, 2026 14:03

AndersonQ marked this pull request as ready for review April 20, 2026 14:03

AndersonQ requested a review from a team as a code owner April 20, 2026 14:03

Copilot AI review requested due to automatic review settings April 20, 2026 14:03

Copilot started reviewing on behalf of AndersonQ April 20, 2026 14:04 View session

Copilot AI reviewed Apr 20, 2026

View reviewed changes

andrewkroh added the Team:Elastic-Agent Platform - Ingest - Agent [elastic/elastic-agent] label Apr 20, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[elastic_agent] Add TSDB dimensions and fix metric_type declarations#18508

[elastic_agent] Add TSDB dimensions and fix metric_type declarations#18508
AndersonQ wants to merge 1 commit intoelastic:mainfrom
AndersonQ:17349-elastic-agent-tsdb-check

AndersonQ commented Apr 17, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

elastic-vault-github-plugin-prod Bot commented Apr 17, 2026

Uh oh!

elasticmachine commented Apr 20, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

elasticmachine commented Apr 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

AndersonQ commented Apr 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed commit message

Checklist

Author's Checklist

How to review the PR

How to test this PR locally

Related issues

Screenshots

Uh oh!

Uh oh!

Uh oh!

elastic-vault-github-plugin-prod Bot commented Apr 17, 2026

🚀 Benchmarks report

Uh oh!

elasticmachine commented Apr 20, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

elasticmachine commented Apr 20, 2026

💚 Build Succeeded

History

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

AndersonQ commented Apr 17, 2026 •

edited

Loading