Kubernetes does not create observability cost by being expensive to run. It creates observability cost by generating identity faster than a static architecture ever did.
Short answer
Kubernetes telemetry can grow faster than the actual workload it describes because the platform continuously creates and destroys the identities — pods, containers, ReplicaSets, Jobs — that metrics, logs and traces attach themselves to. A cluster processing the same request volume in January and June can carry a completely different, larger telemetry footprint by June simply because more pods existed, more labels were introduced, and more ephemeral workloads ran in between — none of which shows up in a request-count graph. The fix is not to strip Kubernetes metadata; most of it is genuinely useful for diagnosis and ownership. The fix is deciding, per telemetry signal, which identity level — pod, workload, service, cluster — actually needs to persist in metrics and dashboards, and which belongs only in traces and logs where individual-instance detail is searchable without being a permanent aggregation dimension.
Why a static mental model breaks on Kubernetes
A conventional infrastructure model is host-to-application: a server exists, an application runs on it, and telemetry describes a relatively stable set of entities over time. Kubernetes replaces that model with a hierarchy that is deliberately not stable: cluster, node, namespace, workload, Deployment, ReplicaSet, pod, container — and several of those layers are designed to be short-lived by construction. A Deployment's ReplicaSet is replaced on every rollout. A pod is replaced on every scale event, every node drain, every restart. A Job or CronJob creates pods that are meant to exist only for the duration of a task and then disappear.
None of this is a design flaw. It is what makes Kubernetes good at what it does. But every one of those transient objects can generate its own telemetry identity, and a telemetry pipeline that treats each one as a first-class, persistent dimension inherits the platform's churn as its own cost driver.
This article is about that mechanism specifically — how dynamic infrastructure amplifies telemetry across logs, metrics and traces. It is not a repeat of Datadog Custom Metrics and Cardinality: How Costs Grow, which covers the detailed mechanics of how tag combinations become billable custom metrics in Datadog specifically; cardinality is one of several mechanisms discussed here, not the whole subject. It is also not the cost-attribution article — Observability FinOps: How to Allocate Costs by Team, Service and Environment covers how to assign Kubernetes-generated cost to an owner once it exists; this article covers why the cost exists in the first place.
The churn mechanism, concretely
Consider a platform running a moderate, genuinely stable request volume across a Kubernetes estate. Over a period of months, without any change in that request volume:
- the cluster autoscales horizontally in response to normal traffic variance, creating and destroying pods continuously even though average load is flat;
- rolling deployments replace ReplicaSets — and therefore the pods behind them — on every release, which for an actively developed service can mean daily or even hourly identity churn;
- scheduled Jobs and CronJobs create short-lived pods for batch work, each with its own pod identity that exists for minutes and then disappears;
- new labels get introduced by newly onboarded services, updated Helm charts, or a platform team adding metadata for a new use case;
- node pools are resized or replaced, changing node-level identity even when the workloads running on them have not changed.
Request volume — the thing the business actually cares about — has not moved. The number of distinct identities that telemetry systems have observed over the same period has grown substantially, because churn is additive: every pod that ever existed, however briefly, can leave a trace in a metrics or logging backend that treats pod identity as a first-class dimension.
This is the core reason a Kubernetes telemetry curve can diverge sharply from a request-volume or transaction-count curve, and why "did traffic increase" is the wrong diagnostic question. The better question is what new identities appeared in the dimensional model during the period being reviewed — a question about churn, not about load.
Where Kubernetes metadata actually enters telemetry
Kubernetes exposes several distinct telemetry surfaces, and conflating them obscures where a given cost is actually coming from.
The kubelet on each node exposes multiple metrics endpoints covering node and pod resource usage, container runtime statistics (commonly surfaced through cAdvisor-derived metrics), and probe results — each a potential source of per-pod or per-container series.
kube-state-metrics generates telemetry from the state of Kubernetes API objects themselves — pods, Deployments, nodes, Jobs and more — independent of what those workloads are doing. Its own documentation explicitly warns that large numbers of frequently updated Kubernetes resources can create high telemetry volume, and recommends controlling which metrics and resource types are exposed rather than collecting everything by default.
Application-level metrics, logs and traces carry whatever labels the instrumentation or a metadata-enrichment step attaches — commonly derived from Kubernetes labels and annotations through an agent, sidecar or Collector processor.
Service discovery — Prometheus's Kubernetes service discovery is the canonical example — continuously discovers scrape targets from the cluster's current state, meaning the set of things being monitored tracks the cluster's actual churn in near real time, by design.
A cost review that looks at only one of these surfaces will consistently underestimate total Kubernetes-driven telemetry growth, because a single spike in pod churn can appear simultaneously in kubelet-derived infrastructure metrics, kube-state-metrics object-state telemetry, and application-level metrics carrying pod identity as a label — three separate cost surfaces reacting to one underlying event.
Labels and annotations: useful metadata, not automatically a metric dimension
Kubernetes' own recommended application labels — name, instance, component, and the broader application a component belongs to — exist to make resources identifiable and are, by design, meant to be attached consistently across an application's resources. That is exactly why they are useful for troubleshooting and ownership, and exactly why teams reach for them when instrumenting metrics: they are already there, already consistent, and already answer "what is this."
The problem is not that this metadata exists. It is the assumption that everything available as Kubernetes metadata should automatically become a persisted dimension on every metric. A label that identifies a stable concept — which service, which environment, which cluster — is a reasonable metric dimension because its value population is bounded and slow-changing. A label or derived identifier tied to something ephemeral — a specific pod name, a specific container ID, a ReplicaSet hash that changes on every deployment — creates a metric dimension whose value population grows continuously for as long as the workload keeps churning, which for an actively deployed service is indefinitely.
The distinction that matters is not "Kubernetes metadata versus application metadata." It is stable identity versus ephemeral identity, and Kubernetes happens to generate both in the same metadata surface, which is why the two get conflated.
Prometheus service discovery and relabeling as the control point
For Prometheus-based metrics pipelines, Kubernetes service discovery finds scrape targets from the cluster's live state, and relabeling is the mechanism that shapes what gets scraped and what labels survive into stored series. This split matters operationally: target-level relabeling decides which pods get scraped at all, while metric-level relabeling — applied after scraping — decides which labels on already-collected samples get kept, renamed or dropped before they become permanent dimensions in the time-series database.
High-churn, high-cardinality Kubernetes-derived labels — pod UID, container ID, image digest, and similar generated identifiers — are common candidates for dropping at the relabeling stage specifically because they identify a transient object rather than a stable operational concept, and because a Prometheus deployment scraping many pods across many containers can otherwise accumulate a very large number of distinct series from exactly this class of label. Relabeling is where that decision gets enforced, close to collection, rather than after the series already exist in storage.
The signal-by-signal question
Rather than applying one blanket policy to all Kubernetes-sourced telemetry, the design question is best asked per signal type, because the right answer differs by signal.
Metrics generally benefit from bounded, stable dimensions — service, environment, cluster, namespace where namespace maps to something operationally meaningful — and generally do not benefit from pod-level or container-level identity as a persisted dimension on application or business metrics. Infrastructure-troubleshooting metrics are a legitimate exception where pod- or node-level granularity is the point.
Logs can carry ephemeral identity more cheaply than metrics, because a log line is a discrete event rather than a contributor to an aggregated, continuously-updated time series — but volume still scales with the number of active pods emitting logs, which is exactly the churn mechanism described above, just expressed as log volume rather than metric cardinality. See Datadog Logs Pricing: Ingestion, Indexing and Retention Explained for how that volume translates into cost once it reaches a platform that separately meters ingestion and indexing.
Traces naturally carry pod- and instance-level identity as span attributes, which is appropriate — a trace exists specifically to describe one execution path through a distributed system, including which instance handled which hop. The risk with traces is less about identity attached to a span and more about instrumentation depth: as services become more distributed, or as auto-instrumentation adds spans for every internal call, the number of spans per request can grow independent of Kubernetes churn specifically, though the two compound in a Kubernetes environment where the number of distinct services communicating is itself often higher than in a monolithic deployment.
Duplicate collection: the other Kubernetes-specific cost multiplier
A pattern that recurs specifically in Kubernetes environments, distinct from the label-cardinality mechanism above, is overlapping telemetry pipelines collecting substantially the same signal. A cluster commonly accumulates a node-level metrics agent, a separate APM or tracing agent, a logging DaemonSet, and sometimes a second monitoring integration added by a different team for a different purpose — each with its own resource footprint and its own path to a billing surface.
Overlap is not automatically waste; different consumers can have legitimate reasons to need their own collection path. But an unreviewed accumulation of collectors is a distinct cost driver from cardinality, and it deserves its own periodic question: does this collector provide incremental diagnostic value that the others do not, or is it duplicate consumption that nobody has stopped to check?
A working decision framework
| Kubernetes telemetry source | Typical operational value | Persist as a metric dimension? |
|---|---|---|
service, environment, namespace (where meaningful) |
High — ownership, alerting, SLOs | Yes |
| Cluster, region, availability zone | High — infrastructure and failure-domain boundaries | Usually yes |
| Deployment / workload name | Moderate to high — deployment-level regression analysis | Often yes, at workload level rather than pod level |
| Pod name | High for live troubleshooting, low for trend analysis | Rarely, on application-level metrics; useful on infrastructure-diagnosis metrics |
| Container ID, ReplicaSet hash, generated pod suffix | Low as a persisted dimension — identifies a transient object | Generally no on metrics; acceptable on logs and traces |
| kube-state-metrics object-state telemetry | High, but volume-sensitive at scale | Scope to the resource types and metrics actually consumed downstream |
Governance, briefly
Kubernetes-driven telemetry growth is ultimately a platform-ownership question as much as a technical one: someone needs to own the default collection profile new services inherit, review new labels and integrations before they reach every workload by default, and periodically ask whether an existing collector still earns its place. That ownership and cost-attribution layer — assigning the Kubernetes-generated bill to the team or service actually responsible for the churn generating it — is covered in full in Observability FinOps: How to Allocate Costs by Team, Service and Environment; this article stops at making the mechanism visible enough for that ownership conversation to have real evidence behind it.
FAQ
Because Kubernetes continuously creates and destroys pods, ReplicaSets and Jobs through normal operation — autoscaling, rolling deployments, scheduled batch work — and each of those transient objects can generate its own telemetry identity. Request volume measures load; churn measures how many distinct identities existed over a period. The two are independent, and churn is what drives telemetry growth that traffic graphs do not explain.
Pod churn is the rate at which pods are created and destroyed — through autoscaling, deployments, restarts, and scheduled Jobs. It matters for telemetry cost because any pipeline that treats pod identity as a persistent dimension in metrics, or that scrapes and stores telemetry per pod without bounding retained identity, accumulates cost proportional to the number of pods that have ever existed, not the number running at any moment.
Generally not on application or business metrics, because their value population grows continuously for as long as a workload keeps deploying and scaling, which is indefinitely for an actively developed service. They remain useful on logs and traces, where they identify a specific event or execution path rather than becoming a persisted aggregation dimension, and on infrastructure-level metrics purpose-built for instance-level troubleshooting.
kube-state-metrics generates telemetry directly from the state of Kubernetes API objects — pods, Deployments, Jobs, nodes and more. Its own documentation warns that large numbers of frequently updated resources can produce high telemetry volume, and recommends scoping which metrics and resource types are collected rather than exposing everything by default.
relabelconfigs operates before scraping, on target metadata from service discovery, and determines which targets are scraped and what labels they carry into collection. metricrelabelconfigs operates after scraping, on the labels of already-collected samples, and determines which metrics and labels are kept, renamed or dropped before being stored. High-cardinality Kubernetes-derived identifiers are commonly dropped at this second stage.
They are related but not identical. Custom-metric cardinality is the billing mechanism that makes unique tag combinations costly on a specific platform, covered in Datadog Custom Metrics and Cardinality: How Costs Grow(/insights/datadog-custom-metrics-cardinality/). Kubernetes churn is one of several sources that feed that mechanism — alongside logs, traces and duplicate collection — and applies across telemetry types and across vendors, not only to Datadog custom metrics specifically.
Sources
- Kubernetes documentation — Workload resources (Deployments, ReplicaSets, Jobs, CronJobs) and recommended application labels.
- kube-state-metrics documentation — resource-state telemetry generation and volume guidance for frequently updated resources.
- Prometheus documentation — Kubernetes service discovery and relabeling (
relabel_configsvs.metric_relabel_configs). - Kubernetes documentation — kubelet metrics endpoints and cAdvisor-derived container metrics.
Kubernetes, Prometheus and vendor-specific agent behavior evolve across versions. Validate current label conventions, service-discovery behavior and collector defaults against official documentation for the versions in use.