Nobody sets out to spend more watching the cluster than running it. It happens because telemetry cost scales with deploy frequency and pod churn, while infrastructure cost scales with load. Those are two different curves, and one crossed the other while you were shipping features. Here's a real shape of it. 40-node EKS cluster, mostly m6i.2xlarge on a savings plan, 8 TB of gp3, a couple of NAT gateways doing real work. Infrastructure lands around $12,400/month. The observability invoice that quarter was $17,900. And 62% of that was custom metrics - from three metric names, all carrying a pod_name label nobody had ever grouped by. The mechanic is multiplication, and it's worth doing the arithmetic once so you never forget it. http_requests_total{service, method, status_code, env} = 40 services x 5 methods x 12 status codes x 3 envs = 7,200 series. Completely fine. Now one engineer adds pod because they wanted to debug a noisy replica: x 250 pods = 1,800,000 series. A 250x multiplication from one label. And pod churns - every deploy retires the old names and mints new ones, so your active series stay at 1.8M while your billable series over a 30-day window climb into the tens of millions. Add request_id or an un-normalised URL path with an ID in it, and it stops being a multiplier and becomes unbounded: cardinality now grows with your traffic, forever. The rule I use: a label is safe if its value set is bounded by something YOU control - a config file, an enum, a deploy. It is unsafe if it's bounded by something your USERS control - traffic, IDs, URLs, error strings. Three commands that tell you where you stand. 1. Total active series: count({__name__=~".+"}) 2. Your worst metrics: topk(20, count by (__name__)({__name__=~".+"})) 3. The precomputed version that won't melt your Prometheus - hit /api/v1/status/tsdb and read seriesCountByMetricName and labelValueCountByLabelName. It's free, it's instant, and labelValueCountByLabelName will name your offender in about four seconds.