Cutting a Kubernetes Bill by 40% Without Touching Reliability

Where does Kubernetes overspend actually come from?

Requested capacity nobody uses. The scheduler bills you for what a pod reserves, not what it consumes, so a service requesting 4 CPU and averaging 0.3 is paying for 4 forever, and every cluster dashboard will show that node as healthily allocated. On the clusters we have audited, the gap between requested and used CPU has run between 60% and 80%.

Nothing below involves running less software. It involves stopping paying for capacity that was never doing anything.

Step 1. Measure the gap before changing anything

You need two numbers per workload: what it requests, and its p95 actual usage over a fortnight. If you run kube-state-metrics and Prometheus you already have both.

# Requested vs actually used CPU, by workload, over 14 days
sum by (namespace, workload) (
  kube_pod_container_resource_requests{resource="cpu"}
)
/
sum by (namespace, workload) (
  quantile_over_time(0.95, rate(container_cpu_usage_seconds_total[5m])[14d:5m])
)

Anything above 3 is a candidate. Sort descending, and work down, cost is almost always concentrated in a handful of workloads, and the top five typically account for most of the recoverable spend.

Step 2. Right-size requests, and treat CPU and memory differently

They fail differently, so tune them differently.

CPU is compressible. A pod that exceeds its CPU limit gets throttled, not killed. Set the request at roughly p95 usage. Leave the CPU limit off entirely on latency-sensitive services. CFS throttling on a bursty request path adds tail latency for no reliability benefit, which is a trade most teams make by accident.

Memory is not compressible. Exceed the limit and the pod is OOM-killed. Set the request at p99 with real headroom, and set the limit equal to the request so the pod lands in the Guaranteed QoS class and is evicted last.

resources:
  requests:
    cpu: "250m"      # p95 measured, not guessed
    memory: "512Mi"
  limits:
    memory: "512Mi"  # equal to request → Guaranteed QoS
    # no CPU limit: throttling costs latency, saves nothing

Step 3. Let the cluster shrink

Right-sizing only saves money if node count follows demand down. Three pieces have to be in place together:

• Cluster autoscaler or Karpenter, with scale-down actually enabled and its aggressiveness tuned. Many clusters run with scale-down effectively disabled by a default nobody revisited.
• Pod Disruption Budgets on everything that matters, so the autoscaler can drain nodes without you fearing it.
• No orphaned blockers. A single pod with local storage, or a PodDisruptionBudget that permits zero disruptions, will pin an otherwise-empty node indefinitely. These are the most common reason a correctly configured autoscaler never scales anything down.

Step 4. Move the tolerant workloads to spot

Spot capacity runs 60–90% cheaper with a short eviction notice. The split we use:

• Spot: stateless services with 3+ replicas, batch jobs, CI runners, async workers
• On-demand: control plane, databases and stateful sets, anything with a warm-up longer than the eviction window, single-replica services

Spread across several instance types and availability zones. A spot pool pinned to one instance type is a single point of failure that happens to be cheap.

What we deliberately did not do

Buy a cost-optimisation platform before doing any of the above. The data needed for all four steps is already in Prometheus, and a tool that surfaces the same numbers behind a subscription does not change them. Buy the platform later, for continuous enforcement, once you know what your baseline should be.

References

• Kubernetes. Managing Resources for Containers.
• Kubernetes. Pod Quality of Service classes.
• Karpenter documentation, on consolidation and node lifecycle.
• FinOps Foundation. Kubernetes cost allocation.