Go Runtime in Containers: GOMAXPROCS, CPU Throttling and the Tail Latency Trap on Kubernetes

There is a trap your dashboard will never show you: the pod’s CPU sits at 40%, every alert is green, and p99 still blows up. The cause isn’t a lack of CPU. It’s that CPU gets locked away in fixed slices of time.

This post starts from that symptom, goes down to the cgroup, and comes back up to how Go 1.25 handles it.

1. The paradox: under half the CPU and p99 already blows up

That night the API’s p99 latency jumped from 180ms to 1.4s for fifteen minutes, then dropped back by itself. No 5xx, no deploy, GC pauses under 2ms. What cost me another two hours was that the pod’s CPU utilization was only around 40%. The first instinct is always “there’s spare CPU, so it can’t be CPU”. That instinct is wrong.

In a container, the CPU a pod may use is a hard budget per time slice, not an average ratio. Utilization is averaged over a window of tens of seconds. What kills requests is the budget being drained in the first few milliseconds of each period, after which the whole process freezes until the period ends. The average of those freezes can still sit below 50%.

The phenomenon is called CPU throttling, and the evidence lives in the pod’s cgroup, not in application metrics.

2. GMP: the P is your permit to burn CPU

Three concepts from the Go scheduler, kept short. G, the goroutine, is the unit of work the runtime manages. M, the machine, is an OS thread, the thing the kernel actually schedules. P, the logical processor, is the execution context needed to run Go code: to run Go code, an M has to be holding a P.

The number of Ps is exactly GOMAXPROCS. Whatever GOMAXPROCS is, the runtime believes it may run that many goroutines in parallel. Each P has a local queue; when it’s empty, the P takes work from the global queue or steals from another P. Work stealing is only efficient when the number of Ps matches the number of real cores.

A goroutine waiting on network I/O doesn’t hold a P: it’s handed to the netpoller and the P is released for other goroutines. When a goroutine makes a blocking syscall, the M detaches from the P, and sysmon reclaims the P after a time threshold.

The runtime, though, has no channel of information about cgroups. It asks the kernel how many logical cores the machine has, sets GOMAXPROCS to that number, and spins up enough Ms for every P to run at once.

3. CFS quota: a hard budget per period

Kubernetes doesn’t divide CPU itself; it writes a few numbers into the cgroup and lets the kernel do the rest. On cgroup v2, the cpu.max file holds the quota and the period, in microseconds. A limit of 2 CPUs becomes a quota of 200000 with a period of 100000: in every 100ms period, this group of processes may spend at most 200ms of CPU time, the equivalent of two cores running flat out. The default period is 100ms and Kubernetes exposes no knob to change it. That’s a very coarse window for a request that takes 5ms.

The enforcement mechanism is the scary part. The kernel accumulates the CPU time of every task in the cgroup during the period. Once the quota is exceeded, the entire cgroup is throttled, not just the thread that burned the most. For a Go service, every running M (the goroutines handling requests, the netpoller, even sysmon) stops at the same moment until the next period begins.

And the process has no idea it’s being throttled: no syscall returns an error, no signal arrives. The threads simply don’t get to run. Measured by wall clock, time passes while CPU time barely moves, and no application-level metric can tell “slow because there’s a lot of work” apart from “slow because I’m forbidden to run”.

Don’t confuse cpu.weight (the CPU request, which is only a share weight when cgroups compete) with cpu.max: only cpu.max is a hard ceiling, and only it produces throttling.

4. Why GOMAXPROCS set from the node’s core count burns the whole quota

Say the node has 32 logical cores and the API pod has a CPU limit of 2. On Go 1.24 and earlier, the runtime reads the machine’s core count, sees 32, and sets GOMAXPROCS to 32. The mismatch starts right there: the runtime believes it holds 32 permits, while the cgroup only grants it 2.

flowchart TB
  subgraph TL["One 100ms period — CPU limit 2 (quota = 200ms CPU time)"]
    direction LR
    A["t=0ms<br/>32 Ms run in parallel"] --> B["~t=6ms<br/>quota = 0"]
    B --> C["t=6..100ms<br/>THROTTLED<br/>whole cgroup frozen"]
    C --> D["t=100ms<br/>new period, quota reset"]
  end

  R["Request"] --> Q{"Quota left?"}
  Q -- "yes" --> RUN["Runs right away on a P/M"]
  Q -- "no" --> WAIT["Queued until next period<br/>+0..94ms latency"]
  D -.-> Q

At the start of each period, when a burst of requests lands, 32 goroutines can run at the same time. The kernel charges the cgroup by CPU time, not by thread count, so 200ms of quota can be drained in about six milliseconds of wall-clock time if the node is idle. From the seventh millisecond to the hundredth, the service is frozen, literally.

A request arriving in that window gets a wait added straight onto its latency, and that wait is a multiple of 100ms rather than a smooth distribution. That’s why p50 barely moves while p99 jumps in steps.

The damage doesn’t stop there. The runtime doesn’t know it’s being blocked, so it keeps behaving as if it were on a 32-core machine: creating more Ms, shuffling goroutines between queues, running preemption, starting GC mark workers sized as a percentage of GOMAXPROCS. Every GC cycle then becomes a CPU burst right when the quota is already gone.

To be blunt: dropping GOMAXPROCS to the limit doesn’t remove throttling. It only removes the systematic overshoot, turning “burned in six milliseconds” into “spread across most of the period”.

5. Measuring: where the real evidence lives

Start from the cgroup’s cpu.stat file, not the application dashboard.

# cgroup v2
cat /sys/fs/cgroup/cpu.max        # 200000 100000  => 2 CPU
cat /sys/fs/cgroup/cpu.stat
# usage_usec 48102933
# user_usec 39201104
# system_usec 8901829
# nr_periods 12480
# nr_throttled 3912
# throttled_usec 812340912

# cgroup v1 — different file names, same meaning
cat /sys/fs/cgroup/cpu/cpu.cfs_quota_us
cat /sys/fs/cgroup/cpu/cpu.stat   # nr_periods, nr_throttled, throttled_time (ns)

How to read it: nr_throttled divided by nr_periods is the share of periods that got throttled, and throttled_usec divided by nr_throttled is how long each throttle lasted. A throttled share above 10% is abnormal for a latency-sensitive service, and an average throttle of tens of milliseconds means requests are genuinely being frozen.

In Prometheus, cAdvisor already exposes these counters:

# Share of periods that were throttled
sum by (pod) (rate(container_cpu_cfs_throttled_periods_total[5m]))
/
sum by (pod) (rate(container_cpu_cfs_periods_total[5m]))

# Throttled time relative to the quota; use alongside the above, not instead of it
sum by (pod) (rate(container_cpu_cfs_throttled_seconds_total[5m]))
/
sum by (pod) (container_spec_cpu_quota / container_spec_cpu_period)

container_cpu_cfs_throttled_seconds_total is a counter in seconds, completely different from container_cpu_usage_seconds_total, which most dashboards use for their CPU charts. This is the metric I put in alerts; utilization is left for capacity planning.

Inside the process, GODEBUG=schedtrace=1000 prints one line per second with the number of Ps, Ms, idle Ps and the global queue. The signs to look for are a long queue while almost no P is running, or lines that skip: if the runtime couldn’t get around to printing its log, that’s also evidence it wasn’t running.

The tool that lasts in production is runtime/metrics:

var samples = []metrics.Sample{
	{Name: "/sched/gomaxprocs:threads"},
	{Name: "/sched/goroutines:goroutines"},
	{Name: "/sched/latencies:seconds"}, // histogram: runnable -> running
	{Name: "/cpu/classes/user:cpu-seconds"},
	{Name: "/cpu/classes/idle:cpu-seconds"},
}

func report() {
	metrics.Read(samples)
	for _, s := range samples {
		switch s.Value.Kind() {
		case metrics.KindUint64:
			log.Printf("%s = %d", s.Name, s.Value.Uint64())
		case metrics.KindFloat64Histogram:
			log.Printf("%s (ms) = %v", s.Name, s.Value.Float64Histogram().Buckets)
		}
	}
}

/sched/latencies:seconds is the most useful one: it measures the time from a goroutine becoming runnable to actually running on a P, so the time spent blocked by the cgroup sits entirely inside it. If its high percentiles jump while the goroutine count and the P count stay the same, the problem is outside the process. /sched/gomaxprocs:threads is the quickest number to compare against the limit: on Go 1.25 they should match.

6. Before Go 1.25: read the quota and set GOMAXPROCS yourself

For years the only way was to do it yourself:

func gomaxprocsFromQuota() int {
	b, err := os.ReadFile("/sys/fs/cgroup/cpu.max")
	if err != nil {
		return 0
	}
	f := strings.Fields(string(b))
	if len(f) != 2 || f[0] == "max" {
		return 0 // no CPU bandwidth limit
	}
	quota, err1 := strconv.ParseInt(f[0], 10, 64)
	period, err2 := strconv.ParseInt(f[1], 10, 64)
	if err1 != nil || err2 != nil || quota <= 0 || period <= 0 {
		return 0
	}
	return int((quota + period - 1) / period) // round up
}

func init() {
	if n := gomaxprocsFromQuota(); n > 0 {
		runtime.GOMAXPROCS(n)
	}
}

This is good enough for cgroup v2, but four more things need handling: the cgroup v1 paths are completely different; the quota can be max, meaning unlimited; you have to read the cgroup that contains the process, not the one of the shell you typed the command in; and it must run before everything else, because worker pools or shard counts computed from GOMAXPROCS at init time will freeze the old number.

The most popular library for this is Uber’s automaxprocs:

import "go.uber.org/automaxprocs/maxprocs"

func main() {
	// Returns an undo function that restores the old value, usually used in tests.
	if _, err := maxprocs.Set(maxprocs.Logger(log.Printf)); err != nil {
		log.Printf("automaxprocs: %v", err)
	}
}

Its weakness is that it’s static: it reads the quota once at startup, so if the pod’s CPU limit changes later, the number inside the runtime doesn’t follow. It’s a dependency you have to add to every binary, and because GOMAXPROCS is set explicitly, all the runtime’s automatic behavior from then on is disabled, even after you upgrade to Go 1.25.

7. Go 1.25: the runtime knows it’s inside a container

Go 1.25 moves this into the runtime. On Linux, the runtime looks at the CPU bandwidth limit of the cgroup containing the process; if that limit is lower than the number of available logical cores, the default GOMAXPROCS equals that limit instead of the node’s core count. In Kubernetes, this value maps directly to the container’s CPU limit option. The runtime does not consider CPU requests: the kernel translates requests into a share weight, not a ceiling, so feeding it into the formula would be semantically wrong. This is the point that causes the most confusion when reading the docs.

The runtime also periodically updates GOMAXPROCS when the number of available logical cores or the cgroup limit changes. From 1.25, GOMAXPROCS is no longer a process-wide constant but a value that can change while the program runs.

Both behaviors are automatically disabled if GOMAXPROCS is set manually, through the environment variable or through a call to runtime.GOMAXPROCS; they can also be turned off explicitly with GODEBUG=containermaxprocs=0 and GODEBUG=updatemaxprocs=0. The practical consequence: a GOMAXPROCS variable left in a manifest, maybe from the days when you had to set it by hand, quietly puts you back on the old behavior, with no warning.

The new behavior is tied to the go directive in go.mod, not to the toolchain you build with: just set Go 1.25.0 or later there and you get it. In other words, “Go 1.25 is installed” isn’t enough. Check go.mod.

There’s a design consequence that gets little attention: code that reads GOMAXPROCS once at init and uses it to shard data or size a worker pool now rests on a value that can change. On Go 1.25, the right approach is to read runtime.GOMAXPROCS(0) at the moment you need it.

8. Operations: what to change in manifests and alerts

First, set the CPU request equal to the CPU limit: the pod gets the Guaranteed QoS class and the scheduler only places it on a node with real room. With Burstable, the pod runs on an idle node and rarely gets throttled, but under contention it’s squeezed by weight and can be throttled heavily. If you can’t commit to that, the healthier choice is to drop the limit and let requests be the only signal.

Second, on Go 1.25 or later, stop setting GOMAXPROCS by hand: audit manifests, Helm values, environment variables in the pipeline, and any numbers left in startup commands. A single leftover variable switches off the new behavior with no signal at all.

Third, move alerting to throttling: alert when the throttled-period share exceeds 10% over fifteen minutes, and when the average throttled time per period exceeds a few milliseconds. Utilization is still useful, but for a different purpose: deciding whether to raise or lower the limit.

There are still cases where setting it by hand is right: multi-container pods sharing one cgroup (sidecars share the quota, so the real ceiling is smaller than the pod’s limit); purely CPU-bound workloads that need fewer Ps to cut scheduling overhead; GC tuning, since the number of mark workers is tied to GOMAXPROCS; and non-Linux environments, where the new behavior doesn’t apply.

One more spot: liveness probes get throttled too. A probe that costs a few milliseconds of CPU can blow past a one-second timeout while the pod is frozen, the kubelet restarts the pod, and the loop keeps spinning.

9. Which metric measures the actual enforcement mechanism

A health metric should measure the same mechanism the system uses to enforce the limit. CPU utilization is an average over a window you choose; the cgroup quota is a hard ceiling per period enforced by the kernel. The two can drift very far apart, and when they do, only one of them tells you why users are waiting.

The same goes for memory limits and OOM kills, connection limits and pool exhaustion, or IOPS and block queues. An average is always a lagging indicator. For Go on Kubernetes, since 1.25 the runtime reads the container’s real ceiling by itself. What’s left is to go through your manifests and check whether any old config line is disabling it.