How the Go Scheduler Works: G, M and P Explained With What You See in pprof

Myth: more goroutines means more parallelism. The Go scheduler runs goroutines (G) on OS threads (M) through a fixed number of logical processors (P), so parallelism is capped at GOMAXPROCS.

Each P has a local run queue, backed by a global queue and work stealing between Ps. When a thread blocks in a syscall, its P is handed off. Below, each piece is checked against the runtime source and shown in pprof and go tool trace.

Table of Contents

Why more goroutines don’t mean more parallelism

The belief goes like this: start more goroutines and more work runs at once. Parallelism is capped at GOMAXPROCS, because a thread needs a P to run Go code and there are exactly GOMAXPROCS Ps.

Take ten thousand goroutines doing CPU-bound work on 8 Ps. You get 8 running, 9,992 runnable, and the memory for 10,000 stacks.

The myth holds up better for I/O-bound work. Many goroutines are fine there, since nearly all of them are parked and cost no thread. The rest of this post shows why, piece by piece.

What G, M and P are

The comment at the top of runtime/proc.go defines the model in three lines. G is a goroutine, M is a worker thread (or machine), and P is a processor, “a resource that is required to execute Go code”.

An M needs a P to run Go code, but can block or sit in a syscall without one. The design is Dmitry Vyukov’s go11sched doc.

Letter What it is How many
G A goroutine: a stack (2 KB to start, stackMin = 2048 in stack.go), a program counter and scheduling state As many as you create
M An OS thread managed by the runtime Grows with demand, capped at 10,000 by default (sched.maxmcount)
P A logical processor: a 256-slot run queue, a runnext slot, a timer heap, an allocation cache Exactly GOMAXPROCS

P exists so that scheduler state is, in the comment’s words, “intentionally distributed (in particular, per-P work queues)”. Most scheduling decisions take no global lock.

How run queues pick the next goroutine

Each P has a local run queue, a ring of 256 goroutine pointers (runq [256]guintptr in runtime2.go). Only its owner pushes to it; other Ps pop from it when they steal. There is also one global queue, sched.runq, behind sched.lock.

Then there is runnext, the part people miss. It’s one extra slot per P for a goroutine made runnable by the goroutine currently running. The comment says it “should be run next instead of what’s in runq if there’s time remaining in the running G’s time slice” and “will inherit the time left in the current time slice”.

The reason is locality: a communicate-and-wait pair gets scheduled as a unit. Two things put a goroutine there:

  • a go statement (newproc calls runqput(pp, newg, true)),
  • waking a parked goroutine, for example by sending on a channel it was receiving from (goready calls ready(gp, traceskip, true)).

When the local queue is full, half of it (plus the new goroutine) moves to the global queue in one batch. When it is empty, findRunnable pulls a batch back.

Every 61st scheduling tick checks the global queue first. Otherwise “two goroutines can completely occupy the local runqueue by constantly respawning each other”.

What happens when a goroutine blocks

This drew the most questions at my talk at the first Golang Calgary meetup. Blocking means three different things to the scheduler.

A blocking syscall

When a goroutine enters a syscall, the G and its M go into the kernel together. The P stays attached for now. If the syscall returns quickly, exitsyscall takes the fast path: the M still has its P and nothing else happens.

If it takes long, sysmon notices. Its retake loop checks each P. Sometimes a P sits in the same syscall for over one sysmon tick (at least 20 microseconds) while other work waits. Then sysmon takes the P away and calls handoffp, which gives it to another M when there is work for it.

When the syscall returns, the M tries to reacquire its old P, then any idle P. If neither is free, the goroutine goes to the global queue and the M parks. The thread blocks, and the processor keeps working.

A network read

A read on a net.Conn with nothing to read does not block a thread. The netpoller wraps epoll, kqueue and their equivalents (listed at the top of netpoll.go).

The goroutine parks on the fd’s poll descriptor, and the M and P move on. When the kernel reports the fd ready, netpoll hands the goroutine back to a run queue. A parked goroutine costs its stack and no thread. That’s how a Go server holds thousands of idle connections on a few threads.

A channel operation

These all go through gopark: a receive on an empty channel, a send on a full one, a held sync.Mutex, or a select with nothing ready. The goroutine is recorded on the channel’s wait queue, and the M picks the next goroutine from the P. No thread blocks and no P changes hands.

When the other side arrives, goready puts it in runnext. So a tight ping-pong between two goroutines shares one time slice until sysmon preempts them.

How preemption went from cooperative to signals

Until Go 1.14 the scheduler could only preempt at a function call, because the check lived in the function prologue. A loop with no calls could not be preempted.

The Go 1.14 release notes changed that: “Goroutines are now asynchronously preemptible. As a result, loops without function calls no longer potentially deadlock the scheduler or significantly delay garbage collection.”

The mechanism is a signal. On Unix the runtime sends SIGURG to the thread running the goroutine (sigPreempt = _SIGURG in signal_unix.go). The handler switches it out at a safe point. The same notes warn that programs now see more EINTR errors from slow syscalls and must retry them.

The time slice is 10 ms. forcePreemptNS in proc.go is “the time slice given to a G before it is preempted”, and retake calls preemptone on any P that has run past it.

How idle threads steal work

A P may find nothing locally, in the global queue or in the netpoller. Then its M becomes a spinning thread and calls stealWork.

It makes up to four passes over the other Ps in random order (stealTries = 4), skipping idle ones. runqsteal takes half of a victim’s local queue. Only on the last pass does it also take timers and the victim’s runnext, which the comment calls “the last resort”.

Spinning is bounded. findRunnable lets an M spin only if 2*nmspinning < gomaxprocs - npidle, limiting spinning Ms “to half the number of busy Ps”.

The design comment explains why spinning exists at all. Handing a readied goroutine straight to a woken thread was rejected. That thread “can be out of work the very next moment”, and the handoff “would destroy locality of computation”.

In a CPU profile, spinning shows up as runtime.findRunnable, runtime.stealWork and runtime.runqsteal. A few percent is normal. A large share means Ps keep starving and then fighting for scraps.

What sysmon does without a P

sysmon is a runtime thread that runs without a P and never executes user code. Its loop sleeps 20 microseconds between iterations, starts doubling after 50 idle iterations, and caps at 10 ms.

Each iteration it:

  • polls the network if nobody has for over 10 ms,
  • runs retake, handing off Ps stuck in syscalls and preempting goroutines past their slice,
  • forces a GC if none ran in two minutes,
  • since Go 1.25, re-checks the GOMAXPROCS default about once a second.

sysmon is also what makes runnext safe. Without preemption, two goroutines sharing a slice would starve everyone else. So on platforms with no sysmon, the runtime does not use runnext at all.

How to set GOMAXPROCS in containers

GOMAXPROCS is the number of Ps. The runtime docs describe it as limiting “the number of operating system threads that can execute user-level Go code simultaneously”. Threads blocked in syscalls “do not count against the GOMAXPROCS limit”.

For years the default was runtime.NumCPU(). Picture a container with a CPU limit of 2 on a 64-core node. That meant 64 Ps sharing two CPUs’ worth of quota, and the kernel throttling the process whenever it ran out. The usual fix was Uber’s automaxprocs, which sets GOMAXPROCS from the container CPU quota.

Go 1.25 changed the default. From the Go 1.25 release notes: “On Linux, the runtime considers the CPU bandwidth limit of the cgroup containing the process, if any. If the CPU bandwidth limit is lower than the number of logical CPUs available, GOMAXPROCS will default to the lower limit.”

This maps to the Kubernetes CPU limit; the runtime “does not consider the ‘CPU requests’ option”. It also “periodically updates GOMAXPROCS” when the limit changes. Both switch off if you set GOMAXPROCS yourself, or with GODEBUG=containermaxprocs=0 and GODEBUG=updatemaxprocs=0.

So on Go 1.25 or later, in a container with a CPU limit, you probably do not need automaxprocs. With requests only and no limit, the runtime still sees every core on the node. More on that in Go microservices in production.

How to see the scheduler in pprof and go tool trace

A CPU profile tells you where the program burns cycles. It can’t tell you why a goroutine waited 40 ms for a P. For that you need the execution tracer, “a tool to detect latency and utilization problems” in the words of the Go diagnostics guide.

This program starts a trace, launches 64 CPU-bound goroutines and collects their results.

package main

import (
	"log"
	"os"
	"runtime/trace"
	"sync"
)

func burn(n int) int {
	s := 0
	for i := 0; i < n; i++ {
		s += i % 7
	}
	return s
}

func main() {
	f, err := os.Create("trace.out")
	if err != nil {
		log.Fatal(err)
	}
	defer f.Close()

	if err := trace.Start(f); err != nil {
		log.Fatal(err)
	}
	defer trace.Stop()

	results := make(chan int, 64)
	var wg sync.WaitGroup
	for i := 0; i < 64; i++ {
		wg.Add(1)
		go func() {
			defer wg.Done()
			results <- burn(20_000_000)
		}()
	}
	wg.Wait()
	close(results)

	total := 0
	for r := range results {
		total += r
	}
	log.Println("total:", total)
}
go run . && go tool trace trace.out

The viewer’s landing page is documented in traceviewer/http.go. The parts I use:

  • “View trace by proc” draws one lane per P, “showing which goroutine (if any) was running on that logical processor at each moment”. With 64 goroutines and 8 Ps you see 8 lanes filled with short bars, and gaps are idle Ps. That is your parallelism, read directly.
  • “Goroutine analysis” groups goroutines by start location. It splits each one’s time into execution, “Sched wait time” (runnable but waiting for a P), syscall time, and “Block time” by reason. If sched wait dominates, you have more runnable goroutines than Ps.
  • Four profiles (network blocking, synchronization blocking, syscall, scheduler latency) cover “causes of delay that prevent a goroutine from running on a logical processor”. go tool trace -pprof=sched trace.out > sched.pprof writes one for pprof. For the program above it is tiny; swap burn for I/O and the sync and net profiles carry the story.

Tracing is cheap enough for short windows in production. The Go team’s post on execution traces puts the overhead at “1–2% for many applications” since Go 1.21. net/http/pprof serves live captures at /debug/pprof/trace. Go 1.25 added runtime/trace.FlightRecorder, a ring buffer you snapshot after a slow request.

In a plain CPU profile the scheduler appears as runtime.schedule, runtime.findRunnable, runtime.mcall and runtime.futex (Ms parking and unparking on Linux). The goroutine profile lists every goroutine with its wait reason in brackets, like [chan receive], [IO wait] or [syscall].

What to do instead of adding goroutines

  • For CPU-bound work, a worker pool the size of GOMAXPROCS is usually right.
  • For I/O-bound work, many goroutines are fine, since nearly all are parked.
  • If the trace shows sched wait dominating, you have more runnable goroutines than Ps.
  • If goroutines are runnable but not running, check GOMAXPROCS against the container’s CPU limit.

Two more GOMAXPROCS misreadings

Raising GOMAXPROCS above the CPU count does not speed up CPU-bound code. The extra Ps share the same cores and now also contend. It can help with heavy cgo or blocking-syscall work, and handoff usually covers that already.

GOMAXPROCS=1 does not make code free of data races. Goroutines still interleave at every channel or mutex operation and, since 1.14, at almost any instruction.

FAQ

What is the difference between G, M and P?

G is a goroutine, and M is an OS thread managed by the runtime. P is a logical processor that an M must hold to run Go code. There are as many Gs as you create, as many Ms as blocking work demands, and exactly GOMAXPROCS Ps.

How many OS threads does Go use?

At least one per P plus sysmon. More appear when goroutines block in syscalls or cgo. The runtime hands the P to another thread while the blocked one stays in the kernel. The default cap is 10,000; debug.SetMaxThreads changes it.

Is the Go scheduler preemptive?

Yes, since Go 1.14, whose release notes state goroutines are “asynchronously preemptible”. On Unix it works by sending SIGURG to the thread. Before 1.14 preemption only happened at function calls, so a loop with no calls could hold a P indefinitely.

What does GOMAXPROCS do?

It sets the number of Ps, which is the maximum number of threads running Go code at once; threads blocked in syscalls do not count. Since Go 1.25 the default on Linux respects the cgroup CPU limit. Before that it was the logical CPU count, which oversubscribed CPU-limited containers.

Do more goroutines make Go code run faster?

Not for CPU-bound work. Parallelism is capped at GOMAXPROCS, so extra goroutines wait as runnable and still cost their stacks. For I/O-bound work many goroutines are fine, since nearly all are parked.

Why is my CPU not fully used with many goroutines?

Most of them are waiting on something other than the CPU: a lock, a channel, the network or a syscall. go tool trace shows this as idle gaps in the per-P lanes and as block time by cause. If they are runnable but not running, check GOMAXPROCS against the container’s CPU limit.