A developer recently documented a production issue caused by enabling swap memory to handle memory spikes in a system running Go processes. The problem centered on Go’s garbage collector being forced to read metadata from swap storage during stop-the-world pauses, causing dramatic latency increases.
In the test setup, a cgroup contained two processes: a Go application performing io.ReadAll and proto.Unmarshal operations to create data structures, and an HTTP server that remained mostly idle. The developer initially hypothesized that with swap enabled, both processes’ memory pages would be evicted roughly equally, limiting the impact. This assumption proved incorrect.
According to the analysis, Go’s garbage collector reads its runtime metadata—stored outside the heap in a region never freed—during stop-the-world pauses. When the kernel evicts pages to swap based on age, these metadata pages can end up on disk. When the GC attempts to read them, page faults occur, forcing the kernel to retrieve them from swap.
Testing on a Hetzner box running kernel 6.8 with MGLRU enabled revealed the severity. Without swap, the median pause was around 51 microseconds. With metadata on NVMe swap storage, the worst-case pause reached 40 milliseconds. Using a BPF script to measure page faults during the worst pause, the developer found 228 faults occurring within 39 milliseconds, with the faults happening during GC bookkeeping operations in runtime functions like spanSet.reset and finishsweep_m.
The developer emphasized that while 40ms might seem minor in isolation, it represents a complete stop-the-world event. During such a pause, every processor (P) in Go’s runtime is blocked. Any goroutines waiting for I/O cannot process completed requests, and the pause was 800 times the median duration. The testing showed two to three such pauses per memory spike.
Additionally, message building operations that normally took 3-5 milliseconds jumped to 105ms on NVMe and 903ms on network-attached storage. While only individual goroutines paid this cost rather than the entire system, it added significant overhead.
The developer noted that Go 1.26’s Green Tea garbage collector showed negligible improvement in how it handles metadata reads from swap. The findings highlight a specific failure mode when combining swap storage with Go’s garbage collection patterns in production environments.
Key facts
- Go GC stop-the-world pauses increased from 51 microseconds to 40 milliseconds when metadata pages were swapped to disk
- 228 page faults during the worst pause accounted for 39 of the 40 milliseconds, occurring during GC bookkeeping operations
- Two to three extreme pauses occurred per memory spike during testing, each 800 times longer than the median pause
- Message building operations slowed from 3-5ms to 105ms (NVMe) or 903ms (network storage) under swap pressure