High p99 Latency in Go Service: Identifying and Resolving Bottlenecks to Prevent System Overload

Introduction: The Latency Challenge In distributed systems, p99 latency often emerges as the silent killer of performance, despite healthy p50 and p95 metrics. This phenomenon is particularly acute in Go services, where the request lifecycle—from client initiation to load balancer routing and service processing—can be disrupted by straggler requests. These stragglers, consuming disproportionate resources, act as systemic bottlenecks, delaying subsequent requests and cascading into degraded user experience. The mechanical process here is straightforward: a single slow request, often due to resource contention or downstream dependency issues, holds up the goroutine scheduler, causing a backlog that amplifies tail latency. Retries, a common mitigation strategy, proved ineffective—and in some cases, counterproductive. The causal chain is clear: retries increase load on already stressed resources, triggering retry storms that exacerbate latency. This is particularly evident in Go’s runtime,