Repository navigation
Distributor: fix goroutine leak merging label results on canceled queries - #7933
Open
amalsaidov wants to merge 1 commit into
Open
amalsaidov wants to merge 1 commit into
amalsaidov wants to merge 1 commit into
Conversation
…ries MergeSlicesParallel splits the per-ingester label name and label value slices into batches and merges them concurrently. It keeps only the first error, over a one-slot channel, but it sent to that channel with a blocking send. When a query is canceled, more than one batch can observe the canceled context and return an error at the same time. The first send fills the channel, the receiver takes that error and returns, and every other worker blocks forever on its send. The goroutine that waits on the WaitGroup before closing the channels then never returns either, so the whole set leaks for the lifetime of the process. Send the error without blocking so a worker that cannot report its error just drops it and exits. The first error is still the one returned. Add a regression test that cancels the context up front and uses goleak to assert no goroutine is left behind. Signed-off-by: amalsaidov <[email protected]>
amalsaidov
force-pushed
the
fix/merge-slices-parallel-goroutine-leak
branch
from
November 13, 2026 11:56
b63d74f to
7a37c90
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does:
MergeSlicesParallelmerges the per-ingester label name and label value slices concurrently. The distributor calls it with parallelism 8 on everyLabelValuesandLabelNamesquery, so the slices are split into several batches merged by separate goroutines.The function keeps only the first error, over a one-slot channel, but it sent to that channel with a blocking send:
When a query is canceled, more than one batch can see the canceled context and return an error at the same time (each batch checks
ctx.Err()once it has mergedCheckContextEveryNIterations= 128 elements). The first send fills the channel, the receiver takes that error and returns, and every other worker that also errored blocks forever on its send. The goroutine that waits on theWaitGroupbefore closing the channels then never returns either, so the whole set leaks for the lifetime of the process. Canceled or timed-out label queries are routine, so this leaks goroutines steadily in production.The fix sends the error without blocking, so a worker that cannot report its error drops it and exits. The first error is still the one returned, and the successful path is unchanged.
The new regression test cancels the context up front and uses
goleakto assert nothing is left behind. It fails before the change (two workers parked onchan sendplus theWaitGroupwaiter) and passes after.Which issue(s) this PR fixes:
No issue filed; found while reading the label-query merge path.
Checklist
CHANGELOG.mdupdateddocs/configuration/v1-guarantees.mdupdated if this PR introduces experimental flagsAI-assisted; directed, reviewed and tested by the author (amalsaidov).