Repository navigation
Conversation
* homa_rpc_reap takes RPCs off dead_rpcs under the socket lock and frees them after unlocking; since d00a93b that is also when it subtracts their frag_bytes from sk_wmem_alloc. homa_sock_destroy only waited for dead_rpcs to be empty (its comment wrongly said that all references were gone), so while homa_timer or softirq still freed the last batch it hit WARN_ON_ONCE(sk_wmem_alloc != 1) and freed the buffer pool under the reaper. * Count the reaps in progress in hsk->reapers. homa_sock_destroy now increments protect_count so that no new reap starts, and waits for hsk->reapers to reach zero before it checks sk_wmem_alloc and frees the pool.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
homa_sock_destroywaits only untildead_rpcsis empty. Buthoma_timeror SoftIRQ can still free the last batch that it took off the list. That reaper returns the buffers to the pool and subtracts theirfrag_bytesfromsk_wmem_alloc. Destroy then hitsWARN_ON_ONCE(sk_wmem_alloc != 1)and frees the pool under that reaper.This patch counts the reaps in progress in
hsk->reapers. Destroy incrementsprotect_count, so a new reap returns at once, and then waits untilreapersis 0.Test on two xl170 nodes: 16 threads each keep 150 8 KB RPCs in flight, then all close their sockets together. On upstream, destroy found
sk_wmem_alloc != 1in 318 of 128000 closes. In one run,homa_timerthen oopsed inhoma_pool_free_bufs(NULLbuffer_pool) and 2 sockets leaked. With this patch: 0 of 144000. cp_node throughput does not change.Reproduction, logs and the oops: https://github.com/breakertt/HomaModule/tree/fix-artifact/destroy-wait-for-reapers