Repository navigation
perf(kernel): skip the Python telemetry client on use_kernel connections - #1050
Open
vuanhphung wants to merge 1 commit into
Open
vuanhphung wants to merge 1 commit into
vuanhphung wants to merge 1 commit into
Conversation
The kernel exports telemetry for use_kernel connections, but every kernel connect still registered a no-op client with TelemetryClientFactory under its global lock (and the first one started the flush thread). In a 200-thread cold burst that lock convoys behind the GIL: initialize_telemetry_client took p50 692 ms / max 1.7 s, and connect p50 was ~3.0 s vs ~1.7 s without it. Kernel connections now use a NoopTelemetryClient directly and skip the factory on close. This also stops a kernel connection opened first from leaving a no-op client registered for the host, which silently disabled Python telemetry for later Thrift connections to that host. Co-authored-by: Isaac <[email protected]> Signed-off-by: Vu Anh Phung <[email protected]>
vuanhphung
temporarily deployed
to
azure-prod
November 13, 2026 19:39 — with
GitHub Actions
Inactive
vuanhphung
temporarily deployed
to
azure-prod
November 13, 2026 19:39 — with
GitHub Actions
Inactive
There was a problem hiding this comment.
Verdict: 1 Low
Looks good, with one low concern. Skipping the factory on kernel connections is correct and consistent: the _kernel_owns_telemetry predicate matches the truthiness check in is_telemetry_enabled, the latency logger is already gated, and the tests cover the refcount and the order of kernel and Thrift connections. The concern is that host-keyed failure telemetry raised from exc.Error can still send kernel-connection errors through a co-resident Thrift client.
vuanhphung
had a problem deploying
to
azure-prod
November 13, 2026 20:01 — with
GitHub Actions
Error
3 tasks done
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What type of PR is this?
Description
On
use_kernel=Truethe kernel exports telemetry itself, andis_telemetry_enabledalready returns False for kernel connections. Every kernelconnect()still registered a no-op client withTelemetryClientFactoryunder its global lock, and the first one also started the flush thread, executor, and excepthook. Concurrent connects serialize on that lock behind the GIL. In a 200-thread cold burst,initialize_telemetry_clienttook p50 45 ms / max 120 ms onmain(p50 692 ms / max 1.7 s before #1024).Registering also caused a bug. When a kernel connection was opened first, its no-op client stayed registered for the host, so a later Thrift connection to that host reused it and never sent Python telemetry. When a Thrift connection was opened first, kernel connections sent their connection log through the Thrift client and overwrote its connection parameters.
Kernel connections now use a
NoopTelemetryClientdirectly and skip the factory on close. Connection-failure telemetry is unchanged (still suppressed when the kernel was attempted, still sent for Thrift). Errors raised withhost_urlstill go through the host-keyed lookup inexc.Error, so in a process that also has a Thrift connection to the same host, a kernel connection's error logs can still go out through the Thrift client. That behavior is unchanged frommainand out of scope here.How is this tested?
New unit tests cover: a kernel connect/close leaves the factory untouched, it doesn't change the refcount of a Thrift client for the same host, and a Thrift client created after a kernel connection is a real client. All three fail on
main. The proxy assertions intest_kernel_http_client_lifecycleare replaced, since kernel connections no longer build Python connection parameters. Full unit suite (-m "not realkernel"), the realkernel sets, andtests/e2e/test_kernel_backend.pypass.Perf,
main(0c1b9e8) vs this branch:c800.py, 1 process × 200 sessions × 60 s against a local mock SEA server (40 ms per statement), alternating order, n=3, median [min–max]:Live runs against a shared Reyden warehouse (1 × 200 × 60 s, n=3) showed no regression, but warehouse load from other clients varied too much to resolve a gain this small.
Live benchmark (staging Large Reyden warehouse)
TPC-H Q6 on a Large Real-Time warehouse, 800 sessions (4 processes x 200 threads), cold start, each session querying back to back for 120 s, 16 client cores. Isolated A/B: only this change vs connector main 0c1b9e8 + kernel 1.4.0, 3 interleaved rounds, nothing else on the warehouse. Measured at 508709b. Median (min-max):
No measurable QPS or CPU change; ~10 MB less per process.
SELECT 1 on an X-Large Real-Time warehouse (driver-bound). Q6 above is limited by the warehouse, so the same harness also ran
SELECT 1, where the client is the limit: X-Large warehouse, 4 processes x 200 threads, 120 s, 16 client cores, this change vs connector main 0c1b9e8 + kernel 1.4.0, measured at 508709b, run in rotation with the other factory PRs, 2 rounds. Median (min-max):Average QPS change per round: -1.2%, -1.2%.
This pull request and its description were written by Isaac.