Repository navigation
Conversation
There was a problem hiding this comment.
📋 Review Summary
This PR fixes a bug where MetadataSpannerConfigFactory copied Code.ABORTED into the retryable codes for the Spanner Change Streams metadata database client, causing Commit RPCs for aborted transactions to be uselessly retried at the gRPC layer until the commit deadline expired instead of immediately propagating to TransactionRunner for transaction replay. The fix and the accompanying mock-service regression test are clean, well-targeted, and directly address the issue.
🔍 General Feedback
- Filtering out
Code.ABORTEDinMetadataSpannerConfigFactoryaccurately scopesABORTEDretries to the primary change stream query config while allowingTransactionRunneron the metadata database to handle aborted read-write transactions as designed. MetadataSpannerConfigFactoryTesteffectively validates the end-to-end interaction betweenMetadataSpannerConfigFactory,SpannerAccessor,TransactionRunner, andMockSpannerServiceImpl.
💡 Tip: Comment /gemini review on this pull request to trigger a new Gemini code review.
|
Assigning reviewers: R: @nielm for label spanner. Note: If you would like to opt out of this review, comment Available commands:
The PR bot will only process comments in the main thread (not review comments). |
1ceb3a5 to
d32e7f0
Compare
SpannerIO.readChangeStream() adds ABORTED to its retryable codes for its streaming queries. The metadata config copied those codes, and they also apply to Commit. An aborted metadata commit was then resent with the same transaction ID until retries ran out, instead of being returned to TransactionRunner. One aborted transaction took about 50 seconds to recover. If the last attempt started with only milliseconds of the 60 second commit deadline left, it failed with DEADLINE_EXCEEDED, which TransactionRunner does not replay. Remove ABORTED from the metadata config. The primary config keeps it for change stream queries.
d32e7f0 to
f26c15f
Compare
SpannerIO.readChangeStream()addsABORTEDto its retryable codes for its streaming change stream queries.MetadataSpannerConfigFactorycopied those codes to the metadata connection, where they also apply toCommit. When Spanner aborted a metadata transaction, the client resent the commit with the same aborted transaction ID until retries ran out, instead of returning the error so thatTransactionRunnercould run the transaction again.With the real Spanner client and a mock server, one aborted transaction took about 50 seconds to recover and sent about 20 commit RPCs. Sometimes the last attempt starts with only milliseconds of the 60 second commit deadline left. It then fails with
DEADLINE_EXCEEDED, whichTransactionRunnerdoes not replay, so the pipeline fails.This matches the
SpannerChangeStreamPlacementTableITfailures in the GCP IO precommit (example, example). They reportCallOptions deadline exceeded after 0.009149773swith a request ID on attempt 20 or 21, and the failed tests take about 66 to 69 seconds. The CI logs do not include the status of the earlier attempts, soABORTEDas the trigger is inferred rather than observed. RepeatedUNAVAILABLEcould use up the same deadline, and this change does not address that case.This change removes
ABORTEDfrom the metadata config. The primary config keeps it for change stream queries, andUNAVAILABLEis still retried on both.With 250 ms of simulated commit latency on the mock server, the old code reproduced the CI failure, with about 8 ms left on the last attempt. With this change, an aborted transaction recovers in about 0.5 seconds with one replay. The new regression test fails before the fix and passes after it. The Spanner unit tests, Spotless, and Checkstyle ran locally. The change stream integration tests need GCP, so they were not run locally. The GCP IO precommit on this PR runs them.
./gradlew :sdks:java:io:google-cloud-platform:test --tests '*MetadataSpannerConfigFactoryTest*'