Skip to content

[Spanner Change Streams] Replay aborted metadata transactions - #40937

Open
bvolpato wants to merge 3 commits into
apache:masterfrom
bvolpato:bvolpato/spanner-metadata-aborted-commit
Open

bvolpato wants to merge 3 commits into
apache:masterfrom
bvolpato:bvolpato/spanner-metadata-aborted-commit

Conversation

@bvolpato

@bvolpato bvolpato commented Nov 2, 2026

Copy link
Copy Markdown
Contributor

SpannerIO.readChangeStream() adds ABORTED to its retryable codes for its streaming change stream queries. MetadataSpannerConfigFactory copied those codes to the metadata connection, where they also apply to Commit. When Spanner aborted a metadata transaction, the client resent the commit with the same aborted transaction ID until retries ran out, instead of returning the error so that TransactionRunner could run the transaction again.

With the real Spanner client and a mock server, one aborted transaction took about 50 seconds to recover and sent about 20 commit RPCs. Sometimes the last attempt starts with only milliseconds of the 60 second commit deadline left. It then fails with DEADLINE_EXCEEDED, which TransactionRunner does not replay, so the pipeline fails.

This matches the SpannerChangeStreamPlacementTableIT failures in the GCP IO precommit (example, example). They report CallOptions deadline exceeded after 0.009149773s with a request ID on attempt 20 or 21, and the failed tests take about 66 to 69 seconds. The CI logs do not include the status of the earlier attempts, so ABORTED as the trigger is inferred rather than observed. Repeated UNAVAILABLE could use up the same deadline, and this change does not address that case.

This change removes ABORTED from the metadata config. The primary config keeps it for change stream queries, and UNAVAILABLE is still retried on both.

With 250 ms of simulated commit latency on the mock server, the old code reproduced the CI failure, with about 8 ms left on the last attempt. With this change, an aborted transaction recovers in about 0.5 seconds with one replay. The new regression test fails before the fix and passes after it. The Spanner unit tests, Spotless, and Checkstyle ran locally. The change stream integration tests need GCP, so they were not run locally. The GCP IO precommit on this PR runs them.

./gradlew :sdks:java:io:google-cloud-platform:test --tests '*MetadataSpannerConfigFactoryTest*'

@github-actions github-actions Bot added java io gcp spanner size/m M: 50-249 lines changed labels Nov 2, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📋 Review Summary

This PR fixes a bug where MetadataSpannerConfigFactory copied Code.ABORTED into the retryable codes for the Spanner Change Streams metadata database client, causing Commit RPCs for aborted transactions to be uselessly retried at the gRPC layer until the commit deadline expired instead of immediately propagating to TransactionRunner for transaction replay. The fix and the accompanying mock-service regression test are clean, well-targeted, and directly address the issue.

🔍 General Feedback

  • Filtering out Code.ABORTED in MetadataSpannerConfigFactory accurately scopes ABORTED retries to the primary change stream query config while allowing TransactionRunner on the metadata database to handle aborted read-write transactions as designed.
  • MetadataSpannerConfigFactoryTest effectively validates the end-to-end interaction between MetadataSpannerConfigFactory, SpannerAccessor, TransactionRunner, and MockSpannerServiceImpl.

💡 Tip: Comment /gemini review on this pull request to trigger a new Gemini code review.

@github-actions

github-actions Bot commented Nov 2, 2026

Copy link
Copy Markdown
Contributor

Assigning reviewers:

R: @nielm for label spanner.
Backup reviewer: chamikaramj (if the main reviewer is unresponsive, you can escalate to them by tagging their username)

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

  • stop reviewer notifications - opt out of the automated review tooling
  • remind me after tests pass - tag the comment author after tests pass
  • waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

@bvolpato
bvolpato force-pushed the bvolpato/spanner-metadata-aborted-commit branch from 1ceb3a5 to d32e7f0 Compare November 3, 2026 03:48
SpannerIO.readChangeStream() adds ABORTED to its retryable codes for its streaming queries. The metadata config copied those codes, and they also apply to Commit. An aborted metadata commit was then resent with the same transaction ID until retries ran out, instead of being returned to TransactionRunner.

One aborted transaction took about 50 seconds to recover. If the last attempt started with only milliseconds of the 60 second commit deadline left, it failed with DEADLINE_EXCEEDED, which TransactionRunner does not replay.

Remove ABORTED from the metadata config. The primary config keeps it for change stream queries.
@bvolpato
bvolpato force-pushed the bvolpato/spanner-metadata-aborted-commit branch from d32e7f0 to f26c15f Compare November 4, 2026 15:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant