Skip to content

Pass explicit timestamps for DirectRow.commit setup writes in Bigtable ITs - #41231

Draft
damccorm wants to merge 1 commit into
apache:masterfrom
damccorm:fix-bigtable-it-idempotent-setup-commit
Draft

damccorm wants to merge 1 commit into
apache:masterfrom
damccorm:fix-bigtable-it-idempotent-setup-commit

Conversation

@damccorm

@damccorm damccorm commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Summary

In google-cloud-bigtable >= 2.44.0 (upgraded in #40042 / #40960), DirectRow.commit() delegates to _DataApiTarget.mutate_row(row_key, mutations), which only retries transient DeadlineExceeded and ServiceUnavailable errors when all(m.is_idempotent() for m in mutations) is True.

When DirectRow.set_cell() is called without an explicit timestamp, it sets timestamp_micros = -1 (_SERVER_SIDE_TIMESTAMP), causing SetCell.is_idempotent() to return False and disabling retries completely. If the single 20-second MutateRow attempt times out while a newly created Bigtable instance/table/column family is warming up (as seen in TestWriteToBigtableXlangIT.test_delete_cells_mutation in run 43118590693), DirectRow.commit() catches GoogleAPICallError, clears self._mutations in finally, and returns a non-OK Status without raising an exception—leaving the table empty and causing actual_row = self.table.read_row('key-1') to return None (AttributeError: 'NoneType' object has no attribute 'find_cells').

Changes

  • Pass an explicit datetime.now(timezone.utc) timestamp to set_cell() in pre-test setup writes in bigtableio_it_test.py so SetCell.is_idempotent() is True and DirectRow.commit() retries transient DeadlineExceeded/ServiceUnavailable errors during Bigtable instance/table warmup.
  • Assert row.commit().code == 0 on pre-test setup commits so any setup failure surfaces immediately with its gRPC status code.

Verification

@github-actions github-actions Bot added python io gcp size/s S: 10-49 lines changed labels Nov 12, 2026
@damccorm damccorm added the test-failure-cleanup Automated test health and flake cleanup PRs label Nov 12, 2026
@damccorm

Copy link
Copy Markdown
Contributor Author

/gemini review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📋 Review Summary

This PR resolves flakiness in Bigtable integration tests (bigtableio_it_test.py) caused by google-cloud-bigtable >= 2.44.0 disabling retries on DirectRow.commit() when set_cell() omits an explicit timestamp. By passing datetime.now(timezone.utc) to setup set_cell() calls and asserting row.commit().code == 0, setup mutations become idempotent (enabling automatic retries on transient warmup errors) and fail fast if a commit still does not succeed.

🔍 General Feedback

  • The root cause analysis and fix are clean, targeted, and consistent with existing patterns in bigtableio_it_test.py.
  • Adding self.assertEqual(0, ...commit().code) on all pre-test setup writes will significantly improve debuggability if non-retryable or persistent gRPC errors occur during test setup.

💡 Tip: Comment /gemini review on this pull request to trigger a new Gemini code review.

@codecov

codecov Bot commented Nov 12, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 57.76%. Comparing base (1878a5b) to head (68bc9f3).

Additional details and impacted files
@@            Coverage Diff            @@
##             master   #41231   +/-   ##
=========================================
  Coverage     57.75%   57.76%           
  Complexity     2617     2617           
=========================================
  Files          1202     1202           
  Lines        171679   171679           
  Branches       2163     2163           
=========================================
+ Hits          99159    99166    +7     
+ Misses        69862    69855    -7     
  Partials       2658     2658           
Flag Coverage Δ
python 80.55% <ø> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

gcp io python size/s S: 10-49 lines changed test-failure-cleanup Automated test health and flake cleanup PRs

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant