Skip to content

Port ElasticsearchIOTest off DoFnTester - #41215

Open
harshilv17 wants to merge 1 commit into
apache:masterfrom
harshilv17:fix/18688-elasticsearch-off-dofntester
Open

harshilv17 wants to merge 1 commit into
apache:masterfrom
harshilv17:fix/18688-elasticsearch-off-dofntester

Conversation

@harshilv17

Copy link
Copy Markdown
Contributor

Fixes #18688.

Ports the Elasticsearch write tests in ElasticsearchIOTestCommon off the deprecated DoFnTester.

TestPipeline doesn't fit here either: as the existing comments say, bundle size is the runner's decision, and testWriteWithMaxBatchSize / testWriteWithMaxBatchSizeBytes check how many documents Elasticsearch holds between elements of the same bundle.

So, like the DatastoreV1Test (#40866) and BatchViewOverridesTest (#41202) ports, the four tests now use a small BulkIOBundleFnRunner that runs BulkIOBundleFn in-process through DoFnRunners.simpleRunner. What differs from those helpers:

  • BulkIOBundleFn creates its REST client in @Setup and closes it in @Teardown, so the helper invokes both through DoFnInvokers.
  • Like DoFnTester, a bundle starts on the first element and finishes at the end of processBundle or on close(), so the tests that feed elements one at a time keep their semantics.
  • It unwraps UserCodeException, so testWriteWithErrors still matches the fn's own exception message.

elasticsearch-tests-common gains an implementation dependency on :runners:core-java for DoFnRunners. The test assertions are unchanged.

Ran locally against Elasticsearch 8 (testcontainers): testWriteWithErrors, testWriteWithAllowableErrors, testWriteWithMaxBatchSize and testWriteWithMaxBatchSizeBytes pass. spotlessApply and checkstyleMain are clean.

  • Small/focused contribution (ICLA not required).
  • Mentions the issue (Fixes #18688).
  • CHANGES.md: not needed, test-only change.

The write tests drive BulkIOBundleFn outside of a runner, because bundle
size is the runner's decision and the batch-size tests check what has
been written to Elasticsearch between elements of one bundle.

Replace DoFnTester with a small BulkIOBundleFnRunner in
ElasticsearchIOTestCommon that runs the fn through
DoFnRunners.simpleRunner, as the DatastoreV1Test and
BatchViewOverridesTest ports do. Unlike those fns, BulkIOBundleFn opens
its client in @setup and closes it in @teardown, so the helper invokes
both, and like DoFnTester it starts a bundle on the first element and
finishes it at the end of processBundle or on close. It unwraps
UserCodeException so testWriteWithErrors still sees the fn's own
exception.

Fixes apache#18688
@github-actions github-actions Bot added java io elasticsearch size/m M: 50-249 lines changed labels Nov 11, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📋 Review Summary

This PR cleanly ports the four BulkIO.BulkIOBundleFn tests in ElasticsearchIOTestCommon off the deprecated DoFnTester by introducing a focused BulkIOBundleFnRunner helper backed by DoFnRunners.simpleRunner and DoFnInvokers. Lifecycle handling (@Setup, @StartBundle, @FinishBundle, and @Teardown in a finally block) and exception unwrapping preserve existing test semantics.

🔍 General Feedback

  • Calling invoker.invokeTeardown() inside a finally block in BulkIOBundleFnRunner.close() is a nice improvement over DoFnTester, ensuring the Elasticsearch RestClient is always closed even when finishBundle() throws an exception (such as in testWriteWithErrors).
  • The changes are well-scoped and consistent with similar DoFnTester migration PRs.

💡 Tip: Comment /gemini review on this pull request to trigger a new Gemini code review.

@github-actions

Copy link
Copy Markdown
Contributor

Assigning reviewers:

R: @ahmedabu98 for label java.
Backup reviewer: Abacn (if the main reviewer is unresponsive, you can escalate to them by tagging their username)

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

  • stop reviewer notifications - opt out of the automated review tooling
  • remind me after tests pass - tag the comment author after tests pass
  • waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Port ElasticSearchIOTest off DoFnTester

1 participant