Repository navigation
Port ElasticsearchIOTest off DoFnTester - #41215
harshilv17 wants to merge 1 commit into
Conversation
The write tests drive BulkIOBundleFn outside of a runner, because bundle size is the runner's decision and the batch-size tests check what has been written to Elasticsearch between elements of one bundle. Replace DoFnTester with a small BulkIOBundleFnRunner in ElasticsearchIOTestCommon that runs the fn through DoFnRunners.simpleRunner, as the DatastoreV1Test and BatchViewOverridesTest ports do. Unlike those fns, BulkIOBundleFn opens its client in @setup and closes it in @teardown, so the helper invokes both, and like DoFnTester it starts a bundle on the first element and finishes it at the end of processBundle or on close. It unwraps UserCodeException so testWriteWithErrors still sees the fn's own exception. Fixes apache#18688
There was a problem hiding this comment.
📋 Review Summary
This PR cleanly ports the four BulkIO.BulkIOBundleFn tests in ElasticsearchIOTestCommon off the deprecated DoFnTester by introducing a focused BulkIOBundleFnRunner helper backed by DoFnRunners.simpleRunner and DoFnInvokers. Lifecycle handling (@Setup, @StartBundle, @FinishBundle, and @Teardown in a finally block) and exception unwrapping preserve existing test semantics.
🔍 General Feedback
- Calling
invoker.invokeTeardown()inside afinallyblock inBulkIOBundleFnRunner.close()is a nice improvement overDoFnTester, ensuring the ElasticsearchRestClientis always closed even whenfinishBundle()throws an exception (such as intestWriteWithErrors). - The changes are well-scoped and consistent with similar
DoFnTestermigration PRs.
💡 Tip: Comment /gemini review on this pull request to trigger a new Gemini code review.
|
Assigning reviewers: R: @ahmedabu98 for label java. Note: If you would like to opt out of this review, comment Available commands:
The PR bot will only process comments in the main thread (not review comments). |
Fixes #18688.
Ports the Elasticsearch write tests in
ElasticsearchIOTestCommonoff the deprecatedDoFnTester.TestPipelinedoesn't fit here either: as the existing comments say, bundle size is the runner's decision, andtestWriteWithMaxBatchSize/testWriteWithMaxBatchSizeBytescheck how many documents Elasticsearch holds between elements of the same bundle.So, like the
DatastoreV1Test(#40866) andBatchViewOverridesTest(#41202) ports, the four tests now use a smallBulkIOBundleFnRunnerthat runsBulkIOBundleFnin-process throughDoFnRunners.simpleRunner. What differs from those helpers:BulkIOBundleFncreates its REST client in@Setupand closes it in@Teardown, so the helper invokes both throughDoFnInvokers.DoFnTester, a bundle starts on the first element and finishes at the end ofprocessBundleor onclose(), so the tests that feed elements one at a time keep their semantics.UserCodeException, sotestWriteWithErrorsstill matches the fn's own exception message.elasticsearch-tests-commongains animplementationdependency on:runners:core-javaforDoFnRunners. The test assertions are unchanged.Ran locally against Elasticsearch 8 (testcontainers):
testWriteWithErrors,testWriteWithAllowableErrors,testWriteWithMaxBatchSizeandtestWriteWithMaxBatchSizeBytespass.spotlessApplyandcheckstyleMainare clean.Fixes #18688).CHANGES.md: not needed, test-only change.