Fix PostCommit Python Xlang Gcp Direct job by Amar3tto · Pull Request #38328 · apache/beam

Amar3tto · 2026-04-29T18:36:42Z

Fixes #35285
Successful run: https://github.com/apache/beam/actions/runs/25123436123

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

gemini-code-assist · 2026-04-29T18:47:56Z

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request improves the reliability of integration tests by ensuring that tests requiring specific Dataflow runner capabilities are skipped when executed in environments where those capabilities are not available. This prevents false negatives in CI pipelines and ensures that tests only run when the necessary infrastructure is present.

Highlights

Test Stability: Added conditional skipping of integration tests that require the Dataflow runner when running on other environments.
Runner Detection: Updated test setup to correctly identify the active runner from pipeline options, enabling accurate conditional test execution.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

Abacn · 2026-04-29T18:48:34Z

Is this caused by #38058? It appears BigQueryIO storage write can no longer run on Direct/Prism runner. It sounds a regression

cc: @ahmedabu98 @reuvenlax is it possible to avoid OnWindowExpiration with some option or, when the runner isn't supported

gemini-code-assist

Code Review

This pull request introduces a mechanism to skip integration tests that require the Dataflow runner, specifically for features like 'on_window_expiration'. This logic was applied to BigQuery and BigTable integration tests. A critical issue was identified in xlang_bigqueryio_it_test.py where the skip_if_not_dataflow_runner method is invoked but not defined, which will cause an AttributeError. Additionally, feedback suggests using self.test_pipeline.get_option('runner') for more idiomatic and safer runner detection instead of manually parsing PipelineOptions, and recommends refactoring the duplicated skip logic into a shared utility module.

gemini-code-assist · 2026-04-29T18:50:09Z

    hamcrest_assert(p, bq_matcher)

  def test_all_types(self):
+    self.skip_if_not_dataflow_runner()


The method skip_if_not_dataflow_runner is called here, but it is not defined in this class or file, unlike in the other test files modified in this PR. Additionally, self._runner is not initialized in this class. This will lead to an AttributeError when the tests are executed. Please ensure the method and its required state are correctly implemented or inherited.

gemini-code-assist · 2026-04-29T18:50:09Z

 from apache_beam.io.gcp.bigquery_tools import BigQueryWrapper
 from apache_beam.io.gcp.internal.clients import bigquery
 from apache_beam.io.gcp.tests.bigquery_matcher import BigqueryFullResultMatcher
+from apache_beam.options.pipeline_options import PipelineOptions


This import is unnecessary if self.test_pipeline.get_option('runner') is used instead of manually instantiating PipelineOptions in setUp.

gemini-code-assist · 2026-04-29T18:50:09Z

+    self.args = self.test_pipeline.get_full_options_as_args()
    self.runner_name = type(self.test_pipeline.runner).__name__
    self.project = self.test_pipeline.get_option('project')
+    self._runner = PipelineOptions(self.args).get_all_options()['runner']


Instead of manually parsing PipelineOptions, use self.test_pipeline.get_option('runner'). This is safer and more idiomatic. Furthermore, self.runner_name is already initialized on line 59 and could be used for the runner check, making self._runner and the additional self.args assignment on line 58 redundant.

Suggested change

self._runner = PipelineOptions(self.args).get_all_options()['runner']

self._runner = self.test_pipeline.get_option('runner')

gemini-code-assist · 2026-04-29T18:50:09Z


    hc.assert_that(p, hc.all_of(*pipeline_verifiers))

+  def skip_if_not_dataflow_runner(self) -> bool:


The skip_if_not_dataflow_runner method is duplicated in sdks/python/apache_beam/io/gcp/bigtableio_it_test.py. Consider refactoring this logic into a shared utility module or a common base class for integration tests to improve maintainability.

gemini-code-assist · 2026-04-29T18:50:09Z

    self.test_pipeline = TestPipeline(is_integration_test=True)
    self.args = self.test_pipeline.get_full_options_as_args()
    self.project = self.test_pipeline.get_option('project')
+    self._runner = PipelineOptions(self.args).get_all_options()['runner']


Directly accessing 'runner' from the options dictionary can be unsafe. It is recommended to use self.test_pipeline.get_option('runner'), which handles missing options gracefully and is more consistent with Beam's test patterns.

Suggested change

self._runner = PipelineOptions(self.args).get_all_options()['runner']

self._runner = self.test_pipeline.get_option('runner')

reuvenlax · 2026-04-29T19:03:27Z

Does Prism not support OnWindowExpiration? If so that's a major gap - in many cases there's no way to avoid data loss without OnWindowExpiration. Also, BigQueryIO already used OnWindowExpiration before (e.g. in StorageApiWriteShardedRecords).

…

On Wed, Apr 29, 2026 at 11:48 AM Yi Hu ***@***.***> wrote: *Abacn* left a comment (apache/beam#38328) <#38328 (comment)> Is this caused by #38058 <#38058>? It appears BigQueryIO storage write can no longer run on Direct/Prism runner. It sounds a regression cc: @ahmedabu98 <https://github.com/ahmedabu98> @reuvenlax <https://github.com/reuvenlax> is it possible to avoid OnWindowExpiration with some option or, when the runner isn't supported — Reply to this email directly, view it on GitHub <#38328 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFAYJVLRRNOZVCUDZPNX2KD4YJFBRAVCNFSM6AAAAACYLE7DECVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHM2DGNBWGYYDONZWHE> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you were mentioned.Message ID: ***@***.***>

reuvenlax · 2026-04-29T19:08:08Z

GroupIntoBatches also uses OnWindowExpiration.

…

On Wed, Apr 29, 2026 at 12:03 PM Reuven Lax ***@***.***> wrote: Does Prism not support OnWindowExpiration? If so that's a major gap - in many cases there's no way to avoid data loss without OnWindowExpiration. Also, BigQueryIO already used OnWindowExpiration before (e.g. in StorageApiWriteShardedRecords). On Wed, Apr 29, 2026 at 11:48 AM Yi Hu ***@***.***> wrote: > *Abacn* left a comment (apache/beam#38328) > <#38328 (comment)> > > Is this caused by #38058 <#38058>? It > appears BigQueryIO storage write can no longer run on Direct/Prism runner. > It sounds a regression > > cc: @ahmedabu98 <https://github.com/ahmedabu98> @reuvenlax > <https://github.com/reuvenlax> is it possible to avoid OnWindowExpiration > with some option or, when the runner isn't supported > > — > Reply to this email directly, view it on GitHub > <#38328 (comment)>, or > unsubscribe > <https://github.com/notifications/unsubscribe-auth/AFAYJVLRRNOZVCUDZPNX2KD4YJFBRAVCNFSM6AAAAACYLE7DECVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHM2DGNBWGYYDONZWHE> > . > Triage notifications on the go with GitHub Mobile for iOS > <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> > or Android > <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. > > You are receiving this because you were mentioned.Message ID: > ***@***.***> >

Abacn · 2026-04-29T19:13:09Z

  def test_read_xlang(self):
+    self.skip_if_not_dataflow_runner()


also read test should not be affected

reuvenlax · 2026-04-29T19:14:24Z

If Prism doesn't support window expiration we should fix this - it's quite a big problem. For now, we can modify BigQueryIO to only expand this portion of the graph if the user has specified schema-update options. That should be a simple change to make.

…

On Wed, Apr 29, 2026 at 12:07 PM Reuven Lax ***@***.***> wrote: GroupIntoBatches also uses OnWindowExpiration. On Wed, Apr 29, 2026 at 12:03 PM Reuven Lax ***@***.***> wrote: > Does Prism not support OnWindowExpiration? If so that's a major gap - in > many cases there's no way to avoid data loss without OnWindowExpiration. > > Also, BigQueryIO already used OnWindowExpiration before (e.g. in > StorageApiWriteShardedRecords). > > On Wed, Apr 29, 2026 at 11:48 AM Yi Hu ***@***.***> wrote: > >> *Abacn* left a comment (apache/beam#38328) >> <#38328 (comment)> >> >> Is this caused by #38058 <#38058>? >> It appears BigQueryIO storage write can no longer run on Direct/Prism >> runner. It sounds a regression >> >> cc: @ahmedabu98 <https://github.com/ahmedabu98> @reuvenlax >> <https://github.com/reuvenlax> is it possible to avoid >> OnWindowExpiration with some option or, when the runner isn't supported >> >> — >> Reply to this email directly, view it on GitHub >> <#38328 (comment)>, or >> unsubscribe >> <https://github.com/notifications/unsubscribe-auth/AFAYJVLRRNOZVCUDZPNX2KD4YJFBRAVCNFSM6AAAAACYLE7DECVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHM2DGNBWGYYDONZWHE> >> . >> Triage notifications on the go with GitHub Mobile for iOS >> <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> >> or Android >> <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. >> >> You are receiving this because you were mentioned.Message ID: >> ***@***.***> >> >

reuvenlax · 2026-04-29T21:46:38Z

#38332 should address the broken Prism test.

…

On Wed, Apr 29, 2026 at 12:14 PM Reuven Lax ***@***.***> wrote: If Prism doesn't support window expiration we should fix this - it's quite a big problem. For now, we can modify BigQueryIO to only expand this portion of the graph if the user has specified schema-update options. That should be a simple change to make. On Wed, Apr 29, 2026 at 12:07 PM Reuven Lax ***@***.***> wrote: > GroupIntoBatches also uses OnWindowExpiration. > > On Wed, Apr 29, 2026 at 12:03 PM Reuven Lax ***@***.***> wrote: > >> Does Prism not support OnWindowExpiration? If so that's a major gap - in >> many cases there's no way to avoid data loss without OnWindowExpiration. >> >> Also, BigQueryIO already used OnWindowExpiration before (e.g. in >> StorageApiWriteShardedRecords). >> >> On Wed, Apr 29, 2026 at 11:48 AM Yi Hu ***@***.***> wrote: >> >>> *Abacn* left a comment (apache/beam#38328) >>> <#38328 (comment)> >>> >>> Is this caused by #38058 <#38058>? >>> It appears BigQueryIO storage write can no longer run on Direct/Prism >>> runner. It sounds a regression >>> >>> cc: @ahmedabu98 <https://github.com/ahmedabu98> @reuvenlax >>> <https://github.com/reuvenlax> is it possible to avoid >>> OnWindowExpiration with some option or, when the runner isn't supported >>> >>> — >>> Reply to this email directly, view it on GitHub >>> <#38328 (comment)>, >>> or unsubscribe >>> <https://github.com/notifications/unsubscribe-auth/AFAYJVLRRNOZVCUDZPNX2KD4YJFBRAVCNFSM6AAAAACYLE7DECVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHM2DGNBWGYYDONZWHE> >>> . >>> Triage notifications on the go with GitHub Mobile for iOS >>> <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> >>> or Android >>> <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. >>> >>> You are receiving this because you were mentioned.Message ID: >>> ***@***.***> >>> >>

github-actions · 2026-04-30T00:55:34Z

Assigning reviewers:

R: @claudevdm for label python.

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

stop reviewer notifications - opt out of the automated review tooling
remind me after tests pass - tag the comment author after tests pass
waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

Amar3tto added 3 commits April 29, 2026 18:59

Skip tests

0da6df3

Add _runner in tests

bf3a2c7

Fix args

64159c0

Amar3tto requested a review from Abacn April 29, 2026 18:36

github-actions Bot added python io gcp labels Apr 29, 2026

gemini-code-assist Bot reviewed Apr 29, 2026

View reviewed changes

Abacn reviewed Apr 29, 2026

View reviewed changes

github-actions Bot added the Next Action: Reviewers label Apr 30, 2026

Amar3tto closed this May 1, 2026

	self._runner = PipelineOptions(self.args).get_all_options()['runner']
	self._runner = self.test_pipeline.get_option('runner')


		hc.assert_that(p, hc.all_of(*pipeline_verifiers))

		def skip_if_not_dataflow_runner(self) -> bool:

		def test_read_xlang(self):
		self.skip_if_not_dataflow_runner()

Conversation

Amar3tto commented Apr 29, 2026

GitHub Actions Tests Status (on master branch)

Uh oh!

gemini-code-assist Bot commented Apr 29, 2026

Summary of Changes

Highlights

Footnotes

Uh oh!

Abacn commented Apr 29, 2026

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist Bot Apr 29, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Apr 29, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Apr 29, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Apr 29, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Apr 29, 2026

Choose a reason for hiding this comment

Uh oh!

reuvenlax commented Apr 29, 2026 via email

Uh oh!

reuvenlax commented Apr 29, 2026 via email

Uh oh!

Abacn Apr 29, 2026

Choose a reason for hiding this comment

Uh oh!

reuvenlax commented Apr 29, 2026 via email

Uh oh!

reuvenlax commented Apr 29, 2026 via email

Uh oh!

github-actions Bot commented Apr 30, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants