Overview

49 Pull requests merged by25 people

• Enable xlang access for SQS read functionality

  #34327 mergedMar 19, 2025

• Add option to disable Kafka metrics

  #34303 mergedMar 19, 2025

• [AnomalyDetection] Support offline detectors

  #34311 mergedMar 18, 2025

• Allow declaration of external dependencies for YAML UDFs.

  #34073 mergedMar 18, 2025

• Bump github.com/nats-io/nats.go from 1.39.0 to 1.39.1 in /sdks

  #34029 mergedMar 18, 2025

• Support directory separator for Python local filesystem

  #34318 mergedMar 18, 2025

• Bump golang.org/x/oauth2 from 0.26.0 to 0.28.0 in /sdks

  #34190 mergedMar 18, 2025

• [AnomalyDetection] Refactor and improve Specifiable

  #34310 mergedMar 18, 2025

• Expose add-modules JVM option in SDKHarnessOptions

  #34289 mergedMar 18, 2025

• [KafkaIO] Fix average record size data race and backlog estimation

  #34165 mergedMar 18, 2025

• Update roadmap page 28342

  #34330 mergedMar 18, 2025

• Bump golang.org/x/net from 0.35.0 to 0.37.0 in /sdks

  #34328 mergedMar 18, 2025

• Fix PostCommit Python ValidatesContainer with RC workflow

  #34332 mergedMar 18, 2025

• Fix Precommit SQL

  #34320 mergedMar 18, 2025

• added test_accessing_valueprovider_info_after_run

  #34315 mergedMar 18, 2025

• Consider workflow flaky if the 5 last runs were failed

  #34321 mergedMar 17, 2025

• Honor JAVA_HOME when it is set

  #34313 mergedMar 17, 2025

• Add Year and Apache Beam in Copyright

  #34306 mergedMar 15, 2025

• Add a STRING format to PubSub reading that interpretes the payload as utf-8 encoded.

  #34301 mergedMar 15, 2025

• Better error messages for unavailable providers.

  #34304 mergedMar 15, 2025

• refactor main.py so that it can be called by runner.run_async().

  #34290 mergedMar 14, 2025

• Fix pre/postprocess type hints

  #34298 mergedMar 14, 2025

• Healthcare Label Update

  #34299 mergedMar 14, 2025

• docs: fix typo in snippets_test.py

  #34297 mergedMar 14, 2025

• docs: update CHANGES.md for Spark Runner

  #34295 mergedMar 14, 2025

• add default port for HostAndPort instances used in Windmill

  #34061 mergedMar 14, 2025

• [AnomalyDetection] Support functions and classes as init arguments in specifiable.

  #34273 mergedMar 14, 2025

• Use Avro 1.8-compatible Schema constructors in Storage Write API translator

  #34281 mergedMar 14, 2025

• Updates 2.64.0 release notes to highlight recent additions to the Managed API

  #34291 mergedMar 13, 2025

• Use BoundedTrie metric to track lineage in IO

  #33891 mergedMar 13, 2025

• Update join beam doc

  #34284 mergedMar 13, 2025

• Spark Runner : Fix invalid translateImpulse

  #34272 mergedMar 13, 2025

• Eliminate springframework dependency in KafkaIO

  #34278 mergedMar 13, 2025

• Fix spotless

  #34279 mergedMar 13, 2025

• Fix Stress Tests Java Bigquery

  #34078 mergedMar 13, 2025

• Remove 2.1 and 2.2 and add 2.8 for Kafka compatibility test

  #34265 mergedMar 13, 2025

• Refactor per worker histogram metrics

  #34244 mergedMar 13, 2025

• Update code-change-guide.md

  #34266 mergedMar 12, 2025

• Fix PostCommit Python ValidatesContainer Dataflow With RC job

  #34246 mergedMar 12, 2025

• Update pinned kafka version to 3.9.0 in Expansion Service.

  #34197 mergedMar 12, 2025

• Update to latest bom

  #34231 mergedMar 12, 2025

• Fix PostCommit Java PVR Spark3 Streaming job

  #34253 mergedMar 12, 2025

• [Java] Added Metrics Configuration Support to Iceberg Data Writers

  #34140 mergedMar 12, 2025

• Add yaml examples generation to the release scripts.

  #34217 mergedMar 12, 2025

• Update DF Python container tag

  #34255 mergedMar 12, 2025

• Putting the default retry settings which was overridden in Dataflow templates

  #34161 mergedMar 12, 2025

• Use same format for next and current release versions

  #34262 mergedMar 12, 2025

• Bump dataflow java container version - 20250312

  #34256 mergedMar 12, 2025

• [AnomalyDetection] Refactor code and add more docstrings

  #34235 mergedMar 12, 2025

29 Pull requests opened by19 people

• [Java]Add Support for Named Schemas in SpannerIO.Write by Enhancing Schema Retrieval

  #34261 openedMar 12, 2025

• Bump @babel/runtime from 7.10.5 to 7.26.10 in /sdks/python/apache_beam/runners/interactive/extensions/apache-beam-jupyterlab-sidepanel

  #34263 openedMar 12, 2025

• [IcebergIO] Filter out data files that have already been committed

  #34264 openedMar 12, 2025

• Bump golang.org/x/net from 0.23.0 to 0.36.0 in /learning/katas/go

  #34267 openedMar 12, 2025

• Fix Republish Released Docker Images workflow

  #34268 openedMar 13, 2025

• Making setServiceFactory public in SpannerConfig for Failure injection testing in Dataflow templates

  #34271 openedMar 13, 2025

• Add sdf kafka poll latencies

  #34275 openedMar 13, 2025

• Update docstring to clarify numShards=0 doesn't neccessarily enable autosharding

  #34280 openedMar 13, 2025

• Fix programming guide python reference for combining pcollection into a single value

  #34282 openedMar 13, 2025

• do not block other threads from health checking if a stream is blocked

  #34283 openedMar 13, 2025

• [Dataflow Streaming] WindmillTimerInternals: Use a single map to store timer data + liveness

  #34292 openedMar 14, 2025

• Bump google.golang.org/api from 0.221.0 to 0.226.0 in /sdks

  #34293 openedMar 14, 2025

• Revert Skip BoundedTrie on Dataflow till service is have BoundedTrie …

  #34294 openedMar 14, 2025

• [Java]Fix NPE in UnboundedSolaceReader.getWatermark during disconnection (fixes #32660)

  #34296 openedMar 14, 2025

• Add STRING format to ReadFromKafka schema transform.

  #34302 openedMar 14, 2025

• More precise binary operation inference.

  #34305 openedMar 15, 2025

• [Java] Add Gauge Metric Extraction to DataflowMetrics

  #34307 openedMar 15, 2025

• Add an experiment named enableLineageRollup which when passed to Java…

  #34312 openedMar 16, 2025

• Fix typo in BigQuery Python Documentation

  #34317 openedMar 16, 2025

• Add 3 examples in playground SQL transform, schema transform and Composite Combine

  #34322 openedMar 17, 2025

• Aggregation option in Kinesis Writer Python sdk

  #34323 openedMar 17, 2025

• Key by paneindex and reshuffle before loading files.

  #34324 openedMar 17, 2025

• Update pypi documentation 30145

  #34329 openedMar 18, 2025

• [KafkaIO] Improve caching in backlog estimation and processing

  #34331 openedMar 18, 2025

• Kafka Consumer Group Properties

  #34334 openedMar 18, 2025

• Remove cancelled tasks from ReadOperation queue when shutting down

  #34335 openedMar 18, 2025

• Fix pandas doctests sensitive to NumpyExtensionArray formatting.

  #34336 openedMar 18, 2025

• Bump cloud.google.com/go/storage from 1.50.0 to 1.51.0 in /sdks

  #34340 openedMar 19, 2025

• Bump github.com/aws/aws-sdk-go-v2/service/s3 from 1.77.0 to 1.78.2 in /sdks

  #34341 openedMar 19, 2025

30 Issues closed by7 people

• [Bug]: Python 3.10 installing `apache-beam==2.43.0` with `multiprocess>=0.70.12` has incompatibility for `dill`

  #24458 closedMar 18, 2025

• [Bug]: Python 3.12 in-compatibility of Apache Beam

  #32617 closedMar 18, 2025

• [Bug]: apache-beam is unusable in recent python due to pinning an old dill library from 2019

  #32842 closedMar 18, 2025

• [Feature Request]: Upgrade or provide ETA on dill

  #22893 closedMar 18, 2025

• Python local filesystem match does not work without directory separator.

  #19741 closedMar 18, 2025

• [Feature Request]: Enable to configure `--add-modules` jvm flag

  #30281 closedMar 18, 2025

• [Bug]: Roadmap page is out of date.

  #28342 closedMar 18, 2025

• [Bug]: beam_PreCommit_PythonDocker misconfigured running on snapshot container

  #33558 closedMar 18, 2025

• The PostCommit Python ValidatesContainer Dataflow With RC job is flaky

  #30525 closedMar 18, 2025

• The PreCommit SQL job is flaky

  #34314 closedMar 18, 2025

• Add test for snippet accessing_valueprovider_info_after_run

  #19731 closedMar 18, 2025

• The PostCommit Java ValidatesRunner Flink Java8 job is flaky

  #32949 closedMar 17, 2025

• Prefer java binary from $JAVA_HOME in JavaJarServer

  #21006 closedMar 17, 2025

• Remove deprecated 'compare' argument from combiners.Top in PyDocs

  #19819 closedMar 16, 2025

• Update Python SDK example tests to use assert_that

  #18029 closedMar 16, 2025

• GCS IO fails with uninformative 'Broken pipe' errors while attempting to write to a GCS bucket without proper permissions.

  #19899 closedMar 15, 2025

• Log GCS upload ID for Python GCS connector

  #21577 closedMar 15, 2025

• The StressTests Java BigQueryIO job is flaky

  #31968 closedMar 15, 2025

• [Bug]: Python doc misses Copyright

  #34287 closedMar 15, 2025

• [Bug]: typo in docs - Python Composition with AfterWatermark example

  #34224 closedMar 14, 2025

• Improve case studies section of website

  #21281 closedMar 14, 2025

• The PostCommit Python job is flaky

  #30513 closedMar 14, 2025

• Beam directRunner documentation java tab has python information

  #19884 closedMar 14, 2025

• Add support for checkpointing in Spark streaming

  #20426 closedMar 13, 2025

• [Task]: Update KafkaIO ConsumerSpEL and eliminate springframework dependencies

  #34277 closedMar 13, 2025

• The Republish Released Docker Images job is flaky

  #33834 closedMar 13, 2025

• The PostCommit Java PVR Spark3 Streaming job is flaky

  #34207 closedMar 12, 2025

• [Feature Request]: [IcebergIO] Configure data writers to track metrics

  #34112 closedMar 12, 2025

• [Failing Test]: dataflow runner worker project test stuck causing Java PreCommit time out

  #28957 closedMar 12, 2025

• [Failing Test]: org.apache.beam.sdk.io.mqtt.MqttIOTest.testReadWithMetadata

  #34175 closedMar 12, 2025

12 Issues opened by11 people

• [Feature Request]: Add support for tracking lineage with BoundedTrie

  #34342 openedMar 19, 2025

• [Bug]: Unauthenticated error when running tests on Dataproc using Flink and Docker with private repos

  #34339 openedMar 19, 2025

• Performance Regression or Improvement: cogbk_python_batch_load_test_2GB_of_100B_records_with_a_single_key:runtime

  #34338 openedMar 18, 2025

• Performance Regression or Improvement: gbk_python_batch_load_test_2gb_of_10B_records:runtime

  #34337 openedMar 18, 2025

• [Bug]: withOutputTags (aka side output) requires ParDo to run on each output separately

  #34333 openedMar 18, 2025

• [Feature Request]: Enable AWS SQS reads for Python SDK

  #34326 openedMar 17, 2025

• [Feature Request]: Record Aggregation option in Kinesis Writer

  #34319 openedMar 17, 2025

• [Bug]: Go Prism Runner has a Data Race in v2.63.0

  #34316 openedMar 16, 2025

• [Task]: Improve README.md for https://github.com/apache/beam-starter-java-provider

  #34308 openedMar 15, 2025

• Big Query Python Documentation Typo

  #34300 openedMar 14, 2025

• [Bug]: Unknown spanner type FLOAT32>(VECTOR_LENGTH=>128

  #34276 openedMar 13, 2025

• [Bug]: Bigquery python streaming insertAll SSLError leads to stuck streaming job

  #34270 openedMar 13, 2025

72 Unresolved conversations

Sometimes conversations happen on old items that aren’t yet closed. Here is a list of all the Issues and Pull Requests with unresolved conversations.

• Add Triton Inference Server Support

  #34252 commented onMar 18, 2025 • 16 new comments

• Fail Fast if Resources Do Not Exist in Kafka Cluster

  #34258 commented onMar 18, 2025 • 7 new comments

• Add streaming example to AlloyDB colab.

  #34251 commented onMar 12, 2025 • 3 new comments

• Fix precommit flink container

  #34250 commented onMar 19, 2025 • 3 new comments

• add vendor to manually shutdown and restart GetWorkerMetadataStream to prevent DEADLINE_EXCEEDED errors

  #34053 commented onMar 14, 2025 • 2 new comments

• Fix: Replace deprecated DataFrame.applymap with map for compatibility

  #33532 commented onMar 18, 2025 • 2 new comments

• [Python] File staging to user worker support

  #34208 commented onMar 13, 2025 • 2 new comments

• Move ExecutionStateTracker.nextBundleLullDurationReportMs reset attempt to resolve race detection

  #34214 commented onMar 18, 2025 • 2 new comments

• Add Encryption When Writing to Iceberg Tables in RecordWriter.java

  #34021 commented onMar 16, 2025 • 1 new comment

• #34009 avro generic record to beam row conversion added support for a…

  #34024 commented onMar 17, 2025 • 1 new comment

• add equals hashCode to BoundedToUnboundedSourceAdapter

  #34057 commented onMar 18, 2025 • 1 new comment

• [Managed Iceberg] unbounded source

  #33504 commented onMar 18, 2025 • 1 new comment

• [DO NOT MERGE] Removed < 1.66 for grpcio

  #34196 commented onMar 16, 2025 • 1 new comment

• feat:large-row-skip-in-bigtable | added experimental options to skip …

  #34245 commented onMar 13, 2025 • 1 new comment

• Request to use BASIC enum when calling tables.get() in BigQuery #34075

  #34249 commented onMar 18, 2025 • 0 new comments

• [Python] Fix WriteToBigQuery transform using CopyJob does not work with WRITE_TRUNCATE write disposition (#34247)

  #34248 commented onMar 18, 2025 • 0 new comments

• Switch to use registerFileSystemsOnce for SerializablePipelineOptions constructor

  #34028 commented onMar 18, 2025 • 0 new comments

• Update allowed lateness in FixedWindows example to 2 days

  #34227 commented onMar 18, 2025 • 0 new comments

• [BEAM-6394] Add support to write protobuf data using ProtoParquetReader

  #34063 commented onMar 15, 2025 • 0 new comments

• Bump github.com/fsouza/fake-gcs-server from 1.52.1 to 1.52.2 in /sdks

  #34094 commented onMar 18, 2025 • 0 new comments

• Update MongoDB driver to mongodb-driver-legacy:5.3.1

  #34100 commented onMar 18, 2025 • 0 new comments

• [Prism] Refactor stageState to a behavior interface to reduce branch combinatorics

  #34132 commented onMar 18, 2025 • 0 new comments

• [Python] Add caching for BigQuery table definitions

  #34135 commented onMar 18, 2025 • 0 new comments

• Fix Docker build error by adding fallback for python3.12-distutils

  #34144 commented onMar 18, 2025 • 0 new comments

• test commit

  #34148 commented onMar 17, 2025 • 0 new comments

• Bump @octokit/plugin-paginate-rest and @octokit/rest in /scripts/ci/issue-report

  #34167 commented onMar 18, 2025 • 0 new comments

• Enable cloudpickle default

  #34223 commented onMar 12, 2025 • 0 new comments

• [Java] Add InsertRetryPolicy for non-successful BigQuery insertAll responses

  #34222 commented onMar 19, 2025 • 0 new comments

• Rethrowing Exception from CassandraIO's ReadFn

  #34191 commented onMar 18, 2025 • 0 new comments

• Fix ProtoCoder NoSuchMethodException

  #34194 commented onMar 12, 2025 • 0 new comments

• [Java] Ensure Pipeline Execution Requires Configuration Options or Logs Warning

  #34220 commented onMar 17, 2025 • 0 new comments

• [KafkaIO] Remove duplicate offset in range check

  #34201 commented onMar 18, 2025 • 0 new comments

• [KafkaIO] Update tracker and watermark for non-visible progress

  #34202 commented onMar 18, 2025 • 0 new comments

• Add Documentation Hint for Template Job Creation in DataflowRunner

  #34204 commented onMar 18, 2025 • 0 new comments

• Add support for top-level table properties table creation

  #34205 commented onMar 16, 2025 • 0 new comments

• [Java] Added tests for S3ReadableSeekableByteChannel

  #34219 commented onMar 17, 2025 • 0 new comments

• [Java] Add parsedData to Hl7v2Message and Update HL7v2IO Docs

  #34213 commented onMar 15, 2025 • 0 new comments

• Bump serialize-javascript and mocha in /sdks/typescript

  #34012 commented onMar 13, 2025 • 0 new comments

• Backlog metrics do not showing up in FlinkRunner

  #25554 commented onMar 16, 2025 • 0 new comments

• The Clean Up GCP Resources job is flaky

  #31846 commented onMar 19, 2025 • 0 new comments

• Make cloudpickle the default pickle library

  #21298 commented onMar 18, 2025 • 0 new comments

• [Task]: Improve project documentation on apache-beam pypi page.

  #30145 commented onMar 18, 2025 • 0 new comments

• [Feature Request]: Replace snappy with other `crc32c ` packages for tfrecordio

  #34226 commented onMar 17, 2025 • 0 new comments

• [Bug]: ./gradlew checkSetup fails with Java 8 on Docker dev image

  #34242 commented onMar 16, 2025 • 0 new comments

• [Bug]: The submission_environment_dependencies.txt file does not get staged when running with Flink runner on Dataproc

  #32743 commented onMar 15, 2025 • 0 new comments

• [Bug]: Update or Remove Integrations page

  #27613 commented onMar 15, 2025 • 0 new comments

• [Bug]: Javadoc still incorrectly renders non-ascii characters

  #25427 commented onMar 14, 2025 • 0 new comments

• [Feature Request]: Running Word-Count with Gradle for PowerShell

  #32307 commented onMar 13, 2025 • 0 new comments

• The PostCommit Java IO Performance Tests job is flaky

  #30527 commented onMar 13, 2025 • 0 new comments

• Beam with intellij: "Could not find implementation class 'org.apache.beam.gradle.BeamModulePlugin' for plugin 'org.apache.beam.module'"

  #21582 commented onMar 13, 2025 • 0 new comments

• [Feature Request]: {Managed IO Iceberg} - Allow users to run streaming reads

  #33092 commented onMar 12, 2025 • 0 new comments

• [Feature Request]: YAML to support creating BQT4AI

  #33726 commented onMar 12, 2025 • 0 new comments

• [YAML][Feature Request]: Integration and Usage of GCP Secret Manager with the different Apache Beam IOs

  #32665 commented onMar 12, 2025 • 0 new comments

• [Failing Test]: beam_PreCommit_Flink_Container not build Python SDK container causing test failure at PR run

  #33942 commented onMar 12, 2025 • 0 new comments

• [Bug]: SpannerIO.Write does not support named schemas

  #32907 commented onMar 12, 2025 • 0 new comments

• Bump @octokit/request-error, @actions/github and @octokit/rest in /scripts/ci/pr-bot

  #33998 commented onMar 18, 2025 • 0 new comments

• Upgrade Kafka client version to 3.7.0

  #33960 commented onMar 18, 2025 • 0 new comments

• Adding credentials option to PubsubMessageMatcher

  #33958 commented onMar 18, 2025 • 0 new comments

• SnowflakeIO: be consistent with backslash escape char

  #33948 commented onMar 18, 2025 • 0 new comments

• Enable timeout setting for Python TestPipeline (#29646)

  #33866 commented onMar 18, 2025 • 0 new comments

• Fix incorrect nullness in FlinkJobInvoker and JobInvoker

  #33713 commented onMar 18, 2025 • 0 new comments

• [Dataflow Streaming] Cache StateNamespace encoded keys using SoftReference

  #33689 commented onMar 18, 2025 • 0 new comments

• Add support for Iceberg table identifiers with special characters

  #33648 commented onMar 18, 2025 • 0 new comments

• Remove type coercion of `FinalizeWrite` in iobase.py

  #33614 commented onMar 18, 2025 • 0 new comments

• Support sharding in WriteToFiles (tested for to_csv)

  #33612 commented onMar 18, 2025 • 0 new comments

• Handle missing tables more gracefully.

  #33610 commented onMar 17, 2025 • 0 new comments

• [WIP] [Do not Merge] [Dataflow Streaming] Support for using virtual harness threads

  #33543 commented onMar 15, 2025 • 0 new comments

• Tour of Beam: update GroupByKey example

  #33242 commented onMar 18, 2025 • 0 new comments

• Read RabbitMQ messages with headers containing nested objects

  #33072 commented onMar 15, 2025 • 0 new comments

• add generics support to AutoValueUtils helpers

  #32977 commented onMar 18, 2025 • 0 new comments

• BigQueryIO uniformize direct and export reads

  #32360 commented onMar 15, 2025 • 0 new comments

• Support writing to Pubsub with ordering key; Add PubsubMessage SchemaCoder

  #31608 commented onMar 15, 2025 • 0 new comments