Uh oh!
There was an error while loading.Please reload this page.
- Notifications
You must be signed in to change notification settings - Fork288
Bazel 8 +rules_java 8 updates, protoc toolchain#1710
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.
Already on GitHub?Sign in to your account
Bazel 8 +rules_java 8 updates, protoc toolchain#1710
Uh oh!
There was an error while loading.Please reload this page.
Conversation
mbland commentedFeb 28, 2025 • edited
Loading Uh oh!
There was an error while loading.Please reload this page.
edited
Uh oh!
There was an error while loading.Please reload this page.
Hmm,an odd failure on the Windows build: ERROR: C:/tools/msys64/home/b/_bazel_b/xknd5zlq/external/com_google_protobuf/src/google/protobuf/compiler/java/BUILD.bazel:87:11: Compiling src/google/protobuf/compiler/java/java_features.pb.cc [for tool] failed: (Exit 2): cl.exe failed: error executing CppCompile command (from target @@com_google_protobuf//src/google/protobuf/compiler/java:java_features_bootstrap) C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.39.33519\bin\HostX64\x64\cl.exe ... (remaining 1 argument skipped)-- | external/com_google_protobuf/src/google/protobuf/compiler/java/java_features.pb.cc(6): fatal error C1083: Cannot open include file: 'google/protobuf/compiler/java/java_features.pb.h': No such file or directory src/google/protobuf/compiler/java/java_features.pb.h most definitely exists in protobuf v29.3. All the other builds seem to pass. If they do, I'll kick this with an empty commit to see if we can get Windows to play along, in case it was some weird transient error. |
Got an odd failure on the Windows build:-https://buildkite.com/bazel/rules-scala-scala/builds/5394#01954e5a-8f5b-4880-befa-1bce0d21d512/75-179```txtERROR: C:/tools/msys64/home/b/_bazel_b/xknd5zlq/external/com_google_protobuf/src/google/protobuf/compiler/java/BUILD.bazel:87:11: Compiling src/google/protobuf/compiler/java/java_features.pb.cc [for tool] failed: (Exit 2): cl.exe failed: error executing CppCompile command (from target @@com_google_protobuf//src/google/protobuf/compiler/java:java_features_bootstrap) C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.39.33519\bin\HostX64\x64\cl.exe ...(remaining 1 argument skipped)-- | external/com_google_protobuf/src/google/protobuf/compiler/java/java_features.pb.cc(6): fatal error C1083: Cannot open include file: 'google/protobuf/compiler/java/java_features.pb.h': No such file or directory````src/google/protobuf/compiler/java/java_features.pb.h` most definitelyexists in protobuf v29.3:-https://github.com/protocolbuffers/protobuf/blob/v29.3/src/google/protobuf/compiler/java/java_features.pb.hAll the other builds passed. Kicking the pull requeest branch with thisempty commit to see if the Windows build will pass, in case it was someweird transient error.
Yep, just windows. And my empty commit bump just failed in the same way. Hmm... why can't it find the |
I looked at the windows issue... the issue is that the full include path for that file is over the char limit (260 chars). Also, I don't remember having to compile those protobuf tools locally before... I feel like they came precompiled before? |
OK, I found the related issue here:protocolbuffers/protobuf#12947. (I'm going to look into it more cause I use MSVC all the time and haven't run into these issues.) I wonder if we can go back to using the prebuilt protobuf libraries, then we don't need to require the c++ compiler in order to use rules_scala. |
@crt-31 Heh, you beat me to it...I snuck a look on my phone while out this evening and found the same issue. Got online to post it, but was too late. 😛 What's more, my nextBzlmod blog post is going to talk about how I had to patch And I saw the paragraph that you just responded to:
The linked announcement makes reference toprotocolbuffers/protobuf#20085: "Breaking Change: Dropping support for Bazel+MSVC". But to your point above:
Idid experiment with using prebuilt Protobuf toolchains inmy proto-toolchainization branch in mbland/rules_scala back in October, when I was first experimenting with Bzlmodification. I think I may dust it off and play with it again in light of this development; it may also provide a performance win, since |
Keeps the `README` guidance in sync with what we're actually using in`WORKSPACE` for consistency's sake.@crt-31 and I found that the Windows build failure forbazel-contrib#1710 mentionedin the earlier commit is related to the Windows/MSVC file path lengthlimit. `src/google/protobuf/compiler/java/java_features.pb.h`, the pathspecified in the error message, doesn't exist until `protobuf` v25.0.-protocolbuffers/protobuf#12947Furthermore, the Protobuf team currently plans to just drop MSVCsupport:-https://protobuf.dev/news/v30/#poison-msvc--bazel-protocolbuffers/protobuf#20085I plan to experiment again with "Protobuf Toolchainization", which I'dtried in October when beginning the Bzlmod experiment. Here are someinteresting background resources before I dig in on that:-bazelbuild/rules_proto#213-bazelbuild/rules_proto#179-https://github.com/bazelbuild/rules_proto/releases/tag/6.0.0-https://github.com/aspect-build/toolchains_protoc/-protocolbuffers/protobuf#20182-protocolbuffers/protobuf#19679-protocolbuffers/protobuf#19558
Registers a precompiled protocol compiler toolchain when`--incompatible_enable_proto_toolchain_resolution` is `True`.Part ofbazel-contrib#1482 andbazel-contrib#1652.Stops `protoc` recompilation, and fixes the build breakage inbazel-contrib#1710 dueto `protobuf` include paths exceeding the Visual Studio path lengthlimit.The updates to `scala_proto/scala_proto_toolchain.bzl` were inspired by:- protocolbuffers/protobuf: bazel: Remove hardcoded dependency on //:protoc from language runtimes #19679protocolbuffers/protobuf#19679The `proto_lang_toolchain` call was inspired by the `README` from:-https://github.com/aspect-build/toolchains_protoc/Adds `scripts/update_protoc_integrity.py` to automatically update`scala/private/protoc/protoc_integrity.bzl`.This should make builds of `rules_scala` much faster all around. Giventhe fact that this feature depends on recent `protobuf` versions, andthe Windows `protobuf` build breaks without it, we have a catch-22. Itlikely can't be separated from the rest ofbazel-contrib#1710, though I would preferthat.It also seems likely that we'd eventually need to do this to continuesupporting Windows, per:-protocolbuffers/protobuf#12947-https://protobuf.dev/news/v30/#poison-msvc--bazel-protocolbuffers/protobuf#20085More background on proto toolchainization:- Proto Toolchainisation Design Dochttps://docs.google.com/document/d/1CE6wJHNfKbUPBr7-mmk_0Yo3a4TaqcTPE0OWNuQkhPs/edit- bazelbuild/bazel: Protobuf repo recompilation sensitivitybazelbuild/bazel#7095- bazelbuild/rules_proto: Implement proto toolchainisationbazelbuild/rules_proto#179- rules_proto 6.0.0 release notes mentioning Protobuf Toolchainisationhttps://github.com/bazelbuild/rules_proto/releases/tag/6.0.0
mbland commentedMar 1, 2025 • edited
Loading Uh oh!
There was an error while loading.Please reload this page.
edited
Uh oh!
There was an error while loading.Please reload this page.
UPDATE: Please skip this comment and readmy new comment about the more recent fully working and polished protoc toolchainization. The good news is, I went ahead and implemented The bad news is,the Windows build is still broken in the same way. defjava_proto_library(**kwattrs):# Only use Starlark rules when they are removed from Bazelifnothasattr(native,"java_proto_library"):_java_proto_library(**kwattrs)else:native.java_proto_library(**kwattrs) I believe this is what@fmeum is trying to fix inprotocolbuffers/protobuf#19679 (which I borrowed from in @simuons Until this gets resolved one way or another, what do you think of making Windows builds optional for now? Then when a fix comes, we can make it required again. Or maybe I can hack something to downgrade the Windows build down to @crt-31 I know you really don't want that to happen, but it's effectively blocking the next @fmeum If you've any better ideas, or my understanding is incorrect in any way, I'm all ears. At any rate, this gives me time to think about maybe adding more tests for this latest |
Hold the presses... I just pushed an experiment that fixed the Windows build. I created a patch fromprotocolbuffers/protobuf#19679, then added a line to Everything else is failing right now (all the tests for all the other |
OK, please ignoremy update from yesterday. I've learned a lot, including that the The majority of the changes in the PR are still the boilerplate updates to
Still, this is alot more than I'd originally planned to include in this pull request. Some of these updates can probably be teased out into separate PRs. And/or, wecould go back to the original 7.x and 8.x release plan. This would mean backing out the In either case, Bzlmod could land in 7.0 or 7.1. I'll prepare my Bzlmod branch, but it would remain a relatively light update on top of whatever changes we decide to include, and when. Anywho, as usual, options...@simuons Let me know what you'd prefer to do here. |
P.S. I can confirm that this branch is still compatible with Bazel 6.5.0, both with the C++ compiler flags and with protocol compiler toolchainization enabled, as well as Bazel 8.1.1. I have protoc toolchainized Bzlmod builds working onmy unstable 'bzlmod-enable' branch and both of my stablebzlmod andbzlmod-bazel-8 branches. Can confirm they work with Bazels 7.5.0, 8.1.1, # rollingStarting local Bazel server (9.0.0-pre.20250210.1) and connecting to it...# last_green2025/03/02 22:24:18 Using unreleased version at commit 15a3526621559170c2a28e9dae5e9d7c72a82f02 2025/03/02 22:24:18 Downloading https://storage.googleapis.com/bazel-builds/artifacts/macos_arm64/15a3526621559170c2a28e9dae5e9d7c72a82f02/bazel... The thing that's at once a bug and a feature of the Bzlmod branches is the fact that I had to manually copy the And of course, in the best possible worlds,protocolbuffers/protobuf#19679 will get merged, and we can upgrade to at least v29.4 and forget about this patching and At any rate, for now, the patch is helping everything work, and I've documented the hell out of it in the |
I've just gone ahead and added these bumps as well, and propagated them tobzlmod-enable,bzlmod, andbzlmod-bazel-8:
|
Just bumped:
I'm also about to try breaking out separate pull requests from this one, in case that will prove more helpful. Then we'll have the option of landing the different pieces in smaller chunks, or at least making more sense of the pile of changes in this PR. |
Dependency version updates that still work with Bazel 6.5.0 and 7.5.0.Broken out frombazel-contrib#1710, and part ofbazel-contrib#1482 andbazel-contrib#1652.Updates `.bazelversion` files to 7.5.0 and the CI builds in`.bazelci/presubmit.yml` to use Bazel 7.5.0.Bumps the following dependencies, which should not cause build breakageson Windows + MSVC:- Go: 1.24.0 => 1.24.1- Scalafmt: 3.9.1 => 3.9.2- `abseil-cpp`: 20220623.1 => 20250127.0- `grpc`: 1.70.0 => 1.71.1- `protobuf-java`: 4.29.3 => 4.30.0- `sbt-compiler-interface`: 1.10.7 => 1.10.8- `sbt-compiler-util`: 1.10.7 => 1.10.10- `google-common-protos`: 2.52.0 => 2.53.0Defers the following updates, which are already present inbazel-contrib#1710:- `protobuf`: v21.7 => v28.3 (or v30.0)- `rules_cc`: 0.0.9 => 0.1.1- `rules_proto`: 6.0.2 => 7.1.0- `rules_python`: 0.38.0 => 1.2.0- `scalapb`: 0.11.17 => 1.0.0-alpha1These deferred updates all need to happen together, as updating only asubset of them will break the build.This change is smaller and more focused thanbazel-contrib#1710, and shouldultimately make that pull request smaller and/or easier to review.
Updates the `protobuf` version to one that's still compatible with Bazel6.5.0 and 7.5.0 without updating `scalapb`. Broken out frombazel-contrib#1710, andpart ofbazel-contrib#1482 andbazel-contrib#1652.Likebazel-contrib#1711, updates `.bazelversion` files to 7.5.0 and the CI builds in`.bazelci/presubmit.yml` to use Bazel 7.5.0.Unlikebazel-contrib#1711, contains only these updates:- `abseil-cpp`: 20220623.1 => 20250127.0- `protobuf`: v21.7 => v25.6This change aims makebazel-contrib#1710 smaller and more focused, and shouldultimately make that pull request smaller and/or easier to review.Specifically, this is an attempt to see whether MSVC will buildsuccessfully with `protobuf` v25.6. If it doesn't, I will update thischange to include the protocol compiler toolchainization changesfrombazel-contrib#1710.
It turns out `protobuf` v26.6 isn't compatibile with ScalaPB 0.9.8 usedby Scala 2.11:```txtCaused by: java.lang.UnsupportedOperationException: As of 2022/09/29 (release 21.7) makeExtensionsImmutable should not be called from protobuf gencode. If you are seeing this message, your gencode is vulnerable to a denial of service attack. You should regenerate your code using protobuf 25.6 or later. Use the latest version that meets your needs. However, if you understand the risks and wish to continue with vulnerable gencode, you can set the system property `-Dcom.google.protobuf.use_unsafe_pre22_gencode` on the command line.See security vulnerability:GHSA-h4h5-3hr4-j3g2```As mentioned inbazel-contrib#1710, we should consider dropping Scala 2.11 support at thispoint, since there's no ScalaPB release for it that supports later versions of`protobuf`. That, and we could remove some of the special case code added inthe following changes, amongst other 2.11 support details:-bazel-contrib#1631-bazel-contrib#1648-bazel-contrib#1687-bazel-contrib#1688
Adds the `examples/overridden_artifacts` repository and thecorresponding `overridden_artifacts_example` test case in`test/shell/test_examples.sh`. Broken out frombazel-contrib#1710, and part ofbazel-contrib#1482andbazel-contrib#1652.@dmivankov noticed the design bug in the upcoming Bzlmod API for`overridden_artifacts` that this change addresses. See:-bazel-contrib#1482 (comment)-bazel-contrib#1482 (comment)Makes `_validate_scalac_srcjar()` and `dt_patched_compiler_setup()` in`scala/private/macros/scala_repositories.bzl` more tolerant ofdictionaries containing keys mapped to `None`. The new`overridden_artifacts_example` test covers this.Sets `.bazelversion` in the new repo to 7.5.0 to match changes inbothbazel-contrib#1710 andbazel-contrib#1711.This change is smaller and more focused thanbazel-contrib#1710, and shouldultimately make that pull request smaller and/or easier to review.
Contains many editorial improvements to and some extra information inthe README, along with a few small improvements to the code. Broken outfrombazel-contrib#1710, and part ofbazel-contrib#1482 andbazel-contrib#1652.Specifically:- Adds more info on translating `@rules_scala` to `@io_bazel_rules_scala` for dependencies via repo mapping.- Merges information about the previously planned `rules_scala` 8.0.0 release into the information for 7.0.0, since it seems we may make only one major release.- Improves information about `protobuf` support for versions before v28, Scala 2.11, and the upcoming Bzlmod `compatibility_level` setting.- In `scala_config.bzl`, changes the private `_default_scala_version()` to the public `DEFAULT_SCALA_VERSION`.- Adds `allow_empty = True` to a `glob` expression in `//test/semanticdb:lib_with_tempsrc`.- Removes Scala 2.11 test cases from `test_thirdparty_version.sh` and `test_version.sh`.This change is smaller and more focused thanbazel-contrib#1710, and shouldultimately make that pull request smaller and/or easier to review.The motivations for the individual changes are:- The public `DEFAULT_SCALA_VERSION` constant makes this value accessible to the upcoming module extension.- `glob` requires an explicit `allow_empty = True` parameter in Bazel 8, in which `--incompatible_disallow_empty_glob` defaults to `True`.- ScalaPB 0.9.8, the last version compatible with Scala 2.11, does not support `protobuf` v25.6 or later. For this reason, we must remove the Scala 2.11 test cases, as documented in the `README.md` updates. See alsobazel-contrib#1712.We should consider dropping Scala 2.11 support at this point, sincethere's no ScalaPB release for it that supports later versions of`protobuf`. That, and we could remove some of the special case codeadded in the following changes, amongst other 2.11 support details:-bazel-contrib#1631-bazel-contrib#1648-bazel-contrib#1687-bazel-contrib#1688
@simuons and@liucijus Before you review, approve, and merge this pull request, consider reviewing and merging these others first, which I just extracted from this one:
Each of those is orthogonal to one another, and is much easier to understand. Merging them will render this pull request much smaller and more focused itself. Also, what do you think of bumping the default Scala version from the current 2.12.20 to 2.13.16? Finally,#1712 (which isnot for review at this point) shows that even |
This is probably for the best, as it enables users to use earlierprotobuf versions by not loading `//protoc:toolchains.bzl`. Leaving itwired into `scala/toolchains_repo.bzl` would've required users to patchit and to remove `protoc/BUILD`.Prompted by:bazel-contrib#1710 (comment)
To be clear, with the latest commit, the protocol compiler toolchain API no longer requires loading and calling # Must come after loading `platforms` and `com_google_protobuf`.load("@rules_scala//protoc:toolchains.bzl","scala_protoc_toolchains")scala_protoc_toolchains() The |
e6627c5 to0ba9ab7Compare@crt-31@simuons OK, last API update, I swear... 🙂 I've now tried updating thebzlmod-enable branch, and landed on making this a pure repo rule. So now the API is this (as documented in # WORKSPACE# Register this toolchain before any others.register_toolchains("@rules_scala_protoc_toolchains//...:all")load("@platforms//host:extension.bzl","host_platform_repo")# Instantiates the `@host_platform` repo to work around:# - https://github.com/bazelbuild/bazel/issues/22558host_platform_repo(name="host_platform")# ...load `com_google_protobuf`, `rules_proto`, etc...load("@rules_scala//protoc:toolchains.bzl","scala_protoc_toolchains")# The name can be anything, but we recommend `rules_scala_protoc_toolchains`.# Only include `platforms` if you need additional platforms other than the# automatically detected host platform.scala_protoc_toolchains(name="rules_scala_protoc_toolchains",platforms= ["linux-x86_64"],) No need to use I've already pushed the corresponding module extension tobzlmod-enable, as |
@mbland, I'll try it out on monday. |
7a25bac to4e90449Compare@mbland , Tried latest version from today on my codebase and it worked. (Note I'm not using scalapb, not calling scala_toolchains()). Also, thanks so much for setting up mechanism for prebuilt protoc. I can't beleive they don't have prebuilt support in the rules_proto or protobuf repo anymore, but awesome that you did that.
|
Awesome! 😄
Heh, thanks. But I view it less as awesomeness on my part than necessity, since the Windows build was dead in the water without it. I'm just lucky I dabbled in exploring precompiled That said, there are hints that precompiled
I wouldn't go that far yet, becauseconsumers might not be prepared for or comfortable with using the precompiled That said, I would imagine many, if not most, consumers might take advantage of the option once they realize it exists. Also, FWIW, I'm not aware that
Sadly, no, not entirely, not as I understand it. The The The good news is that the I wouldhope these
Technically, no, we don't still need them when using the prebuilt
It is, but we can't remove it yet for two reasons: one hard requirement, and one nice to have. And in writing this, I realize I need to add this information to The hard requirement is that the precompiled The nice to have is that it enables both older and newer Specifically, this pull request ended up using doing the following in First, it loads the following directly from load("@com_google_protobuf//:protobuf_version.bzl","PROTOC_VERSION") Then it loads the
load("@rules_proto//proto:proto_common.bzl","toolchains") Finally, it pushes the following load("@com_google_protobuf//bazel/toolchains:proto_lang_toolchain.bzl","proto_lang_toolchain",)load("@com_google_protobuf//bazel/toolchains:proto_toolchain.bzl","proto_toolchain",)
scala_library_classpath= ["@maven//:org_scala_lang_scala_library","@maven//:org_scala_lang_scala_reflect",], If it doesn't contain a |
Dependency version and documentation updates broken out frombazel-contrib#1710.- Bazel: 7.5.0 => 7.6.0- gRPC: 2.53.0 => 2.54.1- Guava: 33.4.0-jre => 33.4.4-jre- Scalafmt: 3.9.2 => 3.9.4- `com-google-protobuf:protobuf-java`: 4.30.0 => 4.30.1- `org.scala-sbt:util-interface`: 1.10.10 => 1.10.11The documentation updates include:- Minor editing updates and typo fixes- The new "Removal of `bind()` aliases for `twitter_scrooge` dependencies" section of `README.md`The Guava 33.4.4-jre update introduces a new dependency on`org.jspecify:jspecify:1.0.0`. 33.4.5-jre changes Guava to be a Javamodule, but that release currently has problems, so we're sticking with34.4.4-jre until that 34.4.6-jre release. The Guava v33.4.1 releasemessage contains all the details.-https://jspecify.dev/-https://dev.java/learn/modules/intro/-https://github.com/google/guava/releases/tag/v33.4.1
Dependency version and documentation updates broken out frombazel-contrib#1710.- Bazel: 7.5.0 => 7.6.0- gRPC: 2.53.0 => 2.54.1- Guava: 33.4.0-jre => 33.4.4-jre- Scalafmt: 3.9.2 => 3.9.4- `com-google-protobuf:protobuf-java`: 4.30.0 => 4.30.1- `org.scala-sbt:util-interface`: 1.10.10 => 1.10.11The documentation updates include:- Minor editing updates and typo fixes- The new "Removal of `bind()` aliases for `twitter_scrooge` dependencies" section of `README.md`The Guava 33.4.4-jre update introduces a new dependency on`org.jspecify:jspecify:1.0.0`. 33.4.5-jre changes Guava to be a Javamodule, but that release currently has problems, so we're sticking with34.4.4-jre until that 34.4.6-jre release. The Guava v33.4.1 releasemessage contains all the details.-https://jspecify.dev/-https://dev.java/learn/modules/intro/-https://github.com/google/guava/releases/tag/v33.4.1
Dependency version and documentation updates broken out from#1710.- Bazel: 7.5.0 => 7.6.0- gRPC: 2.53.0 => 2.54.1- Guava: 33.4.0-jre => 33.4.4-jre- Scalafmt: 3.9.2 => 3.9.4- `com-google-protobuf:protobuf-java`: 4.30.0 => 4.30.1- `org.scala-sbt:util-interface`: 1.10.10 => 1.10.11The documentation updates include:- Minor editing updates and typo fixes- The new "Removal of `bind()` aliases for `twitter_scrooge` dependencies" section of `README.md`The Guava 33.4.4-jre update introduces a new dependency on`org.jspecify:jspecify:1.0.0`. 33.4.5-jre changes Guava to be a Javamodule, but that release currently has problems, so we're sticking with34.4.4-jre until that 34.4.6-jre release. The Guava v33.4.1 releasemessage contains all the details.-https://jspecify.dev/-https://dev.java/learn/modules/intro/-https://github.com/google/guava/releases/tag/v33.4.1
Bumps dependencies to versions that are compatible with both Bazel 7.6.0and 8.0.0, and adds protocol compiler toolchainization in `//protoc` for`protobuf` v29 and later.Closesbazel-contrib#1652. Part ofbazel-contrib#1482.- ScalaPB jars: 0.11.17 => 1.0.0-alpha.1- `rules_python`: 0.38.0 => 1.2.0- `rules_cc`: 0.0.9 => 0.1.1- `rules_java`: 7.12.4 => 8.11.0- `protobuf`: 21.7 => 30.1- `rules_proto`: 6.0.2 => 7.1.0Bazel 6 is officially unsupported as of this change and the upcoming`rules_scala` 7.0.0 release. Updates `.bazelci/presubmit.yml` to bumpthe `7.x` build to `last_rc`.Registers a precompiled protocol compiler toolchain when`--incompatible_enable_proto_toolchain_resolution` is `True`.Otherwise, `register_toolchains("@rules_scala_protoc_toolchains//:all")`is a no-op, as it will be empty.`scripts/update_protoc_integrity.py` automatically updates`scala/private/protoc/protoc_integrity.bzl`. The `protobuf` patch is the`git diff` output fromprotocolbuffers/protobuf#19679, which alsoinspired the updates to `scala_proto/scala_proto_toolchain.bzl`. The`proto_lang_toolchain` call in the `BUILD` file generated by`protoc/private/protoc_toolchain.bzl` was inspired by the `README` from:-https://github.com/aspect-build/toolchains_protoc/Loads `java_proto_library` from `com_google_protobuf`, replacing theofficially deprecated version from `rules_java`.Adds the `scala` parameter to `scala_toolchains()` to control whether itinstantiates the builtin Scala toolchains. Removes the `iflen(toolchains) == 0` check from `_scala_toolchains_repo_impl`. TheScala version check will now happen only when both `scala` and`validate_scala_version` are `True`, which is essentially how theprevious API worked.Updates to `README.md`, and updates to `WORKSPACE` and`third_party/repositories` files precipitated by the dependency updates,comprise the remainder of this change.---We're no longer planning to support Bazel 6 in the next major releaseper@simuons's decision in:-bazel-contrib#1482 (comment)The plan is now to land the Bazel 7 and 8 compatibility updates first,then land the Bzlmod change. This enables us to make only one new majorversion release, instead of two (whereby the first release would'vecontinued supporting Bazel 6).It turns out the two major version plan wouldn't've been possible.Bazel 8 and `rules_java` 8 require `protobuf` >= v29, but this bumpcaused Windows builds to break when compiling `protoc` inbazel-contrib#1710.`src/google/protobuf/compiler/java/java_features.pb.h`, the pathspecified in the error message, doesn't exist until `protobuf` v25.0.@crt-31 and I found that this was related to the Windows/MSVC 260character file path length limit. What's more, the `protobuf` teamplans to drop MSVC support specifically because of this path lengthlimit.The protocol compiler toolchain prevents `protoc` recompilation, whichfixes the Windows breakage while making all builds faster. Since Windowsbuilds break since at least `protobuf` v25, but `protoc`toolchainization requires v29, the version bump and the `protoc`toolchain must land together.Disabling the default Scala toolchain via `scala_toolchains(scala =False)` avoids instantiating any builtin compiler JAR repos orvalidating the Scala version. This enables users defining custom Scalatoolchains using their own JARs to still use other builtin toolchains.This was prompted by:bazel-contrib#1710 (comment)Removing the `if len(toolchains) == 0` covers the case in the upcomingBzlmod implementation whereby the root module may explicitly disable allbuiltin toolchains. This avoids potential breakage of the`register_toolchains("@rules_scala_toolchains//...:all")` call from theupcoming `MODULE.bazel` file. Removing the `scala_register_toolchains()`calls from the `dt_patches/test_dt_patches*/WORKSPACE` files proves thatthose calls were harmless, but ultimately unnecessary.---I tried several things to get protocol compiler toolchainization to workwith `protobuf` v28.2, described below. However, each path only led tothe same suffering described in the new "Why this requires `protobuf`v29 or later" section of the README.I discovered along the way that `protobuf` v30 isn't compatible withBazel 6.5.0 at all. I added an explanation to the "Limited Bazel 6.5.0compatibility" section of `README.md`.---I experimented with using `protobuf` v28.2, `rules_proto` 6.0.2, and`rules_java` 7.12.4 and 8.10.0. I updated the `protobuf` patch for v28.2with the following statements:```pyload("//bazel/common:proto_common.bzl", "proto_common")load("@rules_proto//proto:proto_common.bzl", "toolchains")_PROTO_TOOLCHAIN = "@rules_proto//proto:toolchain_type"_PROTO_TOOLCHAIN_ATTR = "INCOMPATIBLE_ENABLE_PROTO_TOOLCHAIN_RESOLUTION"_PROTOC_TOOLCHAINS = toolchains.use_toolchain(_PROTO_TOOLCHAIN)def _protoc_files_to_run(ctx): if getattr(proto_common, _PROTO_TOOLCHAIN_ATTR, False):```I updated `protoc/private/protoc_toolchain.bzl` to use `proto_common` from `rules_proto`.I also created a `rules_proto` 6.0.2 patch for `proto_toolchain()` tofix a "no such package: //proto" breakage:```diff 6.0.2 patch for `proto_toolchain()`:```diffdiff --git i/proto/private/rules/proto_toolchain.bzl w/proto/private/rules/proto_toolchain.bzlindex a091b80..def2699 100644--- i/proto/private/rules/proto_toolchain.bzl+++ w/proto/private/rules/proto_toolchain.bzl@@ -33,7 +33,7 @@ def proto_toolchain(*, name, proto_compiler, exec_compatible_with = []): native.toolchain( name = name + "_toolchain",- toolchain_type = "//proto:toolchain_type",+ toolchain_type = Label("//proto:toolchain_type"), exec_compatible_with = exec_compatible_with, target_compatible_with = [], toolchain = name,```I tried adding combinations of the following`--incompatible_autoload_externally` flag values to .bazelrc`:```txtcommon --incompatible_autoload_externally=+@protobuf,+@rules_java```Nothing worked.---After the `protobuf` v29 bump, and before the ScalaPB 1.0.0-alpha.1bump, `scala_proto` targets would fail with the following error:```txtERROR: .../external/com_google_protobuf/src/google/protobuf/BUILD.bazel:23:14: ProtoScalaPBRule external/com_google_protobuf/src/google/protobuf/any_proto_jvm_extra_protobuf_generator_scalapb.srcjar failed: (Exit 1): scalapb_worker failed: error executing ProtoScalaPBRule command (from target @@com_google_protobuf//src/google/protobuf:any_proto) bazel-out/.../bin/src/scala/scripts/scalapb_worker ... (remaining 2 arguments skipped)--jvm_extra_protobuf_generator_out: java.lang.NoSuchMethodError: 'java.lang.Object com.google.protobuf.DescriptorProtos$FieldOptions.getExtension(com.google.protobuf.GeneratedMessage$GeneratedExtension)' at scalapb.compiler.DescriptorImplicits$ExtendedFieldDescriptor.fieldOptions(DescriptorImplicits.scala:329) [ ...snip... ]java.lang.RuntimeException: Exit with code 1 at scala.sys.package$.error(package.scala:30) at scripts.ScalaPBWorker$.work(ScalaPBWorker.scala:44) at io.bazel.rulesscala.worker.Worker.persistentWorkerMain(Worker.java:96) at io.bazel.rulesscala.worker.Worker.workerMain(Worker.java:49) at scripts.ScalaPBWorker$.main(ScalaPBWorker.scala:39) at scripts.ScalaPBWorker.main(ScalaPBWorker.scala)ERROR: .../external/com_google_protobuf/src/google/protobuf/BUILD.bazel:23:14 Building source jar external/com_google_protobuf/src/google/protobuf/any_proto_scalapb-src.jar failed: (Exit 1): scalapb_worker failed: error executing ProtoScalaPBRule command (from target @@com_google_protobuf//src/google/protobuf:any_proto) bazel-out/darwin_arm64-opt-exec-ST-a828a81199fe/bin/src/scala/scripts/scalapb_worker ... (remaining 2 arguments skipped)```
39c32df to0aab4bcCompareBumps dependencies to versions that are compatible with both Bazel 7.6.0and 8.0.0, and adds protocol compiler toolchainization in `//protoc` for`protobuf` v29 and later.Closesbazel-contrib#1652. Part ofbazel-contrib#1482.- ScalaPB jars: 0.11.17 => 1.0.0-alpha.1- `rules_python`: 0.38.0 => 1.2.0- `rules_cc`: 0.0.9 => 0.1.1- `rules_java`: 7.12.4 => 8.11.0- `protobuf`: 21.7 => 30.1- `rules_proto`: 6.0.2 => 7.1.0Bazel 6 is officially unsupported as of this change and the upcoming`rules_scala` 7.0.0 release. Updates `.bazelci/presubmit.yml` to bumpthe `7.x` build to `last_rc`.Registers a precompiled protocol compiler toolchain when`--incompatible_enable_proto_toolchain_resolution` is `True`.Otherwise, `register_toolchains("@rules_scala_protoc_toolchains//:all")`is a no-op, as it will be empty.`scripts/update_protoc_integrity.py` automatically updates`scala/private/protoc/protoc_integrity.bzl`. The `protobuf` patch is the`git diff` output fromprotocolbuffers/protobuf#19679, which alsoinspired the updates to `scala_proto/scala_proto_toolchain.bzl`. The`proto_lang_toolchain` call in the `BUILD` file generated by`protoc/private/protoc_toolchain.bzl` was inspired by the `README` from:-https://github.com/aspect-build/toolchains_protoc/Loads `java_proto_library` from `com_google_protobuf`, replacing theofficially deprecated version from `rules_java`.Adds the `scala` parameter to `scala_toolchains()` to control whether itinstantiates the builtin Scala toolchains. Removes the `iflen(toolchains) == 0` check from `_scala_toolchains_repo_impl`. TheScala version check will now happen only when both `scala` and`validate_scala_version` are `True`, which is essentially how theprevious API worked.Updates to `README.md`, and updates to `WORKSPACE` and`third_party/repositories` files precipitated by the dependency updates,comprise the remainder of this change.---We're no longer planning to support Bazel 6 in the next major releaseper@simuons's decision in:-bazel-contrib#1482 (comment)The plan is now to land the Bazel 7 and 8 compatibility updates first,then land the Bzlmod change. This enables us to make only one new majorversion release, instead of two (whereby the first release would'vecontinued supporting Bazel 6).It turns out the two major version plan wouldn't've been possible.Bazel 8 and `rules_java` 8 require `protobuf` >= v29, but this bumpcaused Windows builds to break when compiling `protoc` inbazel-contrib#1710.`src/google/protobuf/compiler/java/java_features.pb.h`, the pathspecified in the error message, doesn't exist until `protobuf` v25.0.@crt-31 and I found that this was related to the Windows/MSVC 260character file path length limit. What's more, the `protobuf` teamplans to drop MSVC support specifically because of this path lengthlimit.The protocol compiler toolchain prevents `protoc` recompilation, whichfixes the Windows breakage while making all builds faster. Since Windowsbuilds break since at least `protobuf` v25, but `protoc`toolchainization requires v29, the version bump and the `protoc`toolchain must land together.Disabling the default Scala toolchain via `scala_toolchains(scala =False)` avoids instantiating any builtin compiler JAR repos orvalidating the Scala version. This enables users defining custom Scalatoolchains using their own JARs to still use other builtin toolchains.This was prompted by:bazel-contrib#1710 (comment)Removing the `if len(toolchains) == 0` covers the case in the upcomingBzlmod implementation whereby the root module may explicitly disable allbuiltin toolchains. This avoids potential breakage of the`register_toolchains("@rules_scala_toolchains//...:all")` call from theupcoming `MODULE.bazel` file. Removing the `scala_register_toolchains()`calls from the `dt_patches/test_dt_patches*/WORKSPACE` files proves thatthose calls were harmless, but ultimately unnecessary.---I tried several things to get protocol compiler toolchainization to workwith `protobuf` v28.2, described below. However, each path only led tothe same suffering described in the new "Why this requires `protobuf`v29 or later" section of the README.I discovered along the way that `protobuf` v30 isn't compatible withBazel 6.5.0 at all. I added an explanation to the "Limited Bazel 6.5.0compatibility" section of `README.md`.---I experimented with using `protobuf` v28.2, `rules_proto` 6.0.2, and`rules_java` 7.12.4 and 8.10.0. I updated the `protobuf` patch for v28.2with the following statements:```pyload("//bazel/common:proto_common.bzl", "proto_common")load("@rules_proto//proto:proto_common.bzl", "toolchains")_PROTO_TOOLCHAIN = "@rules_proto//proto:toolchain_type"_PROTO_TOOLCHAIN_ATTR = "INCOMPATIBLE_ENABLE_PROTO_TOOLCHAIN_RESOLUTION"_PROTOC_TOOLCHAINS = toolchains.use_toolchain(_PROTO_TOOLCHAIN)def _protoc_files_to_run(ctx): if getattr(proto_common, _PROTO_TOOLCHAIN_ATTR, False):```I updated `protoc/private/protoc_toolchain.bzl` to use `proto_common` from `rules_proto`.I also created a `rules_proto` 6.0.2 patch for `proto_toolchain()` tofix a "no such package: //proto" breakage:```diff 6.0.2 patch for `proto_toolchain()`:```diffdiff --git i/proto/private/rules/proto_toolchain.bzl w/proto/private/rules/proto_toolchain.bzlindex a091b80..def2699 100644--- i/proto/private/rules/proto_toolchain.bzl+++ w/proto/private/rules/proto_toolchain.bzl@@ -33,7 +33,7 @@ def proto_toolchain(*, name, proto_compiler, exec_compatible_with = []): native.toolchain( name = name + "_toolchain",- toolchain_type = "//proto:toolchain_type",+ toolchain_type = Label("//proto:toolchain_type"), exec_compatible_with = exec_compatible_with, target_compatible_with = [], toolchain = name,```I tried adding combinations of the following`--incompatible_autoload_externally` flag values to .bazelrc`:```txtcommon --incompatible_autoload_externally=+@protobuf,+@rules_java```Nothing worked.---After the `protobuf` v29 bump, and before the ScalaPB 1.0.0-alpha.1bump, `scala_proto` targets would fail with the following error:```txtERROR: .../external/com_google_protobuf/src/google/protobuf/BUILD.bazel:23:14: ProtoScalaPBRule external/com_google_protobuf/src/google/protobuf/any_proto_jvm_extra_protobuf_generator_scalapb.srcjar failed: (Exit 1): scalapb_worker failed: error executing ProtoScalaPBRule command (from target @@com_google_protobuf//src/google/protobuf:any_proto) bazel-out/.../bin/src/scala/scripts/scalapb_worker ... (remaining 2 arguments skipped)--jvm_extra_protobuf_generator_out: java.lang.NoSuchMethodError: 'java.lang.Object com.google.protobuf.DescriptorProtos$FieldOptions.getExtension(com.google.protobuf.GeneratedMessage$GeneratedExtension)' at scalapb.compiler.DescriptorImplicits$ExtendedFieldDescriptor.fieldOptions(DescriptorImplicits.scala:329) [ ...snip... ]java.lang.RuntimeException: Exit with code 1 at scala.sys.package$.error(package.scala:30) at scripts.ScalaPBWorker$.work(ScalaPBWorker.scala:44) at io.bazel.rulesscala.worker.Worker.persistentWorkerMain(Worker.java:96) at io.bazel.rulesscala.worker.Worker.workerMain(Worker.java:49) at scripts.ScalaPBWorker$.main(ScalaPBWorker.scala:39) at scripts.ScalaPBWorker.main(ScalaPBWorker.scala)ERROR: .../external/com_google_protobuf/src/google/protobuf/BUILD.bazel:23:14 Building source jar external/com_google_protobuf/src/google/protobuf/any_proto_scalapb-src.jar failed: (Exit 1): scalapb_worker failed: error executing ProtoScalaPBRule command (from target @@com_google_protobuf//src/google/protobuf:any_proto) bazel-out/darwin_arm64-opt-exec-ST-a828a81199fe/bin/src/scala/scripts/scalapb_worker ... (remaining 2 arguments skipped)```
0aab4bc to3ddf23bCompareRebased on |
Bumps dependencies to versions that are compatible with both Bazel 7.6.0and 8.0.0, and adds protocol compiler toolchainization in `//protoc` for`protobuf` v29 and later.Closesbazel-contrib#1652. Part ofbazel-contrib#1482.- ScalaPB jars: 0.11.17 => 1.0.0-alpha.1- `rules_python`: 0.38.0 => 1.2.0- `rules_cc`: 0.0.9 => 0.1.1- `rules_java`: 7.12.4 => 8.11.0- `protobuf`: 21.7 => 30.1- `rules_proto`: 6.0.2 => 7.1.0Bazel 6 is officially unsupported as of this change and the upcoming`rules_scala` 7.0.0 release. Updates `.bazelci/presubmit.yml` to bumpthe `7.x` build to `last_rc`.Registers a precompiled protocol compiler toolchain when`--incompatible_enable_proto_toolchain_resolution` is `True`.Otherwise, `register_toolchains("@rules_scala_protoc_toolchains//:all")`is a no-op, as it will be empty.`scripts/update_protoc_integrity.py` automatically updates`scala/private/protoc/protoc_integrity.bzl`. The `protobuf` patch is the`git diff` output fromprotocolbuffers/protobuf#19679, which alsoinspired the updates to `scala_proto/scala_proto_toolchain.bzl`. The`proto_lang_toolchain` call in the `BUILD` file generated by`protoc/private/protoc_toolchains.bzl` was inspired by the `README`from:-https://github.com/aspect-build/toolchains_protoc/Loads `java_proto_library` from `com_google_protobuf`, replacing theofficially deprecated version from `rules_java`.Adds the `scala` parameter to `scala_toolchains()` to control whether itinstantiates the builtin Scala toolchains. Removes the `iflen(toolchains) == 0` check from `_scala_toolchains_repo_impl`. TheScala version check will now happen only when both `scala` and`validate_scala_version` are `True`, which is essentially how theprevious API worked.Updates to `README.md`, and updates to `WORKSPACE` and`third_party/repositories` files precipitated by the dependency updates,comprise the remainder of this change.---We're no longer planning to support Bazel 6 in the next major releaseper@simuons's decision in:-bazel-contrib#1482 (comment)The plan is now to land the Bazel 7 and 8 compatibility updates first,then land the Bzlmod change. This enables us to make only one new majorversion release, instead of two (whereby the first release would'vecontinued supporting Bazel 6).It turns out the two major version plan wouldn't've been possible.Bazel 8 and `rules_java` 8 require `protobuf` >= v29, but this bumpcaused Windows builds to break when compiling `protoc` inbazel-contrib#1710.`src/google/protobuf/compiler/java/java_features.pb.h`, the pathspecified in the error message, doesn't exist until `protobuf` v25.0.@crt-31 and I found that this was related to the Windows/MSVC 260character file path length limit. What's more, the `protobuf` teamplans to drop MSVC support specifically because of this path lengthlimit.The protocol compiler toolchain prevents `protoc` recompilation, whichfixes the Windows breakage while making all builds faster. Since Windowsbuilds break since at least `protobuf` v25, but `protoc`toolchainization requires v29, the version bump and the `protoc`toolchain must land together.Disabling the default Scala toolchain via `scala_toolchains(scala =False)` avoids instantiating any builtin compiler JAR repos orvalidating the Scala version. This enables users defining custom Scalatoolchains using their own JARs to still use other builtin toolchains.This was prompted by:bazel-contrib#1710 (comment)Removing the `if len(toolchains) == 0` covers the case in the upcomingBzlmod implementation whereby the root module may explicitly disable allbuiltin toolchains. This avoids potential breakage of the`register_toolchains("@rules_scala_toolchains//...:all")` call from theupcoming `MODULE.bazel` file. Removing the `scala_register_toolchains()`calls from the `dt_patches/test_dt_patches*/WORKSPACE` files proves thatthose calls were harmless, but ultimately unnecessary.---I tried several things to get protocol compiler toolchainization to workwith `protobuf` v28.2, described below. However, each path only led tothe same suffering described in the new "Why this requires `protobuf`v29 or later" section of the README.I discovered along the way that `protobuf` v30 isn't compatible withBazel 6.5.0 at all. I added an explanation to the "Limited Bazel 6.5.0compatibility" section of `README.md`.---I experimented with using `protobuf` v28.2, `rules_proto` 6.0.2, and`rules_java` 7.12.4 and 8.10.0. I updated the `protobuf` patch for v28.2with the following statements:```pyload("//bazel/common:proto_common.bzl", "proto_common")load("@rules_proto//proto:proto_common.bzl", "toolchains")_PROTO_TOOLCHAIN = "@rules_proto//proto:toolchain_type"_PROTO_TOOLCHAIN_ATTR = "INCOMPATIBLE_ENABLE_PROTO_TOOLCHAIN_RESOLUTION"_PROTOC_TOOLCHAINS = toolchains.use_toolchain(_PROTO_TOOLCHAIN)def _protoc_files_to_run(ctx): if getattr(proto_common, _PROTO_TOOLCHAIN_ATTR, False):```I tried using `proto_common` from `rules_proto`. I also created a`rules_proto` 6.0.2 patch for `proto_toolchain()` to fix a "no suchpackage: //proto" breakage:```diffdiff --git i/proto/private/rules/proto_toolchain.bzl w/proto/private/rules/proto_toolchain.bzlindex a091b80..def2699 100644--- i/proto/private/rules/proto_toolchain.bzl+++ w/proto/private/rules/proto_toolchain.bzl@@ -33,7 +33,7 @@ def proto_toolchain(*, name, proto_compiler, exec_compatible_with = []): native.toolchain( name = name + "_toolchain",- toolchain_type = "//proto:toolchain_type",+ toolchain_type = Label("//proto:toolchain_type"), exec_compatible_with = exec_compatible_with, target_compatible_with = [], toolchain = name,```I tried adding combinations of the following`--incompatible_autoload_externally` flag values to .bazelrc`:```txtcommon --incompatible_autoload_externally=+@protobuf,+@rules_java```Nothing worked.---After the `protobuf` v29 bump, and before the ScalaPB 1.0.0-alpha.1bump, `scala_proto` targets would fail with the following error:```txtERROR: .../external/com_google_protobuf/src/google/protobuf/BUILD.bazel:23:14: ProtoScalaPBRule external/com_google_protobuf/src/google/protobuf/any_proto_jvm_extra_protobuf_generator_scalapb.srcjar failed: (Exit 1): scalapb_worker failed: error executing ProtoScalaPBRule command (from target @@com_google_protobuf//src/google/protobuf:any_proto) bazel-out/.../bin/src/scala/scripts/scalapb_worker ... (remaining 2 arguments skipped)--jvm_extra_protobuf_generator_out: java.lang.NoSuchMethodError: 'java.lang.Object com.google.protobuf.DescriptorProtos$FieldOptions.getExtension(com.google.protobuf.GeneratedMessage$GeneratedExtension)' at scalapb.compiler.DescriptorImplicits$ExtendedFieldDescriptor.fieldOptions(DescriptorImplicits.scala:329) [ ...snip... ]java.lang.RuntimeException: Exit with code 1 at scala.sys.package$.error(package.scala:30) at scripts.ScalaPBWorker$.work(ScalaPBWorker.scala:44) at io.bazel.rulesscala.worker.Worker.persistentWorkerMain(Worker.java:96) at io.bazel.rulesscala.worker.Worker.workerMain(Worker.java:49) at scripts.ScalaPBWorker$.main(ScalaPBWorker.scala:39) at scripts.ScalaPBWorker.main(ScalaPBWorker.scala)ERROR: .../external/com_google_protobuf/src/google/protobuf/BUILD.bazel:23:14 Building source jar external/com_google_protobuf/src/google/protobuf/any_proto_scalapb-src.jar failed: (Exit 1): scalapb_worker failed: error executing ProtoScalaPBRule command (from target @@com_google_protobuf//src/google/protobuf:any_proto) bazel-out/darwin_arm64-opt-exec-ST-a828a81199fe/bin/src/scala/scripts/scalapb_worker ... (remaining 2 arguments skipped)```
3ddf23b to73f71b5CompareBumps dependencies to versions that are compatible with both Bazel 7.6.0and 8.0.0, and adds protocol compiler toolchainization in `//protoc` for`protobuf` v29 and later.Closesbazel-contrib#1652. Part ofbazel-contrib#1482.- ScalaPB jars: 0.11.17 => 1.0.0-alpha.1- `rules_python`: 0.38.0 => 1.2.0- `rules_cc`: 0.0.9 => 0.1.1- `rules_java`: 7.12.4 => 8.11.0- `protobuf`: 21.7 => 30.1- `rules_proto`: 6.0.2 => 7.1.0Bazel 6 is officially unsupported as of this change and the upcoming`rules_scala` 7.0.0 release. Updates `.bazelci/presubmit.yml` to bumpthe `7.x` build to `last_rc`.Registers a precompiled protocol compiler toolchain when`--incompatible_enable_proto_toolchain_resolution` is `True`.Otherwise, `register_toolchains("@rules_scala_protoc_toolchains//:all")`is a no-op, as it will be empty.`scripts/update_protoc_integrity.py` automatically updates`scala/private/protoc/protoc_integrity.bzl`. The `protobuf` patch is the`git diff` output fromprotocolbuffers/protobuf#19679, which alsoinspired the updates to `scala_proto/scala_proto_toolchain.bzl`. The`proto_lang_toolchain` call in the `BUILD` file generated by`protoc/private/protoc_toolchains.bzl` was inspired by the `README`from:-https://github.com/aspect-build/toolchains_protoc/Loads `java_proto_library` from `com_google_protobuf`, replacing theofficially deprecated version from `rules_java`.Adds the `scala` parameter to `scala_toolchains()` to control whether itinstantiates the builtin Scala toolchains. Removes the `iflen(toolchains) == 0` check from `_scala_toolchains_repo_impl`. TheScala version check will now happen only when both `scala` and`validate_scala_version` are `True`, which is essentially how theprevious API worked.Updates to `README.md`, and updates to `WORKSPACE` and`third_party/repositories` files precipitated by the dependency updates,comprise the remainder of this change.---We're no longer planning to support Bazel 6 in the next major releaseper@simuons's decision in:-bazel-contrib#1482 (comment)The plan is now to land the Bazel 7 and 8 compatibility updates first,then land the Bzlmod change. This enables us to make only one new majorversion release, instead of two (whereby the first release would'vecontinued supporting Bazel 6).It turns out the two major version plan wouldn't've been possible.Bazel 8 and `rules_java` 8 require `protobuf` >= v29, but this bumpcaused Windows builds to break when compiling `protoc` inbazel-contrib#1710.`src/google/protobuf/compiler/java/java_features.pb.h`, the pathspecified in the error message, doesn't exist until `protobuf` v25.0.@crt-31 and I found that this was related to the Windows/MSVC 260character file path length limit. What's more, the `protobuf` teamplans to drop MSVC support specifically because of this path lengthlimit.The protocol compiler toolchain prevents `protoc` recompilation, whichfixes the Windows breakage while making all builds faster. Since Windowsbuilds break since at least `protobuf` v25, but `protoc`toolchainization requires v29, the version bump and the `protoc`toolchain must land together.Disabling the default Scala toolchain via `scala_toolchains(scala =False)` avoids instantiating any builtin compiler JAR repos orvalidating the Scala version. This enables users defining custom Scalatoolchains using their own JARs to still use other builtin toolchains.This was prompted by:bazel-contrib#1710 (comment)Removing the `if len(toolchains) == 0` covers the case in the upcomingBzlmod implementation whereby the root module may explicitly disable allbuiltin toolchains. This avoids potential breakage of the`register_toolchains("@rules_scala_toolchains//...:all")` call from theupcoming `MODULE.bazel` file. Removing the `scala_register_toolchains()`calls from the `dt_patches/test_dt_patches*/WORKSPACE` files proves thatthose calls were harmless, but ultimately unnecessary.---I tried several things to get protocol compiler toolchainization to workwith `protobuf` v28.2, described below. However, each path only led tothe same suffering described in the new "Why this requires `protobuf`v29 or later" section of the README.I discovered along the way that `protobuf` v30 isn't compatible withBazel 6.5.0 at all. I added an explanation to the "Limited Bazel 6.5.0compatibility" section of `README.md`.---I experimented with using `protobuf` v28.2, `rules_proto` 6.0.2, and`rules_java` 7.12.4 and 8.10.0. I updated the `protobuf` patch for v28.2with the following statements:```pyload("//bazel/common:proto_common.bzl", "proto_common")load("@rules_proto//proto:proto_common.bzl", "toolchains")_PROTO_TOOLCHAIN = "@rules_proto//proto:toolchain_type"_PROTO_TOOLCHAIN_ATTR = "INCOMPATIBLE_ENABLE_PROTO_TOOLCHAIN_RESOLUTION"_PROTOC_TOOLCHAINS = toolchains.use_toolchain(_PROTO_TOOLCHAIN)def _protoc_files_to_run(ctx): if getattr(proto_common, _PROTO_TOOLCHAIN_ATTR, False):```I tried using `proto_common` from `rules_proto`. I also created a`rules_proto` 6.0.2 patch for `proto_toolchain()` to fix a "no suchpackage: //proto" breakage:```diffdiff --git i/proto/private/rules/proto_toolchain.bzl w/proto/private/rules/proto_toolchain.bzlindex a091b80..def2699 100644--- i/proto/private/rules/proto_toolchain.bzl+++ w/proto/private/rules/proto_toolchain.bzl@@ -33,7 +33,7 @@ def proto_toolchain(*, name, proto_compiler, exec_compatible_with = []): native.toolchain( name = name + "_toolchain",- toolchain_type = "//proto:toolchain_type",+ toolchain_type = Label("//proto:toolchain_type"), exec_compatible_with = exec_compatible_with, target_compatible_with = [], toolchain = name,```I tried adding combinations of the following`--incompatible_autoload_externally` flag values to .bazelrc`:```txtcommon --incompatible_autoload_externally=+@protobuf,+@rules_java```Nothing worked.---After the `protobuf` v29 bump, and before the ScalaPB 1.0.0-alpha.1bump, `scala_proto` targets would fail with the following error:```txtERROR: .../external/com_google_protobuf/src/google/protobuf/BUILD.bazel:23:14: ProtoScalaPBRule external/com_google_protobuf/src/google/protobuf/any_proto_jvm_extra_protobuf_generator_scalapb.srcjar failed: (Exit 1): scalapb_worker failed: error executing ProtoScalaPBRule command (from target @@com_google_protobuf//src/google/protobuf:any_proto) bazel-out/.../bin/src/scala/scripts/scalapb_worker ... (remaining 2 arguments skipped)--jvm_extra_protobuf_generator_out: java.lang.NoSuchMethodError: 'java.lang.Object com.google.protobuf.DescriptorProtos$FieldOptions.getExtension(com.google.protobuf.GeneratedMessage$GeneratedExtension)' at scalapb.compiler.DescriptorImplicits$ExtendedFieldDescriptor.fieldOptions(DescriptorImplicits.scala:329) [ ...snip... ]java.lang.RuntimeException: Exit with code 1 at scala.sys.package$.error(package.scala:30) at scripts.ScalaPBWorker$.work(ScalaPBWorker.scala:44) at io.bazel.rulesscala.worker.Worker.persistentWorkerMain(Worker.java:96) at io.bazel.rulesscala.worker.Worker.workerMain(Worker.java:49) at scripts.ScalaPBWorker$.main(ScalaPBWorker.scala:39) at scripts.ScalaPBWorker.main(ScalaPBWorker.scala)ERROR: .../external/com_google_protobuf/src/google/protobuf/BUILD.bazel:23:14 Building source jar external/com_google_protobuf/src/google/protobuf/any_proto_scalapb-src.jar failed: (Exit 1): scalapb_worker failed: error executing ProtoScalaPBRule command (from target @@com_google_protobuf//src/google/protobuf:any_proto) bazel-out/darwin_arm64-opt-exec-ST-a828a81199fe/bin/src/scala/scripts/scalapb_worker ... (remaining 2 arguments skipped)```
73f71b5 to855222eCompareRebased on |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Thanks@mbland and sorry it took longer than necessary.
38391c1 intobazel-contrib:masterUh oh!
There was an error while loading.Please reload this page.
@simuons Thanks, and no worries. We definitely straightened out a few things in the process. I've rebased and updated my local That said, it will include changes to update the release workflow to publish to the Bazel Central Registry, replete with provenance attestations. (There'll be comments and a |
Uh oh!
There was an error while loading.Please reload this page.
Description
Bumps dependencies to versions that are compatible with both Bazel 7.6.0 and 8.1.1, and adds protocol compiler toolchainization in
//protocforprotobufv29 and later.Closes#1652. Part of#1482.
rules_python: 0.38.0 => 1.2.0rules_cc: 0.0.9 => 0.1.1rules_java: 7.12.4 => 8.11.0protobuf: 21.7 => 30.1rules_proto: 6.0.2 => 7.1.0Bazel 6 is officially unsupported as of this change and the upcoming
rules_scala7.0.0 release. Updates.bazelci/presubmit.ymlto bump the7.xbuild tolast_rc.Registers a precompiled protocol compiler toolchain when
--incompatible_enable_proto_toolchain_resolutionisTrue. Otherwise,register_toolchains("@rules_scala_protoc_toolchains//:all")toolchains is a no-op, as it will be empty.scripts/update_protoc_integrity.pyautomatically updatesscala/private/protoc/protoc_integrity.bzl. Theprotobufpatch is thegit diffoutput fromprotocolbuffers/protobuf#19679, which also inspired the updates toscala_proto/scala_proto_toolchain.bzl. Theproto_lang_toolchaincall in theBUILDfile generated byprotoc/private/protoc_toolchain.bzlwas inspired by theREADMEfrom:Loads
java_proto_libraryfromcom_google_protobuf, replacing the officially deprecated version fromrules_java.Adds the
scalaparameter toscala_toolchains()to control whether it instantiates the builtin Scala toolchains. Removes theif len(toolchains) == 0check from_scala_toolchains_repo_impl. The Scala version check will now happen only when bothscalaandvalidate_scala_versionareTrue, which is essentially how the previous API worked.Updates to
README.md, and updates toWORKSPACEandthird_party/repositoriesfiles precipitated by the dependency updates, comprise the remainder of this change.Motivation
We're no longer planning to support Bazel 6 in the next major release per@simuons's decision in:
The plan is now to land the Bazel 7 and 8 compatibility updates first, then land the Bzlmod change. This enables us to make only one new major version release, instead of two (whereby the first release would've continued supporting Bazel 6).
It turns out the two major version plan wouldn't've been possible. Bazel 8 and
rules_java8 requireprotobuf>= v29, but this bump caused Windows builds to break when compilingprotocin#1710.src/google/protobuf/compiler/java/java_features.pb.h, the path specified in the error message, doesn't exist untilprotobufv25.0.@crt-31 and I found that this was related to the Windows/MSVC 260 character file path length limit. What's more, the
protobufteam plans to drop MSVC support specifically because of this path length limit.The protocol compiler toolchain prevents
protocrecompilation, which fixes the Windows breakage while making all builds faster. Since Windows builds break since at leastprotobufv25, butprotoctoolchainization requires v29, the version bump and theprotoctoolchain must land together.Disabling the default Scala toolchain via
scala_toolchains(scala = False)avoids instantiating any builtin compiler JAR repos or validating the Scala version. This enables users defining custom Scala toolchains using their own JARs to still use other builtin toolchains. This was prompted by:#1710 (comment)Removing the
if len(toolchains) == 0covers the case in the upcoming Bzlmod implementation whereby the root module may explicitly disable all builtin toolchains. This avoids potential breakage of theregister_toolchains("@rules_scala_toolchains//...:all")call from the upcomingMODULE.bazelfile. Removing thescala_register_toolchains()calls from thedt_patches/test_dt_patches*/WORKSPACEfiles proves that those calls were harmless, but ultimately unnecessary.I tried several things to get protocol compiler toolchainization to work with
protobufv28.2, described below. However, each path only led to the same suffering described in the new "Why this requiresprotobufv29 or later" section of the README.I discovered along the way that
protobufv30 isn't compatible with Bazel 6.5.0 at all. I added an explanation to the "Limited Bazel 6.5.0 compatibility" section ofREADME.md.I experimented with using
protobufv28.2,rules_proto6.0.2, andrules_java7.12.4 and 8.10.0. I updated theprotobufpatch for v28.2 with the following statements:I updated
protoc/private/protoc_toolchain.bzlto useproto_commonfromrules_proto.I also created a
rules_proto6.0.2 patch forproto_toolchain()to fix a "no such package: //proto" breakage:I tried adding combinations of the following
--incompatible_autoload_externallyflag values to .bazelrc`:Nothing worked.
After the
protobufv29 bump, and before the ScalaPB 1.0.0-alpha.1 bump,scala_prototargets would fail with the following error: