[Bug]: CI images for Bazel 6.x (and possibly 7.x?) are broken for Java rules

[jmillikin]

What happened?

BCR CI fails for repositories that build Java libraries with Bazel 6.x as part of their presubmit, due to a crash in something called Turbine that appears to be an internal component of Bazel's Java toolchain.

This failure doesn't reproduce for me on any of the actual machines I test on (macOS x86-64, macOS arm64, Ubuntu 20.04 x86-64) so I think it's something specific to the way the container images are constructed, such as an unusual combination of Java and Bazel versions.

This makes the PR process frustrating for modules that have Java functionality, because the presubmit requires manual approval to run but it fails for something that I don't test for and for which verifying a fix is difficult.

Example affected PRs:

• Add javacc and javaparser #2728, which was solved by adding a dependency on rules_java(version = "6.0.0") on the theory that this version included a fixed build of Turbine.
• [email protected] #2904, which tried the same workaround without success.

Version

Development (host) and target OS/architectures:
Happening in BCR CI under an unknown set of conditions, so Linux/macOS/Windows probably.

Output of bazel --version:
bazel 6.5.0, but possibly also affects some Bazel 7.x versions (I noticed a 7.x failure at one point, in PR 2904)

Version of relevant rules from the WORKSPACE or MODULE.bazel file:
n/a

Language(s) and/or frameworks involved:
Java

How to reproduce

Using the same gcr.io/bazel-public/ubuntu2004 OCI container image as BCR CI runs, I was able to reproduce the issue with a "hello world" Java library.

$ podman run -it --rm --net=host -v $PWD/e2e:/src/e2e -w /src/e2e gcr.io/bazel-public/ubuntu2004
root@desktop:/src/e2e# export XDG_CACHE_HOME=/src/e2e/cache
root@desktop:/src/e2e# cat MODULE.bazel 
root@desktop:/src/e2e# cat .bazelversion
6.5.0
root@desktop:/src/e2e# cat .bazelrc
common --enable_bzlmod
common --repository_cache=/src/e2e/cache/repository-cache
root@desktop:/src/e2e# cat BUILD.bazel 
java_library(
    name = "HelloLib",
    srcs = [":HelloLib.java"],
)

java_binary(
    name = "HelloBin",
    srcs = ["HelloBin.java"],
    main_class = "HelloBin",
    deps = [":HelloLib"],
)
root@desktop:/src/e2e# cat HelloLib.java 
public class HelloLib {
	public static void SayHello(String greeting) {
		System.out.println(greeting);
	}
}
root@desktop:/src/e2e# cat HelloBin.java 
public class HelloBin {
	public static void main() {
		HelloLib.SayHello("Hello, world!");
	}
}

Building the Java targets will involve Turbine to get involved and then crash:

root@desktop:/src/e2e# bazel build -s //:HelloBin.jar
INFO: Analyzed target //:HelloBin.jar (66 packages loaded, 1099 targets configured).
INFO: Found 1 target...
SUBCOMMAND: # @rules_java~5.5.1//toolchains:platformclasspath [action 'JavaToolchainCompileClasses external/rules_java~5.5.1/toolchains/platformclasspath_classes', configuration: eae08a0147418bbed67463ae5f68913a92ab5c68abe9a3a6ea98eee64dbbdd8a, execution platform: @local_config_platform//:host]
(cd /root/.cache/bazel/_bazel_root/05153e06462a8431ceb96e94c5b047b4/execroot/_main && \
  exec env - \
  external/rules_java~5.5.1~toolchains~remotejdk11_linux/bin/javac -source 8 -target 8 -Xlint:-options -d bazel-out/k8-fastbuild/bin/external/rules_java~5.5.1/toolchains/platformclasspath_classes external/rules_java~5.5.1/toolchains/DumpPlatformClassPath.java)
# Configuration: eae08a0147418bbed67463ae5f68913a92ab5c68abe9a3a6ea98eee64dbbdd8a
# Execution platform: @local_config_platform//:host
SUBCOMMAND: # @rules_java~5.5.1//toolchains:platformclasspath [action 'JavaToolchainCompileBootClasspath external/rules_java~5.5.1/toolchains/platformclasspath.jar', configuration: eae08a0147418bbed67463ae5f68913a92ab5c68abe9a3a6ea98eee64dbbdd8a, execution platform: @local_config_platform//:host]
(cd /root/.cache/bazel/_bazel_root/05153e06462a8431ceb96e94c5b047b4/execroot/_main && \
  exec env - \
  external/rules_java~5.5.1~toolchains~remotejdk11_linux/bin/java -XX:+IgnoreUnrecognizedVMOptions '--add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED' '--add-exports=jdk.compiler/com.sun.tools.javac.platform=ALL-UNNAMED' '--add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED' -cp bazel-out/k8-fastbuild/bin/external/rules_java~5.5.1/toolchains/platformclasspath_classes DumpPlatformClassPath bazel-out/k8-fastbuild/bin/external/rules_java~5.5.1/toolchains/platformclasspath.jar external/rules_java~5.5.1~toolchains~local_jdk)
# Configuration: eae08a0147418bbed67463ae5f68913a92ab5c68abe9a3a6ea98eee64dbbdd8a
# Execution platform: @local_config_platform//:host
SUBCOMMAND: # //:HelloLib [action 'Compiling Java headers libHelloLib-hjar.jar (1 source file)', configuration: eae08a0147418bbed67463ae5f68913a92ab5c68abe9a3a6ea98eee64dbbdd8a, execution platform: @local_config_platform//:host]
(cd /root/.cache/bazel/_bazel_root/05153e06462a8431ceb96e94c5b047b4/execroot/_main && \
  exec env - \
    LC_CTYPE=en_US.UTF-8 \
    PATH=/src/e2e/cache/bazelisk/downloads/sha256/a40ac69263440761199fcb8da47ad4e3f328cbe79ffbf4ecc14e5ba252857307/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
  external/rules_java~5.5.1~toolchains~remotejdk17_linux/bin/java '--add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED' '--add-exports=jdk.compiler/com.sun.tools.javac.main=ALL-UNNAMED' '--add-exports=jdk.compiler/com.sun.tools.javac.model=ALL-UNNAMED' '--add-exports=jdk.compiler/com.sun.tools.javac.processing=ALL-UNNAMED' '--add-exports=jdk.compiler/com.sun.tools.javac.resources=ALL-UNNAMED' '--add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED' '--add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED' '--add-opens=jdk.compiler/com.sun.tools.javac.code=ALL-UNNAMED' '--add-opens=jdk.compiler/com.sun.tools.javac.comp=ALL-UNNAMED' '--add-opens=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED' '--add-opens=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED' '--add-opens=java.base/java.nio=ALL-UNNAMED' '--add-opens=java.base/java.lang=ALL-UNNAMED' '-Dsun.io.useCanonCaches=false' -XX:-CompactStrings -jar external/rules_java~5.5.1~toolchains~remote_java_tools/java_tools/turbine_direct_binary_deploy.jar --output bazel-out/k8-fastbuild/bin/libHelloLib-hjar.jar --output_deps bazel-out/k8-fastbuild/bin/libHelloLib-hjar.jdeps --bootclasspath bazel-out/k8-fastbuild/bin/external/rules_java~5.5.1/toolchains/platformclasspath.jar --sources HelloLib.java --javacopts -source 8 -target 8 '-XDskipDuplicateBridges=true' '-XDcompilePolicy=simple' -g -parameters -Xep:ReturnValueIgnored:OFF -Xep:IgnoredPureGetter:OFF -Xep:EmptyTopLevelDeclaration:OFF -Xep:LenientFormatStringValidation:OFF -Xep:ReturnMissingNullable:OFF -- --target_label //:HelloLib --reduce_classpath_mode NONE)
# Configuration: eae08a0147418bbed67463ae5f68913a92ab5c68abe9a3a6ea98eee64dbbdd8a
# Execution platform: @local_config_platform//:host
ERROR: /src/e2e/BUILD.bazel:1:13: Compiling Java headers libHelloLib-hjar.jar (1 source file) failed: (Exit 1): java failed: error executing command (from target //:HelloLib) external/rules_java~5.5.1~toolchains~remotejdk17_linux/bin/java '--add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED' '--add-exports=jdk.compiler/com.sun.tools.javac.main=ALL-UNNAMED' ... (remaining 42 arguments skipped)

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
java.lang.ArrayIndexOutOfBoundsException: Index -1 out of bounds for length 1442
	at com.google.turbine.bytecode.ConstantPoolReader.utf8(ConstantPoolReader.java:120)
	at com.google.turbine.bytecode.ClassReader.readMethodParameters(ClassReader.java:229)
	at com.google.turbine.bytecode.ClassReader.readMethods(ClassReader.java:438)
	at com.google.turbine.bytecode.ClassReader.read(ClassReader.java:105)
	at com.google.turbine.bytecode.ClassReader.read(ClassReader.java:55)
	at com.google.turbine.binder.bytecode.BytecodeBoundClass$1.get(BytecodeBoundClass.java:91)
	at com.google.turbine.binder.bytecode.BytecodeBoundClass$1.get(BytecodeBoundClass.java:88)
	at com.google.common.base.Suppliers$NonSerializableMemoizingSupplier.get(Suppliers.java:183)
	at com.google.turbine.binder.bytecode.BytecodeBoundClass$6.get(BytecodeBoundClass.java:194)
	at com.google.turbine.binder.bytecode.BytecodeBoundClass$6.get(BytecodeBoundClass.java:191)
	at com.google.common.base.Suppliers$NonSerializableMemoizingSupplier.get(Suppliers.java:183)
	at com.google.turbine.binder.bytecode.BytecodeBoundClass$7.get(BytecodeBoundClass.java:207)
	at com.google.turbine.binder.bytecode.BytecodeBoundClass$7.get(BytecodeBoundClass.java:204)
	at com.google.common.base.Suppliers$NonSerializableMemoizingSupplier.get(Suppliers.java:183)
	at com.google.turbine.binder.bytecode.BytecodeBoundClass.typeParameters(BytecodeBoundClass.java:221)
	at com.google.turbine.types.Canonicalize.isRaw(Canonicalize.java:152)
	at com.google.turbine.types.Canonicalize.canon(Canonicalize.java:127)
	at com.google.turbine.types.Canonicalize.canonicalizeClassTy(Canonicalize.java:360)
	at com.google.turbine.types.Canonicalize.canonicalize(Canonicalize.java:115)
	at com.google.turbine.types.Canonicalize.canonicalize(Canonicalize.java:77)
	at com.google.turbine.binder.CanonicalTypeBinder.param(CanonicalTypeBinder.java:166)
	at com.google.turbine.binder.CanonicalTypeBinder.parameters(CanonicalTypeBinder.java:153)
	at com.google.turbine.binder.CanonicalTypeBinder.methods(CanonicalTypeBinder.java:127)
	at com.google.turbine.binder.CanonicalTypeBinder.bind(CanonicalTypeBinder.java:66)
	at com.google.turbine.binder.Binder.canonicalizeTypes(Binder.java:319)
	at com.google.turbine.binder.Binder.bind(Binder.java:172)
	at com.google.turbine.binder.Binder.bind(Binder.java:97)
	at com.google.turbine.main.Main.bind(Main.java:259)
	at com.google.turbine.main.Main.compile(Main.java:157)
	at com.google.turbine.main.Main.compile(Main.java:132)
	at com.google.turbine.main.Main.main(Main.java:88)
Target //:HelloBin.jar failed to build
Use --verbose_failures to see the command lines of failed build steps.
ERROR: /src/e2e/BUILD.bazel:6:12 Building HelloBin.jar (1 source file) failed: (Exit 1): java failed: error executing command (from target //:HelloLib) external/rules_java~5.5.1~toolchains~remotejdk17_linux/bin/java '--add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED' '--add-exports=jdk.compiler/com.sun.tools.javac.main=ALL-UNNAMED' ... (remaining 42 arguments skipped)

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
INFO: Elapsed time: 3.159s, Critical Path: 1.82s
INFO: 4 processes: 2 internal, 2 processwrapper-sandbox.
FAILED: Build did NOT complete successfully

Building with --nojava_header_compilation (which if I understand correctly turns off Turbine) works:

root@desktop:/src/e2e# bazel build --nojava_header_compilation //:HelloBin.jar
INFO: Build option --java_header_compilation has changed, discarding analysis cache.
INFO: Analyzed target //:HelloBin.jar (0 packages loaded, 1099 targets configured).
INFO: Found 1 target...
Target //:HelloBin.jar up-to-date:
  bazel-bin/HelloBin.jar
INFO: Elapsed time: 1.233s, Critical Path: 1.04s
INFO: 4 processes: 1 internal, 1 processwrapper-sandbox, 2 worker.
INFO: Build completed successfully, 4 total actions

Any other information?

No response

[jmillikin]

Suspected cause: the container image has Java 21, but Bazel 6.x bundled Java rules are intended for Java 11 and 18.

This means a possible solution is to configure the CI scripts so that when running the build with older versions of Bazel, they set a flag to use a non-host JDK. I verified the "hello world" example builds fine with Bazel 6.x when run with remotejdk_11 and remotejdk_18.

# bazel build //:HelloBin.jar --java_runtime_version=remotejdk_11
[...]
Target //:HelloBin.jar up-to-date:
  bazel-bin/HelloBin.jar
[...]
# bazel build //:HelloBin.jar --java_runtime_version=remotejdk_18
[...]
Target //:HelloBin.jar up-to-date:
  bazel-bin/HelloBin.jar
[...]
root@desktop:/src/e2e# bazel build //:HelloBin.jar --java_runtime_version=remotejdk_21
INFO: Build option --java_runtime_version has changed, discarding analysis cache.
ERROR: /src/e2e/BUILD.bazel:6:12: While resolving toolchains for target //:HelloBin: No matching toolchains found for types @bazel_tools//tools/jdk:runtime_toolchain_type.
To debug, rerun with --toolchain_resolution_debug='@bazel_tools//tools/jdk:runtime_toolchain_type'
If platforms or toolchains are a new concept for you, we'd encourage reading https://bazel.build/concepts/platforms-intro.
ERROR: Analysis of target '//:HelloBin.jar' failed; build aborted: 
[...]
root@desktop:/src/e2e# java -version
openjdk version "21.0.2" 2024-01-16
OpenJDK Runtime Environment (build 21.0.2+13-Ubuntu-120.04.1)
OpenJDK 64-Bit Server VM (build 21.0.2+13-Ubuntu-120.04.1, mixed mode, sharing)