@hunshcn, @linzhp, should we revert the PR until we can find a fix for this? What do you think? I guess we should add a test in the CI to check that we can cross-compile.

Yeah, I am planning to revert our internal diff that upgraded the Gazelle extension. Sorry for not catching this earlier. I thought it was a "pure golang helper", but it turned out to depend heavily on C

I don't know much about cross-compilation, but I tried hermetic_cc_toolchain, and it seems to work normally (compiling linux_amd64 target on my m1 mac).

hunshcn@a52bd0f

bazel build --enable_bzlmod --announce_rc  //python:gazelle_binary --platforms @zig_sdk//platform:linux_amd64 --extra_toolchains @zig_sdk//toolchain:linux_amd64_musl

(If the target platform is darwin_amd64, we may need to make some changes to gazelle_binary. like https://github.com/uber/hermetic_cc_toolchain/blob/13b27b4e1fa461c57ef12d4e84e0ef34d0e697fb/examples/bzlmod/BUILD.bazel#L23C1-L29C8)

Thanks @hunshcn for pointing out... @zig_sdk//platform:linux_amd64 works but @io_bazel_rules_go//go/toolchain:linux_amd64 doesn't, because the rules_go version disables cgo. We need to use @io_bazel_rules_go//go/toolchain:linux_amd64_cgo instead.

Due to the limitation of hermetic_cc_toolchain, cross-compiling C libraries into any darwin platform won't work. Let me update the issue.

@aignas Updated the description. So the main question here is:

are we better off by replacing a Python dependency with a C dependency?

The deepest motivation for me to raise that PR is that py_binary is not hermetic.
I have to exit the conda environment to run gazelle, otherwise I will receive an error.

Maybe darwin build can be supported by uber/hermetic_cc_toolchain#10 (comment) ?

So I guess there are a few alternatives here:

1. Maintain two codebases where the user can chose if they want the pure C tree-sitter implementation + cgo solution or the Python interpreter solution. I personally think that this is not ideal as it makes us maintain 2 implementations.
2. Explain users how they can cross-compile with cgo, but as @linzhp suggests that is non-trivial either.
3. Revert feat(gazelle): pure golang helper #1895 until we have a better way to do the cross-compilation.

The dependence on the system python is only to launch the interpreter that is bundled with the gazelle plugin, but #1599 needs to be addressed before it can work in venv. The dependence on the system python could be also removed if we solve #691.

I am inclined to go with 3. unless there is a better way to avoid the added dependencies here. @hunshcn, do you know if there are other ways to work around this issue?

Just retell the current situation. Can't cross-compile to darwin-any, other systems are ok. Because hermetic_cc_toolchain does not support sysroot (even on macos machines). This may be a future direction, but no one provides support. I'm interested, but it's beyond my knowledge. It would be best if someone could achieve this.
If you need to compile the binary of the whole platform, you need at least one darwin_amd64 and one darwin_arm64.

I think rules_python decides how much cross-compilation should be supported. If the answer is that we need complete cross-compilation ability, then we may need revert.

1 is also an option, but it can be a user operation (I mean patch by the user himself, this part of the code is relatively simple, and I think it is feasible. If revert, I will do it.)

A related issue from the upstream bindings package is here: smacker/go-tree-sitter#120

This issue is a tough one, the fact that the cross-compilation does not work means that it is not entirely hermetic on all platforms. So I think we should revert the #1895 especially since #1929 is in the works.

As discussed in #1931, the main idea to address this would be to use https://github.com/go-python/gpython to try and use a pure go implementation to fix this issue.

Within the maintainers meeting today we discussed that this issue is affecting a small subset of rules_python users and that subset who needs to cross-compile gazelle most likely have access to required hardware or know how how to build the plugin in the mean time.

So to summarize:

• For cross-compiling gazelle plugin for windows you may need a windows machine or a properly setup CC toolchain.
• For building gazelle for linux, you will need a sysroot and a properly setup toolchain, usually doable with hermetic_cc_toolchain and others.
• Fo building gazelle for mac, you will need a mac machine.

I'll pin this issue as it is a known issue.

PR's are welcome for this.

FYI that the Aspect CLI has a configure command which is a pre-compiled Gazelle. This way end-users aren't responsible for setting up a toolchain to build a local Gazelle binary themselves. see https://docs.aspect.build/cli/commands/aspect_configure

Here's a demo for a python project, setup from scratch:
https://asciinema.org/a/663610