Artifacts for Linux Environment

[illiw99]

Hello everbody!

I am trying to set up a Gitlab Pipeline, where I first build a C++ Library.

This build is used to create python binding. Since the build files are not avalible in my python folder, they are not avalible inside the manylinux docker container.

Is there a way to pass these build C++ folder into the cibuildwheel environment, like artefacts/caches or as a volume? Otherwise I need to get them inside on another way? (Dont really know whats the right way there!)

I hope you can help me :)

[scott-guthridge]

I'm having my own build problems with MacOS right now, but Linux and Windows are working, at least, and I should be able to help you with this. As you're finding, cibuildwheel sets up an isolated environment in which to do the build, and you have to get your C++ library into there.

My github workflow is here: https://github.com/scott-guthridge/pylibvna/blob/main/.github/workflows/release.yml If you're building only for Linux, it's simpler: mainly, you just need to set up the matrix build for the versions of python you want to target. You can start with something like this:

name: Make Release
on: workflow_dispatch

jobs:
  build_wheels:
    name: Build wheel for ${{matrix.python.cp}}-${{matrix.buildplat.sys}}
          ${{matrix.buildplat.arch}}
    runs-on: ${{ matrix.buildplat.runs_on }}
    strategy:
      matrix:
        buildplat:
        - { runs_on: ubuntu-20.04, sys: manylinux, arch: x86_64 }
        python:
        - { cp: "cp39",   rel: "3.9" }
        - { cp: "cp310",  rel: "3.10" }
        - { cp: "cp311",  rel: "3.11" }
        - { cp: "cp312",  rel: "3.12" }
    steps:
    - name: Checkout Code
      uses: actions/[email protected]

    - uses: actions/setup-python@v5
      with:
        python-version: 3.11

    - name: Install Python Packages
      run: |
        python -m pip install --upgrade pip
        pip install cibuildwheel requests

    - name: Build Wheels
      env:
        CIBW_BUILD: ${{ matrix.python.cp }}-${{ matrix.buildplat.sys }}*
        CIBW_ARCHS: ${{ matrix.buildplat.arch }}
      run: python -m cibuildwheel --output-dir wheelhouse

Then in pyproject.toml (https://github.com/scott-guthridge/pylibvna/blob/main/pyproject.toml), start with the usual stuff to describe the build system, dependencies and package details. Then, the cibuildwheel part can look something like this:

[tool.cibuildwheel]
build-frontend = "build"

[tool.cibuildwheel.linux]
before-build = """\
    set -e\n\
    yum install -y gcc libyaml-devel\n\
    pip install requests\n\
    python3 {project}/tools/install-libvna-rpm\n\
"""
manylinux-aarch64-image = "manylinux2014"
manylinux-x86_64-image = "manylinux2014"

The key is the before-build line above. This sets things up within the isolated environment. In my case, my C library is built in its own github workflow and all I do here is download the RPM and install it using the tools/install-libvna-rpm script, but if that library source were part of this repo, I could just as easily have compiled the library in that before-build step instead of installing it from RPM.

I hope this helps.