Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

installation: Switch to UV for dependency management #3294

Open
wants to merge 41 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 35 commits
Commits
Show all changes
41 commits
Select commit Hold shift + click to select a range
24707fb
install: Switch to using UV for dependency management
EdmundGoodman Oct 12, 2024
a4f1043
ci: Switch ci-core workflow to using UV
EdmundGoodman Oct 12, 2024
1484229
ci: Debug ci-core workflow
EdmundGoodman Oct 12, 2024
0c38cdb
ci: Fix uv python version matrix
EdmundGoodman Oct 12, 2024
5fccc9d
ci: Re-enable pytest on ci-core workflow
EdmundGoodman Oct 12, 2024
13d29aa
ci: Adopt uv in ci-mlir workflow
EdmundGoodman Oct 12, 2024
0c0dd79
ci: Fix installation paths in ci-mlir workflow
EdmundGoodman Oct 12, 2024
0729097
ci: Re-enable ci-notebooks workflow
EdmundGoodman Oct 12, 2024
f62890c
ci: Add uv run prefix to fix python venvs
EdmundGoodman Oct 12, 2024
484e030
ci: Re-enable ci-pyright-fails workflow
EdmundGoodman Oct 12, 2024
840ed43
ci: Activate virtual environment for uv
EdmundGoodman Oct 12, 2024
e4e4e73
ci: Fix virtual environment path
EdmundGoodman Oct 12, 2024
60b6204
ci: Revert to uv run for stubgen
EdmundGoodman Oct 12, 2024
842a69b
ci: Further fix virtual environment path
EdmundGoodman Oct 12, 2024
07773c4
ci: Re-enable code-formatting workflow
EdmundGoodman Oct 12, 2024
b6cf9dd
ci: Enable running code formatting on non-main branches
EdmundGoodman Oct 12, 2024
e435414
ci: Re-enable jupyterlite workflow
EdmundGoodman Oct 12, 2024
d3be64f
ci: Re-enable remaining workflows
EdmundGoodman Oct 12, 2024
c57bc59
ci: Revert action invocation cases
EdmundGoodman Oct 12, 2024
1d5efe3
ci: Revert python version changes
EdmundGoodman Oct 12, 2024
22a9f4d
misc: Switch to inline make phonies and uv run
EdmundGoodman Oct 13, 2024
4e92d77
misc: Constrain venv creation based on uv implementation
EdmundGoodman Oct 13, 2024
23dfdd3
misc: Add phony target to check if uv is installed
EdmundGoodman Oct 13, 2024
a6694ef
Merge branch 'uv-packaging' of github.com:EdmundGoodman/xdsl into uv-…
EdmundGoodman Oct 13, 2024
bc747ee
misc: Make uv-installed phony fail when not installed
EdmundGoodman Oct 13, 2024
18a50f8
docs: Update readme to reflect uv usage
EdmundGoodman Oct 13, 2024
0218bba
tests: Add uv run to lit.cfg file
EdmundGoodman Oct 15, 2024
507d2df
Revert "docs: Update readme to reflect uv usage"
EdmundGoodman Oct 15, 2024
5cdd709
Revert "Revert "docs: Update readme to reflect uv usage""
EdmundGoodman Oct 15, 2024
14a407e
Revert "tests: Add uv run to lit.cfg file"
EdmundGoodman Oct 15, 2024
111873d
install: Remove extraneous extra imports
EdmundGoodman Oct 16, 2024
ef480c9
Add uv-installed dependency to all targets using uv run
EdmundGoodman Oct 16, 2024
6b9b5a2
ci: Add update-bot to replace Dependabot behaviour for uv
EdmundGoodman Oct 26, 2024
d0e6877
ci: Add assignees to version bump PR and disable dependabot
EdmundGoodman Oct 26, 2024
b803524
install: Add jax and riscv as default extras, as the are required for…
EdmundGoodman Oct 26, 2024
b12a76a
ci: Add superlopuh to update bot pr list
EdmundGoodman Oct 26, 2024
242d587
ci: Remove dependabot config file
EdmundGoodman Oct 26, 2024
e3dd09d
Update flake for uv
alexarice Oct 29, 2024
ea40cb4
Revert "ci: Remove dependabot config file"
EdmundGoodman Oct 31, 2024
37d2919
docs: Update docs to add describe old mechanism of using pip
EdmundGoodman Nov 1, 2024
9b7493a
Fix pip install instructions in README
EdmundGoodman Nov 1, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what is the impact of this change?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we just delete it?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dependabot will no longer run - as it would do nothing anyway or throw an error as the package ecosystem is no longer pip as it expects.

An approximation to the behaviour of Dependabot is then supported by the new custom update bot GitHub Action workflow.

When Dependabot (hopefully) eventually supports uv, it can be reverted and the package ecosystem changed to uv

If not disabling Dependabot is a strong constraint (which is fairly reasonable), then moving to uv is blocked till it offers support for it

Copy link
Collaborator Author

@EdmundGoodman EdmundGoodman Oct 26, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oops, didn't see the follow up comment - this is would make it marginally easier to (remember to?) revert it when support is added, but realistically git revert would make more sense. I'll just delete it

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah I'm pretty sure I'll see on either the twitter or GH when they enable Dependabot for UV, so I don't think it's a big risk.

File renamed without changes.
27 changes: 11 additions & 16 deletions .github/workflows/ci-core.yml
Original file line number Diff line number Diff line change
Expand Up @@ -28,28 +28,23 @@ jobs:
packages: mesa-vulkan-drivers
version: 1.0

- name: Set up Python
uses: actions/setup-python@v5
- name: Install uv
uses: astral-sh/setup-uv@v3
with:
python-version: ${{ matrix.python-version }}
cache: 'pip' # caching pip dependencies
cache-dependency-path: |
setup.py
requirements.txt
enable-cache: true
cache-dependency-glob: "uv.lock"

- name: Upgrade pip
run: |
pip install --upgrade pip
- name: Set up Python ${{ matrix.python-version }}
run: uv python install ${{ matrix.python-version }}

- name: Install the package locally
run: pip install -r requirements.txt
- name: Install the project
run: uv sync --all-extras

- name: Test with pytest
run: |
pytest -W error
run: uv run pytest -W error

- name: Execute lit tests
run: |
export PYTHONPATH=$(pwd)
lit -v tests/filecheck/
lit -v docs/Toy/examples/
uv run lit -v tests/filecheck/
uv run lit -v docs/Toy/examples/
35 changes: 17 additions & 18 deletions .github/workflows/ci-mlir.yml
Original file line number Diff line number Diff line change
Expand Up @@ -33,24 +33,23 @@ jobs:
packages: mesa-vulkan-drivers
version: 1.0

- name: Python Setup
uses: actions/setup-python@v5
- name: Install uv
uses: astral-sh/setup-uv@v3
with:
python-version: ${{ matrix.python-version }}
cache: 'pip' # caching pip dependencies
cache-dependency-path: |
setup.py
requirements.txt
enable-cache: true
cache-dependency-glob: "xdsl/uv.lock"

- name: Upgrade pip
- name: Set up Python ${{ matrix.python-version }}
run: |
pip install --upgrade pip
# Change directory so that xdsl-opt can be found during installation.
cd xdsl
uv python install ${{ matrix.python-version }}

- name: Install the package locally and nbval
run: |
# Change directory so that xdsl-opt can be found during installation.
cd xdsl
pip install -r requirements.txt
uv sync --all-extras

- name: Cache binaries
id: cache-binary
Expand Down Expand Up @@ -99,36 +98,36 @@ jobs:
- name: Test with pytest and generate code coverage
run: |
cd xdsl
pytest -W error --cov --cov-config=.coveragerc .
uv run pytest -W error --cov --cov-config=.coveragerc .

- name: Execute lit tests
run: |
cd xdsl
# Add mlir-opt to the path
export PATH=$PATH:${GITHUB_WORKSPACE}/llvm-project/build/bin/
lit -v tests/filecheck/ -DCOVERAGE
lit -v docs/Toy/examples/ -DCOVERAGE
uv run lit -v tests/filecheck/ -DCOVERAGE
uv run lit -v docs/Toy/examples/ -DCOVERAGE

- name: Test MLIR dependent examples/tutorials
run: |
cd xdsl
# Add mlir-opt to the path
export PATH=$PATH:${GITHUB_WORKSPACE}/llvm-project/build/bin/
pytest --nbval docs/mlir_interoperation.ipynb --maxfail 1 -vv
uv run pytest --nbval docs/mlir_interoperation.ipynb --maxfail 1 -vv

- name: Test MLIR dependent marimo notebooks
run: |
cd xdsl
# Add mlir-opt to the path
export PATH=$PATH:${GITHUB_WORKSPACE}/llvm-project/build/bin/
make tests-marimo-mlir
uv run make tests-marimo-mlir

- name: Combine coverage data
run: |
cd xdsl
coverage combine --append
coverage report
coverage xml
uv run coverage combine --append
uv run coverage report
uv run coverage xml

- name: Upload coverage to Codecov
if: matrix.python-version == '3.10'
Expand Down
29 changes: 15 additions & 14 deletions .github/workflows/ci-notebooks.yml
Original file line number Diff line number Diff line change
Expand Up @@ -21,23 +21,24 @@ jobs:

steps:
- uses: actions/checkout@v4
- name: Set up Python
uses: actions/setup-python@v5

- name: Install uv
uses: astral-sh/setup-uv@v3
with:
python-version: ${{ matrix.python-version }}
cache: 'pip' # caching pip dependencies
cache-dependency-path: |
setup.py
requirements.txt
- name: Upgrade pip
run: |
pip install --upgrade pip
- name: Install the package locally
run: pip install -r requirements.txt
enable-cache: true
cache-dependency-glob: "uv.lock"

- name: Set up Python ${{ matrix.python-version }}
run: uv python install ${{ matrix.python-version }}

- name: Install the project
run: uv sync --all-extras

- name: Run examples/tutorials
run: |
# mlir_interoperation.ipynb is dependent on MLIR, and is tested in the MLIR-enabled workflow.
pytest -W error --nbval -vv docs --ignore=docs/mlir_interoperation.ipynb
uv run pytest -W error --nbval -vv docs --ignore=docs/mlir_interoperation.ipynb

- name: Test marimo notebooks
run: |
make tests-marimo
uv run make tests-marimo
26 changes: 16 additions & 10 deletions .github/workflows/ci-pyright-fails.yml
Original file line number Diff line number Diff line change
Expand Up @@ -23,19 +23,25 @@ jobs:

steps:
- uses: actions/checkout@v4
- name: Set up Python
uses: actions/setup-python@v5

- name: Install uv
uses: astral-sh/setup-uv@v3
with:
python-version: ${{ matrix.python-version }}
cache: 'pip' # caching pip dependencies
cache-dependency-path: |
setup.py
requirements.txt
- name: Install the package locally
run:
pip install -r requirements.txt
enable-cache: true
cache-dependency-glob: "uv.lock"

- name: Set up Python ${{ matrix.python-version }}
run: uv python install ${{ matrix.python-version }}

- name: Install the project and activate its virtual environment
run: |
uv sync --all-extras
source .venv/bin/activate
echo "$PWD/.venv/bin" >> $GITHUB_PATH

- name: Generate IRDL stubs
run: xdsl-stubgen

- name: Pyright
uses: jakebailey/pyright-action@v2
with:
Expand Down
18 changes: 7 additions & 11 deletions .github/workflows/code-formatting.yml
Original file line number Diff line number Diff line change
Expand Up @@ -21,18 +21,14 @@ jobs:
steps:
- uses: actions/checkout@v4

- name: Set up Python
uses: actions/setup-python@v5
- name: Install uv
uses: astral-sh/setup-uv@v3
with:
python-version: ${{ matrix.python-version }}
cache: 'pip' # caching pip dependencies
cache-dependency-path: |
setup.py
requirements.txt

- name: Upgrade pip
run: |
pip install --upgrade pip
enable-cache: true
cache-dependency-glob: "uv.lock"

- name: Set up Python ${{ matrix.python-version }}
run: uv python install ${{ matrix.python-version }}

- name: Run code formatting checks with pre-commit
uses: pre-commit/[email protected]
18 changes: 7 additions & 11 deletions .github/workflows/jupyterlite.yml
Original file line number Diff line number Diff line change
@@ -1,8 +1,7 @@
name: Deploy JupyterLite Page

on:
# Trigger the workflow on push or pull request,
# but only for the master branch
# Trigger the workflow every day at 4:15am
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

schedule:
- cron: '15 4 * * *'

Expand All @@ -15,18 +14,15 @@ jobs:
with:
path: xdsl

- name: Setup Python
uses: actions/setup-python@v5
with:
python-version: "3.11"
cache: 'pip' # caching pip dependencies
cache-dependency-path: |
setup.py
requirements.txt
- name: Install uv
uses: astral-sh/setup-uv@v3

- name: Set up Python
run: uv python install 3.11

- name: Install dependencies
run: |
python -m pip install jupyterlite-core==0.2.3 jupyterlite-pyodide-kernel==0.2.1 libarchive-c build pyodide-build==0.24.1 jupyter-server
uv tool install jupyterlite-core==0.2.3 jupyterlite-pyodide-kernel==0.2.1 libarchive-c build pyodide-build==0.24.1 jupyter-server

- name: Build xDSL source distribution
run: |
Expand Down
17 changes: 8 additions & 9 deletions .github/workflows/pythonpublish.yml
Original file line number Diff line number Diff line change
Expand Up @@ -9,18 +9,17 @@ jobs:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4

- name: Install uv
uses: astral-sh/setup-uv@v3

- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: '3.x'
cache: 'pip' # caching pip dependencies
cache-dependency-path: |
setup.py
requirements.txt
run: uv python install 3.11

- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install setuptools wheel twine
uv tool install setuptools wheel twine

- name: Build and publish
run: |
python setup.py sdist bdist_wheel
Expand Down
38 changes: 38 additions & 0 deletions .github/workflows/update-bot.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
name: Update Bot

on:
workflow_dispatch:
# Set the schedule, for example every week at 8:00am on Monday
schedule:
- cron: 0 8 * * 1

permissions:
contents: write
pull-requests: write

jobs:
lock:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4

- uses: astral-sh/setup-uv@v3

- run: |
echo "\`\`\`" > uv_output.md
uv lock &>> uv_output.md
echo "\`\`\`" >> uv_output.md

- name: Create pull request
uses: peter-evans/create-pull-request@v7
with:
token: ${{ secrets.GITHUB_TOKEN }}
commit-message: Update uv lockfile
title: Update uv lockfile
body-path: uv_output.md
branch: update-uv
base: main
labels: install
delete-branch: true
add-paths: uv.lock
assignees: math-fehr, georgebisbas
EdmundGoodman marked this conversation as resolved.
Show resolved Hide resolved
Loading
Loading