installation: Switch to UV for dependency management

[EdmundGoodman]

Summary

Adopt uv for dependency management, replacing the existing pip approach.

Motivation

uv is developed by Astral, who also wrote ruff which is already in use in xDSL CI pipelines. It offers the benefits of much faster dependency resolution and installation, along with leveraging the more modern pyproject.toml configuration format defined in PEPs 518 and 618.

Requested by @superlopuh

Changes

This replacement requires the following changes:

• Modification to the venv target of the Makefile to create the virtual environment
• Update CI flows to install with uv as opposed to pip, including testing
  • ci-core.yaml
  • ci-mlir.yaml
  • ci-notebooks.yaml
  • ci-pyright-fails.yaml
  • code-formatting.yaml
  • jupyterlite.yaml (cannot easily be spawned due to workflow triggers)
  • pythonpublish.yaml (cannot easily be spawned due to required secrets)
  • release-notes.yaml (requires no changes)
• Update documentation to reflect these changes
• Test across operating systems
  • Windows
  • MacOS
  • NixOS
• Resolve changes required to Dependabot

Post-merge checklist

Once merged, we should check the following things:

• The release flow worked correctly (including versioneer)
• Users are able to switch to uv and instructions are clear
• The update-bot workflow creates helpful and correct PRs in a timely manner

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 90.11%. Comparing base (f2ba5f7) to head (9b7493a).
Report is 63 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3294      +/-   ##
==========================================
+ Coverage   90.01%   90.11%   +0.10%     
==========================================
  Files         445      450       +5     
  Lines       55850    56792     +942     
  Branches     5372     5478     +106     
==========================================
+ Hits        50272    51177     +905     
- Misses       4169     4172       +3     
- Partials     1409     1443      +34     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[superlopuh]

Could you please also migrate the makefile? Maybe with a phony target that checks whether uv is installed, and directs developers to installation instructions. The goal would be for make tests to be runnable outside of an activated venv

[math-fehr]

Thanks a lot for the PR!
A few quick questions/requests I have:

• Can we move the removed requirements.txt in the pyproject.toml? That way we can still track them with dependabot, and also this is the only change that's missing to still support virtualenv. I'd personally would like to still continue supporting venv if possible, as for now it is still way more popular than uv.
• Do you think we can automate the update of uv.lock with dependabot? Currently, we get daily automated updates of dependencies with dependabot, and it would quite annoying to have to update the uv.lock manually every time.
• We will probably have to test pythonpublish after the PR gets merged. Just ping me and I'll try to run it to see if it works or not, but looking at it this should be fine I think.

[math-fehr]

Are these dependencies moved to uv.lock?
I feel that this should rather go to the .toml file so that we can track them with dependabot, it was also bad before when they were here, but at least we had a list of them.

[superlopuh]

No, these are in the pyproject.toml, and are installed via --all-extras

[alexarice]

We've been using uv for our quantum stack recently and it seems to work nicely. I have a couple of comments:

• When to use uv run. It seems in the test ci you are activating the venv anyway so I don't see the reason to us uv run everywhere.
• If you did want to use uv run everywhere, you would need to make the filecheck tests use it (i.e. every xdsl-opt should be uv run xdsl-opt). You can get most of these by modifying lit.cfg. I'm not sure if this is actually worth it?
• I've always had problems with the wgpu package and had to manually remove it. Will there be an easy way to do that with the new set up?
• uv should be added to flake.nix (I can do this afterwards if necessary)

[EdmundGoodman]

Ok - I think I see the problem. All the make targets using uv need to have the uv-installed dependency, not just make venv.

I assumed that people would be coming in without having cloned it already, make their venv (and install uv then), then do the other targets. In your case, you already have a venv so went straight to the other targets and it failed.

I will update the PR tonight when I'm back from a commitment to hopefully fix this.

[EdmundGoodman]

Now the Makefile stuff is hopefully fixed, I've also hacked together something which might be able to be a stopgap for Dependabot until it officially supports uv.

I've made a demo repo here: https://github.com/EdmundGoodman/update-bot, but the idea is a cron job which runs every week and creates a PR to bump the uv lockfile. This slightly differs from Dependabot in it is scheduled and does all version bumps not just security ones, but might give enough functionality for now?

If it seems reasonable, the change will just be adding that workflow YAML file to this PR

[superlopuh]

This is beautiful. If I understand you correctly, the idea would be to loosen the dependabot requirements, and update the dev versions in the lockfile once a week? My understanding is that we are already too fine-grained in our dependency constraints.

[EdmundGoodman]

This is beautiful.

😃 ! I have added the new workflow to the PR now

If I understand you correctly, the idea would be to loosen the dependabot requirements, and update the dev versions in the lockfile once a week?

I think Dependabot doesn't support uv at all currently, so we would turn it off (now in the PR), then just bump any dependencies with new versions weekly by merging the automatically generated PR if it passes all the tests.

My understanding is that we are already too fine-grained in our dependency constraints.

I guess some things like filecheck==1.0.1 would need to be modified to be filecheck>=1.0.1 as otherwise uv alone wouldn't bump them even if they are known to require a security patch, yes

[superlopuh]

what is the impact of this change?

[superlopuh]

Should we just delete it?

[EdmundGoodman]

Dependabot will no longer run - as it would do nothing anyway or throw an error as the package ecosystem is no longer pip as it expects.

An approximation to the behaviour of Dependabot is then supported by the new custom update bot GitHub Action workflow.

When Dependabot (hopefully) eventually supports uv, it can be reverted and the package ecosystem changed to uv

If not disabling Dependabot is a strong constraint (which is fairly reasonable), then moving to uv is blocked till it offers support for it

[EdmundGoodman]

Oops, didn't see the follow up comment - this is would make it marginally easier to (remember to?) revert it when support is added, but realistically git revert would make more sense. I'll just delete it

[superlopuh]

Yeah I'm pretty sure I'll see on either the twitter or GH when they enable Dependabot for UV, so I don't think it's a big risk.

[superlopuh]

What's left on the TODO list to open it up for review?

[EdmundGoodman]

What's left on the TODO list to open it up for review?

I think mostly just the stuff in the top comment:

• Testing against non-MacOS operating systems (I will need help with this)
• Checking the release flow/hard-to-run pipelines won't explode because of the changes (I will need help with this)
• Checking the update-bot workflow crontab schedules correctly in this instance (I am doing this now -- not listed in the top comment) all looks ok

It may also be worth ensuring there is buy-in from maintainers and key users before what is a fairly impactful if not large change.

[alexarice]

Hi, I added a commit to update the nix flake, hope that is ok. make tests works for me locally

[superlopuh]

This LGTM, I guess Mathieu will test the release situation once this is merged

[EdmundGoodman]

Hi, I added a commit to update the nix flake, hope that is ok. make tests works for me locally

Amazing, thanks!

[EdmundGoodman]

This LGTM, I guess Mathieu will test the release situation once this is merged

Sounds reasonable - if anything explodes for users when it gets merged feel free to ping me here/on Zulip and I can try to help them debug things (but hopefully everything will go smoothly 😄)

[superlopuh]

@math-fehr

[math-fehr]

Almost good for me, I just want to be sure we are not regressing too much with dependabot, and keep the documentaiton for doing a pip install.

I'll check the release once it's merged, hopefully it won't break!

[math-fehr]

Could we actually keep dependabot, though maybe making the schedule to "weekly" instead?

I feel that update-bot is not as strong as dependabot, as it will not update the actual version, it will just update the minor ones (and miss security updates).
Having both at the same time is probably what we want here.

[EdmundGoodman]

tl;dr, I've just now caught up with you in realising it might be possible to use Dependabot in this case despite the docs saying its not supported. I'll try to make changes to go back to just using Dependabot and see if something breaks

Long chain of thought to justify this to myself when I come back to it tomorrow:

I've just had a further stare at this and am wondering if Dependabot might just work for uv despite not being explicitly supported?

On modern python projects the requirements are often loosely defined in the pyproject.toml file (e.g. package = ^1.2.1), then the specific versions set in a lock file such as Pipenv.lock/poetry.lock/uv.lock - then Dependabot acts on this lock file. Since uv isn't a supported lock file type, Dependabot can't be used like this.

However, I've just now realised that since this project is pinning dependency versions in pyproject.toml (as pip doesn't have a lockfile), Dependabot acts on the pyproject.toml directly -- which I hadn't seen before so didn't think to check. As such if we either: don't commit the uv lock file; or use Dependabot in conjunction with update-bot to bump the lock file, it should all work correctly

In future, it might be worth considering using a lock file instead of pinning dependencies in pyproject.toml -- but for now I think it saves us from this problem so we can just leave it.