Explore uv as an alternative to pip, pip-compile and poetry #16
Comments
legoktm
added a commit
to freedomofpress/securedrop
that referenced
this issue
Sep 23, 2024
The `make update-python3-dependencies` step is slow for two main reasons: container image layering and pip-compile. Currently, each `pip-compile` step is run in a separate dev-shell, which means after each one, if anything changed in the requirements/ folder, it needs to rebuild the slim image and reinstall all the pip dependencies. Now I've moved the steps to a separate script that execute in a single dev-shell, so no image rebuilding happens during the updates, it'll just be needed once afterwards. For an additional boost, switch to the new uv tool, which reimplements pip-compile in a much faster way. The output is basically the same, except the sorting is smarter (e.g. pytest comes before pytest-cov) and package names are properly normalized. We can also drop the `--allow-unsafe` because uv is entirely independent of setuptools and pip-tools. uv is still quite new to the Python ecosystem, but this allows us to begin using it without any lock-in, it should be trivial to swap back to pip-tools if needed. Overall `make update-python3-dependencies` now takes seconds to run instead of minutes \o/ Refs <freedomofpress/securedrop-tooling#16>.
6 tasks
|
freedomofpress/securedrop#7234 swaps out pip-tools for uv and it's basically a drop-in replacement with no noticable downside yet. As I mentioned there, it's pretty conservative and still uses pip for the actual installation, but even Reading through the docs, I note that dependabot support (dependabot/dependabot-core#10039) will probably be a barrier for further adoption. |
legoktm
added a commit
to freedomofpress/securedrop
that referenced
this issue
Sep 23, 2024
The `make update-python3-dependencies` step is slow for two main reasons: container image layering and pip-compile. Currently, each `pip-compile` step is run in a separate dev-shell, which means after each one, if anything changed in the requirements/ folder, it needs to rebuild the slim image and reinstall all the pip dependencies. Now I've moved the steps to a separate script that execute in a single dev-shell, so no image rebuilding happens during the updates, it'll just be needed once afterwards. For an additional boost, switch to the new uv tool, which reimplements pip-compile in a much faster way. The output is basically the same, except the sorting is smarter (e.g. pytest comes before pytest-cov) and package names are properly normalized. We can also drop the `--allow-unsafe` because uv is entirely independent of setuptools and pip-tools. uv is still quite new to the Python ecosystem, but this allows us to begin using it without any lock-in, it should be trivial to swap back to pip-tools if needed. Overall `make update-python3-dependencies` now takes seconds to run instead of minutes \o/ Refs <freedomofpress/securedrop-tooling#16>.
legoktm
added a commit
to freedomofpress/securedrop
that referenced
this issue
Oct 7, 2024
The `make update-python3-dependencies` step is slow for two main reasons: container image layering and pip-compile. Currently, each `pip-compile` step is run in a separate dev-shell, which means after each one, if anything changed in the requirements/ folder, it needs to rebuild the slim image and reinstall all the pip dependencies. Now I've moved the steps to a separate script that execute in a single dev-shell, so no image rebuilding happens during the updates, it'll just be needed once afterwards. For an additional boost, switch to the new uv tool, which reimplements pip-compile in a much faster way. The output is basically the same, except the sorting is smarter (e.g. pytest comes before pytest-cov) and package names are properly normalized. We can also drop the `--allow-unsafe` because uv is entirely independent of setuptools and pip-tools. uv is still quite new to the Python ecosystem, but this allows us to begin using it without any lock-in, it should be trivial to swap back to pip-tools if needed. Overall `make update-python3-dependencies` now takes seconds to run instead of minutes \o/ Refs <freedomofpress/securedrop-tooling#16>.
legoktm
added a commit
to freedomofpress/securedrop
that referenced
this issue
Oct 9, 2024
The `make update-python3-dependencies` step is slow for two main reasons: container image layering and pip-compile. Switch to the new uv tool, which reimplements pip-compile in a much faster way. The output is basically the same, except the sorting is smarter (e.g. pytest comes before pytest-cov) and package names are properly normalized. We can also drop the `--allow-unsafe` because uv is entirely independent of setuptools and pip-tools. Because uv is independent of the system Python (it's pure Rust), we can run it on the host and outside the container, so no image rebuilding happens during the updates, it'll just be needed once afterwards. uv is still quite new to the Python ecosystem, but this allows us to begin using it without any lock-in, it should be trivial to swap back to pip-tools if needed. Overall `make update-python3-dependencies` now takes seconds to run instead of minutes \o/ Refs <freedomofpress/securedrop-tooling#16>.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
https://docs.astral.sh/uv/
From the same creators as ruff, uv can be a much faster replacement for pip and pip-compile, but also has poetry-like dependency management features. There is concern that it is a VC-funded tool for something that is so critical to the ecosystem, but it's all based on PEPs and also should be forkable.
The text was updated successfully, but these errors were encountered: