Broken dependency evaluation in pip 24.3.1. Pip will not recognize installed prerelease versions

[maarre]

Description

I want to be able to test multiple prerelease versions together without having to shoehorn installations.

I really don't want to put prerelease versions in my pyproject.toml.

Current version of pip will only allow prerelease dependencies using --no-deps flag.

$ python --version
Python 3.11.2
$ python -m pip --version
pip 24.3.1 from xxxxx\Python311-64\Lib\site-packages\pip (python 3.11)
$ pip list | grep spdbtools
spdbtools                 0.7.1a20241118121953
$ python -m pip install dist/postgresqlpatches-0.4.0a20241118163330-py3-none-any.whl
Looking in indexes: https://python.repo.sfa.se/repository/all/simple
Processing c:\fkapps\repo\git\ios\python-postgresql-patches\dist\postgresqlpatches-0.4.0a20241118163330-py3-none-any.whl
Requirement already satisfied: psycopg2<3,>=2.9.5 in xxx\python311-64\lib\site-packages (from postgresqlpatches==0.4.0a20241118163330) (2.9.9)
INFO: pip is looking at multiple versions of postgresqlpatches to determine which version is compatible with other requirements. This could take a while.
ERROR: Could not find a version that satisfies the requirement spdbtools<1.0,>=0.7.1 (from postgresqlpatches) (from versions: 0.4.1, 0.5.0, 0.5.1, 0.5.2, 0.6.2, 0.6.3, 0.6.4)
ERROR: No matching distribution found for spdbtools<1.0,>=0.7.1
$ python -m pip install dist/postgresqlpatches-0.4.0a20241118163330-py3-none-any.whl --pre
Looking in indexes: https://python.repo.sfa.se/repository/all/simple
Processing c:\fkapps\repo\git\ios\python-postgresql-patches\dist\postgresqlpatches-0.4.0a20241118163330-py3-none-any.whl
Requirement already satisfied: psycopg2<3,>=2.9.5 in c:\fkapps\python311-64\lib\site-packages (from postgresqlpatches==0.4.0a20241118163330) (2.9.9)
INFO: pip is looking at multiple versions of postgresqlpatches to determine which version is compatible with other requirements. This could take a while.
ERROR: Could not find a version that satisfies the requirement spdbtools<1.0,>=0.7.1 (from postgresqlpatches) (from versions: 0.4.1, 0.5.0, 0.5.1, 0.5.2, 0.6.2, 0.6.3, 0.6.4)
ERROR: No matching distribution found for spdbtools<1.0,>=0.7.1
$ python -m pip install dist/postgresqlpatches-0.4.0a20241118163330-py3-none-any.whl --no-deps
Looking in indexes: https://python.repo.sfa.se/repository/all/simple
Processing xxxx\git\ios\python-postgresql-patches\dist\postgresqlpatches-0.4.0a20241118163330-py3-none-any.whl
Installing collected packages: postgresqlpatches
Successfully installed postgresqlpatches-0.4.0a20241118163330

Expected behavior

If there is an alpha, beta or release candidate installed i want pip to accept this installation when doing the dependency evaluation. The installations should succeed without having to use --no-deps switch.

pip version

24.3.1

Python version

3.11.2

OS

Windows, Linux

How to Reproduce

1 Create a package with an alpha version.´
2 Build the first package
3 Install the first package
4 Create another package with a dependency to the first package
5 The pyproject.toml file should reference the version of the first package without the alpha specifier.
6 Build the second package
7 Install the second package

Output

No response

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[pfmoore]

Doesn’t the --pre option do exactly this?

[maarre]

$ python -m pip install dist/postgresqlpatches-0.4.0a20241118163330-py3-none-any.whl --pre
Looking in indexes: https://python.repo.sfa.se/repository/all/simple
Processing c:\fkapps\repo\git\ios\python-postgresql-patches\dist\postgresqlpatches-0.4.0a20241118163330-py3-none-any.whl
Requirement already satisfied: psycopg2<3,>=2.9.5 in c:\fkapps\python311-64\lib\site-packages (from postgresqlpatches==0.4.0a20241118163330) (2.9.9)
INFO: pip is looking at multiple versions of postgresqlpatches to determine which version is compatible with other requirements. This could take a while.
ERROR: Could not find a version that satisfies the requirement spdbtools<1.0,>=0.7.1 (from postgresqlpatches) (from versions: 0.4.1, 0.5.0, 0.5.1, 0.5.2, 0.6.2, 0.6.3, 0.6.4)
ERROR: No matching distribution found for spdbtools<1.0,>=0.7.1

[pfmoore]

Hang on - your requirement is spdbtools<1.0,>=0.7.1, but you have 0.7.1a20241118121953. According to the version specification:

Within a numeric release (1.0, 2.7.3), the following suffixes are permitted and MUST be ordered as shown:
.devN, aN, bN, rcN, <no suffix>, .postN

So your alpha release does not satisfy the constraint >=0.7.1.

Pip is behaving according to spec here.

[notatallshaw]

Yeah, it's a part of the spec that I see a lot of people confused by, but 1.0a1 is not satisfied by >=1.0 even with pre release available.

You must specify >0.7.0 and enable pre releases (--pre).

[potiuk]

Pip is behaving according to spec here.

Yes. I looked very closely at those in the past. It's according to the specification. And specification is good. But still it does not make it easy for the workflow described by @maarre.

You must specify >0.7.0 and enable pre releases (--pre).

Yes, but this is a bit problematic becasue pip does not support per-package --pre release - you can only enable --pre for all resolution, not for individual packages.

By adding --pre you allow pre-releases of ALL packages - which means that you will install pre-releases that you might absolutely not want - if you have many packages as dependencies and you have it specified as >= and your resolution normally returns latest available version of the packages, you are almost guaranteed to break something by using --pre - because someone is testing something and pushed a broken pre-release to pypI. Been there, done that (on both sides).

And it does make it very tricky when you have a workflow of releasing two packages at the same time and want to make rc candidates of package A that might become final candidate and you want it to depend on the new version of the package B - that you also want to release as an rc candidate at the same time.

Say: you releae A==0.1.rc1 that should (when final) depend on B>=0.2 - but you release the B==0.2.rc1 at the same time.

We have this very problem very often in Airflow - when it often happens that at the same we want to relese a new common package with new feature (say airflow-providers-common-sql) and another package that depends on the new features added in airflow-providers-common-sql (say airflow-providers-postgres==1.0.0rc1 that depends on features added in airflow-providers-common-sql==2.1.0rc1). The final dependency will have to be apache-alrflow-providers-postgres==1.0.0 should depend on apache-airlfow-providers-common.sql>=2.1.0. But if our postgres 1.0.0rc1 provider has common-sql>=2.1.0 - that will not work.

And this is problematic because you effectively have to modify your dependencies in either git repo or relased packages between the RC and final version. That's not good workflow - for security and relase process especially - modifying code between rc and final version is problematic especially if you already started to develop new version and you release from main (it basically forces you to create branches or dynamic dependency modification while you are building the package).

Our solution in airflow (really a workaround) that we found working for us is to dynamically modify the rc packages with rc dependencies for all our rc packages. In the example above when we generate apache-airflow-providers-postgres==1.0.0rc1 we check all the required dependencies it has and if we find any apache-airflow-providers-*>=X.Y.Z - we dynamically change them to apache-airlfow-provider-....>=X.Y.Zrc1 (so in this case apache-airflow-providers-common-sql>=2.1.0 will be dynamically modified to apache-airflow-providers-common-sql>=2.1.0rc1. Since 2.1.0rc1 is the only one that satisfies the requirement, it will be used even without --pre flag. This is a special case where using --pre is not even needed to install pre-release version.

THis way in our pyproject.toml/hatch_build.py we keep the final dependency (apache-airflow-providers-common-sql>=2.1.0) and we do not have to worry about modifying it between rc and final version.

It's a workable solution and works good for us in Airflow - so maybe you can adapt it as well @maarre - however it requires dynamic generation of requirements and quite some automation of your release process.

Another, better solution that maybe we might see in the future is to be able to specify selectively which packages should be treated differently, say if we could specify which packages could be treated differently with pre-releases we could use somethign like --allow-pre-releases-of "apache-airflow-providers-common-sql,apache-airflow-providers-common-io" - and for those packages automatically treat >=0.7.2 as (>=0.7.2.dev0).

Another option is what uv does - where you can selectively override specific dependencies when you install packages https://docs.astral.sh/uv/concepts/resolution/#dependency-overrides - that would effectively serve even more use cases (though I think this one gives far too much freedom to the users, and it's a little bit of footgun for uv when implemented this way). And I would not recommend it.

[notatallshaw]

By adding --pre you allow pre-releases of ALL packages - which means that you will install pre-releases that you might absolutely not want

There's a workaround to get per package prerelease, create a constraints file prereleases.txt:

package>=0.0dev0

Then you can add -c prereleases.txt, the prerelease specifier now allows prereleases for just that package.

[potiuk]

Then you can add -c prereleases.txt, the prerelease specifier now allows prereleases for just that package.

This is only half of a solution. You'd still need to modify dynamically your RC package requirement. In the example above, I want postgres 1.0.0rc1 depend on common-sql>=2.1.0. Adding constraints file common-sql>=2.1.0.rc1 will not change anything because common-sql>=2.1.0 will exclude common-sqll 2.1.0.rc1 (because rc1 is lower than 2.1.0).

So the only solution I see now is to dynamically modify rc1 packages to modify their dependencies >=X.Y.Z.rc1 - all while RC packages are being built.

Again - I want to avoid to have to bump >=X.Y.Z.rc1 to >=X.YZ in the repository between RC and final version, so if I "naively" build RC package from the same sources as final it will always have >=X.YZ.

The flag that could solve it will have to dp two things:

• allow pre-release flags
• allow to install X.Y.Z.rc1/.dev0 etc. packages even if the requirement is >=X.Y.Z. Basically it means violating the ordering of pre-releases for X.YZ if >=X.Y.Z is specified for that package as requirement.

[notatallshaw]

As I read this it appears to me what you're asking for is a "light" override of the requirements. It's an interesting idea, but I don't see if getting any kind of traction with pip maintainers unless it became a standard. As maintainers have strongly expressed not wanting to help user install "broken" requirements.

There has been discussion of override interfaces before, e.g. #8076 (comment) where I proposed an exactly equivalent interface to uv's --override, back in 2022.

[potiuk]

As I read this it appears to me what you're asking for is a "light" override of the requirements. It's an interesting idea, but I don't see if getting any kind of traction with pip maintainers unless it became a standard. As maintainers have strongly expressed not wanting to help user install "broken" requirements.

Yes. but with a twist. I do not want (and I agree with maintainers here) to allow for broken dependencies. That would be very bad.

What I would see as a possible solution is to handle specific case where you want to treat pre-release candidates version comparision differently for the packages that have not yet been released (only pre-released). Simply recognising the fact that there is a use case where >=1.2 should also include 1.2.rc when there is no 1.2 released yet (maybe controlled by a flag).

[notatallshaw]

Simply recognising the fact that there is a use case where >=1.2 should also include 1.2.rc when there is no 1.2 released yet (maybe controlled by a flag).

That's a spec change, so the discussion must happen on the packaging forum, as it happens there is a discussion that is going on right now that has devolved into including that very question: https://discuss.python.org/t/proposal-intersect-and-disjoint-operations-for-python-version-specifiers/71888/20

I'm hoping though that discussion gets moved to it's own dedicated thread, I'll post back here if it does.

[pfmoore]

I'm hoping though that discussion gets moved to it's own dedicated thread

It will, and I intend to propose a clarified version of the spec. However, I will say right now that changing the fact that 1.2rc1 is before 1.2 (and hence doesn't satisfy >=1.2) is firmly off the table. There's no lack of clarity in the current spec over this behaviour, and I intend to strictly separate "clarifications" from "rules changes". Someone else would have to pick that up, after the current discussion has run its course.

[maarre]

Hang on - your requirement is spdbtools<1.0,>=0.7.1, but you have 0.7.1a20241118121953. According to the version specification:

Within a numeric release (1.0, 2.7.3), the following suffixes are permitted and MUST be ordered as shown:
.devN, aN, bN, rcN, <no suffix>, .postN

So your alpha release does not satisfy the constraint >=0.7.1.

Pip is behaving according to spec here.

You are wrong.

The ordering just defines which one to choose. No suffix is more worth than any prerelease.

[maarre]

https://packaging.python.org/en/latest/specifications/version-specifiers/#handling-of-pre-releases

Pre-releases of any kind, including developmental releases, are implicitly excluded from all version specifiers, unless they are already present on the system, explicitly requested by the user, or if the only available version that satisfies the version specifier is a pre-release.

[maarre]

@potiuk you almost nailed it. Very well described. See my previous post.

[pfmoore]

You are wrong.

I am not, sorry. Pip follows the standards, and this is what the standard says tools must do. The section of the standard you quoted is not relevant here, as that only covers situations where the ordering rules mean that a pre-release would match the specifier. It's the ordering rules which state that alpha releases come before final releases, and they leave no room for ambiguity on that matter.

The ordering just defines which one to choose. No suffix is more worth than any prerelease.

You may be correct in saying this, but if so, then the spec is wrong, not pip. I'd support someone asking for a change to the spec, and if such a change was approved, pip would be changed to behave as required by the new spec, but the point is that the spec must be changed first.

@potiuk you almost nailed it. Very well described.

Yes, but the solution proposed by @potiuk would require pip to support a flag that explicitly caused us to violate the spec. We won't do that.

To be absolutely clear - if you want version 0.7.1a20241118121953 to satisfy a version specifier of <1.0,>=0.7.1, you will have to submit a PEP suggesting a change to the packaging version specifiers standard, which makes it valid for tools to behave like that. There are reasons why such a change could be reasonable, so this is not a way of making it impossible for you to get what you want - I'm simply telling you the correct process if you want to take this further.

[potiuk]

To be absolutely clear - if you want version 0.7.1a20241118121953 to satisfy a version specifier of <1.0,>=0.7.1, you will have to submit a PEP suggesting a change to the packaging version specifiers standard, which makes it valid for tools to behave like that. There are reasons why such a change could be reasonable, so this is not a way of making it impossible for you to get what you want - I'm simply telling you the correct process if you want to take this further.

That's very fair expectation. I will take a look at the discussion that @notatallshaw pointed us to (thanks - it's next to impossible for us who have 100s of others things to follow all the discussions in packaging) and see if I can propose something (as a follow up as @pfmoore suggested)

[notatallshaw]

It should also be noted, it is Packaging which implements the behavior here, pip mostly inherits from it. Another reason why discussing the spec here is not that useful.

For changing this behavior you describe, which all implementations of versions specifiers follow (packaging, poetry-core, and uv), there would need to be a consensus reached on discuss (and a PEP of it was agreed a change to the spec) and then packaging would need to be updated and then pip would need to vendor that version of packaging.

See a recent example where I updated packaging where it was agreed it didn't follow the spec (so no discuss discussion was required): pypa/packaging#788 / pypa/packaging#794. This behavior will be vendored into the next version of pip, but no pip specific logic is updated.

[maarre]

I don't care if it is you or packaging owning the bug. This is from the spec which I linked in my previous post. This is a bug.

https://packaging.python.org/en/latest/specifications/version-specifiers/#handling-of-pre-releases

Handling of pre-releases
Pre-releases of any kind, including developmental releases, are implicitly excluded from all version specifiers, unless they are already present on the system, explicitly requested by the user, or if the only available version that satisfies the version specifier is a pre-release.

By default, dependency resolution tools SHOULD:

accept already installed pre-releases for all version specifiers

accept remotely available pre-releases for version specifiers where there is no final or post release that satisfies the version specifier

exclude all other pre-releases from consideration

Dependency resolution tools MAY issue a warning if a pre-release is needed to satisfy a version specifier.

Dependency resolution tools SHOULD also allow users to request the following alternative behaviours:

accepting pre-releases for all version specifiers

excluding pre-releases for all version specifiers (reporting an error or warning if a pre-release is already installed locally, or if a pre-release is the only way to satisfy a particular specifier)

Dependency resolution tools MAY also allow the above behaviour to be controlled on a per-distribution basis.

Post-releases and final releases receive no special treatment in version specifiers - they are always included unless explicitly excluded.

[potiuk]

I don't care if it is you or packaging owning the bug. This is from the spec which I linked in my previous post. This is a bug.

As much as I would like to have similar behaviour, I am afraid you are wrong @maarre and @pfmoore and @notatallshaw are right. The chapter you quoted is irrelevant. If the package has >1.3 according to odering rules, pre-releases are excluded. And the fact whether you classify it as a bug does not matter - this is open source, it's you who are responsible to make it happen if you want to make it happen (and if specification needs to be changed - someone has to roll their sleeves up and make it happen.

Whether it's you, me or anyone else, it does not matter, but someone has to do it. And it's clear that pip mainteiners expects those who want it, to actually spend their time and energy to do it, rather than expecting them to spend their volunteer effort on it. Which is fair expectation and the way how open-source works.

So if you really want it to happen writing it is a bug does not move a needle. Doing something about it, discussing it in packaging forum might. So the ball is on your (and maybe my) side if I decide to spend my - also volunteer and unpaide by anyone - time for it.

Also if you wish to get it done, and have no time for it yourself, one other viable option is to pay for someone's time. I for one take Github Sponsorship, so If you would like to support me in that - I am all for taking sponsorship to lead such an effort.

Are you willing to do so?

[pfmoore]

I'm intimately familiar with that section of the spec, thank you. "Accept" here means "consider" (as opposed to excluding them without consideration, which is what the first sentence says). Nothing in that section, though, says that pre-releases should be viewed as satisfying a condition that the ordering relationship says they don't satisfy. In fact, the spec is explicit in saying (here):

Within a numeric release (1.0, 2.7.3), the following suffixes are permitted and MUST be ordered as shown:
.devN, aN, bN, rcN, <no suffix>, .postN

By this rule, 0.7.1a20241118121953 is smaller than 0.7.1, and so does not satisfy the condition ">=0.7.1".

While I understand your frustration - as I've said, I can see valid arguments for changing the spec - neither your tone nor your refusal to listen to people trying to explain to you the correct process if you want a change in this are are helping make your case.

You've been given the information on how to progress this, if you want to. What you do next is up to you, but continuing to demand that someone else fix this for you will get you nowhere.

[notatallshaw]

I don't care if it is you or packaging owning the bug.

Pip maintainers (I am not one) don't agree it's a bug, and I am telling you even if they did they couldn't do anything about it, it would require Packaging maintainers to agree it's a bug and accept a PR.

I am telling you this to give you somewhere that might make progress, if you decide it's worth spending time and resources on.

Remember, this is open source software built by volunteers. Neither Pip nor Packaging are maintained by some big corporation with dedicated support teams. We're all users here, contributing as we can.

Pip relies on Packaging to follow the spec. It doesn't implement the spec itself. Therefore, there's nothing more to be done here.