Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Private Google Cloud PyPi proxy causes unnecessary package downloads #3275

Open
1 task
pineapple-pokopo opened this issue Nov 11, 2024 · 1 comment
Open
1 task
Labels
🐛 bug Something isn't working

Comments

@pineapple-pokopo
Copy link

pineapple-pokopo commented Nov 11, 2024

Describe the bug

We use a Google Artifact Registry (GAR) repository as a proxy for the public PyPi repository. We have noticed that adding/updating packages is much slower using the GAR proxy than public PyPi. Inspecting the logging from pdm update -v reveals PDM seems to be redownloading packages when resolving while this does not happen using public PyPi.

Example of such download (full logging below):

unearth.preparer: Downloading <Link https://<REGION>-python.pkg.dev/<PROJECT>/<REPO>/pytest/pytest-8.3.3-py3-none-any.whl (from https://<REGION>-python.pkg.dev/artifacts-downloads/namespaces/<PROJECT>/repositories/<REPO>/downloads/<LONG_RANDOM_STRING>)> (342 kB)

To reproduce

Minimal pyproject.toml:

[project]
name = "test-pdm-private-pypi"
requires-python = "==3.12.*"
dependencies = [
    "pytest==8.3.3",
    "ruff==0.7.2",
]

[tool.pdm]
distribution = false

[[tool.pdm.source]]
name = "pypi"
url = "https://<REGION>-python.pkg.dev/<PROJECT>/<REPO>/simple/"
verify_ssl = true

Expected Behavior

PDM should not redownload (cached) packages

Environment Information

PDM version:
  2.20.0.post1
Python Interpreter:
  /Users/<USER>/workspaces/pdm-test/.venv/bin/python (3.12)
Project Root:
  /Users/<USER>/workspaces/pdm-test
Local Packages:
{
  "implementation_name": "cpython",
  "implementation_version": "3.13.0",
  "os_name": "posix",
  "platform_machine": "arm64",
  "platform_release": "23.6.0",
  "platform_system": "Darwin",
  "platform_version": "Darwin Kernel Version 23.6.0: Thu Sep 12 23:36:55 PDT 2024; root:xnu-10063.141.1.701.1~1/RELEASE_ARM64_T8112",
  "python_full_version": "3.12.7",
  "platform_python_implementation": "CPython",
  "python_version": "3.12",
  "sys_platform": "darwin"
}

pdm -v output

Running pdm update -v --no-sync ruff

With public PyPy:

Updating packages: ruff.
pdm.termui: ======== Start resolving requirements ========
pdm.termui:   pytest==8.3.3
pdm.termui:   ruff==0.7.2
pdm.termui:   Adding requirement python==3.12.*
pdm.termui:   Adding requirement pytest==8.3.3
pdm.termui:   Adding requirement ruff==0.7.2
pdm.termui: ======== Starting round 0 ========
pdm.termui: Adding new pin: python None
pdm.termui: ======== Starting round 1 ========
pdm.termui:   Adding requirement iniconfig(from pytest 8.3.3)
pdm.termui:   Adding requirement packaging(from pytest 8.3.3)
pdm.termui:   Adding requirement pluggy<2,>=1.5(from pytest 8.3.3)
pdm.termui:   Adding requirement colorama; sys_platform == "win32"(from pytest 8.3.3)
pdm.termui: Adding new pin: pytest 8.3.3
pdm.termui: ======== Starting round 2 ========
pdm.termui: Adding new pin: ruff 0.7.2
pdm.termui: ======== Starting round 3 ========
pdm.termui: Adding new pin: pluggy 1.5.0
pdm.termui: ======== Starting round 4 ========
pdm.termui: Adding new pin: colorama 0.4.6
pdm.termui: ======== Starting round 5 ========
pdm.termui: Adding new pin: iniconfig 2.0.0
pdm.termui: ======== Starting round 6 ========
pdm.termui: Adding new pin: packaging 24.2
pdm.termui: ======== Starting round 7 ========
pdm.termui: ======== Resolution Result ========
pdm.termui:      python None
pdm.termui:      pytest 8.3.3
pdm.termui:        ruff 0.7.2
pdm.termui:      pluggy 1.5.0
pdm.termui:    colorama 0.4.6
pdm.termui:   iniconfig 2.0.0
pdm.termui:   packaging 24.2
pdm.termui: Fetching hashes for [email protected]

With GAR PyPi proxy:

Updating packages: ruff.
pdm.termui: ======== Start resolving requirements ========
pdm.termui:   pytest==8.3.3
pdm.termui:   ruff==0.7.2
pdm.termui:   Adding requirement python==3.12.*
pdm.termui:   Adding requirement pytest==8.3.3
pdm.termui:   Adding requirement ruff==0.7.2
pdm.termui: ======== Starting round 0 ========
unearth.preparer: Downloading <Link https://<REGION>-python.pkg.dev/<PROJECT>/<REPO>/pytest/pytest-8.3.3-py3-none-any.whl (from https://europe-west4-python.pkg.dev/artifacts-downloads/namespaces/<PROJECT>/repositories/<REPO>/downloads/<RANDOM_STRING>)> (342 kB)
pdm.termui: Adding new pin: python None
pdm.termui: ======== Starting round 1 ========
pdm.termui:   Adding requirement iniconfig(from pytest 8.3.3)
pdm.termui:   Adding requirement packaging(from pytest 8.3.3)
pdm.termui:   Adding requirement pluggy<2,>=1.5(from pytest 8.3.3)
pdm.termui:   Adding requirement colorama; sys_platform == "win32"(from pytest 8.3.3)
pdm.termui: Adding new pin: pytest 8.3.3
pdm.termui: ======== Starting round 2 ========
unearth.preparer: Downloading <Link https://<REGION>-python.pkg.dev/<PROJECT>/<REPO>/iniconfig/iniconfig-2.0.0-py3-none-any.whl (from https://<REGION>-python.pkg.dev/<PROJECT>/<REPO>/simple/iniconfig/)> (5.9 kB)
unearth.preparer: Downloading <Link https://<REGION>-python.pkg.dev/<PROJECT>/<REPO>/packaging/packaging-24.2-py3-none-any.whl (from https://<REGION>-python.pkg.dev/<PROJECT>/<REPO>/simple/packaging/)> (65 kB)
unearth.preparer: Downloading <Link https://<REGION>-python.pkg.dev/<PROJECT>/<REPO>/pluggy/pluggy-1.5.0-py3-none-any.whl (from https://<REGION>-python.pkg.dev/<PROJECT>/<REPO>/simple/pluggy/)> (20 kB)
unearth.preparer: Downloading <Link https://<REGION>-python.pkg.dev/<PROJECT>/<REPO>/colorama/colorama-0.4.6-py2.py3-none-any.whl (from https://<REGION>-python.pkg.dev/<PROJECT>/<REPO>/simple/colorama/)> (25 kB)
pdm.termui: Adding new pin: ruff 0.7.2
pdm.termui: ======== Starting round 3 ========
pdm.termui: Adding new pin: pluggy 1.5.0
pdm.termui: ======== Starting round 4 ========
pdm.termui: Adding new pin: colorama 0.4.6
pdm.termui: ======== Starting round 5 ========
pdm.termui: Adding new pin: iniconfig 2.0.0
pdm.termui: ======== Starting round 6 ========
pdm.termui: Adding new pin: packaging 24.2
pdm.termui: ======== Starting round 7 ========
pdm.termui: ======== Resolution Result ========
pdm.termui:      python None
pdm.termui:      pytest 8.3.3
pdm.termui:        ruff 0.7.2
pdm.termui:      pluggy 1.5.0
pdm.termui:    colorama 0.4.6
pdm.termui:   iniconfig 2.0.0
pdm.termui:   packaging 24.2
pdm.termui: Fetching hashes for [email protected]

Additional Context

Note that GAR does a redirect when fetching the HTML with packages versions. For example, https://<REGION>-python.pkg.dev/<PROJECT>/<REPO>/simple/ruff/ redirects to https://<REGION>-python.pkg.dev/artifacts-downloads/namespaces/<PROJECT>/repositories/<REPO>/downloads/<RANDOM_STRING> with some random string at the end of the URL. Might this be interfering with the caching PDM does?

Are you willing to submit a PR to fix this bug?

  • Yes, I would like to submit a PR.
@pineapple-pokopo pineapple-pokopo added the 🐛 bug Something isn't working label Nov 11, 2024
@pineapple-pokopo
Copy link
Author

Hi @frostming, do you have any idea on this?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
🐛 bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant