Python 3.12 update. #58

dumol · 2024-05-28T09:03:25Z

Scope

This was supposed to fix #57.

Changes

Updated Python to version 3.12.7, as there's no pywin32 for Python 3.13 for now. Fixes CVE-2024-28757, CVE-2024-45490, CVE-2024-45491 and CVE-2024-45492.

Also updating OpenSSL for Python's ssl module to 3.0.15 on all platforms, fixing CVE-2024-6119, CVE-2024-6119. On Windows, CVE-2024-6119 and CVE-2024-6119 are now fixed too.

Python modules version updates:

pip to 24.2
setuptools to ~~75.1.0~~ (70.3.0 to match the version in server repo)
psutil to 6.0.0.

Updated the Alpine Linux version to build on to 3.15 (with musl 1.2.2). Therefore, musl 1.1.x distributions are no longer supported. This follows upstream cryptography, which only builds musl 1.2 wheels since version 43.0.0.

Drive-by changes:

Added support for building beta and rc Python versions.
Enabled building against libedit on macOS, where it's a system lib.
Removed safety and added requirements.txt to leverage GitHub's own security checks.
The requirements.txt file is auto-updated when building successfully on Windows through GHA.
Updated pythia.* from server repo with minor improvements.
Trimmed the packages by installing only the minimum xz stuff, and by bzip'ing copied header files.

Testing

Automated.

…re, take two.

adiroiban

Thanks for the changes

Moving to 3.12 is an important step toward 3.13

Maybe, in the future we should always focus on gettting the next Python version, instead of trying to jump from 3.11 to 3.13 for example

Can we update the docker tests, to include centos 8 and 9 ?

I have found this issue related to the new libnsl dependency

conda-forge/rasterio-feedstock#220

I hope that we can get the "-dev" packages removed from our build system, and then the Python configure script will disable those modules

adiroiban · 2024-10-08T10:57:20Z

.github/workflows/bare.yaml

@@ -36,6 +40,12 @@ jobs:
      timeout-minutes: 5
      run: bash ./build.sh test

+    # Commit changed requirements.txt back to the repository
+    - uses: chevah/git-auto-commit-action@HEAD


I am not happy with this ... but we can look to fix this in a separate PR

The CI run should not modify the source code

Can we have the "requirements.txt" manually updated before a push ?

I guess that this is a question for #62

What we can do as part of the CI, is make sure that "requirements.txt" doesn't need an update during a CI run

We have something similar for chevah/server

What we can do as part of the CI, is make sure that "requirements.txt" doesn't need an update during a CI run

We have something similar for chevah/server

Not sure what that would be. I know it's used for cache updates, but what do you mean by "making sure it doesn't need an update"?

adiroiban · 2024-10-08T11:02:43Z

requirements.txt

@@ -0,0 +1,5 @@
+pip==24.2
+psutil==6.0.0


Why do we need to distribute psutil with pythia ?

For the long term, my expectation is that pythia, will be just python + embedded pip

No other requirements.

setuptools / pywin32 / pycparser can all be installed later in chevah/server and chevah/compat via pip from our PyPI mirror

It has been added as requested through chevah/python-package#125.

It can also be removed as requested. I've added a ticket to deal with all this: #66

Note though that psutil doesn't have a musl wheel, so we need to build it until upstream changes its mind. More at giampaolo/psutil#2126

thanks. That's ok. We just need to add a comment that psutils is still needed and link to the upstream bug report.

In theory, on alpine, we can build the wheel for psutil and then copy it to bin.chevah.com .. it will be available as wheel for Alpine just for us

adiroiban · 2024-10-08T11:03:24Z

src/xz/chevahbs

@@ -37,7 +37,10 @@ chevahbs_try() {


 chevahbs_cp() {


this needs a comment ... I have no idea why we need this and what it does

It's copying needed files after a successful build. All the commands in chevahbs for this library are also cp commands.

adiroiban · 2024-10-08T11:05:01Z

src/xz/chevahbs

@@ -37,7 +37,10 @@ chevahbs_try() {


 chevahbs_cp() {
-    execute "${MAKE_CMD[@]}" install DESTDIR="$INSTALL_DIR"
+    # xz's installation copies binaries too, but there's no need for them.
+    execute cp -R src/liblzma/api/* "$INSTALL_DIR"/include/


Do we need these files in the install dir ?

What is the "install dir" ?

I expect them to be needed in the build dir , but they should not be needed for the final package

$INSTALL_DIR is where the built libraries and Python are installed (or simply copied as needed).

Pruning files needed there but not wanted in the final dist package is done through cleanup_install_dir() in functions_build.sh:

pythia/functions_build.sh

Line 172 in ba80a30

cleanup_install_dir() {

Added #67 for this.

adiroiban · 2024-10-08T11:07:29Z

src/chevah-python-tests/test_python_binary_dist.py

            allowed_deps=[
                'libc.so.6',
                'libcrypt.so.1',
                'libdl.so.2',
                'libm.so.6',
+                'libnsl.so.1',


Is this library directly needed by Python 3.12?

Can we build python without NIS module

https://docs.python.org/3/library/nis.html

This module is not used by the Chevah repos ... and will be removed soon

Didn't bother much about it, because it's so small and unimportant, soon to be removed.

We used to patch setup.py for disabling such modules, but that file is gone. Other ideas of how to do this with 3.12?

Fixed through https://github.com/chevah/pythia/pull/58/files/2303a256768d84384699427d6498fe9fb580eb2e..ac6595fba8c420b6f6a5a63559d7398cc29e4a6f.

.github/workflows/docker.yaml

dumol · 2024-10-08T11:19:33Z

Ooops… needs-changes

dumol · 2024-10-08T12:14:00Z

Thanks for the review!

Can we update the docker tests, to include centos 8 and 9 ?

These are not actually "tests", but rather builds. So we build the musl and glibc Linux packages through Docker on Alpine Linux 3.15 and Amazon Linux 2.

Building on other platforms would result in false positives, e.g. for RHEL 8/9 and clones, the build is going to be contaminated with unwanted dependencies.

I have found this issue related to the new libnsl dependency

conda-forge/rasterio-feedstock#220

I've seen that this is part of glibc on Amazon Linux 2, thus not removable. Checked a number of distro containers, all had libnsl libraries, but I see now that some have libnsl.so.2 as an independent package installed by default, with libnsl.so.1 available through a legacy package, similar to libxcrypt-compat.

Obviously, this is a problem that might arise again in the future. Not sure how to best check for this, now that tests in the server repo are not done on supported platforms. But why didn't compat tests catch this issue?

I hope that we can get the "-dev" packages removed from our build system, and then the Python configure script will disable those modules

Well, on Amazon Linux 2 this would be glibc-headers… I'm going to try the same hack as for avoiding building dbm and gdbm modules.

dumol · 2024-10-08T13:30:58Z

The dependency on libnsl.so.1 was removed, the nis module is no longer built on Amazon Linux 2. I've commented above on this, I'm bothered that it wasn't caught by the tests in the compat repo at chevah/compat#700. What shall we do about it?

I've added tickets for the other suggestions, but left the conversations open until your resolution. Let me know if I missed something. Thanks!

needs-review

adiroiban · 2024-10-08T13:41:40Z

I'm bothered that it wasn't caught by the tests in the compat repo at chevah/compat#700. What shall we do about it?

Maybe Oracle Linux 8 has that library ... I think that we are ok.

These are not actually "tests", but rather builds.

True. Makes sense.

well... what we can do is create separate docker test runs as part of GitHub actions.
It will work something like this

After the binary .tar.gz is created for each OS, upload that file as an GitHub Action artifact
After all the build jobs are done, trigger the docker test job
THe docker test job will download the GitHub Action Artifacts for this build and will save them in cache
The docker test job can then run a few smoke tests to make sure we can start Python on other OSes ... for example run our custom test file

That's all

With GitHub Action Artifacts we can "publish" the pythia .tar.gz files in kind of a temporary location

More info about GitHub Artifacts here https://github.com/actions/upload-artifact

adiroiban

Thanks. Let's have this released.

We can then merge compat ...and also I am looking forward to have this included in chevah/server

Many thanks again! Great work!

Zlib 1.3.1 update.

111024a

dumol linked an issue May 28, 2024 that may be closed by this pull request

Python 3.13 update. #57

Closed

dumol self-assigned this May 28, 2024

dumol added the high-priority label May 28, 2024

dumol added 10 commits May 28, 2024 09:09

Pip 24.0 update.

ff41dd8

Debug outdated macOS Homebrew hacks.

a102aee

No more Homebrew hacking on GitHub's macOS setups.

d5240fa

Make it possible to ignore preset Safety IDs.

3a2291c

Make it possible to ignore preset Safety IDs, take two.

4a5cb66

Unified debugging for testing on GitHub's bare OS'es.

12b79af

Update macOS setup for latest GitHub labels.

543d85f

Shellcheck fixes.

f6e873c

Shellcheck fixes and more.

40291f7

Put back Homebrew hacks for x86 macOS.

87713dd

dumol force-pushed the 57-python-313-update branch from d25e663 to 87713dd Compare May 31, 2024 14:18

dumol added 15 commits June 3, 2024 11:27

Do not install binary wheels.

e7e0fb6

Do not install binary wheels, take two.

8aaeaa3

Also ignore the jinja2 disputed safety issue.

e515cfc

Properly generate SAFETY_IGNORDE_OPTS for multiple Safety IDs to ignore.

c562a70

Properly generate SAFETY_IGNORDE_OPTS for multiple Safety IDs to igno…

bf771c3

…re, take two.

Try debugging safety opts.

1ee3fd8

Process Safety IDs the Bash way.

f5e176a

Make it possible to skip tests during build.

e0675ce

Switched to latest OpenSSL 3.0.x on non-Windows platforms.

e61df38

More updated libs.

041ee81

Latest versions for remaining Python modules.

2724f91

Manual merge with master

238dda3

Try building Python 3.13.0b2.

abdeddf

Try building Python 3.13.0b2, take two.

0feae9d

Python 3.13 deprecates crypt and spwd.

1018d97

chevah-robot added the needs-review label Oct 4, 2024

dumol mentioned this pull request Oct 7, 2024

[#699] Python 3.12 update. chevah/compat#700

Merged

adiroiban requested changes Oct 8, 2024

View reviewed changes

chevah-robot added needs-changes and removed needs-review labels Oct 8, 2024

dumol requested a review from adiroiban October 8, 2024 11:15

chevah-robot added needs-review and removed needs-changes labels Oct 8, 2024

chevah-robot added needs-changes and removed needs-review labels Oct 8, 2024

chevah-robot removed the request for review from adiroiban October 8, 2024 11:19

dumol mentioned this pull request Oct 8, 2024

Stop packing Python modules. #66

Open

One more hack for not building a default Python module.

aada621

One more hack for not building a default Python module, take two.

ac6595f

dumol mentioned this pull request Oct 8, 2024

Improved pruning of the packages. #67

Open

chevah-robot added needs-review and removed needs-changes labels Oct 8, 2024

dumol requested a review from adiroiban October 8, 2024 13:32

adiroiban approved these changes Oct 8, 2024

View reviewed changes

chevah-robot added needs-merge and removed needs-review labels Oct 8, 2024

dumol mentioned this pull request Oct 10, 2024

Test the Pythia packages on more platforms #68

Open

dumol added 2 commits October 10, 2024 07:42

Added a comment for psutil.

e45dcd7

Changes after own review.

77ba463

dumol merged commit bb41ace into master Oct 10, 2024
5 checks passed

dumol deleted the 57-python-313-update branch October 10, 2024 09:18

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Python 3.12 update. #58

Python 3.12 update. #58

dumol commented May 28, 2024 •

edited

Loading

adiroiban left a comment

adiroiban Oct 8, 2024

dumol Oct 8, 2024

adiroiban Oct 8, 2024

dumol Oct 8, 2024 •

edited

Loading

dumol Oct 8, 2024

adiroiban Oct 8, 2024

adiroiban Oct 8, 2024

dumol Oct 8, 2024 •

edited

Loading

adiroiban Oct 8, 2024

dumol Oct 8, 2024

dumol Oct 8, 2024

adiroiban Oct 8, 2024

dumol Oct 8, 2024

dumol Oct 8, 2024

dumol commented Oct 8, 2024

dumol commented Oct 8, 2024 •

edited

Loading

dumol commented Oct 8, 2024

adiroiban commented Oct 8, 2024

adiroiban left a comment

		@@ -0,0 +1,5 @@
		pip==24.2
		psutil==6.0.0

Python 3.12 update. #58

Python 3.12 update. #58

Conversation

dumol commented May 28, 2024 • edited Loading

Scope

Changes

Testing

adiroiban left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dumol Oct 8, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dumol Oct 8, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dumol commented Oct 8, 2024

dumol commented Oct 8, 2024 • edited Loading

dumol commented Oct 8, 2024

adiroiban commented Oct 8, 2024

adiroiban left a comment

Choose a reason for hiding this comment

dumol commented May 28, 2024 •

edited

Loading

dumol Oct 8, 2024 •

edited

Loading

dumol Oct 8, 2024 •

edited

Loading

dumol commented Oct 8, 2024 •

edited

Loading