Python 3.10 update. (#39)

* Only build generic Linux packages.

* Only build x64 Windows packages.

* Try with cryptography 39.0.2 wheels.

* Retry Windows matrix.

* Use linux instead of lnx for Linux packages.

* Use latest OpenSSL 1.1.1 sources for Python's ssl module.

* Check for cryptography's OpenSSL version everywhere.

* Fixed cryptography's OpenSSL version to check for.

* Try fixing Safety errors.

* Updated all Python modules to latest version.

* Try building on CentOS 6.10.

* Disable OpenSSL tests for now.

* Updated macOS deps.

* Try CentOS 7.

* Revert "Try CentOS 7."

This reverts commit 861817c.

* Revert "Try building on CentOS 6.10."

This reverts commit 9d2551e.

* Use a get-pip.py that still works on CentOS 5.

* Build the generic Linux version on Ubuntu 18.04.

* Forward port the latest cleanups from python-package.

* Also clean up on Windows.

* Put include/ back after packing the distributable.

* Also remove ssl/ if found, e.g. on macOS.

* Re-enabled OpenSSL 1.1.x self tests.

* Re-enabled x86 Windows build.

* Run the macOS build from the beginning.

* Changes after own review.

* Removed subprocess32.

* Python 3.10.

* Updated libffi to version 3.4.4.

* Run the checks in all PRs.

* Updated Python version patching.

* Disabled readline for macOS too.

* Updated xz and SQLite to latest versions.

* Changes after own review.

* One small correction in a comment.

* Updated cryptography to newly-released 40.0.1.

* Updated OpenSSL version to check for.

* Check for cffi module too.

* Build and check for PyNaCl module.

* Syntax fix for cffi check.

* Collect PyNaCl when building.

* Added charset-normalizer, markupsafe, pywin32.

* Update src/chevah-python-test/test_python_binary_dist.py

* Enable debug

* Apply suggestions from code review

* Update src/Python-Windows/chevahbs

* Update src/Python-Windows/chevahbs

* fix 3.10 path

* Update src/Python-Windows/chevahbs

* enable debug

* Update src/Python-Windows/chevahbs

* Update test_python_binary_dist.py

* Apply suggestions from code review

* Apply suggestions from code review

* Update .github/workflows/bare.yaml

* Update src/Python-Windows/chevahbs

* The -3 check is no longer needed.

* Removed Windows matrix from bare workflow.

* Renamed docker workflow job.

* Updated Python to version 3.10.11.

* Updated Python to version 3.10.11 (bis).

* Actually remove the build path for 32bit Windows packages.

* Document why MarkupSafe is pinned to an older version.

* Try -latest in the bare workflow.

* Better document supported Linux distributions.

* Build generic Linux on Amazon 2 with glibc 2.26.

* Lower case yum parameter.

* Use gcc 10.4 instead of 7.2 for building the generic Linux package.

* Link gcc 10.4 in the default path.

* Back to using gcc 7.2 on Amazon Linux 2.

* Try building an ARM64 package.

* Try building an ARM64 package, take 2.

* Removed ARM64 job.

* Try building on Apple Silicon.

* Removed third-party action from the macos-m1 job.

* Replaced third-party action with out fork for the other jobs.

* Changes after own review.

---------

Co-authored-by: Adi Roiban <[email protected]>

.github/workflows/bare.yaml

@@ -2,18 +2,13 @@
 # GitHub actions for building and testing on bare hardware.
 # Typically, Windows and macOS builds are done this way.
 # Look for the Linux builds in the "docker" workflow.
-#
-# Don't use `-latest` for runners, pin specific OS versions instead. More at
-# https://help.github.com/en/actions/reference/virtual-environments-for-github-hosted-runners.
 
 name: Bare
 
 on:
   push:
     branches: [ main ]
-
   pull_request:
-    branches: [ main ]
 
 concurrency:
   group: bare-${{ github.ref }}
@@ -30,13 +25,7 @@ env:
 jobs:
   windows:
     # The type of runner that the job will run on
-    runs-on: ${{ matrix.runs-on }}
-    if: github.event.inputs.job == '' || github.event.inputs.job == 'windows'
-    strategy:
-      # Workflow won't be cancelled at the first failed job.
-      fail-fast: false
-      matrix:
-        runs-on: [ windows-2022, windows-2019 ]
+    runs-on: windows-latest
     timeout-minutes: 15
 
     # Steps represent a sequence of tasks that will be executed as part of the job
@@ -74,7 +63,7 @@ jobs:
     # Command line debugging through Tmate. v3 works on Windows too.
     - name: Tmate debug on failure
       if: failure() && env.TMATE_DEBUG == 'yes'
-      uses: mxschmitt/action-tmate@v3
+      uses: chevah/action-tmate@v3
       with:
         limit-access-to-actor: true
 
@@ -97,7 +86,7 @@ jobs:
 
 
   macos:
-    runs-on: macos-10.15
+    runs-on: macos-latest
     timeout-minutes: 30
     steps:
     - uses: actions/checkout@v2
@@ -137,6 +126,30 @@ jobs:
 
     - name: Tmate debug on failure
       if: failure() && env.TMATE_DEBUG == 'yes'
-      uses: mxschmitt/action-tmate@v3
+      uses: chevah/action-tmate@v3
       with:
         limit-access-to-actor: true
+
+
+  macos-m1:
+    runs-on: m1
+    timeout-minutes: 30
+    steps:
+    - uses: actions/checkout@v2
+      with:
+        fetch-depth: 2
+
+    - name: Build Pythia
+      run: ./build.sh build
+
+    - name: Test Pythia
+      run: ./build.sh test
+
+    - name: Upload testing package
+      run: |
+        mkdir -pv ~/.ssh/
+        touch priv_key
+        chmod 600 priv_key
+        echo "${{ secrets.SFTPPLUS_BIN_PRIV_KEY }}" > priv_key
+        echo "${{ secrets.SFTPPLUS_BIN_HOST_KEY }}" > ~/.ssh/known_hosts
+        ./publish_dist.sh ; rm priv_key

.github/workflows/docker.yaml

@@ -10,7 +10,6 @@ on:
   push:
     branches: [ main ]
   pull_request:
-    branches: [ main ]
 
 concurrency:
   group: docker-${{ github.ref }}
@@ -24,14 +23,14 @@ env:
 # Using a job name that doesn't contain the OS name, to minimize the risk of
 # confusion with the OS names of the containers, which are the relevant ones.
 jobs:
-  latest:
+  linux:
     runs-on: ubuntu-latest
     container: ${{ matrix.container }}
     strategy:
       fail-fast: false
       matrix:
-        # Alpine 3.12 has musl 1.1.24, Ubuntu 18.04 has glibc 2.27.
-        container: [ 'alpine:3.12', 'ubuntu:18.04' ]
+        # Alpine 3.12 has musl 1.1.24, Amazon 2 has glibc 2.26.
+        container: [ 'alpine:3.12', 'amazonlinux:2' ]
     timeout-minutes: 30
     steps:
 
@@ -45,6 +44,12 @@ jobs:
         curl --output /usr/local/bin/paxctl https://bin.chevah.com:20443/third-party-stuff/alpine/paxctl-3.12
         chmod +x /usr/local/bin/paxctl
 
+    - name: Amazon setup
+      if: startsWith(matrix.container, 'amazonlinux')
+      run: |
+        yum -y upgrade
+        yum -y install git-core gcc make m4 patch tar unzip perl perl-Test-Simple
+
     - name: Ubuntu setup
       if: startsWith(matrix.container, 'ubuntu')
       run: |
@@ -84,7 +89,7 @@ jobs:
     # Not working on Alpine (not supported) and CentOS 5 (glibc too old).
     - name: Tmate debug on failure
       if: failure() && env.TMATE_DEBUG == 'yes'
-      uses: mxschmitt/action-tmate@v3
+      uses: chevah/action-tmate@v3
       with:
         sudo: false
         limit-access-to-actor: true

README.rst

@@ -21,13 +21,16 @@ Note that compat tests are currently only working on the ``python2.7`` branch.
 Supported platforms
 -------------------
 
-* Windows Server 2012 R2 and newer (x86 and x64)
-* macOS 10.13 and newer (Intel Macs only)
-* all glibc-based Linux distributions (glibc 2.17+)
+* Windows Server 2012 R2 and newer
+* macOS 10.13 and newer (both Intel and Apple Silicon)
+* Red Hat Enterprise Linux 8 and newer (including clones)
+* Ubuntu 18.04 LTS and newer
+* Amazon Linux 2 and newer
 * Alpine Linux 3.12 and newer.
 
 Platforms that should work, but are not regularly tested:
 
+* all glibc-based Linux distributions (glibc 2.26+)
 * all musl-based Linux distributions (musl 1.1.24+)
 * FreeBSD 12 and newer
 * OpenBSD 6.7 and newer

build.conf

@@ -5,30 +5,40 @@ BUILD_DIR="build"
 DIST_DIR="dist"
 
 # Python and lib versions.
-PYTHON_BUILD_VERSION="3.8.6"
-LIBFFI_VERSION="3.4.2"
+PYTHON_BUILD_VERSION="3.10.11"
+LIBFFI_VERSION="3.4.4"
 ZLIB_VERSION="1.2.13"
 BZIP2_VERSION="1.0.8"
-XZ_VERSION="5.2.5"
+XZ_VERSION="5.4.1"
 # Statically build the BSD libedit on selected platforms to get the
 # readline module available without linking to the GPL-only readline libs.
+# If there's a need to reenable this, our libedit patch for Python 3.9 was
+# https://github.com/chevah/pythia/pull/5/commits/09c128154d23feb6b1a7cb5a8d79.
+# A newer patch is available at https://github.com/python/cpython/issues/57710.
 LIBEDIT_VERSION="20170329-3.1"
 OPENSSL_VERSION="1.1.1t"
 # To match the unusual naming scheme for SQLite downloads, eliminate dots from
-# the regular SQLite version, then add 3 zeros. E.g. "3.33.0" -> "3330000".
+# the regular version, then add zeros where missing. E.g. "3.41.1" -> "3410100".
 # When updating this, also update the year in src/sqlite/chevahbs, if needed.
-SQLITE_VERSION="3380300"
+# To check for a SHA3-256 signature: openssl dgst -sha3-256 file.tar.gz.
+SQLITE_VERSION="3410100"
 # These are used by get-pip.py.
 PIP_VERSION="23.0.1"
-SETUPTOOLS_VERSION="67.6.0"
+SETUPTOOLS_VERSION="67.6.1"
 
 # Python modules versions to be used everywhere possible.
 # Latest cryptography/bcrypt require Rust, use only wheels for them.
 CFFI_VERSION="1.15.1"
-CRYPTOGRAPHY_VERSION="39.0.2"
+CRYPTOGRAPHY_VERSION="40.0.1"
+PYNACL_VERSION="1.5.0"
 BCRYPT_VERSION="4.0.1"
 PSUTIL_VERSION="5.9.4"
 SETPROCTITLE_VERSION="1.3.2"
+# Version 1.1.1 pinned since it works on both py2 and py3.
+MARKUPSAFE_VERSION="1.1.1"
+# Version 2.0.11 pinned since it works on both py2 and py3.
+CHARSETNORMALIZER_VERSION="2.0.11"
+PYWIN32_VERSION="306"
 
 # pycparser is explicitly installed to work around setuptools auto dependencies.
 PYCPARSER_VERSION="2.21"
@@ -41,17 +51,20 @@ BUILD_LIBFFI="no"
 BUILD_ZLIB="no"
 BUILD_BZIP2="yes"
 BUILD_XZ="no"
-BUILD_LIBEDIT="yes"
+BUILD_LIBEDIT="no"
 BUILD_OPENSSL="no"
 BUILD_SQLITE="yes"
 
 # List of default Python modules installed using pip.
 PIP_LIBRARIES="\
     cffi==${CFFI_VERSION} \
     cryptography==${CRYPTOGRAPHY_VERSION} \
+    PyNaCl==${PYNACL_VERSION} \
     bcrypt==${BCRYPT_VERSION} \
     psutil==${PSUTIL_VERSION} \
     setproctitle==${SETPROCTITLE_VERSION}
+    MarkupSafe==${MARKUPSAFE_VERSION} \
+    charset-normalizer==${CHARSETNORMALIZER_VERSION} \
     "
 PIP_ARGS="\
     --index-url=${PIP_INDEX_URL} \

functions_build.sh

@@ -66,7 +66,7 @@ download_sources(){
             echo "## Unpacking archive ${archive_filename}... ##"
             execute $ZIP_CMD "$archive_filename" -d ../../build/
             ;;
-        exe|amd64*|win32*)
+        exe|amd64*)
             # No need to use ../../build/"$project_name"-"$project_ver"/ here.
             echo "    Nothing to unpack in build/ for ${archive_filename}."
             ;;

os_quirks.sh

@@ -13,8 +13,10 @@ case $OS in
         # But not all are from pypi.org. Wheels copied from other places:
         #   * "setproctitle" from https://www.lfd.uci.edu/~gohlke/pythonlibs/
         export BUILD_BZIP2="no"
-        export BUILD_LIBEDIT="no"
         export BUILD_SQLITE="no"
+        PIP_LIBRARIES="$PIP_LIBRARIES \
+            pywin32==${PYWIN32_VERSION} \
+            "
         # On Windows, only one of the installers is downloaded.
         export SHA_CMD="$SHA_CMD --ignore-missing"
         ;;
@@ -32,7 +34,6 @@ case $OS in
         export BUILD_LIBFFI="yes"
         export BUILD_ZLIB="yes"
         export BUILD_XZ="yes"
-        export BUILD_LIBEDIT="no"
         export BUILD_OPENSSL="yes"
         ;;
     macos)

pythia.sh

@@ -86,7 +86,7 @@ ARCH='not-detected-yet'
 PYTHON_CONFIGURATION='NOT-YET-DEFINED'
 PYTHON_VERSION='not.defined.yet'
 PYTHON_PLATFORM='unknown-os-and-arch'
-PYTHON_NAME='python3.8'
+PYTHON_NAME='python3.10'
 BINARY_DIST_URI='https://github.com/chevah/pythia/releases/download'
 PIP_INDEX_URL='https://pypi.org/simple'
 BASE_REQUIREMENTS=''
@@ -106,7 +106,7 @@ check_source_folder() {
 # Called to trigger the entry point in the virtual environment.
 # Can be overwritten in pythia.conf
 execute_venv() {
-    ${PYTHON_BIN} $PYTHON3_CHECK -c 'from paver.tasks import main; main()' "$@"
+    ${PYTHON_BIN} -c 'from paver.tasks import main; main()' "$@"
 }
 
 
@@ -626,12 +626,12 @@ check_glibc_version(){
     local supported_glibc2_version
 
     # Supported minimum minor glibc 2.X versions for various arches.
-    # For x64, we build on Ubuntu 18.04 with glibc 2.27.
-    # For arm64, we build on Ubuntu 16.04 with glibc 2.23.
+    # For x64, we build on Amazon 2 with glibc 2.26.
+    # For arm64, we used to build on Ubuntu 16.04 with glibc 2.23.
     # Beware we haven't normalized arch names yet.
     case "$ARCH" in
         "amd64"|"x86_64"|"x64")
-            supported_glibc2_version=27
+            supported_glibc2_version=26
             ;;
         "aarch64"|"arm64")
             supported_glibc2_version=23
@@ -782,17 +782,6 @@ detect_os() {
             ;;
         "amd64"|"x86_64")
             ARCH="x64"
-            case "$OS" in
-                win)
-                    # 32bit build on Windows 2019, 64bit otherwise.
-                    # Should work with a l10n pack too (tested with French).
-                    win_ver=$(systeminfo.exe | head -n 3 | tail -n 1 \
-                        | cut -d ":" -f 2)
-                    if [[ "$win_ver" =~ "Microsoft Windows Server 2019" ]]; then
-                        ARCH="x86"
-                    fi
-                    ;;
-            esac
             ;;
         "aarch64")
             ARCH="arm64"
@@ -839,15 +828,6 @@ if [ "$COMMAND" == "deps" ] ; then
     install_base_deps
 fi
 
-case $COMMAND in
-    test_ci|test_py3)
-        PYTHON3_CHECK='-3'
-        ;;
-    *)
-        PYTHON3_CHECK=''
-        ;;
-esac
-
 set +e
 execute_venv "$@"
 exit_code=$?

src/Python-Windows/chevahbs

@@ -18,18 +18,16 @@ INSTALL_DIR=$3/lib
 
 # Construct the relevant download link for the Windows version.
 EXT="amd64.zip"
-if [ "$ARCH" = "x86" ]; then
-    EXT="win32.zip"
-fi
 ARCHIVE="python-${VER}.${EXT}"
 LINK="https://www.python.org/ftp/python/${VER}/python-${VER}-embed-${EXT}"
 
-# Download without unpacking (amd64.zip/win32.zip EXTs are treated specially).
+# Download without unpacking (amd64.zip extension is treated specially).
 download_sources $NAME $VER $LINK $EXT
 
 # Installation consists of unpacking the ZIP'ed embeddable distribution.
 echo "## Extracting ZIP archive to ${INSTALL_DIR}... ##"
 execute unzip -q "$ARCHIVE" -d "${INSTALL_DIR}"
 
 echo "## Adding site-packages to python*._pth file... ##"
-echo "lib\\site-packages" >> ${INSTALL_DIR}/python38._pth
+echo "import site" >> ${INSTALL_DIR}/python310._pth
+echo "site.main()" >> ${INSTALL_DIR}/python310._pth

src/Python-Windows/sha512.sum

@@ -1,2 +1 @@
-3b463ef64397326ffd5f20009be6c1304a3bd8646e5bf05c0a8be5ed4aabe7b0a5a6133e962f538292e6290a04fbe07b1caa6c09c54d012b0d07b7f025772921 *Python-3.8.6.amd64.zip
-fb270ce1f1d0bcb95e6c192a35dda276005f74e3c9d89a3a8f799296e9a1db925155437be977a3f3181c25eb522025895563d2333bc6606353bc024b51379107 *Python-3.8.6.win32.zip
+40cbc98137cc7768e3ea498920ddffd0b3b30308bfd7bbab2ed19d93d2e89db6b4430c7b54a0f17a594e8e10599537a643072e08cfd1a38c284f8703879dcc17 *Python-3.10.11.amd64.zip

src/Python/chevahbs

@@ -69,7 +69,7 @@ chevahbs_configure() {
     execute cp Modules/getbuildinfo.c Modules/getbuildinfo.c.orig
     # Don't use 'sed -i' here, it's not supported on macOS 10.13.
     execute sed -e \
-        s/gitid\ =\ \"default\"/gitid\ =\ \"$PYTHIA_VERSION\"/g \
+        s/gitid\ =\ \"main\"/gitid\ =\ \"$PYTHIA_VERSION\"/g \
         Modules/getbuildinfo.c.orig > Modules/getbuildinfo.c
     execute ./configure --prefix="" $CONFIG_ARGS
 }

src/Python/disabled_modules.patch

@@ -1,12 +1,11 @@
-diff -ur Python-3.8.6.orig/setup.py Python-3.8.6.disabled_modules/setup.py
---- Python-3.8.6.orig/setup.py	2020-09-23 12:36:32.000000000 +0000
-+++ Python-3.8.6.disabled_modules/setup.py	2020-10-30 11:48:50.512622149 +0000
-@@ -24,8 +24,16 @@
- TEST_EXTENSIONS = True
+diff -ur Python-3.10.10/setup.py Python-3.10.10-disabled-modules/setup.py
+--- Python-3.10.10/setup.py	2023-02-07 14:05:45.000000000 +0200
++++ Python-3.10.10-disabled-modules/setup.py	2023-03-17 12:41:45.252791918 +0200
+@@ -58,7 +58,16 @@
+ TEST_EXTENSIONS = (sysconfig.get_config_var('TEST_MODULES') == 'yes')
 
  # This global variable is used to hold the list of modules to be disabled.
 -DISABLED_MODULE_LIST = []
--
 +DISABLED_MODULE_LIST = [
 +    '_curses',
 +    '_curses_panel',
@@ -18,5 +17,5 @@ diff -ur Python-3.8.6.orig/setup.py Python-3.8.6.disabled_modules/setup.py
 +    'readline',
 +    ]
 
- def get_platform():
-     # Cross compiling
+ # --list-module-names option used by Tools/scripts/generate_module_names.py
+ LIST_MODULE_NAMES = False