Use Docker to build the deb packages

[amercader]

This replaces the old Ansible / Vagrant setup with Docker. All the building steps are run in the relevant ubuntu:X.Y image, and the resulting deb package is copied to the client host. As with the old scripts, you can pass build args to control the process. The full command is:

docker buildx build \
    --output type=local,dest=.  \
    --build-arg CKAN_VERSION=2.11 \
    --build-arg CKAN_BRANCH=dev-v2.11 \
    --build-arg DATAPUSHER_VERSION=0.0.21 \
    --build-arg DISTRIBUTION=noble \
    --build-arg UBUNTU_VERSION=24.04 \
    .

For now we are sticking with a single Dockerfile for all Ubuntu versions as there is only one minor difference in focal, but if there are more differences added in the future we will switch to different Dockerfiles.

All the target combinations for the next release build fine, full script below for reference:

CKAN version	Ubuntu version
2.10	20.04 (focal)
2.10	22.04 (jammy)
2.11	22.04 (jammy)
2.11	24.04 (noble)

TODO:

• Adapt ckan-package helper script
• Finish README
• Automate via actions
• Publish packages (as a Github release and / or s3 packaging.ckan.org )

docker buildx build --output type=local,dest=./output --build-arg CKAN_VERSION=2.10 --build-arg CKAN_BRANCH=dev-v2.10 --build-arg DATAPUSHER_VERSION=0.0.21 --build-arg DISTRIBUTION=focal --build-arg ITERATION=1 --build-arg UBUNTU_VERSION=20.04 .

docker buildx build --output type=local,dest=./output --build-arg CKAN_VERSION=2.10 --build-arg CKAN_BRANCH=dev-v2.10 --build-arg DATAPUSHER_VERSION=0.0.21 --build-arg DISTRIBUTION=jammy --build-arg ITERATION=1 --build-arg UBUNTU_VERSION=22.04 .

docker buildx build --output type=local,dest=./output --build-arg CKAN_VERSION=2.11 --build-arg CKAN_BRANCH=dev-v2.11 --build-arg DATAPUSHER_VERSION=0.0.21 --build-arg DISTRIBUTION=jammy --build-arg ITERATION=1 --build-arg UBUNTU_VERSION=22.04 .

docker buildx build --output type=local,dest=./output --build-arg CKAN_VERSION=2.11 --build-arg CKAN_BRANCH=dev-v2.11 --build-arg DATAPUSHER_VERSION=0.0.21 --build-arg DISTRIBUTION=noble --build-arg ITERATION=1 --build-arg UBUNTU_VERSION=24.04 .

[kowh-ai]

Wonderful wonderful work here @amercader! I have quickly built 4 debian packages on my Mac OS machine:

Here are the branches I have built (just to test)

1. dev-v2.11 (noble)
2. dev-v2.11 (jammy)
3. ckan-2.10.4 (jammy)
4. ckan-2.10.4 (focal)

I'm happy to test these debian packages on fresh VM's if you want me to.

I can further test and merge after you finish the TODO above or when give me the word

[amercader]

@kowh-ai great! Testing the deb packages would be great, this is what's important for the release. The versions you tested are correct, but use the dev-v2.10 branch otherwise we won't get the latest changes included in 2.10.5.

Let's do at least the first two items before merging, actions can be a separate PR

[kowh-ai]

Testing of deb packages

• dev-v2.10 on focal (Ubuntu 20.04)
• dev-v2.10 on jammy (Ubuntu 22.04)
• dev-v2.11 on jammy (Ubuntu 22.04)
• dev-v2.11 on noble (Ubuntu 24.04)

[kowh-ai]

@amercader - had an issue installing dev-v2.11 on jammy. Seems the file: /usr/lib/ckan/default/lib/python3.10/site-packages/ckanext/activity/subscriptions.py incurred some corruption:

I tried a brand new VM and re-built this debian package but I got the same result.

To fix I just removed the corrupted lines in the subscriptions.py file...after that everything was all hunky dory

I'll continued with the others now

[amercader]

That's really weird. This looks like a merge error but the HEAD at the dev-v2.11 branch doesn't have these lines:

https://github.com/ckan/ckan/blob/dev-v2.11/ckanext/activity/subscriptions.py#L167

These lines were removed 3 weeks ago:

ckan/ckan@af60df9

Try using --no-cache when building to see if that helps

[kowh-ai]

Yes the --no-cache option seems to fix this corruption. Previously I ran a docker system prune and when building the deb package looked like it was compiling/installing everything from scratch again...but maybe it wasn't..

[kowh-ai]

@amercader - Looks to be a problem with running dev-v2.11 on noble (Ubuntu 24.04)

the deb package for noble includes Python 3.12 which has a problem with find_module

This is the error:

I'll see if I can work out how to include (build) Python 3.11 in the package rather than 3.12

[amercader]

This again looks like a caching issue. The py 3.12 compatibility issue with loader.find_module() was fixed in ckan/ckan#8357, and backported to dev-v2.11 (ckan/ckan#8368), meaning that the fix is in the dev-v2.11 branch.

Docker with cache the RUN git clone ... step, which is fine when we are using tags but problematic when targeting the release branch. We could use some of the hacks described here but for now it's just easier to always use --no-cache.

[kowh-ai]

@amercader - OK, I thought I had used the --no-cache option in my Makefile but then again maybe I didn't...Good to see Python 3.12 can remain...thanks

[amercader]

This is now complete.

There are two new GitHub workflows:

• build.yml builds the deb packages (based on the versions supplied in VERSIONS.json) and stores them as artifacfts in the workflow run page. This is triggered on every push.
  

• Additionally, when a tag is pushed, publish.yml also builds the packages and:

  1. Uploads them to the S3 bucket powering https://packaging.ckan.org
  2. Creates a new GitHub release with the packages attached as asset