Merge branch 'develop' into main

clemlesne · Apr 6, 2023 · 381925c · 381925c
2 parents 9549159 + 883f1e9
commit 381925c
Show file tree

Hide file tree

Showing 15 changed files with 190 additions and 69 deletions.
diff --git a/.github/workflows/pipeline.yaml b/.github/workflows/pipeline.yaml
@@ -112,11 +112,11 @@ jobs:
       matrix:
         include:
           - os: bullseye
-            arch: linux/amd64,linux/arm/v7,linux/arm64/v8
+            arch: linux/amd64,linux/arm64/v8
           - os: focal
-            arch: linux/amd64,linux/arm/v7,linux/arm64/v8
+            arch: linux/amd64,linux/arm64/v8
           - os: jammy
-            arch: linux/amd64,linux/arm/v7,linux/arm64/v8
+            arch: linux/amd64,linux/arm64/v8
           - os: ubi8
             arch: linux/amd64,linux/arm64/v8
     steps:

diff --git a/README.md b/README.md
@@ -20,7 +20,7 @@ Features:
 - Agent restart itself if it crashes.
 - Auto-scale based on Pipeline usage (with [KEDA](https://keda.sh), not required).
 - Can run air-gapped (no internet access).
-- Cheap to run (scale to 0, dynamic provisioning of agents, can scale from 0 to 100+ in few seconds).
+- Cheap to run (dynamic provisioning of agents, can scale from 0 to 100+ in few seconds).
 - Compatible with Debian, Ubuntu and Red Hat LTS releases.
 - SBOM (Software Bill of Materials) is packaged with each container image.
 - System updates are applied every days.
@@ -62,10 +62,62 @@ helm upgrade --install agent clemlesne-azure-pipelines-agent/azure-pipelines-age
 
 - [Azure Pipelines agent](https://github.com/microsoft/azure-pipelines-agent) (see env var `AGENT_VERSION` on the container images) + [requirements](https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/docker?view=azure-devops#linux)
 - [ASP.NET Core](https://github.com/dotnet/aspnetcore) runtime (required by the Azure Pipelines agent)
-- [Azure CLI](https://github.com/Azure/azure-cli) (required by the Azure Pipelines agent)
+- [Azure CLI](https://github.com/Azure/azure-cli) (required by the Azure Pipelines agent) + requirements ([Python 3.8](https://www.python.org/downloads/release/python-380), [Python 3.9](https://www.python.org/downloads/release/python-390), [Python 3.10](https://www.python.org/downloads/release/python-3100), depending of the system, plus C/Rust build tools for libs non pre-built on the platforms)
 - [Powershell](https://github.com/PowerShell/PowerShell), [bash](https://www.gnu.org/software/bash) and [zsh](https://www.zsh.org) (for inter-operability)
 - [gzip](https://www.gnu.org/software/gzip), [make](https://www.gnu.org/software/make), [tar](https://www.gnu.org/software/tar), [unzip](https://infozip.sourceforge.net/UnZip.html), [wget](https://www.gnu.org/software/wget), [yq](https://github.com/mikefarah/yq), [zip](https://infozip.sourceforge.net/Zip.html), [zstd](https://github.com/facebook/zstd) (for developer ease-of-life)
 
+### Capabilities
+
+Capabilities are declarative variables you can add to the agents, to allow developers to select the right agent for their pipeline ([official documentation](https://learn.microsoft.com/en-us/azure/devops/pipelines/process/demands?view=azure-devops&tabs=yaml)).
+
+Note, you can add multiple Helm instances to the same agent pool. Then, disctinct them by capabilities. For examples:
+
+- A pool of AMD64 agents, and a pool of ARM64 agents
+- A pool of agents with GPU, and a pool of agents without GPU
+- A pool of agents with low performance (standard usage), and a pool of agents with high performance (IA training, intensive C/Rust/GraalVM compilation, ...), with distinct Kubernetes Node pool, scaling to 0 when not used ([AKS documentation](https://learn.microsoft.com/en-us/azure/aks/cluster-autoscaler))
+
+#### Example: ARM64 agents
+
+Take the assumption we want to host a specific instance pool to ARM servers.
+
+```yaml
+# values.yaml
+pipelines:
+  pool: Kubernetes
+  capabiliies:
+    - arch-arm64
+
+affinity:
+  nodeAffinity:
+    requiredDuringSchedulingIgnoredDuringExecution:
+      nodeSelectorTerms:
+        - matchExpressions:
+          - key: kubernetes.io/arch
+            operator: In
+            values:
+              - arm64
+```
+
+Deploy the Helm instance:
+
+```bash
+❯ helm upgrade --install agent-arm64 clemlesne-azure-pipelines-agent/azure-pipelines-agent -f values.yaml
+```
+
+Update the Azure Pipelines file in the repository to use the new pool:
+
+```yaml
+# azure-pipelines.yaml
+pool:
+  name: Kubernetes
+  demands:
+    - Agent.OS -equals Linux
+    - arch-arm64
+
+stages:
+  ...
+```
+
 ### Helm values
 
 | Parameter | Description | Default |
@@ -75,7 +127,7 @@ helm upgrade --install agent clemlesne-azure-pipelines-agent/azure-pipelines-age
 | `autoscaling.cooldown` | Time in seconds the automation will wait until there is no more pipeline asking for an agent. Same time is then applied for system termination. | `60` |
 | `autoscaling.enabled` | Enable the auto-scaling, requires [KEDA](https://keda.sh). | `true` |
 | `autoscaling.maxReplicas` | Maximum number of pods, remaining jobs will be kept in queue. | `100` |
-| `autoscaling.minReplicas` | Minimum number of pods. If autoscaling not enabled, the number of replicas to run. If auto-scaling is enabled, it is recommended to set the value to `0`. | `1` |
+| `autoscaling.minReplicas` | Minimum number of pods. If autoscaling not enabled, the number of replicas to run. If `pipelines.capabiliies` is defined, cannot be set to `0`. | `1` |
 | `extraVolumeMounts` | Additional volume mounts for the agent container. | `[]` |
 | `extraVolumes` | Additional volumes for the agent pod. | `[]` |
 | `fullnameOverride` | Overrides release fullname | `""` |
@@ -88,6 +140,7 @@ helm upgrade --install agent clemlesne-azure-pipelines-agent/azure-pipelines-age
 | `nodeSelector` | Node labels for pod assignment | `{}` |
 | `pipelines.cacheSize` | Total cache the pipeline can take during execution, by default [the same amount as the Microsoft Hosted agents](https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/hosted?view=azure-devops&tabs=yaml#hardware). | `10Gi` |
 | `pipelines.cacheType` | Disk type to attach to the agents, see your cloud provider for mor details  ([Azure](https://learn.microsoft.com/en-us/azure/aks/concepts-storage#storage-classes), [AWS](https://docs.aws.amazon.com/eks/latest/userguide/storage-classes.html)). | `managed-csi` (Azure compatible) |
+| `pipelines.capabiliies` | Add [demands/capabilities](https://learn.microsoft.com/en-us/azure/devops/pipelines/process/demands?view=azure-devops&tabs=yaml) to the agent | `[]` |
 | `pipelines.pat` | Personal Access Token (PAT) used by the agent to connect. | *None* |
 | `pipelines.pool` | Agent pool to which the Agent should register. | *None* |
 | `pipelines.url` | The Azure base URL for your organization | *None* |

diff --git a/example/values-amd64.yaml b/example/values-amd64.yaml
@@ -0,0 +1,17 @@
+pipelines:
+  pat: ojemi3pd7adocbqeeuzr5o7ldz2jlwxxbmtrrj7gq5mifhmafpsa
+  pool: Kubernetes
+  url: https://dev.azure.com/shopping-cart-devops-demo
+  cacheType: azurefile-csi
+  capabiliies:
+    - arch-amd64
+
+affinity:
+  nodeAffinity:
+    requiredDuringSchedulingIgnoredDuringExecution:
+      nodeSelectorTerms:
+        - matchExpressions:
+          - key: kubernetes.io/arch
+            operator: In
+            values:
+              - amd64
diff --git a/example/values-arm64.yaml b/example/values-arm64.yaml
@@ -0,0 +1,17 @@
+pipelines:
+  pat: ojemi3pd7adocbqeeuzr5o7ldz2jlwxxbmtrrj7gq5mifhmafpsa
+  pool: Kubernetes
+  url: https://dev.azure.com/shopping-cart-devops-demo
+  cacheType: azurefile-csi
+  capabiliies:
+    - arch-arm64
+
+affinity:
+  nodeAffinity:
+    requiredDuringSchedulingIgnoredDuringExecution:
+      nodeSelectorTerms:
+        - matchExpressions:
+          - key: kubernetes.io/arch
+            operator: In
+            values:
+              - arm64
diff --git a/example/values-high-perf.yaml b/example/values-high-perf.yaml
@@ -0,0 +1,17 @@
+pipelines:
+  pat: ojemi3pd7adocbqeeuzr5o7ldz2jlwxxbmtrrj7gq5mifhmafpsa
+  pool: Kubernetes
+  url: https://dev.azure.com/shopping-cart-devops-demo
+  cacheType: azurefile-csi
+  capabiliies:
+    - node-high-perf
+
+affinity:
+  nodeAffinity:
+    requiredDuringSchedulingIgnoredDuringExecution:
+      nodeSelectorTerms:
+        - matchExpressions:
+          - key: kubernetes.azure.com/agentpool
+            operator: In
+            values:
+              - highperf
diff --git a/example/values.yaml b/example/values.yaml
diff --git a/src/docker/Dockerfile-bullseye b/src/docker/Dockerfile-bullseye
@@ -1,23 +1,33 @@
-# Fetch the vendor with the builder platform to avoid QEMU issues
-FROM --platform=$BUILDPLATFORM mcr.microsoft.com/dotnet/aspnet:6.0-bullseye-slim
+FROM mcr.microsoft.com/dotnet/aspnet:6.0-bullseye-slim
 
 ENV DEBIAN_FRONTEND=noninteractive
 
 # Install:
 # - Azure Pipelines agent system requirements
 # - zsh, for inter-operability
 # - gzip, make, tar, unzip, wget, yq, zip, zstd for developer ease-of-life
+# - Azure CLI system requirements (Python 3.9, plus C/Rust build tools for libs non pre-built on this platform)
 RUN apt-get update -q \
     && apt-get install -y -q --no-install-recommends \
+        build-essential \
         ca-certificates \
+        cargo \
         curl \
         git \
         git-lfs \
         gnupg \
         gzip \
         iputils-ping \
         jq \
+        libffi-dev \
+        libssl-dev \
+        lsb-release \
         make \
+        pkg-config \
+        python3-dev=3.9.* \
+        python3-pip \
+        python3=3.9.* \
+        software-properties-common \
         sudo \
         tar \
         unzip \
@@ -34,13 +44,9 @@ RUN chmod +x arch.sh \
     && ./arch.sh
 
 # Install Azure CLI, then verify installation
-RUN curl -LsSf https://packages.microsoft.com/keys/microsoft.asc | gpg --dearmor > /etc/apt/trusted.gpg.d/microsoft.asc.gpg \
-    && echo "deb [arch=$(./arch.sh)] https://packages.microsoft.com/repos/azure-cli $(. /etc/os-release; echo ${VERSION_CODENAME}) main" > /etc/apt/sources.list.d/azure-cli.list \
-    && apt-get update -q \
-    && apt-get install -y -q --no-install-recommends \
-        azure-cli \
-    && apt-get clean \
-    && rm -rf /var/lib/apt/lists \
+RUN python3 -m pip install --upgrade pip setuptools wheel \
+    && python3 -m pip install azure-cli \
+    && rm -rf $HOME/.cache/pip \
     && az --version
 
 # Install Powershell, then verify installation

diff --git a/src/docker/Dockerfile-focal b/src/docker/Dockerfile-focal
@@ -1,23 +1,33 @@
-# Fetch the vendor with the builder platform to avoid QEMU issues
-FROM --platform=$BUILDPLATFORM mcr.microsoft.com/dotnet/aspnet:6.0-focal
+FROM mcr.microsoft.com/dotnet/aspnet:6.0-focal
 
 ENV DEBIAN_FRONTEND=noninteractive
 
 # Install:
 # - Azure Pipelines agent system requirements
 # - zsh, for inter-operability
 # - gzip, make, tar, unzip, wget, yq, zip, zstd for developer ease-of-life
+# - Azure CLI system requirements (Python 3.8, plus C/Rust build tools for libs non pre-built on this platform)
 RUN apt-get update -q \
     && apt-get install -y -q --no-install-recommends \
+        build-essential \
         ca-certificates \
+        cargo \
         curl \
         git \
         git-lfs \
         gnupg \
         gzip \
         iputils-ping \
         jq \
+        libffi-dev \
+        libssl-dev \
+        lsb-release \
         make \
+        pkg-config \
+        python3-dev=3.8.* \
+        python3-pip \
+        python3=3.8.* \
+        software-properties-common \
         sudo \
         tar \
         unzip \
@@ -34,13 +44,9 @@ RUN chmod +x arch.sh \
     && ./arch.sh
 
 # Install Azure CLI, then verify installation
-RUN curl -LsSf https://packages.microsoft.com/keys/microsoft.asc | gpg --dearmor > /etc/apt/trusted.gpg.d/microsoft.asc.gpg \
-    && echo "deb [arch=$(./arch.sh)] https://packages.microsoft.com/repos/azure-cli $(. /etc/os-release; echo ${VERSION_CODENAME}) main" > /etc/apt/sources.list.d/azure-cli.list \
-    && apt-get update -q \
-    && apt-get install -y -q --no-install-recommends \
-        azure-cli \
-    && apt-get clean \
-    && rm -rf /var/lib/apt/lists \
+RUN python3 -m pip install --upgrade pip setuptools wheel \
+    && python3 -m pip install azure-cli \
+    && rm -rf $HOME/.cache/pip \
     && az --version
 
 # Install Powershell, then verify installation

diff --git a/src/docker/Dockerfile-jammy b/src/docker/Dockerfile-jammy
@@ -1,23 +1,33 @@
-# Fetch the vendor with the builder platform to avoid QEMU issues
-FROM --platform=$BUILDPLATFORM mcr.microsoft.com/dotnet/aspnet:6.0-jammy
+FROM mcr.microsoft.com/dotnet/aspnet:6.0-jammy
 
 ENV DEBIAN_FRONTEND=noninteractive
 
 # Install:
 # - Azure Pipelines agent system requirements
 # - zsh, for inter-operability
 # - gzip, make, tar, unzip, wget, yq, zip, zstd for developer ease-of-life
+# - Azure CLI system requirements (Python 3.10, plus C/Rust build tools for libs non pre-built on this platform)
 RUN apt-get update -q \
     && apt-get install -y -q --no-install-recommends \
+        build-essential \
         ca-certificates \
+        cargo \
         curl \
         git \
         git-lfs \
         gnupg \
         gzip \
         iputils-ping \
         jq \
+        libffi-dev \
+        libssl-dev \
+        lsb-release \
         make \
+        pkg-config \
+        python3-dev=3.10.* \
+        python3-pip \
+        python3=3.10.* \
+        software-properties-common \
         sudo \
         tar \
         unzip \
@@ -34,13 +44,9 @@ RUN chmod +x arch.sh \
     && ./arch.sh
 
 # Install Azure CLI, then verify installation
-RUN curl -LsSf https://packages.microsoft.com/keys/microsoft.asc | gpg --dearmor > /etc/apt/trusted.gpg.d/microsoft.asc.gpg \
-    && echo "deb [arch=$(./arch.sh)] https://packages.microsoft.com/repos/azure-cli $(. /etc/os-release; echo ${VERSION_CODENAME}) main" > /etc/apt/sources.list.d/azure-cli.list \
-    && apt-get update -q \
-    && apt-get install -y -q --no-install-recommends \
-        azure-cli \
-    && apt-get clean \
-    && rm -rf /var/lib/apt/lists \
+RUN python3 -m pip install --upgrade pip setuptools wheel \
+    && python3 -m pip install azure-cli \
+    && rm -rf $HOME/.cache/pip \
     && az --version
 
 # Install Powershell, then verify installation

diff --git a/src/docker/Dockerfile-ubi8 b/src/docker/Dockerfile-ubi8
@@ -1,15 +1,18 @@
-# Fetch the vendor with the builder platform to avoid QEMU issues
-FROM --platform=$BUILDPLATFORM registry.access.redhat.com/ubi8/ubi-minimal:8.7
+FROM registry.access.redhat.com/ubi8/ubi-minimal:8.7
 
 # Install:
 # - Azure Pipelines agent system requirements
+# - Azure CLI system requirements (Python 3.9, plus C/Rust build tools for libs non pre-built on this platform)
 # - ASP.NET Core runtime
 # - zsh, for inter-operability
 # - gzip, make, tar, unzip, wget, yq, zip, zstd for developer ease-of-life
 RUN microdnf install -y --refresh --nodocs --setopt=install_weak_deps=0 \
         aspnetcore-runtime-6.0 \
         ca-certificates \
+        cargo \
         curl \
+        gcc \
+        gcc-c++ \
         git-core \
         git-lfs \
         gnupg \
@@ -18,6 +21,11 @@ RUN microdnf install -y --refresh --nodocs --setopt=install_weak_deps=0 \
         iputils \
         jq \
         make \
+        openssl-devel \
+        pkg-config \
+        python39 \
+        python39-devel \
+        python39-pip \
         sudo \
         tar \
         unzip \
@@ -28,13 +36,10 @@ RUN microdnf install -y --refresh --nodocs --setopt=install_weak_deps=0 \
     && microdnf clean all \
     && rm -rf /var/cache/yum
 
-# Install Azure CLI with RPM, then verify installation
-RUN rpm --import https://packages.microsoft.com/keys/microsoft.asc \
-    && curl -LsSf https://packages.microsoft.com/config/rhel/8/prod.repo > /etc/yum.repos.d/azure-cli.repo \
-    && microdnf install -y --refresh --nodocs --setopt=install_weak_deps=0 \
-        azure-cli \
-    && microdnf clean all \
-    && rm -rf /var/cache/yum \
+# Install Azure CLI, then verify installation
+RUN python3 -m pip install --upgrade pip setuptools wheel \
+    && python3 -m pip install azure-cli \
+    && rm -rf $HOME/.cache/pip \
     && az --version
 
 # Copy helper script, then verify installation

diff --git a/src/docker/arch.sh b/src/docker/arch.sh
@@ -2,7 +2,6 @@
 set -e
 
 ARCH_AMD64="${ARCH_AMD64:-x64}"
-ARCH_ARM32="${ARCH_ARM32:-arm}"
 ARCH_ARM64="${ARCH_ARM64:-arm64}"
 
 SYS_ARCH="$(arch)"
@@ -11,8 +10,6 @@ if [[ $SYS_ARCH == x86_64 ]]; then
   SYS_ARCH=$ARCH_AMD64
 elif [[ $SYS_ARCH == aarch64 || $SYS_ARCH == arm64 ]]; then
   SYS_ARCH=$ARCH_ARM64
-elif [[ $SYS_ARCH == arm* ]]; then
-  SYS_ARCH=$ARCH_ARM32
 else
   echo 1>&2 "Unsupported architecture $SYS_ARCH"
   exit 1