Merge branch 'develop' into main

README.md

@@ -195,7 +195,7 @@ These methods can be used to build a container image, at the time of writing:
 | [img](https://github.com/genuinetools/img#running-with-kubernetes), [BuildKit](https://github.com/moby/buildkit) | 🟩🟩🟩 | 🟩🟩🟥 | 🟩🟥🟥 | Local CLI | CLI to build the images. Can build different architectures on a single machine. Requires [Seccomp](https://en.wikipedia.org/wiki/Seccomp) disabled and [AppArmor](https://apparmor.net) disabled. |
 | Docker in docker | 🟩🟩🟩 | 🟥🟥🟥 | 🟩🟩🟩 | Local CLI | Before Kubernetes 1.20, it was possible to build container images in the agent, using the Docker socket. This is not possible anymore, as Kubernetes [deprecated the Docker socket](https://kubernetes.io/blog/2020/12/02/dont-panic-kubernetes-and-docker) in favor of the [Container Runtime Interface](https://kubernetes.io/blog/2016/12/container-runtime-interface-cri-in-kubernetes). |
 
-We choose [BuildKit](https://github.com/moby/buildkit) for this project. [Its licence](https://raw.githubusercontent.com/moby/buildkit/v0.11.5/LICENSE) allows commercial use, and the project and mainly maintained, as the time of writing, by Docker, Netlix and Microsoft.
+We choose BuildKit for this project. [Its licence](https://raw.githubusercontent.com/moby/buildkit/v0.11.5/LICENSE) allows commercial use, and the project and mainly maintained, as the time of writing, by Docker, Netlix and Microsoft.
 
 #### How to use the bundled BuildKit
 
@@ -228,28 +228,31 @@ steps:
 
 Out of the box, argument `--opt platform=linux/amd64,linux/arm64` can be added to build an image compatible with multiple architectures ([more can be specified](https://github.com/moby/buildkit/blob/v0.11.5/docs/multi-platform.md)). Multiple cache strategies [are available](https://github.com/moby/buildkit/tree/v0.11.5#cache) (including container registry, Azure Storage Blob, AWS S3).
 
+#### BuildKit and the performance
+
+BuildKit works by virtualization in the user space. You can't expect build times as short as native (on your laptop for example). [QEMU](https://www.qemu.org) is used as a backend. This has the advantage of being able to create images for different architectures than your processor. Virtualization-wise, not all CPU models are equivalent, you can [refer to the official project documentation](https://www.qemu.org/docs/master/system/qemu-cpu-models.html) to select the most appropriated CPU model for your Kubernetes Node Pool.
+
 ### Helm values
 
 | Parameter | Description | Default |
 |-|-|-|
-| `additionalEnv` | Additional environment variables for the agent container. | `null` |
-| `affinity` | Node affinity for pod assignment | `null` |
-| `annotations` | Add custom annotations to the Pod. | `null` |
+| `additionalEnv` | Additional environment variables for the agent container. | `[]` |
+| `affinity` | Node affinity for pod assignment | `{}` |
+| `annotations` | Add custom annotations to the Pod. | `{}` |
 | `autoscaling.cooldown` | Time in seconds the automation will wait until there is no more pipeline asking for an agent. Same time is then applied for system termination. | `60` |
-| `autoscaling.enabled` | Enable the auto-scaling. Requires [KEDA](https://keda.sh), but can be started without. | `true` |
+| `autoscaling.enabled` | Enable the auto-scaling. Requires [KEDA](https://keda.sh), but can be started without. Be warning, disabling auto-scaling implies a shutdown of the existing agents during a Helm instance upgrade, according to `pipelines.timeout`. | `true` |
 | `autoscaling.maxReplicas` | Maximum number of pods, remaining jobs will be kept in queue. | `100` |
-| `autoscaling.minReplicas` | Minimum number of pods. If autoscaling not enabled, the number of replicas to run. If `pipelines.capabilities` is defined, cannot be set to `0`. | `1` |
-| `extraVolumeMounts` | Additional volume mounts for the agent container. | `null` |
-| `extraVolumes` | Additional volumes for the agent pod. | `null` |
-| `fullnameOverride` | Overrides release fullname | `null` |
-| `image.flavor` | Container image tag | `bullseye` |
+| `extraVolumeMounts` | Additional volume mounts for the agent container. | `[]` |
+| `extraVolumes` | Additional volumes for the agent pod. | `[]` |
+| `fullnameOverride` | Overrides release fullname | `""` |
+| `image.flavor` | Container image tag, can be `bullseye`, `focal`, `jammy`, or `ubi8`. | `bullseye` |
 | `image.pullPolicy` | Container image pull policy | `IfNotPresent` |
 | `image.repository` | Container image repository | `ghcr.io/clemlesne/azure-pipelines-agent:bullseye` |
 | `image.version` | Container image tag | *Version* |
-| `imagePullSecrets` | Use secrets to pull the container image. | `null` |
-| `initContainers` | InitContainers for the agent pod. | `null` |
-| `nameOverride` | Overrides release name | `null` |
-| `nodeSelector` | Node labels for pod assignment | `null` |
+| `imagePullSecrets` | Use secrets to pull the container image. | `[]` |
+| `initContainers` | InitContainers for the agent pod. | `[]` |
+| `nameOverride` | Overrides release name | `""` |
+| `nodeSelector` | Node labels for pod assignment | `{}` |
 | `pipelines.cacheSize` | Total cache to attach to the Azure Pipelines standard directory. By default, [same amount as the Microsoft Hosted agents](https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/hosted?view=azure-devops&tabs=yaml#hardware). | `10Gi` |
 | `pipelines.cacheType` | Disk type to attach to the Azure Pipelines standard directory. See your cloud provider for types ([Azure](https://learn.microsoft.com/en-us/azure/aks/concepts-storage#storage-classes), [AWS](https://docs.aws.amazon.com/eks/latest/userguide/storage-classes.html)). | `managed-csi` (Azure compatible) |
 | `pipelines.capabilities` | Add [demands/capabilities](https://learn.microsoft.com/en-us/azure/devops/pipelines/process/demands?view=azure-devops&tabs=yaml) to the agent | `[]` |
@@ -259,12 +262,13 @@ Out of the box, argument `--opt platform=linux/amd64,linux/arm64` can be added t
 | `pipelines.tmpdirSize` | Total size of the [standard `TMPDIR` directory](https://en.wikipedia.org/wiki/TMPDIR).  | `1Gi` |
 | `pipelines.tmpdirType` | Disk type to attach to the [standard `TMPDIR` directory](https://en.wikipedia.org/wiki/TMPDIR). See your cloud provider for types ([Azure](https://learn.microsoft.com/en-us/azure/aks/concepts-storage#storage-classes), [AWS](https://docs.aws.amazon.com/eks/latest/userguide/storage-classes.html)). | `managed-csi` (Azure compatible) |
 | `pipelines.url` | The Azure base URL for your organization | *None* |
-| `podSecurityContext` | Security rules applied to the Pod ([more details](https://kubernetes.io/docs/concepts/security/pod-security-standards)). | `null` |
+| `podSecurityContext` | Security rules applied to the Pod ([more details](https://kubernetes.io/docs/concepts/security/pod-security-standards)). | `{}` |
+| `replicaCount` | Default fixed amount of agents deployed. Those are not auto-scaled. | `3` |
 | `resources` | Resource limits | `{ "resources": { "limits": { "cpu": 2, "memory": "4Gi" }, "requests": { "cpu": 1, "memory": "2Gi" } }}` |
-| `securityContext` | Security rules applied to the container ([more details](https://kubernetes.io/docs/concepts/security/pod-security-standards)). | `null` |
+| `securityContext` | Security rules applied to the container ([more details](https://kubernetes.io/docs/concepts/security/pod-security-standards)). | `{}` |
 | `serviceAccount.create` | Create ServiceAccount | `true` |
 | `serviceAccount.name` | ServiceAccount name | *Release name* |
-| `tolerations` | Toleration labels for pod assignment. | `null` |
+| `tolerations` | Toleration labels for pod assignment. | `[]` |
 
 ## [Security](./SECURITY.md)
 

src/docker/Dockerfile-bullseye

@@ -32,7 +32,6 @@ RUN --mount=target=/var/lib/apt/lists,type=cache,sharing=locked --mount=target=/
         jq \
         libffi-dev \
         libssl-dev \
-        libssl1.1 \
         lsb-release \
         make \
         pkg-config \

src/docker/Dockerfile-focal

@@ -31,7 +31,6 @@ RUN --mount=target=/var/lib/apt/lists,type=cache,sharing=locked --mount=target=/
         jq \
         libffi-dev \
         libssl-dev \
-        libssl1.1 \
         lsb-release \
         make \
         pkg-config \

src/docker/Dockerfile-jammy

@@ -31,7 +31,6 @@ RUN --mount=target=/var/lib/apt/lists,type=cache,sharing=locked --mount=target=/
         jq \
         libffi-dev \
         libssl-dev \
-        libssl3 \
         lsb-release \
         make \
         pkg-config \
@@ -75,6 +74,13 @@ RUN git clone --depth 1 --branch v${ROOTLESSKIT_VERSION} https://github.com/root
 
 FROM base
 
+# Azure CLI doesn't work with OpenSSL 3.0 (https://github.com/Azure/azure-cli/issues/22230)
+COPY jammy-misc/libssl* .
+RUN dpkg -i libssl1.1_1.1.1f-1ubuntu2_$(ARCH_X64=amd64 bash arch.sh).deb \
+    && cat /etc/ssl/openssl.cnf \
+    && rm -rf libssl* \
+    # https://github.com/microsoft/azure-pipelines-agent/issues/3834#issuecomment-1164461892
+    && sed -i 's/openssl_conf = openssl_init/#openssl_conf = openssl_init/g' /etc/ssl/openssl.cnf
 # Install Azure CLI, then verify installation
 ARG AZURE_CLI_VERSION
 ENV AZURE_CLI_VERSION ${AZURE_CLI_VERSION}

src/docker/Dockerfile-ubi8

@@ -11,6 +11,7 @@ ENV HOME /home/${USER}
 # - zsh, for inter-operability
 # - fuse-overlayfs, iptables, shadow-utils, for BuildKit
 # - gzip, make, tar, unzip, wget, yq, zip, zstd for developer ease-of-life
+# - compat-openssl10, openssl, Azure CLI doesn't work with OpenSSL 3.0 (https://github.com/Azure/azure-cli/issues/22230)
 RUN --mount=target=/var/cache/yum,type=cache,sharing=locked \
     microdnf install -y --refresh --nodocs --setopt=install_weak_deps=0 \
         aspnetcore-runtime-6.0 \
@@ -31,7 +32,7 @@ RUN --mount=target=/var/cache/yum,type=cache,sharing=locked \
         jq \
         make \
         openssl-devel \
-        openssl-libs \
+        openssl \
         pkg-config \
         python39 \
         python39-devel \

src/helm/azure-pipelines-agent/templates/_helpers.tpl

@@ -60,3 +60,102 @@ Create the name of the service account to use
 {{- default "default" .Values.serviceAccount.name }}
 {{- end }}
 {{- end }}
+
+{{- define "this.podSharedTemplate" -}}
+{{- with .Values.imagePullSecrets }}
+imagePullSecrets:
+  {{- toYaml . | nindent 2 }}
+{{- end }}
+serviceAccountName: {{ include "this.serviceAccountName" . }}
+{{- with .Values.podSecurityContext }}
+securityContext:
+  {{- toYaml . | nindent 2 }}
+{{- end }}
+{{- with .Values.initContainers }}
+initContainers:
+  {{- toYaml . | nindent 2 }}
+{{- end }}
+terminationGracePeriodSeconds: {{ .Values.pipelines.timeout | int | required "A value for .Values.pipelines.timeout is required" }}
+containers:
+  - name: azp-agent
+    securityContext:
+      runAsUser: 0
+      {{- with .Values.securityContext }}
+      {{- toYaml . | nindent 6 }}
+      {{- end }}
+    image: "{{ .Values.image.repository | required "A value for .Values.image.repository is required" }}:{{ .Values.image.flavor | required "A value for .Values.image.flavor is required" }}-{{ default .Chart.Version .Values.image.version }}"
+    imagePullPolicy: {{ .Values.image.pullPolicy }}
+    lifecycle:
+      preStop:
+        exec:
+          command: [bash, -c, "bash ${AZP_HOME}/config.sh remove --auth PAT --token $AZP_TOKEN"]
+    env:
+      - name: VSO_AGENT_IGNORE
+        value: AZP_TOKEN
+      - name: AZP_AGENT_NAME
+        value: {{ include "this.fullname" . }}-template
+      - name: AZP_URL
+        valueFrom:
+          secretKeyRef:
+            name: {{ include "this.fullname" . }}
+            key: url
+      - name: AZP_POOL
+        value: {{ .Values.pipelines.pool | quote | required "A value for .Values.pipelines.pool is required" }}
+      - name: AZP_TOKEN
+        valueFrom:
+          secretKeyRef:
+            name: {{ include "this.fullname" . }}
+            key: pat
+      # Agent capabilities
+      - name: flavor_{{ .Values.image.flavor | required "A value for .Values.image.flavor is required" }}
+      {{- range .Values.pipelines.capabilities }}
+      - name: {{ . }}
+      {{- end }}
+      {{- with .Values.additionalEnv }}
+      {{- toYaml . | nindent 6 }}
+      {{- end }}
+    resources:
+      {{- toYaml .Values.resources | nindent 6 | required "A value for .Values.resources is required" }}
+    volumeMounts:
+      - name: azp-work
+        mountPath: /home/root/azp-work
+      - name: local-tmp
+        mountPath: /home/root/.local/tmp
+      {{- with .Values.extraVolumeMounts }}
+      {{- toYaml . | nindent 6 }}
+      {{- end }}
+volumes:
+  - name: azp-work
+    ephemeral:
+      volumeClaimTemplate:
+        spec:
+          accessModes: [ "ReadWriteOnce" ]
+          storageClassName: {{ .Values.pipelines.cacheType | required "A value for .Values.pipelines.cacheType is required" }}
+          resources:
+            requests:
+              storage: {{ .Values.pipelines.cacheSize | required "A value for .Values.pipelines.cacheSize is required" }}
+  - name: local-tmp
+    ephemeral:
+      volumeClaimTemplate:
+        spec:
+          accessModes: [ "ReadWriteOnce" ]
+          storageClassName: {{ .Values.pipelines.tmpdirType | required "A value for .Values.pipelines.tmpdirType is required" }}
+          resources:
+            requests:
+              storage: {{ .Values.pipelines.tmpdirSize | required "A value for .Values.pipelines.tmpdirSize is required" }}
+  {{- with .Values.extraVolumes }}
+  {{- toYaml . | nindent 2 }}
+  {{- end }}
+{{- with .Values.nodeSelector }}
+nodeSelector:
+  {{- toYaml . | nindent 2 }}
+{{- end }}
+{{- with .Values.affinity }}
+affinity:
+  {{- toYaml . | nindent 2 }}
+{{- end }}
+{{- with .Values.tolerations }}
+tolerations:
+  {{- toYaml . | nindent 2 }}
+{{- end }}
+{{- end }}

src/helm/azure-pipelines-agent/templates/deployment.yaml

@@ -0,0 +1,27 @@
+{{- if not (and (.Values.autoscaling.enabled) (.Capabilities.APIVersions.Has "keda.sh/v1alpha1")) }}
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: {{ include "this.fullname" . }}-{{ .Release.Revision }}
+  labels:
+    {{- include "this.labels" . | nindent 4 }}
+spec:
+  selector:
+    matchLabels:
+      {{- include "this.selectorLabels" . | nindent 6 }}
+  replicas: {{ .Values.replicaCount | int | required "A value for .Values.replicaCount is required" }}
+  template:
+    metadata:
+      labels:
+        {{- include "this.selectorLabels" . | nindent 8 }}
+      annotations:
+        # Cluster autoscaler never evicts this Pod
+        cluster-autoscaler.kubernetes.io/safe-to-evict: "false"
+        {{- with .Values.annotations }}
+        {{- toYaml . | nindent 8 }}
+        {{- end }}
+    spec:
+      # The pod will shut down at each run, we need to restart it
+      restartPolicy: Always
+      {{- include "this.podSharedTemplate" . | nindent 6 }}
+{{- end }}

src/helm/azure-pipelines-agent/templates/hpa.yaml

@@ -35,106 +35,10 @@ spec:
           {{- toYaml . | nindent 10 }}
           {{- end }}
       spec:
-        {{- with .Values.imagePullSecrets }}
-        imagePullSecrets:
-          {{- toYaml . | nindent 10 }}
-        {{- end }}
-        serviceAccountName: {{ include "this.serviceAccountName" . }}
-        {{- with .Values.podSecurityContext }}
-        securityContext:
-          {{- toYaml . | nindent 10 }}
-        {{- end }}
-        {{- with .Values.initContainers }}
-        initContainers:
-          {{- toYaml . | nindent 10 }}
-        {{- end }}
-        terminationGracePeriodSeconds: {{ .Values.pipelines.timeout | int | required "A value for .Values.pipelines.timeout is required" }}
-        containers:
-          - name: azp-agent
-            securityContext:
-              runAsUser: 0
-              {{- with .Values.securityContext }}
-              {{- toYaml . | nindent 14 }}
-              {{- end }}
-            image: "{{ .Values.image.repository | required "A value for .Values.image.repository is required" }}:{{ .Values.image.flavor | required "A value for .Values.image.flavor is required" }}-{{ default .Chart.Version .Values.image.version }}"
-            imagePullPolicy: {{ .Values.image.pullPolicy }}
-            lifecycle:
-              preStop:
-                exec:
-                  command: [bash, -c, "bash ${AZP_HOME}/config.sh remove --auth PAT --token $AZP_TOKEN"]
-            env:
-              - name: VSO_AGENT_IGNORE
-                value: AZP_TOKEN
-              - name: AZP_AGENT_NAME
-                valueFrom:
-                  fieldRef:
-                    apiVersion: v1
-                    fieldPath: metadata.name
-              - name: AZP_URL
-                valueFrom:
-                  secretKeyRef:
-                    name: {{ include "this.fullname" . }}
-                    key: url
-              - name: AZP_POOL
-                value: {{ .Values.pipelines.pool | quote | required "A value for .Values.pipelines.pool is required" }}
-              - name: AZP_TOKEN
-                valueFrom:
-                  secretKeyRef:
-                    name: {{ include "this.fullname" . }}
-                    key: pat
-              # Agent capabilities
-              - name: flavor_{{ .Values.image.flavor | required "A value for .Values.image.flavor is required" }}
-              {{- range .Values.pipelines.capabilities }}
-              - name: {{ . }}
-              {{- end }}
-              {{- with .Values.additionalEnv }}
-              {{- toYaml . | nindent 14 }}
-              {{- end }}
-            resources:
-              {{- toYaml .Values.resources | nindent 14 | required "A value for .Values.resources is required" }}
-            volumeMounts:
-              - name: azp-work
-                mountPath: /home/root/azp-work
-              - name: local-tmp
-                mountPath: /home/root/.local/tmp
-              {{- with .Values.extraVolumeMounts }}
-              {{- toYaml . | nindent 14 }}
-              {{- end }}
-        volumes:
-          - name: azp-work
-            ephemeral:
-              volumeClaimTemplate:
-                spec:
-                  accessModes: [ "ReadWriteOnce" ]
-                  storageClassName: {{ .Values.pipelines.cacheType | required "A value for .Values.pipelines.cacheType is required" }}
-                  resources:
-                    requests:
-                      storage: {{ .Values.pipelines.cacheSize | required "A value for .Values.pipelines.cacheSize is required" }}
-          - name: local-tmp
-            ephemeral:
-              volumeClaimTemplate:
-                spec:
-                  accessModes: [ "ReadWriteOnce" ]
-                  storageClassName: {{ .Values.pipelines.tmpdirType | required "A value for .Values.pipelines.tmpdirType is required" }}
-                  resources:
-                    requests:
-                      storage: {{ .Values.pipelines.tmpdirSize | required "A value for .Values.pipelines.tmpdirSize is required" }}
-          {{- with .Values.extraVolumes }}
-          {{- toYaml . | nindent 10 }}
-          {{- end }}
-        {{- with .Values.nodeSelector }}
-        nodeSelector:
-          {{- toYaml . | nindent 10 }}
-        {{- end }}
-        {{- with .Values.affinity }}
-        affinity:
-          {{- toYaml . | nindent 10 }}
-        {{- end }}
-        {{- with .Values.tolerations }}
-        tolerations:
-          {{- toYaml . | nindent 10 }}
-        {{- end }}
-  maxReplicaCount: {{ sub (.Values.autoscaling.maxReplicas | int | required "A value for .Values.autoscaling.maxReplicas is required") (.Values.autoscaling.minReplicas | int | required "A value for .Values.autoscaling.minReplicas is required") }}
+        # The job cannot crash, this is not functionally intended; a shutdown is functionally a pipeline end
+        restartPolicy: Never
+        {{- include "this.podSharedTemplate" . | nindent 8 }}
+  maxReplicaCount: {{ .Values.autoscaling.maxReplicas | int | required "A value for .Values.autoscaling.maxReplicas is required" }}
   minReplicaCount: 0
   pollingInterval: 15
   rollout: