Skip to content

Commit

Permalink
Merge branch 'develop' into main
Browse files Browse the repository at this point in the history
  • Loading branch information
clemlesne committed Apr 12, 2023
2 parents 6c06c16 + cd5bf21 commit 8a17577
Show file tree
Hide file tree
Showing 19 changed files with 713 additions and 193 deletions.
12 changes: 12 additions & 0 deletions .github/workflows/pipeline.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,14 @@ env:
TINI_VERSION: 0.19.0
# https://github.com/mikefarah/yq/releases
YQ_VERSION: 4.33.2
# https://go.dev/dl
GO_VERSION: 1.20.3
# https://github.com/rootless-containers/rootlesskit/releases
ROOTLESSKIT_VERSION: 1.1.0
# https://github.com/moby/buildkit/releases
BUILDKIT_VERSION: 0.11.5
# https://github.com/shadow-maint/shadow/releases
SHADOW_VERSION: 4.12.3

jobs:
build-helm:
Expand Down Expand Up @@ -187,7 +195,11 @@ jobs:
with:
build-args: |
"AGENT_VERSION=${{ env.AGENT_VERSION }}"
"BUILDKIT_VERSION=${{ env.BUILDKIT_VERSION }}"
"GO_VERSION=${{ env.GO_VERSION }}"
"POWERSHELL_VERSION=${{ env.POWERSHELL_VERSION }}"
"ROOTLESSKIT_VERSION=${{ env.ROOTLESSKIT_VERSION }}"
"SHADOW_VERSION=${{ env.SHADOW_VERSION }}"
"TINI_VERSION=${{ env.TINI_VERSION }}"
"YQ_VERSION=${{ env.YQ_VERSION }}"
cache-from: type=gha
Expand Down
158 changes: 136 additions & 22 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,18 +16,22 @@

Features:

- Agent register itself with the Azure DevOps server.
- Agent restart itself if it crashes.
- Auto-scale based on Pipeline usage (with [KEDA](https://keda.sh), not required).
- Agent register and restart itself.
- Allow to build containers inside the agent using [BuildKit](https://github.com/moby/buildkit).
- Can run air-gapped (no internet access).
- Cheap to run (dynamic provisioning of agents, can scale from 0 to 100+ in few seconds).
- Compatible with Debian, Ubuntu and Red Hat LTS releases.
- Cheap to run (dynamic provisioning of agents, can scale from 0 to 100+ in few seconds with [KEDA](https://keda.sh)).
- Performances can be customized depending of the engineering needs, which goes far beyond the Microsoft-hosted agent.
- Pre-built with Debian, Ubuntu and Red Hat Enterprise Linux releases.
- SBOM (Software Bill of Materials) is packaged with each container image.
- System updates are applied every days.
- Systems are based on [Microsoft official .NET images](https://mcr.microsoft.com/en-us/product/dotnet/aspnet/about) and [Red Hat Universal Base Image](https://catalog.redhat.com/software/containers/ubi8/ubi-minimal/5c359a62bed8bd75a2c3fba8).

## Usage

Deployment steps:

1. [Prepare the token for allowing access from the Agent to Azure DevOps.](https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/v2-linux?view=azure-devops#permissions)
2. Deployment in Kubernetes using Helm

### Deployment in Kubernetes using Helm

Minimal configuration:
Expand Down Expand Up @@ -64,6 +68,7 @@ helm upgrade --install agent clemlesne-azure-pipelines-agent/azure-pipelines-age
- [ASP.NET Core](https://github.com/dotnet/aspnetcore) runtime (required by the Azure Pipelines agent)
- [Azure CLI](https://github.com/Azure/azure-cli) (required by the Azure Pipelines agent) + requirements ([Python 3.8](https://www.python.org/downloads/release/python-380), [Python 3.9](https://www.python.org/downloads/release/python-390), [Python 3.10](https://www.python.org/downloads/release/python-3100), depending of the system, plus C/Rust build tools for libs non pre-built on the platforms)
- [Powershell](https://github.com/PowerShell/PowerShell), [bash](https://www.gnu.org/software/bash) and [zsh](https://www.zsh.org) (for inter-operability)
- [BuildKit](https://github.com/moby/buildkit) + requirements ([dbus-user-session](https://dbus.freedesktop.org), [fuse-overlayfs](https://github.com/containers/fuse-overlayfs), [iptables](https://www.netfilter.org/projects/iptables/index.html), [shadow-utils](https://github.com/shadow-maint/shadow), [uidmap](https://github.com/shadow-maint/shadow))
- [gzip](https://www.gnu.org/software/gzip), [make](https://www.gnu.org/software/make), [tar](https://www.gnu.org/software/tar), [unzip](https://infozip.sourceforge.net/UnZip.html), [wget](https://www.gnu.org/software/wget), [yq](https://github.com/mikefarah/yq), [zip](https://infozip.sourceforge.net/Zip.html), [zstd](https://github.com/facebook/zstd) (for developer ease-of-life)

### Capabilities
Expand All @@ -85,7 +90,7 @@ Take the assumption we want to host a specific instance pool to ARM servers.
```yaml
# values.yaml
pipelines:
pool: onprem_kubernetes
pool: private_kube
capabilities:
- arch_arm64

Expand All @@ -111,46 +116,155 @@ Update the Azure Pipelines file in the repository to use the new pool:
```yaml
# azure-pipelines.yaml
pool:
name: onprem_kubernetes
name: private_kube
demands:
- Agent.OS -equals Linux
- arch_arm64

stages:
...
```
#### Example: Use different agents on specific jobs
In that example:
- We are using a default agent on ARM64
- Semgrep, our SAST tool, is not compatible with ARM64, let's use X64 pool
- Our devs are working on a Java project, built with GraalVM, and the container is built locally with [img](https://github.com/genuinetools/img#running-with-kubernetes): we need a system lot of RAM and multipe CPUs for building the application
Our problematic:
- Is it possible to reconcile the efficiency of these different architectures, without restricting ourselves?
- Do we necessarily have to install high performance agents when the use of these large constructions is only a small part of the total execution time (tests, deployments, monitoring, rollback, external services, ...)?
We decide to dpeloy these agents:
| Details | Efficiency (cost, perf, energy) | Capabilities |
|-|-|-|
| Standard performance, ARM64 | ≅ x1 | `arch_arm64`, `perf_standard` |
| Standard performance, X64 | ≅ x1.5 | `arch_x64`, `perf_standard` |
| High performance, ARM64 | ≅ x10 | `arch_x64`, `perf_high` |
| High performance, X64 | ≅ x15 | `arch_arm64`, `perf_high` |

The developer can now use:

```yaml
# azure-pipelines.yaml
pool:
name: private_kube
demands:
- arch_arm64
- perf_standard
stages:
- stage: build
jobs:
- job: sast
# Use X64 Linux agent because Semgrep is not available on ARM64
# See: https://github.com/returntocorp/semgrep/issues/2252
pool:
name: private_kube
demands:
- arch_x64
- perf_standard
- job: unit_tests
- job: container
# Use high performance agent as Java GraalVM compilation is complex
pool:
name: private_kube
demands:
- arch_x64
- perf_high
- stage: deploy
jobs:
- job: upgrade
- job: dast
- job: integration_tests
```

### Build container images in the agent

#### Introduction

These methods can be used to build a container image, at the time of writing:

| Software | Ease | Security | Perf | Run location | Description |
|-|-|-|-|-|-|
| [Azure Container Registry task](https://learn.microsoft.com/en-us/azure/container-registry/container-registry-tasks-overview#quick-task), [Google Cloud Build](https://cloud.google.com/build/docs/building/build-containers) | 🟩🟩🟥 | 🟩🟩🟩 | 🟩🟩🟩 | Managed environment | A managed service build the container image in a dedicated environment. |
| [Kaniko](https://github.com/GoogleContainerTools/kaniko#running-kaniko-in-a-kubernetes-cluster) | 🟩🟥🟥 | 🟩🟩🟩 | 🟩🟩🟥 | Self-hosted Kubernetes | A Pod is created for each build, taking care of building and pushing the container to the registry. No security drawbacks. |
| [img](https://github.com/genuinetools/img#running-with-kubernetes), [BuildKit](https://github.com/moby/buildkit) | 🟩🟩🟩 | 🟩🟩🟥 | 🟩🟥🟥 | Local CLI | CLI to build the images. Can build different architectures on a single machine. Requires [Seccomp](https://en.wikipedia.org/wiki/Seccomp) disabled and [AppArmor](https://apparmor.net) disabled. |
| Docker in docker | 🟩🟩🟩 | 🟥🟥🟥 | 🟩🟩🟩 | Local CLI | Before Kubernetes 1.20, it was possible to build container images in the agent, using the Docker socket. This is not possible anymore, as Kubernetes [deprecated the Docker socket](https://kubernetes.io/blog/2020/12/02/dont-panic-kubernetes-and-docker) in favor of the [Container Runtime Interface](https://kubernetes.io/blog/2016/12/container-runtime-interface-cri-in-kubernetes). |

We choose [BuildKit](https://github.com/moby/buildkit) for this project. [Its licence](https://raw.githubusercontent.com/moby/buildkit/v0.11.5/LICENSE) allows commercial use, and the project and mainly maintained, as the time of writing, by Docker, Netlix and Microsoft.

#### How to use the bundled BuildKit

There are two components, the backend, `buildkitd`, and the CLI, `buildctl`.

```yaml
# azure-pipelines.yaml
variables:
- name: container_name
value: my-app
- name: container_registry_domain
value: my-app-registry.azurecr.io
steps:
- bash: |
# Start buildkitd
rootlesskit buildkitd --oci-worker-no-process-sandbox --addr $BUILDKIT_HOST &
# Wait for buildkitd to start
while ! buildctl debug workers; do sleep 1; done
displayName: Run BuildKit
- bash: |
buildctl build \
--frontend dockerfile.v0 \
--local context=. \
--local dockerfile=. \
--output type=image,name=$(container_registry_domain)/$(container_name):latest,push=true
displayName: Build and push the image
```

Out of the box, argument `--opt platform=linux/amd64,linux/arm64` can be added to build an image compatible with multiple architectures ([more can be specified](https://github.com/moby/buildkit/blob/v0.11.5/docs/multi-platform.md)). Multiple cache strategies [are available](https://github.com/moby/buildkit/tree/v0.11.5#cache) (including container registry, Azure Storage Blob, AWS S3).

### Helm values

| Parameter | Description | Default |
|-|-|-|
| `additionalEnv` | Additional environment variables for the agent container. | `[]` |
| `affinity` | Node affinity for pod assignment | `{}` |
| `additionalEnv` | Additional environment variables for the agent container. | `null` |
| `affinity` | Node affinity for pod assignment | `null` |
| `annotations` | Add custom annotations to the Pod. | `null` |
| `autoscaling.cooldown` | Time in seconds the automation will wait until there is no more pipeline asking for an agent. Same time is then applied for system termination. | `60` |
| `autoscaling.enabled` | Enable the auto-scaling, requires [KEDA](https://keda.sh). | `true` |
| `autoscaling.enabled` | Enable the auto-scaling. Requires [KEDA](https://keda.sh), but can be started without. | `true` |
| `autoscaling.maxReplicas` | Maximum number of pods, remaining jobs will be kept in queue. | `100` |
| `autoscaling.minReplicas` | Minimum number of pods. If autoscaling not enabled, the number of replicas to run. If `pipelines.capabilities` is defined, cannot be set to `0`. | `1` |
| `extraVolumeMounts` | Additional volume mounts for the agent container. | `[]` |
| `extraVolumes` | Additional volumes for the agent pod. | `[]` |
| `fullnameOverride` | Overrides release fullname | `""` |
| `extraVolumeMounts` | Additional volume mounts for the agent container. | `null` |
| `extraVolumes` | Additional volumes for the agent pod. | `null` |
| `fullnameOverride` | Overrides release fullname | `null` |
| `image.flavor` | Container image tag | `bullseye` |
| `image.pullPolicy` | Container image pull policy | `IfNotPresent` |
| `image.repository` | Container image repository | `ghcr.io/clemlesne/azure-pipelines-agent:bullseye` |
| `image.version` | Container image tag | *Version* |
| `initContainers` | InitContainers for the agent pod. | `[]` |
| `nameOverride` | Overrides release name | `""` |
| `nodeSelector` | Node labels for pod assignment | `{}` |
| `pipelines.cacheSize` | Total cache the pipeline can take during execution, by default [the same amount as the Microsoft Hosted agents](https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/hosted?view=azure-devops&tabs=yaml#hardware). | `10Gi` |
| `pipelines.cacheType` | Disk type to attach to the agents, see your cloud provider for mor details ([Azure](https://learn.microsoft.com/en-us/azure/aks/concepts-storage#storage-classes), [AWS](https://docs.aws.amazon.com/eks/latest/userguide/storage-classes.html)). | `managed-csi` (Azure compatible) |
| `imagePullSecrets` | Use secrets to pull the container image. | `null` |
| `initContainers` | InitContainers for the agent pod. | `null` |
| `nameOverride` | Overrides release name | `null` |
| `nodeSelector` | Node labels for pod assignment | `null` |
| `pipelines.cacheSize` | Total cache to attach to the Azure Pipelines standard directory. By default, [same amount as the Microsoft Hosted agents](https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/hosted?view=azure-devops&tabs=yaml#hardware). | `10Gi` |
| `pipelines.cacheType` | Disk type to attach to the Azure Pipelines standard directory. See your cloud provider for types ([Azure](https://learn.microsoft.com/en-us/azure/aks/concepts-storage#storage-classes), [AWS](https://docs.aws.amazon.com/eks/latest/userguide/storage-classes.html)). | `managed-csi` (Azure compatible) |
| `pipelines.capabilities` | Add [demands/capabilities](https://learn.microsoft.com/en-us/azure/devops/pipelines/process/demands?view=azure-devops&tabs=yaml) to the agent | `[]` |
| `pipelines.pat` | Personal Access Token (PAT) used by the agent to connect. | *None* |
| `pipelines.pat` | Personal Access Token (PAT) used by the agent to connect to the Azure DevOps server (both SaaS and self-hosted). | *None* |
| `pipelines.pool` | Agent pool to which the Agent should register. | *None* |
| `pipelines.timeout` | Time in seconds after a agent will be stopped, the same amount of time is applied as a timeout for the system to shut down. | `3600` (1 hour) |
| `pipelines.tmpdirSize` | Total size of the [standard `TMPDIR` directory](https://en.wikipedia.org/wiki/TMPDIR). | `1Gi` |
| `pipelines.tmpdirType` | Disk type to attach to the [standard `TMPDIR` directory](https://en.wikipedia.org/wiki/TMPDIR). See your cloud provider for types ([Azure](https://learn.microsoft.com/en-us/azure/aks/concepts-storage#storage-classes), [AWS](https://docs.aws.amazon.com/eks/latest/userguide/storage-classes.html)). | `managed-csi` (Azure compatible) |
| `pipelines.url` | The Azure base URL for your organization | *None* |
| `podSecurityContext` | Security rules applied to the Pod ([more details](https://kubernetes.io/docs/concepts/security/pod-security-standards)). | `null` |
| `resources` | Resource limits | `{ "resources": { "limits": { "cpu": 2, "memory": "4Gi" }, "requests": { "cpu": 1, "memory": "2Gi" } }}` |
| `securityContext` | Security rules applied to the container ([more details](https://kubernetes.io/docs/concepts/security/pod-security-standards)). | `null` |
| `serviceAccount.create` | Create ServiceAccount | `true` |
| `serviceAccount.name` | ServiceAccount name | *Release name* |
| `tolerations` | Toleration labels for pod assignment. | `[]` |
| `tolerations` | Toleration labels for pod assignment. | `null` |

## [Security](./SECURITY.md)

Expand Down
3 changes: 3 additions & 0 deletions example/azure-pipelines/build-container-buildctl/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
FROM hub.docker.com/library/nginx:1.23-alpine

COPY index.html /usr/share/nginx/html/
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
# Use ARM64 Linux agents because they are cheap
pool:
name: onprem_aks
demands:
- flavor_bullseye
- nested_virtualization

trigger:
- "*"

variables:
- name: branch_sanitized
value: ${{ replace(replace(replace(variables['Build.SourceBranch'], 'refs/heads/', ''), '/', '-'), '_', '-') }}
- name: container_name
value: my-app
- name: container_registry_service_connection
value: my-app-registry
- name: container_registry_domain
value: my-app-registry.azurecr.io

stages:
- stage: build
displayName: Build
jobs:
- job: build
displayName: Build
steps:
- template: step-checkout.yaml

- template: step-setup-docker.yaml

- task: Docker@2
displayName: Login to ACR
inputs:
command: login
containerRegistry: $(container_registry_service_connection)

- bash: |
buildctl build \
--export-cache type=inline \
--frontend dockerfile.v0 \
--import-cache type=registry,ref=$(container_registry_domain)/$(container_name):$(branch_sanitized) \
--local context=. \
--local dockerfile=. \
--opt platform=linux/amd64,linux/arm64/v8 \
--output type=image,\"name=$(container_registry_domain)/$(container_name):latest,$(container_registry_domain)/$(container_name):$(branch_sanitized)\",push=true
displayName: Build the image
10 changes: 10 additions & 0 deletions example/azure-pipelines/build-container-buildctl/index.html
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
<!DOCTYPE html>
<html>
<head>
<title>Page Title</title>
</head>
<body>
<h1>Hello World!</h1>
<p>This is a super small HTML page.</p>
</body>
</html>
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
steps:
- task: KubectlInstaller@0
inputs:
# https://github.com/kubernetes/kubectl/releases
kubectlVersion: 1.26.3

- bash: |
ARCH=$(ARCH_X64=amd64 bash cicd/arch.sh)
# Install
mkdir -p "$(Agent.TempDirectory)/kubelogin"
cd "$(Agent.TempDirectory)/kubelogin"
curl -LsSf https://github.com/Azure/kubelogin/releases/download/v${KUBELOGIN_VERSION}/kubelogin-linux-${ARCH}.zip -o kubelogin.zip
unzip kubelogin.zip -d .
mkdir -p /usr/local/kubelogin/bin
mv bin/linux_${ARCH}/kubelogin /usr/local/kubelogin/bin/kubelogin
ln -s /usr/local/kubelogin/bin/kubelogin /usr/local/bin
# Test the install
kubelogin --version
displayName: Setup Kubelogin
env:
# https://github.com/Azure/kubelogin/releases
KUBELOGIN_VERSION: 0.0.28
- task: HelmInstaller@1
displayName: Setup Helm
inputs:
# https://github.com/helm/helm/releases
helmVersionToInstall: 3.11.2
23 changes: 11 additions & 12 deletions example/values-arm64.yaml → example/helm/arm64.yaml
Original file line number Diff line number Diff line change
@@ -1,17 +1,16 @@
pipelines:
pat: ojemi3pd7adocbqeeuzr5o7ldz2jlwxxbmtrrj7gq5mifhmafpsa
pool: onprem_aks
url: https://dev.azure.com/shopping-cart-devops-demo
cacheType: azurefile-csi
capabilities:
- arch_arm64

affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/arch
operator: In
values:
- arm64
- key: kubernetes.io/arch
operator: In
values:
- arm64

pipelines:
capabilities:
- arch_arm64
pat: your-pat
pool: private_kube
url: https://dev.azure.com/shopping-cart-devops-demo
29 changes: 29 additions & 0 deletions example/helm/container-build.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
annotations:
container.apparmor.security.beta.kubernetes.io/azp-agent: unconfined

extraVolumeMounts:
- mountPath: /home/root/.local/share/buildkit
name: buildkitd

extraVolumes:
- emptyDir: {}
name: buildkitd

securityContext:
seccompProfile:
type: Unconfined

resources:
limits:
cpu: 4
memory: 8Gi
requests:
cpu: 2
memory: 4Gi

pipelines:
capabilities:
- buildkit
pat: your-pat
pool: private_kube
url: https://dev.azure.com/shopping-cart-devops-demo
Loading

0 comments on commit 8a17577

Please sign in to comment.