Prerequisites

Install Omni using this guide.

Create the machine classes in Omni. The machine classes in machine-class.yaml define a controller as a node with 8GB of memory and a worker as a node with 16GB or more of memory. Worker nodes are split up into two categories, NVMe and SDB. These two categories are manually applied with labels either via the installation media or by adding a label to the machine in the Omni UI. The machine class is used within the template.yaml to create countpoints for use later by Longhorn.

omnictl apply -f machine-class.yaml

Install brew

• amd64

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

• arm64 (unsupported)

sudo apt-get install build-essential procps curl file git
mkdir homebrew && curl -L https://github.com/Homebrew/brew/tarball/master | tar xz --strip-components 1 -C homebrew
echo 'eval "$(homebrew/bin/brew shellenv)"' >> ~/.bashrc
echo 'export PATH="$HOME/homebrew/bin:$PATH"' >> ~/.bashrc
brew update --force --quiet
chmod -R go-w "$(brew --prefix)/share/zsh"

Install talosctl & omnictl sidero & (use-kubectl-with-omni)

brew install siderolabs/tap/talosctl
brew install siderolabs/tap/omnictl
brew install int128/kubelogin/kubelogin

Download talosconfig & omniconfig from the Omni UI and confirm set up is working

talosctl --talosconfig talosconfig.yaml --nodes \<Node IP Address> dashboard
omnictl --omniconfig omniconfig.yaml get clusters

Install wslu - Utilities for Windows Subsystem for Linux

sudo apt install gnupg2 apt-transport-https
wget -O - https://pkg.wslutiliti.es/public.key | sudo gpg -o /usr/share/keyrings/wslu-archive-keyring.pgp --dearmor
echo "deb [signed-by=/usr/share/keyrings/wslu-archive-keyring.pgp] https://pkg.wslutiliti.es/debian \
$(. /etc/os-release && echo "$VERSION_CODENAME") main" | sudo tee /etc/apt/sources.list.d/wslu.list
sudo apt update
sudo apt install wslu
echo 'export BROWSER=wslview' >> ~/.bashrc

Install ArgoCD Cli Option 1: brew

brew install argocd

Option 2:

• amd64

curl -sSL -o argocd-linux-arm64 https://github.com/argoproj/argo-cd/releases/latest/download/argocd-linux-amd64
sudo install -m 555 argocd-linux-amd64 /usr/local/bin/argocd
rm argocd-linux-amd64

• arm64

curl -sSL -o argocd-linux-arm64 https://github.com/argoproj/argo-cd/releases/latest/download/argocd-linux-arm64
sudo install -m 555 argocd-linux-arm64 /usr/local/bin/argocd
rm argocd-linux-arm64

Install additional command line utilities

brew install kustomize
brew install yq
brew install sops
brew install helm

Components

• ksops - A sops implementation using Kustomize and ArgoCD deployed using a patch. Secrets are encrypted locally using Age and commited to the git repo. ArgoCD uses the private key stored in the cluster to decrypt the secrets and create the Kubernetes secrets.
• Cilium - CNI, LB, KubeProxy replacement using either L2 announcements or BGP as configured with use_cilium_bgp in setup.sh
• Istio - Gateway API & service mesh enabled (Cilium can be used for gateway API as well however, it currently lacks support for the TCPRoute which is used for accessing Postgres outside of the cluster).
• Kiali - Istio Service Mesh visualization
• Longhorn - CSI for distributed node storage
• Cert-Manager - Management of certificates used with the gateway API
• ArgoCD - Bootstrap and continuously deploy apps from git
• CloudNativePG - Postres operator
• Kubernetes Dashboard
• node-identifier - App for testing the configuration and load balancing. It returns the node and pod name (on which the request was handled) formatted as JSON.

Folder structure (also see here)

📁 _initial-setup    # All manifests used by the cluster through apps and patches
├──📁 config         # Manifests to be configured before copying to the deployment folder
├──📁 patches        # Manifests in full form to be configured and transformed into a Talos patch
📁 deployment        # Directory watched by ArgoCD to deploy applications
│  └──📁 apps        # Application manifests specific to the ArgoCD project "apps"
📁 patches           # Patches applied though Omni to the cluster

Configuration & cluster setup

Create the Age key for use with ksops (Note: replace public key in .sops.yaml)

age-keygen -o age.agekey
cp age.agekey ~/.config/sops/age/keys.txt

Update the values within the manifests in the config folder and setup.sh, run setup.sh and commit the files updated in the deployment folder to git.

Setup.sh will handle copying manifests from the config folder, encrypting secrets (secret.enc.yaml) and changing . The deployment folder will be watched by ArgoCD setup within the bootstrap manifest.

chmod u+x _initial-setup/setup.sh
./_initial-setup/setup.sh

Create the cluster

omnictl cluster template sync --file template.yaml

Create a secret in Kubernetes with the Age private key once node have passed the booting state

cat ~/.config/sops/age/keys.txt | kubectl --kubeconfig kubeconfig.yaml create secret generic sops-age --namespace=argocd --from-file=keys.txt=/dev/stdin

Get the initial admin password for ArgoCD

argocd --kubeconfig ./kubeconfig.yaml admin initial-password -n argocd

Get token for Kubernetes Dashboard login

kubectl  --kubeconfig kubeconfig.yaml get secret admin-user -n kubernetes-dashboard -o jsonpath="{.data.token}" | base64 -d