Runner Scale Set takes long time to create pods for pending jobs

[omri-shilton]

Hello,
runner-scale-set chart version 0.6.1
controller version 0.6.1
kubernetes mode

when a job starts, lets say with 10 jobs simultaneously, it starts to create pods for each job, but it takes more than 1-2 minutes each pod. and its doing it one by one. this makes each workflow take a lot of time.
I would assume it would try to start all the pods at the same time.

When im looking at AutoscalingRunnerSet and EphemeralRunnerSet for my runner i see a lot of pending runners, and very few running runners

job_startup_duration_seconds metric is on average 6 minutes

on the github side it looks like "waiting for runner to come online"

Edit:
After some research it appears that i takes a very long time for the ephemeralrunner pods to be created.
Anyone has any idea why could that be happening?

Am i missing something?

I'm using GHA-Runner-Scale-Set and have this value file:

minRunners: 0
maxRunners: 15
runnerScaleSetName: custom-runner-2
controllerServiceAccount:
  namespace: gha-runner-scale-set
  name: gha-runner-scale-set-controller-gha-rs-controller
containerMode:
  type: "kubernetes"
  kubernetesModeWorkVolumeClaim:
    accessModes: ["ReadWriteOnce"]
    storageClassName: "gp2-encrypted"
    resources:
      requests:
        storage: 10Gi
template:
  metadata:
    annotations:
      karpenter.sh/do-not-disrupt: "true"
  spec:
    nodeSelector:
      type: arc
    tolerations:
      - key: "type"
        operator: "Equal"
        value: "arc"
        effect: "NoSchedule"
    initContainers:
      - name: kube-init
        image: ghcr.io/actions/actions-runner:latest
        command: ["/bin/sh", "-c"]
        args:
          - |
            sudo chown -R 1001:123 /home/runner/_work
        volumeMounts:
          - name: work
            mountPath: /home/runner/_work
    containers:
    - name: runner
      image: ghcr.io/actions/actions-runner:latest
      command: ["/home/runner/run.sh"]
      resources:
        requests:
          cpu: 2000m
      env:
        - name: ACTIONS_RUNNER_CONTAINER_HOOKS
          value: /home/runner/k8s/index.js
        - name: ACTIONS_RUNNER_POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: ACTIONS_RUNNER_REQUIRE_JOB_CONTAINER
          value: "false"

[nixlim]

At the moment, we have the same problem - a bit worse than yours though. It takes 14 mins for the pod to be created. Looking into it but have no ideas at the moment.

[omri-shilton]

I ended up upgrading all the eks addons (vpc,ebs,kubeproxy,coredns) and upgrading the cluster to 1.28 and the issue was solved.

[nixlim]

We run on AKS, cluster at 1.29.2. But great to know that yours was solved. Any debugging suggestions (or basically where to look for to see what is taking time)?

[omri-shilton]

My gut feeling was that it had something to do with networking. And an overall issue with the cluster rather than with the controller