WIP: 🌱 Use equinix runner instead of ubuntu for e2e tests

[kashifest]

No description provided.

[metal3-io-bot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please ask for approval from kashifest. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[tuminoid]

/retest

[tuminoid]

Please add proper description and link to the relevant GH doc.

It seems, e2e pipeline did not get a runner, as they are just "failed" with no run logs. Do we really want to use Equinix as they seem to be off most of the time?

[kashifest]

Please add proper description and link to the relevant GH doc.

Its a WIP so not ready for review, I will add description once we know it works.

[kashifest]

It seems, e2e pipeline did not get a runner, as they are just "failed" with no run logs. Do we really want to use Equinix as they seem to be off most of the time?

We were suggested to try them in the CNCF cluster issue here cncf/cluster#266 (comment) . I have replied back what we experienced here.

[Rozzii]

/retest

[Rozzii]

/ok-to-test

[kashifest]

/retest

[kashifest]

/retest

[tuminoid]

/retest

Still queued after 2.5 hours.

[idvoretskyi]

@kashifest we'll investigate what's going on with the runners and get back to you.

[RobertKielty]

@tuminoid @kashifest I am troubleshooting this right now, thank you for your patience.

The two pending jobs are

https://github.com/metal3-io/baremetal-operator/actions/runs/11459855378/job/31885474716?pr=1775

and

https://github.com/metal3-io/baremetal-operator/actions/runs/11459855378/job/31885475330?pr=1775

[RobertKielty]

We are still working on determining the root cause of the problem.

Since the last update, we have reviewed and checked org-level settings and restarted a cluster listener that should be picking this job up. I canceled and restarted the failed jobs a few times. We will continue to work on this and update you again in an hour.

Thanks again for your patience. Let me know if this is an immediate blocker for the project. If not we appreciate your adoption of the runner and giving us this opportunity to figure out how to fix this.

[RobertKielty]

We think that changing this line back to

runs-on: ${{ inputs.runner }}

will enable the job to be picked up by the appropriate CNCF Managed External GitHub Action runner.

[tuminoid]

It seems to have picked it up, but fails very early.

Run sudo usermod -a -G libvirt $USER
Usage: usermod [options] LOGIN
...

[kashifest]

It seems to have picked it up, but fails very early.

Run sudo usermod -a -G libvirt $USER
Usage: usermod [options] LOGIN
...

Yeah, @RobertKielty what kind of OS and which versions are these runners using? It would be good if you can point us to detailed information on these runners.

[RobertKielty]

@kashifest, you are correct that documentation is needed to describe the runners; full end-user documentation is not yet ready, as this iteration of the runners was deployed last week.

The cluster and external runners are defined in the ci directory on the cncf/automation repo.

So we can review the code that created the runners.

The equinix-4cpu-16gb runner is deployed from the following directory which contains a Helm Chart and associated values file. (Runner Helm Charts and related components are deployed using ArgoCD but that should not be relevant to this specific problem)

https://github.com/cncf/automation/tree/main/ci/cluster/equinix/runners/4cpu-16gb

The Container Image is defined here
https://github.com/cncf/automation/tree/main/ci/gha-runner-image

Looking at the Dockerfile we have

USER root
RUN apt-get update -y \
    && apt-get install -y --no-install-recommends \
    build-essential \
    npm \
    git \
    curl \
    jq \
    unzip \
    kmod

USER runner

The GHA runner image we use is based on ghcr.io/actions/actions-runner:latest from

https://github.com/actions/runner

We add typically needed packages as root and then revert to the runner user

Apologies again for not having full documentation in place; it is on our roadmap. Any and all feedback is very much appreciated!

[tuminoid]

Testing Prow, ignore.
/retest

[tuminoid]

/retest

[tuminoid]

/retest

[tuminoid]

/retest

[tuminoid]

OK, we're still missing Prow image bumps, no wonder it did not work. I'll retry later.

[tuminoid]

/retest

[tuminoid]

/retest

[tuminoid]

One last try.

/retest

[tuminoid]

/test all