Provide suite of tests for validating Fulcio releases and deployments

[k4leung4]

Description

To better streamline releases and deployments, we need a suite of tests that we can run to validate releases and deployments.

@haydentherapper Can you outline what tests you would like to see and what tests we have and which ones we are lacking.

[haydentherapper]

The gRPC test suite code is fairly thorough. It tests all endpoints with different types of issuers. For each supported issuer for the production environment (email, SPIFFE, GitHub and K8S currently), we should test a successful certificate issuance. We should also test the other GET endpoints.

Tests over HTTP could be considered, but we'd effectively be testing the gRPC-HTTP bridge, which is not necessary. If we did test using HTTP, we should at least test the V1 API.

[k4leung4]

Are you proposing we should have the equivalent of the gRPC test suite, but pointed at a live instance?

As for supported issuer, do we have documentation on how to test each of the issuer? with the lack of documentation and existing test, do we actually know if the spiffe actually works, other than a user telling us it doesnt? im worried that we have previously added support for issuers but never had a way to continuously verify it works.

sounds like HTTP testing would be a nice to have, but not high on the priority list.

To better prioritize them, are these all tests that we are willing to block our next release/deployment on, or only some?

I'm trying to set a baseline of what are we comfortable doing a release/deployment with, versus what do we want to aim to have.

[haydentherapper]

Are you proposing we should have the equivalent of the gRPC test suite, but pointed at a live instance?

Yea, I'd like a subset of the suite tests pointed against the live instance, exercised before a release:

• Issuing a certificate with the Dex provider
• Issuing a certificate with the GitHub provider
• Issuing a certificate with a K8S provider
• Checking api/v2/TrustBundle can be used to verify an issued certificate

Some of those may be harder than others to set up, particularly GitHub and K8S. If we at least cover Dex initially, that'd be good. I'm removing SPIFFE from the list because it's a federated provider currently.

As for supported issuer, do we have documentation on how to test each of the issuer? with the lack of documentation and existing test, do we actually know if the spiffe actually works, other than a user telling us it doesnt? im worried that we have previously added support for issuers but never had a way to continuously verify it works.

I can add more docs/help write the test suite, just need to know if there's a certain way to write it - Bash script? Go script?

We don't have tests for each supported OIDC issuer, though I would say it's not Fulcio's responsibility to test any federated providers (the SPIFFE ones primarily). We should make sure the default providers - GitHub, Dex, Google, K8S - are working.

To better prioritize them, are these all tests that we are willing to block our next release/deployment on, or only some?

The list above is what I'd like to have block a release. Not sure when we're planning to cut a new release. If we need a new release soon, we might need to manually test, but I think automation should be in scope for GA. Does that seem fair?

[k4leung4]

I would like to see new tests in Go, as it is easier to maintain, and less likely to grow organically in an unmanageable fashion.

Agree on the verifying of GitHub/Dex/Google/K8s providers.

I think it is fair to have automation in scope for GA.

Moving forward, I would like to push to have e2e integration tests for all new features so we can avoid the situation where we don't feel confident about releases/deployments.

When do you think we will have the tests or docs ready, whether it is for the prod migration or just another general release?

[haydentherapper]

Note to self: We need to also test:

• Values of the certificate
• Successful entry in a transparency log, successful verification of the SCT, and verify inclusion proof

[loosebazooka]

Questions/things so I can frame this in the client perspective and then also put man power behind this:

1. Should e2e tests live independently of the service (fulcio/rekor) repos? (in GA repo?)
2. These test will run against prod and staging?
3. Should/can staging always just be latest relase on each of the service repos? (is it already?)
   • alternatively can there be latest, staging, prod, where "latest" is this potentially chaotic env.
4. I would like the test suite to be extensible (maybe via a config file or just code) to execute tests from other repo clients

   repo: "github.com/sigstore/sigstore-java"
   tests: "./gradlew e2eTests"
   

[k4leung4]

1. I think it makes sense to for e2e tests to live in a common repo that the service repos can access. Not sure if sigstore/sigstore would make sense or having its own repo. If it is a blocker, I would just choose one a service repo and we can migrate to another repo at a later point.
2. The goal is to run it against staging, and as necessary against prod. I think running against prod initially will be necessary, depending on the nature of the tests.
3. Will definitely have a latest env, and I think some version of it will need to live in the service repo,

• in cosign: test against [prod|staging] rekor|fulcio at head
• in rekor: test against [prod|staging] fulcio and prod cosign at head
• in fulcio: test against [prod|staging] rekor and prod cosign at head

4. +1

[dlorenc]

IMO the probers we have are sufficient for GA, although they can always be improved.

[trixor]

@lukehinds @dlorenc what are your thoughts on if this needs to be a GA Blocker or not?

[dlorenc]

Not a GA blocker IMO.

[trevrosen]

I'm not seeing this as a GA blocker, but it would be a good fast-follow issue, possibly for my team to tackle.

cc: @codysoyland @kommendorkapten

[haydentherapper]

Not a blocker sounds good.