fix(ci): Multiple data and test issues #1737

dcaravel · 2024-12-10T02:36:38Z

Description

Fixes multiple CI issues

Removes CVSSv3 metadata from various test cases for Microsoft images that no longer have data avail in the NVD legacy feeds.
Adjusts sanity test logic to skip validating CVSSv3 fields when not provided by test case.
Increase test timeout to avoid panic: test timed out after 20m0s (such as this one).
Removes CVE-2021-26291 from the ose-jenkins image test case, it is no longer appearing in scan results (ref)

$ rctl image scan -f --image=quay.io/rhacs-eng/qa:ose-jenkins 2>/dev/null | jq -r '.scan.components[] | select(.name == "jenkins-2-plugins").vulns[].cve' 
RHSA-2022:6531
RHSA-2022:7865
RHSA-2022:9098
RHSA-2023:0560
RHSA-2023:0697
RHSA-2023:1655
RHSA-2023:1866
RHSA-2023:3362
RHSA-2023:3625

Testing

CI

Logs from CI showing CVSSv3 comparison skips.

=== RUN   TestImageSanity/mcr.microsoft.com/dotnet/core/runtime:3.1.2
=== RUN   TestImageSanity/mcr.microsoft.com/dotnet/core/runtime:3.1.2/microsoft.netcore.app/3.1.2
    sanity_test.go:40: WARN: No CVSSv3 data provided for "CVE-2021-1721", skipping CVSSv3 field validation.
    sanity_test.go:40: WARN: No CVSSv3 data provided for "CVE-2021-1723", skipping CVSSv3 field validation.
    sanity_test.go:40: WARN: No CVSSv3 data provided for "CVE-2021-24112", skipping CVSSv3 field validation.
    sanity_test.go:40: WARN: No CVSSv3 data provided for "CVE-2021-26701", skipping CVSSv3 field validation.
    sanity_test.go:40: WARN: No CVSSv3 data provided for "CVE-2021-31204", skipping CVSSv3 field validation.
=== RUN   TestImageSanity/mcr.microsoft.com/dotnet/core/sdk:3.1.100@sha256:091126a93870729f4438ee7ed682ed98639a89acebed40409af90f84302c48dd
=== RUN   TestImageSanity/mcr.microsoft.com/dotnet/core/sdk:3.1.100@sha256:091126a93870729f4438ee7ed682ed98639a89acebed40409af90f84302c48dd/microsoft.aspnetcore.app/3.1.0
    sanity_test.go:40: WARN: No CVSSv3 data provided for "CVE-2020-1045", skipping CVSSv3 field validation.
    sanity_test.go:40: WARN: No CVSSv3 data provided for "CVE-2021-1723", skipping CVSSv3 field validation.
=== RUN   TestImageSanity/mcr.microsoft.com/dotnet/core/sdk:3.1.100@sha256:091126a93870729f4438ee7ed682ed98639a89acebed40409af90f84302c48dd/microsoft.netcore.app/3.1.0
    sanity_test.go:40: WARN: No CVSSv3 data provided for "CVE-2021-1721", skipping CVSSv3 field validation.
    sanity_test.go:40: WARN: No CVSSv3 data provided for "CVE-2021-1723", skipping CVSSv3 field validation.
    sanity_test.go:40: WARN: No CVSSv3 data provided for "CVE-2021-24112", skipping CVSSv3 field validation.
    sanity_test.go:40: WARN: No CVSSv3 data provided for "CVE-2021-26701", skipping CVSSv3 field validation.
    sanity_test.go:40: WARN: No CVSSv3 data provided for "CVE-2021-31204", skipping CVSSv3 field validation.

dcaravel · 2024-12-11T01:31:13Z

/test e2e-tests

dcaravel · 2024-12-11T01:39:37Z

Last failure appears to be a flake:

=== RUN   TestStackroxVulnImages/quay.io/rhacs-eng/qa:appdynamics
Handling connection for 8443
    utils.go:49: 
        	Error Trace:	/go/src/github.com/stackrox/scanner/e2etests/utils.go:49
        	            				/go/src/github.com/stackrox/scanner/e2etests/utils.go:65
        	            				/go/src/github.com/stackrox/scanner/e2etests/vuln_test.go:39
        	            				/go/src/github.com/stackrox/scanner/e2etests/vuln_test.go:265
        	Error:      	Received unexpected error:
        	            	rpc error: code = Unknown desc = could not advance in the tar archive: archive/tar: invalid tar header
        	Test:       	TestStackroxVulnImages/quay.io/rhacs-eng/qa:appdynamics

jvdm · 2024-12-11T21:15:47Z

e2etests/sanity_test.go

+		// When expected vuln has no CVSSv3 data, do not try to compare it.
+		// This was added when NVD stopped returning CVSSv3 data for some vulns which we had test cases for.


Can you confirm your intention is to continue ignoring V3 if/when NVD bring it back?

Good Q. Main intention was to get CI green again 😉 . Given that this scanner is on 'life support' I didn't spend too much time considering far into the future.

As it's currently written it would indeed continue ignoring V3 if/when NVD brings it back. I did consider reversing the condition such that if the NVD data has V3 then do the compare, otherwise ignore - I changed that assuming we would want CI failures when the data we're expecting to exist goes away so that we could investigate further (like we did with these).

I suppose we could change the condition to fail the test if NVD V3 data was provided but our test case isn't comparing it so that if/when the data comes back we'll have failures again.

Any preference?

RTann · 2024-12-11T23:17:37Z

e2etests/testcase_test.go

-									"Score":               6.5,
-									"Vectors":             "CVSS:3.1/AV:N/AC:L/PR:N/UI:R/S:U/C:N/I:N/A:H",
-								},
+								// NVD stopped returning this


Can you create a ticket to track this?

I may instead change the sanity test to fail if this data comes back so that we will not need a ticket and can rely on CI failures instead.

RTann · 2024-12-11T23:19:45Z

e2etests/testcase_test.go

@@ -3887,31 +3899,6 @@ Applications using RegexRequestMatcher with '.' in the regular expression are po
 				AddedBy:       "sha256:3fa3f612bdcb92746bf76be1b9c9e1c1c80de777aedaf48b7068f4a129ded3c2",
 				FixedBy:       "4.10.1685679861-1.el8",
 				Vulnerabilities: []apiV1.Vulnerability{
-					{


You know I have seen this flap sometimes, and I have yet to dig into why this happens. Any ideas? I recall this was a key vulnerability we wanted to track because it's an unfixed OpenShift-related vulnerability which we did not track for some time. It'd be nice to keep this, but since we are focused more on Scanner V4, I'm ok with commenting this out with a ticket to go with it to track it

Hrm, interesting, I'll see if I have an old copy of the data for comparing to see what changed - I did check and none of the RHSA's listed in the current scan results for this image include this particular CVE.

I did notice CI uses a new 'genesis dump' each run - and if I remember correctly scanner currently does not handle 'deleted' vulns when processing a diff. So this would only appear in scanners that include a new genesis dump as of whatever date when this vuln was seemingly removed from the feeds.

RTann · 2024-12-11T23:21:25Z

e2etests/sanity_test.go

+		}
+
+		// When expected vuln has no CVSSv3 data, do not try to compare it.
+		// This was added when NVD stopped returning CVSSv3 data for some vulns which we had test cases for.


maybe another ticket here while we're at it :D

Sure, what would the ticket be for? Rechecking the vulns to see if NVD added the data back?

openshift-ci bot added the do-not-merge/work-in-progress label Dec 10, 2024

adjusts test cases and logic to handle removed NVD data

73b0613

dcaravel force-pushed the dc/ci-fix branch from d25245a to 73b0613 Compare December 10, 2024 21:32

dcaravel changed the title ~~[WIP DO NOT MERGE] chore(ci): Fix CI due to NVD feed changes~~ chore(ci): Fix CI due to NVD feed changes Dec 10, 2024

openshift-ci bot removed the do-not-merge/work-in-progress label Dec 10, 2024

dcaravel changed the title ~~chore(ci): Fix CI due to NVD feed changes~~ chore(ci): Fix CI due to NVD legacy feed changes Dec 10, 2024

dcaravel changed the title ~~chore(ci): Fix CI due to NVD legacy feed changes~~ fix(ci): NVD legacy feed changes broke CI Dec 10, 2024

dcaravel added 2 commits December 10, 2024 16:22

increase test timeouts

e7d0542

fix jenkins test case

85cdbb0

dcaravel requested review from a team, jvdm, BradLugo, daynewlee and RTann December 11, 2024 02:16

dcaravel mentioned this pull request Dec 11, 2024

ROX-26604: Add layer lineage to RHEL v2 data model #1720

Open

dcaravel changed the title ~~fix(ci): NVD legacy feed changes broke CI~~ fix(ci): Multiple data and test issues Dec 11, 2024

jvdm reviewed Dec 11, 2024

View reviewed changes

dcaravel requested a review from jvdm December 11, 2024 22:35

RTann reviewed Dec 11, 2024

View reviewed changes

RTann mentioned this pull request Dec 13, 2024

chore: add manual dotnet entries #1731

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(ci): Multiple data and test issues #1737

fix(ci): Multiple data and test issues #1737

dcaravel commented Dec 10, 2024 •

edited

Loading

dcaravel commented Dec 11, 2024

dcaravel commented Dec 11, 2024 •

edited

Loading

jvdm Dec 11, 2024

dcaravel Dec 11, 2024 •

edited

Loading

RTann Dec 11, 2024

dcaravel Dec 11, 2024

RTann Dec 11, 2024

dcaravel Dec 11, 2024

RTann Dec 11, 2024

dcaravel Dec 11, 2024 •

edited

Loading

		// When expected vuln has no CVSSv3 data, do not try to compare it.
		// This was added when NVD stopped returning CVSSv3 data for some vulns which we had test cases for.

fix(ci): Multiple data and test issues #1737

Are you sure you want to change the base?

fix(ci): Multiple data and test issues #1737

Conversation

dcaravel commented Dec 10, 2024 • edited Loading

Description

Testing

dcaravel commented Dec 11, 2024

dcaravel commented Dec 11, 2024 • edited Loading

jvdm Dec 11, 2024

Choose a reason for hiding this comment

dcaravel Dec 11, 2024 • edited Loading

Choose a reason for hiding this comment

RTann Dec 11, 2024

Choose a reason for hiding this comment

dcaravel Dec 11, 2024

Choose a reason for hiding this comment

RTann Dec 11, 2024

Choose a reason for hiding this comment

dcaravel Dec 11, 2024

Choose a reason for hiding this comment

RTann Dec 11, 2024

Choose a reason for hiding this comment

dcaravel Dec 11, 2024 • edited Loading

Choose a reason for hiding this comment

dcaravel commented Dec 10, 2024 •

edited

Loading

dcaravel commented Dec 11, 2024 •

edited

Loading

dcaravel Dec 11, 2024 •

edited

Loading

dcaravel Dec 11, 2024 •

edited

Loading