Jv integration tests for runtime config #1899

JoukoVirtanen · 2024-10-18T18:20:16Z

Description

Integration tests for runtime configuration. The tests create various versions of the runtime configuration file and check that the configuration introspection endpoint returns the correct expected configuration.

Checklist

Investigated and inspected CI test results
Updated documentation accordingly

Automated testing

Added unit tests
Added integration tests
Added regression tests

If any of these don't apply, please comment below.

Testing Performed

Ran the integration tests locally 2,000 times without a failure using the following script

#!/usr/bin/env bash
set -eou pipefail

for ((i = 0; i < 2000; i=i+1)); do
        echo "i= $i"
        sudo -E make -C integration-tests TestRuntimeConfigFile
done

JoukoVirtanen · 2024-11-06T17:57:59Z

collector/lib/ConfigLoader.cpp

+  } catch (const YAML::Exception& e) {
+    CLOG(ERROR) << "Failed to parse the configuration file: " << config_file << ". Error: " << e.what();
+  } catch (const std::exception& e) {
+    CLOG(ERROR) << "An unknown error occured while loading the configuration file: " << config_file << ". Error: " << e.what();


Molter73

I'm less than half way through, I'll finish the review tomorrow.

collector/lib/ConfigLoader.cpp

Molter73 · 2024-11-06T18:05:56Z

collector/test/ConfigLoaderTest.cpp

@@ -67,12 +67,20 @@ TEST(CollectorConfigTest, TestYamlConfigToConfigInvalid) {
 }

 TEST(CollectorConfigTest, TestYamlConfigToConfigEmpty) {


Please update this test name.

Molter73 · 2024-11-06T18:07:30Z

integration-tests/pkg/collector/collector_docker.go

@@ -41,6 +41,7 @@ func NewDockerCollectorManager(e executor.Executor, name string) *DockerCollecto
 		"/host/etc:ro":              "/etc",
 		"/host/usr/lib:ro":          "/usr/lib",
 		"/host/sys/kernel/debug:ro": "/sys/kernel/debug",
+		"/etc/stackrox:ro":          "/tmp",


Consider creating a separate directory for this like /tmp/collector-test and use that instead of the full /tmp directory.

Molter73 · 2024-11-06T18:16:26Z

integration-tests/pkg/collector/introspection.go

+func AssertExternalIps(t *testing.T, enable bool, collectorIP string) {
+	AssertRepeated(t, func() bool {
+		body := QueryConfig(t, collectorIP)
+		var response types.RuntimeConfig
+		err := json.Unmarshal(body, &response)
+		assert.NoError(t, err)
+
+		return response.Networking.ExternalIps.Enable == enable
+	})
+}
+
+func AssertNoRuntimeConfig(t *testing.T, collectorIP string) {
+	AssertRepeated(t, func() bool {
+		body := QueryConfig(t, collectorIP)
+		return strings.TrimSpace(string(body)) == "{}"
+	})
+}
+
+// TODO: This should be in its own package
+func AssertRepeated(t *testing.T, condition func() bool) {
+	tick := time.NewTicker(1 * time.Second)
+	timer := time.After(3 * time.Minute)
+
+	for {
+		select {
+		case <-tick.C:
+			if condition() {
+				// Condition has been met
+				return
+			}
+
+		case <-timer:
+			// TODO: This message should be passed in rather than hard coded here
+			t.Log("Timeout reached: Runtime configuration was not updated")
+			t.FailNow()
+		}
+	}
+}


We probably want to move all assertions to a separate pkg/assert or similar.

Molter73 · 2024-11-06T18:19:02Z

integration-tests/pkg/collector/introspection.go

+func QueryConfig(t *testing.T, collectorIP string) []byte {
+	log.Info("Querying: /state/config")
+	body, err := IntrospectionQuery(collectorIP, "/state/config")
+	assert.NoError(t, err)
+	log.Info("Response: %q", body)
+	return body
+}


I'm not 100% sold this is something we need, from a test we could just do:

body, err := IntrospectionQuery(s.Collector().IP(), "/state/config") s.Require().NoError(err)

This is simple enough and you can add the logging if you deem it necessary.

JoukoVirtanen · 2024-11-07T02:26:12Z

integration-tests/pkg/mock_sensor/expect_conn.go

@@ -33,7 +33,7 @@ loop:
 		case <-timer:
 			// we know they don't match at this point, but by using
 			// ElementsMatch we get much better logging about the differences
-			return assert.ElementsMatch(t, expected, s.Connections(containerID), "timed out waiting for networks")
+			return assert.ElementsMatch(t, expected, s.Connections(containerID), "timed out waiting for network connections")


The changes in this file were made here https://github.com/stackrox/collector/pull/1902/files

JoukoVirtanen · 2024-11-07T02:26:25Z

integration-tests/pkg/mock_sensor/server.go

@@ -32,7 +32,6 @@ const (
 // us to use any comparable type as the key)
 type ProcessMap map[types.ProcessInfo]interface{}
 type LineageMap map[types.ProcessLineage]interface{}
-type ConnMap map[types.NetworkInfo]interface{}


The changes in this file were made here https://github.com/stackrox/collector/pull/1902/files

Molter73 · 2024-11-07T12:17:08Z

integration-tests/pkg/assert/assert.go

+func AssertExternalIps(t *testing.T, enable bool, collectorIP string) {
+	AssertRepeated(t, func() bool {
+		body, err := collector.IntrospectionQuery(collectorIP, "/state/config")


We don't need to do this here, but the IntrospectionQuery method needs to be part of a collector manager object, it would change this to be a lot simpler/clearer:

func AssertExternalIps(t *testing.T, enable bool, collector *CollectorManager) { AssertRepeated(t, func() bool { body, err := collector.IntrospectionQuery("/state/config")

I think it can be done in another PR. Maybe we should have an Introspection struct that could be a member of a Collector interface. The introspection object would get the IP address from the collector object.

integration-tests/pkg/types/runtime_config.go

integration-tests/suites/runtime_config_file.go

Molter73

Thanks for splitting the tests up, as silly as it may sound, it's a lot easier to understand what is going on now.

The comments I left now are mostly on moving some comments around to leave the code blocks on their own. This should help reduce clutter a bit more, since the description of what will be done is provided up front and the code is left as clean as possible.

integration-tests/suites/runtime_config_file.go

Molter73 · 2024-11-11T16:56:03Z

integration-tests/suites/runtime_config_file.go

+	// The runtime config file was deleted before starting collector so there should not be any config
+	assert.AssertNoRuntimeConfig(s.T(), collectorIP)
+	// Since there is no config the default is used, which means external IPs is disabled and we should
+	// expect a normalized connection


Suggested change

// The runtime config file was deleted before starting collector so there should not be any config

assert.AssertNoRuntimeConfig(s.T(), collectorIP)

// Since there is no config the default is used, which means external IPs is disabled and we should

// expect a normalized connection

// The runtime config file was deleted before starting collector.

// Default configuration is external IPs disabled.

// We expect normalized connections.

assert.AssertNoRuntimeConfig(s.T(), collectorIP)

This same change can be applied to the other 2 tests.

This was not applied to all tests.

It has now been applied to all tests.

integration-tests/suites/runtime_config_file.go

JoukoVirtanen · 2024-11-12T23:15:17Z

CI results

Run 1
Many Konflux integration tests failed. rhcos failed with

=== RUN   TestProcfsScraperDisableFeatureFlag/TestProcfsScraper
    expect_conn.go:181: 
        	Error Trace:	/tests/pkg/mock_sensor/expect_conn.go:181
        	            				/tests/suites/procfs_scraper.go:72
        	Error:      	elements differ
        	            	
        	            	extra elements in list A:
        	            	([]interface ***) (len=1) ***
        	            	 (types.EndpointInfo) ***
        	            	  Protocol: (string) (len=15) "L4_PROTOCOL_TCP",
        	            	  Address: (types.ListenAddress) ***
        	            	   AddressData: (string) (len=4) "\x00\x00\x00\x00",
        	            	   Port: (int) 80,
        	            	   IpNetwork: (string) (len=5) "\x00\x00\x00\x00 "
        	            	  ***,
        	            	  CloseTimestamp: (string) (len=5) "<nil>",
        	            	  Originator: (types.ProcessOriginator) ***
        	            	   ProcessName: (string) "",
        	            	   ProcessExecFilePath: (string) "",
        	            	   ProcessArgs: (string) ""
        	            	  ***
        	            	 ***
        	            	***
        	            	
        	            	
        	            	listA:
        	            	([]types.EndpointInfo) (len=1) ***
        	            	 (types.EndpointInfo) ***
        	            	  Protocol: (string) (len=15) "L4_PROTOCOL_TCP",
        	            	  Address: (types.ListenAddress) ***
        	            	   AddressData: (string) (len=4) "\x00\x00\x00\x00",
        	            	   Port: (int) 80,
        	            	   IpNetwork: (string) (len=5) "\x00\x00\x00\x00 "
        	            	  ***,
        	            	  CloseTimestamp: (string) (len=5) "<nil>",
        	            	  Originator: (types.ProcessOriginator) ***
        	            	   ProcessName: (string) "",
        	            	   ProcessExecFilePath: (string) "",
        	            	   ProcessArgs: (string) ""
        	            	  ***
        	            	 ***
        	            	***
        	            	
        	            	
        	            	listB:
        	            	([]types.EndpointInfo) ***
        	            	***
        	Test:       	TestProcfsScraperDisableFeatureFlag/TestProcfsScraper
        	Messages:   	timed out waiting for endpoints

The other konflux integration tests failed with either problems pulling the image. Both konflux and non-konflux cos integration tests failed with verifier errors.

Run 2:
Similar errors as above. No procfs test error. Collector also fails for fedora coreos.

Run 3:
rhcos had the following error

=== RUN   TestUdpNetworkFlow/TestUdpNetorkflow/sendto_recvmmsg
    udp_networkflow.go:274: 
        	Error Trace:	/tests/suites/udp_networkflow.go:274
        	            				/tests/suites/udp_networkflow.go:244
        	            				/tests/suites/udp_networkflow.go:105
        	            				/tests/suites/udp_networkflow.go:96
        	            				/go/pkg/mod/github.com/stretchr/[email protected]/suite/suite.go:115
        	Error:      	Received unexpected error:
        	            	Error response from daemon: container create: creating container storage: the container name "udp-server" is already in use by 0b88d4d870e77d5fed05bd5bad8a1dd1e5391d4e229d7ffb3d92a5bd57d0febf. You have to remove that container to be able to reuse that name: that name is already in use
        	            	create udp-server

I did not see a previous failure that caused this error. There were other failures unrelated to the changes in the PR.

Molter73

Changes look ok, but #1902 still has comments on it that haven't been addressed, if your intent is to merge the changes from that PR with this one, please close it so I can move my comments to this PR.

Stringy · 2024-11-13T10:36:01Z

integration-tests/pkg/assert/assert.go

+	tick := time.NewTicker(1 * time.Second)
+	timer := time.After(3 * time.Minute)


Can these timeouts be configurable via some parameters?

JoukoVirtanen · 2024-11-14T04:02:00Z

Changes look ok, but #1902 still has comments on it that haven't been addressed, if your intent is to merge the changes from that PR with this one, please close it so I can move my comments to this PR.

I have addressed the PR review comments in the other PR.

integration-tests/pkg/assert/assert.go

integration-tests/pkg/mock_sensor/server.go

Molter73

LGTM, but holding approval until #1902 is merged, so this can be rebased on top of that.

Molter73

LGTM!

Just a small comment on a TODO that I think we could solve quickly before merging, but we can also address it on the next PR that needs to use the helper function.

Molter73 · 2024-11-28T09:36:46Z

integration-tests/pkg/assert/assert.go

+			// TODO: This message should be passed in rather than hard coded here
+			log.Error("Timeout reached: Runtime configuration was not updated")


This TODO should be relatively simple to solve, add a format and args parameter to the function like this:

func AssertRepeated(t *testing.T, tickTime time.Duration, timeout time.Duration, condition func() bool, msg string, args ...interface{}) {

Then here you can just call log.Error like this:

log.Error("Timeout reached: " + msg, args...)

If this ends up being too verbose on the call sites with the function being defined inline, you can always define the function and assign it to a variable, then pass that in.

Molter73 · 2024-11-28T09:41:50Z

Also, there are some tests on the multiarch for konflux, but those are due to the integration test image not being rebuilt for those archs, this is not a blocker for this PR and I'm trying to address it here: #1974

Stringy · 2024-12-02T09:41:24Z

integration-tests/pkg/assert/assert.go

 )

+func AssertExternalIps(t *testing.T, enable bool, collectorIP string) {


[nit] should probably be enabled rather than enable (the latter implies we're turning it on)

Stringy · 2024-12-02T09:44:56Z

integration-tests/pkg/types/runtime_config.go

+		ExternalIps struct {
+			Enable bool
+		}
+	}


just a note that we'll need to add PerContainerRateLimit (probably in a follow up PR)

We should not be defining a RuntimeConfig struct here. We should be importing the go code generated from the protobuf definitions. The same for EndpointInfo, NetworkInfo and other similar structs. That should be done in a different PR.

Stringy · 2024-12-02T09:47:00Z

integration-tests/suites/runtime_config_file.go

+	cmd := "mkdir " + dir
+	s.execShellCommand(cmd)


Can't we do this via the standard library? same for deleteFile, setRuntimeConfig

This way we get the logging that comes with execShellCommand. Also if there are changes to how commands need to be executed, using the standard library might not be the best method. E.g we might want to run the commands in a VM, locally, in a container, or somewhere else and changes to those things might result in the standard library not working. However, you did remind me that this is not the correct place for these functions and I have moved them to base.go.

We've gone to great lengths to remove the various contexts in which things run, and today we're always running on the same host with the test containers so I don't think we should care about how commands run, or whether to restrict ourselves to using the shell instead of the stdlib. If this ever changes, then we'd have to overhaul the way the tests work again, so a few file system operations would be the least of our worries.

If these specific operations need to run in various contexts, then execShellCommand is still inappropriate because it is intended for running a local shell command (via exec.Command) and we'd have to change these functions anyway.

It's just that a shell command feels very expensive for an operation as simple as making a directory or deleting a file, and if we need logging then we can add logging here.

Also, we would have a better handle on the errors if we do it via the stdlib. We wouldn't need to worry about failures in exec'ing the shell or the command, for example.

we're also just suppressing all errors at the moment -- we should probably return them to the caller

Stringy · 2024-12-02T09:50:15Z

integration-tests/suites/runtime_config_file.go

+	configStr := `networking:
+  externalIps:
+    enable: ` + strconv.FormatBool(enable)
+	s.setRuntimeConfig(runtimeConfigFile, configStr)


It would be better for to construct the object and serialize it to a string, and in fact it may be best to have the caller pass the structure directly to setRuntimeConfig

func (s *RuntimeConfigFileTestSuite) setRuntimeConfig(path string, config types.RuntimeConfig) error

e8bb38d Starting work 23ee1b0 Working with config introspection endpoint. No checks for connections yet 1b919ee Moved code for introspection endpoint from mock sensor to own directory 91ee35e Testing with an invalid config ae8bf37 Creates a container with an external connection and adds tests for it. Also made changes so that external connections can be checked for 34247e2 Mock server no longer maintains just a map of connections seen, but all connections edf7ca3 Should be able to handle cidr blocks and non-cidr blocks 4b227bd Able to handle the case when there is only port data dbed0a1 Brought in changes to server.go to handle external ips fd24fe8 Apply suggestions from code review 3eb5e48 Cleanup 13f8539 Able to control if connections should be ordered or not. Also added comments f9ea3fc Listening for new connection events instead of using a ticker 6d06416 Using ExpectSameElementsConnections instead of ExpectExactConnections 9c3f013 More error handling for reading the configuration. Using a volume mount a4fe63b Fixed lint error 8dce19c The runtime config file is only modified on the host machine 1c05a55 There is now only one source code file for runtime config introspection 7b77e23 Minor fixes for k8s e53509f Added comments and sleeps when config does not change so we get a scrape interval 2da34ae Increased the afterglow period ca6dd2c Update collector/lib/ConfigLoader.cpp 6aa60e0 Renamed unit test. Integration tests use a dedicated directory a6b0c57 Created a new assert package 949534d Apply suggestions from code review 09b13aa Split test into smaller tests 9346776 Apply suggestions from code review 3586964 Removed unneeded sort 5e95832 Using slices.ContainsFunc in HasConnection 76e244d Made timings in AssertRepeated configurable 5e94d5c Always sort the connections when comparing 12c04f3 Not using assertMismatch 129308e Updated comments 0b34466 Apply suggestions from code review e2fd23c Sorting connections inside Connections instead of SortedConnections a544a68 Created ElementsMatchFunc in assert pkg 6d1a737 Using spew for logging 2ae0101 Apply suggestions from code review 416b7f2 Importing logging package 9241f09 Simplified by removing checkIfConnectionsMatchExpected ba6babc Changed assert.True to assert.Fail

…eConfig object to a config string rather than writing the string

Stringy · 2024-12-03T16:09:12Z

integration-tests/suites/runtime_config_file.go

+	s.execShellCommand(cmd)
+}
+
+func (s *RuntimeConfigFileTestSuite) setExternalIpsEnable(runtimeConfigFile string, enable bool) {


This might sound like a nit, but from a separation of concerns perspective, this function shouldn't do the marshalling - it should be handled by setRuntimeConfig

This function should just construct types.RuntimeConfig object and pass it to setRuntimeConfig. Otherwise, when we inevitably come to adding another field to these tests, we'll have to duplicate the marshaling logic.

Stringy · 2024-12-03T16:19:16Z

integration-tests/suites/runtime_config_file.go

+	cmd := "mkdir " + dir
+	s.execShellCommand(cmd)


We've gone to great lengths to remove the various contexts in which things run, and today we're always running on the same host with the test containers so I don't think we should care about how commands run, or whether to restrict ourselves to using the shell instead of the stdlib. If this ever changes, then we'd have to overhaul the way the tests work again, so a few file system operations would be the least of our worries.

If these specific operations need to run in various contexts, then execShellCommand is still inappropriate because it is intended for running a local shell command (via exec.Command) and we'd have to change these functions anyway.

It's just that a shell command feels very expensive for an operation as simple as making a directory or deleting a file, and if we need logging then we can add logging here.

Stringy · 2024-12-03T16:41:12Z

integration-tests/suites/runtime_config_file.go

+	cmd := "mkdir " + dir
+	s.execShellCommand(cmd)


Also, we would have a better handle on the errors if we do it via the stdlib. We wouldn't need to worry about failures in exec'ing the shell or the command, for example.

we're also just suppressing all errors at the moment -- we should probably return them to the caller

…ctions for createDir and removeFile

integration-tests/pkg/types/runtime_config.go

JoukoVirtanen · 2024-12-09T17:17:00Z

integration-tests/suites/runtime_config_file.go

+
+func (s *RuntimeConfigFileTestSuite) setRuntimeConfig(runtimeConfigFile string, configStr string) {
+	cmd := "echo '" + configStr + "' > " + runtimeConfigFile
+	s.execShellCommand(cmd)


Use os.write.

https://pkg.go.dev/os#example-WriteFile

Stringy

As discussed, there's a last couple of minor changes but otherwise LGTM!

openshift-ci bot added the do-not-merge/work-in-progress label Oct 18, 2024

JoukoVirtanen force-pushed the jv-integration-tests-for-runtime-config branch from 316fecb to ae8bf37 Compare October 20, 2024 18:12

JoukoVirtanen mentioned this pull request Oct 22, 2024

ROX-26711: Collector mock server should handle external ips #1901

Merged

5 tasks

JoukoVirtanen force-pushed the jv-integration-tests-for-runtime-config branch from 4b6cf76 to 4b227bd Compare October 22, 2024 20:43

JoukoVirtanen mentioned this pull request Oct 23, 2024

ROX-26726: Collector integration tests should record all connections #1902

Merged

5 tasks

JoukoVirtanen force-pushed the jv-integration-tests-for-runtime-config branch from 6d06416 to 075a4df Compare November 1, 2024 18:59

JoukoVirtanen commented Nov 6, 2024

View reviewed changes

Molter73 reviewed Nov 6, 2024

View reviewed changes

JoukoVirtanen force-pushed the jv-integration-tests-for-runtime-config branch from b446ed0 to 347114d Compare November 6, 2024 19:21

JoukoVirtanen changed the base branch from master to jv-ROX-26726-collector-integration-tests-should-record-all-connections November 7, 2024 02:18

JoukoVirtanen changed the base branch from jv-ROX-26726-collector-integration-tests-should-record-all-connections to master November 7, 2024 02:23

JoukoVirtanen commented Nov 7, 2024

View reviewed changes

JoukoVirtanen requested a review from Molter73 November 7, 2024 03:30

Molter73 reviewed Nov 7, 2024

View reviewed changes

JoukoVirtanen requested a review from Molter73 November 11, 2024 04:05

Molter73 reviewed Nov 11, 2024

View reviewed changes

JoukoVirtanen requested a review from Molter73 November 12, 2024 02:28

JoukoVirtanen marked this pull request as ready for review November 12, 2024 23:16

JoukoVirtanen requested a review from a team as a code owner November 12, 2024 23:16

openshift-ci bot removed the do-not-merge/work-in-progress label Nov 12, 2024

Molter73 reviewed Nov 13, 2024

View reviewed changes

Stringy reviewed Nov 13, 2024

View reviewed changes

JoukoVirtanen force-pushed the jv-integration-tests-for-runtime-config branch from 8420864 to 9b4f72e Compare November 14, 2024 01:54

JoukoVirtanen requested review from Molter73 and Stringy November 14, 2024 04:13

JoukoVirtanen force-pushed the jv-integration-tests-for-runtime-config branch from d734a7c to f092de4 Compare November 16, 2024 00:27

Molter73 reviewed Nov 20, 2024

View reviewed changes

integration-tests/pkg/assert/assert.go Outdated Show resolved Hide resolved

integration-tests/pkg/mock_sensor/server.go Outdated Show resolved Hide resolved

JoukoVirtanen requested a review from Molter73 November 21, 2024 03:27

JoukoVirtanen force-pushed the jv-integration-tests-for-runtime-config branch from 4a03f7b to 9241f09 Compare November 21, 2024 05:20

Molter73 reviewed Nov 25, 2024

View reviewed changes

JoukoVirtanen force-pushed the jv-integration-tests-for-runtime-config branch from ba6babc to 72fb150 Compare November 28, 2024 03:14

Molter73 approved these changes Nov 28, 2024

View reviewed changes

Stringy reviewed Dec 2, 2024

View reviewed changes

JoukoVirtanen added 7 commits December 2, 2024 15:55

X-Smart-Branch-Parent: master

57d2af9

Cleanup after rebase

8000bbc

Expected and observed connections are passed in the correct order

58c5c1c

Not hard coding error message in AssertRepeated

65be359

Changed enable to enabled in AssertExternalIps

bc20e2a

Moved createDirectory and deleteFile to base.go. Serializing a Runtim…

9a30908

…eConfig object to a config string rather than writing the string

JoukoVirtanen force-pushed the jv-integration-tests-for-runtime-config branch from b836088 to 9a30908 Compare December 2, 2024 23:55

JoukoVirtanen requested a review from Stringy December 3, 2024 16:01

Stringy reviewed Dec 3, 2024

View reviewed changes

Broke setExtenalIpsEnabled into smaller functions. Using built in fun…

596a2e3

…ctions for createDir and removeFile

JoukoVirtanen requested a review from Stringy December 6, 2024 21:25

JoukoVirtanen commented Dec 9, 2024

View reviewed changes

integration-tests/pkg/types/runtime_config.go Outdated Show resolved Hide resolved

JoukoVirtanen commented Dec 9, 2024

View reviewed changes

Stringy approved these changes Dec 9, 2024

View reviewed changes

JoukoVirtanen added 3 commits December 9, 2024 09:24

Fixed minor typo

e7d60e8

Using os.WriteFile instead of execShellCommand and echo

348116b

Made change to make things consistent with recent typo fix

a4c8d06

JoukoVirtanen merged commit a979ab4 into master Dec 9, 2024
51 of 53 checks passed

JoukoVirtanen deleted the jv-integration-tests-for-runtime-config branch December 9, 2024 21:00

		@@ -67,12 +67,20 @@ TEST(CollectorConfigTest, TestYamlConfigToConfigInvalid) {
		}

		TEST(CollectorConfigTest, TestYamlConfigToConfigEmpty) {

		tick := time.NewTicker(1 * time.Second)
		timer := time.After(3 * time.Minute)

		// TODO: This message should be passed in rather than hard coded here
		log.Error("Timeout reached: Runtime configuration was not updated")

		)

		func AssertExternalIps(t *testing.T, enable bool, collectorIP string) {

Jv integration tests for runtime config #1899

Jv integration tests for runtime config #1899

Conversation

JoukoVirtanen commented Oct 18, 2024 • edited Loading

Description

Checklist

Testing Performed

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Molter73 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Molter73 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JoukoVirtanen commented Nov 12, 2024 • edited Loading

Molter73 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JoukoVirtanen commented Nov 14, 2024

Molter73 left a comment

Choose a reason for hiding this comment

Molter73 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Molter73 commented Nov 28, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JoukoVirtanen Dec 2, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Stringy left a comment

Choose a reason for hiding this comment

JoukoVirtanen commented Oct 18, 2024 •

edited

Loading

JoukoVirtanen commented Nov 12, 2024 •

edited

Loading

JoukoVirtanen Dec 2, 2024 •

edited

Loading