Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

rosa-sts-ovn dags must wait for nodes to update before running benchmarks #344

Open
afcollins opened this issue Jul 25, 2023 · 0 comments
Open

Comments

@afcollins
Copy link
Member

node-density fails because p99 latency is > 5s. The reason the latency is so high is bc worker nodes are still being updated when the test starts.

We must ensure the worker MCP is completely updated (READY == UPDATED).
We can speed up this process by setting maxUnavailable=100%.

It would also be a plus to pause all MCPs after the cluster and all day2 operations are complete, so any platform maintenance will not disrupt the tests.

Recent failure log: msg="❗ P99 Ready latency (11.52s) higher than configured threshold: 5s

I thought we could increase or remove the pod-threshold from the test, but that would give us bad data and increase confusion when looking at results. It is best to fix this during install/postinstall tasks up front.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant