-
Notifications
You must be signed in to change notification settings - Fork 4.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Disable linux-x64 dev-innerloop #108581
Disable linux-x64 dev-innerloop #108581
Conversation
Tagging subscribers to this area: @hoyosjs |
I'm fine with disabling until we either fix the mega-build into pieces or bump the memory. |
Cc: @ericstj. This is related to the OOM issue we encountered with the Could we consider splitting the build into groups? For example, we could run multiple |
Do we know what consumes the memory and whether it is expected? We should check that this is not a symptom of a product issue. |
@agocke suggested we may have broken something in main and to see how it behaves after #107772 went in. |
LKG PR was merged 20 hours ago, since than I have seen multiple PRs running into the innerloop timeout issue. Not sure if that is related to -allConfigurations leg (which is purely about libs subset building all assemblies for every possible platform combination), but I might be missing something. |
Yeah, I guess that wasn't it. However, I notice that release/9.0 runs seem to be succeeding. So I'm still worried this is a product issue of some sort. |
Agree with what @jkotas says. There's no reason for us to think that the product shouldn't be capable of building the |
CI machine is running out of memory after building assemblies for 50-60 mins. Locally allConfigurations build succeeds because I have 32 gigs on my linux box (and not many running processes). I don't think anyone was suggesting that the entire product is incapable.. |
Right, it suggests that there is "leak" that accumulates to be big-enough to take the CI machine down with long-enough build. A single assembly takes no more than 10's seconds to build. All memory required to build given assembly should be released after we are done building it if there is a memory pressure. |
@jkotas should we merge this until dotnet/dnceng#3879 is resolved since it's failure rate is high these days? If you think we should keep it enabled, lets close this PR and track it in the issue. cc @dotnet/dotnet-hotreload-utils-admin |
I think we should have this leg disabled against an issue in this repo that is specific to linux_x64_dev_innerloop dotnet/dnceng#3879 looks too generic, not very actionable. |
Opened #108821. |
Next steps are figuring out how we're going to go about investigating and who is going to do it. |
This reverts commit 7b69459.
Lately, this leg just times out with pending status on GItHub:
From dotnet/dnceng#3879, it is currently the only leg running into this issue.