Ensure building dotnet/runtime works on Windows ARM64

[adamsitnik]

Hello everyone!

Some time ago I've received my very first ARM64 machine - Surface Pro X. I was very happy about it as I was hoping that it would make ARM64 performance investigations easier for me.

Recently the right opportunity has arisen - I've identified a few ARM64-specific regressions (#41699, #41704, #41706) and decided to prepare a simple workshop where I would explain how to: run benchmarks, profile, and get the disassembly on ARM64. To make sure that everyone on the team can solve simple ARM64-specific performance issues.

Since it's currently impossible to install Linux on Surface Pro X and our performance tooling is more mature on Windows, I decided to prepare a Windows workshop first. I did not want to use WSL as it adds another layer of abstraction and none of our customers run .NET Core apps via WSL in production.

I've tried running some BenchmarkDotNet benchmarks and using PerfView to profile them. Everything worked perfectly fine.

As we all know, after a performance bottleneck is identified, the next step is to apply a fix, rebuild the product and run the benchmarks against the modified build.

And this is where I've failed.

After running a very simple command:

build.cmd -c Release -arch arm64

The build got stuck after printing a few lines to the output. It turned out to be a known issue: #38902. I remembered that @eiriktsarpalis mentioned to me a while ago that the new SDK should contain a fix and that @ViktorHofer send an email about updating the SDK version just a few days ago. So I've pulled Viktor's PR: #41947 and it solved this particular problem.

Then I've faced another problem, read the docs about Windows ARM requirements and installed missing VS components (this doc is great 👍 )

Another problem was gone, but a new one has arisen:

    This version of C:\Projects\runtime\artifacts\bin\coreclr\Windows_NT.arm64.Release\x64\dactabletools\InjectResource.exe is not compatible with the version of Windows you're running. Check your computer's system information and then contact the software publisher.
  C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\MSBuild\Microsoft\VC\v160\Microsoft.CppCommon.targets(234,5): error MSB6006: "cmd.exe" exited with code 216. [C:\Projects\runtime\artifacts\obj\coreclr\Windows_NT.arm64.Release\src\dlls\mscoree\coreclr\inject_debug_resources.vcxproj]
  BUILD: Error: native component build failed. Refer to the build log files for details.

The full log is available here

Here I decided to stop and asked @eiriktsarpalis for help. Eirik told me that from what he knows building the runtime repo on Windows ARM64 is currently not supported and the only workaround is to cross-build the runtime on x64 machine and copy the binaries to the ARM machine... Or switch to WSL2 and build it for Linux as it works fine.

Building a product on a different machine and copying it over to my Surface box hurts my productivity a lot. I am not the only person who got Surface Pro X. I imagine that other team members (@pgovind @carlossanlop @eiriktsarpalis @kunalspathak) would also like to have the possibility to just build the product locally, attach the debugger, run the tests, etc. Just get the work done smoothly and move to the next task.

@ericstj @danmosemsft Is there any chance for improving the developer experience on Windows ARM64?

[ghost]

Tagging subscribers to this area: @ViktorHofer
See info in area-owners.md if you want to be subscribed.

[ViktorHofer]

cc @mikem8361 @jkoritzinsky for the dactabletools issue

[danmoseley]

building the runtime repo on Windows ARM64 is currently not supported

Does this mean we have decided we won't get it to work? Or that we haven't yet got it to work? It would be nice to continue to understand what blocks it - assuming that we do support the shipping SDK on this OS.

[eiriktsarpalis]

Does this mean we have decided we won't get it to work? Or that we haven't yet got it to work?

I think it's probably the latter. It would be nice if we could try to get a CI job running the build on ARM64 natively.

[danmoseley]

cc @ericstj it would be good to have this on our infra backlog. Maybe it's one of the tasks for the next couple months.

[jkoritzinsky]

My guess here is that the build-runtime.cmd script still tries to do a cross-build for arm64 even when building on an arm64 machine. So it's erroring out when trying to run the x64 tools built earlier on the arm64 binaries since Windows 10 ARM64 only has an x86 emulator, not x64.

[ericstj]

It would be nice to continue to understand what blocks it

Reading through @adamsitnik's issues it's just a mix of product bugs, which got fixed, then repo bugs in individual components. Those should just get filed against the component owners and fixed.

Sure someone on the infra team can drive this. Once we get clean we should add a CI leg (if hardware is available) to protect it.

[adamsitnik]

Those should just get filed against the component owners and fixed

The problem is that I can report only 1 problem at a time (I hit a bug, report it, it gets fixed and then I hit another and can report it) and if we don't increase the priority it's going to take many months (it took two months to fix #38902 and I expect that there are at least few other undiscovered problems)

Sure someone on the infra team can drive this. Once we get clean we should add a CI leg

this would be perfect 💯

[ViktorHofer]

My guess here is that the build-runtime.cmd script still tries to do a cross-build for arm64 even when building on an arm64 machine. So it's erroring out when trying to run the x64 tools built earlier on the arm64 binaries since Windows 10 ARM64 only has an x86 emulator, not x64.

@jkoritzinsky do we have a tracking issue for that?

[jkoritzinsky]

I don't believe we do. We could use this as the tracking issue or we can create a new one.

[ViktorHofer]

@jkoritzinsky who would be the best person to look into this?

[jkoritzinsky]

I think either myself, @hoyosjs, @davidwrighton, or someone on the JIT team. I don't think I have the cycles at the moment to work on this.

[hoyosjs]

I could do this as a side work-item. However I don't have access to an ARM64 device capable of building something like runtime, I'd need to ask if there's any devices available.

[ViktorHofer]

@MattGal might know about arm64 hardware. AFAIK we have a few physical devices but I don't know if you could connect to them.

[MattGal]

@ilyas1974 can help with this. We have VMs now that you could connect to, but you'll need a jumpbox on the corporate network to remote to it.

[ilyas1974]

Taking this offline with @hoyosjs to get him a system

[ilyas1974]

We've taken ddarm64-046 out of helix rotation and are using it for this investigation.

[ViktorHofer]

Thanks for following-up on this that quickly!

[AntonLapounov]

Has this issue been resolved?

[hoyosjs]

Has this issue been resolved?

@AntonLapounov Not yet. There's an arm64 sdk, but the native build needed some work. I'll be taking a look next week

[hoyosjs]

I haven't been able to take a look yet (few PRI 1 came after release and then the test reliability effort, will address as soon as those clear out).

[danmoseley]

Thanks @hoyosjs this came up yesterday during .NET 5 retrospective and we feel that it would be valuable, especially for library devs where an incremental build is fairly quick so should be less sensitive to hardware performance.

[hoyosjs]

Sadly I haven't been able to take a stab at this. There's a few things I have that take priority over this. I'm removing the assignment in case someone else has bandwidth to pick it up.

[AntonLapounov]

What issues did we have? I just tried build -s clr and it worked fine for a long time until hit this issue with DacTableGen:

BUILDMSG: System.BadImageFormatException: An attempt was made to load a program with an incorrect format. (0x8007000B)
   at Dia.Util.DiaFile.DllGetClassObject(Guid rclsid, Guid riid)
   at Dia.Util.DiaFile.GetDiaSourceClass() in C:\Runtime\src\coreclr\ToolBox\SOS\DacTableGen\diautil.cs:line 421

[eiriktsarpalis]

Last time I checked the issue was primarily due to x64 build tool dependencies. Upcoming x64 emulation support on Windows ARM should fix the problem eventually. According to this post it should already be available in the Dev channel of insiders preview.

[hoyosjs]

It was mostly that. Things like the DAC table generation would require to use x86 as there isn't such a DIA based solution on arm64 in-box in VS. If you run that as an x86 process there would be no issue, but then all is running under emulation. The native build would always need the emulation though afaik, given that there's no ARM64 VS. That's also the case for WiX (I believe all their binaries are x86). Choosing and matching were what I saw. I didn't get to investigate the failures as much though.

[AntonLapounov]

DAC table generation would require to use x86 as there isn't such a DIA based solution on arm64 in-box in VS

@hoyosjs I do have the ARM64 version at "%VSINSTALLDIR%DIA SDK\bin\arm64\msdia140.dll". It might have been installed with ARM64 build tools, which are a prerequisite for ARM64 builds. I am going to submit a PR and ask you for CR.

[GrabYourPitchforks]

@adamsitnik @eiriktsarpalis Can we double-check the contents of https://github.com/dotnet/runtime/blob/main/docs/workflow/requirements/windows-requirements.md to verify that they're still applicable to arm64 Windows devices?

[AntonLapounov]

Yes, they are applicable. In fact, the minimal VS installation you need for building the repo without tests is Build Tools edition with C++ ARM64 build tools and Windows 10 SDK, which can be installed this way:

curl -LO https://aka.ms/vs/16/release/vs_buildtools.exe
vs_buildtools.exe --installPath C:\VS2019 --add Microsoft.VisualStudio.Component.VC.Tools.ARM64 Microsoft.VisualStudio.Component.Windows10SDK.19041 --passive --norestart --nocache

Alternatively you may use vs_community.exe to install the Community edition with IDE.

x86 version of Git for Windows does not work on my Windows ARM64 though. Git installed with the VS does work.

[GrabYourPitchforks]

@AntonLapounov The previous release of Git for Windows had some regressions on arm64. I had to install a nightly to unblock myself. But according to git-for-windows/git#3107, looks like there's going to be a flurry of work on this in the near future. So things should get better. :)