Windows 386 build support

[alchem1ster]

Are there plans to add x86 support?

• Windows 10 Pro 22H2 x64
• tinygo version 0.26.0 windows/amd64 (using go version go1.19.4 and LLVM version 14.0.0)
• GCC 12.2.0 + LLVM/Clang/LLD/LLDB 14.0.6 + MinGW-w64 10.0.0 (MSVCRT)

Input:
$Env:GOARCH=386
tinygo build -no-debug -o result.exe
Result:
error: failed to hash file: open O:\TinyGo\src\runtime\asm_386_windows.S: The system cannot find the file specified.

[aykevl]

Correct, windows/386 is not currently supported. (But we should really be printing a more useful error message here).

I think windows/386 support can be added, but maintaining it will be some work. Can you describe why you need it?

[alchem1ster]

Can you describe why you need it?

Creating tiny standalone utilities for x86-based Windows systems, with very weak CPUs (similar to Core 2 Duo) and limited RAM (up to 1GB). I can use the full Go, but tinyGo allows me to significantly reduce memory consumption.

[BieHDC]

I started working on this and have a simple hello world printing before it crashes for yet unknown reasons.
I would be willing to keep working on this, however i am afraid i will require the support of someone who has deeper knowledge in the subject.

[BieHDC]

I did the other things i wanted to achieve, now it is just about that crash i cant debug.
And i took down a semi fixme in the os_windows aswell while i was at it.

I parked the current work here for anyone who wants to take a look.

[aykevl]

My first guess would be that you use the wrong calling convention in task_stack_386_windows.S. You can try to build with -scheduler=none to skip that code and see if it works any better. I don't know exactly which calling convention is used on windows/386, there are a lot of them (see https://en.wikipedia.org/wiki/X86_calling_conventions).
Another thing you can try is use -gc=leaking to avoid including a GC, there's a small chance that might also skip some brokenness on windows/386.

[BieHDC]

I tried preserving xmm6 and xmm7 as mentioned here and also a hail mary approach with pushf/popf and pusha/popa, but it didnt help.

When doing -gc=leaking it fails to find _tinygo_scanstack and when doing -scheduler=none it fails to find _tinygo_pause. From my current code you can see i just double defined such symbols twice in the .S files to work around it.
I took the amd64 as template for symbol convention when doing this port and i dont know why it has trouble resolving them correctly on 386.
So for now i commented out the symbols it did not find in the .S parts of the code for quick testing and the crash is the same between all 3 variants still.
But i would really like to know how to resolve those linker issues.

[BieHDC]

One thing i found when appending --verbose to the linker flag:
amd64 -> tinygo:ld.lld: Entry name inferred: mainCRTStartup
386 -> tinygo:ld.lld: Entry name inferred: _mainCRTStartup

So it seems for 386 mingw adds a leading underscore by default to things and there is --[no-]leading-underscore Set explicit symbol underscore prefix mode on my system mingw, but this is not present on tinygos lld.
Looks like it requires this patch https://reviews.llvm.org/D152363

[BieHDC]

After some more testing it turns out the crash happens when the tinygo program exits, not during the run. Which just makes it much weirder. With scheduler and gc completely disabled and looking at the binary in ghidra, it really just printlns the message and then exits with code 0 as you would expect. (it looks just like a C hello world to be precise)
So the fault happens on teardown for some reason.

Unhandled exception: page fault on execute access to 0x7ffd1000 in wow64 32-bit code (0x7ffd1000).
0114:fixme:dbghelp:elf_search_auxv can't find symbol in module
Register dump:
 CS:0023 SS:002b DS:002b ES:002b FS:0063 GS:006b
 EIP:7ffd1000 ESP:0051ff58 EBP:0051ff58 EFLAGS:00010246(  R- --  I  Z- -P- )
 EAX:00000000 EBX:7ffd1000 ECX:00404008 EDX:00000000
 ESI:00000000 EDI:00000000
Stack dump:
0x0051ff58:  0051ff6c 7efa7c9f 7ffd1000 00000000
0x0051ff68:  00000000 0051ffec 7efa900e 7eef8920
0x0051ff78:  004017a7 7ffd1000 00000000 00000000
0x0051ff88:  00000000 ffffffff 7efbc650 7ef785e0
0x0051ff98:  00000000 0051ffec 7ffd1000 00000000
0x0051ffa8:  00000000 0051ff70 7efa8fc2 00000000
Backtrace:
=>0 0x7ffd1000 (0x0051ff58)
  1 0x7efa7c9f in ntdll (+0x57c9f) (0x0051ff6c)
  2 0x7efa900e in ntdll (+0x5900e) (0x0051ffec)
0x7ffd1000: add %al, (%eax)
Modules:
Module  Address                 Debug info      Name (13 modules)
PE        400000-  417000       Deferred        hello
ELF     7e52d000-7e600000       Deferred        msvcrt.dll.so
  \-PE  7e550000-7e600000       \               msvcrt
ELF     7e600000-7e8bb000       Deferred        kernelbase.dll.so
  \-PE  7e620000-7e8bb000       \               kernelbase
ELF     7eeb3000-7ef31000       Deferred        kernel32.dll.so
  \-PE  7eed0000-7ef31000       \               kernel32
ELF     7ef31000-7f000000       Export          ntdll.dll.so
  \-PE  7ef50000-7f000000       \               ntdll
ELF     f7c00000-f7e27000       Deferred        libc.so.6
ELF     f7e79000-f7f31000       Export          ntdll.so
ELF     f7f33000-f7f68000       Deferred        ld-linux.so.2
ELF     f7f68000-f7f6d000       Deferred        <wine-loader>

---

However, the linker issue above makes it impossible to write complex programs as you can not even import "time" without ending up in unresolved function calls due to the underscore issue.
It is looking for 3 underscores while it should actually be looking for 2 underscores.

tinygo:ld.lld: error: undefined symbol: ___udivdi3
>>> referenced by /home/biehdc/code/tinygo/src/runtime/print.go:86
>>>               /tmp/tinygo2458429745/main.exe.lto.main.o:(_runtime.printuint32)

tinygo:ld.lld: error: undefined symbol: ___divdi3
>>> referenced by /home/biehdc/code/tinygo/src/runtime/runtime_windows.go:134
>>>               /tmp/tinygo2458429745/main.exe.lto.main.o:(_runtime.runMain)
failed to run tool: ld.lld

I tried importing the relevant patches from upstream llvm into the local version which itself apparenly is a fork of llvm with patches and then recompile llvm, but it did not pick up on the option.

[aykevl]

I tried preserving xmm6 and xmm7 as mentioned here and also a hail mary approach with pushf/popf and pusha/popa, but it didnt help.

xmm6 and xmm7 aren't mentioned there, where did you get them from?
In any case, the registers that need to be preserved are the exact opposite of what need to be preserved in a regular function call. So with this description:

Functions must preserve all registers, except for eax, ecx, and edx, which can be changed across a function call, and esp, which must be updated according to the calling convention.

...means that stack switching needs to preserve eax, ecx, edx, and esp. The rest will have been preserved already by the caller.

[aykevl]

How I usually approach this is by familiarizing myself closely with the used calling convention, and then single stepping through the stack switching code to see where it goes wrong.

[BieHDC]

xmm6 and xmm7 aren't mentioned there, where did you get them from?

Sorry, wrong link. Its that quote from stackoverflow XMM6 and 7 are call-preserved, the rest are call-clobbered scratch registers. but it was the only thing i found that so directly mentioned that and his source does not mention it at all.

In any case, the registers that need to be preserved are the exact opposite of what need to be preserved in a regular function call. So with this description:

Functions must preserve all registers, except for eax, ecx, and edx, which can be changed across a function call, and esp, which must be updated according to the calling convention.

...means that stack switching needs to preserve eax, ecx, edx, and esp. The rest will have been preserved already by the caller.

Oh i see, i need to preserve the so-called scratch registers.
If the different conventions end up being an issue, that would just mean i have to account for all then?

[aykevl]

Oh i see, i need to preserve the so-called scratch registers.

Sorry I made a mistake before. When I said:

In any case, the registers that need to be preserved are the exact opposite of what need to be preserved in a regular function call. So with this description:

That's wrong, the same registers need to be saved that are typically saved in a function call.

Basically, the code is to switch to a different goroutine. It is called like a regular function call, and needs to preserve the same registers that are typically preserved in a function call. This means scratch registers can be clobbered, but all other registers need to be saved and restored.

That means, I believe the set of registers that need to be saved and restored are: https://godbolt.org/z/79WTzvWE5 (plus of course the stack pointer).

If the different conventions end up being an issue, that would just mean i have to account for all then?

There is probably only one calling convention, you just have to figure out which.

I also recommend looking at the stack switching code for other architectures, it might help to figure out the pattern.
For example: https://github.com/tinygo-org/tinygo/blob/release/src/internal/task/task_stack_386.S
Also make sure task_stack_386_windows.go matches the assembly file.

[BieHDC]

I also recommend looking at the stack switching code for other architectures, it might help to figure out the pattern. For example: https://github.com/tinygo-org/tinygo/blob/release/src/internal/task/task_stack_386.S Also make sure task_stack_386_windows.go matches the assembly file.

Currently its a 1:1 copy from this.

But until llvm is updated, this will be on hold.

[alchem1ster]

@BieHDC
If we're talking about --no-leading-underscore option, then llvm/llvm-project@fb19fa2
Maybe try to use MSVCRT build from https://winlibs.com
Its now LLVM/Clang/LLD/LLDB 17.0.4

[BieHDC]

@BieHDC If we're talking about --no-leading-underscore option, then llvm/llvm-project@fb19fa2 Maybe try to use MSVCRT build from https://winlibs.com Its now LLVM/Clang/LLD/LLDB 17.0.4

Yes it is about that option, but tinygo did not like linking against it.
I am using the make llvm-source llvm-build option and not the system-llvm version.
How i tried upstream llvm is to git checkout the latest llvm upstream release into the llvm-source folder and then running llvm-build again. It gave me a working suite, but it would refuse to link against it. Updating the go-llvm dependency did not resolve it.
Does it handle differently when you use the system llvm version?

[aykevl]

How i tried upstream llvm is to git checkout the latest llvm upstream release into the llvm-source folder and then running llvm-build again. It gave me a working suite, but it would refuse to link against it.

TinyGo needs to be updated to statically link against a particular LLVM version, unfortunately.

You could however try to apply the llvm/llvm-project@fb19fa2 patch manually to the LLVM 16 tree.

[BieHDC]

You could however try to apply the llvm/llvm-project@fb19fa2 patch manually to the LLVM 16 tree.

I tried that too together with that other patch that this one depends on, but in the end it still said that the option is unknown.

Edit: i did try to play with that again by hardcoding AddUnderscores to false or true and it results in ether it not finding the mingw function definitions or it not finding tinygos exports because they do not have underscores (for example tinygo:ld.lld: error: undefined symbol: _VirtualAlloc)

[BieHDC]

I am picking this up again as i got some time. Here is the current branch dev...BieHDC:tinygo:dev
Edit: Actually thats not the crash source, but something about loading dlls.

[aykevl]

Did some experimenting myself, see: #4629