[RFC] Building LLVM for WebAssembly

LLVM Project
21

You could use cmake ... -DLLVM_TOOL_LLVM_DRIVER_BUILD=ON -DLLVM_DISTRIBUTION_COMPONENTS=clang;lld and that would create a multi-call driver with just clang and lld inside.

This ends up with clang, ld.lld and llvm binaries in {build_folder}/bin/, which all will be exactly the same. When calling them on the command line they will act a bit differently because of their file name. Internally however – with the “inprocess” branch above I’ve suggested – clang will be able to call lld internally (or any other tool that you might need later).

Also just worth mentionning LLVM_USE_SYMLINKS cmake option which will create the llvm binary, making clang and ld.lld symlinks to that binary.

2 Likes
22

Thanks! This is really exciting, I can see a lot of use for more expansive multi-call builds than just -fintegrated-lld. I’m a bit concerned though about the patchset since it seems fairly sizeable to me; do you think it’s something that will land quickly? Have you gathered feedback on it already?

I will definitely need -fintegrated-lld as soon as possible to ship a usable toolchain but I could carry that around as a patch on ToT if it’s just for a while until the “inprocess” branch lands.

23

Thank you for this!
This would be very interesting for online code playgrounds, like the popular godbolt.org. I have one myself at play.tinygo.org and being able to run the compiler client-side would give a number of benefits, if the compiler toolchain binaries aren’t too big.

1 Like
24

Adding to the collection of WebAssembly use cases: I could really use a WASI version of llvm-cxxfilt. I would like to embed it into the vscode-bazel extension to demangle C++ symbol names from coverage results

1 Like
25

They’re on the order of ~100M (uncompressed; haven’t tested compression yet but I’d expect 20-30M) for the entire distribution. Not tiny, but bearable, and they cache well.

I was spot on: a zstd-compressed tarball of clang and lld is 31M.

3 Likes
26

I built trunk clang and llvm targeting wasi a couple of weeks ago. Needed to pass various no threads arguments to cmake and hack the compiler-rt build to not check libc exists but otherwise I think it worked. At least I ran clang under wasm3 and it didn’t immediately fall over.

Keen to have that as a more out of the box solution, there probably is a tail of missing ifdef and cnake flags. Some cmake could probably be inferred from other arguments being passed, I remember some hand holding needed to disable libunwind.

Happy to review and/or implement part of this proposal, thanks for the thread!

2 Likes
27

If that is the ground reality for the platform, perhaps defined(__wasm__) might work better.

  • to avoid cost of CMake compile-time tests
  • make the platform requirement clear. HAVE_XXX is sometimes used for “we provide a similar API in a different name”

Is it cleaner to define LLVM_ON_UNIX for __wasm__ or not?

28

OK, I can do the change. Does everybody else agree?

(It will be defined(__wasi__) since defined(__wasm__) checks for the architecture and defined(__wasi__) checks for the environment. There could in principle be a non-WASI Wasm environment that is more Unix-like.)

I think it is, but I’ll check again now that I’ve done the complete port.

29

I’m a bit late to this thread but just wanted to say that I’m also generally supportive of this. And to mention that there is a non-WASI environment that’s (a bit) more Unix-like: namely Emscripten, which targets JS/Web environments (whereas WASI mostly targets non-web environment). They share many of the same limitations (no processes, async signals, sockets, etc), although Emscripten has a larger built-in library of stubs, and in some configurations filesystem support actually works. Because of this, Wasi and Emscripten currently have different OS fields in the target triple. So in many cases __wasi__ will be the right define to check for rather than __wasm__. I haven’t looked at the PR yet but will be happy to do so, to help with refining things.

1 Like
30

I recognize that; the PR uses both __wasm__ and __wasi__ where appropriate.

1 Like
31

I’ve updated the PR to use almost exclusively defined(__wasi__)/defined(__wasm__), with the exception of HAVE_SETJMP, where you could conceivably have both kinds of builds on WebAssembly depending on your needs.

1 Like
32

I’ve also updated the repository which contains the script to build the Wasm-hosted, Wasm-targeting Clang and a fully functional sysroot with libc and libc++. It allows a C++ “Hello, world” to be built, like this:

$ wasmtime run --dir wasi-prefix --dir . wasi-prefix/bin/clang.wasm -resource-dir wasi-prefix --sysroot wasi-prefix -fno-exceptions -O2 -c test.cc
$ wasmtime run --dir wasi-prefix --dir . -S threads wasi-prefix/bin/wasm-ld.wasm "-m" "wasm32" "-Lwasi-prefix/lib/wasm32-unknown-wasip1" "-Lwasi-prefix/lib/wasm32-wasip1" "wasi-prefix/lib/wasm32-wasip1/crt1-command.o" "test.o" "-lc++" "-lc++abi" "-lc" "wasi-prefix/lib/wasm32-unknown-wasip1/libclang_rt.builtins.a" "-o" "a.out"
$ wasmtime run a.out
Hello, world!
33

@whitequark Your latest tweet about this looks great! The patch also looks clean, love it’s just a +155 −21.

Could you please share the build instructions? For example, is -fintegrated-lld needed? Not sure where/how to pass. I assume at the moment this patch needs threads?

Alternatively, could you please share a binary? I’d love to see if I can replace the existing port I got in w2c2/examples/clang at main · turbolent/w2c2 · GitHub and get it to work with w2c2’s threading support.

Thanks again for your effort!

34

Give me a few minutes, I’m preparing something y’all could use easily.

Note that I’ve currently punted on thread support since it’s not super important for me and including wasi-sdk into my build has proven more difficult than expected; I’m building the entire sysroot myself in YoWASP Clang and I don’t enable threads since I don’t want to go through the mess of having multiple targets available. (The compiler driver is doing something weird to support that and I’m not in the mood to figure out exactly why today.)

1 Like
35

FWIW GitHub - olimpiadi-informatica/wasm-compilers: Tools to run LLVM in a WASI environment builds a wasi-threads version of llvm (plus a couple of other useful things) - it’s not based on @whitequark’s patches (yet), but it will be soon :slight_smile:

1 Like
36

OK, for everyone who wants to try this out, I’ve added a horrifically cursed “compiler driver driver” that makes the Wasm Clang look like any other Clang by intercepting the output of -### and redirecting the (impossible in the Wasm sandbox) linker subprocess call. To use it, open a shell on a Linux machine with CMake, cURL, git, Python 3, and Wasmtime installed, and run:

$ git clone https://github.com/YoWASP/clang yowasp-clang
$ cd yowasp-clang
$ ./build.sh
... wait about one hour...

Once this is all done, you should be able to have the following test succeed:

$ cat >test.cc <<END
#include <iostream>

int main() {
    std::cout << "Hello, world!" << std::endl;
}
END
$ ./wasi-prefix/bin/clang++ test.cc -o test.wasm
$ wasmtime test.wasm
Hello, world!

Note that the “compiler driver driver” (what’s installed as wasi-prefix/bin/clang and wasi-prefix/bin/clang++) is a truly vile Python script I wrote in half a hour and while it should work reasonably well, it may also break in some funny way I’m not responsible for. It should not be in any way dangerous (you can read the entire thing), just kind of fragile, don’t expect it to survive being fed to some random ./configure (although you may try, I’m curious).

2 Likes
37

I have now compiled yosys.wasm using clang.wasm! The built yosys.wasm is able to compile a blinky (hardware equivalent of Hello World) just as well as one compiled with native clang. I’m going to call this a success, I think!

For everyone who doesn’t want to build their own packages, this workflow run should have artifacts available in a matter of minutes now (caution: expands to multiple directories). It includes latest changes to the (sigh) compiler driver driver that let you compile a moderately complex project like Yosys nearly unmodified.

In my experience, there are two things to watch out for:

  • *.d files are probably broken because clang includes paths into the sysroot in them, and the sysroot is currently mounted under a path that make (running outside of the sandbox) won’t see. This is fixable but not fixed currently.
  • WASI doesn’t have realpath so sometimes the build will break in a place where both an absolute path and a relative path are expected to point to the same file, often in a somewhat obscure way. The workaround is to use either absolute or relative paths uniformly.
2 Likes
38

I realized I haven’t answered these questions.

-fintegrated-lld doesn’t exist yet. The “compiler driver driver” that I wrote is a crude replacement for a proper solution that @aganea was working on and that I really hope to use.

Only LLD does. It looks like Clang never spawns a thread or uses atomics and an LTO build optimizes all of them out or something like that–I’m not completely sure what’s happening, only that the resulting Wasm file doesn’t require threads after all.

1 Like
39

I did try looking into the threading issue a little bit more and found the following:

  • In general, LLVM_ENABLE_THREADS doesn’t guard the usage of types like std::mutex, just the actual spawning of threads. Because of this, it seems extremely invasive to even try to make things compile without -pthread
  • clang still ends up using atomic WASM instructions as a result but never actually spawns a thread
  • LLD ends up with potential calls to pthread_create in exactly two places:
    • lld::unlinkAsync – can this performance hack just be removed for WASI? It also appears as if the file doesn’t actually get unlinked if ThreadsRequested == 1???
    • ThreadPoolInterface::asyncImpl when creating an async future. Not sure what to do about this one, but it’s also only used in a handful of places in Mach-O support and in LTO support This code path seems to be unreachable. It involves the __make_async_assoc_state function somewhere within libc++, but the ThreadPoolInterface only calls std::async with std::launch::deferred

Could the need for threading be solved with better stubs inside WASI which permit use of types like std::mutex, implements all of them as no-ops, and refuses to spawn additional threads?

40

If I apply the following patch on top of whitequark’s commit 826c21d6070bbb629edc525340f10c2ad068f4b8 and build with LTO, the resulting WASM files no longer import wasi::thread-spawn, although they do still require atomic opcodes

diff --git a/lld/Common/Filesystem.cpp b/lld/Common/Filesystem.cpp
index c2d3644191c9..c6ef5df01beb 100644
--- a/lld/Common/Filesystem.cpp
+++ b/lld/Common/Filesystem.cpp
@@ -75,6 +75,8 @@ void lld::unlinkAsync(StringRef path) {
       sys::fs::remove(tmpName);
   }
   sys::fs::remove(path);
+#elif defined(__wasi__)
+  sys::fs::remove(path);
 #else
   if (parallel::strategy.ThreadsRequested == 1)
     return;