Using LLVM to compile system programs

Chris, et al.,

I'm having a bit of difficulty using LLVM to compile system programs
(e.g. rsh) on FreeBSD 5.x. Basically 'lli' is bombing out with
"ERROR: Program used external function 'blah' which could not be
resolved!" I was hoping that you could please point me to a sample
Makefile that might both use external (i.e. system) libraries as well
as the LLVM libraries & programs. At this point, LLVM makefiles seem
clear, as do the makefiles that the system programs use, but merging
them is proving complicated. Thanks.

There are three basic options here:

1. Build all your code with LLVM.
2. Build the non-llvm parts into a dynamic library, and use "lli -load
    x.so <other options>" to load it.
3. Link the non-llvm parts into LLI.

The best one depends on what you're trying to do. If you're interested in the LLVM makefiles, there is a good doc in About — LLVM 16.0.0git documentation on the makefiles.

-Chris

Thanks, Chris.

> I'm having a bit of difficulty using LLVM to compile system programs
> (e.g. rsh) on FreeBSD 5.x.

There are three basic options here:

1. Build all your code with LLVM.
2. Build the non-llvm parts into a dynamic library, and use "lli -load
    x.so <other options>" to load it.
3. Link the non-llvm parts into LLI.

Option #2 turned out to be the easiest option for a single program,
however if I were interested in recompiling _all_ of the system
programs, I suspect that it would not be. However I assume, option #3
would require linking LLI with _all_ of the system's libraries, which
seems nutty, and option #1 is likely to fail for at least some system
libraries due to inline assembly. Do my assumptions seem correct?
Has an OS (or at least the sytem applications on it) ever been
compiled with LLVM?

Thanks,
Sean

libraries due to inline assembly. Do my assumptions seem correct?

yes

Has an OS (or at least the sytem applications on it) ever been
compiled with LLVM?

I'm not aware of anyone who has built a whole os with LLVM yet, e.g. due to inline asm issues. However, many system utils (e.g. finger and core utils) have been built with LLVM. The key difference is that we didn't try to use the JIT, so native code was no problem. :slight_smile:

If building native code is an option for you, try passing -Wl,-native or -Wl,-native-cbe to llvm-gcc.

-Chris

We haven't compiled anything containing inline assembly, but John Criswell has compiled a large part of the Linux kernel with LLVM (after replacing inline assembly with intrinsic functions). This is for a research project and is capable of booting the kernel on top of LLVM and running with limited functionality in multi-user mode.

To do this, we have defined an API of LLVM intrinsic functions that support the features a kernel normally gets from hardware via machine instructions. John implemented this API as a library on x86 and ported Linux to this API.

As a side note, a subset of this API can be used to support a thread library directly on top of LLVM.

--Vikram Adve

I'm not aware of anyone who has built a whole os with LLVM yet, e.g. due
to inline asm issues. However, many system utils (e.g. finger and core
utils) have been built with LLVM. The key difference is that we didn't
try to use the JIT, so native code was no problem. :slight_smile:

Though unfortunately, this still requires customizing all the
Makefiles to be LLVM-style, unless I'm missing something.... Now,
some sort of ability to batch-compile/instrument source from the
system source tree somehow by just specifying the directory would be
REALLY cool.

If building native code is an option for you, try passing -Wl,-native or
-Wl,-native-cbe to llvm-gcc.

Absolutely, native code is no problem. Better, in fact. And thanks,
that worked perfectly!

Most makefiles allow you to just add this to CFLAGS or something. Try 'make CFLAGS=-Wl,-native'. Alternatively, since this really needs to be passed in at link time, try LDFLAGS. CFLAGS is more widely supported, but LDFLAGS may also work.

-Chris