compile linux kernel

Hi,

I'm trying to compile linux-2.6.23.16 with llvm-2.3. Is it at all
possible? I get "Not an ELF" error. I pass "-emit-llvm" option to spit
LLVM IR, which can be JITed at runtime

$ make CROSS_COMPILE=llvm- CFLAGS="-O2 -emit-llvm"
Error: not ELF
make[2]: *** [scripts/mod/elfconfig.h] Error 1
make[1]: *** [scripts/mod] Error 2
make: *** [scripts] Error 2

using the verbose option -

$ make V=1 CROSS_COMPILE=llvm- CFLAGS="-O2 -emit-llvm"
/bin/bash scripts/checksyscalls.sh llvm-gcc
-Wp,-MD,./.missing-syscalls.d -nostdinc -isystem
/home/ashish/llvm/llvm-gcc4.2-2.3.source/obj/../install/lib/gcc/x86_64-unknown-linux-gnu/4.2.1/include
-D__KERNEL__ -Iinclude -include include/linux/autoconf.h -O2
-emit-llvm -D"KBUILD_STR(s)=#s"
-D"KBUILD_BASENAME=KBUILD_STR(missing_syscalls)"
-D"KBUILD_MODNAME=KBUILD_STR(missing_syscalls)"
make -f scripts/Makefile.build obj=scripts
make -f scripts/Makefile.build obj=scripts/genksyms
make -f scripts/Makefile.build obj=scripts/mod
  scripts/mod/mk_elfconfig x86_64 < scripts/mod/empty.o >
scripts/mod/elfconfig.h
Error: not ELF
make[2]: *** [scripts/mod/elfconfig.h] Error 1
make[1]: *** [scripts/mod] Error 2
make: *** [scripts] Error 2

$cat scripts/Makefile.build
hostprogs-y := modpost mk_elfconfig
always := $(hostprogs-y) empty.o
modpost-objs := modpost.o file2alias.o sumversion.o
# dependencies on generated files need to be listed explicitly
$(obj)/modpost.o $(obj)/file2alias.o $(obj)/sumversion.o: $(obj)/elfconfig.h
quiet_cmd_elfconfig = MKELF $@
      cmd_elfconfig = $(obj)/mk_elfconfig $(ARCH) < $< > $@
$(obj)/elfconfig.h: $(obj)/empty.o $(obj)/mk_elfconfig FORCE
  $(call if_changed,elfconfig)
targets += elfconfig.h

It seems like Makefile.build is trying to make an ELF file out of
scripts/mod/empty.o which has been compiled using LLVM to emit LLVM
code.

How to fix this? Am I missing something? Please help.

Thanks,
Ashish

If you want to build a kernel you can't use the JIT anyway, so why not
just build native code?

Best regards,
--Edwin

I plan to use virtualization infrastructure to JIT VM Kernel Code.
Michael Engel has done that in the past to have dynamic aspect in the
Kernel: http://llvm.org/pubs/2005-03-14-ACP4IS-AspectsKernel.pdf

He used L4 based Hypervisor and Virtual Machines. Can this not be done
or am I missing something?

-Ashish

I think you'll need to modify the kernel build scripts to not expect ELF
output, such as mk_elfconfig.
LLVM outputs its own bytecode format when you use -emit-llvm, and not an
ELF format.

Best regards,
--Edwin

can kernel b JITed using hypervisor at runtime??? also can llvm be run
on l4 as native L4 application? which flavor of l4 does it support?

any doc on how to go about it? should i just remove mk_elf from the
makefile?? i actually tried removing mk_elf from the build scripts but
then it fails when it doesn't find global symbols emitted during
preprocessing using ## e.g. per_cpu_## in percpu.h in the kernel.

thanks,
ashish

Yes, but it requires significant hacking, and the result for 2.6 is a
mostly bitcode kernel with a few userspace shared libraries linked in
as objcode (yes, the kernel builds .so files and includes them in the
binary). There are a few changes here and there you can make to the
kernel code to produce significantly nicer bitcodes too (e.g. rather
than having module level asm that makes symbols weak, you can just
have gcc do it).

The modification to the make system is not difficult, but it requires
some work and an understanding of the kernel build system.

Andrew

Thanks for the help. I've a couple of questions though:

How does LLVM deal with inline assembly?

I'm trying to compile kernel and I get this error probably because
LLVM is not able to handle inline assembly. I'm using LLVM-2.3

code snippet from "arch/x86_64/kernel/asm-offsets.c"
....

#define DEFINE(sym, val) \
        asm volatile("\n->" #sym " %0 " #val : : "i" (val))
....

    DEFINE(__NR_syscall_max, sizeof(syscalls) - 1);
    return 0;

# make CROSS_COMPILE=llvm- V=1

  llvm-gcc -Wp,-MD,arch/x86_64/kernel/.syscall.o.d -nostdinc -isystem
/home/ashish/llvm/llvm-gcc4.2-2.3.source/obj/../install/lib/gcc/x86_64-unknown-linux-gnu/4.2.1/include
-D__KERNEL__ -Iinclude -include include/linux/autoconf.h -emit-llvm
-Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing
-fno-common -Werror-implicit-function-declaration -O2 -mtune=generic
-m64 -mno-red-zone -mcmodel=kernel -pipe -Wno-sign-compare
-fno-asynchronous-unwind-tables -funit-at-a-time -mno-sse -mno-mmx
-mno-sse2 -mno-3dnow -maccumulate-outgoing-args
-fomit-frame-pointer -g -fno-stack-protector
-Wdeclaration-after-statement -Wno-pointer-sign
-D"KBUILD_STR(s)=#s" -D"KBUILD_BASENAME=KBUILD_STR(syscall)"
-D"KBUILD_MODNAME=KBUILD_STR(syscall)" -c -o
arch/x86_64/kernel/.tmp_syscall.o arch/x86_64/kernel/syscall.c
arch/x86_64/kernel/syscall.c:22: error: '__NR_syscall_max' undeclared
here (not in a function)
arch/x86_64/kernel/syscall.c:24: error: array index in initializer not
of integer type
arch/x86_64/kernel/syscall.c:24: error: (near initialization for
'sys_call_table')
In file included from arch/x86_64/kernel/syscall.c:25:
include/asm-x86_64/unistd.h:16: error: array index in non-array initializer
include/asm-x86_64/unistd.h:16: error: (near initialization for
'sys_call_table')
include/asm-x86_64/unistd.h:18: error: array index in non-array initializer

Note that I use "-emit-llvm" option with llvm-gcc. Is this correct of
am I missing something?
How can I fix this error?

Thanks,
Ashish

Thanks for the help. I've a couple of questions though:

How does LLVM deal with inline assembly?

It's been implemented piece by piece on an as-needed basis. At this point most of the things people actually use should work. llvm-gcc has seen the Linux kernel, so most usages in there ought to work.

The symptoms here look more like a preprocessor problem than an asm problem. Your asm snippet works fine with llvm's top of tree, and I suspect the real problem is something else. If you compile with -save-temps what does the .i file look like?

Thanks. I actually checked the IR code generated, it seems inline
assembly is being handled correctly. The preprocessing is also being
done correctly. Here is the asm-offsets.i file snippet..

...
builtin_offsetof(struct crypto_tfm,__crt_ctx)));
asm volatile("\n->" : : );
asm volatile("\n->" "__NR_syscall_max" " %0 " "sizeof(syscalls) - 1"
: : "i" (sizeof(syscalls) - 1));
return 0;
}
....

and here is the IR code snippet -

...
    tail call void asm sideeffect "\0A->__NR_syscall_max $0
sizeof(syscalls) - 1", "i,~{dirflag},~{fpsr},~{flags}"( i64 285 )
nounwind
    ret i32 0
}
....

However, the build system is trying to generate asm-offset.s from
asm-offset.c and then it populates asm-offsets.h header file for
correct values. Here is the code from Kbuild -

"define cmd_offsets
    (set -e; \
     echo "#ifndef __ASM_OFFSETS_H__"; \
     echo "#define __ASM_OFFSETS_H__"; \
     echo "/*"; \
     echo " * DO NOT MODIFY."; \
     echo " *"; \
     echo " * This file was generated by Kbuild"; \
     echo " *"; \
     echo " */"; \
     echo ""; \
     sed -ne $(sed-y) $<; \
     echo ""; \
     echo "#endif" ) > $@
endef
"
Since this does "sed" on asm-offsets.s file and retrieves the offsets
of the symbols, it fails when it gets LLVM generated "asm-offsets.s"
IR file. As a result the asm-offsets.h file generated is empty and
doesn't contain the symbol __NR_SYSCALLS_max.

If I use GCC to generate asm-offsets.s file, then the build system go
ahead but fails when it generates .so files as Andrew pointed out.

...
llvm-gcc -m elf_x86_64 -nostdlib -fPIC -shared
-Wl,-soname=linux-vdso.so.1 -Wl,--hash-style=sysv
-Wl,-z,max-page-size=4096 -Wl,-z,common-page-size=4096
-Wl,-T,arch/x86_64/vdso/vdso.lds arch/x86_64/vdso/vdso-start.o
arch/x86_64/vdso/vdso-note.o arch/x86_64/vdso/vclock_gettime.o
arch/x86_64/vdso/vgetcpu.o arch/x86_64/vdso/vvar.o -o
arch/x86_64/vdso/vdso.so
arch/x86_64/vdso/vvar.o: file not recognized: File format not recognized
collect2: ld returned 1 exit status
....

Now, If the build systems generates .so files, then it will be
difficult o actually generate fully arch-independent kernel code,
isn't it?

thanks,
ashish

If I use GCC to generate asm-offsets.s file, then the build system go
ahead but fails when it generates .so files as Andrew pointed out.

You have to generate this with gcc. .c -> .s and .s -> .o need gcc,
.c -> .o can use llvm-gcc. The combination has to be fixed up in
final linking.

Now, If the build systems generates .so files, then it will be
difficult o actually generate fully arch-independent kernel code,
isn't it?

A llvm compiled kernel is in no way arch-independent. It is very very
arch dependent.

Andrew

does that mean .o generated with gcc (.c -> .s and .s -> .o) will not
contain llvm ir?

i meant, final kernel bitcode ir arch independent and can be JIT with
any arch-specific backend. Is it not the case?

thanks,
ashish

No, this is not the case.

Just because you compile something to LLVM IR does not make the thing
you compiled work on every architecture.
You may even be able to retarget it to any architecture (it depends),
but this in no way means the result will *actually work*.
The LLVM IR generated by llvm-gcc is very architecture dependent.
Theoretically you could make a C compiler that was mostly C compliant
and produced architecture independent LLVM IR.
llvm-gcc is not this compiler, however.

Now, If the build systems generates .so files, then it will be
difficult o actually generate fully arch-independent kernel code,
isn't it?

Even if the LLVM IR itself was guaranteed to be
architecture-transparent (it isn't), all architectures use different
kernel code, there's no way to make a vmlinux binary which has the
code for all the architectures at once.
Even compiling the UML (using ARCH=um), where the kernel runs in
userspace like a regular program, won't make a 'universal binary' as
it depends on some x86[_64] inline assembly.

And the code of the Linux kernel compiled by LLVM is also very arch depended,
despite any loader and hw access will not work inside the standard LLVM JIT,
all the MMU and low-level access stuff will make no sense on a machine other
than what was enabled while compiling the kernel. E.g. you would need a whole
virtualized HW architecture ala Qemu (et al.) along the JIT.

Of course in theory one could setup such an infrastructure to JIT low level,
kernel and driver code, just throwing the i386 linux kernel source on LLVM
will not bring you there.

Yours,

Daniel Berlin wrote:

Watching this thread, it occurs to me that the "V" in "LLVM" is creating
confusion. So far as I know, LLVM is the first project to use "virtual"
to refer to the instruction set of the intermediate form. I understand
why this labeling made sense (sort of), but it was unfortunate. The
machine is abstract, not virtual, and the use of "virtual" here is so
out of keeping with any other use of the term that it really does
generate confusion.

Is this worth a FAQ entry?

Watching this thread, it occurs to me that the "V" in "LLVM" is creating
confusion. So far as I know, LLVM is the first project to use "virtual"
to refer to the instruction set of the intermediate form. I understand
why this labeling made sense (sort of), but it was unfortunate. The
machine is abstract, not virtual, and the use of "virtual" here is so
out of keeping with any other use of the term that it really does
generate confusion.

The topic whether LLVM bitcode is independent of the target platform was raised several times on the mailing list, but it was never discussed in detail. I would appreciate learning more about the following questions:

- Is the architecture dependence of LLVM IR only an artifact of llvm-gcc producing architecture dependent results?
- What architecture-specific features are present in the IR that prevent running the same LLVM bitcode on different architectures?

Is this worth a FAQ entry?

I would definitely appreciate such a FAQ entry.

Best regards,
  Christian

does that mean .o generated with gcc (.c -> .s and .s -> .o) will not
contain llvm ir?

The kernel uses .c -> .s steps to build target dependent asm including
things such as struct offsets. The kernel has some .s files (often
including the ones generated from .c files) that are target dependent.

Finally simple things such as interrupt controllers and clocks and io
architecture and mmu control are all target independent.

i meant, final kernel bitcode ir arch independent and can be JIT with
any arch-specific backend. Is it not the case?

The SVA project (http://sva.cs.uiuc.edu/SVA_files/page0002.html see
top two papers) made inroads on this with significant extensions to
the instruction set to virtualize many hardware elements the OS must
deal with.

Andrew

- Is the architecture dependence of LLVM IR only an artifact of llvm-
gcc producing architecture dependent results?

No.
It also is an artifact of code compiling architecture and OS dependent
features based on what they detect at configure time
It is an artifact of compiling non-type safe languages.
It is an artifact of system headers including inline asm.
It is an artifact of the endianness of the system.

- What architecture-specific features are present in the IR that
prevent running the same LLVM bitcode on different architectures?

A better question is: what architecture-abstracting features would
make writing target independent LLVM bitcode easier? There is 1 that
I think is critical, and 3 more that would make life much easier
(though technically redundant).

hton and ntoh intrinsics. These are needed to allow target code to
deal with endianness in a target independent way. (Ok, you could
potentially write code that detected endiannes at runtime and chose
multiversioned code based on that, but that is ugly and optimization
prohibiting).

redundant, but greatly simplifying:

iptr aliased type. There are legitamate cases where you want to
perform arithmetic and comparisons on pointers that the semantics of
GEP make illegal so the only way to do so in a target independent way
is to either cast to an int you hope is >= than any pointer, or
violate the GEP semantics (which is generally works).

GBP instruction (GetBasePointer). The inverse of a GEP. A GEP
selects an offset into a object in a target independent way based on
the type. What GBP would do would be to get a pointer to the base of
an object based on a pointer to field, a type, and the same specifier
as the GEP would use to get the field.
x == GBP (GEP x, 0, 1, 1), typeof(x), 0, 1, 1
This would make upcasts or any conversion from an embedded object to a
parent object not need arch dependent offsets and raw pointer
manipulation. (yes you could figure out the offset from GEP off null
trick and use raw pointer manipulation and casts)

sizeof instruction. Again, you can use the GEP off null trick, but
this isn't very obvious, but since it doesn't involve raw pointer
manipulation.

Andrew

Hello Jonathan,

Actually it is the second project that I know of that used the term Virtual to describe the intermediate instruction set. Virtual Processor was part of a failed attempt by the Tao Group and Amiga Inc. to bring a common set of runtimes and instruction sets to handheld devices (called AmigaDE). Later, certain optimizations appeared in Java allowing it to catch on instead. In an effort to minimize their losses, Amiga released a subset of it with a hosted operating system called AmigaAnywhere and a few titles were released on it.

To make a long story short, VP Assembler was replaced by some GCC cross-compilers and renamed AmigaAnywhere 2.0 . For more information about this, refer to http://amiga.com/developers/ .

This is trivia however as there isn't even a Wikipedia article about VP Assembler now.

--Sam

hton and ntoh intrinsics. These are needed to allow target code to
deal with endianness in a target independent way. (Ok, you could
potentially write code that detected endiannes at runtime and chose
multiversioned code based on that, but that is ugly and optimization
prohibiting).

Why not add types with explicit endianess? A trick I use for reading binary
files across platforms is to define the types int32, int32_le and int32_be :
int32 is platform-native, _le and _be are little and big endian,
respectively. I use and #ifdef in my types.hpp to determine which of _le and
_be is a typedef for the standard uint32, and the other is implemented as a
class with operator int32(). "add i32_be %X, 8" looks elegant to me, and
quite easy (for someone writing the ir output to a text file, like me :slight_smile: to
bolt on to existing code.

- Sherief