Size and performance figures for LLVM?

Project Infrastructure LLVM Dev List Archives
#1

I gave a short presentation on LLVM for a couple of
people here, and they had questions I could not immediately
answer. The questions are rather obvious, so I'm hoping that
someone has already found out the answers. So here we go:

1) What is the relative size of LLVM bitcode files and the
    corresponding native binaries? Are there significant
    differences between targets (e.g. x86, ARM, Thumb,...)

2) What is the relative performance of code generated by LLVM
    and gcc respectively?

I am not looking for exact answers, rough ballpark figures
are quite sufficient.

#2

2) What is the relative performance of code generated by LLVM
   and gcc respectively?

See llvm.org/nightlytest Many testers run various benchmarks and produces comparison numbers daily.

#3

> 2) What is the relative performance of code generated by LLVM
> and gcc respectively?

See llvm.org/nightlytest

Which does not compare GCC vs. LLVM. (I haven't seen this announced as
the nightly tester's purpose either.)

Many testers

4 machines with a total of 15 comparisons have been reporting on a
regular basis.
Sorry, but this is far from "many".

run various benchmarks and
produces comparison numbers daily.

Regards,
Jo

#4

2) What is the relative performance of code generated by LLVM
  and gcc respectively?

See llvm.org/nightlytest

Which does not compare GCC vs. LLVM. (I haven't seen this announced as
the nightly tester's purpose either.)

Sure it does. See the GCC/LLC columns.

Many testers

4 machines with a total of 15 comparisons have been reporting on a
regular basis.
Sorry, but this is far from "many".

I count 12 testers reporting in the last day alone. What specifically are you looking for? If you care about a specific metric, you should measure it yourself.

-Chris

#5

>
>>
>>> 2) What is the relative performance of code generated by LLVM
>>> and gcc respectively?
>>
>> See llvm.org/nightlytest
>
> Which does not compare GCC vs. LLVM. (I haven't seen this announced as
> the nightly tester's purpose either.)

Sure it does. See the GCC/LLC columns.

I must be particularly blind to not find these.

Here is where I looked:

URL is http://llvm.org/nightlytest/ .
Machine (actually test configuration) names are links; following any of
these leads me to a URL like
http://llvm.org/nightlytest/machine.php?machine=230 . No GCC nor LLC
column though.

>> Many testers
>
> 4 machines with a total of 15 comparisons have been reporting on a
> regular basis.
> Sorry, but this is far from "many".

I count 12 testers reporting in the last day alone.

I based that on the "Test machines with recent submissions" section of
http://llvm.org/nightlytest .
I see 16 reports there. (Yesterday, I didn't count S3_AMD64 in because
it had just three results, but it seems that it is reporting on a
regular basis.)
I had tried to group machines by tester, giving me four (now five)
names: lattner, grawp, grue, laurov, and s3lap.

What specifically
are you looking for? If you care about a specific metric, you should
measure it yourself.

Sorry.

I think there's some lingering frustration on my side at work. I had
tried to get the nightly tester to run, found it difficult, had
postponed it until I find the time to properly diagnose the problems,
and now I find that my entire project was stalled because of this.

The concrete problem is that Ubuntu's way of doing cross compilations
seems incompatible with LLVM's way of using the autoconf machinery.
I've been told that this is Ubuntu's fault, but I'm sceptical: I have
seen LLVM's autoconf do things that autoconf shouldn't do if properly
set up.
Unfortunately, nobody stepped up to clear this up, and I'm increasingly
frustrated because I've been lacking the time to do it myself.

Devang's answer didn't seem to give any of the information he claimed it
would, and that simply tipped me off.

Sorry again, I didn't intend to vent my frustration.

Regards,
Jo

#6

Each row in that table is a separate nightly run. If you click on "view details" for any particular run, and then "See Full Test Results", you'll get the output complete w/ GCC/LLC column.

--Owen

#7

What specifically
are you looking for? If you care about a specific metric, you should
measure it yourself.

Sorry.

I think there's some lingering frustration on my side at work. I had
tried to get the nightly tester to run, found it difficult, had
postponed it until I find the time to properly diagnose the problems,
and now I find that my entire project was stalled because of this.

Surely you must appreciate that it isn't our problem :).

LLVM is an open source project. As such, you should consider all the people responding to your mails and doing stuff for you as volenteers. We are not slaves, we are people trying to help you out. Please be courteous as you would be to anyone else you are asking a favor of.

The concrete problem is that Ubuntu's way of doing cross compilations
seems incompatible with LLVM's way of using the autoconf machinery.
I've been told that this is Ubuntu's fault, but I'm sceptical: I have
seen LLVM's autoconf do things that autoconf shouldn't do if properly
set up.
Unfortunately, nobody stepped up to clear this up, and I'm increasingly
frustrated because I've been lacking the time to do it myself.

Ok. This is an open source project, which means you either do it yourself, convince someone to help you, or pay someone to do it for you. I strongly encourage you to word your emails politely, which increases chances of someone stepping forward to help you. I know nothing about ubuntu, or it's cross compilation scheme. If you would like some help, please describe in detail what the problem is.

Devang's answer didn't seem to give any of the information he claimed it
would, and that simply tipped me off.

Sorry again, I didn't intend to vent my frustration.

Everyone has a bad day, no worries.

-Chris

#8

Devang Patel wrote:

See llvm.org/nightlytest

Thanks, that is exactly the kind of information I was looking for.

#9

Sorry to step into this in the middle of a thread, but what exactly is LLVM's autoconf doing that "autoconf shouldn't do if properly set up"?

I have some experience with autoconf. I don't know if I've ever seen one "properly set up" before. :slight_smile: It's a tool that is used so that one can do their best to fit the compilation to a machine and its environment. As such, it's not great, but is frequently adequate for the job.

-bw

#10

Sorry to step into this in the middle of a thread, but what exactly is
LLVM's autoconf doing that "autoconf shouldn't do if properly set up"?

1) Variables like CC, CXX, CFLAGS don't work properly when submitted via
the command line. The handling seems to be inconsistent, i.e. it seems
that parts of the toolchain receive these flags, others don't.
This is just a cosmetic problem, of course.
2) The setting of --build should be the default for --host, and the
value of --host should be the default for --target. The build machinery
seems to take the defaults for --host and --target from elsewhere,
specifying just --build or --host will cause amd64 assembly to be fed to
an assembler running in i686 mode.
3) On a 64-bit Ubuntu, setting --build/host/target=i686-pc-linux-gnu
will work when compiling with gcc, but llvm-gcc will try to feed 64-bit
machine code to the assembler in this situation. At least that's what
happens when bootstrapping.
4) I also dimly recall having seen llvm-gcc use 64-bit library paths for
linking the 32-bit binaries (with ensuing hilarity, of course). I had
postponed further investigation with this issue until after I'd get the
assembly stage working - this approach had worked well with gcc, too.

I'm under the impression that problems 1-3 are caused by autoconf as
used in LLVM.
I don't know about problem 4; it might be because Ubuntu's handling of
library paths is inherently broken, or it might be a consequence of
problem 3.

If autoconf in LLVM can be cleaned up, I'd be really like to do it.
Having expertise in that area might just be what's been missing, so if
you're willing to give a helping hand to an autoconf newbie, this might
finally work out.

I have some experience with autoconf. I don't know if I've ever seen
one "properly set up" before. :slight_smile:

I can understand that, it's a beast and few if any people ever set it up
for cross compilation anyway.
I assume that a cross compilation capable tool like LLVM needs a
particularly careful setup in autoconf to avoid breakage like the above.

Regards,
Jo

#11

See llvm.org/nightlytest Many testers run various benchmarks
and produces comparison numbers daily.

Can I trust those numbers? For example, right now I'm looking at
http://llvm.org/nightlytest/machine.php?machine=230.

"CVS checkout time" might be wrong, as LLVM is now in SVN.

This column also jitters heavily. However, it's not really
important.

"Configure time CPU": is always 0, which must be a blantant
lie :slight_smile:

"Build time CPU" is usually higher than "Build time wall". Isn't
wall=cpu+system, so do we have negative system times here? I
could be completely wrong, but this looks strange. However, for
the Dejagnu test, it hold's that CPU time < wall time.

Now I look at Chris' PPC box:
http://llvm.org/nightlytest/machine.php?machine=153. And here
suddenly "Dejagnu CPU time" > "Dejagnu wall time". Now I'm
confused.

#12

SMP build hosts?

— Gordon

#13

Sorry to step into this in the middle of a thread, but what exactly is
LLVM's autoconf doing that "autoconf shouldn't do if properly set up"?

1) Variables like CC, CXX, CFLAGS don't work properly when submitted via
the command line. The handling seems to be inconsistent, i.e. it seems
that parts of the toolchain receive these flags, others don't.
This is just a cosmetic problem, of course.

Is this specifying CC, CXX, and CFLAGS during the "configure" process
or during the "make" process? The Makefiles seem to be using them in a
fairly consistent manner, as far as I can see. Which parts of the
toolchain don't get the flags? Also, are we talking LLVM or LLVM-GCC?

2) The setting of --build should be the default for --host, and the
value of --host should be the default for --target. The build machinery
seems to take the defaults for --host and --target from elsewhere,
specifying just --build or --host will cause amd64 assembly to be fed to
an assembler running in i686 mode.

I don't have much of an explanation for this. I would suggest
specifying all three (build/host/target) parameters if you're running
into this problem. It might help in the short term while we figure out
the issue.

3) On a 64-bit Ubuntu, setting --build/host/target=i686-pc-linux-gnu
will work when compiling with gcc, but llvm-gcc will try to feed 64-bit
machine code to the assembler in this situation. At least that's what
happens when bootstrapping.

(We derive our configure script for llvm-gcc from gcc's configure
script, so there shouldn't be any drastic changes.) If you're
cross-compiling, I don't think you can do a bootstrap. If you're
compiling natively, you don't need to specify the build/host/target
flags.

Try disabling bootstrapping to see if this helps your problem.

4) I also dimly recall having seen llvm-gcc use 64-bit library paths for
linking the 32-bit binaries (with ensuing hilarity, of course). I had
postponed further investigation with this issue until after I'd get the
assembly stage working - this approach had worked well with gcc, too.

I'm under the impression that problems 1-3 are caused by autoconf as
used in LLVM.
I don't know about problem 4; it might be because Ubuntu's handling of
library paths is inherently broken, or it might be a consequence of
problem 3.

If autoconf in LLVM can be cleaned up, I'd be really like to do it.
Having expertise in that area might just be what's been missing, so if
you're willing to give a helping hand to an autoconf newbie, this might
finally work out.

I suggest looking at the autoconf/configure.ac (for LLVM) and
configure.in & gcc/configure.ac (for LLVM-GCC) scripts. You can find
documentation here:

  http://www.gnu.org/software/autoconf/manual/autoconf.html

The documentation will help you understand what the different macros
in the .ac/.in files are up to and how they work. I'll try to help as
much as I can, but I warn that I'm not an expert on GCC's configure
scripts (they do a lot of...interesting things there).

I have some experience with autoconf. I don't know if I've ever seen
one "properly set up" before. :slight_smile:

I can understand that, it's a beast and few if any people ever set it up
for cross compilation anyway.
I assume that a cross compilation capable tool like LLVM needs a
particularly careful setup in autoconf to avoid breakage like the above.

-bw

#14

See llvm.org/nightlytest Many testers run various benchmarks
and produces comparison numbers daily.

Just an FYI.. I've been working on the nightly tester a bit and you can also view it here. Its much faster.:
http://llvm.org/nightlytest2/

PLEASE KEEP IN MIND THAT THIS IS NOT FINISHED AND I DO NOT CLAIM IT WORKS 100%! I plan on sending an announcement once its completed. Do not send me bug reports or anything :wink:

Can I trust those numbers? For example, right now I'm looking at
http://llvm.org/nightlytest/machine.php?machine=230.
"CVS checkout time" might be wrong, as LLVM is now in SVN.

Its svn time, but its just labled wrong.

"Configure time CPU": is always 0, which must be a blantant
lie :slight_smile:
"Build time CPU" is usually higher than "Build time wall". Isn't
wall=cpu+system, so do we have negative system times here? I
could be completely wrong, but this looks strange. However, for
the Dejagnu test, it hold's that CPU time < wall time.

I believe that the cvs/svn and configure time are just in the wrong columns. I noticed this when I was rewriting the scripts. So SVN cpu time is 0.

-Tanya

#15

Bill Wendling wrote:

#16

>
>> Sorry to step into this in the middle of a thread, but what exactly is
>> LLVM's autoconf doing that "autoconf shouldn't do if properly set up"?
>
> 1) Variables like CC, CXX, CFLAGS don't work properly when submitted via
> the command line. The handling seems to be inconsistent, i.e. it seems
> that parts of the toolchain receive these flags, others don't.
> This is just a cosmetic problem, of course.

Is this specifying CC, CXX, and CFLAGS during the "configure" process
or during the "make" process?

That was during configure.

Which parts of the
toolchain don't get the flags? Also, are we talking LLVM or LLVM-GCC?

Both.
I seem to recall that LLVM and LLVM-GCC were failing in slightly
different ways, but I'll have to verify this more precisely.

Would --debug=j be a good way to log all commands as they are executed?
Somehow I overlooked that option and didn't use it during my earlier
tests, which made diagnosing this kind of problem somewhat difficult.

> 2) The setting of --build should be the default for --host, and the
> value of --host should be the default for --target. The build machinery
> seems to take the defaults for --host and --target from elsewhere,
> specifying just --build or --host will cause amd64 assembly to be fed to
> an assembler running in i686 mode.

I don't have much of an explanation for this. I would suggest
specifying all three (build/host/target) parameters if you're running
into this problem. It might help in the short term while we figure out
the issue.

I've been doing that all the time, and it got LLVM unwedged.
It wasn't enough to get LLVM-GCC to compile though.

> 3) On a 64-bit Ubuntu, setting --build/host/target=i686-pc-linux-gnu
> will work when compiling with gcc, but llvm-gcc will try to feed 64-bit
> machine code to the assembler in this situation. At least that's what
> happens when bootstrapping.

(We derive our configure script for llvm-gcc from gcc's configure
script, so there shouldn't be any drastic changes.) If you're
cross-compiling, I don't think you can do a bootstrap.

Oh. What would be the problems with that

If you're
compiling natively, you don't need to specify the build/host/target
flags.

Try disabling bootstrapping to see if this helps your problem.

I wanted to avoid issues with compiling LLVM for 64 bits. (I'm running a
64-bit Linux here.)

There was a warning that you have to disable PIC generation in that
situation, and I wasn't sure whether that wouldn't get me into other
kinds of trouble.

> If autoconf in LLVM can be cleaned up, I'd be really like to do it.
> Having expertise in that area might just be what's been missing, so if
> you're willing to give a helping hand to an autoconf newbie, this might
> finally work out.
>
I suggest looking at the autoconf/configure.ac (for LLVM) and
configure.in & gcc/configure.ac (for LLVM-GCC) scripts. You can find
documentation here:

  http://www.gnu.org/software/autoconf/manual/autoconf.html

The documentation will help you understand what the different macros
in the .ac/.in files are up to and how they work. I'll try to help as
much as I can, but I warn that I'm not an expert on GCC's configure
scripts (they do a lot of...interesting things there).

:slight_smile:

Thanks. I hope to find some time for this tomorrow.

Regards,
Jo

#17

>
>> Sorry to step into this in the middle of a thread, but what exactly is
>> LLVM's autoconf doing that "autoconf shouldn't do if properly set up"?
>
> 1) Variables like CC, CXX, CFLAGS don't work properly when submitted via
> the command line. The handling seems to be inconsistent, i.e. it seems
> that parts of the toolchain receive these flags, others don't.
> This is just a cosmetic problem, of course.

Is this specifying CC, CXX, and CFLAGS during the "configure" process
or during the "make" process?

That was during configure.

Which parts of the
toolchain don't get the flags? Also, are we talking LLVM or LLVM-GCC?

Both.
I seem to recall that LLVM and LLVM-GCC were failing in slightly
different ways, but I'll have to verify this more precisely.

Would --debug=j be a good way to log all commands as they are executed?
Somehow I overlooked that option and didn't use it during my earlier
tests, which made diagnosing this kind of problem somewhat difficult.

I've never used the "--debug=j" option before. I didn't know it
existed either. :slight_smile: That might help matters.

> 2) The setting of --build should be the default for --host, and the
> value of --host should be the default for --target. The build machinery
> seems to take the defaults for --host and --target from elsewhere,
> specifying just --build or --host will cause amd64 assembly to be fed to
> an assembler running in i686 mode.

I don't have much of an explanation for this. I would suggest
specifying all three (build/host/target) parameters if you're running
into this problem. It might help in the short term while we figure out
the issue.

I've been doing that all the time, and it got LLVM unwedged.
It wasn't enough to get LLVM-GCC to compile though.

> 3) On a 64-bit Ubuntu, setting --build/host/target=i686-pc-linux-gnu
> will work when compiling with gcc, but llvm-gcc will try to feed 64-bit
> machine code to the assembler in this situation. At least that's what
> happens when bootstrapping.

(We derive our configure script for llvm-gcc from gcc's configure
script, so there shouldn't be any drastic changes.) If you're
cross-compiling, I don't think you can do a bootstrap.

Oh. What would be the problems with that

For llvm-gcc, the bootstrapping takes the compiler it built and tries
to rebuild the compiler using those executables. If you're building
for another target, the compiler you built won't run on your system.

If you're
compiling natively, you don't need to specify the build/host/target
flags.

Try disabling bootstrapping to see if this helps your problem.

I wanted to avoid issues with compiling LLVM for 64 bits. (I'm running a
64-bit Linux here.)

There was a warning that you have to disable PIC generation in that
situation, and I wasn't sure whether that wouldn't get me into other
kinds of trouble.

The only way to find out is to try. :slight_smile:

> If autoconf in LLVM can be cleaned up, I'd be really like to do it.
> Having expertise in that area might just be what's been missing, so if
> you're willing to give a helping hand to an autoconf newbie, this might
> finally work out.
>
I suggest looking at the autoconf/configure.ac (for LLVM) and
configure.in & gcc/configure.ac (for LLVM-GCC) scripts. You can find
documentation here:

  http://www.gnu.org/software/autoconf/manual/autoconf.html

The documentation will help you understand what the different macros
in the .ac/.in files are up to and how they work. I'll try to help as
much as I can, but I warn that I'm not an expert on GCC's configure
scripts (they do a lot of...interesting things there).

:slight_smile:

Thanks. I hope to find some time for this tomorrow.

-bw

#18

What variable are you setting? Here's what the clang project uses in its Makefiles

CPPFLAGS += -I$(PROJ_SRC_DIR)/../../include

I *think* that specifying CPPFLAGS on the command line (like this

  $ make CPPFLAGS=-I/my/own/include/dir

will replace the CPPFLAGS' initial value in the Makefiles.

-bw

#19

>> If you're
>> cross-compiling, I don't think you can do a bootstrap.
>
> Oh. What would be the problems with that
>
For llvm-gcc, the bootstrapping takes the compiler it built and tries
to rebuild the compiler using those executables.

That's the crucial step in a bootstrap does, right :slight_smile:

If you're building
for another target, the compiler you built won't run on your system.

Ah, right. I wasn't thinking far enough and too focused on my personal
situation (an amd64 machine that can run i686 code natively).

Regards,
Jo

#20

Bill Wendling wrote:

Bill Wendling wrote:
    

Sorry to step into this in the middle of a thread, but what exactly is
LLVM's autoconf doing that "autoconf shouldn't do if properly set up"?

1) Variables like CC, CXX, CFLAGS don't work properly when submitted via
the command line. The handling seems to be inconsistent, i.e. it seems
that parts of the toolchain receive these flags, others don't.
This is just a cosmetic problem, of course.

Is this specifying CC, CXX, and CFLAGS during the "configure" process
or during the "make" process? The Makefiles seem to be using them in a
fairly consistent manner, as far as I can see. Which parts of the
toolchain don't get the flags? Also, are we talking LLVM or LLVM-GCC?

As an example: When running make from my own project that uses llvm (and
the llvm makefiles), I am unable to add extra include paths to the
build. It seems that they get into the makefiles ok, but not through to
the compiler itself.

What variable are you setting? Here's what the clang project uses in its Makefiles

CPPFLAGS += -I$(PROJ_SRC_DIR)/../../include

I *think* that specifying CPPFLAGS on the command line (like this

  $ make CPPFLAGS=-I/my/own/include/dir

will replace the CPPFLAGS' initial value in the Makefiles.

I'm trying:

    $ make CPPFLAGS += -I/some/dir

I see the -I/some/dir showing up in Compile.CXX and Compile.C when I make 'printvars', however the build fails as if the path is not in the command line. When I build with TOOL_VERBOSE=1, I see that it's not making it through to the comand line at all.

I had a dig through the Makefiles but couldn't see where it was falling out. I get the same results if I add CPPFLAGS += -I/some/dir to the Makefile directly.

Thanks
dominic