Makefile.rules Changes / automake update (IMPORTANT!)

Hello,

I've closed PR106 (use automake) as WONTFIX. I've already delineated the problems with automake in previous posts but as of now, all the automake related stuff has been removed from the repository.

In an effort to start making our makefile system better, I've committed changes to Makefile.rules and a few library Makefiles that nearly double the speed of our compilations. I can now build all of LLVM (lib, utils, tools, runtime, examples) in just over 17 minutes on my system. This used to take over 30 minutes. There are two main speed ups: (a) we do automatic dependency checking in one pass instead of two, and (b) we use "ar" to archive libraries, not libtool. The first change makes the build generally go faster in all directories. The second change speeds up links slightly because libraries are properly searched for symbols. I have also removed redundant variable definitions and sub-shell invocations (replaced with GNU Make features).

I have tested these changes quite extensively including the Stacker, sample, llvm-test, Java and reopt projects. If you have another project, you MIGHT be impacted because of a few variable name changes.

All changes will (soon) be documented in a new "LLVM Makefiles" document that I'll write, but quickly, here's what changed from a USER's perspective:

* ExtraSource variable became BUILT_SOURCES (standard variable name)
* SourceDir variable eliminated (redundant with BUILD_SRC_DIR, few uses)
* Properly allow standard variables to be used (C,CFLAGS,CXX,CXXFLAGS,CPP,
   CPPFLAGS, LD, LDFLAGS, AR, ARFLAGS, etc) to alter compilation settings on a
   per directory basis.
* Build BUILT_SOURCES first (ensures *.td -> *.inc translation occurs)
* Dependency information is now, by default, generated on every compile. This
   keeps it constantly up to date. System headers are not tracked so as to
   reduce make's load time. There is no separate dependency generation pass.
* Implement auto-reconfiguration (if the configure script changes, you get
   an automatic reconfiguration instead of a message saying that you need to
   do so.)
* Implement auto update of Makefile.config (if you change Makefile.config.in
   it automatically gets regenerated to Makefile.config)
* Added tblgen build rules to Makefile.rules and removed from the X86 and
   PPC target Makefiles. There was some overlap. Standardizing these will
   encourage future targets to use them. These rules are available if you set
   the TARGET variable.
* $(DEST*) variables are gone unless they specifically refer to the
   installation destination. These conflicted with standard makefile usage
* $(BUILD_OBJ_DIR)/$(CONFIGURATION) became $(OBJDIR) to simplify it. This is
   where object files should go.
* $(LIBDIR) is where libraries should go
* $(TOOLDIR) is where tools (executables) should go
* VERBOSE=1 mode is now even quieter (a few rules were missing $(VERB)
* Build messages have been corrected and made more indicative
* The "prdirs" target has been replaced with the "printvars" target which
   prints values for interesting make variables, not just the directories.

Brian: I've emailed your one Makefile change that you need to use in WholeReoptimizer library after you take the Makefile.rules update.

Alkis: Some very minor changes were needed in your Java project, but since I don't have access to your cvs repository any more, I can't tell you what changed (my file timestamps are all the same for some reason). So, try building it, if it isn't obvious what's wrong, let me know.

Misha/Brian: I didn't try compiling LLVM-TV with this, but I doubt you'll run into any significant problems, the user changes are few

Everyone: llvm-test has successfully compiled and run ONCE with these changes. However, please note that llvm-test no longer depends on llvm/Makefile.rules. I copied the OLD Makefile.rules to llvm-test/Makefile.rules. The ONLY remaining dependency between llvm-test and llvm proper is the Makefile.confg(.in) file. This was the only "safe/quick" way to ensure I didn't break llvm-test. As time goes on we need to straighten out the Makefile mess in llvm-test but that's another project.

Reid.

Hi Reid, just a quick note while you're doing this, a while ago I ran
into a problem that the standard makefiles weren't building libraries
(like, say, a new pass) correctly on OS X - they were being built as
shared libraries and not bundles, so they couldn't be loaded with
dlopen. The discussion is in the archives if you want more details...

I fixed it locally by doing the following hack:

% diff Makefile.rules.old Makefile.rules
326c326
< Link := $(LIBTOOL) --mode=link $(CXX)

s/building libraries/building loadable modules/

Libaries work fine out of the box, it's modules you'd pass to e.g. opt
-load that don't.

Okay, so you're suggesting that for Darwin, we have a platform specific
parameter to CXX, -module, that builds the right kind of loadable shared
library (a module?). Nate, do you agree?

Reid.

The changes to Makefile.rules have been validated on three platforms
(Linux, Solaris, Darwin) AS LONG AS BUILD_OBJ_DIR != BUILD_SRC_DIR

If you build in the same directory that you check out into, you may
experience some "circular dependency" issues. I'm looking into this now.
In the mean time, just create a FRESH build directory and build there,
separately from your source directory.

Reid.

In an effort to start making our makefile system better, I've committed changes
to Makefile.rules and a few library Makefiles that nearly double the speed of
our compilations. I can now build all of LLVM (lib, utils, tools, runtime,
examples) in just over 17 minutes on my system. This used to take over 30

Nice!!

* Implement auto-reconfiguration (if the configure script changes, you get
   an automatic reconfiguration instead of a message saying that you need to
   do so.)

I haven't looked at your implementation, but does this work if the user
passes arguments (e.g. --with-llvmgccdir) to the configure script?

...

* Added tblgen build rules to Makefile.rules and removed from the X86 and
   PPC target Makefiles. There was some overlap. Standardizing these will
   encourage future targets to use them. These rules are available if you set
   the TARGET variable.

Nice! Thanks for all of the great improvements Reid!

-Chris

> In an effort to start making our makefile system better, I've committed changes
> to Makefile.rules and a few library Makefiles that nearly double the speed of
> our compilations. I can now build all of LLVM (lib, utils, tools, runtime,
> examples) in just over 17 minutes on my system. This used to take over 30

Nice!!

yeah, a little productivity enhancement never hurts.

> * Implement auto-reconfiguration (if the configure script changes, you get
> an automatic reconfiguration instead of a message saying that you need to
> do so.)

I haven't looked at your implementation, but does this work if the user
passes arguments (e.g. --with-llvmgccdir) to the configure script?

Yes, it does. It uses "config.status --recheck" which is supposed to
DTRT, including passing original arguments to configure.

> * Added tblgen build rules to Makefile.rules and removed from the X86 and
> PPC target Makefiles. There was some overlap. Standardizing these will
> encourage future targets to use them. These rules are available if you set
> the TARGET variable.

Nice! Thanks for all of the great improvements Reid!

You ain't seen nothing yet. This commit was just to clear up the
obfuscation and misuse. Next comes the documentation, and then the new
targets (dist, dist-check, dist-clean, package, install, uninstall,
check, etc.)

Reid