Difficulties preparing libtooling based workshop

Hi,

I'm trying to do a presentation next Wednesday (yes, in 3 days) for
the Utah Software Craftsmen group on building a C++ refactoring tool
using libtooling. This is a dry-run for the presentation I proposed
to C++ Now 2014 (proposal decisions will be announced Feb. 15th).
The tool is based on llvm/tools/clang/tools/extra/remove-cstr-calls.
I'm running into roadblocks that seem unnecessary.

1) I must build from source because the prebuilt binaries don't include
libtooling headers or libraries, only libclang. Why?

This is a live coding workshop and my participants will chew up 30-45
minutes of time just to download source and build the base libraries to
start coding.

2) following the "Getting Started" instructions fails the build
under Debug on Windows. The asan runtime has a line of code in it like
this in asan_win.cc:

#if defined(_DEBUG)
#error Please build the runtime with a non-debug CRT: /MD or /MT
#endif

...which means the whole solution fails to build in debug and this
contingency isn't handled in CMakefiles.txt

3) there are a bunch of tests that fail on Windows along the lines of:

'echo': command not found

...which in turn fails the build for the projects that run these
tests, and in turn causes the whole solution to fail.

Attempting to do exec (CreateProcess) on 'echo' is not going to work
on Windows because 'echo' is an internal command to cmd.exe and is not
an executable in your path. I'm not even certain it will work under
cygwin because echo is also an internal command under bash and is n ot
an executable in the path either.

A good chunk of my audience are Windows users and this isn't going to
leave a good taste in their mouth.

I can work around problems 2) and 3) myself. (I'm trying a build without
compiler-rt right now and this build problem must have been there for
a while because I found it in my experimental checkout from 5 months ago.)

I was able to build remove-cstr-calls without compiler-rt's asan
runtime, so obviously that's not a requirement.

What's really more of a problem for me is the lack of prebuilt
binaries for libtooling. Is there enough packaging infrastructure in
place that I can get the necessary headers and libraries packaged up
with a simple command?

I did a check before the build was finished and I had up to 8 GB of built
files on my machine, so clearly providing a pre-built virtual machine or
a downloadable tarball simply isn't going to cut it.

I also get this:

284>------ Build started: Project: check-clang, Configuration: Debug Win32 ------
....
284> Unexpected Failures: 20

Many of which appear to be due to the 'echo' problem, but it's unclear
how many of them are real failures.

  • timur (on asan_win.cc)

+Hans who did the CMake files for ASan on Windows.

Hi,

I worked Sunday preparing this presentation. This is still a work in
progress... this will be available even if the talk is not accepted for
C++ Now 2014. I have a walkthrough up to the point of making the
refactoring tool perform the first simple transformation.

Feedback on the presentation is most welcome.

<https://github.com/LegalizeAdulthood/remove-void-args&gt;

Hi Richard,

I think the truth of the matter is that not many people have worked on
the tooling stuff in a Windows environment, and that's why it's such a
rough experience.

I'm trying to do a presentation next Wednesday (yes, in 3 days) for
the Utah Software Craftsmen group on building a C++ refactoring tool
using libtooling. This is a dry-run for the presentation I proposed
to C++ Now 2014 (proposal decisions will be announced Feb. 15th).
The tool is based on llvm/tools/clang/tools/extra/remove-cstr-calls.
I'm running into roadblocks that seem unnecessary.

1) I must build from source because the prebuilt binaries don't include
libtooling headers or libraries, only libclang. Why?

The Windows binaries (both the 3.4 release and the weekly snapshots)
focus on shipping an LLVM-based toolchain, not headers and libraries.
It would be great to have the latter but it hasn't been first
priority.

This is a live coding workshop and my participants will chew up 30-45
minutes of time just to download source and build the base libraries to
start coding.

2) following the "Getting Started" instructions fails the build
under Debug on Windows. The asan runtime has a line of code in it like
this in asan_win.cc:

#if defined(_DEBUG)
#error Please build the runtime with a non-debug CRT: /MD or /MT
#endif

...which means the whole solution fails to build in debug and this
contingency isn't handled in CMakefiles.txt

This sounds like a bug that we should fix. In the meantime, as you've
noticed, you don't need compiler-rt for tooling.

3) there are a bunch of tests that fail on Windows along the lines of:

'echo': command not found

...which in turn fails the build for the projects that run these
tests, and in turn causes the whole solution to fail.

I run the test suite in cygwin. I don't know how well it works when
run in cmd.exe or from Visual Studio.

What's really more of a problem for me is the lack of prebuilt
binaries for libtooling. Is there enough packaging infrastructure in
place that I can get the necessary headers and libraries packaged up
with a simple command?

I don't know if the cmakefiles for tooling is set up to do this, but
what I would try is to build the "install" target (you might want to
set CMAKE_INSTALL_PREFIX first) and see if that includes everything
necessary for tooling. If it does, just zip that up. If not, the
cmakefiles need work.

Thanks,
Hans

Hi Hans,

In article <CAB8jPhdmfgBwhOw-HxhhXX8_m+0eaL5H4Mxd7dCvTpWhyQRuEw@mail.gmail.com>,
    Hans Wennborg <hans@chromium.org> writes:

> What's really more of a problem for me is the lack of prebuilt
> binaries for libtooling. Is there enough packaging infrastructure in
> place that I can get the necessary headers and libraries packaged up
> with a simple command?

I don't know if the cmakefiles for tooling is set up to do this, but
what I would try is to build the "install" target (you might want to
set CMAKE_INSTALL_PREFIX first) and see if that includes everything
necessary for tooling. If it does, just zip that up. If not, the
cmakefiles need work.

Thanks for this suggestion. I'll try it tonight and see how far I
get. If I need to do some CMakeLists.txt hacking, I'll give that a go
and contribute a patch for review after the workshop.

In particular, I don't want to distribute a whole pile of static libraries
in the standard toolchain build of LLVM and Clang.

There are a couple of ways to avoid it, both of which require work:
1. Split into a libLLVM.dll, clang.dll, and clangTooling.ll
2. Create a separate installation with the usual static libraries

IMO doing 1 is the best, because then we can shrink the installation
footprint even further by avoiding 4 copies of a statically linked clang:
clang.exe, clang-cl.exe, clang++.exe, and msbuild-bin/cl.exe

The problem is that there's no easy way to just blithely export everything
on Windows like we would do on Linux, and we don't have any existing
annotations for our C++ APIs.

In article <CACs=tyLv3xa6T=7SjcZobRzkdcXPW8U3zris_HwyXScYjAAT8g@mail.gmail.com>,
    Reid Kleckner <rnk@google.com> writes:

In particular, I don't want to distribute a whole pile of static libraries
in the standard toolchain build of LLVM and Clang.

There are a couple of ways to avoid it, both of which require work:
1. Split into a libLLVM.dll, clang.dll, and clangTooling.ll
2. Create a separate installation with the usual static libraries

IMO doing 1 is the best, because then we can shrink the installation
footprint even further by avoiding 4 copies of a statically linked clang:
clang.exe, clang-cl.exe, clang++.exe, and msbuild-bin/cl.exe

The problem is that there's no easy way to just blithely export everything
on Windows like we would do on Linux, and we don't have any existing
annotations for our C++ APIs.

This is something I can contribute longer-term. I'll take a look at
what's necessary for doing it.

In the meantime, I'm going to look into a means for packaging headers
and standard libraries for a refactoring tool separate from the
existing package for clang.

Hi,

In discussion with the organizers of Utah Software Craftsmen, we've
decided to postpone this libtooling workshop due to the excessive
setup time. This was really useful feedback because the same problem
would be faced at C++ Now 2014 if this session is accepted. (I prefer
a hands-on exploration of code instead of watching someone else's
power-point deck on a screen, don't you?)

So I'm going to finish off my existing walk through on github and then
take a look at the packaging. My intention is to contribute a patch
that will package up everything you need (headers and built libraries)
in order to write your own refactoring tool using libtooling without
requiring you to build clang yourself first. This would be an add-on
package to the existing binary packages.

We can tackle the idea of packaging a shared object to avoid repeated
copies of static libraries in executables and reduce package size as a
"package refactoring" that can be applied once the libraries are
packaged.

Thanks everyone for your suggestions so far.

-- Richard