XRay: Demo on x86_64/Linux almost done; some questions.

Project Infrastructure LLVM Dev List Archives
1

Hi,

The following three patches, when applied cleanly to LLVM+Clang+compiler-rt will now allow for building and running binaries with a very simple version of XRay included:

http://reviews.llvm.org/D21612 (compiler-rt)

http://reviews.llvm.org/D20352 (clang)

http://reviews.llvm.org/D19904 (llvm)

To use this on x86_64-unknown-linux-gnu you’d need to set the flags ‘-fxray-instrument’ and if you’d like to instrument all functions, set ‘-fxray-instruction-threshold=1’. Apply those flags to all your object files, and link statically for best effect.

Now, some questions before I continue:

  • What is the preferred way of controlling the behaviour of the runtime library for something like XRay? Do the sanitizers check/use environment variables or commandline-flags to control behaviour at runtime/init?
  • We would like to be able to trigger the patching/unpatching routines at runtime in a portable manner. In Linux and other UNIX-like environments signals (SIGUSR1 and SIGUSR2) might be good candidates for signalling the tracing infrastructure to start/stop at user-controlled times (useful for long-running servers). Is there a preference for this, or are there alternatives in this space that might make better sense?
  • Currently we are using ‘printf’ for the logging, but could use a simple thread-local in-memory log that flushes to disk when full. Any other preferred ways of doing this?
  • Documentation for how to use/run XRay may need to live in a central location, but since the changes to the LLVM pieces are currently in three different places, are there suggestions for where the docs should live?

Cheers

PS. An example log of how to use XRay with clang+llvm+compiler-rt with the above patches:

[16-06-28 17:21:02] dberris@dberris: ~/xray/llvm-build% cat test.cc
#include
#include

[[clang::xray_always_instrument]] void foo() { std::printf(“Hello, XRay!\n”); }

[[clang::xray_never_instrument]] void bar() { std::printf(“Not instrumented\n”); }

extern void baz(); // defined in other.cc

int main(int argc, char* argv) {
printf(“main has started.\n”);
bar();
foo();
baz();
}
[16-06-28 17:30:13] dberris@dberris: ~/xray/llvm-build% cat other.cc
#include

[[clang::xray_always_instrument]] void baz() {
std::printf(“Welcome!\n”);
}

[16-06-28 17:30:16] dberris@dberris: ~/xray/llvm-build% ./bin/clang -c test.cc -x c++ -std=c++11 -fxray-instrument
[16-06-28 17:30:34] dberris@dberris: ~/xray/llvm-build% ./bin/clang -c other.cc -x c++ -std=c++11 -fxray-instrument
[16-06-28 17:30:46] dberris@dberris: ~/xray/llvm-build% ./bin/clang -o test.bin test.o other.o -fxray-instrument
[16-06-28 17:30:55] dberris@dberris: ~/xray/llvm-build% ./test.bin
__xray_instr_map@0x400f5b…0x400fdb
400cf0 E * @function(400cf0)
400d1c X * @function(400cf0)
400da0 E * @function(400da0)
400dcc X * @function(400da0)
main has started.
Not instrumented
4133: [9699491610577808] E1
Hello, XRay!
4133: [9699491610592074] X1
4133: [9699491610599840] E2
Welcome!
4133: [9699491610611254] X2

2

- What is the preferred way of controlling the behaviour of the runtime
library for something like XRay? Do the sanitizers check/use environment
variables or commandline-flags to control behaviour at runtime/init?

The sanitizers each use a different, single environment variable:

IMO this works pretty well, and it supports multiple flags.

- We would like to be able to trigger the patching/unpatching routines at
runtime in a portable manner. In Linux and other UNIX-like environments
signals (SIGUSR1 and SIGUSR2) might be good candidates for signalling the
tracing infrastructure to start/stop at user-controlled times (useful for
long-running servers). Is there a preference for this, or are there
alternatives in this space that might make better sense?

No idea, maybe others have a better idea.

- Currently we are using 'printf' for the logging, but could use a simple
thread-local in-memory log that flushes to disk when full. Any other
preferred ways of doing this?

Should XRay use libc? Are there re-entrancy issues that we should be
worried about? Most sanitizer code avoids calling libc directly, and most
functionality (printf) is implemented internally. Maybe you should link
sanitizer_common and use that.

- Documentation for how to use/run XRay may need to live in a central
location, but since the changes to the LLVM pieces are currently in three
different places, are there suggestions for where the docs should live?

The sanitizer docs are kind of spread out, unfortunately. There's some in
clang, some on the github wiki, and probably some in LLVM. Personally I
think the documentation should be in clang in rst.

3

Awesome, thanks!

I suspect the bare minimum here would be to expose a single function that could be invoked to enable or disable tracing. Then that could be called in many different ways according to the application or through some supported, configurable, with sane defaults for the platforms supported.

I wondered about that, and I think avoiding libc as much as possible is a good thing to do.

I’ll work with sanitizer_common and re-use stuff from there instead, thanks for the pointer. :slight_smile:

I think following current practice is fine for consistency (i.e. document the many different parts in many different places) but then have some sort of high-level documentation to present the use-cases and user scenarios. I’m fine with the user-facing documentation to be in clang too (i.e. how to xray-enable your application/libraries, then how to configure, etc.).

Would love to know what others think about this particular point, but will proceed to add documentation through clang in rst.

Cheers