JIT, incremental 'linking' and global variables.

Hi,

[This time with the missing attachment]

I attempting (in the context of Cling, our C++ interpreter) to
incrementally compile and link user code,
including global variable declarations. For example, I would like to
compile, link, load and execute
the C++ statement:

       int i = 3;

and then compile, link, load and execute a C++ function that uses this
global variable, for example:

   extern int i;
   int main() {
     printf("%d\n",i);
   }

But I was unable to find a workable solution.

With the example attached (the Makefile, as is, works only on Macos),
you can reproduce my result by doing:

    tar xfz linking_test.tar.gz
    cd linking_test
    gmake

and then run 'tester' with 0 through 3 as argument and you should get:

$ ./tester 0
Loading declaration.cxx
Loading main.cxx
LLVM ERROR: Could not resolve external global address: i

$ ./tester 1
Loading declaration.cxx
Loading main.cxx
5

$ ./tester 2
Loading declaration.cxx
Loading main.cxx
5

$ ./tester 3
Loading declaration.cxx
[i] Failure: dlopen(main.so, 9): Symbol not found: _i
  Referenced from: /Users/pcanal/root_working/code/cling_src/test/main.so
  Expected in: dynamic lookup

The only 'working' combination is "tester 2", in an ideal world "tester
0" and "tester 3" should also
have printed the same output.

In 'tester 0', I tried to link the 2 source files independently. When
executing the 2nd module it does
not find the global variable defined in the 1st module.

In 'tester 1', I tried to link the 2nd module against the 1st module.
The execution works okay
__BUT__ all global initialization from the first module are 'redone'.

In 'tester 2', I tried using an actual shared library containing the
code in 'declaration.cxx' and it works
well (i.e. the loading of the module containing main.cxx properly finds
the 'compiled' global variable).

In 'tester 3', I tried to JIT the 1st module and then load a shared
library containing the code of main.cxx,
here the dynamic loader does not find the declaration that was made in
the 1st module. Is this model
(shared library being able to see symbol that have been 'JITed')
supported? If not, is there any plan to
ever support it? As an alternative, we can get this behavior if we use
as a 'just in compiler' a forked call
to g++/clang to generate a shared library; we were hoping to be able to
skip this fork steps.

Ideally I would like to get 'tester 0' and 'tester 3' to work as I
expected: i.e. same output as 'tester 2'.

Does anybody know what I missed or mis-configured or if it is simply not
supported (and if it is not, it is
planned to be supported)?

Thanks,
Philippe.

PS. As a side note, clang inappropriately 'optimize away' something like:

int func1() { return prinf("loading the file\n"); }
static int loader_1 = func1();
int loader_2 = func2();

where loading a shared library containing this file should lead to the
string "loading file" being printed
twice, while with clang it is printed only once (the static int ... is
not executed).

linking_test.tar.gz (4 KB)

Hi,

[This time with the missing attachment]

I attempting (in the context of Cling, our C++ interpreter) to
incrementally compile and link user code,
including global variable declarations. For example, I would like to
compile, link, load and execute
the C++ statement:

     int i = 3;

and then compile, link, load and execute a C++ function that uses this
global variable, for example:

extern int i;
int main() {
   printf("%d\n",i);
}

But I was unable to find a workable solution.

Ok.

In 'tester 0', I tried to link the 2 source files independently. When
executing the 2nd module it does
not find the global variable defined in the 1st module.

In 'tester 1', I tried to link the 2nd module against the 1st module.
The execution works okay
__BUT__ all global initialization from the first module are 'redone'.

In 'tester 2', I tried using an actual shared library containing the
code in 'declaration.cxx' and it works
well (i.e. the loading of the module containing main.cxx properly finds
the 'compiled' global variable).

In 'tester 3', I tried to JIT the 1st module and then load a shared
library containing the code of main.cxx,
here the dynamic loader does not find the declaration that was made in
the 1st module. Is this model
(shared library being able to see symbol that have been 'JITed')
supported? If not, is there any plan to
ever support it? As an alternative, we can get this behavior if we use
as a 'just in compiler' a forked call
to g++/clang to generate a shared library; we were hoping to be able to
skip this fork steps.

The JIT has a lot of default behavior. By default if you load a module that has a global variable with an initializer, the first reference to it will cause its initializer to get splatted out to memory and considered to be the canonical location for that variable.

If you don't want this to happen, you can easily customize this by poking ExecutionEngine::addGlobalMapping. With this API you can say "any reference to this global should use this physical address". You can also query to global address map etc to see where a previous instance of the global was splatted out (and possibly updated by executing code!) so that later modules can reference it.

PS. As a side note, clang inappropriately 'optimize away' something like:

int func1() { return prinf("loading the file\n"); }
static int loader_1 = func1();
int loader_2 = func2();

where loading a shared library containing this file should lead to the
string "loading file" being printed
twice, while with clang it is printed only once (the static int ... is
not executed).

it looks like clang codegen for this is wrong, I filed: http://llvm.org/bugs/show_bug.cgi?id=5396

Please file bugs when you run into them, thanks!

-Chris