Follow-up on: Dynamic updates of current executed code


I am coming back to the below discussion again, regarding the LLVM support of Ruby dynamics. The initial problem description is as follow, to refresh your memory :slight_smile: After that my questions come:

-----------BEGIN Initial problem description---------------

Problem is though, that the Ruby compiler is integrated in the compilation of the program being executed, to be able to parse & compile dynamic code at run-time. Therefore the calls to ExecutionEngine::getPointerToGlobal(F) need to be made in LLVM code. Here is a detailed simplistic example in pseudocode of what we want to do:

First Ruby code is entered at run-time, received as a string, parsed and compiled into the following code:

; External function
declare int %printf(sbyte*, ...)

; Custom function
int %puts_kernel( sbyte* %string )
%tmp.0 = call int (sbyte*, ...)* %printf( sbyte* %string )
ret int 0

This code is represented in the string variable
%dynamically_compiled_function_code below:

%dynamically_compiled_function_code = internal constant [LENGTH x sbyte] c"--String with the function code--\0A\00"

; Table of function pointer(s)
%kernel = type { int ( sbyte* )* }

int %main() {

; Create the kernel in memory, and get pointer to first function pointer
%theKernel = malloc %kernel
%FirstFunctionPTR = getelementptr %kernel* %theKernel, long 0, ubyte 0
;Load code
%myNewFunction = %getPointerToGlobal(%dynamically_compiled_function_code)

; Write memory address of myNewFunction() into kernel struct
store RETURNTYPE (PARAMTYPE*)* %myNewFunction, RETURNTYPE (PARAMTYPE*)** %FirstFunctionPTR

;Any code using first function element in %kernel is now_
;using dynamically updated function!?

ret int 0


The questionmark is at this pseudocode row:
%myNewFunction = %getPointerToGlobal(%dynamically_compiled_function_code)

Is there an llvm version of the getPointerToGlobal() function as outlined, and can the %myNewFunction pointer be used as described?
Also, does the getPointerToGlobal() take human readable code (.ll) or only binary byte code (.bc)? Is there a specification of how to write binary byte code directly, so we do not have to externally call the llvm-as utility?

Best regards
-----------END Initial problem description---------------

-------------------Previous Discussion:

Is there an llvm version of the getPointerToGlobal() function as
outlined, and can the %myNewFunction pointer be used as described?

Yes, see llvm/lib/ExecutionEngine/ExecutionEngine.cpp . It will compile
the given Function* to machine code, and return a pointer to you. In
your LLVM code, you will have to first cast your function pointer before
storing it, but it should work as you have it.

1) Can a modified version of ParseAssemblyFile(InputFilename) that takes a pointer to LLVM code in memory instead be used to generate a Function* for use by the getPointerToGlobal()? I understand that it currently returns a Module*. Is it possible to make it return an F* instead, and if so how would the implementation roughly look like?

2) Can this function be compiled into LLVM independently and used as a stand-alone function by the run-time compiler?

One thing is that you need to have a pointer to the currently-running
instance of ExecutionEngine to call methods on it... a easy quick hack
would be to create a static variable that the ExecutionEngine
constructor writes its own pointer (this) into, and have a C function
(in ExecutionEngine.cpp) that returns that pointer. Your code can then
access the currently-running ExecutionEngine through that pointer. There
should be a cleaner solution to this...

We aim for an integrated binary solution with compiler-application-LLVMjitter, (if at all possible), so my question is:

1) How is it possible to make LLVM calls to C++ methods? Can the Ruby-LLVM compiler, applied on itself thus residing in LLVM-form, call the getPointerToGlobal() using the pointer to the currently-running instance of ExecutionEngine from the fast hack above? :slight_smile: In this process it is passing the F* from the modified ParseAssemblyFile(InputFilename), finally getting a pointer to the function, which can be called using standard LLVM syntax?


Here's an idea:

In tools/llvm-as/llvm-as.cpp, you can see how the assembly parser really
does its job: it calls ParseAssemblyFile(InputFilename) which returns it
a Module*.

ParseAssemblyFile is defined in llvm/lib/AsmParser/Parser.cpp; something
similar to it, but one that accepts a string instead of a filename could
be implemented.

However, there is a catch (as I see it, Chris may correct me):

The entry to the parser is to parse a whole Module. The new function
that you write may reference other functions. When you parse a Module,
it has to be complete, so functions that are not defined in the module
are "external". Having a definition for those functions will make those
external functions different (in terms of their location in memory) and
hence different value for Function*, which will mess with the
ExecutionEngine as it uses a map Function* => memory address of
translation. Other problems would be with types, same idea.

(Is this making sense?)

A solution would be perhaps to use a different entry point into a parser
that would just parse a Function instead of a whole Module. It should
then resolve the types it finds to the already-parsed Module in memory.
I don't think this has been done yet.

Misha Brukman :: ::

I hope you are able to roughly describe this scenario! We do not aim for implementing it at this point, but only to know if it is possible, and roughly how.

Best regards
Anders Alexandersson
RubyComp project

getPointerToGlobal can not take either of those (a .bc or .ll file). It
must take an LLVM "Function*" to tell it what to compile.

I think the way you really want to approach this is to write your ruby
compiler in C++, and have a public function that you expose

Your compiler would lex and parse the ruby code in the string, translate
it to LLVM (to a Function *) and then ask the JIT to compile it for you,
returning a pointer to the function.