Hi everyone, Rafael has graciously given me some pointers for helping
out on the ARM/MC .s emission infrastructure, and I am volunteering to
do so.
It looks like as of yesterday, the MC obj emitter for ARM is also
incomplete (there does not seem to be a ARMMCCodeEmitter.cpp, for
example)
So if anyone already has started looking into this, I'd like to pool
info so as to not step on toes.
Any add'l tips and pointers would be greatly appreciated.
Thanks!
-jasonkim
Hi Jason,
I've just started actively working on this. Coordinating to get things moving even faster sounds great! Can you elaborate a bit on your ultimate goals and use cases are? That might help us better determine a natural breakdown and separation of tasks. Evan and Chris may have suggestions there, too, as I know they're both very interested in getting this stuff fleshed out and working properly.
My current high level plan is something like the following:
1. MC-ized Assembly Printer
a. Work through the "make check" failures using the MC inst printer. Some of these are false failures due to mnemonic choice differences, but most are real failures.
b. Repeat (a) except running the full nightly regression suite, not just "make check."
c. Clean up the printer and instruction lowering code to make much less use of the "Modifier" string (which causes lots of tight coupling between the instruction selection code and the asm printer).
d. Nuke the old asm printer once the above is complete and enable the MC printer as the One True Printer(tm).
2. Object File Emission
a. Add basic infrastructure for the object file emitter. i.e. just construct the classes needed and put in lots of asserts() and FIXMEs.
b. Add ARM-specific object file format bits.
c. Run through the test suite using the object file emitter, disassembling the output and comparing it to what comes out from the .s->assembler->disassembler execution path. Crush all differences.
3. JIT. Same basic approach as (1) to replace the current object code emitter with an MC based one.
a. Hand-wavey stuff. I haven't thought much about this bit yet except in broad terms.
b. More hand waving.
4. ???
5. Profit.
-Jim
Hi Jim!
Hi Jason,
I've just started actively working on this. Coordinating to get things moving even faster sounds great! Can you elaborate a bit on your ultimate goals and use cases are? That might help us better determine a natural breakdown and separation of tasks. Evan and Chris may have suggestions there, too, as I know they're both very interested in getting this stuff fleshed out and working properly.
It looks like the MC obj emission and MC .s emission are two naturally
related, but separate tasks. I think the overall goal of having the MC
.s/.o drive the entire flow is probably a great idea. Your plan for
MC.s and MC.o sounds spot on as well. I humbly suggest you tackle the
.s emission, and I tackle the .o emission - I haven't thought about
how this will interact with generating arch-specific .so's directly
from LLVM...
(I am sure there's a blog post about it somewhere on llvm.org 
My current high level plan is something like the following:
1. MC-ized Assembly Printer
a. Work through the "make check" failures using the MC inst printer. Some of these are false failures due to mnemonic choice differences, but most are real failures.
b. Repeat (a) except running the full nightly regression suite, not just "make check."
c. Clean up the printer and instruction lowering code to make much less use of the "Modifier" string (which causes lots of tight coupling between the instruction selection code and the asm printer).
d. Nuke the old asm printer once the above is complete and enable the MC printer as the One True Printer(tm).
2. Object File Emission
a. Add basic infrastructure for the object file emitter. i.e. just construct the classes needed and put in lots of asserts() and FIXMEs.
b. Add ARM-specific object file format bits.
c. Run through the test suite using the object file emitter, disassembling the output and comparing it to what comes out from the .s->assembler->disassembler execution path. Crush all differences.3. JIT. Same basic approach as (1) to replace the current object code emitter with an MC based one.
a. Hand-wavey stuff. I haven't thought much about this bit yet except in broad terms.
b. More hand waving.
Lots of handwavy stuff - fast incremental feedback based optimization <blah>
Hi Jim!
Hi Jason,
I've just started actively working on this. Coordinating to get things moving even faster sounds great! Can you elaborate a bit on your ultimate goals and use cases are? That might help us better determine a natural breakdown and separation of tasks. Evan and Chris may have suggestions there, too, as I know they're both very interested in getting this stuff fleshed out and working properly.
It looks like the MC obj emission and MC .s emission are two naturally
related, but separate tasks. I think the overall goal of having the MC
.s/.o drive the entire flow is probably a great idea. Your plan for
MC.s and MC.o sounds spot on as well. I humbly suggest you tackle the
.s emission, and I tackle the .o emission
Sounds perfectly reasonable to me. We can always adjust later if need be for some reason. Looking forward to working with you.
- I haven't thought about
how this will interact with generating arch-specific .so's directly
from LLVM...
Semi off the top of my head, I'd expect the normal code path for that to still go through the linker rather than being emitted directly. If nothing else, to resolve any symbols that need to be brought in from static libs. That said, in combination with the LTO and perhaps some additional restrictions (no static symbol dependencies, etc.), I don't see any reason why there couldn't be a .so emitter.
Regards,
Jim
Semi off the top of my head, I'd expect the normal code path for that to still go through the linker rather than being emitted directly. If nothing else, to resolve any symbols that need to be brought in from static libs. That said, in combination with the LTO and perhaps some additional restrictions (no static symbol dependencies, etc.), I don't see any reason why there couldn't be a .so emitter.
That is more or less the idea we were playing with. When producing a
.so for PNaCl, all files going into the library must be bitcode files,
but there can be external .so ELF dependencies. Using gold we should
be able to record the dependencies somewhere and output a combined
bitcode.
A custom translator can then be written that takes the combined
bitcode and the dependencies and produces a .so in one step. Ideally
the dependencies would be encoded in the .bc itself, so we would have
a single file going in for each .so.
Regards,
Jim
Cheers,
Excellent. I'll look forward to seeing what you come up with.
-Jim
Hi Jason, Jim, Rafael,
We had plans to work on ARM MC (asm and elf), good to see there is an
interest, and we'd be glad to help.
The ARM part of MC is almost empty, so there's a lot of work in there,
probably even before writing the assembly/ELF writers. Also, AFAIK,
there is no ELF writer for MC, so wouldn't be good to write one
specific for the ARM platform, but keep things generic as much as
possible.
As I got it, MC is generic and the file writers only get the
instructions and sections and "print" them to file. Does it mean MC
already have all the section, relocation, attributes, properties that
both ASM and ELF have? If that's so, even though they're separate
projects (as Jason pointed out), a higher degree of communication will
be required to synchronize the MC part.
Also, we want to make sure that the generated ASM/ELF is not only GNU
compatible, but ARM ABI too (if the user require, in the target
triple). We have done some changes to the ISelLowering to conform to
EABI, a patch is on its way. Is it correct to assume that, if the user
chooses a triple with "eabi-none", we can generate ARM EABI (including
RT-ABI intrinsics) code?
1. MC-ized Assembly Printer
Would be good to make it generic enough, so we could (hypothetically)
add ARM assembly as a choice. But that could come as a great weight in
the first draft, so I won't be so sad if we can't get it straight
away. 
Irrespective of the asm syntax, we need better support for build
attributes, relocation information, stack unwinding, exception tables,
etc. That alone should prove a good challenge.
In the meantime, we have worked on support for some of these things
(build attr, reloc.) for the old writer. I'm preparing some of them as
patches and will send soon.
2. Object File Emission
That will also need support for features mentioned above, and also in
the disassembler.
GCC ignores most of the build attributes and "guess" everything else
from cpu, fpu and the instructions used. This is far from ideal and we
shouldn't follow the same route. If we need to guess anything, lets do
it in MC, so when you print assembly, the user can later edit if they
so want to (or pass the correct cmd-line options to create them right
in the first place). The disassembler should also understand that and
*always* give preference to user attributes.
Because ASM and ELF writers will use MC, we need explicit relocation
information in MC, so the writer can print it correctly to the file. I
saw that the Dwarf printer has this working, but the old ARM AsmWriter
doesn't (I sent a patch for this a while ago). Maybe we can join them
all in MC this time.
3. JIT. Same basic approach as (1) to replace the current object code emitter with an MC based one.
We're not overly concerned about JIT at the moment, but it's good to
keep in mind that we also have that route.
I would as far as to suggest to create a third data section in the IR,
after "target-data-layout" and "target-triple", maybe called
"target-info" or "target-options". It could be a target dependent
string, but would be best if it was shared among architectures, so you
could run IR generated to ARM in an 32bit x86, for instance.
We could also pass target specific features (such as build
attributes), how to handle unions and ByVal parameters, and maybe some
optimization assumptions when generating IR.
That would allow the codegen phase to detect not only what was
generated, but why, and make the appropriate changes if the execution
target does not conform to the intended target's specifications. That
is very liberating for JIT environments, but I do realise this is
extremely complex.
I'd be happy to have just the build attributes on it for now, though.
cheers,
--renato
The ARM part of MC is almost empty, so there's a lot of work in there,
probably even before writing the assembly/ELF writers. Also, AFAIK,
there is no ELF writer for MC, so wouldn't be good to write one
specific for the ARM platform, but keep things generic as much as
possible.
There is one for x86/x86-64 on the works. It has some limitations, but can
produce working binaries for small programs like screen.
As I got it, MC is generic and the file writers only get the
instructions and sections and "print" them to file. Does it mean MC
already have all the section, relocation, attributes, properties that
both ASM and ELF have? If that's so, even though they're separate
projects (as Jason pointed out), a higher degree of communication will
be required to synchronize the MC part.
I am just starting with MC, but as I understand it, the interface to the
streamer is basically one call per assembly directive. Things like
relocation have to be recorded by the streamer implementation. The
ELF one has to be extended to create the ARM specific relocations.
Also, we want to make sure that the generated ASM/ELF is not only GNU
compatible, but ARM ABI too (if the user require, in the target
triple). We have done some changes to the ISelLowering to conform to
EABI, a patch is on its way. Is it correct to assume that, if the user
chooses a triple with "eabi-none", we can generate ARM EABI (including
RT-ABI intrinsics) code?
That is almost all decided before getting to the MC streamer I think. So there
might be some work for supporting other ABIs, but I suspect most of it would
be in the IL to DAG conversion.
1. MC-ized Assembly Printer
Would be good to make it generic enough, so we could (hypothetically)
add ARM assembly as a choice. But that could come as a great weight in
the first draft, so I won't be so sad if we can't get it straight
away.Irrespective of the asm syntax, we need better support for build
attributes, relocation information, stack unwinding, exception tables,
etc. That alone should prove a good challenge.
Tables in general are produced one level before MC I think.
In the meantime, we have worked on support for some of these things
(build attr, reloc.) for the old writer. I'm preparing some of them as
patches and will send soon.2. Object File Emission
That will also need support for features mentioned above, and also in
the disassembler.GCC ignores most of the build attributes and "guess" everything else
from cpu, fpu and the instructions used. This is far from ideal and we
shouldn't follow the same route. If we need to guess anything, lets do
it in MC, so when you print assembly, the user can later edit if they
so want to (or pass the correct cmd-line options to create them right
in the first place). The disassembler should also understand that and
*always* give preference to user attributes.Because ASM and ELF writers will use MC, we need explicit relocation
information in MC, so the writer can print it correctly to the file. I
saw that the Dwarf printer has this working, but the old ARM AsmWriter
doesn't (I sent a patch for this a while ago). Maybe we can join them
all in MC this time.3. JIT. Same basic approach as (1) to replace the current object code emitter with an MC based one.
We're not overly concerned about JIT at the moment, but it's good to
keep in mind that we also have that route.I would as far as to suggest to create a third data section in the IR,
after "target-data-layout" and "target-triple", maybe called
"target-info" or "target-options". It could be a target dependent
string, but would be best if it was shared among architectures, so you
could run IR generated to ARM in an 32bit x86, for instance.We could also pass target specific features (such as build
attributes), how to handle unions and ByVal parameters, and maybe some
optimization assumptions when generating IR.That would allow the codegen phase to detect not only what was
generated, but why, and make the appropriate changes if the execution
target does not conform to the intended target's specifications. That
is very liberating for JIT environments, but I do realise this is
extremely complex.I'd be happy to have just the build attributes on it for now, though.
Can you give an example of what attributes you have in mind?
cheers,
--renato
Cheers,
Hi Jason, Jim, Rafael,
We had plans to work on ARM MC (asm and elf), good to see there is an
interest, and we'd be glad to help.The ARM part of MC is almost empty, so there's a lot of work in there,
probably even before writing the assembly/ELF writers. Also, AFAIK,
there is no ELF writer for MC, so wouldn't be good to write one
specific for the ARM platform, but keep things generic as much as
possible.
Yeah, not to mention, the Apple folks are probably not interested much
on ELF as mach.o, so there's probably even more genericity to be
considered.
There is one for x86/x86-64 on the works. It has some limitations, but can
produce working binaries for small programs like screen.
Hi Rafael,
This is good, at least we don't start from scratch and we can
cooperate with the other team to make sure the generic ELF
infrastructure remains generic.
I am just starting with MC, but as I understand it, the interface to the
streamer is basically one call per assembly directive. Things like
relocation have to be recorded by the streamer implementation. The
ELF one has to be extended to create the ARM specific relocations.
My previous relocation patch decides which relocation to use based on
the section and expression type, but that might not be the best way to
go for it. I think we should try to get it *before* ELF or ASM, so we
only implement that once.
If we have to extend some generic relocation mechanism, even better,
but that has to be generic in the MC level, not ASM/ELF streamers,
IMHO.
Can you give an example of what attributes you have in mind?
The EABI attributes that define the compilation options / platform
definitions, to propagate them to assembler and linker in case you use
multiple steps to compile.
Normally, just by choosing Cortex-A8, you can assume NEON exists, but
not all A8 have NEON and you can end up wrongly generating vector
instructions to a platform that doesn't support it. The "solution" is
to compile all at one step, but that's also far from ideal.
In effect, only having "armv7a-eabi-none" as a triple is very limiting
in ARM world. And since our designs can be extended in an very large
number of combinations, adding it to the triple is not the way to
go...
The attributes can be generically used by other platforms, that, once
it's required, it can help on generating correct code (sometimes
better code), like in the case of unions discussed earlier.
If front-ends ignore it, they can still generate code and rely on the
back-end knowing what they meant, but front-ends that want to
communicate to back-ends via IR/ASM/ELF can do so, especially for JIT
and multi-step compilations.
CodeSourcery's GCC already generates and understands that (to a
point), so there is no compatibility issues.