Using LLVM to compile Objective-C on an Xbox 360

We are looking at using Objective-C/C++ in a new game engine. Objective C’s duality of being both very dynamic and very “C” gives us exactly what we need to make the SDK and engineering of games simpler.

This means that we will need a way to compile it on all platforms our games will target. Currently the major platforms we are concerned with include… PC, Mac, XBox 360, PS3, iPhone. Now the PC, Mac, iPhone and PS3 are fairly simple. If we build our own OpenStep libraries we can simply use LLVM to compile directly to these platforms; the PS3 although proprietary uses GCC as a C/C++ compiler so I am assuming Objective-C can be used fairly simply. This leaves us with the XBox 360.

The 360 is a special chip (PowerPC based) with, as far as I have researched, a special ABI (Windows derivative). I haven’t the faintest clue of whether code from the LLVM PPC backend would even work on the 360, much less interoperate with the system libraries. So my formulated solution has become this: use an LLVM backend to output C code and then compile that code with using MS’s XBox 360 compiler. I believe I have read that LLVM has a C backend already but I don’t know how to select it.

If I can get a proof of concept showing Objective-C code running on the 360 we are off to the races. Any help is appreciated just not sure if all the pieces/parts exist and/or what I am missing. So… is this feasible? If so… how do I get LLVM to output C code?

Thanks,
Kevin

llvm can output C code, but that target has bitrotted severely over the last few months and nobody seems to be interested in fixing it. You may need to do some work there. Alternatively you could implement the PPC ABI that you need. There are several examples of supporting multiple ABIs on the same hardware, x86 being the most obvious. A lot of simple stuff will probably Just Work with the existing PPC ABI, more complicated stuff may not. (Only 32-bit PPC is really maintained, though, there are probably lots of problems with 64-bit.)

Objective C in llvm, AFAIK, is only used on the MacOSX targets and only tested there. There are sufficient secret handshakes between the compiler and the ObjC runtime that it is unlikely to Just Work in an untested environment. OpenStep has a familial relationship to MacOSX ObjC runtime, but they aren't the same and are unlikely to still be binary compatible at this point. You may need to do some work there also.

Summary, you can probably get this approach to work, but it's not as easy as you're hoping.

I'm slightly confused here, just to be clear: you're planning to go ObjC -> LLVM (via Clang or whatever) -> C, then use the 360 SDK compiler to build that, right? If that's the case, I think it'd be a better option than to use LLVM to directly generate XEXs, and you won't give the TCR guys fits (I checked the BAS TCRs, none of them say you can't do that, but the platform has some requirements regarding binary code that I doubt you'll get right without a lot of effort). If that is the pipeline you had in mind, have you attempted to run a Hello World app yet? Ran into any errors?

- Sherief

This is certainly an option... but keeping up an LLVM backend for the 360 seems like a bit much work. Using Clang's rewriter requires no real low-level maintenance, just creating and maintaining a C library for support functions emitted by the compiler (e.g. objc_msgSend).

It seems to create a backend for LLVM targeting the 360 I wouldn't need to create something that outputs XEX's anyway. My custom backend would effectively create object files that the 360's linker would have to be able to recognize. This would mean not only outputting all the correct ABI conforming code but the correct PDB debugging information as well. It seems much less, and easier, work to implement the support functions and run it through the standard xbox compiler as the ABI and debugging information are taken care of.

Implementing the backend (or editing the current PPC backend as needed) is a definite option. This seems to be the real question... which is easier... maintaining the PPC backend or maintaining the rewriter. Currently (in admittedly trivial tests) I have gotten the rewriter to work and output C code. There are some outstanding issues to do with linking and accessing the reflection information objective-c provides (and the rewriter properly outputs) but I am sure the person/people who wrote the rewriter can answer those fairly simply (or else the rewriter would be fairly useless).

As far as the runtime goes there is an implementation for windows, linux and most likely the PS3 (since it is already a linux derivative using GCC as mentioned). If we have to create a runtime for PS3 and XBox that seems trivial as the functions are very basic in nature.

My comments on OpenStep were more meant to point to the fact that we would write our own library; really just throwing all notion of OpenStep away. Instead of NSObject we would create our own base DObject or something along with a DString, DAarry, DSet, DMap, etc. Truthfully this would be our plan anyway because we want to follow the lead of OSX and provide these objects "toll free bridged"... meaning we would implement them using a std c++ library object (e.g. std::basic_string or std::vector) as the first member of the object which will allow us to cast a DString to std::string or DArray to std::vector without any manual or automatic marshaling. As mentioned this is how OSX implements its NS classes, by using the equivalent CF version which again allows casting between the objects.

This is certainly an option… but keeping up an LLVM backend for the 360 seems like a bit much work. Using Clang’s rewriter requires no real low-level maintenance, just creating and maintaining a C library for support functions emitted by the compiler (e.g. objc_msgSend).

The clang rewriter is not the same than LLVM C backend. AFAIK, clang rewriter don’t use llvm at all (as it does not perform codegen). It is just a source to source conversion.
I don’t know how it work yet, but if it is design to output code for the NeXT Runtime, you may have to tweak it to output code compatible with your runtim.

It seems to create a backend for LLVM targeting the 360 I wouldn’t need to create something that outputs XEX’s anyway. My custom backend would effectively create object files that the 360’s linker would have to be able to recognize. This would mean not only outputting all the correct ABI conforming code but the correct PDB debugging information as well. It seems much less, and easier, work to implement the support functions and run it through the standard xbox compiler as the ABI and debugging information are taken care of.

I’m slightly confused here, just to be clear: you’re planning to go ObjC → LLVM (via Clang or whatever) → C, then use the 360 SDK compiler to build that, right? If that’s the case, I think it’d be a better option than to use LLVM to directly generate XEXs, and you won’t give the TCR guys fits (I checked the BAS TCRs, none of them say you can’t do that, but the platform has some requirements regarding binary code that I doubt you’ll get right without a lot of effort). If that is the pipeline you had in mind, have you attempted to run a Hello World app yet? Ran into any errors?

  • Sherief

We are looking at using Objective-C/C++ in a new game engine. Objective C’s duality of being both very dynamic and very “C” gives us exactly what we need to make the SDK and engineering of games simpler.

This means that we will need a way to compile it on all platforms our games will target. Currently the major platforms we are concerned with include… PC, Mac, XBox 360, PS3, iPhone. Now the PC, Mac, iPhone and PS3 are fairly simple. If we build our own OpenStep libraries we can simply use LLVM to compile directly to these platforms; the PS3 although proprietary uses GCC as a C/C++ compiler so I am assuming Objective-C can be used fairly simply. This leaves us with the XBox 360.

The 360 is a special chip (PowerPC based) with, as far as I have researched, a special ABI (Windows derivative). I haven’t the faintest clue of whether code from the LLVM PPC backend would even work on the 360, much less interoperate with the system libraries. So my formulated solution has become this: use an LLVM backend to output C code and then compile that code with using MS’s XBox 360 compiler. I believe I have read that LLVM has a C backend already but I don’t know how to select it.

If I can get a proof of concept showing Objective-C code running on the 360 we are off to the races. Any help is appreciated just not sure if all the pieces/parts exist and/or what I am missing. So… is this feasible? If so… how do I get LLVM to output C code?

Thanks,

Kevin

– Jean-Daniel

Implementing the backend (or editing the current PPC backend as needed) is a definite option. This seems to be the real question… which is easier… maintaining the PPC backend or maintaining the rewriter. Currently (in admittedly trivial tests) I have gotten the rewriter to work and output C code. There are some outstanding issues to do with linking and accessing the reflection information objective-c provides (and the rewriter properly outputs) but I am sure the person/people who wrote the rewriter can answer those fairly simply (or else the rewriter would be fairly useless).

As far as the runtime goes there is an implementation for windows, linux and most likely the PS3 (since it is already a linux derivative using GCC as mentioned). If we have to create a runtime for PS3 and XBox that seems trivial as the functions are very basic in nature.

My comments on OpenStep were more meant to point to the fact that we would write our own library; really just throwing all notion of OpenStep away. Instead of NSObject we would create our own base DObject or something along with a DString, DAarry, DSet, DMap, etc. Truthfully this would be our plan anyway because we want to follow the lead of OSX and provide these objects “toll free bridged”… meaning we would implement them using a std c++ library object (e.g. std::basic_string or std::vector) as the first member of the object which will allow us to cast a DString to std::string or DArray to std::vector without any manual or automatic marshaling. As mentioned this is how OSX implements its NS classes, by using the equivalent CF version which again allows casting between the objects.

This is not how the CFBridge works. Actually, the first member of an object MUST be a pointer to its Class (the isa pointer). If you want to be able to put something else, you will have to rewrite the runtime to support it.

You can read this great article about the bridge here:

http://ridiculousfish.com/blog/archives/2006/09/09/bridge/#fish_made_a_mess

and an other one here:

http://www.mikeash.com/pyblog/friday-qa-2010-01-22-toll-free-bridging-internals.html

And you can also have a look at the stripped down version of the CoreFoundation sources to get some other clues about how it works now.

http://www.opensource.apple.com/source/CF/CF-550.13/

– Jean-Daniel

Thanks for the articles! There is definitely some stuff in there I hadn’t thought of; also, it quite obviously points out that it’s actually not “toll free” at all but “very low cost”!

This won’t affect us too much though since we will be creating our own objective-c classes from scratch and we are not trying to maintain backward compatibility. A simple solution would just be to have a method that returns our underlying implementation object (e.g. a std::basic_string for GString) and/or make the ivar implementation object public and let anybody access it.

I believe you are correct, the Clang rewriter seems to stay completely inside the Clang universe. Although, I think you can argue whether or not Clang is part of LLVM (it is listed on the main page as a primary sub-project).

Either way my point was to compare the “ease” of the two methods, the Clang rewriter or modifying LLVM’s PPC backend to output object code.

PS3 is not "a Linux derivative." The compilers supplied by SCE for PS3 game development are highly customized and support a customized ABI that will take some time to adjust LLVM and Clang to support.

You'd likely also run afoul of a TRC or two, similar to the problems you'll face with Microsoft TCRs mentioned earlier.

Alex

By linux derivative I meant that it borrows the linux GCC ABI... and it does. You can compile with an off the shelf GCC cross compiler and link the resultant object files ones compiled with the PS3 provided version. We have done it.

Also, as both an XBox 360 and PS3 developer, there seems to me to be nothing in the TCRs/TRCs that preclude us from using a different compiler. There are rules about symbol inclusion and other resultant binary requirements... but as of yet I have not found specific ones stopping us from using a compiler that works. In either case the LLVM rewriter solves any other these issues as we would be just be compiling C.

Please create a thread on DevNet to discuss this further.

Alex

Just to be clear, nothing in the MS TCRs require that you use a certain compiler - it does, however, require that you link with the 360-specific C runtime libraries. That being said, I still believe that direct binary generation is not a feasible option, and the ObjC => C route is preferred.

Kevin, hate to be asking this but do you have access to the XDK or are you just "aiming" for it?

- Sherief

Why? I think the discussion belongs here, and the topic doesn't seem to include proprietary information - a lot of 360 info has been made public through MS material. Other info can be discussed without much disclosure (we can refer to TCRs by three-letter category and number, etc).

- Sherief

Kevin, there're some unwritten rules on the 360 that seem to interfere with code generation - pointer load-stores seem to require zeroing the most significant 32 bits, for example - that was one issue that I ran into a while ago while fooling around with code generation on the 360 (and don't blame me if I'm slightly off the mark, it was quite a while ago). I didn't seem to find it mentioned in the XDK docs anywhere. Again, you'll be better off generating C code and using the XDK compiler to compile that.

- Sherief

At this point I am thinking, for the PS3 and 360, that the ObjC to C route is preferred also.

Yes, as I alluded to, we are an XBox and PlayStation developer so I have both a 360 and a PS3 dev kit.

FYI, DevNet is for the SCE (i.e. Sony) :wink:

But I am kind of in agreement... although I will discuss on DevNet also because there may be some knowledge there that does include proprietary information. Sony and MS seem to think any information about their respective consoles is proprietary and falls under NDA.

Yep... I have been a 360 developer since the days when it was a actual Macintosh G5. I went to all the early tech. seminars and learned way more about that wacky processor and MS's seemingly crazy ABI choices than I care to (I know, I know, they had their reasons for them). Knowing this is why I quickly turned toward rewriting objc instead of attempting to implement a backend.

I have also had a PS3 dev kit since the days it was a big grey can of fire-hazard. I am much better versed in the 360 but it seems Sony made many more sane decisions to build on existing technology. Although, it sounds like Alex is alluding to the fact that I may be wrong on just how much they built on and how much they rewrote; on the surface though trivial integration is certainly possible between generic GCC and PS3 tools. I will get on DevNet later today and discuss that with him.

Overall, as we both have said before, the ObjC => C route seems like the best idea. It just means that we need to build/maintain our own objective-c runtime for those platforms that require them and make sure to keep the "-rewrite-objc" option in clang maintained and ensure it produces workable code for all platforms (their is currently some very GCC specific code being output). This allows each proprietary platform to generate code in their own fashion and it really is a minor extra step to first run our code through clang.

To ease our development process, actually hide it completely, we are planning on leveraging the open nature of Clang and LLVM to create a driver for these platforms that first rewrites ObjC to C and then calls the platform's C compiler passing through all command line options. This way the ObjC stage should be fairly transparent.

Kevin,

My apologies... the original question was directed here because I didn't know about the Clang rewriter. Now that the direction has sharply tacked toward using that method I will direct any further questions/discussions to that list.

Thanks