Here are my notes from the Embedded Toolchains open discussion, with some additions by Jordan.
- Embedded Linux/LLVM-libc
- Is there a set of packages that we need to set up
- Picolibc and newlib are existing libcs
- PicoSDK is the most complete thing LLVM-libc can build
- Just barely there, still missing some symbols that are needed for users
- Musl is more complete, but also much larger in general
- Might be good to have porting guide to LLVM-libc
- Peter Smith gave a good talk on the differences between gnu ld and lld
- It would be nice to have something similar for LLVM-libc vs other common libcs
- Do we have a point of reference for code size limits?
- How much space does a Raspberry Pi Pico have, for example?
- For configuration, LLVM-libc generally does configuration as options and not by target
- So if you want a size-optimized library you set the size optimization flags (e.g. disable floats in printf)
- In future we’d like to have “profiles” that provide a default set of flags
- E.g. picking the size-optimized profile which sets a lot of flags
- Might also swap out implementations, e.g. qsort can actually be heapsort
- Microcontrollers tend to have “shared” properties. Clusters of embedded properties.
- As long as the “centroids” of these spaces are chosen well, the spaces for these should be fine.
- Probably end up with 2-3 implementations.
- Need to figure out which configurations are valuable
- Locales are expensive, as are pthread cancellation checks
- Need proper testing (buildbots) for each well supported config.
- Embedded threading
- Libc++ supports providing your own threading API
- LLVM-libc would like to support something similar
- There are some users who want more complex threading APIs
- Users generally want a familiar threading API, so C11 threads, C++ std::thread, or pigweed’s PW_THREAD
- What’s the smallest platform we want to support?
- Full posix?
- Stripped down linux?
- Minimal RTOS?
- Baremetal?
- Embedded systems are deeply heterogeneous, and they are not multi-modal. They’re fundamentally different on what they do/don’t support.
- Without a build bot, I don’t know if I shouldn’t be allowed to break it. Where is the market share? Who makes these build bots?
- If LLVM-libc chooses a generic API that people plug into, then we’re potentially locking ourselves into that API since changing it might break people
- It may be best to call directly into existing APIs, then people can connect to that.
- Regardless, buildbots are the best way to make sure something is maintained. We can’t fix what we don’t know is broken.
- There are also a lot of different RTOSes in common use, even within one project.
- Counterpoint: Embedded developers are more tolerant of breakages, may be okay with doing some amount of rewrites.
- How do we handle headers?
- Some RTOSes have their whole API in a single header, how do we handle platform support?
- May need them to build their own LLVM-libc compatible headers
- What’s the status of LLVM-libc and a loader?
- Not really needed for embedded, but it is needed for larger systems
- LLVM-libc doesn’t have one, but there is a loader in progress at Google within Fuchsia.
- There’s some discussion around rewriting it for LLVM, but nobody’s currently working on it.
- If people need a loader soon, there will need to be significant effort.
- If the internal threading API is similar or the same as C11 threads it may be easier to port.
- What’s the ABI like for LLVM-libc?
- The user ABI is currently not required to be stable since we only support static linking.
- Once we do dynamic linking we’ll have to be more careful
- For platforms they do need a stable ABI, but LLVM-libc is still growing so it may not be completely stable.
- Currently it’s fairly straightforward, e.g. “write to stdout/stderr”, “read from stdin”, and “errno”
- The user ABI is currently not required to be stable since we only support static linking.
- LLVM-libc needs users, but more importantly we need bug reports
- We don’t have good visibility into what people want
- From the user perspective a bug report is significant effort, having a “+1 I am missing this symbol” button might be useful.
- But also people on stable releases won’t get updates for ~6 months.
- Bugs on file allows for new LLVM-libc contributors.
- To convince users to switch, LLVM-libc needs a feature that other libcs don’t have
- The most obvious example would be a clang/libc integration
- E.g. fixed point (which we did), or the compiler picking which parts of printf are needed from the format string.
- We’d need a target triple for LLVM-libc
- The most obvious example would be a clang/libc integration
- Multilib and build
- Arm has been adding multilib support that works for listing some known list of architecture-related options.
- There’s problems with setting config options (e.g. turning off threading) which need to adjust macros
- Multilib solves the problem of “I have a target which needs specific flags”
- It may be useful to ship sources along with the multilib so that if a specific flag configuration isn’t available it can be built from source.
- Possible problem: This starts moving clang into a build system, which is a lot of complexity.
- LLD can already do this to some extent if you provide raw bitcode and LTO it
- Doesn’t work for defines, which limits it somewhat.
- LLD can already do this to some extent if you provide raw bitcode and LTO it
- Could also just build your libc from source with your application
- Solves the problem of multilibs, but this is difficult to scale, especially across build systems
- Build performance isn’t great
- It’s all cmake
- The runtimes build is cmake invoking cmake invoking cmake
- Turns out it’s slow
- If you build lots of different targets, it takes a lot of time.
- Cmake now has a feature called “multigenerators”
- Lets you set up multiple builds at the same time
- It has some issues, which nobody else is running into.
- Nobody else is building a compiler with cmake.
- Ninja doesn’t properly disambiguate rules for different targets.
- Multilibs are keyed off of the target triple, but if you want the same target and different flags it doesn’t work
- Changing the LLVM cmake would break out of tree builds.
- Currently the multilib format is marked experimental, when will it not be?
- When there’s confidence about it being stable.
- It may be at that point, and there’s a version number as part of the yaml file.
- Arm has been adding multilib support that works for listing some known list of architecture-related options.
- Position independent code
- PC relative vs base relative
- PC relative is relative to the program counter, better for embedded
- Base relative is relative to the base image.
- Is there interest? In what type?
- ARM has put in effort for PC relative
- PIC is mostly relevant for ASLR (Address space layout randomization)
- Also relevant if you’re building things as modules then linking them.
- Base relative could be useful for patching
- Many times, PIC is used in a transition from baremetal to a more full OS.
- PC relative vs base relative
- Community
- It’s important to speak about being an embedded user, so that it’s clear there is demand.
- There also needs to be embedded testing
- LLVM-libc may be able to be the first with that.
- Where do you talk about things?
- Github issues vs discourse vs monthly meetings vs discord
- The monthly sessions are good if you want to ask questions
- Discourse is where to go for RFCs