LLVM Embedded Toolchains Working Group sync up


LLVM Embedded Toolchains Working Group sync-up calls and meeting minutes are tracked here as replies below. Migrated from Google Docs.

Meeting Details

On Thursdays at 9am PST/5pm BST/6pm CET every 4 weeks starting on Mar 3rd, 2022.

Join Zoom Meeting

Meeting ID: 932 9643 6903

Passcode: 918316

One tap mobile

+442034815240,93296436903#,*918316# United Kingdom

Dial by your location

+44 203 481 5240 United Kingdom

+1 346 248 7799 US (Houston)

+1 408 638 0968 US (San Jose)

+1 646 518 9805 US (New York)

Find your local number: https://armltd.zoom.us/u/adPiWtNQFa

Calendar Invites

Link to share the ICS file and direct link.

Link to shared Google Calendar (gcal).

Past meeting minutes migrated from Google Docs:

2022-03-03 Multilib support


  1. Requirements and design options for multilib support in clang.
    Multilib toolchains contain multiple sysroot, each having a version of the target libraries for different architecture/ABI variants that are selected based on the target and other command line options.
  2. Profiling support.
  3. AOB


  1. Volodomyr Turanskyy
  2. Peter Smith
  3. Alex Brachet
  4. David Spickett
  5. Hafiz Abid Qadeer
  6. Marcel Achim
  7. Nigel Perks
  8. Petr Hosek
  9. Stephen Hines
  10. Pirama
  11. Son Tuan Vu
  12. Daniel Thornburgh
  13. Shivam Gupta

Follow up from previous meeting


Discussion (minutes)

  1. Multilib support (presentation)

  2. Want to avoid having to add multilibs each time

  3. Tablegen file for generating multilib

  4. GCC implementation isn’t great. Possible for a cleaner solution.

  5. Implementation examples llvm-project/Gnu.cpp at bd1917c88a32c0930864d04f4e71155dcc3fa592 · llvm/llvm-project · GitHub

  6. llvm-project/Fuchsia.cpp at bd1917c88a32c0930864d04f4e71155dcc3fa592 · llvm/llvm-project · GitHub

  7. Fuchsia Multilib for LLVM runtimes (runtimes exceptions/no-exceptions etc.)

  8. CMake build to generate all the multilibs, only going to work for LLVM build system.

  9. Auto searching will only work for GNU sysroots (libc++ and libstdc++).

  10. Reasonable to go in the direction, need to get the right design, used in several drivers. Each one may have existing constraints. Few iterations, try different designs. Keep several alpha/experimental.

  11. History of downstream forks for embedded. Get people to test it out. Future iterations. Reaching the right audience.

  12. Provide downstream users an opportunity to migrate

  13. Start an RFC in discourse.


  1. Fuchsia not using libclang rt profiling, kernel has its own runtime with shared include file (structs). Linux is thought to do a similar thing.

  2. Problem is that it relies on libc (calls into a few functions). Files, mmaps. Instrument libc itself (recursive case). sanitizer_common has alternative implementations. Proposed rewriting profile runtime on top of sanitizer common.

  3. Breaking up profile into smaller parts.

  4. Fuchsia has a custom runtime for low-level use cases (bootloader and kernel) ​​https://cs.opensource.google/fuchsia/fuchsia/+/main:src/lib/llvm-profdata

  5. There’s currently little reuse between our runtime and compiler-rt runtime, but we are interested in refactoring libclang_rt.profile so we can reuse more parts and reduce the duplication.

  6. Not clear if there is a universal runtime. Can we share the basic things, common logic. Extract into headers in the clang installations. Can do downstream integration for embedded systems.

  7. Symbolization produce a stack trace then invoke llvm-symbolizer, can’t usually do that in embedded system. Fuchsia uses offline symbolization through symbolizer markup, markup goes to serial output, no runtime support for symbolization is needed.

  8. This is already supported in sanitizer_common llvm-project/sanitizer_symbolizer_markup.cpp at b223e5f8468cbed5cffe0d872de8feac2a73b030 · llvm/llvm-project · GitHub

  9. RTEMS has adopted the same format (since removed from compiler-rt, likely bitrotted).

  10. Fuchsia team is interested in implementing support for the markup directly in llvm-symbolizer. Is there community interest?

2022-03-31 Build bots, symbolizer


  1. Build and test bots for embedded toolchains.

  2. Background:

1. https://lab.llvm.org/buildbot/#/console
2. https://llvm.org/docs/HowToAddABuilder.html
  1. Intention:
1. Provide pre-commit tests for bare-metal configurations to flag any breaking changes early or preferably even before they land.
  1. Using GNU toolchain driver via the bare-metal driver, see discussion Switching to GCC C runtime linkage for the baremetal driver

  2. Symbolizer markup

  3. Google sent out an RFC to implement symbolizer markup support in llvm-symbolizer: RFC: Log Symbolizer


  1. Volodymyr Turanskyy
  2. Mikhail Maltsev
  3. Peter Smith
  4. Petr Hosek
  5. Daniel Thorn
  6. Paul Kirth
  7. Pirama
  8. Roland McGrath
  9. Stephen Hines

Follow up from previous meeting

  1. Multilib - Arm team wants to come up with an RFC, busy with a release right now.
  2. Profiling - needs further discussion?


  1. Build Bots

  2. There was a question about xcore build bot regarding missing REQUIRED in tests and what does it mean to use the default target for cross-compilation targets.

  3. There are toolchain builders (easy as they are cross compilation toolchains) and runtimes (more difficult since they depend on the target architecture).

  4. Are there cross compilation bots of compiler-rt? Looks like not.

1. Fuchsia team is investigating this now, cannot make to work fully yet. Need to prefix each execution with a tool like QEMU or scp/ssh. Need a way to batch together (build all - copy all - run all)? Otherwise running one-by-one is very slow.
2. How to handle and report accurately test failures when batched?
3. May be possible to run tests on a different model, e.g. Arm M-profile test code on Arm A-profile target which is more capable.
4. Libc++ and libc++abi have a different test runner/wrapper around lit test runner, which is easier to tailor.
5. Difficult would be sanitizer tests, coverage tests.
  1. How to make people in the community respond to failures in these bots?
1. How to provide people with a reproducer and a way to debug their failures? Use emulators?
2. Required to fix vs best practice?
3. At a minimum we must have really good docs, step-by-step how to reproduce a failure.
4. Provide a VM to reproduce and debug, if possible.
  1. Arm Embedded Toolchain for now only provides a smoke test, we want to provide a public bot with much more tests/regular LLVM tests.
1. Internally, Arm Compiler has a patch to the libc++ test runner to invoke the model to execute. It may be possible to upstream.
2. Big part of the lit tests can be run without POSIX, so in bare-metal - would it make sense to define this subset in upstream to make it easier to run/maintain? Sounds like a good idea.
  1. Using GNU toolchain driver

  2. Is this a common use case to add into the LLVM driver itself?

  3. Escaping linker options to avoid the need to double-escape may be useful.

  4. Pigweed library uses LLVM baremetal driver.

1. LLVM baremetal driver does not seem to be very well thought out/designed. There are opportunities to simplify/improve, remove duplication by lifting to the parent class.
2. History of the driver: was started by Code Sourcery, does not seem to have an active owner now.
3. Anyone interested in doing reviews for such refactorings? Arm team (Peter Smith may be a contact, Mikhail Maltsev as well) is happy to collaborate. Hafiz Abid Qadeer was adding RISC-V support to the driver so may be interested too. Petr Hosek will post something for review.
  1. Symbolizer

  2. Petr and Daniel: RFC was sent (link in the agenda):

1. Defer symbolization until later time maybe on a different machine with more debug info.
2. This approach has been used in Fuchsia since 2017, so is proven.
3. Design by Roland McGrath.
4. In Fuchsia info goes out via serial for later processing on a host machine.
5. It is supported in sanitizer runtime for Fuchsia, makes sense to extend support.
6. Whoever finds this interesting, please review the RFC, implementation will follow. As a separate library in the LLVM project.
7. Target binary emits markup elements that describe where segments are mapped, PCs from the stack, i.e. enough context. Some other machine offline can take the markup and generate the traditional stack trace. The markup should be standardised.
8. Arm team is interested in this - will review the RFC.

2022-04-28 LLVM libc


  1. Choice of C library for embedded use cases. LLVM libc as an option.
  2. Roundtable at EuroLLVM?


  1. Peter Smith
  2. Siva Chandra
  3. Alex Brachet
  4. Mikhail Maltsev
  5. Nigel Perks
  6. Petr Hosek
  7. Pirama
  8. Simon Wallis
  9. Volodymyr Turanskyy

Follow up from previous meeting

  1. Build bots - below.
  2. GNU driver - N/A
  3. Symbolizer - N/A


  1. Roundtable at EuroLLVM

  2. Petr and Nigel are going.

  3. So far not many people are on the call.

  4. Should we try to advertise in the LLVM Discourse to see if more people are interested?

  5. Lib C options.

  6. How suitable LLVM libc is for embedded toolchains?

  7. Newlib is OK for OSS projects, but not commercial toolchains.

  8. Arm team is looking into LLVM libc see now, apparently there are other people interested as well, suggesting some patches upstream.

  9. Siva works on LLVM libc:

1. Just started looking into potential embedded and bare-metal uses.
2. Happy to learn from embedded experiences.
3. Peter posted a comment on the Arm32 build of LLVM libc - how to build with LLVM Embedded Toolchain for Arm - https://discourse.llvm.org/t/building-llvm-libc-for-arm32/62092
  1. There is a large design space for embedded libc in terms of code size vs performance. Peter will try to summarise in a Discourse post.
  2. Zephyr as an example: uses very minimalistic libc. LLVM libc is now tuned for server performance, people may be most interested in something in between. So it is important that the library should allow tailoring to the specific use case.
  3. Arm Compiler example: we offer two different libraries: standard compliant one (but bigger) and “micro” one that has very limited functionality, but very small.
  4. LLVM libc strives to ideally cover both use cases where standard compliance is important and very minimal code size is important.
  5. The more tuning points there are, the more library variants will need to be built, thus all of them need to be tested. So there is a practical limit to configurability.
1. Pragmatically, there will be some more used configuration vs less used to direct testing.
  1. LLVM libc - how startup is addressed (for bare-metal, e.g. stack setup)? So far it is Linux focused, startup was not addressed yet. More details on requirements/approaches would be useful to share. Usually, hooks for users to define/customise stack and such are provided.
  2. Now many customers ask for POSIX support in embedded. LLVM libc seems to provide some POSIX functions - how easy is it to extend to add more coverage of POSIX for high-end embedded systems?
1. POSIX functions that do not require OS: should be easy to add.
2. POSIX functions that rely on OS: Zephyr and FreeRTOS have a level of support, newlib has some support too. Libc may need something like a porting layer to allow adaptation to a particular OS.
  1. Sanitizer runtimes. Is there an intersection with LLVM libc? Aim is to instrument everything from libc. Fuchsia has its own libc as a fork of musl that includes interaction with sanitizers. The API for sanitizers is very low-level so that libc itself can be instrumented to use sanitizers. There is a discussion about other libc projects adopting the same API, or rather come up with one shared API. Long term plan is to have this API implemented in LLVM libc too.
1. https://cs.opensource.google/fuchsia/fuchsia/+/main:zircon/third_party/ulib/musl/include/zircon/sanitizer.h is the API that Fuchsia uses.
  1. How complete in regard to C standards is LLVM libc now?
1. String is mostly complete, no locale.
2. Maths work is underway to complete it by ~Sep 2022.
3. Aim is to eventually build full compliant library, the team is happy to prioritise what is important for specific use cases with good rationale.
  1. Modular structure: static libc vs missing pieces used from the shared system libc. There may be some subsets that are all-or-nothing, e.g. IO subsystem. There is the “full build” mode that assumes LLVM libc is the only C library.

  2. Build system: native or cross compilation? Now focus only on native builds and tests, but cross compilation is possible.

  3. Let us raise any topics of interest in Discourse.

  4. Build bots

  5. Embedded target vs host for running the tests. Object emission was missing. There is the REQUIRES (restored) directive to guard object emission in tests.

2022-05-26 EuroLLVM, LLVM libc


  1. Admin: Move to Discourse for agenda and meeting minutes, example LLVM Pointer Authentication sync-ups - #4 by rjmccall ?

  2. Follow up from EuroLLVM 2022 round tables.

  3. Upstreaming of build scripts for embedded toolchain, potential issues/acceptance criteria.

  4. What embedded use cases can be tested within llvm-project itself without external dependencies, e.g. building compiler_rt for embedded targets. Such tests should be easy to establish within existing LLVM build bots.

  5. 64-bit source code locations to handle AUTOSAR generated code (https://reviews.llvm.org/D97204) lack of interest/reviewers.

  6. llvm-libc update and embedded systems (won’t get time to get through all of these)

  7. A summary of a comparison versus an existing embedded toolchain’s C-library.

  8. Update of where we are with providing requirements for embedded systems.

  9. The parts of llvm-libc that have been implemented so far are independent of each other. Things like printf and locale will need to communicate. Some size optimized embedded libraries do not include locale, have there been any thoughts about implementations that may not include some parts of the library?

  10. libc++ on embedded systems


  1. Peter Smith
  2. Volodymyr Turanskyy
  3. Simon Cook
  4. Alex Brachet
  5. David Finkelstein
  6. Ed Jones
  7. Lewis Revill
  8. Michael RJ
  9. Nigel Perks
  10. Petr Hosek
  11. Saleem
  12. Simon Wallis
  13. Yvan Roux

Follow up from previous meeting

  1. LLVM libc below.


  1. Admin - agreed to move, Volodymyr will take care and update.

  2. EuroLLVM 2022 round tables follow ups:

  3. Libraries discussion, multilib support like in GCC.

1. Simon is working on new multilib configurations in runtime. Would be great to have an RFC.
2. compier_rt with multilib - how to build? Manually?
3. Petr: Fuchsia builds ~50 different multilibs, can share experience.
4. Link from Saleem: https://github.com/apple/swift/blob/main/cmake/caches/Windows-x86_64.cmake#L30-L45
  1. Extension of config files to allow more configurability? RFC would be useful to start the discussion too.

  2. Build scripts for embedded

  3. Embedded toolchain for Arm is in a separate github repo that only has build scripts (and few patches) - would it be useful to try upstream such scripts into LLVM to keep in one place and make it easier to create the build bots?

1. Downloads external (non LLVM repo) library.
2. Builds multiple versions of this library in a loop.
  1. Make use of CMake?

  2. Needs a concrete example to start discussion?

  3. Need to consider Windows: split between runtime and SDK components.

  4. → RFC on Discourse.

  5. What can we test within current LLVM repo in embedded configs

  6. Compiler_rt? Build only? Run on a model?

  7. Libcxx + tests, run on a model.

  8. LLVM test suite - factor out tests that depend on POSIX that is missing in embedded configs.

  9. Would community be willing to keep such buildbots green?

  10. Tests may be difficult to run on real HW, so models are used.

  11. Update the lit tool to batch running the tests? Fuchsia did some prototyping for this, still needs work.

  12. 64-bit source code locations

  13. Lack of reviewers, anyone is interested, please ping in the review.

  14. LLVM libc

  15. Had a closer look, compared to Arm library.

  16. Next, post at Discourse.

  17. How printf and locale should be tied together? May need to be optimized together.

  18. Arm library has modular printf: can send via serial port, can skip a lof of features like FP support.

1. Example, LLVM libc is designed it to let one specify which conversions they want:[ https://github.com/llvm/llvm-project/blob/main/libc/src/stdio/printf_core/converter.cpp](https://github.com/llvm/llvm-project/blob/main/libc/src/stdio/printf_core/converter.cpp)
  1. Clang can optimize more if building specifically for LLVM libc by knowing a bit more about it.

  2. Libc++ on embedded systems

  3. Not including features that are not needed.

  4. AUTOSAR adaptive platform is built on C++14 including libraries, so there is a lot of uptake in use in automotive domain.

  5. Invite the libc++ maintainer to start a discussion about scaling down to embedded systems.

  6. Peter had a discussion with Louis Dionne recently, ideas: subsets can be very useful, e.g. one that does not allocate memory - can these be separated and only relevant tests run? Overall, he seems to be interested in building such support.

  7. Agreed to reach out to Louis and invite him. Volodymyr to organize.

1 Like


Apparently, the event in this post shows the time 1 hour off - I cannot edit it now, will try to fix for next time.

The correct time is in the calendar: “On Thursdays at 9am PST/5pm BST/6pm CET”

2022-06-23 LLVM libc, MC/DC coverage


  1. Nigel Perks
  2. Rouxy
  3. Alan Phipps
  4. Simon Wallis
  5. Daniel Thornburgh
  6. Simon Butcher
  7. Peter Smith
  8. Siva Chandra
  9. Petr Hosek
  10. Shivam Gupta
  11. Volodymyr Turanskyy
  12. Mikhail Maltsev
  13. Claudio Bantaloukas


  1. Discourse event time is shown incorrectly at the main page, some people were confused, sorry about that, Volodymyr will see how to fix this.
  2. Libc++ for embedded.
  • Louis was not able to join this time, Volodymyr to check if he will be available to attend next time.
  1. Peter: Follow up on past LLVM libc discussion.
  • After the investigation and internal discussion, Arm team confirmed that LLVM libc looks very promising, so wants to contribute.

  • Some of the topics that we want to touch/discuss upstream below.

  • Diffrernt size variants of some functions like smaller memcpy are needed for MCUs.

  • Customization - how to manage interrelated pieces of functionality?

  • HW abstraction layer - if there is no file system, but there is a serial port or debug interface of semihosting, how to redirect?

  • Process/app startup when there is no OS:

    – Peter: Is this something that libc would include or just document how to add the startup code?
    – Siva: if there is a standard/convention for startup code, then happy to add to the library. Question: How to test it? Need to add buildbots, then how people could debug failures in such a buildbot?

  • Some aspects of startup - setup stack, heap, etc - are architecture specific, but required. Other libs can be used as example, e.g. newlib.

  1. Alan: Upstream support for MC/DC code coverage, which is useful for embedded functional safety applications.
  • MC/DC is Modified Condition/Decision Coverage (Modified condition/decision coverage - Wikipedia), it is required for functional safety code of higher ASIL levels.

  • An implementation will be upstreamed in next few months.

  • Who is interested? WOuld be great to help with code review.

  • Peter: Arm is interested to help with review. Team in Qualcomm probably can be interested too.

  • Will it work for bare-metal? Compiler-rt implementation is not suitable for bare-metal now - should be the next step to add support.

  • Petr: happy to help with code coverage too, Fuchsia team have some experience. Fuchsia uses coverage in the kernel.

    – Now there is a lot of similar code in sanitizers, code coverage, profiler - a refactoring was suggested last year on Discourse (Using C++ and sanitizer_common in profile runtime): profiler runtime different from compiler-rt so suggested to rebuild it in C++ (from C) and on top of sanitizers common code. Sanitizer_common is already good at abstracting from the underlying OS/target. So migrating and improving would improve all use cases together.
    – Fuchsia may work on this in the coming months. Pure refactoring would be useful too, e.g. remove multiple ifdefs, etc - structure the code better.
    – Meta (company) team is doing interesting work in coverage now too. It is intended for mobile phones, e.g. adding boolean coverage mode (instead of counters). Fuchsia team is talking to them to coordinate.

  • Peter: Important point is how to get the requirements defined to fit all use cases from MCU to more high end systems. E.g. if there is small memory, have an interface to write the coverage/profile info to the debugger interface.

  • What are the key changes in the MD/DC coverage patch? Additional level of analysis for boolean expressions + additional counter object to keep track of.

Update on the incorrect event time: Turned out there is a related known issue, so until it is fixed, the event summary panel is removed.

Please use the calendar invite to get the correct date and time.

Sorry about the confusion yesterday and thank you for joining anyway!