LLVM Embedded Toolchains Working Group sync up

Hi everyone!

As mentioned on a couple of the embedded LLVM calls, my changes supporting MC/DC are presently in phabricator (quoted above):

Since the Developers’ Meeting last November, I’ve been hearing from more folks who are interested in seeing this functionality upstream but don’t have the LLVM expertise to contribute meaningfully to the reviews, unfortunately, so I could really use some help in getting things reviewed.

So far, @ellishg has been able to look at some of the back-end work and provide some good feedback. @smithp35 provided some good suggestions for the preliminary review I added, which I incorporated into the clang-specific review linked here.

Of course, I don’t want to trivialize the fact that everybody is busy, and many of you have upstreaming work of your own. I appreciate the feedback you have and whatever time y’all are able to contribute to this effort! I’m also on Discord if you want to chat about MC/DC.



1 Like

2023-02-02: LLD


  1. Petr Hosek

  2. Siva Chandra

  3. Prabhu Rajasekaran

  4. Peter Smith

  5. Tue Ly

  6. Fangrui Song

  7. Michael Platings

  8. Garrett Van Mourik

  9. Vince Del Vecchio

  10. Stan

  11. Henry Cox

  12. Yung-Chia Lin

  13. Zhi Zhuang

  14. Volodymyr Turanskyy

  15. Amilendra Kodithuwakku


  1. LLD key embedded features by Peter Smith

  2. Multilib implementation code reviews by Michael Platinigs

  3. Other code reviews


LLD key embedded features by Peter Smith

Two major areas:

  • Observability/discoverability - more understandable output, better usability.

  • Additional features:

    • Disjoint memory regions: multiple memory banks with different properties => possible linker script extension to distribute code over multiple free spaces in different regions.

    • RW data compression - copy RW data from ROM to RAM and decompress, can save ROM => could add to LLD or have a separate utility. It is important that compressions and decompression algorithms match! Maintaining multiple algorithms may add to overheads.

    • Memory-mapped variables - placing a section at a particular address, e.g. to access IO ports directly.


  • Petr Hosek:

    • GSoC project proposed for usability improvements [LLD] Linker Improvements for Embedded

    • In practice many issues are up to linker scripts issues (difference in behaviour of BFD vs LLD), thus being able to debug linker scripts easily helps a lot.

    • Disjoint memory - distributing by hand is very tedious, indeed.

    • Compression is helpful.

    • LTO support with embedded constraints of placement is another interesting area - there was a presentation by TI recently.

    • Another GSoC project idea is for machine readable format, e.g. JSON, for debug output (also link map, that is different between linkers now, thus tedious to parse) so that people can create their own visualizers/analyzers. Would be nice to convince the GNU community to implement the same format as well.

Demo by Peter Smith how the features mentioned above work in armlink (Arm proprietary linker).

  • armlink has 3 compression algorithms, one very basic run-length for 0’s which is already very helpful.

  • armlink supports placement attributes from C code, i.e. saves on manually editing linker script files (called scatter files for armlink).

  • armlink can show useful debug info like call graph/stack depth required, also breakdown of code/data sizes including the libraries to analyse code size issues.

  • armlink can trace symbols to show why a particular one was included.

Multilib implementation code reviews by Michael Platings

  • ⚙ D142933 Add -print-multi-selection-flags argument is about the proposed syntax for multi lib options not using actual command line names directly. It allows more limited, bit more stable API. Feedback is welcome!

  • Petr is reviewing and will get back with more feedback. Could we reuse tablegen here? May result in too much/complex dependencies.

  • We may consider making the feature experimental for the first LLVM release to allow later adaptation as per feedback from users.

Other code reviews


2023-03-02: Code reviews


  1. Michael Platings

  2. Prabhu Rajasekaran

  3. Anton Rapetov

  4. Henry Cox

  5. Jason Liu

  6. Michael Jones

  7. Petr Hosek

  8. Simon Butcher

  9. Siva Chandra

  10. Stan

  11. Tue Ly

  12. Vince Del Vecchio

  13. Yung-Chia Lin

  14. Yvan Roux

  15. Alan Phipps

  16. Todd Snider

  17. Peter Smith

  18. Volodymyr Turanskyy


  1. Multilib implementation code reviews by Michael Platinigs.

  2. MC/DC implementation code review by Alan Phipps.

  3. FatLTO by Petr Hosek.

  4. Other.


Multilibs code review

  • Michael:

    • Patches in review, few rounds of discussions happened and comments addressed.

    • One patch landed, 6 more to finish.

    • How to speed up or accept the current version with the intent to improve/address any issues?

  • Feedback form Petr:

    • The team reviewed the RFC in detail, the response will be posted on Discourse in coming days.

    • Suggestion: There are changes to internal API and adding new file formats (which are UVB - user visible behavior), so for internal changes it should be OK to land, UVB may need a bit more discussion.

  • Michael: Could/should we be more aggressive: accept a format now as an experimental feature, so warn that it may and likely will change in the future? May commit now, but review/refine before LLVM17 release to have it as stable as possible by the next release.

  • Peter: It would be nice to be able to give it a try with real projects and see if it works, rather than keep overthinking.

Agreed: Petr posts the response on Discourse, then if after the Discourse discussion there are no blockers, we commit the current format and try to refine it for LLVM17.

MC/DC code review

  • Petr: Someone on the team is reviewing the patches, it goes a bit slower than wanted, but in progress, not forgotten.


  • Petr: FatLTO is progressing, there is an RFC and patches will be available soon. Approach aligned with LTO maintainers.

  • The idea of FatLTO is for object files to contain information for both normal and LTO linking (i.e. binary and IR code).

  • TI presented a revised version of LTO for embedded/linker scripts recently, their solution is similar to/compatible with FatLTO.

  • Peter: Someone reported an issue with using LTO for embedded recently, see LLVM Embedded Toolchain for Arm issue Could you please include llvm-link, llc and opt? · Issue #187 · ARM-software/LLVM-embedded-toolchain-for-Arm · GitHub - they are using llvm-link, llc and opt manually to avoid the pitfalls of the default LTO.

  • Todd explained the details of the TI solution from the presentation - the two teams will talk to each other to further align the approach and implementation.


  • Peter: FOSDEM embedded developers were asking about a way to embed a section, e.g. a checksum, into the output image at the link time.

    • Petr: why is build-id not enough? Looks like something very custom/special.

    • Suggested that it would make sense to start a topic on Discourse to explain the use case, then consider possible solutions.

  • Peter: Use of TLS (thread local storage) in embedded projects. Picolibc uses TLS and initializes it in the linker script. The linker script and the library need to agree on the calculations of relevant addresses. LLD and GNU LD disagree on this - Peter is looking to create a reduced reproducer.

    • Is anyone using TLS in embedded apps? Vince: No, but had similar issues.

    • Is this going to change with C11 used more in embedded? Something to look out for in the future.

    • Peter will post an issue with the reproducer upstream.

1 Like

2023-03-30: Multilib, profiling runtime


  1. Gulfem Savrun Yeniceri

  2. Henry Cox

  3. Mandeep Singh Grang

  4. Michael Jones

  5. Nathan Sidwell

  6. Prabhu Rajasekaran

  7. Paul Kirth

  8. Petr Hosek

  9. Pierre

  10. Siva Chandra

  11. Stan

  12. Vince Del Vecchio

  13. Yung-Chia Lin

  14. Peter Smith

  15. Volodymyr Turanskyy


  1. Multilib code reviews.

  2. Other code reviews in progress.

  3. Embedded profiling runtime. Include profiling lib? · Issue #197 · ARM-software/LLVM-embedded-toolchain-for-Arm · GitHub and Profiling contribution by rgrr · Pull Request #204 · ARM-software/LLVM-embedded-toolchain-for-Arm · GitHub

  4. Building runtimes for bare-metal.


Multilibs code review

  • RFC and list of patches [RFC] Multilib - #5 by mplatings

  • Peter: The reviews are accepted by Arm, need confirmation from others in the community.

  • Petr: Will follow up on remaining reviews shortly.

  • Peter: A related question: In case there is a newlib installed from a distro package: how to make it work with clang?

    • Option could be to provide the config file to point there.

    • Could we inject an external multilib config file to use an existing set of multilibs?

    • Petr: There was a comment in the review that now the location of the yaml file is hardcoded - would be great to allow configuring it via a command line option, would solve this use case as well.

Other code reviews

Profiling runtime

  • Peter: A request raised for the LLVM Embedded Toolchains for Arm Issue #197

  • One option is to create a trivial runtime that would dump the counters somewhere as suggested in the issue discussion thread.

  • Wider question is how to add bare-metal support to the compiter_rt?

  • The PR Pull Request #204 suggests an implementation based on reusing compiler_rt pieces, which goes in the right direction, but only provides a very narrow Arm semihosting-specific implementation. How to generalise?

  • Can we provide an interface inside compiler_rt that can be used to tailor actual implementation of storing the data, suitable for bare-metal use cases as well?

  • Petr: The idea makes sense, the profile runtime is not in the best shape now, it would be great to refactor it and rewrite in C++. Would be good to have a header-only minimal implementation to allow easy reuse between actual implementations.

  • The team is very much interested in the implementation, but there was a lack of time to progress.

  • https://cs.opensource.google/fuchsia/fuchsia/+/main:src/lib/llvm-profdata/llvm-profdata.cc is an example of a minimal runtime we use for our kernel, we would like to break it up and upstream individual pieces so it can be reused for other embedded targets.

  • Exists a local patch in progress, the team will need help to progress it upstream.

  • Best way to start would be to do clean up/refactoring.

  • People who have downstream modifications - would be useful to know what kind of changes are there and why, i.e. how to refactor to accommodate for these? Examples:

    • Split of data to minimise the size of the resulting executable.

    • Size of counters: 32 vs 64 bits.

    • One runtime is used for both profiling and code coverage, thus maintains data for both - could be configurable.

  • Petr may post on Discourse a list of ideas for refactoring based on internal discussions.

  • A good topic to discuss in EuroLLVM 2023.

Building runtimes for bare-metal