<atomic> for bare-metal development on ARM microcontrollers

Dear libc++ developers,

I am experimenting with clang++ together with libc++ on ARM M4 microcontrollers, with the intent to test the viability of the new C++20 coroutines on them (using the -fcoroutines-ts switch of clang). I have had some moderate success, resulting in a usable toolchain based on the archetypical newlib C-library: https://github.com/burnpanck/docker-llvm-armeabi .

However, currently, I build libc++ without threading support, because I'm targeting bare-metal without RTOS. Unfortunately, that immediately disables `<atomic>`. This is a pity, because even if there is no fully-fledged threading available, bare-metal development has to deal with concurrency in the form of interrupts. Here, library support for `<atomic>` would be much appreciated.

Is there any particular reason why `<atomic>` needs threading support under all circumstances? I understand that a library implementation in absence of compiler intrinsics requires at least a "unique_lock". But on a microcontroller, one could just disable interrupts for the duration of the atomic operation, as there is no need for memory synchronisation. Would it make sense to try detaching atomics from threads to support vendor supplied atomics, similarly to `<__external_threading>`?

The alternative, trying implement threading support for that target (with working synchronisation primitives but no actual threads) seems a much heavier task. At least, it also pulls in the requirement for a monotonic clock.

I would love to hear your thoughts on the subject.

Thanks and best regards

Yves

I think you want libc++ support for “freestanding”. We’ve discussed this e.g. in “D56913: decoupling Freestanding atomic<T> from libatomic.a”. Olivier started some of this work for atomic. It sounds like you’d like this work to continue :slight_smile:

I think you want libc++ support for “freestanding”.

It looks like that's what I want :-).

We’ve discussed this e.g. in “D56913: decoupling Freestanding atomic<T> from libatomic.a”. Olivier started some of this work for atomic.

Great - I had a quick glance over D56913 and the current bleeding edge of `<atomic>`. It seems that this should work for me. Except that there is still a "# error <atomic> is not supported on this single threaded system” at the top of `<atomic>`. Why is that? It seems that Olivier Giroux’s implementation contains it’s own lock, requiring no further thread-support. Anyway, I will give bleeding edge a try, simply removing that check and see how far I get.

It sounds like you’d like this work to continue :slight_smile:

I certainly do. I strongly believe in C++ for small embedded and it’s a pity that MCU SDKs are mostly C. It would be great to get access to all the niceties of a modern C++ library just by implementing a handful of C primitives for the platform (one may always dream, right?). Is there anything I (as someone without deeper knowledge of libc++) could help with?

I think you want libc++ support for “freestanding”.

It looks like that's what I want :-).

We’ve discussed this e.g. in “D56913: decoupling Freestanding atomic<T> from libatomic.a”. Olivier started some of this work for atomic.

Great - I had a quick glance over D56913 and the current bleeding edge of `<atomic>`. It seems that this should work for me. Except that there is still a "# error <atomic> is not supported on this single threaded system” at the top of `<atomic>`. Why is that? It seems that Olivier Giroux’s implementation contains it’s own lock, requiring no further thread-support. Anyway, I will give bleeding edge a try, simply removing that check and see how far I get.

I think it’s just that nobody’s tried to make it work.

It sounds like you’d like this work to continue :slight_smile:

I certainly do. I strongly believe in C++ for small embedded and it’s a pity that MCU SDKs are mostly C. It would be great to get access to all the niceties of a modern C++ library just by implementing a handful of C primitives for the platform (one may always dream, right?). Is there anything I (as someone without deeper knowledge of libc++) could help with?

Sure! Patches would be great since you’re trying it out. I’m happy to help review with other libc++ folks as I think this is important.

I think you want libc++ support for “freestanding”.

It looks like that's what I want :-).

We’ve discussed this e.g. in “D56913: decoupling Freestanding atomic<T> from libatomic.a”. Olivier started some of this work for atomic.

Great - I had a quick glance over D56913 and the current bleeding edge of `<atomic>`. It seems that this should work for me. Except that there is still a "# error <atomic> is not supported on this single threaded system” at the top of `<atomic>`. Why is that? It seems that Olivier Giroux’s implementation contains it’s own lock, requiring no further thread-support. Anyway, I will give bleeding edge a try, simply removing that check and see how far I get.

I think it’s just that nobody’s tried to make it work.

Right, I just looked at it and I don’t see any reason why the combination of -D_LIBCPP_FREESTANDING / -D_LIBCPP_HAS_NO_THREADS would not work. Yves, would you be willing to check it out and report on whether it works? We may be able to get a simple patch in.

Louis

I have an important project to wrap up until mid next week, I won’t have time before that. Afterwards, I’ll definitely give it a try. I should be able to report until the end of the coming week, if that’s ok.

Yves

[…] Yves, would you be willing to check it out and report on whether it works? We may be able to get a simple patch in.

I have an important project to wrap up until mid next week, I won’t have time before that. Afterwards, I’ll definitely give it a try. I should be able to report until the end of the coming week, if that’s ok.

Absolutely. It’s not like you owe us anything :). Looking forward to your report.

Louis

Hi all,

I managed to compile a static build of libc++ bleeding edge (just before the forking of the release branch) for armv7em baremetal supporting `<atomic>`, with a few minimal changes. I know, the timing with respect to the release of 9.0.0 is terrible. Nonetheless, the corresponding patch is attached to this eMail. It consists essentially of three changes:
1. Do not blindly refuse <atomic> when _LIBCPP_HAS_NO_THREADS is defined. I removed corresponding `#error` at the top of `<atomic>`. Furthermore, I changed the logic in `<__config>` where _LIBCPP_HAS_NO_ATOMIC_HEADER is determined to ignore _LIBCPP_HAS_NO_THREADS.
2. In `<__config>`, define _LIBCPP_ATOMIC_ONLY_USE_BUILTINS when _LIBCPP_HAS_NO_THREADS is defined (instead of just when _LIBCPP_FREESTANDING is defined), since there is no other way to implement <atomic> without thread support. Arguably, when building without thread support, one is effectively doing a freestanding compilation, so one could require the user to supply “-ffreestanding”. However, it seems that this is not usually done on the systems I work with. Furthermore, it is conceivable that one wants to build libc++ for a hosted platform that does not supports threads but still would want to have support for atomics.
3. In `memory.cpp`, change `#ifdef _LIBCPP_HAS_NO_ATOMIC_HEADER` to `#ifdef _LIBCPP_HAS_NO_THREADS`. Since this ifdef protects against the use of threading primitives, the latter seems more correct, while the former leads to compilation errors with the new logic for _LIBCPP_HAS_NO_ATOMIC_HEADER. It seems to indicate that the implementation of `<memory>` uses threading primitives instead of atomics. Therefore, in standalone builds, where concurrency comes in the form of interrupts rather than threads, one has to be careful with the use of `<memory>`. If at some point libc++ on baremetal becomes officially supported, that issue should probably be documented somewhere.

I would like to contribute this patch, is this mailing list a good place to initiate the process. Of course, so far my patch fails to provide tests for the issue, which I’m not sure how to proceed with. I have never run any of the LLVM tests myself, as this seems to be extra challenging for cross-builds for baremetal.

In order to build a static library of libc++abi including libunwind, I had to work around a separate issue in libc++abi for out-of-tree builds. The problem is that the final merge step for libc++abi expects to find the location of the libunwind archive from the CMake target, which only exists for in-tree builds. To me, that seems like a bug in the CMake scripts for libc++abi - I would assume that the intention is to support static builds for out-of-tree builds. Note that in-tree builds didn’t work for me because the configure step chokes on the baremetal compiler as the “host compiler" to build LLVM - ironically, because it determines that the “host compiler” doesn’t support atomics. I worked around that by introducing another CMake variable that let’s me specify the location of libunwind explicitly. Those changes are also in the attachment as a separate patch. However, I’m not sure that my work around is really an acceptable solution to the issue. Consider this just an attempt at a bug-report. I will share the details of my CMake configuration at GitHub - burnpanck/docker-llvm-armeabi: Dockerised ARM Mircocontroller toolchain using only LLVM tools once I have cleaned up the scripts a little.

Yves

0001-enable-atomic-header-on-thread-less-builds.patch (2.33 KB)

0001-explicitly-specify-location-of-libunwind-in-static-b.patch (1.58 KB)

Hi Yves,

Hi all,

I managed to compile a static build of libc++ bleeding edge (just before the forking of the release branch) for armv7em baremetal supporting <atomic>, with a few minimal changes. I know, the timing with respect to the release of 9.0.0 is terrible. Nonetheless, the corresponding patch is attached to this eMail. It consists essentially of three changes:

  1. Do not blindly refuse when _LIBCPP_HAS_NO_THREADS is defined. I removed corresponding #error at the top of <atomic>. Furthermore, I changed the logic in <__config> where _LIBCPP_HAS_NO_ATOMIC_HEADER is determined to ignore _LIBCPP_HAS_NO_THREADS.
  2. In <__config>, define _LIBCPP_ATOMIC_ONLY_USE_BUILTINS when _LIBCPP_HAS_NO_THREADS is defined (instead of just when _LIBCPP_FREESTANDING is defined), since there is no other way to implement without thread support. Arguably, when building without thread support, one is effectively doing a freestanding compilation, so one could require the user to supply “-ffreestanding”. However, it seems that this is not usually done on the systems I work with. Furthermore, it is conceivable that one wants to build libc++ for a hosted platform that does not supports threads but still would want to have support for atomics.
  3. In memory.cpp, change #ifdef _LIBCPP_HAS_NO_ATOMIC_HEADER to #ifdef _LIBCPP_HAS_NO_THREADS. Since this ifdef protects against the use of threading primitives, the latter seems more correct, while the former leads to compilation errors with the new logic for _LIBCPP_HAS_NO_ATOMIC_HEADER. It seems to indicate that the implementation of <memory> uses threading primitives instead of atomics. Therefore, in standalone builds, where concurrency comes in the form of interrupts rather than threads, one has to be careful with the use of <memory>. If at some point libc++ on baremetal becomes officially supported, that issue should probably be documented somewhere.

I would like to contribute this patch, is this mailing list a good place to initiate the process.

The best place to contribute a patch is Phabricator, as detailed here: https://llvm.org/docs/Contributing.html

Of course, so far my patch fails to provide tests for the issue, which I’m not sure how to proceed with. I have never run any of the LLVM tests myself, as this seems to be extra challenging for cross-builds for baremetal.

I’ll let the libc++ maintainers chime in on that part.

I managed to compile a static build of libc++ bleeding edge (just before the forking of the release branch) for armv7em baremetal supporting `<atomic>`, with a few minimal changes.

I want to add that I did manually test the built libc++ in an simple (“blinky”) project that uses hardware timer interrupts to trigger state changes, relying on <atomic> for synchronisation. It appears to work, contrary to the non-atomic version of it.

Hi Yves,

Hi all,

I managed to compile a static build of libc++ bleeding edge (just before the forking of the release branch) for armv7em baremetal supporting <atomic>, with a few minimal changes. I know, the timing with respect to the release of 9.0.0 is terrible. Nonetheless, the corresponding patch is attached to this eMail. It consists essentially of three changes:

  1. Do not blindly refuse when _LIBCPP_HAS_NO_THREADS is defined. I removed corresponding #error at the top of <atomic>. Furthermore, I changed the logic in <__config> where _LIBCPP_HAS_NO_ATOMIC_HEADER is determined to ignore _LIBCPP_HAS_NO_THREADS.
  2. In <__config>, define _LIBCPP_ATOMIC_ONLY_USE_BUILTINS when _LIBCPP_HAS_NO_THREADS is defined (instead of just when _LIBCPP_FREESTANDING is defined), since there is no other way to implement without thread support. Arguably, when building without thread support, one is effectively doing a freestanding compilation, so one could require the user to supply “-ffreestanding”. However, it seems that this is not usually done on the systems I work with. Furthermore, it is conceivable that one wants to build libc++ for a hosted platform that does not supports threads but still would want to have support for atomics.
  3. In memory.cpp, change #ifdef _LIBCPP_HAS_NO_ATOMIC_HEADER to #ifdef _LIBCPP_HAS_NO_THREADS. Since this ifdef protects against the use of threading primitives, the latter seems more correct, while the former leads to compilation errors with the new logic for _LIBCPP_HAS_NO_ATOMIC_HEADER. It seems to indicate that the implementation of <memory> uses threading primitives instead of atomics. Therefore, in standalone builds, where concurrency comes in the form of interrupts rather than threads, one has to be careful with the use of <memory>. If at some point libc++ on baremetal becomes officially supported, that issue should probably be documented somewhere.

I would like to contribute this patch, is this mailing list a good place to initiate the process.

The best place to contribute a patch is Phabricator, as detailed here: https://llvm.org/docs/Contributing.html

Of course, so far my patch fails to provide tests for the issue, which I’m not sure how to proceed with. I have never run any of the LLVM tests myself, as this seems to be extra challenging for cross-builds for baremetal.

I’ll let the libc++ maintainers chime in on that part.

Yup, let’s start by opening a Phabricator review and we’ll let you know what needs to satisfy our testing requirements!

Louis

Hi Bastien,

The best place to contribute a patch is Phabricator, as detailed here: https://llvm.org/docs/Contributing.html

I should have RTFM more attentively - sorry. Had registered at Phabricator, didn’t understand the “Differential” button, skimmed “Contributing” and noted “mailinglist” somewhere, and here I was. Anyway, it’s now up: https://reviews.llvm.org/D65348 . For the potential libc++abi bug I mentioned, I’ll try to get into the LLVM bug-tracker tomorrow.

Thanks, Yves