Monthly LLVM libc meeting

Runtimes C
1

Hello,

I’m setting up a regular (every 4 weeks) meeting for developers of, and anyone interested in, LLVM’s libc (partly inspired by the one being set up for libc++).

The goal is to provide a space for future and current libc developers and users where we can discuss what we’re working on, what is open for more work, and what is blocking us.

There should be an event on the LLVM community events calendar by the time this is posted.

The meeting will be held on Google Meet (meet.google.com/otv-dmps-mcc) at 09:00 PST/PDT which is also 12:00 EST/EDT and 18:00 CET/CEST. The first meeting will be on Thursday, October 26th, 2023, and then every four weeks after that (excluding holidays, such as US Thanksgiving).

The meeting notes will be posted on this thread after each meeting, and the agenda will be posted before each meeting. If you want to add to the agenda, the easiest way is to send a message in the #libc channel in discord, or send me a message on discourse/discord.

4 Likes
2

Here’s the agenda for tomorrow’s meeting:

  • Introductions
    • Explaining the structure of this meeting
      • Start with the topics in the agenda
      • If we get through those, move to Q&A/open discussion
      • If we finish with that, end the meeting, even if it’s early.
  • (michaelrj): How many people are interested in working on the project, and how many are interested in using the project?
    • How can we enable what you’re doing?
    • What needs to be done before the libc is “complete” for your purposes?
  • Here’s what we’re working on:
    • (michaelrj): working on finalizing printf for google use
    • (lntue): working on double precision math functions
  • (jhuber): GPU math status?
  • (mcgrathr): How should special compiler flags be handled?
    • Should we allow changes that make certain compiler flags necessary for optimal performance?
      • This can cause problems for customers that don’t use our build system
3

Here are the notes from today’s meeting (lightly edited):

  • What needs to be done before the libc is “complete” for your use case?
    • Jhuber: Using LLVM libc for GPUs
      • Waiting on compiler fixes to land before varargs are available, as well as math tests for the GPU.
      • Tue is working on improving the exhaustive math tests so that they can test against the host system, since if we test that against MPFR then we can know they all match.
    • Phosek: Using LLVM libc for Fuchsia and Pigweed
      • Fuchsia is a microkernel OS
        • Currently using an old fork of musl, they want to move to something with better non-linux support, and getting updates.
        • Fuchsia needs a complete C library, and ideally C++ as well.
        • The Fuchsia libc has been slowly replacing parts of musl with functions from LLVM libc.
      • Pigweed is an embedded framework
        • LLVM libc would be a replacement for newlib
        • Most embedded systems don’t need all of libc
          • Memory functions, math, maybe sprintf
        • The libc might be complete for embedded, testing is ongoing.
        • The pixel buds pro are the first product they’re trying to move to LLVM libc, and it looks like all the functions they need are available.
        • There are other ones in progress, but they may need more. Every embedded system is different.
  • What are people interested in working on?
    • Lucian: Curious about streams, security and performance critical functions.
      • For security critical functions: The networking functions are very relevant, as well as having more fuzzers for existing string functions.
      • Having better support for operating with sanitizers, which traditionally intercept the libc functions since other libcs like glibc don’t cooperate with the sanitizer
        • Fuchsia builds sanitizers into their libc, since they build from source, and they’d be willing to upstream their internal API.
        • They’d also be interested in (if possible) working with several libc vendors to make this API more generally available.
      • Fortify source is something we’d like to support, but haven’t yet.
        • We’d like to coordinate our fortify source development with clang, since that would make the implementation cleaner within libc.
      • Another option would be matching the hardening modes that libc++ has.
  • Here’s what we’re working on:
    • (michaelrj): working on finalizing printf for google use
    • (lntue): working on single precision math functions and some initial support for C23 math functions
    • (jhuber): Poked at scanf, but has had issues getting it to compile. Also would like to build libc for GPUs and CPUs at the same time.
      • One of the important things for the GPU build is that it has to be built with an up-to-date clang since it requires some magic builtins only added to clang.
        • The current problem is that the libc cmake assumes that there’s only one target at a time.
        • It might be easier to just run cmake multiple times as separate builds using the runtimes build.
        • We might need to add ordering to the runtimes build, since it currently builds in any order.
        • The current way that libc++ can do multiple builds for multiple targets is by putting them in lib/target-triple/…
        • Another way would be to build the libcs with different suffixes (e.g. libc-target-triple.a)
        • One of the things being considered for the runtimes build is having a list of targets to build for, as well as a list of which of those can run tests.
  • Status of GPU math functions
    • The next step is getting exhaustive testing running so that we can test against the openCL standard.
    • We’d like to be able to provide multiple implementations so that we can provide a switch between speed and accuracy.
  • How do we balance having broad hardware support vs having the best performance
    • Using ifuncs to select a specific implementation is the standard method, but not necessarily the best one.
    • One way to handle this is to treat libc sources as part of your program, which allows for more targeted optimization but harms portability
    • How do we inform external users about special flags that certain files need?
      • Some users don’t use our provided build system, so having flags that are necessary for optimal performance can be a stumbling block if they aren’t made aware
      • We should probably add those flags to our config system since it’s JSON and so fairly easy to consume for downstream users
  • Lucian brought up the idea of benchmarking functions using shortcuts through undefined behavior, such as what strlen does: https://github.com/llvm/llvm-project/blob/main/libc/src/string/string_utils.h#L61
    • We currently have benchmarking for memory functions, math functions, and strtof.
    • Having more benchmarks would be nice.
      • Are we benchmarking to check our own performance or to compare against other libcs?
  • It would be nice to make our test suite available to use with other libc implementations
    • (michaelrj) I’d like to follow the design that libc++ uses, which is having the main test suite and marking some parts as testing internal details.
      • This allows us to have tests that check our implementation thoroughly, but also work for other libcs that only match us in the public API.
4

Just as a heads up: There will be no meeting on November 23rd since it falls on US Thanksgiving, currently the plan is to have the next meeting on December 21st.

5

Here is the current list of topics for tomorrow’s meeting:

  • (michaelrj): Most of the Google contributors are going to be on vacation from Dec 22, 2023 to Jan 1, 2023
  • (jhuber): remove the special handling of -ffreestanding
  • (jhuber): need a way to build the GPU and CPU libraries in the same LLVM config
  • (SchrodingerZhu): wants to know priorities for symbols/syscall wrappers
  • (SchrodingerZhu): interested in new buildbot configurations
    • “hermetic build on archlinux fails again because __stack_chk_fail is not included in integration test and hermetic builds”
6

Here are the notes from today’s meeting:

  • -ffreestanding
    • Need to talk to Guillaume (gchatelet)
    • The original reason for the flag was to avoid the compiler calling builtin memcpy inside memcpy
    • -fno-builtin prevents the memcpy inlining, but it also prevents inlining in later, and also causes issues with LTO
      • LTO and inlining are very important in GPU builds, but also anyone else who is building from source
    • Original patch: ⚙ D74162 [Inliner] Inlining should honor nobuiltin attributes
    • It would be nice to have a reproducer so it can be checked if the problem is fixed
  • Need a way to build CPU/GPU libraries at the same time
    • Basically we’d need something similar to runtimes which lets you do two similar builds at the same time
    • lntue is working on fixing our cmake, but there’s a lot of “if target_arch”
    • Runtimes already supports this
      • The target triple for your host is the “default” runtimes build, and ends up in the normal spot
      • There’s another cmake variable that has a list of target triples, and this will build those as well.
        • You can also have target specific build flags passed through
      • There’s basic support for multilibs
    • Right now the GPU target isn’t entirely based on the triple, there are also specific arguments it needs.
    • See for example llvm-project/clang/cmake/caches/Fuchsia-stage2.cmake at 11140cc238b8c4124e6f9efacb1601f81da096a0 · llvm/llvm-project · GitHub
    • It would be nice to look into building the libc with FatLTO
    • The runtimes build is being moved away from being as dependent on target triples
    • There is a brute-force version of math tests working on GPUs
  • What are the priorities for new development?
    • Syscall wrappers are always useful
    • The symbols necessary to build for libc++ would be nice
    • Dynamic linking/loading isn’t a priority, but it would be nice to have eventually
      • Fuchsia has their own loader which has a compatible license, but hasn’t upstreamed it. That would be a good starting point.
7

Here’s the current list of topics for tomorrow’s meeting:

  • (michaelrj): List of things Google users have asked for:
    • Finishing pthreads
      • (michaelrj) I am currently working on this
    • Finish up stdio design
    • Finishing floating point functions
      • Double precision, float128, float16?
      • In progress by lntue
    • Fixed point functions
    • Wide character support
    • Embedded support
      • Code size optimization
      • arm32/risc-v 32 support
      • Various non-standard embedded utilities (e.g. sys/queue.h)
    • First-class sanitizer support
      • Maybe phosek can talk about this?
  • (michaelrj): What do external users need?
8

Notes from today’s meeting:

  • First-class Sanitizer support
    • Still in the exploration phase
    • Motivated by experience from Fuchsia
    • Existing sanitizers work by intercepting libc calls, which has similar challenges to llvm-libc’s overlay mode
    • The Fuchsia sanitizers use lower level apis which go underneath the libc to use the sanitizers
    • They also instrument most libc functions
    • This experience could be reused for LLVM-libc
    • We could also try to use this for embedded since it would save binary size and performance overhead
  • What do external users need
    • Jhuber: needs to finish the freestanding compile flag fix with gchatelet
      • [RFC] libc -ffreestanding / -fno-builtin
      • Doing -fno-builtin isn’t good enough since the GPU builds need to include stdint.h and that may pull in extra pieces without -ffreestanding
      • This may end up needing clang changes
      • We should likely look into making an LTO buildbot
      • This may be a good use case for fat-LTO
        • fat-LTO embeds the bitcode alongside the machine code into the object so that it can be used for optimized linking
      • We should look into disabling loop idiom recognition for certain sections of code
        • We don’t want the memcpy implementation to be turned into a builtin call
      • There is some special handling on some special functions that can’t get eliminated during LTO
    • Mikhail
      • Working on porting to risc-v
      • The buildbot is currently broken, working on fixing it
      • A few patches to hopefully enable fullbuild on risc-v 32
  • Build system changes
    • Added -werror
    • Had some troubles since rooting the build in the llvm directory was setting some default flags
      • It might be good to remove LLVM libc from the enable projects list and move it exclusively to runtimes
      • LLVM 18 branch is the 23rd, so we could add a warning that we’re removing it in the next major version (6 months).
        • We’d need to figure out a way to build headergen, or replace it with something that doesn’t need tablegen
    • We’d like to enable crossbuilding fullbuild, but we need to figure out how to set up getting kernel headers
  • C23 additions
    • Bitwise operations in the library
      • It would be good to have these since it might be a good differentiator
  • Math status
9

Here’s the current list of topics for tomorrow’s meeting:

10

Notes from today’s meeting:

  • Commit Policy
    • People seem in agreement that review being strongly recommended is fine
    • Jhuber: being able to push in an emergency is important
      • Michaelrj: fix forward can be good, but also can break things more, be careful with it
  • “Good first issue”
  • Fixed Point:
    • Support for fixed point arithmetic in libc · Issue #74560 · llvm/llvm-project · GitHub
    • Currently in the process of setting up support
    • Compiler support is spotty
    • Leonard Chan has been working on compiler-side support
      • Currently we are only supporting clang 18+ for fixed point, since that’s the only compiler we are sure will work
    • Using fixed points instead of soft floats gives significant performance improvements on Cortex M0s and other embedded targets
    • Fixed point support isn’t in the standard, it’s technically still just a proposal
      • The proposal hasn’t changed since 2006, so it’s likely to be stable
      • Clang and gcc have support currently
        • Gcc requires a recompile to use it
    • As far as we know, there’s currently no open-source libc that supports it right now, so we will be the first
      • Erikson has a toolchain that supports fixed point support, but their libc is proprietary
  • Update on dependencies, compile + linking options, and flags propagation
    • Cmake has been fragile
    • lntue has been working through the technical debt
    • Planning on removing all the generator expressions
    • There may be a few small patches, but there will have to be a big patch doing the main change
    • Planning to have a new file for compile options
      • Put the test options in there
  • Fuzztest
  • Freestanding
    • Currently can’t do LTO with a libc since the compiler doesn’t want to inline libc
    • Look into having specific builtin disabling flags on specific functions
    • Jhuber wants ffreestanding since that simplifies the GPU build
      • The whole GPU environment needs to be freestanding
    • Look into making the inliner recognize specific “-no-builtin” functions to avoid inlining them
    • Being able to LTO the entire application is an important feature
  • Jhuber: working on updating the GPU build
  • Future cmake design
    • Currently the cmake assumes we’re building for the host, in future we’d like to make cross compiling and emulator testing easier
    • Ideally we’d match how libc++ and compiler-rt do cross compiling so that we can all work well together
  • GSoC
    • Proposals are still open (possibly)
    • Need to set up your idea as a mentor, then get a participant to write up a proposal
    • Deadline for the proposals is end of February
      • LLVM had a soft deadline of February 5th
      • If we want to do it we need to do it as soon as possible.
    • Idea for proposal:
      • Float16 support
      • Work on GPU support?
  • C23 support
    • Stdbit is in progress
      • We have most of it done
    • There are internal customers asking for C23 support (since it’s not as available elsewhere)
  • Nick is working on some integration for Bionic (Android)
    • Currently working on continuous release
  • For emulation
11

Here’s the current list of topics for tomorrow’s meeting:

12

Notes from today’s meeting:

  • Hand In Hand
    • Link to RFC: [RFC] Project Hand In Hand (LLVM-libc/libc++ code sharing)
    • What are the goals?
      • We want to close the gap on libc building for libc++
      • We also want to stop duplicating code between the projects
    • Libc++ currently depends on a conforming libc
      • This may be like adding a new libc that it supports
      • Some functions will probably have options to use the system libc, but others may not require this
        • Calling ctype functions on the default c locale is the same in all libcs
        • Similarly, string to float should always give the same result
    • Currently, libc++ can’t be built fully on top of LLVM-libc
      • Embedded targets and some Fuchsia uses
    • The new design would change the libc surface that libc++ uses
      • Some platform independent functions would always come from LLVM-libc
      • Some other functions would still come from the public libc interface
    • This would allow us to share code that can’t go through the public interface
      • The from_chars interface is similar to the strtod interface, but from_chars can’t easily call strtod
    • Libc++ already depends on other directories
      • Specifically libcxxabi and libunwind
      • Adding a new interface will hopefully not be too large
    • Some of the duplicated interfaces are in installed headers
      • We need to (at least initially) avoid touching those since we don’t want to leak the libc internal interface
    • Build system integration
      • We need to be careful of the case where users take just individual folders for individual libraries
        • Examples:
          • Fuchsia uses libc as an individual folder
          • Pigweed uses libc and libcxx as individual slices
          • Chromium builds libcxx as part of chromium
        • We’ll need to decide on include path policies, and warn downstream users so that they can update their build systems
      • Officially libc++ doesn’t support users who take just an individual folder from the monorepo
        • Their policy is basically just copy any extra folders too
    • The existing libcxxabi interface is problematic
      • There is automated code copying, and some parallel headers
      • We don’t want our project to end up like that
    • The libc++ project has previously tried to share code that shows up in the installed headers
      • It was a lot of effort, there’s a lot of macros and other specific code style choices that need to be kept in sync
    • Shared underlying API design
      • Mutexes in libc currently have a flag to mark it as recursive or not
        • This means that lock/unlock always has a branch to check
      • Libc++ has two different types for mutex and recursive mutex
        • Since they sit on top of libc, they have to use the mutex type with flag under the hood, causing performance bottlenecks
      • A shared underlying interface could avoid these bottlenecks
        • Louis Dionne would be very interested in this
    • If the existing C interface is designed in a way that doesn’t work well with C++, it is possible, though difficult, to change the C standard
      • Android went through the process to make this change with pthread clock waits, and got changes in bionic+glibc and libc++ and POSIX
      • the specific “this solves an impedance mismatch problem between libc and libc++” use case was sufficiently obvious+convincing that it “only” took about a year start to end — everybody “got it”.
  • Petr Hosek is working on porting code for a baremetal codebase
  • Nick Desaulniers: libc and libc++ interface layer
    • Libc++ currently needs ~80 symbols that LLVM-libc doesn’t provide
    • Carve outs can remove some of these, but not all
  • SchrodingerZhu Macro policy inside of LLVM-libc
    • Existing:
      • Headers in libc/include have dependencies on macros and types
        • Some macros are defined in UAPI headers, but some we provide ourselves
        • Some macros are defined in llvm-libc-macros, some in API.td
      • For internal usage
        • Current use is split between using the public header vs depending on our internal header directly
    • Future
      • We should try to provide a complete set of macros for C23
        • We may need to use linux UAPI headers for specific macros/types
          • But also some are unusable since they don’t match the userspace interface
        • Linux headers are mostly stable between versions
        • Posix headers often have a lot of implementation defined pieces, we’ll need to keep that in mind
      • For internal usage
        • Types and macros should always be defined in llvm-libc-types/llvm-libc-macros
        • Ideally headergen would only be used for functions
  • Nick Desaulniers: Lint rules
    • We have a rule that checks if a header comes from the system
      • We need to clean up code that currently is doing this
13

Here’s the current list of topics for tomorrow’s meeting:

Feel free to send me a message if you have another topic, or bring it up during the meeting.

14

Notes from today’s meeting:

  • Freestanding Design:
    • Inlining isn’t the only problem for doing optimizations with libc
      • Some libc functions are “magic” in that the compiler will identify them and handle them differently
        • E.g. sin is a “nowrite” function
    • We might have to make the compiler not do its magic if the symbol is available
    • The meaning of freestanding may need to be changed from what it currently is
      • The idea would be freestanding turns off libcall optimizations, since any given function (e.g. strlen) should be defined locally and may be whatever the user defines.
      • We may want to have a different flag for “the libc is in the module, but the semantics are the same as normal” so that it can do libcall optimizations, but using the provided libc.
    • Freestanding implies nobuiltins, which can prevent inlining in LTO
    • We want inlining, libcall optimizations and LTO, but the compiler generally doesn’t handle having the libc built with the user code well
      • We don’t want a libc function to get inlined, then loop idiom recognized back to a call of the libc function
        • E.g. call to strlen gets inlined, then the optimizer sees the loop for strlen and turns it back into a call to strlen
      • One design for this would be to have the optimizer mark which functions it’s touched (“poisoning” it) so it doesn’t try to do more to it
    • In different stages, we need to handle these functions differently
      • During constant folding (for example) we still want these functions to be treated as magic (e.g. sin(pi) should still be the constant 0)
      • During LTO (for example) we want these to just be regular functions
        • Maybe inlined or otherwise transformed for better performance
    • It may be useful to have a way to mark a function as the “canonical” version of a libc function
      • That way the compiler can treat it as the abstract libc function
      • But also the compiler might have multiple copies of the same function so it can be optimized/inlined as a regular function as well
    • The linux kernel has already handled some of these issues so that it can have LTO and libcall optimizations, but there are specific kludges to get that working.
  • Baremetal libc++
  • Raising Compiler Version
    • Currently the compiler version we require is based on what our buildbots have
      • The builtbots got updated recently, so the minimum version would need to be updated
    • If we want to move to C++20 we need to increase the minimum compiler version
    • The proposal is to move to something like what libc++ does, which is the most recent two versions of clang and the most recent version of GCC
      • Matching libc++ would let us move towards LLVM runtimes having a unified support window
    • So far we don’t have any users clamoring for old compiler support
    • The GPU server code needs to be built in the libc projects phase
      • This kinda puts it into the overall LLVM compiler support window, which goes back to much older versions.
      • It also reaches into libc sources right now, including touching the printf converters which have a lot of dependencies in __support
    • While libc is still available in projects, we should probably survey the linux distributions for which compilers we can use
    • LLVM 19 will be branching in July, so we could add a warning to not use the libc as a project now and then remove project support after the branch point.
      • The GPU build might be able to make this deadline, but it depends on the offloading runtime. There’s complexity there.
  • Policy on assembly
    • The plan is to avoid assembly for performance (since it tends to be brittle)
      • It’s generally better to improve the compiler instead
    • For functions that specifically need assembly (E.g. setjmp/longjmp, syscall) we need to decide how we’ll write that
      • Things like startup code will also need assembly
15

Here’s the current list of topics for tomorrow’s meeting:

  • Improving extern build integration support (Roland McGrath)
    • Header generator vs external builds
    • File / header lists
    • Per-function test vs specific test file support
  • LibcTest.h vs gtest/gmock APIs (Roland McGrath)
    • Ability to use EXPECT_/ASSERT_ in helper functions (outside fixture methods).
    • Custom matcher API: explainError vs MatchAndExplain, etc.
    • Death tests.
  • Builds (Roland McGrath)
    • Would like to only support -DLLVM_ENABLE_RUNTIMES=”libc” and …/runtimes
      • No more -DLLVM_ENABLE_PROJECTS=“libc”
        • Libc-hdrgen
      • No more …/llvm
      • Documentation is all over the place.
      • Too many build configurations to test
      • Can we make the bootstrap build explicit?
  • Fenv tests (Roland McGrath)
  • Landing the LTO/inlining fix soon (Joseph Huber)
  • Performance Testing (Joseph Huber)
  • Libc++ status (Joseph Huber)

Since this has more topics than usual, I’m planning to try to get through them relatively quickly.

16

Notes from today’s meeting:

  • Extern build integration support:
    • Users of the LLVM-libc code but providing their own build system
    • Header Generator:
      • Hasn’t been a problem yet since users haven’t been using fullbuild, but they’d like to move to fullbuild in the future
      • External users want a more standalone header generator
      • I (Michael) am hosting interns this summer, they start in ~2 weeks and will be rewriting the header generator to be more standalone
    • File organization
      • Currently external builds need to hand maintain their build system
      • Every time a new file is created they need to add it manually
        • New interdependencies cause a lot of issues
      • Most of the code has a uniform structure
        • Headers, sources, and tests all have a consistent pattern
        • But there are exceptions, e.g. new fenv functions
          • There are also some tests that depend on multiple functions.
          • It would be good for us to try to keep that uniform structure.
      • It would be nice to have a way to export the dependencies to simplify the downstream build maintenance
        • Something machine ingestible
        • It would also be nice to have annotations for what dependencies a function needs (e.g. OS, architecture)
        • This could also potentially be used by bazel
      • Enforcing fine grained dependencies inside cmake is in progress
        • Tue is working on it
        • Once this is done there should be a cmake tool to print out the dependencies
      • Ideally the bazel build will be moved to match the structure of the existing libc directory.
  • Libctest
    • Not a subset of gtest anymore, it’s just a similar framework that’s standalone at this point
    • There are some incompatibilities,
      • specifically in gtest all the assert macros are allowed to be within sub functions, whereas our current framework doesn’t.
      • The custom matchers also don’t work as gtest matchers
    • We should probably move to being gtest compatible
      • If we do, we should have an upstream gtest buildbot
    • Roland is working on some patches to fix the assert macro issue, but would like some help on the matchers.
  • LTO/Inlining change
  • Performance testing
    • There’s some code in tree for performance testing already, but it might not be usable for GPUs
      • The existing method on GPUs requires a significant amount of inline asm to measure clock cycle latency
      • It might need to be put in a GPU subdirectory
    • The end goal is to measure the latency of things like math calls on the GPU.