Organizing Security Support for LLVM - Discussion

LLVM Project
1

Hello to everyone in the llvm community!

I’m Derek, founder of ostif.org and we are interested in assisting the llvm project with meaningful security support for llvm.

We’d like to start some discussion here about how we can best use our resources to help without being burdensome on the maintainers or finding non-issues. It is extremely important to us that if we spend time and resources on llvm that it is relevant to you and actually improves your project.

I’ve tried to learn about llvm, the community, and the project’s general security posture through as much documentation and community discussion that I can find. I do have some early observations that I’d like to confirm.

-The project is running coverity scans regularly and addressing concerns as they are raised.
-The project is on Google ossfuzz but has a backlog of issues that are unaddressed and most of the scans appear to be a few years old. (We see this often. This is usually because the fuzzers are mostly spamming non-issues that are outside of the security model of the project.)
-I’ve heard (anecdotally) that because llvm is mostly run locally on trusted source, or it is run remotely on hermetic VMs that are spun up every run that reliability issues such as generic crashes and assert triggers are not considered to be security issues. This makes sense to me.

However, this introduces a problem, as triggering crashes is how fuzzing finds issues, which means that easily triggered crashes act as blockers for the fuzzers to find more serious problems underneath.

We’ve talked extensively to our advisory council about how to best help llvm and how to address this problem without being a burden on the community. What we’ve come up with is building and documenting a threat model for llvm. One that looks at risks for the various functionality that the llvm core provides, and specifically spells out classes of bugs that might be excluded from that threat model because they aren’t relevant to llvm. The goal here is to identify what a security problem in llvm actually looks like, and also to identify what a non-issue is to eliminate reports that outside of the scope of your threat model.

If there is any supporting documentation that the community has related to “what is a security bug in llvm” or even a previous threat model, it would really help us throughout this process. If not, we can build something from scratch and we have a lot of experience working from zero.

Once we have a threat model established, we’d like to work on the ossfuzz implementation to achieve multiple goals. We want to expand code coverage to the riskiest areas in “llvm core” if they are not currently covered, and build a stronger set of fuzzers that don’t report as many non issues. This would have to be coupled with our team working on crash fixes as merge requests to sidestep the fuzzing issues created by simple reliability issues like asset triggers, or low/informational issues discovered by the fuzzers. As coverage expands, if we find any serious issues, we would report those through the regular llvm security channels for addressing.

This seems to be the least burdensome solution that we can come up with without your feedback so far.

  1. Build a threat model
  2. Work on fuzzers with prioritization for the identified high risk functions
  3. Tighten up existing fuzzers to reduce reports that are non-issues
  4. Submit MRs where necessary to fix minor issues (following your contributing guidelines and regular processes of course)
    4a. Any major security findings get reported through regular channels responsibly
  5. If this work is deemed successful and valuable by all parties involved, look into scheduling more follow up work to continue improving things.

I do have a couple of questions for the community, and I’m perfectly aware that there’s a lot of opinions here and there may be some disagreement on of the finer points, these questions are just intended to help guide us along our way to help do meaningful work.

  1. What do we consider to be “llvm core?” Can you point us to the repos that everyone MUST install to do anything with llvm? If we want to go one step further, which repos would be installed with a “typical” setup of llvm? We can also do this on our own by following basic setup guides and looking at what people are using llvm for the most, but we don’t want to miss any common use cases or omit a repo that is important to most users.

  2. Does this approach sound like it will help? If there are complications introduced by the idea, or if we create an undue burden on the maintainers through something we’ve missed, let us know so that we can revise the plan and make it more useful.

  3. Has a threat model for llvm ever been created before? Or are there docs identifying what the security model for llvm looks like? This is crucial because it will help us eliminate all of the non-issues and spam reports that you could get from us, as we can screen them carefully and only report things that matter. This is important for both the fuzzer design and our manual reporting of more serious issues.

  4. Do you have any questions for me? You can post them here, or reach out to me directly via email. I’m at firstname @ orgs url. We are open to all discussion and collaboration here, and anything we can do to help is on the table.

Thank you all for your tireless work on llvm. It is one of the best projects out there and i’m looking forward to helping if we can!

2

Hi Derek,

If you haven’t found it yet, the LLVM Security Group has a page here that might answer some of your questions, particularly our current take on what is a security issue, and how to report a security issue.

IMO the fuzzing effort for LLVM was not really focused on finding security issues, but rather more general robustness problems. My impression is that a number of things that it found early on were addressed, but that there has not been a concerted effort to keep the fuzzer happy.

3

This is great information!

I see at the bottom of the page that no sections of code have been defined as security centric. Are there proposals to make some pieces of llvm core secure? We could focus our work there to help and it sounds like it would be valuable.

IMO the fuzzing effort for LLVM was not really focused on finding security issues, but rather more general robustness problems. My impression is that a number of things that it found early on were addressed, but that there has not been a concerted effort to keep the fuzzer happy.

My fuzzing assumptions were pulled from the ossfuzz data here: Monorail - oss-fuzz - OSS-Fuzz: Fuzzing the planet - Monorail

Some limited info about llvm fuzzing here:
https://llvm.org/docs/FuzzingLLVM.html

as well as a few random discussions that I came across from years ago.

IMO the fuzzing effort for LLVM was not really focused on finding security issues, but rather more general robustness problems.

Would you say (in your opinion of course) that increasing the reliability of llvm generally is a valid goal even though it is not explicitly a security goal?

4

This is a hard question: the llvm-project is a mono-repo of multiple projects related to compilers and toolchains. The top-level README says:

The LLVM project has multiple components. The core of the project is itself called “LLVM”. This contains all of the tools, libraries, and header files needed to process intermediate representations and convert them into object files. Tools include an assembler, disassembler, bitcode analyzer, and bitcode optimizer.

At the top-level you have directories which map to high-level components. However inside the llvm directory, which is the historical starting point of the project (and referred to as the “core” of the project), there is a range of things that could be considered separated layers:

  • some support libraries (Filesystem abstractions, data structures, etc.)
  • some utilities libraries that forms the basis of “binutils” programs (so reading and modifying objects files, or debug info / DWARF, so that we have implementation or ar, objdump, nm, …).
  • the whole LLVM IR and the associated compiler optimizations
  • A series of backends for the “code generation” part of the compiler, going from LLVM IR to native assembly / binaries for various HW targets supported in-tree.

The other subprojects outside of the llvm/ directory may reuse part of the above. Some subprojects are completely disconnected from the llvm/ folder though (compiler-rt, libc++ for example).

We should improve the README and provide a high-level “map” of the repo I think.

5

That’s correct, although the Security Group is certainly open to changing that. We have accepted some security issues so IMO we should think about defining some areas as security relevant.

“make it secure” without specifying a context (which might or might not include a threat model) isn’t meaningful. The vast majority of LLVM code is used in unprivileged command-line tools where “is it secure?” isn’t a useful question. It would be like asking “how do we make grep secure?”

Now I believe there are some parts of LLVM that show up (or could show up) in some sort of web service, at which point all the usual web-service hardening kinds of things should come into play. But again we haven’t determined what any of those parts are, or whether it is on the LLVM project itself to engage in that hardening.

I think any professional software developer would say that increasing reliability is a valid goal. Fuzzing is one tool that can help achieve that goal. I personally think it’s clear from the number of open reports that the community as a whole doesn’t rate the fuzzers as hugely important.

But if you want to contribute to improving the usefulness and actionability of the fuzzing reports, that’s great and I’m sure people will welcome it!

6

LLVM already has plenty of known bugs. I think where we need help is in fixing them :slight_smile:

Besides the obvious (potentially exploitable) crashes in the LLVM core, a much more serious issue is related with incorrect optimizations and mismatch between the semantics offered by LLVM and assumed by frontends. But it’s very hard to realize which of these problems may have security implications. There’s evidence of such problems leading to vulnerabilities in compiled programs, but there’s no systematic way of evaluating the risk of bugs.

In the end, performance is always the #1 priority of generalist compilers. JIT compilers may have a different mindset. Don’t get me wrong: bugs are actively fixed in LLVM, but the hard ones don’t get prioritized (particularly because it’s hard to judge the impact in practice and the cost of fixing them is pretty high). Any research on bug impact would be very welcome!

2 Likes
7

I definitely misspoke there. What I should have said was, “has there been any proposed or draft proposals for defining any areas of llvm as having a security priority?” This would be so that we could align our observations with the views of the community on what is or isn’t a problem, and what is a problem but lies outside of the responsibility of llvm.

And I 1000% agree that context is needed when defining anything as “secure.” If you’re executing your own trusted code locally, that is completely different from a live service that uses llvm in some way.

One of the interesting things about developing a threat model is that it also defines all of the things that you’re not responsible for, which is both important for anyone performing security research, but also for users/admins who have to understand what risks they need to address themselves when they use llvm for something.

8

This is definitely something that we picked up on while learning more about the project, and we can do our best to find resources to help address that problem.

9

Potentially one of the main difficulties with defining a threat model is that
LLVM is a set of reusable libraries, which indeed do get reused in lots of
varied contexts. Furthermore, I think that whatever threat model we would come
up with would have to be (at least implicitly) accepted by a large majority of
the LLVM developers to be meaningful. It seems to me it wouldn’t work well if
the threat model would declare certain classes of bugs as security-sensitive if
a large group of LLVM developers would consider them to be just normal bugs.

With that in mind, I think the following might be the most obvious areas that
most of us could agree with to be security sensitive:

  1. Memory safety bugs in the parts of LLVM that are libraries that are linked in
    with most end-user applications built with LLVM-based tools. I believe
    examples are the sub-projects compiler-rt, libunwind, libc++, libc++abi,
    libc, flang/runtime. There probably are more.

  2. Somewhat more controversial are issues with security hardening features in
    the code generator. Examples of some security hardening features I’m thinking
    of are stack canaries, pointer authentication, speculative load hardening,
    etc.

    My experience is that people who use these features get upset when they
    discover a corner case where the hardening feature doesn’t do what they think
    the feature promises them. OTOH, many people dealing with security incidence
    response (e.g. CVEs) seem to classify these issues as merely a security
    issue, but not a vulnerability. See some more on that in @siddhesh’s
    recent GNU Cauldron presentation on Security Processes for the GNU toolchain project.

If you or anyone else also would want to discuss this further more
interactively, you could do so in the monthly online public sync-up call for the
LLVM security group, where everyone is welcome to join:
https://llvm.org/docs/GettingInvolved.html#online-sync-ups. The next one will be
held on November 21st, at 19:00 UTC (as indicated by the
LLVM community calendar).

10

(post deleted by author)

11

This is great info! Thank you!

Building a threat model for llvm is indeed complex. We would be most concerned about the use-cases that are security sensitive for obvious reasons.

Furthermore, I think that whatever threat model we would come
up with would have to be (at least implicitly) accepted by a large majority of
the LLVM developers to be meaningful.

Everything we do whether it comes to auditing, building tools, or documentation like threat models comes only as recommendations and we get your feedback before we publish anything. We will work with the community on draft recommendations and usually the final product is a simplified amalgam of recommendations from the researchers and the community. A great example of this is the threat model that we did with cURL here: (linking directly to the pdf of this report made the automoderator angry, removing to fix. The pdf is linked at the bottom of the curl article on our website.)

While making a full threat model for llvm would be mostly frivolous because most use-cases of llvm are “safe” because you’re running trusted code on a local machine, we need to identify the cases where the use-case introduces security risks. From there, we can identify how to scope our research to have the most security impact for real-world users of llvm.

The main thing that we’d need help understanding is how LLVM is commonly used in ways that require security, for example if any components of the STRIDE model are required to make the use of LLVM safe. (I know that this is nearly an impossible question, as LLVM is used in so many contexts and it’s impossible for any project to know how others are implementing it, but i’m hoping that we can make some educated guesses. The info from kbeyls is great!)

This would help us to narrow our focus to spend time on the things that will help the most.

I will try to make it to the online sync-up so that we can discuss things further as well as async here!