Meeting minutes for the 16th of April 2024 meeting
A proposal to make LLVM a CNA (CVE Numbering Authority). @tstellar has proposed steps to make LLVM a CVE Numbering Authority. It seems it’s a relatively simple process by filling in this form.
A main motivation for this is to be able to dispute CVEs raised by others against LLVM that are clearly outside of LLVM’s threat model.
Would becoming a CNA set the expectation that the security group would need to start raising CVEs for the issues it accepts as security issues? There’s an observation that on quite a few toolchain-related security issues, there is no general consensus on whether CVEs should be created for them.
Do we need to learn something from the xz supply chain attack and take some action?
Is it OK for there being so many binary files in LLVM’s source repository?
Having these binary files results in LLVM getting a lower OpenSSF score. see OpenSSF scorecard report, under Binary-Artefacts, to see all binary files in the repository.
It seems most, if not all, binary files are tests. It seems we could have fewer binary files (e.g. by making more use of yaml2obj), but we probably cannot remove all binary files.
Would it be helpful from a supply chain security point-of-view to only allow some trusted users to touch these files?
It’s currently not possible to implement this restriction.
Another possibly useful restriction is to only allow binary files in test directories. If then our test/build/package workflows would not include test directories, the risk of supply chain attack making use of binary files might be reduced.
As a first easy step, it probably would be helpful to explicitly state in the developer policy that binary files are only allowed in test directories.
We currently cannot create an audit trail of who pushed code into the repository. The reason is that we allow direct push access, without going through a pull request.
The only information on who pushed a particular commit for direct pushes (i.e. not using a pull request) is the committer email address in the git commit. That information can very easily be changed by an attacker. No information gets stored on which github account pushed the commit.
If we required pull requests for all commits, it tracks which github accounts merged the pull request.
We discussed a bit about why people like doing direct pushes. It seems there are 2 reasons:
being able to quickly revert patches when they break bots.
We thought it wouldn’t be too hard to create a script that creates a pull request and merges it for you using the github cli.
Furthermore, if all commits are guaranteed to go through pull requests, it’s also possible to use the button on the github web interface to easily do a revert. In other words, it may mean the reason for direct pushes for reverts mostly goes away.
The real or perceived overhead of just directly pushing in small changes.
We thought it wouldn’t be too hard to create a script that creates a pull request and merges it for you using the github cli.
We thought that these direct pushes are also problematic because there is no track record of who (if anyone) reviewed the changes, either pre- or post-commit.
Moving away from the chromium issue tracker to a github-based workflow.
@kbeyls points to a recent blog post which highlighted that the github-based mechanism to report security issues privately does not allow to pull in people not on the security group into the ongoing discussions on these issues. The LLVM security group often has to do so.
Overall, it seems we have two plausible options: go for an email-based workflow (which is what most other toolchain-like projects such as Rust, Swift, Gcc, Go, etc do), or use the github-based flow.
A draw-back of the email-based flow is that it does not automatically create a unique id for each reported issue. It would probably also require a bit more work to create transparency reports.
We checked the status of the open security issues and took a few actions to progress them.
Meeting minutes for the 16th of April 2024 meeting- A proposal to make LLVM a CNA (CVE Numbering Authority).
@tstellar has proposed steps to make LLVM a CVE Numbering Authority. It seems it’s a relatively simple process by filling in this form.
A main motivation for this is to be able to dispute CVEs raised by others against LLVM that are clearly outside of LLVM’s threat model.
Would becoming a CNA set the expectation that the security group would need to start raising CVEs for the issues it accepts as security issues? There’s an observation that on quite a few toolchain-related security issues, there is no general consensus on whether CVEs should be created for them.
Do we need to learn something from the xz supply chain attack and take some action?
Is it OK for there being so many binary files in LLVM’s source repository?
Having these binary files results in LLVM getting a lower OpenSSF score. see OpenSSF scorecard report, under Binary-Artefacts, to see all binary files in the repository.
It seems most, if not all, binary files are tests. It seems we could have fewer binary files (e.g. by making more use of yaml2obj), but we probably cannot remove all binary files.
It’s worth mentioning that the yaml side of yaml2obj is still partially binary because the section data is represented as hex strings. Sometimes it’s just zeros or something like 0xDEADC0DE but some are machine code.
Would it be helpful from a supply chain security point-of-view to only allow some trusted users to touch these files?
It’s currently not possible to implement this restriction.
Another possibly useful restriction is to only allow binary files in test directories. If then our test/build/package workflows would not include test directories, the risk of supply chain attack making use of binary files might be reduced.
As a first easy step, it probably would be helpful to explicitly state in the developer policy that binary files are only allowed in test directories.
We currently cannot create an audit trail of who pushed code into the repository. The reason is that we allow direct push access, without going through a pull request.
The only information on who pushed a particular commit for direct pushes (i.e. not using a pull request) is the committer email address in the git commit. That information can very easily be changed by an attacker. No information gets stored on which github account pushed the commit.
If we required pull requests for all commits, it tracks which github accounts merged the pull request.
We discussed a bit about why people like doing direct pushes. It seems there are 2 reasons:
being able to quickly revert patches when they break bots.
- We thought it wouldn’t be too hard to create a script that creates a pull request and merges it for you using the github cli.
- Furthermore, if all commits are guaranteed to go through pull requests, it’s also possible to use the button on the github web interface to easily do a revert. In other words, it may mean the reason for direct pushes for reverts mostly goes away.
The real or perceived overhead of just directly pushing in small changes.
- We thought it wouldn’t be too hard to create a script that creates a pull request and merges it for you using the github cli.
- We thought that these direct pushes are also problematic because there is no track record of who (if anyone) reviewed the changes, either pre- or post-commit.
Moving away from the chromium issue tracker to a github-based workflow.
@kbeyls points to a recent blog post which highlighted that the github-based mechanism to report security issues privately does not allow to pull in people not on the security group into the ongoing discussions on these issues. The LLVM security group often has to do so.
Overall, it seems we have two plausible options: go for an email-based workflow (which is what most other toolchain-like projects such as Rust, Swift, Gcc, Go, etc do), or use the github-based flow.
A draw-back of the email-based flow is that it does not automatically create a unique id for each reported issue. It would probably also require a bit more work to create transparency reports.
We checked the status of the open security issues and took a few actions to progress them.
This doesn’t sound right – GitHub advisories have a “collaborators” field where additional people can be added.
At least the last time I looked the problem was the reverse one: Only admins could see submitted security reports. There was no possibility to have a security team that can see them / is notified by default. Not sure whether there have been any changes to this in the meantime.
Thank you for pointing that out. Indeed after adding someone to the “collaborators” field, they can see the issue and add comments.
We are experimenting with a setup where a new “artificial” repository is created in the LLVM github organization, which is only used to report security issues. Then the people on the security group can be made admins of that repo, so that they will get notified about new security reports.
So, in summary, I don’t think there are any changes to what you describe.
I should expand on this a bit. If actual instructions are in there, it might be better to have the input be asm source rather than yaml. Lots of debug-info dumping tests are written that way, for example, because yaml doesn’t have a way to specify DWARF input in a readable way. The asm isn’t hugely readable either–we could really use a proper “DWARF assembler”–but at least you can annotate the hex to show what’s going on.
Moving away from the chromium issue tracker to a github-based workflow.
Kristof experimented with the github-based workflow and shared impressions with the LLVM security group.
After the outcome of these experiments, no-one objects to migrating from the chromium issue tracker to a github-based workflow. @kbeyls will work on implementing this migration, which presumably consists of:
@Tulio_Magno_Quites_M introduced himself as a candidate to join the LLVM security group.
Does anyone have experience with creating US government-style SBOMs for a compiler built using LLVM components?
A few people from different vendors on the call have gotten similar requests.
Different vendor toolchains built on LLVM probably include slightly different dependencies and/or parts of LLVM, and hence their SBOMs might be somewhat different.
It seemed no-one on the call knew exactly what to include and what not in an SBOM.
If enough vendors get these requests, there might be scope for sharing the cost of creating or maintaining SBOMs upstream somehow?
Maybe clang could grow a feature that helps with automatically building an SBOM as a side artefact while building a C/C++ project?
It seems this presentation from the 2022 developer meeting covers this topic:
We checked the status of the open security issues and took a few actions to progress them.
We should probably make sure that we archive the content of all the issues in the chromium issues tracker somehow, as it isn’t clear whether issues will be retained there forever. Issue #96039 was raised to cover this action.
All security issues and published advisories for all of the github repositories under the github LLVM organization will be filed under the llvm-security-repo repository. Currently, at least some of the other repositories, including the mono-repo, show an “advisories” tab under the “security” tab. This might be confusing. Could we do something to make it very clear to all visitors of the project exactly where to look for security advisories? Issue #96050 was raised to explore what is possible.
We checked the status of the open security issues and took a few actions to progress them.
We have switched over to to using github rather than chromium.org for reporting security issues, and we’re trying to figure out some of the details on how to process security issues in this new setting. A specific issue we’re running into is how to best close out a privately reported issue for which we ended up concluding that we do not consider it to be a security issue. Just closing the ticket will not make its content public; there is not straightforward way to make the discussion on the ticket public. On the call we thought of the following options:
It seems it is not possible to simply convert a security-reported issue to a regular issue (that would be publicly visible).
Option 1 is to manually recreate a public issue with all the information and comments that were made on the private issue. (This would be quite a bit of work for each issue we close out in this state).
Option 2 is to create a public issue with the same title as the security issue, and to attach a PDF (which should easily be created using a “print-to-pdf” method in the browser) containing all comments. The downside is that the details of the discussion wouldn’t be searchable. The upside is that this is a small amount of work, so feasible to do for every issue we close out. Furthermore having all the discussion available in at least some form publicly helps with transparency.
Option 3 is to not create a public issue at all, but just record a summary of a few lines in the transparency report.
Option 4 is to invite the whole LLVM github organization as a “collaborator” on the issue, so that everyone in the LLVM github organization would get visibility of the issue (but not anyone else). Presumably this would result in everyone in the LLVM organization to get notified every time we do this.
In the call, the consensus reached was to try option 2 (create public issue with attached PDF with all the details). We still need to propose/have agreement on how we’d mark such “issues reported as security issues, but ultimately getting rejected as ‘not a security issue’” in the regular LLVM issue tracker. Options include a specific label, a marker in the title of the issue, or some other solution.
We checked the status of the open security issues and took a few actions to progress them.
In the github “security advisory” workflow we switched to, there is the ability to create a temporary private fork of a repository to develop a patch for a security issue privately. However, we use a separate repository (GitHub - llvm/llvm-security-repo: Repo to test filing security issues) to track security issues across all repositories in the LLVM organization. When a private fork is created, it is therefore a private fork of this (almost empty) repo, not of, for example, the main monorepo. We discussed a few alternatives on how we could best develop patches for security issues under embargo. We decided to stick with just putting patches under development in this empty repository, and do reviews that way. If we need a better solution at some point, we’ll explore options then.
Meeting minutes for the 17th of September 2024 meeting
We discussed how to best prepare the LLVM security group’s yearly transparency report after migrating from the Chromium issue tracker to github to track security issues. When using the Chromium issue tracker, the end state for all issues, independently of whether they were accepted as security issues or not, was for them to be public. Therefore, the transparency reports were relatively easy to create: simply point to the issues in the Chromium issue tracker.
The Github-based flow keeps all issues and their comments private. When an issue is considered to be a major enough security issue, a public advisory gets published. For all other issues, all information will remain non-public. Therefore, we’ll probably have to do more work to make meaningful transparency reports.
To increase the probability of actually creating meaningful transparency reports (by this volunteer-based group), @kbeyls proposes to have a living transparency document, which we aim to update with a relevant summary every time we close an issue.
There was consensus on the call to try out this approach. @kbeyls takes an action to set up such a live transparency document. @tulio pointed out that a tool like GitHub - psss/did: What did you do last week, month, year? could be useful to automatically create a transparency report, assuming support for github security issues was added to it.
There was agreement on the call to organize a round table at the upcoming dev meeting on the LLVM security group and related topics. @kbeyls takes an action to propose this round table.
We went through the list of open reported issues and @smithp35, @tulio, @whuhn, and @kbeyls took actions to progress the status of open issues.
