how to auto-report LLVM bugs found by fuzzing?

Hi,

We have several llvm fuzz targets running on OSS-Fuzz, a continuous automated fuzzing service:
https://github.com/google/oss-fuzz
https://www.usenix.org/sites/default/files/conference/protected-files/usenixsecurity17_slides_serebryany.pdf

It has reported a few bugs in cxa_demangler, clang, and dwarfdump already,
and we expect to add more fuzz targets to it soon (llvm-isel-fuzzer, clang-format-fuzzer, …)

A question to everyone: how do we report these bugs properly?
OSS-Fuzz files bugs automatically into a separate bug tracker, it can not file bugs to bugzilla.

By default, the bug reports are private for security reasons, and only those CC-ed explicitly can see them.

Should we make the bug reports public by default?
We can set things differently for the llvm project (llvm, clang, etc) and libcxxabi (demangler):https://github.com/google/oss-fuzz/tree/master/projects/llvm
https://github.com/google/oss-fuzz/tree/master/projects/llvm_libcxxabi

Should we automatically CC the bugs to any of the llvm maliing lists (e.g. llvm-dev)?

If a bug is CC-ed to a list, everyone will see the bug report summary in e-mail,
but if the bug remains private the reproducer for the bug will remain private.

Who wants to be CC-ed explicitly?
(please add yourself to https://github.com/google/oss-fuzz/blob/master/projects/llvm/project.yaml)

Examples of bug reports follow.

Thanks!

–kcc

dwarfdump:

https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=3216&q=label%3AProj-llvm%20dwarfdump-fuzzer&colspec=ID%20Type%20Component%20Status%20Proj%20Reported%20Owner%20Summary

Crash Type: ASSERT
Crash Address:
Crash State:
result <= UINT32_MAX
llvm::object::WasmObjectFile::parseStartSection
llvm::object::WasmObjectFile::parseSection

Crash Type: Heap-buffer-overflow READ 1
Crash Address: 0x60200000009a
Crash State:
llvm::object::WasmObjectFile::parseCustomSection
llvm::object::WasmObjectFile::parseSection
llvm::object::WasmObjectFile::WasmObjectFile

Crash Type: Heap-buffer-overflow READ 1
Crash Address: 0x604000000776
Crash State:
llvm::StringMapImpl::LookupBucketFor
std::pair<llvm::StringMapIterator, bool> llvm::StringMap<unsigned
llvm::DWARFContext::create

Crash Type: Heap-buffer-overflow READ 4
Crash Address: 0x60300000011c
Crash State:
llvm::identify_magic
llvm::object::ObjectFile::createObjectFile
_start

clang-fuzzer:
https://bugs.chromium.org/p/oss-fuzz/issues/list?can=1&q=label%3AProj-llvm+clang-fuzzer&colspec=ID+Type+Component+Status+Proj+Reported+Owner+Summary&cells=ids

Crash Type: Stack-buffer-overflow READ 1
Crash Address: 0x7f79e7b71760
Crash State:
clang::Lexer::SkipLineComment
clang::Lexer::LexTokenInternal
clang::Lexer::Lex

Crash Type: Direct-leak
Crash Address:
Crash State:
clang::Parser::ParseParameterDeclarationClause
clang::Parser::ParseFunctionDeclarator
clang::Parser::ParseDirectDeclarator

Crash Type: Stack-overflow
Crash Address: 0x7ffc78d69f48
Crash State:
clang::StmtVisitorBase<clang::make_const_ptr, IntExprEvaluator, bool>::Visit
Evaluate
IntExprEvaluator::VisitBinaryOperator

Crash Type: ASSERT
Crash Address:
Crash State:
!Prev.isAmbiguous() && “Cannot have an ambiguity in previous-declaration lookup”
DiagnoseInvalidRedeclaration
clang::Sema::ActOnFunctionDeclarator

cxa_demangler:

https://bugs.chromium.org/p/oss-fuzz/issues/list?can=1&q=label%3AProj-llvm_libcxxabi&colspec=ID+Type+Component+Status+Proj+Reported+Owner+Summary&cells=ids

Crash Type: Heap-buffer-overflow READ 8
Crash Address: 0x619000000078
Crash State:
__cxxabiv1::parse_encoding
__cxxabiv1::demangle
__cxa_demangle

Kostya Serebryany <kcc@google.com> writes:

Hi,

We have several llvm fuzz targets running on OSS-Fuzz, a continuous
automated fuzzing service:
https://github.com/google/oss-fuzz
https://www.usenix.org/sites/default/files/conference/protected-files/usenixsecurity17_slides_serebryany.pdf

It has reported a few bugs in cxa_demangler, clang, and dwarfdump already,
and we expect to add more fuzz targets to it soon (llvm-isel-fuzzer,
clang-format-fuzzer, ...)

A question to everyone: how do we report these bugs properly?
OSS-Fuzz files bugs automatically into a separate bug tracker, it can not
file bugs to bugzilla.
By default, the bug reports are private for security reasons, and only
those CC-ed explicitly can see them.

Should we make the bug reports public by default?
We can set things differently for the llvm project (llvm, clang, etc) and
libcxxabi (demangler):
https://github.com/google/oss-fuzz/tree/master/projects/llvm
https://github.com/google/oss-fuzz/tree/master/projects/llvm_libcxxabi

At least some of these should probably just be public by default. Things
like llvm-isel-fuzzer or clang-fuzzer aren't really looking for security
bugs, so I wouldn't expect them to find stuff that falls under the
responsible disclosure umbrella.

This should be thought about on a case by case basis, of course.

Should we automatically CC the bugs to any of the llvm maliing lists (e.g.
llvm-dev)?

Perhaps we could CC them to llvm-bugs? That's the same list that new
bugzilla bugs are announced to.

If a bug is CC-ed to a list, everyone will see the bug report summary in
e-mail,
but if the bug remains private the reproducer for the bug will remain
private.

Who wants to be CC-ed explicitly?
(please add yourself to
https://github.com/google/oss-fuzz/blob/master/projects/llvm/project.yaml)

Can this be set up to CC per-fuzz-target or so? I'm sure some people are
interested in, say, clang, but not necessarily cxa_demangler, or
vice-versa.

Kostya Serebryany <kcc@google.com> writes:
> Hi,
>
> We have several llvm fuzz targets running on OSS-Fuzz, a continuous
> automated fuzzing service:
> https://github.com/google/oss-fuzz
> https://www.usenix.org/sites/default/files/conference/protected-files/
usenixsecurity17_slides_serebryany.pdf
>
> It has reported a few bugs in cxa_demangler, clang, and dwarfdump
already,
> and we expect to add more fuzz targets to it soon (llvm-isel-fuzzer,
> clang-format-fuzzer, ...)
>
> A question to everyone: how do we report these bugs properly?
> OSS-Fuzz files bugs automatically into a separate bug tracker, it can not
> file bugs to bugzilla.
> By default, the bug reports are private for security reasons, and only
> those CC-ed explicitly can see them.
>
> Should we make the bug reports public by default?
> We can set things differently for the llvm project (llvm, clang, etc)
and
> libcxxabi (demangler):
> https://github.com/google/oss-fuzz/tree/master/projects/llvm
> https://github.com/google/oss-fuzz/tree/master/projects/llvm_libcxxabi

At least some of these should probably just be public by default. Things
like llvm-isel-fuzzer or clang-fuzzer aren't really looking for security
bugs, so I wouldn't expect them to find stuff that falls under the
responsible disclosure umbrella.

So, how about making all LLVM bugs public by default and leaving
cxa_demangler bugs private?
(I can't make it finer-grained, see below)

This should be thought about on a case by case basis, of course.

> Should we automatically CC the bugs to any of the llvm maliing lists
(e.g.
> llvm-dev)?

Perhaps we could CC them to llvm-bugs? That's the same list that new
bugzilla bugs are announced to.

Ah, good idea.
Unless someone objects I'll add llvm-bugs to the spam^W list :slight_smile:

> If a bug is CC-ed to a list, everyone will see the bug report summary in
> e-mail,
> but if the bug remains private the reproducer for the bug will remain
> private.
>
> Who wants to be CC-ed explicitly?
> (please add yourself to
> https://github.com/google/oss-fuzz/blob/master/projects/
llvm/project.yaml)

Can this be set up to CC per-fuzz-target or so? I'm sure some people are
interested in, say, clang, but not necessarily cxa_demangler, or
vice-versa.

Sadly, no.

We can distinguish llvm_cxxabi (cxa_demangler) from everything else because
these are currently two independent projects on oss-fuzz.
Making it finer-grained would require setting up separate oss-fuzz projects
which is harder to maintain and would not be welcome on oss-fuzz side.
The automatic e-mails announce the fuzz target's name, so filters will be
easy to set up.

Please don't, llvm-bugs is relatively low-traffic today. There will be a
mountain of fuzzer bugs. Right now llvm-bugs is only issues that actual
humans took the time to file. I volunteer my time to help humans with real
issues, but I don't necessarily want to triage every fuzzer bug ever.

Actually, maybe go ahead and do it. I'll just filter them out of my inbox
like everyone else does.

Ok, we have an objection. Not adding llvm-bugs. :frowning:
Other suggestions?

Kostya Serebryany <kcc@google.com> writes:

Kostya Serebryany <kcc@google.com> writes:
> Hi,
>
> We have several llvm fuzz targets running on OSS-Fuzz, a continuous
> automated fuzzing service:
> https://github.com/google/oss-fuzz
> https://www.usenix.org/sites/default/files/conference/protected-files/
usenixsecurity17_slides_serebryany.pdf
>
> It has reported a few bugs in cxa_demangler, clang, and dwarfdump
already,
> and we expect to add more fuzz targets to it soon (llvm-isel-fuzzer,
> clang-format-fuzzer, ...)
>
> A question to everyone: how do we report these bugs properly?
> OSS-Fuzz files bugs automatically into a separate bug tracker, it can not
> file bugs to bugzilla.
> By default, the bug reports are private for security reasons, and only
> those CC-ed explicitly can see them.
>
> Should we make the bug reports public by default?
> We can set things differently for the llvm project (llvm, clang, etc)
and
> libcxxabi (demangler):
> https://github.com/google/oss-fuzz/tree/master/projects/llvm
> https://github.com/google/oss-fuzz/tree/master/projects/llvm_libcxxabi

At least some of these should probably just be public by default. Things
like llvm-isel-fuzzer or clang-fuzzer aren't really looking for security
bugs, so I wouldn't expect them to find stuff that falls under the
responsible disclosure umbrella.

So, how about making all LLVM bugs public by default and leaving
cxa_demangler bugs private?
(I can't make it finer-grained, see below)

This sounds good to me.

This should be thought about on a case by case basis, of course.

> Should we automatically CC the bugs to any of the llvm maliing lists
(e.g.
> llvm-dev)?

Perhaps we could CC them to llvm-bugs? That's the same list that new
bugzilla bugs are announced to.

Ah, good idea.
Unless someone objects I'll add llvm-bugs to the spam^W list :slight_smile:

> If a bug is CC-ed to a list, everyone will see the bug report summary in
> e-mail,
> but if the bug remains private the reproducer for the bug will remain
> private.
>
> Who wants to be CC-ed explicitly?
> (please add yourself to
> https://github.com/google/oss-fuzz/blob/master/projects/
llvm/project.yaml)

Can this be set up to CC per-fuzz-target or so? I'm sure some people are
interested in, say, clang, but not necessarily cxa_demangler, or
vice-versa.

Sadly, no.

We can distinguish llvm_cxxabi (cxa_demangler) from everything else because
these are currently two independent projects on oss-fuzz.
Making it finer-grained would require setting up separate oss-fuzz projects
which is harder to maintain and would not be welcome on oss-fuzz side.
The automatic e-mails announce the fuzz target's name, so filters will be
easy to set up.

Fair enough.

Bugs found by oss-fuzz in llvm are now public:
https://bugs.chromium.org/p/oss-fuzz/issues/list?q=proj-llvm
(and the new ones will be public too).
I’ve also added llvm-bugs@lists.llvm.org to the list of e-mail recipients:
https://github.com/google/oss-fuzz/blob/master/projects/llvm/project.yaml

Bugs found by oss-fuzz in llvm are now public:
Monorail - oss-fuzz - OSS-Fuzz: Fuzzing the planet - Monorail
(and the new ones will be public too).
I've also added llvm-bugs@lists.llvm.org to the list of e-mail recipients:
https://github.com/google/oss-fuzz/blob/master/projects/llvm/project.yaml

.. and now mailing to llvm-bugs actually works:
http://lists.llvm.org/pipermail/llvm-bugs/2017-September/058151.html
http://lists.llvm.org/pipermail/llvm-bugs/2017-September/058152.html

Is it possible to put different tags in the subject line for the LLVM / Clang / clang-format fuzz targets, so that mail filters can identify the ones of interest?

Is it possible to put different tags in the subject line for the LLVM /
Clang / clang-format fuzz targets, so that mail filters can identify the
ones of interest?

In subject -- probably not easy (at least not desirable -- some target
names are pretty long).
Can you filter by the message contents? ("Fuzz target binary: clang-fuzzer")

Or maybe we want to change the subjects indeed…
Filed https://github.com/google/oss-fuzz/issues/883 for discussion.