Random non-reproducible errors in ClangdTests

Hi,

For a while now we randomly see failures when running the ClangdTests test.

E.g the below but sometimes other failures:

One reason of non-reproducible errors in a program is mainly "uninitialized
variables" .
These variables uses whatever remains in the memory from previous
allocations and used by the new program .
Therefore when these values are used , program behavior is unexpected .

Another may be "array index showing outside of an array" . Especially in
loops , if loop passes array boundaries toward up or down .
Or , when a value is directly used to index an array , but it is out of
bounds .

The above reasons are not , "all" , but "some" .

Mehmet Erol Sanliturk

Hi Mikael,

Thanks for pointing that out! Could you share information about what compiler + system you used to build and run tests? Sometimes, UB is relatively easy to reproduce using the same setup.

As Mehmet mentioned, it looks that there might be uninitialised variable given the Valgrind log that you provided.

Could you please attach the whole log so that it would be easier for us to understand the cause of problem(s)?

+clangd-dev

Kind regards,
Kirill

Hi Mikael,

Thanks for pointing that out! Could you share information about what
compiler + system you used to build and run tests? Sometimes, UB is
relatively easy to reproduce using the same setup.

As Mehmet mentioned, it looks that there might be uninitialised variable
given the Valgrind log that you provided.

Could you please attach the whole log so that it would be easier for us to
understand the cause of problem(s)?

+clangd-dev

Kind regards,
Kirill

The following reasons may also be considered :

A memory chip or memory management chip or a line in the main board if
failing slowly .

If some values are read from disk and used , disk surface failures or cable
connection point oxidization may also cause arbitrary data corruptions .

Mehmet Erol Sanliturk

Hi,

    Hi,

    For a while now we randomly see failures when running the
    ClangdTests test.

    E.g the below but sometimes other failures:

    --------
    FAIL: Extra Tools Unit Tests ::
    clangd/./ClangdTests/WorkspaceSymbolsTest.Namespaces (16172 of 39758)
    ******************** TEST 'Extra Tools Unit Tests ::
    clangd/./ClangdTests/WorkspaceSymbolsTest.Namespaces' FAILED
    ********************
    Note: Google Test filter = WorkspaceSymbolsTest.Namespaces
    [==========] Running 1 test from 1 test case.
    [----------] Global test environment set-up.
    [----------] 1 test from WorkspaceSymbolsTest
    [ RUN ] WorkspaceSymbolsTest.Namespaces
    ../tools/clang/tools/extra/unittests/clangd/FindSymbolsTests.cpp:151: Failure
    Value of: getSymbols("a")
    Expected: has 4 elements and there exists some permutation of
    elements such that:
      - element #0 (named "ans1") and (in container ""), and
      - element #1 (named "ai1") and (in container "ans1"), and
      - element #2 (named "ans2") and (in container "ans1"), and
      - element #3 (named "ai2") and (in container "ans1::ans2")
      Actual: {}
    ../tools/clang/tools/extra/unittests/clangd/FindSymbolsTests.cpp:153: Failure
    Value of: getSymbols("::")
    Expected: has 1 element that (named "ans1") and (in container "")
      Actual: {}
    ../tools/clang/tools/extra/unittests/clangd/FindSymbolsTests.cpp:155: Failure
    Value of: getSymbols("::a")
    Expected: has 1 element that (named "ans1") and (in container "")
      Actual: {}
    ../tools/clang/tools/extra/unittests/clangd/FindSymbolsTests.cpp:158: Failure
    Value of: getSymbols("ans1::")
    Expected: has 2 elements and there exists some permutation of
    elements such that:
      - element #0 (named "ai1") and (in container "ans1"), and
      - element #1 (named "ans2") and (in container "ans1")
      Actual: {}
    ../tools/clang/tools/extra/unittests/clangd/FindSymbolsTests.cpp:160: Failure
    Value of: getSymbols("::ans1")
    Expected: has 1 element that (named "ans1") and (in container "")
      Actual: {}
    ../tools/clang/tools/extra/unittests/clangd/FindSymbolsTests.cpp:163: Failure
    Value of: getSymbols("::ans1::")
    Expected: has 2 elements and there exists some permutation of
    elements such that:
      - element #0 (named "ai1") and (in container "ans1"), and
      - element #1 (named "ans2") and (in container "ans1")
      Actual: {}
    ../tools/clang/tools/extra/unittests/clangd/FindSymbolsTests.cpp:165: Failure
    Value of: getSymbols("::ans1::ans2")
    Expected: has 1 element that (named "ans2") and (in container "ans1")
      Actual: {}
    ../tools/clang/tools/extra/unittests/clangd/FindSymbolsTests.cpp:167: Failure
    Value of: getSymbols("::ans1::ans2::")
    Expected: has 1 element that (named "ai2") and (in container
    "ans1::ans2")
      Actual: {}
    [ FAILED ] WorkspaceSymbolsTest.Namespaces (39 ms)
    [----------] 1 test from WorkspaceSymbolsTest (39 ms total)

    [----------] Global test environment tear-down
    [==========] 1 test from 1 test case ran. (40 ms total)
    [ PASSED ] 0 tests.
    [ FAILED ] 1 test, listed below:
    [ FAILED ] WorkspaceSymbolsTest.Namespaces

      1 FAILED TEST

    It can fail and then at once pass again so something funny is going on.

    I ran valgrind on ClangdTests and then I got a bunch of complaints,
    the first being:

    [ RUN ] ClangdVFSTest.SearchLibDir
    Updating file /clangd-test/foo.cpp with command [/clangd-test] clang
    -ffreestanding -xc++ -target x86_64-linux-unknown -m64
    --gcc-toolchain=/randomusr -stdlib=libstdc++ /clangd-test/foo.cpp
    -resource-dir=/data/repo/llvm-patch/build-all/tools/clang/tools/extra/unittests/clangd/../lib/clang/8.0.0
    Preamble for file /clangd-test/foo.cpp cannot be reused. Attempting
    to rebuild it.
    Built preamble of size 175020 for file /clangd-test/foo.cpp
    ==14605== Thread 2 worker:foo.cpp:
    ==14605== Conditional jump or move depends on uninitialised value(s)
    ==14605== at 0xCDF8A6:
    clang::Sema::SetCtorInitializers(clang::CXXConstructorDecl*, bool,
    llvm::ArrayRef<clang::CXXCtorInitializer*>) (in
    /data/repo/llvm-patch/build-all/tools/clang/tools/extra/unittests/clangd/ClangdTests)
    ==14605== by 0xD01CB9:
    clang::Sema::DefineImplicitDefaultConstructor(clang::SourceLocation,
    clang::CXXConstructorDecl*) (in
    /data/repo/llvm-patch/build-all/tools/clang/tools/extra/unittests/clangd/ClangdTests)
    ==14605== by 0xEA3B93:
    PerformConstructorInitialization(clang::Sema&,
    clang::InitializedEntity const&, clang::InitializationKind const&,
    llvm::MutableArrayRef<clang::Expr*>,
    clang::InitializationSequence::Step const&, bool&, bool, bool,
    clang::SourceLocation, clang::SourceLocation) (in
    /data/repo/llvm-patch/build-all/tools/clang/tools/extra/unittests/clangd/ClangdTests)
    ==14605== by 0xE9AA3F:
    clang::InitializationSequence::Perform(clang::Sema&,
    clang::InitializedEntity const&, clang::InitializationKind const&,
    llvm::MutableArrayRef<clang::Expr*>, clang::QualType*) (in
    /data/repo/llvm-patch/build-all/tools/clang/tools/extra/unittests/clangd/ClangdTests)
    ==14605== by 0xBFA3C4:
    clang::Sema::ActOnUninitializedDecl(clang::Decl*) (in
    /data/repo/llvm-patch/build-all/tools/clang/tools/extra/unittests/clangd/ClangdTests)
    ==14605== by 0x13C5EFF:
    clang::Parser::ParseDeclarationAfterDeclaratorAndAttributes(clang::Declarator&,
    clang::Parser::ParsedTemplateInfo const&,
    clang::Parser::ForRangeInit*) (in
    /data/repo/llvm-patch/build-all/tools/clang/tools/extra/unittests/clangd/ClangdTests)
    ==14605== by 0x13C40B7:
    clang::Parser::ParseDeclGroup(clang::ParsingDeclSpec&,
    clang::DeclaratorContext, clang::SourceLocation*,
    clang::Parser::ForRangeInit*) (in
    /data/repo/llvm-patch/build-all/tools/clang/tools/extra/unittests/clangd/ClangdTests)
    ==14605== by 0x13AAF5B:
    clang::Parser::ParseDeclOrFunctionDefInternal(clang::Parser::ParsedAttributesWithRange&,
    clang::ParsingDeclSpec&, clang::AccessSpecifier) (in
    /data/repo/llvm-patch/build-all/tools/clang/tools/extra/unittests/clangd/ClangdTests)
    ==14605== by 0x13AAA1D:
    clang::Parser::ParseDeclarationOrFunctionDefinition(clang::Parser::ParsedAttributesWithRange&,
    clang::ParsingDeclSpec*, clang::AccessSpecifier) (in
    /data/repo/llvm-patch/build-all/tools/clang/tools/extra/unittests/clangd/ClangdTests)
    ==14605== by 0x13A9861:
    clang::Parser::ParseExternalDeclaration(clang::Parser::ParsedAttributesWithRange&,
    clang::ParsingDeclSpec*) (in
    /data/repo/llvm-patch/build-all/tools/clang/tools/extra/unittests/clangd/ClangdTests)
    ==14605== by 0x13A8629:
    clang::Parser::ParseTopLevelDecl(clang::OpaquePtr<clang::DeclGroupRef>&)
    (in
    /data/repo/llvm-patch/build-all/tools/clang/tools/extra/unittests/clangd/ClangdTests)
    ==14605== by 0x13A840F:
    clang::Parser::ParseFirstTopLevelDecl(clang::OpaquePtr<clang::DeclGroupRef>&)
    (in
    /data/repo/llvm-patch/build-all/tools/clang/tools/extra/unittests/clangd/ClangdTests)
    ==14605==
    Updating file /clangd-test/foo.cpp with command [/clangd-test] clang
    -ffreestanding -xc++ -target x86_64-linux-unknown -m64
    --gcc-toolchain=/randomusr -stdlib=libstdc++ /clangd-test/foo.cpp
    -resource-dir=/data/repo/llvm-patch/build-all/tools/clang/tools/extra/unittests/clangd/../lib/clang/8.0.0
    Reusing preamble for file /clangd-test/foo.cpp
    [ OK ] ClangdVFSTest.SearchLibDir (952 ms)

    It's not obvious to me what the problem really is. Does anyone here
    have a clue?

    Regards,
    Mikael
    _______________________________________________

One reason of non-reproducible errors in a program is mainly "uninitialized variables" .
These variables uses whatever remains in the memory from previous allocations and used by the new program .
Therefore when these values are used , program behavior is unexpected .

Another may be "array index showing outside of an array" . Especially in loops , if loop passes array boundaries toward up or down .
Or , when a value is directly used to index an array , but it is out of bounds .

The above reasons are not , "all" , but "some" .

Sure, that's why I ran valgrind to get some help, and it does complain a number of times, but even so it's not obvious to me (who doesn't know the code) what the problem really is.

Since this happens when I compile on trunk I thought someone else also have seen this and that someone who knows Sema could possibly make more out of the valgrind info.

/Mikael

Hi,

Hi Mikael,

Thanks for pointing that out! Could you share information about what compiler + system you used to build and run tests?

I've seen the failures when compiling with clang 3.6.0 on RHEL 6.9 as well as gcc 5.4.0 on SLED 11.2.

The valgrind run was made on a binary compiled with clang 3.6.0 on Ubuntu 14.04.

Sometimes, UB is relatively easy to reproduce using the same setup.

As Mehmet mentioned, it looks that there might be uninitialised variable given the Valgrind log that you provided.

Could you please attach the whole log so that it would be easier for us to understand the cause of problem(s)?

Sure!

Thanks,
Mikael

ClangdTests.log.gz (18.5 KB)

Thanks for the information!

I was able to reproduce many failures in tests using TSan and MSan. I’m attaching logs for TSan for this particular test that was originally mentioned (WorkspaceSymbolsTest.Namespaces): it looks like a data race in Trace and TUScheduler. The pattern looks to be the same for all other tests.

+CC Ilya who is knowledgeable about TUScheduler.

I will attach stdout logs of the test failure and whole TSan output.

Kind regards,
Kirill

thread-sanitizer.logs

stdout-clangd-thread-sanitizer.logs (1.31 KB)

Thanks for the logs. Will take a look.

There was a data race in another test, fixed in r341538.
Might be causing these failures, but I couldn’t reproduce the original ones, so can’t be sure.

The TSAN logs were produced with non-sanitized standard library, I couldn’t reproduce the failures with a sanitized build of libc++.

Please check if r341538 fixes the error, there is a good chance it was the root cause.
If it doesn’t help, please let us know, we’ll investigate further.

+cfe-dev

Hi,

Thanks! (I've just joined clangd-dev but haven't been approved yet so I didn't see the mails there)

We'll pull in your fix and hopefully the problems will be gone then. I'll get back if not.

Thanks,
Mikael