Running clang tests

Hi Team,
I have checked out the clang and llvm source code and built the executables using the visual studio 2015 community edition. I am using Windows as my platform.

However I see that there are some test cases under the clang test folder in the LLVM.sln. Eg AstMatcherTest,ASTTests etc. I see that these tests make use of the Google test framework.

In my visual studio I have installed Google Test Adapter extension for running the Google tests.

But the tests under AstMatcherTest,ASTTests projects won’t show up in the Test Explorer view in visual studio even after building these projects.So I wanted to know the reason for the same.

So can you please let me know if it possible to run these tests in visual studio. If not how to run them.

Thanks in advance.

Thanks and Regards
Sudhindra Kulkarni

Hi,

LLVM requires Visual Studio 2017 or 2019: https://llvm.org/docs/GettingStartedVS.html#software – you could use Community 2019.

I don’t recommend building LLVM inside Visual Studio. However you can generate or update the VS solution once in a while:

  • mkdir buildVS && cd buildVS
  • cmake {your_llvm_root}/llvm -G"Visual Studio 16 2019" -Thost=x64 -DLLVM_ENABLE_PROJECTS=clang;llvm -DLLVM_OPTIMIZED_TABLEGEN=ON

Use Ninja instead for building: https://ninja-build.org/ it’s a lot faster - keep Visual Studio only for debugging.

Ensure ninja.exe is in %PATH%.

From a Visual Studio shell (“x64 Native Tools Command Prompt for VS 2019” in the Start menu) run the following:

  • mkdir buildNinjaRel && cd buildNinjaRel
  • cmake {your_llvm_root}/llvm -G"Ninja" -DLLVM_ENABLE_PROJECTS=clang;llvm -DCMAKE_BUILD_TYPE=RelWithDebInfo -DLLVM_OPTIMIZED_TABLEGEN=ON
  • ninja check-all

You can then debug the test in VS: {your_llvm_root}\buildNinjaRel\tools\clang\unittests\ASTMatchers\ASTMatchersTests.exe

Hi,

I’d just like to interject to say that building within Visual Studio isn’t really that bad. Running the lit tests is a bit painful because the LLVM build tools that are integrated with the build system don’t play nice with msbuild. Particularly, I’ve never been able to actually cancel an invocation of lit or tablegen via visual studio. That said, there is a huge upside to building with Visual Studio: actually being able to use the debugger.

Honestly, I find this “just use ninja” attitude frustrating and troubling. If there are pain points with using MSVC or make or whatever, we should be trying to fix then, not just throwing our hands up in the air giving up. In my opinion, most of the issues surrounding using make or msbuild stem from the fact that nobody is using these build systems, so nobody is fixing the issues. I feel that we should be encouraging people to use the build system that they prefer, and encouraging people to fix the build system to work well with their chosen build system. This is the only way that things will get better.

Sudhindra, to answer your question, I would also recommend upgrading to Visual Studio 2019 if you can. I just tried to directly run ASTMatcherTests from Visual Studio 2019 and it worked for me. I have no experience with the GTest IDE plugin, I’ve always just ran GTest binaries directly. There’s also a check-clang (I think that’s the one you want. It might be called something else but it’s definitely there) target in there that is equivalent to the Ninja check-clang target. These check targets are weird because to run them you have to do “build” from Visual Studio. If you try to actually run the target it won’t work. This is one of those things that would probably get fixed if people actually used the LLVM Visual Studio project.

Thanks,

Christopher Tetreault

Chris,

We are fixing the issues :slight_smile: clang-cl doesn’t currently support /MP which makes building of each project single-threaded with MSBuild. I have a patch ongoing here: https://reviews.llvm.org/D52193 – I am working on it since last week to iron out the last issues, I’ll update it soon.

Without that you have to resort to increasing the “maximum number of parallel builds” to a higher value than the default 2.

Interesting. I’ve always heard that “Ninja is faster” but never seen actual numbers before, so I appreciate you taking the time to post them. That said, ~10 extra minutes with cl.exe/msbuild for a full rebuild is a price that I personally am willing to pay to be able to use the visual studio debugger.

Hopefully you get /MP support for clang-cl soon. Visual Studio is my preferred development environment for C++, so anything that makes that situation better with open source tools is a win in my book.

It is technically possibly to integrate ninja into Visual Studio, through the NMake bindings. We’re doing it with Fastbuild:

cd $(SolutionDir)…\Code\ & …\Bin\Windows-x64\fbuild -vs -dist -monitor -fastcancel -cache $(ProjectName)-$(Configuration)

cd $(SolutionDir)…\Code\ & …\Bin\Windows-x64\fbuild -vs -dist -monitor -fastcancel -cache -clean $(ProjectName)-$(Configuration)

WIN32_LEAN_AND_MEAN;WIN32;_WIN32;WINDOWS;WIN64;DEBUG;PROFILING_ENABLED;

…\Code;…\External\SDK\VisualStudio\2017\Professional\VC\Tools\MSVC\14.16.27023\include;…\External\SDK\Windows\10\Include\10.0.17763.0\ucrt;…\External\SDK\Windows\10\Include\10.0.17763.0\um;…\External\SDK\Windows\10\Include\10.0.17763.0\shared;

$(SolutionDir)\Temp

$(SolutionDir)\Temp

In this case, we generate both .vcxprojs with NMake properties, and Fastbuild .bff files (which are the equivalent to .vcxprojs). NMake invokes Fastbuild instead of using MSBuild.

I am not aware though how/if cmake can do that, I don’t think it supports this kind of scenario. You’d have to generate once the .sln/.vcxprojs and then invoke cmake again with -GNinja, and hope that the former properly invokes the latter. It is possible, but someone would have to make that work.

I find it easier to build and use them separately. I keep Visual Studio open to browse and edit LLVM files/projects, and use cmd-line ninja in various configurations. You can actually use the VS debugger, however you need to manually change the paths to use the ninja-built EXEs (on the “Debugging” property page).

Sure, visual studio’s debugger can connect to external processes or binaries. But at some point, the “it just works out of the box” factor of the CMake generated msbuild project starts to outweigh any speed gains from Ninja. Usually, you only do a full build once, so the speedups associated with Ninja are much less when you’re only compiling ~10 files. I also know that Visual Studio has built in CMake support that uses ninja builds by default. In my experience, it’s not very good support, and it requires hacking the LLVM build system to actually work for subprojects other than llvm.

I certainly don’t expect people to go out of their way to support these minority build systems, but they do work, and more eyes on them will ensure that they work better. Make is also fine, I use it on my linux machine because I find it easier to deal with.

(Long thread, didn’t read it all, apologies for missing context…)

I wanted to mention that this is possible, and to add that I think the gn build also has support for this: to make VS project files that shell out to ninja for the real build work, but still have a project.

If there are pain points with using MSVC or make or whatever, we should be trying to fix then, not just throwing our hands up in the air giving up.

The biggest “pain point” for those of us who have made a good-faith effort to go down the MSVC route, has been in how MSVC and/or MSBuild checks and handles dependencies. In practice, during a build, the Visual Studio stack tends to spend a lot of time rebuilding things, that do not necessarily require it. In practice, I find that iteration using Visual Studio is an order of magnitude slower than iterating on a Ninja-based system.

If you need to support Microsoft-based technologies, then Visual Studio Code treats LLVM as a first-class customer on Windows. In particular, the clangd plugin within VSCode, provides C++ highlighting and refactoring on par with MSVC. That, plus the CMake plugin, makes VSCode an entirely viable alternative to MSVC, supporting all its key features and adding a few, with quicker iteration time.

The VSCode CMake plugin allows you to switch between VC and LLVM compilers trivially, for compatibility testing or whatever, in VSCode.

After spending too much time, trying to do LLVM development in Visual Studio, I switched to VSCode and never looked back. Not a paid shill by any means. Just a happier user after making the switch.