Running tests on windows?

Does anybody run tests on Windows? I was able to build lldb on windows using cmake + ninja by following the instructions at http://lldb.llvm.org/build.html#BuildingLldbOnWindows

Running nina check-lldb however fails:

[1/1] cmd.exe /C “cd /D C:\Users\chuckr\source\build_llvm_debug\tools\lldb\test && C:\Users\chuckr\source\Python-2.7.8\PCbuild\python.exe c:/Users/chuckr/source/llvm/tools/lldb/test/dosep.py -o “-q --arch=i686 --executable C:/Users/chuckr/source/build_llvm_debug/bin/lldb.exe -s C:/Users/chuckr/source/build_llvm_debug/lldb-test-traces -u CXXFLAGS -u CFLAGS -C C:/Program Files (x86)/Microsoft Visual Studio 12.0/VC/bin/cl.exe””

FAILED: cmd.exe /C “cd /D C:\Users\chuckr\source\build_llvm_debug\tools\lldb\test && C:\Users\chuckr\source\Python-2.7.8\PCbuild\python.exe c:/Users/chuckr/source/llvm/tools/lldb/test/dosep.py -o “-q --arch=i686 --executable C:/Users/chuckr/source/build_llvm_debug/bin/lldb.exe -s C:/Users/chuckr/source/build_llvm_debug/lldb-test-traces -u CXXFLAGS -u CFLAGS -C C:/Program Files (x86)/Microsoft Visual Studio 12.0/VC/bin/cl.exe””

usage: dotest.py [-h] [-A arch] [-C compiler] [-E extra-flags] [-D]

[-N {dwarf,dsym}] [-a] [+a] [+b] [-b blacklist]

[-f filterspec] [-g] [-l] [-m] [+m] [-p pattern]

[-X directory] [-G category] [–skip-category category]

[-c config-file] [–framework framework-path]

[–executable executable-path] [–libcxx directory]

[-e benchmark-exe] [-k command] [-R dir] [-r dir] [-s name]

[-x breakpoint-spec] [-y count] [-# sharp]

[–platform-name platform-name] [–platform-url platform-url]

[–platform-working-dir platform-working-dir] [-d] [-F] [-i]

[-n] [-P] [-q] [-S] [-t] [-u variable] [–env variable] [-v]

[-w] [-T] [–enable-crash-dialog] [–show-inferior-console]

[test-dir [test-dir …]]

dotest.py: error: unrecognized arguments: c:\Users\chuckr\source\llvm\tools\lldb\test\api\check_public_api_headers

usage: dotest.py [-h] [-A arch] [-C compiler] [-E extra-flags] [-D]

[-N {dwarf,dsym}] [-a] [+a] [+b] [-b blacklist]

[-f filterspec] [-g] [-l] [-m] [+m] [-p pattern]

[-X directory] [-G category] [–skip-category category]

[-c config-file] [–framework framework-path]

[–executable executable-path] [–libcxx directory]

[-e benchmark-exe] [-k command] [-R dir] [-r dir] [-s name]

[-x breakpoint-spec] [-y count] [-# sharp]

[–platform-name platform-name] [–platform-url platform-url]

[–platform-working-dir platform-working-dir] [-d] [-F] [-i]

[-n] [-P] [-q] [-S] [-t] [-u variable] [–env variable] [-v]

[-w] [-T] [–enable-crash-dialog] [–show-inferior-console]

[test-dir [test-dir …]]

dotest.py: error: unrecognized arguments: c:\Users\chuckr\source\llvm\tools\lldb\test\functionalities\breakpoint\cpp

usage: dotest.py [-h] [-A arch] [-C compiler] [-E extra-flags] [-D]

[-N {dwarf,dsym}] [-a] [+a] [+b] [-b blacklist]

[-f filterspec] [-g] [-l] [-m] [+m] [-p pattern]

[-X directory] [-G category] [–skip-category category]

[-c config-file] [–framework framework-path]

[–executable executable-path] [–libcxx directory]

[-e benchmark-exe] [-k command] [-R dir] [-r dir] [-s name]

[-x breakpoint-spec] [-y count] [-# sharp]

[–platform-name platform-name] [–platform-url platform-url]

[–platform-working-dir platform-working-dir] [-d] [-F] [-i]

[-n] [-P] [-q] [-S] [-t] [-u variable] [–env variable] [-v]

[-w] [-T] [–enable-crash-dialog] [–show-inferior-console]

[test-dir [test-dir …]]

dotest.py: error: unrecognized arguments: c:\Users\chuckr\source\llvm\tools\lldb\test\expression_command\persistent_types

usage: dotest.py [-h] [-A arch] [-C compiler] [-E extra-flags] [-D]

[-N {dwarf,dsym}] [-a] [+a] [+b] [-b blacklist]

[-f filterspec] [-g] [-l] [-m] [+m] [-p pattern]

[-X directory] [-G category] [–skip-category category]

[-c config-file] [–framework framework-path]

[–executable executable-path] [–libcxx directory]

[-e benchmark-exe] [-k command] [-R dir] [-r dir] [-s name]

[-x breakpoint-spec] [-y count] [-# sharp]

[–platform-name platform-name] [–platform-url platform-url]

[–platform-working-dir platform-working-dir] [-d] [-F] [-i]

[-n] [-P] [-q] [-S] [-t] [-u variable] [–env variable] [-v]

[-w] [-T] [–enable-crash-dialog] [–show-inferior-console]

[test-dir [test-dir …]]

dotest.py: error: unrecognized arguments: c:\Users\chuckr\source\llvm\tools\lldb\test\functionalities\data-formatter\data-formatter-enum-format

usage: dotest.py [-h] [-A arch] [-C compiler] [-E extra-flags] [-D]

[-N {dwarf,dsym}] [-a] [+a] [+b] [-b blacklist]

[-f filterspec] [-g] [-l] [-m] [+m] [-p pattern]

[-X directory] [-G category] [–skip-category category]

[-c config-file] [–framework framework-path]

[–executable executable-path] [–libcxx directory]

[-e benchmark-exe] [-k command] [-R dir] [-r dir] [-s name]

[-x breakpoint-spec] [-y count] [-# sharp]

[–platform-name platform-name] [–platform-url platform-url]

[–platform-working-dir platform-working-dir] [-d] [-F] [-i]

[-n] [-P] [-q] [-S] [-t] [-u variable] [–env variable] [-v]

[-w] [-T] [–enable-crash-dialog] [–show-inferior-console]

[test-dir [test-dir …]]

……

I run tests on Windows and if something is wrong then I definitely want to fix it.

That said, I’ve never seen these particular failures before. I will say that the test suite only passes about 50% on Windows, but at the end you will see a lot of failures. But they won’t be these failures, they will be actual test failures like couldn’t set a watchpoint.

One thing that seems suspicious to me from looking at your command line is that, although you appear to have built python from source (which is correct), LLDB is finding a release version of Python. If you are building a Debug version of LLDB, this won’t work. LLDB needs to find a debug version of python. I don’t think that’s the actual issue you’re experiencing here though, but it might be. To fix this I recommend explicitly specifying both PYTHON_EXECUTABLE, PYTHON_LIBRARY, and PYTHON_INCLUDE_DIR on your CMake command line.

The next thing I would do is add a line to dosep.py that prints out the command line it uses for dotest.py. dosep.py is a lightweight shim that just spawns many instances of dotest.py in different processes, and it seems to be forming the command line incorrectly. So the key is going to see what command line it’s using for dotest.py.

What version of Windows are you using if I might ask? And are you running it from a cmd prompt or a Powershell prompt?

Also, are you using Cygwin? (It doesn’t look like it from your command line, but as cygwin is a completely untested path, I have no idea what will happen if you use it)

I am doing this on Windows 8.1, running from command prompt. I have specified all three of those python cmake options to the debug versions that I built.

Are you sure you set PYTHON_EXECUTABLE to python_d.exe? Because it’s using python.exe on your command line.

I notice you also don’t have a test compiler specified. I assume you’re building with MSVC, which means you’ve run vcvarsall, which means that the test suite will try to use cl.exe to compile. This also won’t work, you will need to use clang as the compiler. Since you’ve already built clang as part of building LLDB, this shouldn’t be too difficult, you just need to set a CMake variable to use clang. I should probably make the CMake a little bit smarter and attempt to find this automatically. I’ll add a note to do that soon. In the meantime set -DLLDB_TEST_COMPILER=C:/Users/chuckr/source/build_llvm_debug/bin/clang.exe

Ahh actually you do have the compiler specified, and it is cl.exe. So yea, this is definitely wrong and I think it actually the problem for two reasons. The first reason is what I mentioned about the test suite not working with cl.exe at all (because the Makefiles that build test programs assume gcc-style command line options), and for a second reason because there’s a space in the path, and dosep doesn’t support that (although it could be made to). So if you fix the LLDB_TEST_COMPILER issue and set it to clang, that should fix your problem.

I have been using the cmake GUI and it seems to have some quirks. For example, if I added the PYTHON_EXECUTABLE entry and then generated, it would get dropped. I have cleaned and reconfigured my cmake build with your suggestions. Thanks!

Yea I had issues with the CMake GUI a while ago as well. I wrote a batch file that just regenerates for me, and it seems to work better.

Would you be willing to share that batch file?

Additionally, do you do an external lldb on Windows? That seemed to work several months ago but was broken when I tried it today.

Heh. It doesn’t do anything special. This is what it looks like:

cmake -G %1 -DLLDB_TEST_DEBUG_TEST_CRASHES=1 -DPYTHON_LIBRARY=D:\python_src\Python-2.7.8\PCbuild\python27_d.lib -PYTHON_EXECUTABLE=D:\python_src\Python-2.7.8\PCbuild\python_d.exe -DPYTHON_INCLUDE_DIR=D:\python_src\Python-2.7.8\Include -DLLDB_TEST_COMPILER=D:\src\llvmbuild\ninja\bin\clang.exe …..\llvm

You can probably remove the LLDB_TEST_DEBUG_TEST_CRASHES unless you’re interested in fixing test failures with core LLDB on Windows functionality. Although I’m here to help if you are interested in doing that :slight_smile: By setting it to 1 it will cause tests that crash to generate a crash dialog so you can attach your favorite JIT debugger. Default is 0, so that tests silently fail.

The %1 is so I can use both Ninja and “Visual Studio 12” generators. I run my builds with the former since Ninja compiles faster than MSBuild, and I use the latter just for having a Visual Studio solution opened that I can work in (they need to be generated into different folders though obviously)

Also, I’m not sure what you mean by “external lldb”?

Thanks! By external lldb, I mean building lldb standalone outside of the llvm tree and just point it to the llvm/clang source/libs from somewhere else.

Ahh. I don’t think anyone’s using that workflow, at least on Windows. You can try to make it work if you want and submit patches to the CMake, but it’s a difficult proposition since LLDB depends heavily on Clang and LLVM, and we can only really guarantee that ToT LLDB will compile against ToT LLVM and Clang.

I am still having issues running tests on windows. The issue appears to be with building debuggees.

ERROR: test_persistent_types (TestPersistentTypes.PersistenttypesTestCase)

Test that lldb persistent types works correctly.

When you originally checked out clang and llvm, did you also check out lld? i.e. is llvm\tools\lld a folder that exists on your disk? If not, you need to clone lld the same way you cloned clang. The reason this is necessary is because clang outputs DWARF, not PDB, and MSVC’s linker doesn’t know how to link dwarf, only lld does. So running the test suite requires that you build lld.