windows testing - lit.py failure

[dawn]

I build clang from the command-line with cmake on Windows 7 in a different
build directory but am unable to run lit.py. The command I use is:

    python c:/llvmpath/lit.py -sv --no-progress-bar --param=build_config=Debug c:/clangpath/test/Sema/wchar.c

as directed in the "Testing on the Command Line" section of the "Hacking on Clang" html.

I get the error:
lit.py: lit.cfg:87: fatal: No site specific configuration available!

On Linux it seems I have to run all the tests first with "make test" before
I can run a single test with lit.py to set up the various configuration
files it needs. Do I need to do that on Windows? If so, how would I do this
from the command-line with cmake?

Thanks,
-Dawn

[_utf-8_Q_C3_93scar_F]

dawn@burble.org writes:

[snip]

On Linux it seems I have to run all the tests first with "make test" before
I can run a single test with lit.py to set up the various configuration
files it needs. Do I need to do that on Windows?

Yes.

If so, how would I do this from the command-line with cmake?

On the build directory:

cmake --build . --config Debug --target check

"Debug" is the build configuration you are testing. you need that
parameter if you generated Visual Studio project files (you don't need
if you are working with makefiles).

[dawn]

Thanks for the quick response.

This won't work for us however, as we don't use the Visual Studio
project files. We use cmake and nmake entirely from the command-line,
because we run the tests on remote build machines. We also build in a
separate directory outside llvm. We set up certian env vars that MSVC
wants, then run cmake as:

    cmake -G \"NMake Makefiles\" ..\\..\\..\\llvm -DCMAKE_BUILD_TYPE=debug

How would I use this technique to configure the clang tests?

Thanks again,
-Dawn

[_utf-8_Q_C3_93scar_F]

dawn@burble.org writes:

Thanks for the quick response.

This won't work for us however, as we don't use the Visual Studio
project files.

Have you tried it?

We use cmake and nmake entirely from the command-line,
because we run the tests on remote build machines. We also build in a
separate directory outside llvm. We set up certian env vars that MSVC
wants, then run cmake as:

    cmake -G \"NMake Makefiles\" ..\\..\\..\\llvm -DCMAKE_BUILD_TYPE=debug

How would I use this technique to configure the clang tests?

cmake --build <build-directory> --target <target>

just invokes whatever tool is appropiate for building the <target>
(msbuild.exe, nmake.exe, etc) But if you prefer to directly run nmake:

for Clang:

nmake clang-test clang-c++tests

for LLVM:

nmake check

and

nmake check-all

will execute both LLVM and Clang tests.

[_utf-8_Q_C3_93scar_F]

dawn@burble.org writes:

I build clang from the command-line with cmake on Windows 7 in a different
build directory but am unable to run lit.py. The command I use is:

    python c:/llvmpath/lit.py -sv --no-progress-bar --param=build_config=Debug c:/clangpath/test/Sema/wchar.c

as directed in the "Testing on the Command Line" section of the "Hacking on Clang" html.

I get the error:
lit.py: lit.cfg:87: fatal: No site specific configuration available!

On Linux it seems I have to run all the tests first with "make test" before
I can run a single test with lit.py to set up the various configuration
files it needs. Do I need to do that on Windows? If so, how would I do this
from the command-line with cmake?

After looking at the CMake scripts related to testing it seems to me
that you don't need to execute `make check' for generating the
confiration files for lit.

Are you trying to test an installed Clang or LLVM? (that's not
supported) Is the python interpreter found when you configure your build
directory with cmake?

Those c:/llvmpath and c:/clangpath on your command looks suspicious to
me.

[_utf-8_Q_C3_93scar_F]

dawn@burble.org writes:

> I build clang from the command-line with cmake on Windows 7 in a different
> build directory but am unable to run lit.py. The command I use is:
>
> python c:/llvmpath/lit.py -sv --no-progress-bar --param=build_config=Debug c:/clangpath/test/Sema/wchar.c
>
> as directed in the "Testing on the Command Line" section of the "Hacking on Clang" html.
>
> I get the error:
> lit.py: lit.cfg:87: fatal: No site specific configuration available!
>
> On Linux it seems I have to run all the tests first with "make test" before
> I can run a single test with lit.py to set up the various configuration
> files it needs. Do I need to do that on Windows? If so, how would I do this
> from the command-line with cmake?

After looking at the CMake scripts related to testing it seems to me
that you don't need to execute `make check' for generating the
confiration files for lit.

Are you trying to test an installed Clang or LLVM? (that's not
supported) Is the python interpreter found when you configure your build
directory with cmake?

No, it's a clang I built using cmake and nmake.

Sure, but did you installed the resulting binaries with execute `nmake
install' ?

Those c:/llvmpath and c:/clangpath on your command looks suspicious to
me.

What's wrong with that? It comes stright from the online instructions.

The Clang sources are supposed to live within the LLVM source tree
(under tools/clang) not outside the LLVM source directory, unless you
are using the recently added feature for building Clang as an
independent project.

Can you show the exact sequence of commands you executed from the
LLVM&Clang source code checkout until the failed attempt at testing
(both inclusive)?

[dawn]

dawn@burble.org writes:

>> > I build clang from the command-line with cmake on Windows 7 in a different
>> > build directory but am unable to run lit.py. The command I use is:
>> >
>> > python c:/llvmpath/lit.py -sv --no-progress-bar --param=build_config=Debug c:/clangpath/test/Sema/wchar.c
>> >
>> > as directed in the "Testing on the Command Line" section of the "Hacking on Clang" html.
>> >
>> > I get the error:
>> > lit.py: lit.cfg:87: fatal: No site specific configuration available!
>> >
>> > On Linux it seems I have to run all the tests first with "make test" before
>> > I can run a single test with lit.py to set up the various configuration
>> > files it needs. Do I need to do that on Windows? If so, how would I do this
>> > from the command-line with cmake?
>>
>> After looking at the CMake scripts related to testing it seems to me
>> that you don't need to execute `make check' for generating the
>> confiration files for lit.
>>
>> Are you trying to test an installed Clang or LLVM? (that's not
>> supported) Is the python interpreter found when you configure your build
>> directory with cmake?
>
> No, it's a clang I built using cmake and nmake.

Sure, but did you installed the resulting binaries with execute `nmake
install' ?

Oops. No, I didn't know I needed to do that. I just did...
but it made no difference - the nmake and cmake commands you gave me both fail
in lit.cfg.

>> Those c:/llvmpath and c:/clangpath on your command looks suspicious to
>> me.
>
> What's wrong with that? It comes stright from the online instructions.

The Clang sources are supposed to live within the LLVM source tree
(under tools/clang) not outside the LLVM source directory, unless you
are using the recently added feature for building Clang as an
independent project.

clang lives in llvm/tools as usual.

Can you show the exact sequence of commands you executed from the
LLVM&Clang source code checkout until the failed attempt at testing
(both inclusive)?

cd c:\dev
svn co http://llvm.org/svn/llvm-project/llvm/trunk llvm
cd llvm\tools
svn co http://llvm.org/svn/llvm-project/cfe/trunk clang

cd c:\dev
mkdir build
cd build

cmd /c msvcenv.cmd
@rem where mscvenv.cmd is:
@rem ------------------------
@rem script to build clang libraries
set ROOT=%cd%\..
set PATH=%ROOT%\..\msvc\bin;%ROOT%\..\cmake\bin;%PATH%
set INCLUDE=%ROOT%\..\msvc\include;%ROOT%\..\msvc\sdkinclude;
set LIB=%ROOT%\..\msvc\lib
set VCINSTALLDIR=%ROOT%\..\msvc
set VSINSTALLDIR=%ROOT%\..\msvc

@rem generate cmake makefiles
cmake -G "NMake Makefiles" ..\llvm -DCMAKE_BUILD_TYPE=debug

@rem build clang
set MFLAGS=
set MAKEFLAGS=
nmake MAKEFLAGS=

@rem ---- this is new ----

@rem build the test files
set PATH=%ROOT%\..\shared\tools\python;%PATH%
cmake --build . --config Debug --target check

@rem install the files so we can run the tests
@rem Yuk! this installs in c:/Program Files (x86)/LLVM
nmake MAKEFLAGS= install

@rem now run the tests
nmake MAKEFLAGS= clang-test clang-c++tests

[_utf-8_Q_C3_93scar_F]

dawn@burble.org writes:

Have you tried it?

Yes, it failed at in the lit python script with:

[100%] Running LLVM regression tests
File "C:/path/llvm/utils/lit/lit.py", line 5
[...]
File "C:/path/llvm/test/lit.cfg", line 166, in <module>
site_exp[sub].replace('-fno-expections', '')))
KeyError: 'compile_cxx'
NMAKE: fatal error U1077:
[...]

The cmake --build command worked. What failed is lit. That appearence of
-fno-exceptions looks like a bug.

[_utf-8_Q_C3_93scar_F]

dawn@burble.org writes:

>> Are you trying to test an installed Clang or LLVM? (that's not
>> supported) Is the python interpreter found when you configure your build
>> directory with cmake?
>
> No, it's a clang I built using cmake and nmake.

Sure, but did you installed the resulting binaries with execute `nmake
install' ?

Oops. No, I didn't know I needed to do that. I just did...
but it made no difference - the nmake and cmake commands you gave me both fail
in lit.cfg.

No, you don't need to that. Actualy, as explained above, running tests
on the installed files of LLVM/Clang is not supported. Tests must be run
from the build directory (as you are doing)

>> Those c:/llvmpath and c:/clangpath on your command looks suspicious to
>> me.
>
> What's wrong with that? It comes stright from the online instructions.

The Clang sources are supposed to live within the LLVM source tree
(under tools/clang) not outside the LLVM source directory, unless you
are using the recently added feature for building Clang as an
independent project.

clang lives in llvm/tools as usual.

So what's c:/llvmpath and c:/clangpath ?

Can you show the exact sequence of commands you executed from the
LLVM&Clang source code checkout until the failed attempt at testing
(both inclusive)?

cd c:\dev
svn co http://llvm.org/svn/llvm-project/llvm/trunk llvm
cd llvm\tools
svn co http://llvm.org/svn/llvm-project/cfe/trunk clang

cd c:\dev
mkdir build
cd build

cmd /c msvcenv.cmd
@rem where mscvenv.cmd is:
@rem ------------------------
@rem script to build clang libraries
set ROOT=%cd%\..
set PATH=%ROOT%\..\msvc\bin;%ROOT%\..\cmake\bin;%PATH%
set INCLUDE=%ROOT%\..\msvc\include;%ROOT%\..\msvc\sdkinclude;
set LIB=%ROOT%\..\msvc\lib
set VCINSTALLDIR=%ROOT%\..\msvc
set VSINSTALLDIR=%ROOT%\..\msvc

@rem generate cmake makefiles
cmake -G "NMake Makefiles" ..\llvm -DCMAKE_BUILD_TYPE=debug

@rem build clang
set MFLAGS=
set MAKEFLAGS=
nmake MAKEFLAGS=

@rem ---- this is new ----

@rem build the test files
set PATH=%ROOT%\..\shared\tools\python;%PATH%
cmake --build . --config Debug --target check

Python must be in the PATH before you generate the makefiles, that means
before the

cmake -G "NMake Makefiles" ..\llvm -DCMAKE_BUILD_TYPE=debug

(also replace `debug' by `Debug', just in case)

@rem install the files so we can run the tests
@rem Yuk! this installs in c:/Program Files (x86)/LLVM
nmake MAKEFLAGS= install

You'll better install after testing succeeds.

[dawn]

dawn@burble.org writes:

>> >> Are you trying to test an installed Clang or LLVM? (that's not
>> >> supported) Is the python interpreter found when you configure your build
>> >> directory with cmake?
>> >
>> > No, it's a clang I built using cmake and nmake.
>>
>> Sure, but did you installed the resulting binaries with execute `nmake
>> install' ?
>
> Oops. No, I didn't know I needed to do that. I just did...
> but it made no difference - the nmake and cmake commands you gave me both fail
> in lit.cfg.

No, you don't need to that. Actualy, as explained above, running tests
on the installed files of LLVM/Clang is not supported. Tests must be run
from the build directory (as you are doing)

>> >> Those c:/llvmpath and c:/clangpath on your command looks suspicious to
>> >> me.
>> >
>> > What's wrong with that? It comes stright from the online instructions.
>>
>> The Clang sources are supposed to live within the LLVM source tree
>> (under tools/clang) not outside the LLVM source directory, unless you
>> are using the recently added feature for building Clang as an
>> independent project.
>
> clang lives in llvm/tools as usual.

So what's c:/llvmpath and c:/clangpath ?

llvmpath = c:\dev\llvm
clangpath = c:\dev\llvm\tools\clang

>> Can you show the exact sequence of commands you executed from the
>> LLVM&Clang source code checkout until the failed attempt at testing
>> (both inclusive)?
>
> cd c:\dev
> svn co http://llvm.org/svn/llvm-project/llvm/trunk llvm
> cd llvm\tools
> svn co http://llvm.org/svn/llvm-project/cfe/trunk clang
>
> cd c:\dev
> mkdir build
> cd build
>
> cmd /c msvcenv.cmd
> @rem where mscvenv.cmd is:
> @rem ------------------------
> @rem script to build clang libraries
> set ROOT=%cd%\..
> set PATH=%ROOT%\..\msvc\bin;%ROOT%\..\cmake\bin;%PATH%
> set INCLUDE=%ROOT%\..\msvc\include;%ROOT%\..\msvc\sdkinclude;
> set LIB=%ROOT%\..\msvc\lib
> set VCINSTALLDIR=%ROOT%\..\msvc
> set VSINSTALLDIR=%ROOT%\..\msvc
>
> @rem generate cmake makefiles
> cmake -G "NMake Makefiles" ..\llvm -DCMAKE_BUILD_TYPE=debug
>
> @rem build clang
> set MFLAGS=
> set MAKEFLAGS=
> nmake MAKEFLAGS=

This all works great - no errors.

>
> @rem ---- this is new ----
>
> @rem build the test files
> set PATH=%ROOT%\..\shared\tools\python;%PATH%
> cmake --build . --config Debug --target check

Python must be in the PATH before you generate the makefiles, that means
before the

cmake -G "NMake Makefiles" ..\llvm -DCMAKE_BUILD_TYPE=debug

(also replace `debug' by `Debug', just in case)

That cmake command works just great.
It's the 2nd cmake that fails in lit.

> @rem install the files so we can run the tests
> @rem Yuk! this installs in c:/Program Files (x86)/LLVM
> nmake MAKEFLAGS= install

You'll better install after testing succeeds.

Ok - I was just trying the various things you suggested. I'll delete
that line now that I know it's not needed.

[dawn]

So what should I try next? How can I fix this?

-Dawn

[_utf-8_Q_C3_93scar_F]

dawn@burble.org writes:

> @rem generate cmake makefiles
> cmake -G "NMake Makefiles" ..\llvm -DCMAKE_BUILD_TYPE=debug
>
> @rem build clang
> set MFLAGS=
> set MAKEFLAGS=
> nmake MAKEFLAGS=

This all works great - no errors.

>
> @rem ---- this is new ----
>
> @rem build the test files
> set PATH=%ROOT%\..\shared\tools\python;%PATH%
> cmake --build . --config Debug --target check

Python must be in the PATH before you generate the makefiles, that means
before the

cmake -G "NMake Makefiles" ..\llvm -DCMAKE_BUILD_TYPE=debug

(also replace `debug' by `Debug', just in case)

That cmake command works just great.
It's the 2nd cmake that fails in lit.

CMake just detects what's available. If Python is not found in the
system, it will simply display a line of text reporting that python was
not found and then proceeds to generate the makefiles. But as `nmake
clang-test' works I guess that python already is on the PATH when `cmake
-G ..' is executed.

As for the python exception thrown from lit.cfg, I suggest filing a bug
report (or, better, fixing the bug yourself

[snip]

[David_Neto]

I just came across this issue myself.

To recap:
- if you use Cmake to generate Nmake makefiles (-G "NMake Makefiles")
- and you're using a Microsoft compiler (maybe this is not essential)
- then lit.py fails on startup

This is the error:

[100%] Running LLVM regression tests
Traceback (most recent call last):
  File "d:/llvm/llvm/utils/lit/lit.py", line 5, in <module>
    lit.main()
  File "d:\llvm\llvm\utils\lit\lit\main.py", line 515, in main
    testSuiteCache, localConfigCache)[1])
  File "d:\llvm\llvm\utils\lit\lit\main.py", line 230, in getTests
    ts,path_in_suite = getTestSuite(path, litConfig, testSuiteCache)
  File "d:\llvm\llvm\utils\lit\lit\main.py", line 199, in getTestSuite
    ts, relative = search(item)
  File "d:\llvm\llvm\utils\lit\lit\main.py", line 183, in search
    cache[path] = res = search1(path)
  File "d:\llvm\llvm\utils\lit\lit\main.py", line 174, in search1
    cfg = TestingConfig.frompath(cfgpath, None, litConfig, mustExist = True)
  File "d:\llvm\llvm\utils\lit\lit\TestingConfig.py", line 45, in frompath
    exec f in cfg_globals
  File "D:\llvm\llvm-nmake-dbg\test\lit.site.cfg", line 20, in <module>
    lit.load_config(config, "d:/llvm/llvm/test/lit.cfg")
  File "d:\llvm\llvm\utils\lit\lit\LitConfig.py", line 65, in load_config
    config = config)
  File "d:\llvm\llvm\utils\lit\lit\TestingConfig.py", line 45, in frompath
    exec f in cfg_globals
  File "d:/llvm/llvm/test/lit.cfg", line 166, in <module>
    site_exp[sub].replace('-fno-exceptions', '')))
KeyError: 'compile_cxx'

I tracked it down to Cmake generating a test/site.exp with nmake-style
continuation lines, and llvm/test/lit.cfg not handling those
continuation lines.

For example, in my site.exp I get the following (on LLVM 2.9):

set compile_c ""
set compile_cxx "c:/devstudio2008/VC/bin/amd64/cl.exe @ /Fd -c
<< -ID:/llvm/llvm-nmake-dbg/include -Id:/llvm/llvm/include"
set link ""
set llvmgcc ""
set llvmgxx ""

But lit.cfg can't parse the "<< " line as a continuation of the
definition of compile_cxx.

I have a patch for lit.cfg that works against LLVM 2.8. I'll file a
bug with the fix once I get download a trunk image.

thanks,
david

[_utf-8_Q_C3_93scar_F]

David Neto <dneto.llvm@gmail.com> writes:

As for the python exception thrown from lit.cfg, I suggest filing a bug
report (or, better, fixing the bug yourself

I just came across this issue myself.

To recap:
- if you use Cmake to generate Nmake makefiles (-G "NMake Makefiles")
- and you're using a Microsoft compiler (maybe this is not essential)
- then lit.py fails on startup

[snip]

But lit.cfg can't parse the "<< " line as a continuation of the
definition of compile_cxx.

I have a patch for lit.cfg that works against LLVM 2.8. I'll file a
bug with the fix once I get download a trunk image.

Thanks for the detailed diagnostic. I'm looking forward to your bug
report and patch.

[David_Neto]

I've created http://llvm.org/bugs/show_bug.cgi?id=9680 and attached a
trunk patch there and to this email.

thanks,
david

pr9680-key-error.txt (1.33 KB)

[_utf-8_Q_C3_93scar_F]

David Neto <dneto.llvm@gmail.com> writes:

I've created 9680 – KeyError: 'compile_cxx' when using cmake -G "NMake Makefiles" and attached a
trunk patch there and to this email.

Someone who knows about lit.py please review this patch.

[David_Neto]

Ack! I'm withdrawing this patch. Sorry. It turns out the resulting
compilation line is still quite badly broken.

I've been struggling to get the MSVC 9 + NMake flow going. Only now
have I been able to run tests in bulk.

I need to investigate some more.
Again, I apologize.

david

[_utf-8_Q_C3_93scar_F]

David Neto <dneto.llvm@gmail.com> writes:

Ack! I'm withdrawing this patch. Sorry. It turns out the resulting
compilation line is still quite badly broken.

I've been struggling to get the MSVC 9 + NMake flow going. Only now
have I been able to run tests in bulk.

I need to investigate some more.

I followed a different approach: fix the problem from the cmake
file. This patch still fails some tests, but if you can figure out what
the problem is with the command line it generates maybe I could extend
it for a definitive fix. Please note that you don't need to know cmake,
just do a `nmake check' and see what's wrong with the commands shown for
the failed tests.

--- a/test/CMakeLists.txt
+++ b/test/CMakeLists.txt
@@ -37,11 +37,15 @@ if(PYTHONINTERP_FOUND)
   foreach(INC_DIR ${INC_DIRS})
     set(IDIRS "${IDIRS} -I${INC_DIR}")
   endforeach()
- string(REPLACE "<CMAKE_CXX_COMPILER>" "${CMAKE_CXX_COMPILER}" TEST_COMPILE_CXX_CMD ${CMAKE_CXX_COMPILE_OBJECT})
+ string(REPLACE "<CMAKE_CXX_COMPILER>" "\"${CMAKE_CXX_COMPILER}\"" TEST_COMPILE_CXX_CMD ${CMAKE_CXX_COMPILE_OBJECT})
+ string(REPLACE "@" "" TEST_COMPILE_CXX_CMD ${TEST_COMPILE_CXX_CMD})
   string(REPLACE "<DEFINES>" "${DEFS}" TEST_COMPILE_CXX_CMD ${TEST_COMPILE_CXX_CMD})
   string(REPLACE "<FLAGS>" "${CMAKE_CXX_FLAGS}" TEST_COMPILE_CXX_CMD ${TEST_COMPILE_CXX_CMD})
   string(REPLACE "-o" "" TEST_COMPILE_CXX_CMD ${TEST_COMPILE_CXX_CMD})
   string(REGEX REPLACE "<[^>]+>" "" TEST_COMPILE_CXX_CMD ${TEST_COMPILE_CXX_CMD})
+ message(STATUS "cmd: ${TEST_COMPILE_CXX_CMD}")
+ string(REPLACE "\n<<" " " TEST_COMPILE_CXX_CMD ${TEST_COMPILE_CXX_CMD})
+ message(STATUS "cmd: ${TEST_COMPILE_CXX_CMD}")
   set(TEST_COMPILE_CXX_CMD "${TEST_COMPILE_CXX_CMD} ${IDIRS}")
   if(NOT MSVC)
     set(TEST_COMPILE_CXX_CMD "${TEST_COMPILE_CXX_CMD} -x c++")

Again, I apologize.

No problem.

[David_Neto]

I agree. I have updated PR9680 with a new patch to fix how
test/CMakeLists.txt handles MSVC options.
Now I can get the right compile_cxx definition into site.exp and need
no updates to lit.cfg.

Please review.

thanks,
david

PR9680-fix-cmake-nmake-msvc-flow.patch (1.8 KB)