Parsing VC++ headers with tool/libtooling

Clang Frontend
1

I have a custom tool that uses libTooling. I am running the tool on Windows. When I attempt to run the tool over my code I get many errors of the following form:

2> In file included from C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\include\vector:6:
2> In file included from C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\include\xmemory:6:
2> In file included from C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\include\xmemory0:1015:
2> In file included from C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\include\intrin.h:24:
2> C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\include\immintrin.h:764:28: error: invalid token in macro parameter list
2> #define _mm256_loadu2_m128(/* float const* / hiaddr,
2> ^
2> C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\include\immintrin.h:768:29: error: invalid token in macro parameter list
2> #define _mm256_loadu2_m128d(/ double const* / hiaddr,
2> ^
2> C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\include\immintrin.h:772:29: error: invalid token in macro parameter list
2> #define _mm256_loadu2_m128i(/ __m128i const* / hiaddr,
2> ^
2> C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\include\immintrin.h:782:29: error: invalid token in macro parameter list
2> #define _mm256_storeu2_m128(/ float* / hiaddr, / float* / loaddr,
2> ^
2> C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\include\immintrin.h:790:30: error: invalid token in macro parameter list
2> #define _mm256_storeu2_m128d(/ double* / hiaddr, / double* / loaddr,
2> ^
2> C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\include\immintrin.h:798:30: error: invalid token in macro parameter list
2> #define _mm256_storeu2_m128i(/ __m128i* / hiaddr, / __m128i* / loaddr,
2> ^
2> C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\include\immintrin.h:1117:25: error: invalid token in macro parameter list
2> #define _mm256_set_m128(/ __m128 / hi, / __m128 / lo)
2> ^
2> C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\include\immintrin.h:1120:26: error: invalid token in macro parameter list
2> #define _mm256_set_m128d(/ __m128d / hi, / __m128d / lo)
2> ^
2> C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\include\immintrin.h:1123:26: error: invalid token in macro parameter list
2> #define _mm256_set_m128i(/ __m128i / hi, / __m128i */ lo)
2> ^
2> 9 errors generated.

Is there some way to get my tool to handle this gracefully? My command-line for my tool looks like:
MyTool.exe – -IC:/Path/To/Includes -DSOME_MACRO

2

You probably need to pass some combination of -fms-compatibility -fms-extensions -fdelayed-template-parsing and a MS abi triple to -target to be able to parse SDK headers. You can try running clang-cl -c some_file.cc -### to see which cc1 parameters clang-cl passes by default.

Actually, looking at lib/Driver/Tools.cpp, it looks like the driver will use good defaults as long as the triple thinks that isWindowsMSVCEnvironment() is true. Try adding something like “-target x86_64-pc-windows-msvc18.0.0” to your flag, that’s probably enough. Alternatively, you could also try passing "–driver-mode=cl " as first parameter, then the tool will behave like clang-cl in general (but it’ll expect clang-cl flags, see bin/clang-cl /? for a list. -I and -D should work).

Nico

3

This seems more like failure to prefer Clang’s immintrin.h over MSVC’s. Getting that to work in right Clang tools is annoying.

4

Note that to make writing compiler specific tools simpler, tooling has grown a function addTargetAndModeForProgramName (where you can hopefully pass clang-cl for it to figure out a windows triple - if this doesn’t work, let me know and we’ll figure out how to fix that)

5

I wrote a quick program to play with addTargetAndModeForProgramName, and it appears to not work:

#include <clang/Tooling/Tooling.h>

int main()
{
std::vectorstd::string args{ “clang-cl”, “main.cpp” };
clang::tooling::addTargetAndModeForProgramName(args, “i686-pc-windows-msvc18.0.0”);
return 0;
}

Any suggestions on what I am doing wrong? The args variable does not get updated by the function call.

6

The second parameter is the name of the executable the compiler was “invoked as”. This should be “clang-cl.exe” probably.

7

When I changed to second parameter to “clang-cl.exe” I get an output argument list of {“clang-cl”, “–driver-mode=cl”, “main.cpp”}.

When I add that additional driver to the invocation of my tool I still get the same set of errors.

Tool usage:
MyTool.exe main.cpp – --driver-mode=cl

8

That seems to indicate this function doesn’t work for clang-cl yet. Luke, ideas?

9

I’m in the process of setting up a Windows environment to reproduce/debug this.

Are you adding --driver-mode=cl first in the arguments list, or are you appending it to the arguments you read in from your compilation database?

Are you using response files for this compilation?

What happens if you add “-fms-compatibility” and “-fms-extensions” to the end of the arguments list? (I’d have thought these would be implicit with --driver-mode=cl, but maybe not.)

10

On Windows, clang has more or less the same defaults for -fms-compatibility
and -fms-extensions as clang-cl.

I wasn't able to reproduce Daniel's error about invalid tokens in macro
lists locally with clang, clang-check, or clang-tidy. =/ Any way to
reproduce that error with VC's intrinsic headers would be useful.

11

My exact call to my tool is:
MyTool.exe main.cpp – --driver-mode=cl -fms-compatibility -fms-extensions

I get:
error : unknown argument: ‘-fms-compatibility’
error : unknown argument: ‘-fms-extensions’

And I get the same set of errors from earlier. Also, if I run it from the command line I get a different set of errors than if I run it from a Visual Studio build.

12

For this source code:
#include
std::vector v;

I run:
clang-tidy x.cpp – --driver-mode=cl

From a command prompt opened with the link:
Developer Command Prompt for VS2015

I get this output:
2 errors generated.
Error while processing D:\CMakeTest\src\x.cpp.
C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\INCLUDE\iosfwd:267:21: error: use of undeclared identifier ‘char16_t’ [clang-diagnostic-error]
struct char_traits<char16_t>
^
C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\INCLUDE\iosfwd:276:21: error: use of undeclared identifier ‘char32_t’ [clang-diagnostic-error]
struct char_traits<char32_t>
^
C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\INCLUDE\limits:615:33: error: use of undeclared identifier ‘char16_t’ [clang-diagnostic-error]
template<> class numeric_limits<char16_t>
^
C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\INCLUDE\limits:905:33: error: use of undeclared identifier ‘char32_t’ [clang-diagnostic-error]
template<> class numeric_limits<char32_t>
^
C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\INCLUDE\xstddef:408:14: error: use of undeclared identifier ‘char16_t’ [clang-diagnostic-error]
struct hash<char16_t>
^
C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\INCLUDE\xstddef:414:14: error: use of undeclared identifier ‘char32_t’ [clang-diagnostic-error]
struct hash<char32_t>
^
C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\INCLUDE\xstring:2623:22: error: use of undeclared identifier ‘char16_t’ [clang-diagnostic-error]
typedef basic_string<char16_t, char_traits<char16_t>, allocator<char16_t> >
^
C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\INCLUDE\xstring:2623:53: error: expected unqualified-id [clang-diagnostic-error]
typedef basic_string<char16_t, char_traits<char16_t>, allocator<char16_t> >
^
C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\INCLUDE\xstring:2625:22: error: use of undeclared identifier ‘char32_t’ [clang-diagnostic-error]
typedef basic_string<char32_t, char_traits<char32_t>, allocator<char32_t> >
^
C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\INCLUDE\xstring:2625:53: error: expected unqualified-id [clang-diagnostic-error]
typedef basic_string<char32_t, char_traits<char32_t>, allocator<char32_t> >
^
C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\INCLUDE\xtr1common:231:22: error: use of undeclared identifier ‘char16_t’ [clang-diagnostic-error]
struct _Is_integral<char16_t>
^
C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\INCLUDE\xtr1common:237:22: error: use of undeclared identifier ‘char32_t’ [clang-diagnostic-error]
struct _Is_integral<char32_t>
^

13

When I run my own tool with the same set of arguments, and the same setup, I get the following errors:
In file included from D:\CMakeTest\src\x.cpp:1:
In file included from C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\INCLUDE\vector:6:
In file included from C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\INCLUDE\xmemory:6:
In file included from C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\INCLUDE\xmemory0:1015:
In file included from C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\INCLUDE\intrin.h:24:
C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\INCLUDE\immintrin.h:764:28: error:
invalid token in macro parameter list
#define _mm256_loadu2_m128(/* float const* / hiaddr,
^
C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\INCLUDE\immintrin.h:768:29: error:
invalid token in macro parameter list
#define _mm256_loadu2_m128d(/ double const* / hiaddr,
^
C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\INCLUDE\immintrin.h:772:29: error:
invalid token in macro parameter list
#define _mm256_loadu2_m128i(/ __m128i const* / hiaddr,
^
C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\INCLUDE\immintrin.h:782:29: error:
invalid token in macro parameter list
#define _mm256_storeu2_m128(/ float* / hiaddr, / float* / loaddr,
^
C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\INCLUDE\immintrin.h:790:30: error:
invalid token in macro parameter list
#define _mm256_storeu2_m128d(/ double* / hiaddr, / double* / loaddr,
^
C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\INCLUDE\immintrin.h:798:30: error:
invalid token in macro parameter list
#define _mm256_storeu2_m128i(/ __m128i* / hiaddr, / __m128i* / loaddr,
^
C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\INCLUDE\immintrin.h:1117:25: error:
invalid token in macro parameter list
#define _mm256_set_m128(/ __m128 / hi, / __m128 / lo)
^
C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\INCLUDE\immintrin.h:1120:26: error:
invalid token in macro parameter list
#define _mm256_set_m128d(/ __m128d / hi, / __m128d / lo)
^
C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\INCLUDE\immintrin.h:1123:26: error:
invalid token in macro parameter list
#define _mm256_set_m128i(/ __m128i / hi, / __m128i */ lo)
^
9 errors generated.

14

For this source code:
#include <vector>
std::vector<int> v;

I run:
clang-tidy x.cpp -- --driver-mode=cl

From a command prompt opened with the link:
Developer Command Prompt for VS2015

I get this output:
2 errors generated.
Error while processing D:\CMakeTest\src\x.cpp.
C:\Program Files (x86)\Microsoft Visual Studio
14.0\VC\INCLUDE\iosfwd:267:21: error: use of undeclared identifier
'char16_t' [clang-diagnostic-error]
        struct char_traits<char16_t>
                           ^
C:\Program Files (x86)\Microsoft Visual Studio
14.0\VC\INCLUDE\iosfwd:276:21: error: use of undeclared identifier
'char32_t' [clang-diagnostic-error]
        struct char_traits<char32_t>
                           ^
C:\Program Files (x86)\Microsoft Visual Studio
14.0\VC\INCLUDE\limits:615:33: error: use of undeclared identifier
'char16_t' [clang-diagnostic-error]
template<> class numeric_limits<char16_t>
                                ^
C:\Program Files (x86)\Microsoft Visual Studio
14.0\VC\INCLUDE\limits:905:33: error: use of undeclared identifier
'char32_t' [clang-diagnostic-error]
template<> class numeric_limits<char32_t>
                                ^
C:\Program Files (x86)\Microsoft Visual Studio
14.0\VC\INCLUDE\xstddef:408:14: error: use of undeclared identifier
'char16_t' [clang-diagnostic-error]
        struct hash<char16_t>
                    ^
C:\Program Files (x86)\Microsoft Visual Studio
14.0\VC\INCLUDE\xstddef:414:14: error: use of undeclared identifier
'char32_t' [clang-diagnostic-error]
        struct hash<char32_t>
                    ^
C:\Program Files (x86)\Microsoft Visual Studio
14.0\VC\INCLUDE\xstring:2623:22: error: use of undeclared identifier
'char16_t' [clang-diagnostic-error]
typedef basic_string<char16_t, char_traits<char16_t>, allocator<char16_t> >
                     ^
C:\Program Files (x86)\Microsoft Visual Studio
14.0\VC\INCLUDE\xstring:2623:53: error: expected unqualified-id
[clang-diagnostic-error]
typedef basic_string<char16_t, char_traits<char16_t>, allocator<char16_t> >
                                                    ^
C:\Program Files (x86)\Microsoft Visual Studio
14.0\VC\INCLUDE\xstring:2625:22: error: use of undeclared identifier
'char32_t' [clang-diagnostic-error]
typedef basic_string<char32_t, char_traits<char32_t>, allocator<char32_t> >
                     ^
C:\Program Files (x86)\Microsoft Visual Studio
14.0\VC\INCLUDE\xstring:2625:53: error: expected unqualified-id
[clang-diagnostic-error]
typedef basic_string<char32_t, char_traits<char32_t>, allocator<char32_t> >
                                                    ^
C:\Program Files (x86)\Microsoft Visual Studio
14.0\VC\INCLUDE\xtr1common:231:22: error: use of undeclared identifier
'char16_t' [clang-diagnostic-error]
        struct _Is_integral<char16_t>
                            ^
C:\Program Files (x86)\Microsoft Visual Studio
14.0\VC\INCLUDE\xtr1common:237:22: error: use of undeclared identifier
'char32_t' [clang-diagnostic-error]
        struct _Is_integral<char32_t>
                            ^

Clang doesn't know if the STL or Clang should provide char16_t/char32_t
without -fms-compatibility-version. Using -fms-compatibility-version=19
would inform the compiler that we are targeting VS2015 which doesn't supply
these types which would cause Clang to provide them.

15

Using both --driver-mode=cl and -fms-compatibility-version=19 I still get the errors about invalid token in macro parameter list.

16

What version of clang and VS 2015 are you using? I've tried with clang-tidy
and your sources, and I'm not able to reproduce this.

Furthermore, I don't understand why clang is rejecting this macro
definition. Clang accepts this on Linux:
#define _mm256_set_m128i(/* __m128i */ hi, /* __m128i */ lo) \
  (a + b)
int main() {
  int a, b;
  _mm256_set_m128i(a, b);
}

There doesn't seem to be MS-specific behavior around '#define
foo(/*comment*/ argname)', so far as I can tell.

17

Source code for Clang is checked out of:
http://llvm.org/svn/llvm-project/llvm/tags/RELEASE_370/final
http://llvm.org/svn/llvm-project/cfe/tags/RELEASE_370/final
http://llvm.org/svn/llvm-project/clang-tools-extra/tags/RELEASE_370/final

I am using Visual Studio 2015 Enterprise.

clang-tidy that I have installed is:
D:\CMakeTest\src>clang-tidy --version
LLVM (http://llvm.org/):
LLVM version 3.8.0-r251293
DEBUG build with assertions.
Built Oct 26 2015 (10:01:02).
Default target: i686-pc-windows-gnu
Host CPU: sandybridge