Clang-cl.exe and the VC++ preprocessor

Clang Frontend
1

I have been told, via my bug report on a preprocessor error in the latest version of clang-cl.exe, that clang-cl.exe is trying to emulate at least some of the bugs of the VC++ preprocessor.

NO ! Please do not do that.

I know, from having worked on variadic macros in Boost PP and having created my own variadic macro data library, how broken the VC++ preprocessor is when it comes to preprocessor metaprogramming. Paul Mensonides, the creator of Boost PP and an exceptional expert on the C++ preprocessor, also knows and has bemoaned countless times how difficult it is to do preprocessor metaprogramming if one has to deal with the VC++ preprocessor. Not only will clang-cl.exe's emulation of the brokeness of the VC++ preprocessor set back further efforts in metaprogramming preprocessor code and libraries but there is absolutely no chance, if it means anything to clang-cl developers, that any effort will be made to accomodate clang-cl's preprocessor bugs into Boost PP ( or my own variadic macro data library ).

This is completely the wrong direction for clang-cl to go. Please stop and only consider producing as close to as possible a 100% C++ conformant preprocessor, as the rest of clang no doubt has as its goal.

2

I have the opposite view. Having an MSVC-compatible preprocessor is an
excellent thing for interoperability on the platform. It increases the
likelihood of adopting clang as a native toolset on Windows, and
reduces the barrier to entry, because it allows me to take code which
compiles with MSVC and instead run it through clang.

This functionality is behind a flag (-fms-compatibility), and it would
be a step backwards to say "please turn on compatibility with MSVC,
except for the preprocessor" when that flag is enabled. It would also
prevent using MSVC's STL (IIRC, <type_traits> relies on this behavior
in MSVC 2012 due to the compiler's lack of support for variadic
templates).

I would not be opposed to a more fine-grained flag that allowed you to
disable the quirks for the processor when -fms-compatibility is turned
on, akin to -fno-delayed-template-parsing.

~Aaron

3

So, I feel like we're between a rock and a hard place here.

At a fundamental level there are bugs we need to be bug-for-bug compatible
with because there is large amounts of existing code in the wild relying on
those bugs.

At the same time, I'm quite sympathetic to not wanting to make things less
conforming.

But I don't see an easy way to have both. Here are my suggestions, but none
of them are great:

1) Make sure that any bug-compatibility is inspired by actual code relying
on the bug. I suspect it is in this case, but its always good to check.

2) It may be desirable for Boost (and Boost users) to opt into a more
strict mode. I get the flags backwards all the time (Reid or someone else
will correct me) but we have two flags to control compatibility: one for
essentially conforming (or at least not horrible broken) extensions, and
another for bug-for-bug compatibility. I think the latter is
'-fms-compatibility' while the former is '-fms-extensions', but again, i
get them backwards. Anyways, perhaps Boost would want to use
'-fno-ms-compatibility' where possible to get a more conforming
implementation.

3) I think we should then make sure we expose macros you can use to detect
the bug-for-bug compat mode, and use the (horrible) cl.exe implementations
of things like Boost PP, even though the host compiler is Clang.

Does this make sense?

4

I have been told, via my bug report on a preprocessor error in the latest
version of clang-cl.exe, that clang-cl.exe is trying to emulate at least
some of the bugs of the VC++ preprocessor.

NO ! Please do not do that.

I know, from having worked on variadic macros in Boost PP and having
created my own variadic macro data library, how broken the VC++
preprocessor is when it comes to preprocessor metaprogramming. Paul
Mensonides, the creator of Boost PP and an exceptional expert on the C++
preprocessor, also knows and has bemoaned countless times how difficult it
is to do preprocessor metaprogramming if one has to deal with the VC++
preprocessor. Not only will clang-cl.exe's emulation of the brokeness of
the VC++ preprocessor set back further efforts in metaprogramming
preprocessor code and libraries but there is absolutely no chance, if it
means anything to clang-cl developers, that any effort will be made to
accomodate clang-cl's preprocessor bugs into Boost PP ( or my own variadic
macro data library ).

This is completely the wrong direction for clang-cl to go. Please stop
and only consider producing as close to as possible a 100% C++ conformant
preprocessor, as the rest of clang no doubt has as its goal.

So, I feel like we're between a rock and a hard place here.

At a fundamental level there are bugs we need to be bug-for-bug compatible
with because there is large amounts of existing code in the wild relying on
those bugs.

At the same time, I'm quite sympathetic to not wanting to make things less
conforming.

But I don't see an easy way to have both. Here are my suggestions, but
none of them are great:

1) Make sure that any bug-compatibility is inspired by actual code relying
on the bug. I suspect it is in this case, but its always good to check.

2) It may be desirable for Boost (and Boost users) to opt into a more
strict mode. I get the flags backwards all the time (Reid or someone else
will correct me) but we have two flags to control compatibility: one for
essentially conforming (or at least not horrible broken) extensions, and
another for bug-for-bug compatibility. I think the latter is
'-fms-compatibility' while the former is '-fms-extensions', but again, i
get them backwards. Anyways, perhaps Boost would want to use
'-fno-ms-compatibility' where possible to get a more conforming
implementation.

3) I think we should then make sure we expose macros you can use to detect
the bug-for-bug compat mode, and use the (horrible) cl.exe implementations
of things like Boost PP, even though the host compiler is Clang.

Shouldn't that just be _MSC_VER?

5

It might be desirable to be able to compile some Windows system headers
(that rely on _MSC_VER, but not on all of the crazy in the bug-for-bug
compatibility) in this mode.

6

I have been told, via my bug report on a preprocessor error in the latest
version of clang-cl.exe, that clang-cl.exe is trying to emulate at least
some of the bugs of the VC++ preprocessor.

NO ! Please do not do that.

I know, from having worked on variadic macros in Boost PP and having created
my own variadic macro data library, how broken the VC++ preprocessor is when
it comes to preprocessor metaprogramming. Paul Mensonides, the creator of
Boost PP and an exceptional expert on the C++ preprocessor, also knows and
has bemoaned countless times how difficult it is to do preprocessor
metaprogramming if one has to deal with the VC++ preprocessor. Not only will
clang-cl.exe's emulation of the brokeness of the VC++ preprocessor set back
further efforts in metaprogramming preprocessor code and libraries but there
is absolutely no chance, if it means anything to clang-cl developers, that
any effort will be made to accomodate clang-cl's preprocessor bugs into
Boost PP ( or my own variadic macro data library ).

This is completely the wrong direction for clang-cl to go. Please stop and
only consider producing as close to as possible a 100% C++ conformant
preprocessor, as the rest of clang no doubt has as its goal.

I have the opposite view. Having an MSVC-compatible preprocessor is an
excellent thing for interoperability on the platform.

Am MSVC-compatible preprocessor is a broken preprocessor. How can this be an excellent thing ? If the idea is to emulate VC++ in every respect, there is no point in using clang-cl. Really !

It increases the
likelihood of adopting clang as a native toolset on Windows, and
reduces the barrier to entry, because it allows me to take code which
compiles with MSVC and instead run it through clang.

Again there is no point in using clang if you are just going to emulate VC++. Is not the idea to produce a standard conforming compiler for people to use on their VC++ code which is better and cleaner than VC++ with all its bugs.

This functionality is behind a flag (-fms-compatibility), and it would
be a step backwards to say "please turn on compatibility with MSVC,
except for the preprocessor" when that flag is enabled. It would also
prevent using MSVC's STL (IIRC, <type_traits> relies on this behavior
in MSVC 2012 due to the compiler's lack of support for variadic
templates).

I would not be opposed to a more fine-grained flag that allowed you to
disable the quirks for the processor when -fms-compatibility is turned
on, akin to -fno-delayed-template-parsing.

So you want a default that produces the same bugs as VC++ and only a switch will produce a standard conforming compiler. I cannot tell you how amazing I find that and how wrong that is.

Honestly if that is the case, aka "let's just create a completely compatible compiler with VC++", I haven't the faintest idea why clang developers would have spent a second of their time doing that since anybody can use VC++ instead.

7

Being able to migrate large, existing code bases off of MSVC requires
compatibility with MSVC in practice, which is what drives me (and
likely quite a few others) to contribute to clang.

As Chandler has pointed out, this is a balancing act. We try to avoid
adding nonconforming changes to the compiler simply to be bug-for-bug
compatible with another compiler. But we don't shy away from adding
them when there are valid reasons to do so. This particular
compatibility piece is in place in order to migrate code using MSVC
2012's <type_traits> header. We add others to be compatible with gcc
from time to time as well.

~Aaron

8

    I have been told, via my bug report on a preprocessor error in the
    latest version of clang-cl.exe, that clang-cl.exe is trying to
    emulate at least some of the bugs of the VC++ preprocessor.

    NO ! Please do not do that.

    I know, from having worked on variadic macros in Boost PP and having
    created my own variadic macro data library, how broken the VC++
    preprocessor is when it comes to preprocessor metaprogramming. Paul
    Mensonides, the creator of Boost PP and an exceptional expert on the
    C++ preprocessor, also knows and has bemoaned countless times how
    difficult it is to do preprocessor metaprogramming if one has to
    deal with the VC++ preprocessor. Not only will clang-cl.exe's
    emulation of the brokeness of the VC++ preprocessor set back further
    efforts in metaprogramming preprocessor code and libraries but there
    is absolutely no chance, if it means anything to clang-cl
    developers, that any effort will be made to accomodate clang-cl's
    preprocessor bugs into Boost PP ( or my own variadic macro data
    library ).

    This is completely the wrong direction for clang-cl to go. Please
    stop and only consider producing as close to as possible a 100% C++
    conformant preprocessor, as the rest of clang no doubt has as its goal.

So, I feel like we're between a rock and a hard place here.

At a fundamental level there are bugs we need to be bug-for-bug
compatible with because there is large amounts of existing code in the
wild relying on those bugs.

At the same time, I'm quite sympathetic to not wanting to make things
less conforming.

In the case of the VC++ preprocessor you really have to be a masochist, and someone with little pride in correct C++, to want to emulate the bugs there.

But I don't see an easy way to have both. Here are my suggestions, but
none of them are great:

1) Make sure that any bug-compatibility is inspired by actual code
relying on the bug. I suspect it is in this case, but its always good to
check.

This is a fool's errand, sir. You have no idea of all the "existing code in the wild". Do you really think because someone has written a macro that relies on the non-conformance of the VC++ preprocessor that it is clang-cl's responsibility to reproduce that same preprocessing bug ? Or even a theoretical someone.

2) It may be desirable for Boost (and Boost users) to opt into a more
strict mode. I get the flags backwards all the time (Reid or someone
else will correct me) but we have two flags to control compatibility:
one for essentially conforming (or at least not horrible broken)
extensions, and another for bug-for-bug compatibility. I think the
latter is '-fms-compatibility' while the former is '-fms-extensions',
but again, i get them backwards. Anyways, perhaps Boost would want to
use '-fno-ms-compatibility' where possible to get a more conforming
implementation.

I do not ultimately control the Boost PP library, Paul Mensonides does although I am a contributor, but I can almost guarantee you that he has no inclination, and I know I have none, to make any changes to accomodate clang-cl's emulation of VC++ preprocessor bugs no matter what compatibility options and emulation of whatever VC++ preprocessor bugs clang-cl wants to promote. And if others are willing to go into that Boost PP code and hack around with it given the ridiculousness of clang-cl attempting to emulate the brokeness of the VC++ preprocessor in whatever respects, I wish them the best of luck through their hair-pulling and tears.

3) I think we should then make sure we expose macros you can use to
detect the bug-for-bug compat mode, and use the (horrible) cl.exe
implementations of things like Boost PP, even though the host compiler
is Clang.

Does this make sense?

Not to me. The only thing that makes sense, quite honestly, is that clang, and clang-cl, produce a C++ standard conforming preprocessor. Furthermore there is no way you will ever get Microsoft to fix their broken preprocessor if you think that emulating their own preprocessing bugs is the way to go. If you want to make a switch in clang-cl to produce a standard conforming preprocessor, rather than have it be the default, my message to anyone using Boost will be simply that without this switch there is no point in using Boost in their code. Of course if this does not matter they can do what they want.

Please, please reconsider the notion that you can produce a compiler worthy of clang, to be used by VC++ programmers, by reproducing VC++ preprocessor bugs. This is a realy bad path to take.

9

Since much of Boost is header only, the user controls the compile
flags, so that's not possible.

10

    I have been told, via my bug report on a preprocessor error in the
    latest version of clang-cl.exe, that clang-cl.exe is trying to
    emulate at least some of the bugs of the VC++ preprocessor.

    NO ! Please do not do that.

    I know, from having worked on variadic macros in Boost PP and having
    created my own variadic macro data library, how broken the VC++
    preprocessor is when it comes to preprocessor metaprogramming. Paul
    Mensonides, the creator of Boost PP and an exceptional expert on the
    C++ preprocessor, also knows and has bemoaned countless times how
    difficult it is to do preprocessor metaprogramming if one has to
    deal with the VC++ preprocessor. Not only will clang-cl.exe's
    emulation of the brokeness of the VC++ preprocessor set back further
    efforts in metaprogramming preprocessor code and libraries but there
    is absolutely no chance, if it means anything to clang-cl
    developers, that any effort will be made to accomodate clang-cl's
    preprocessor bugs into Boost PP ( or my own variadic macro data
    library ).

    This is completely the wrong direction for clang-cl to go. Please
    stop and only consider producing as close to as possible a 100% C++
    conformant preprocessor, as the rest of clang no doubt has as its
goal.

So, I feel like we're between a rock and a hard place here.

At a fundamental level there are bugs we need to be bug-for-bug
compatible with because there is large amounts of existing code in the
wild relying on those bugs.

At the same time, I'm quite sympathetic to not wanting to make things
less conforming.

In the case of the VC++ preprocessor you really have to be a masochist,
and someone with little pride in correct C++, to want to emulate the bugs
there.

I don't have much of an opinion on this topic (as long as clang-cl can
parse MS headers), but please drop the hyperbolic language and stick to
technical arguments.

11

_MSC_VER isn't useful for Boost here as it's also defined by older
versions of clang which don't emulate the preprocessor.

12

I have been told, via my bug report on a preprocessor error in the latest
version of clang-cl.exe, that clang-cl.exe is trying to emulate at least
some of the bugs of the VC++ preprocessor.

NO ! Please do not do that.

I know, from having worked on variadic macros in Boost PP and having
created
my own variadic macro data library, how broken the VC++ preprocessor is
when
it comes to preprocessor metaprogramming. Paul Mensonides, the creator of
Boost PP and an exceptional expert on the C++ preprocessor, also knows
and
has bemoaned countless times how difficult it is to do preprocessor
metaprogramming if one has to deal with the VC++ preprocessor. Not only
will
clang-cl.exe's emulation of the brokeness of the VC++ preprocessor set
back
further efforts in metaprogramming preprocessor code and libraries but
there
is absolutely no chance, if it means anything to clang-cl developers,
that
any effort will be made to accomodate clang-cl's preprocessor bugs into
Boost PP ( or my own variadic macro data library ).

This is completely the wrong direction for clang-cl to go. Please stop
and
only consider producing as close to as possible a 100% C++ conformant
preprocessor, as the rest of clang no doubt has as its goal.

I have the opposite view. Having an MSVC-compatible preprocessor is an
excellent thing for interoperability on the platform.

Am MSVC-compatible preprocessor is a broken preprocessor. How can this be an
excellent thing ? If the idea is to emulate VC++ in every respect, there is
no point in using clang-cl. Really !

It increases the
likelihood of adopting clang as a native toolset on Windows, and
reduces the barrier to entry, because it allows me to take code which
compiles with MSVC and instead run it through clang.

Again there is no point in using clang if you are just going to emulate
VC++. Is not the idea to produce a standard conforming compiler for people
to use on their VC++ code which is better and cleaner than VC++ with all its
bugs.

This functionality is behind a flag (-fms-compatibility), and it would
be a step backwards to say "please turn on compatibility with MSVC,
except for the preprocessor" when that flag is enabled. It would also
prevent using MSVC's STL (IIRC, <type_traits> relies on this behavior
in MSVC 2012 due to the compiler's lack of support for variadic
templates).

I would not be opposed to a more fine-grained flag that allowed you to
disable the quirks for the processor when -fms-compatibility is turned
on, akin to -fno-delayed-template-parsing.

So you want a default that produces the same bugs as VC++ and only a switch
will produce a standard conforming compiler. I cannot tell you how amazing I
find that and how wrong that is.

Honestly if that is the case, aka "let's just create a completely compatible
compiler with VC++", I haven't the faintest idea why clang developers would
have spent a second of their time doing that since anybody can use VC++
instead.

Being able to migrate large, existing code bases off of MSVC requires
compatibility with MSVC in practice, which is what drives me (and
likely quite a few others) to contribute to clang.

The problem with the VC++ preprocessor bugs is that there are many situations where VC++ simply does not work correctly doing macro expansion. Even figuring out why VC++ is incorrectly doing macro expansion is next to impossible unless you have their source code for their preprocessor, which I have my doubts that clang has. So now clang-cl wants to emulate, let's say, some of those buggy preprocessor macro expansions which VC++ does but clearly not all of them since it is impossible to figure out all of them. This now creates one more dialect of a buggy preprocessor for preprocessor metaprogrammers to deal with.

As Chandler has pointed out, this is a balancing act. We try to avoid
adding nonconforming changes to the compiler simply to be bug-for-bug
compatible with another compiler. But we don't shy away from adding
them when there are valid reasons to do so. This particular
compatibility piece is in place in order to migrate code using MSVC
2012's <type_traits> header. We add others to be compatible with gcc
from time to time as well.

Would you please point out to me what in the VC12 <type_traits> does not compile with a standard conforming C++ preprocessor ?

13

C'mon Edward, surely you're joking? Even if it emulates msvc bugs clang
still compiles faster, has better diagnostics, has better language support,
has static analyzer... There's a number of reasons for people to use it.

Nikola

14

It was the old faux-variadic macros which went away in 2013, I believe.
http://llvm.org/PR14981

15

Basically anything involving the _VARIADIC_EXPAND_XXX macros they used
in place of variadic templates.

http://llvm.org/bugs/show_bug.cgi?id=14981

~Aaron

16

Also it’s worth pointing out that the idea is not to emulate VC++ in every respect. Only in every respect where not doing so will break vast amounts of code. There are plenty of places where not emulating VC++ will fix vasts amounts of code. I imagine the goal is to depart with VC++ in those respects.

There are examples where not emulating VC++'s bugs means we can’t even include standard windows headers. Surely we agree that a Windows compiler that cannot include <windows.h> would not be particularly useful.

Any transition from using something broken to using something not broken involves a migration path. Years ago VC++ left for-loop variables in scope after the loop completed. The solution isn’t to simply turn fix it and break old code, it’s to provide a migration path behind a switch. It’s the same here. The goal is that if code compiles with VC++, it compiles with clang-cl. That’s the migration path.

17

I do not understand why clang gives the error in the bug report after looking at the macros in the VC++ files. Would someone enlighten me about this ? No doubt I have missed something concerning the expansion of the macros.

18

I can see that Boost Wave gives the same error as clang so I will study the Boost Wave output.

19

I've done a lot of preprocessor metaprogramming in the past and I am
extremely sympathetic here. A few years ago, I think in in the context of
trying to get Paul to get Chaos into the Boost review queue (one of several
times), one of his reasons for not doing so was because he did not want to
support broken preprocessors -- especially Microsoft. It still should get
in anyway, but that's another story.

I've never implemented a C preprocessor (thankfully), but one thing that
I've wondered about that may actually be feasible and may allow for a way
forward, is to have a pragma that allows you to toggle between the buggy
MS-style preprocessor and a compliant one. The idea is that when any macros
are defined while the buggy mode is on, those macros internally behave as
they would if defined in Microsoft's broken implementation whenever they
are expanded. Similarly, if the toggle is the other way (specifying
compliance), then everything works as expected in a compliant preprocessor
implementation. In other words, preprocessor definitions, depending on the
state of the toggle at the time they are defined, internally behave
differently upon expansion. If a compliant-mode definition invokes a
non-compliant macro, the MS-emulation only happens internal to the broken
macro's expansion. Similar-but-opposite behavior would be expected when the
composition of the macros is flipped (note that this would be less
important to support than the latter). Hypothetically there could also be a
way at the command-line to specify that certain include paths should always
have this broken preprocessor option on for the duration of the include,
whereas by default, other files do not (the default could be switched, but
I think this would be ideal).

The idea of all of this is that in the vast majority of cases (I have no
numbers on this, but I think it's a reasonable hunch) the only reason
people care about MS preprocessor emulation is because they need to
preprocess MS headers or other libraries that they, themselves, have not
written. I'd be surprised if a lot of people had preprocessor-heavy code
that is difficult to fix, but if they did, they could opt-in for the broken
preprocessor if they so chose. At the very least, libraries like
Boost.Preprocessor could use the pragma so that they don't need to
bend-over-backwards to try to support another implementation (clang's
preprocessor is already awesome in this respect).

Again, I have not thought through this very deeply and do not have the
experience here necessary to say whether this is actually feasible or not,
but I present it as an option hoping that someone out there who does have
the experience can at least consider it. The whole point of this is that
emulating MS's preprocessor bugs allows "proper" preprocessing of
broken-but-important headers and so its importance cannot be stressed
enough. On the other side, producing a sort-of-like-MS preprocessor that
everyone is stuck with just because of Windows headers is itself a horror.
I'm not entirely convinced that we can't somehow have our cake and eat it
too in this particular instance.

20

cl-clang will give microsoft the possibility of fixing the bug because
there will be more and more cl-clang builds that provide standard-conform versions
microsoft will never force people to switch to a better solution, but will always try to follow