Warnings are different on preprocessed file

Hi,

is this expected behavior?

cat preproc_warning.sh
#!/bin/bash

echo "-- direct compilation:"
clang++ -c preproc_warning.cpp -o /dev/null
echo "-- preprocessing:"
clang++ -E preproc_warning.cpp -o preproc_out.cpp
echo "-- compilation of preprocessed source:"
clang++ -c preproc_out.cpp -o /dev/null

cat preproc_warning.cpp
#include <stddef.h>

#define CHECK(a) ( a ? 1 : NULL)

int func1()
{
    int a;
    return NULL; // produces warning during direct compilation and compilation of preprocessed source
}

int func2()
{
    int a;
    return (CHECK(a)); // produces warning only on preprocessed source
}

shows this output:
-- direct compilation:
preproc_warning.cpp:8:12: warning: implicit conversion of NULL constant to 'int' [-Wnull-conversion]
    return NULL; // produces warning during direct compilation and compilation of preprocessed source
    ~~~~~~ ^~~~
           0
1 warning generated.
-- preprocessing:
-- compilation of preprocessed source:
preproc_warning.cpp:8:12: warning: implicit conversion of NULL constant to 'int' [-Wnull-conversion]
    return __null;
    ~~~~~~ ^~~~~~
           0
preproc_warning.cpp:14:23: warning: implicit conversion of NULL constant to 'int' [-Wnull-conversion]
    return (( a ? 1 : __null));
    ~~~~~~ ^~~~~~
                      0
2 warnings generated.

Best regards,
Martin

Hi,

is this expected behavior?

...

#define CHECK(a) ( a ? 1 : NULL)

int func1()
{
    int a;
    return NULL; // produces warning during direct compilation and
compilation of preprocessed source }

int func2()
{
    int a;
    return (CHECK(a)); // produces warning only on preprocessed source
}

http://llvm.org/viewvc/llvm-project?view=revision&revision=156861

That's not the only case when Clang suppresses a warning for something coming
from a macro, so a separate preprocessing step results in a warning that
wouldn't be otherwise reported. On the other hand, why would one actually
need a separate preprocessing step for normal usage in practice anyway.

http://llvm.org/viewvc/llvm-project?view=revision&revision=156861

That's not the only case when Clang suppresses a warning for something coming
from a macro, so a separate preprocessing step results in a warning that
wouldn't be otherwise reported. On the other hand, why would one actually
need a separate preprocessing step for normal usage in practice anyway.

Thanks for the info. I agree that preprocessing is usually not done
separately, but distributed compile clusters like icecream rely on it
(preprocess locally, compile remotely).

So it would be great to have consistent warning behavior. I don't
want to disable the warning, are there any other workarounds?

Best regards,
Martin

>http://llvm.org/viewvc/llvm-project?view=revision&revision=156861
>
> That's not the only case when Clang suppresses a warning for something
> coming from a macro, so a separate preprocessing step results in a
> warning that wouldn't be otherwise reported. On the other hand, why would
> one actually need a separate preprocessing step for normal usage in
> practice anyway.

Thanks for the info. I agree that preprocessing is usually not done
separately, but distributed compile clusters like icecream rely on it
(preprocess locally, compile remotely).

They actually don't, icecream uses -frewrite-includes and does "normal"
compile on the remote host if the version is recent enough ('icecc --help'
should mention ICECC_CLANG_REMOTE_CPP, v1.0 should do, although the newer
icecream and clang the better, given that even current clang still has
minor -frewrite-include bugs pending). My current icecream and clang builds
manage to have identical stderr output in the local and remote case for
anything I've tried.

If you use ccache and have similar problems because of preprocessing, use
CCACHE_CPP2=1.

So it would be great to have consistent warning behavior. I don't
want to disable the warning, are there any other workarounds?

I don't think there's any other than not using a separate preprocessing step.
I was looking into that when making icecream work well with clang, and it's
pretty much a lost fight. Even if you managed to get all the warnings right,
which is questionable if that's possible, you still would have problems e.g.
with caret diagnostics, as they would report preprocessed input in messages.
That could be worked around by using linemarks and reading the original
source file (as gcc-4.8+ does), but that doesn't work with the remote
icecream case (to-be-v1.1 icecream "supports" gcc caret diagnostics by
compiling locally again if it detects the need).

So if you still have valid use case for separate preprocessing step, I think
a better chance is with finding a workaround that avoids that.

Hi Lubos.

Thanks for your reply and your explanations and sorry for my late reply. I
am a colleague of Martin Richtarsky, and actually I am the one who asked
Martin to report this bug.

First, I would like to give you some more background. We have implemented
our own compile cluster for several reasons (better performance, higher
reliability, integrated distributed object files cache, remote cluster
administration, support for Windows builds, etc.). However, in order to
provide these cluster features (the cache features in particular and support
for Windows), the preprocessing step is absolutely mandatory!

I would like to add two thoughts regarding this bug report:

1) One might discuss, whether the compiler should emit or skip warnings, if
these warnings only occur after macro expansion. My personal opinion is,
that I would like to see really all pieces of code that cause warnings, even
if these warnings only occur after macro expansion. When implementing or
using macros, I would like to really know, if these macros are
"warning-free". However, experience shows that discussions about such topics
quickly turn out to be "religious discussions", some love it, some hate it,
and the best way is to satisfy both sides by giving the freedom of choice.
Therefore I guess, a great clang feature would be to add a new compiler
option like "-Wstrict-macro-warnings" or similar, that controls the
compiler's behavior in that case.

2) However, warning behavior after macro expansion isn't really the topic of
this bug report. The key topic here is, that compiling a CPP file directly
gives different warnings than first preprocessing CPP file and then
compiling preprocessed file. I've been working with many different C++
compilers, and I cannot remember any compiler behaving that way. All
compilers, I am aware of, do NOT emit different warnings/errors, when
compiling directly vs. compiling preprocessed file. Furthermore, C++
standard only talks about "preprocessing tokens", which indicates that
preprocessing is a purely lexical operation and shouldn't have any impact on
compiler semantics. I couldn't find any hint in C++ standard indicating that
preprocessing would potentially modify compiler semantics. Therefore, I
would highly appreciate, if you could find a way to fix this issue. If the
suggestion in (1) can be easily implemented (additional compiler flag for
controlling the behavior), that would probably be a fix for the topic
discussed in (2).

Thanks + best regards,
Volker

Hi Lubos.

Thanks for your reply and your explanations and sorry for my late reply. I
am a colleague of Martin Richtarsky, and actually I am the one who asked
Martin to report this bug.

First, I would like to give you some more background. We have implemented
our own compile cluster for several reasons (better performance, higher
reliability, integrated distributed object files cache, remote cluster
administration, support for Windows builds, etc.). However, in order to
provide these cluster features (the cache features in particular and
support for Windows), the preprocessing step is absolutely mandatory!

Given that -frewrite-includes output is basically all the input the
compilation will get, and that ccache can work with clang too, I somewhat
doubt that, but it's your code, so if you insist.

I would like to add two thoughts regarding this bug report:

1) One might discuss, whether the compiler should emit or skip warnings, if
these warnings only occur after macro expansion.

...

2) However, warning behavior after macro expansion isn't really the topic
of this bug report. The key topic here is, that compiling a CPP file
directly gives different warnings than first preprocessing CPP file and
then compiling preprocessed file.

I'm not a Clang developer, so you don't need to convince me. And guessing
from the lack of any other comments, I'd say your convicing will need to take
the form of patches.

Hi,

is this expected behavior?

To answer this original question: Yes, it is expected behavior that
preprocessing source and compiling it does not produce the same
results as compiling the preprocessed source directly.

Certain warnings produce (too many) false positives without taking
into account whether they were expanded from a macro, etc.

Take, for example:

#define FUNC_TEST(arg) (func(arg) ? GOOD : BAD)

Perhaps this macro is the idiomatic way to call 'func' in this
library? (ie: 'func' is an internal name, only ever meant to be
accessed via the macro)

Sometimes the user wants the result:

if (FUNC_TEST(42) == GOOD) { ... }

but sometimes it's not needed:

FUNC_TEST(42);

Which all seems fine. But if the user explicitly wrote:

func(arg) ? GOOD : BAD;

we would like to warn the user that they have unused-values there?
(Why would anyone /ever/ write that code? The whole conditional
operator and the values GOOD and BAD are discarded, the user might as
well just write "func(arg);")

So using the presence/absence of a macro helps diagnose the bad cases
while not needlessly warning on reasonable cases.

To your later comment about "no other compiler does this" - I suspect
some do, but also one of Clang's engineering novelties is that it
keeps track of all this macro expansion information so it /can/ reason
about warnings using this information. Many other compilers don't have
this information available to make such choices about when to emit and
suppress warnings.

- David

Hi David.
Great to hear that clang development reserves it&#39;s right to be different and to ignore objections regarding C++ standard Let&#39;s wait and see to where this policy leeds. I have enough professional experience to know what will come out of this attitude.
BR
Volker

  Hi David.
Great to hear that clang development reserves it&#39;s right to be
different and to ignore objections regarding C++ standard Let&#39;s wait
and see to where this policy leeds. I have enough professional experience
to know what will come out of this attitude.

Sorry, but you've lost a lot of credibility through this message. We do
care deeply about following the relevant language standards, but:
1) This warning is not part of any standard we follow,
2) The -E flag (or, indeed, any preprocessing-only mode) is not part of
the C++ standard (but is part of POSIX),
3) Given our desire to minimize false positives from our warning, it's our
judgement that our users are best served by suppressing the warning in
certain cases involving macros, and
4) Given the behavior of -E (as defined by POSIX), it must expand macros
by default.

If you're going to be passing the result of -E back to Clang or another
compiler, use -frewrite-includes to avoid problems such as this one --
that's the very reason why we have that flag, and other distributed build
systems are using this to great effect.

BR

Well, the string &quot;preprocess&quot; occurrs roughly 200 times in C++ standard, therefore I doubt that preprocessing is &quot;not part of the C++ standard&quot;.

True enough, C++ standard is not always very clear about preprocessing and leaves much space for interpretation. However, as already mentioned, C++ standard only talks about &quot;preprocessing tokens&quot;, what (in my understanding) suggests, that preprocessing is a purely lexical operation and shouldn&#39;t have any impact on semantics (as is the case with clang).

I never received a response to my question, from which statement of C++ standard (or any other standards for that matter) clang developers come to the conclusion, that preprocessing might have impact on compiler&#39;s warning semantics.

Furthermore, if C++ standard (or any other standard) is &quot;fuzzy&quot; and leaves room for interpretation, I would expect an implementation of that standard to consider the ambiguity and provide options / switches for enforcing this or that interpretation. Hey,&nbsp; I only asked for a compiler switch in order to make clang warnings in case of compilation of preprocessed files compatible with clang&#39;s warnings in case of direct compilation of source files. I did not ask for changing clang&#39;s default behavior, nor did I ask for any incompatible changes, nor did I question the reasons, why clang team implemented it the way, it currently is. But again, I didn&#39;t get any response to my question, and I don&#39;t get the point yet, why such a compiler switch would be a &quot;bad thing&quot; for clang.

If someone told me, that it&#39;s technically not possible or too much effort for this or that technical reason, I would (have to) live with it, but simply saying &quot;No&quot; on the ground that someone considers her/his interpretation of C++ standard superior to other people&#39;s interpretation doesn&#39;t seem to be a good solution (at least to me).

And finally... Software, that is not flexible enough to adapt to different user&#39;s requirements, will not survive long-term. That&#39;s my conviction and part of my professional experience.

&nbsp;