Clang-cl.exe support for C++ modules

Clang Frontend
,
61

@sharadhr Here’s a full repro with CMake, with failing clang-cl command in the end. (checked with real Windows and MSVC, not under Linux/Wine).

CMakeLists.txt:

cmake_minimum_required(VERSION 3.29)
project(testrc)
add_library(testrc STATIC test.cpp test.rc)

test.cpp and test.rc are just empty files.

Building:

cmake -S . -B build -G Ninja -DCMAKE_C_COMPILER=clang-cl -DCMAKE_CXX_COMPILER=clang-cl
cmake --build build

With stock clang-cl.exe it succeeds, but when clang-cl.exe is replaced with the patched one (I prepend a folder with it to the PATH), it fails:

C:\test\testrc>cmake --build build
[2/3] Building RC object CMakeFiles\testrc.dir\test.rc.res
FAILED: CMakeFiles/testrc.dir/test.rc.res
C:/PROGRA~1/MICROS~2/2022/COMMUN~1/Common7/IDE/COMMON~1/MICROS~1/CMake/CMake/bin/cmcldeps.exe RC C:\test\testrc\test.rc CMakeFiles\testrc.dir\test.rc.res.d CMakeFiles\testrc.dir\test.rc.res "Note: including file: " "C:/test/testrc/bin/clang-cl.exe" C:\PROGRA~2\WI3CF2~1\10\bin\100226~1.0\x64\rc.exe   -DWIN32 -D_DEBUG /fo CMakeFiles\testrc.dir\test.rc.res C:\test\testrc\test.rc
error: unable to open output file 'CMakeFiles\testrc.dir\test.rc.res.obj': 'no such file or directory'
1 error generated.
ninja: build stopped: subcommand failed.

I tried to trace it, it seems for .rc files CMake calls its own program cmcldeps.exe (see source) with cmdline above which tries to generate a list of includes in .i file by calling clang-cl like this:

clang-cl.exe /P /DRC_INVOKED /nologo /showIncludes /TC -DWIN32 -FoCMakeFiles\testrc.dir\test.rc.res.obj C:\test\testrc\test.rc

When I try to run this command manually, it succeeds with the unpatched clang-cl and fails with the patched one.

1 Like
62

Hi, just wanted to report that I just successfully used the PRs for LLVM and CMake to build some C++ modules-based projects of mine (an installable library and a few projects dependent on it), as well as a bunch of third-party non-module dependencies. Specifically, I used the @sharadhr LLVM PR 121046 and CMake MR 9762 mentioned before in this thread. The original versions were LLVM 19.1.7 and CMake 3.31.6, and the patches were applied as is. See the commit to my Nix-based cross-building toolchain if interested.

The only thing that doesn’t work is RC compilation I reported earlier (so I had to patch out any mentions of .rc files from all the projects). Otherwise, this combination successfully builds real-world projects.

1 Like
63

It sounds really good. In the PR [clang-cl] Accept `cl`-style output arguments (`/Fo`, `-Fo`) for `--fmodule-output` by sharadhr · Pull Request #121046 · llvm/llvm-project · GitHub it looks close to land and @sharadhr may be away from the community for a while. And if you want, I think you can take it to finish it. It should be fine as long as you signed @sharadhr 's name.

64

I’m not very sure about that. There are still edge cases involving PCHs, and now RC compilation. I think it’s worth having a word with the CMake authors about how the flags are meant to be used before passing them in. And it would really help if I could have a word with the Clang-CL authors/maintainers so I can understand how I can fit this change in. It appears trivial at first glance but the driver translation unit is just very complex.

65

Depending on the PCH thing, CMake may not care. We don’t support modules when building PCH files. Basically, -include or -FI paths cannot introduce import statements. I don’t know if clang care, but CMake doesn’t (today). .rc compilation is probably more relevant.

66

Some summary observations, after thinking about this and reading every manual and relevant PR.

  1. --precompile and -fmodule-output are mutually exclusive, and they should be marked as such in the driver, regardless of driver mode or target.
    1. Hence, the above tests are axiomatically incorrect, as is CMake’s behaviour. It is unclear why CMake needs both flags, and this hasn’t been well explained.
  2. clang-cl can straightforwardly compile BMIs as XMake has demonstrated with clang-cl /std:c++20 --precompile /path/to/foo.cppm /clang:-o /clang:/path/to/foo.pcm. CMake is a little more wary and this particular invocation fails certain tests:
    1. Scanning for dependencies with precompiled headers,
    2. and RC files, as demonstrated above.
  3. cl.exe uses /Fo (or -Fo; in the driver, they are aliases) strictly for object files—that is, a.obj as a result of compiling a.c or a.cpp. clang-cl generally follows this behaviour.
  4. clang++ is much more loose with its usage of -o, with --precompile accepting it as a ‘sub-argument’ to produce built module interfaces (BMIs), that is, a.pcm. Hence, -o is not a good substitute for -Fo.

A suggestion was made to follow clang-cl’s command line, while understanding that clang-cl’s behaviour diverges from and is binary-incompatible with cl.exe’s behaviour.

I propose a middle-ground, an entirely new command-line argument: /pcmOutput, aliased to -pcmOutput that is paired with --precompile and allows the clang-cl driver to specify the output BMI file correctly, without conflicting with -o, and this will straightforwardly allow PCHs to be correctly managed while doing dependency scanning, as the -o argument will not be reused.

There will be no changes to the mechanism of -fmodule-output, which works as expected in clang-cl after my previous set of PRs.

67

A note for the peanut gallery that there is only proposed support for clang-cl in CMake at this time; nothing has been merged yet.

68

I have a small proof-of-concept for /pcmOutput here.

I recalled that the CMake MR changed all invocations of -o to /clang:-o, which I reverted, and at this point the PCH tests pass, but the RunCMake/examples/export-transitive-modules test and other related ones fail. I should probably have a look at those tests and understand why clang-scan-deps isn’t doing the right thing (which probably means clang-cl isn’t doing the right thing).

69

That may be because that is testing that the CMake collator gives the full transitive closure of modules (i.e., if a imports b which imports c, compiling a needs to know about c’s location even without a literal import c; in its own source. See the clang and msvc module map formats.

70

Cheers. Is the following minimum example equivalent:

// c.cppm
export module c;

export auto c() -> int
{
  return 3;
}

// b.cppm
export module b;

import c;

export auto b() -> int
{
  return 2 + c();
}

// a.cpp
import b;

auto main() -> int
{
  return 1 + b(); // returns 6
}

And what would be the appropriate clang-scan-deps call?

71

I can’t recall off the top of my head; I’d say to generate the test with clang and use ninja -v to see what it is doing. You can inspect all of the .ddi files in the build tree to see what scanning reports.

72

Right, I reworked the Windows-Clang.cmake compiler module and everything but the scan-with-pch test passes. I think CMake now needs to learn how to differentiate between flags; a simple find-replace will not suffice.

73

differentiate between flags

Could you please be more specific? Do you mean differences between cl and clang-cl?

74

Sorry, allow me to clarify: CMake needs to differentiate between /Fo and /pcmOutput (and also /Fo and -o in general).

75

A quick (if unfortunate) update.

I haven’t got the bandwidth to continue working on this, and I have personally moved back to MSVC on Windows. Sorry to anyone who was relying on this being merged; I simply haven’t had the time to contribute. I am not intimately familiar with the driver code, and iteration is really long. A good summary is here; this is the state of affairs now.

The problem appears straightforward—set up a new argument for Clang-CL to accept module units/PCMs/JSONs within the compiler driver and within clang-scan-deps (which uses the same driver mechanism).

Additionally this requires stakeholders (build systems, Clang driver developers) to come to a conclusion about expected behaviour for all the edge cases, before I as a user-developer can implement something.

76

Thanks for what you done. For better visibility, if possible, you can change the title to [help wanted] clang-cl.exe ...

77

@asb hi would you like to mention this in LLVM Weekly to make this more visible? This is about asking more helpers in help clang-cl with c++ modules. This is almost a driver thing.

78

I think I can close this thread as complete: it looks like both XMake and CMake are happy enough with Clang-CL, and the basic command-line interface works well enough.

79

Assuming your reply, the current state of the clang-cl, and that XMake and CMake have done support of their site, that will be valid to assume that any attempts to connect clang-cl to Visual Studio should be done by Microsoft (assuming, with an attempt to reopen an already closed issue)?

80

Indeed. At this point it is MSBuild that needs to wire up the support, and I have opened an issue in the tracker.

1 Like