[RFC] Extensions to export macros/(preprocessor states) for C++20 modules

[ChuanqiXu]

Update in 2025-3-21

See [RFC] Extensions to export macros/(preprocessor states) for C++20 modules - #62 by ChuanqiXu for the updated proposal.

---

Abstract

C++20 modules was designed to not leak macro definitions (or preprocessor states). But clang’s implementation limits users to use C++20 modules for existing and complex projects in the real world efficiently. (See discussion below). To ease such limitations, we implemented an extension to export macros/(preprocessor states) for C++20 modules in the downstream. It works very well. It helped us to adopt C++20 modules in existing projects with over 7M LoC. It helped our products to C++20 modules in scale stably for more than 3 years. Our experience using this extension makes us believe that exporting macros for C++20 modules is very helpful.

The problems

I concluded the challenges, from implementor’s perspective, to use modules without exporting macros/preprocessors as the following problems:

1. Force user to introduce duplicated declarations between module units.
2. The import-before-include problem.
3. Handling 3rd party libraries.
4. The source location space.

The problem [1] is the key point. We can think problem[2] is a problem of QoI. The problem [3] is a result of problem[1] and problem [2]. The problem [4] is already existing but C++20 modules make it more explicit.

I heard MSVC and GCC has similar problems on [2]. But I am not sure if they have the same reasons. I’ll focus on clang here.

Problem 1: Duplicated declarations between module units

The idea to use macros in modules is easy: if you want it, include the corresponding header in GMF.

But in practice, many existing library may define macros with other declarations in the same header. This implies when we include a header to introduce a macro, we might probably introduced the declarations we don’t intended introduce. Then we might not be in an optimal state (see Standard C++ Modules — Clang 22.0.0git documentation for details). We wasted space and we will pay more at compilation time due to our logics to handle redeclarations.

This is unfortunate especially we’re dealing with 3rd party libraries that we can’t touch really.

While we can argue all other problems here are QoI, I do think this issue is the key point that we need to export macros. Otherwise, the modules won’t get best performance. Although modules have many other advantages, the compilation performance is the killing point.

Problem 2: The import-before-include problem

The problem is, the compiler may not be able to handle the following code well:

import a; // containing #include <a.h> directly or indirectly.
#include <a.h>

There were a lot such issue reports. I’ve fixed some of them. But I never say the issue is fixed. Since I think the issue may be a long standing one.

The logic is, when we inlcude-before-import, or we have multiple import contain the same declaration, we need to handle the declaration merging in ASTReader. But when we import-before-include, we would handle the declaration merging in Sema. So we’re doing 2 similar things in 2 different path. So we might do well in ASTReader but we might not do well in Sema due to the using practice in vendors.

So I assume the problem exist if we still have such divergence.

By exporting macro/preprocessor states to C++20 modules, we can avoid such redeclarations naturally.

Problem 3: Wrapping 3rd party libraries

Another issue to use modules in complex real world project is, the 3rd party libraries. Then it will be a problem if these 3rd party libraries doesn’t provide a module.

If we wrap them simply, like:

module;
#include <thirdparty>
export module thirdparty;

...

It will introduce declarations not belong to thirdparty to the module file of third party. At least, generally, all libraries will dependent on std module. Simply, we will meet the problem[1], the duplicated declaration in different module units.

With exporting macros, we can wrapping third party modules super easily in the most efficient way:

module;
import std;
#include <thirdparty>
export module thirdparty;

...

And also we can introduce an export-all extension so that we don’t need to write export using ... for third party declarations, if we want. That is much easier than this. (while this extension is already simple enough)

Problem 4: the source location space

Currently we use 32 bits to represent the source locations in the current compilation. One bit is used to represent the location are valid/invalid. So we have 31 bits, about 2G space for the source locations in a compilation.

And with modules, the problem is, the same header included by different module units, have different entry values in different module module, then when we imported these module files, the source locations for the same header will occupy N times than before (N is the number of imported modules includes the header). Then it is much easier than before to reach the hard limitations.

By exporting macros, we can avoid a lot of unnecessary include. So that we have more chance to not face the limitations.

It is another topic to increase the limitation though.

Implementation and Maintainance cost

Clang header modules, header units and C++20 modules share the same serialization/deserialization framework. Given both clang headers modules and header units can export macros, it should be natural to expect that C++20 modules can share macro from the same framework easily.

And also, given the preprocessor system is pretty stable, the maintainance cost is expected to be pretty low.

In fact, in the downstream, after I made it in early 2022, I rarely maintain it and it works consistently well.

This is important. There are a lot of things good. But we can’t make all of them due to the implementation pressure. And if we can have a pretty simple solution that works well, let’s adopt it.

Impacts

As an extension, it must be opt-in. It won’t affect anything if this is not enabled explicitly.

If this is enabled, it will affect:

1. Build systems (tools) scanning.
2. Preprocess

For scanning, this is the major concern for the past few years when we discuss whether or not export macros from named modules.

The question is:

// a.cc
import A;
#ifdef M
import B;
#endif

If A is allowed to export macros, how can the build system to decide whether or not the current TU may dependent on module B?

The pre-assumption is, the CURRENT scanning model used by build systems now can scan all source files in parallel. Then we can find the ability to export macros from named modules breaks the above model. Although technically we can solve it, there were deep concerns about the scanning performance.

Then what’s the proposed solution here? None. In fact, we just ignored the issue. And it worked very well in practice. Here are some thoughts:

1. The problem has a pretty clear boundary that users can understand well. The new implicit rule is simple. And in downstream, we’ve already banned to put import in headers and #if blocks. (e.g., we can get import-relation ship by a grep command)
2. In the worst case, the exported macro affects dependency relationship, it is very likely to be a compilation error. The chance to make it a runtime error should be pretty rare in practice.
3. It leaves space for tools to progress later. My feeling for the states of modules over the years is: this is a big (not precise) egg-and-chicken problem. Different parts of the ecosystem are waiting for each other to take the first step. By implementing the extension, the tools can have a chance to experiment the new scanning model in practice instead of imaging with a paper.

My conclusion for the problem is, this is a real problem but not a blocking one.

For preprocessor problem, we can illustrate it as:

// A.cppm
export module A;
#define A_VALUE = 43

// a.cc
import A;
int func() {
   return A_VALUE;
}

then what is the expected preprocessing output for a.cc? Previously, we can preprocess a.cc without building A.cppm first. But after this extension, we have to build A.cppm first to preprocess a.cc.

Is this really bad? I feel it might not be the case as I realize it in the first place. Since after introducing modules, the TUs have dependencies on each other naturally. All tools that need to deal with semantics have to deal with the dependency somehow. And now we just add preprocess to these classes.

Header Units

Readers may feel that this extension take the place of header units. It is true for the Ideal header units. But not true for header units today.

Except for the lack support from build systems, the key problem is still the duplicated declarations from different units. The problem is:

// a.h
#include "base.h"

// b.h
#include "base.h"

// a.cc
import "a.h"
import "b.h"

...

(If you know the implicit transition of header include to header import, ignore it temporarily)

And let’s compile it with

clang++ a.h ... -o a.pcm
clang++ b.h ... -o b.pcm
clang++ a.cc -fmodule-file=a.pcm -fmodule-file=b.pcm ...

In the current implementation today, the a.pcm and b.pcm may contain duplicated declaration from base.h, it wastes space. In the last compilation, the compiler may also waste time to deal with redeclarations from different .pcm files. As we stated before, this is inefficient.

In the dream of header units in early days, we can solve the above problem by an action called “implicit transition”. It means, when the vendor see an inclusion, the vendor is allowed to treat the inclusion as if an import. The condition to do the transition is unspecified.

So for the above example, the build system were expected to build .pcm file for base.h and pass base.pcm to the compilation of a.pcm and b.pcm and hope the compiler to can transit the inclusion of base.h to base.pcm.

But now, the build system don’t build base.pcm for base.h. (In fact, there are rare build systems support header units) and the compiler won’t transit inclusion of base.h to base.pcm in headers. Also there are many unexplored areas and possible questions. e.g, what if we can’t use single .pcm file to present a header in different context. It seems complex. And I believe it is a long way to go.

And if we wish compilers to do all these jobs, we fall back to the discussion of implicit clang header modules VS explicit clang header modules. It seems there are different voices. But it seems not ended and complex.

After all, this thread is not about header units. I want to say the path to land header units is much longer. Personally, from implementor’s point of view, it is even better to ignore header units, to make the community to be more focus, given the resources are really limited.

User interface

Initially I want to make things simple. So I’ll send what we implemented and used. We can make more complex construct later.

The proposed interface consists of the producer side and the consumer side.

In the producer side, now we can offer an option -fmodules-export-macros for importable module unit to emit macros. Or we can add a pragma #pragma clang module "export-macros" in the source code.

In future, maybe we can introduce construct like #pragma clang module "export-macro-begin" and #pragma clang module "export-macro-end" to export macros conditionally.

In the consumer side, now we can use an option like -try-load-module-file-when-preprocess. Without this, the compiler can save a lookup-and-load process when the option is not enabled.

In the future, maybe we can have some construct like:

import A [[clang::import_macro]]; // import macro from A;
import B; // Don't import macro from B.

Summary

Without exporting macros for C++20 modules, it is hard to use C++20 modules in real world project efficiently. We proposed an extension to address this. The proposed extension have been implemented and used in production over years. We strongly believe this will help users of modules and potential users of modules significantly.

[ben.boeckel]

See gdr’s P3041R0. This offers a way to tell the build system that a header is implemented by importing a module and giving sidecar macros through some -include-like mechanism that injects macros at the import of the relevant header. But it works through importing headers, not named modules. This I can support building (because scanning has support for finding the macros).

Given that [[clang::import_macro]] is just asking for:

#ifdef from_A
import B;
#endif

which is unbuildable in general, I’m hesitant to support such a thing.

@ChuanqiXu

The pre-assumption is, the CURRENT scanning model used by build systems now can scan all source files in parallel. Then we can find the ability to export macros from named modules breaks the above model. Although technically we can solve it, there were deep concerns about the scanning performance.

I doubt the “technically we can solve it” without dropping support for static build graph tools (namely ninja). I cannot write a build.ninja file that can do this because I need to scan before I can scan to make sure that A’s BMI is available before I scan a.cc. If B then also has macros that may make C available, I need to schedule a second wait for the final scan. Ninja does not support such dynamic node generation and upstream refuses to do so (for reasonable reasons given its goals).

[ChuanqiXu]

https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2023/p3041r0.pdf only aims for standard headers and it doesn’t address the requirement for macro generally. And also it lacks an implementation.

Given that [[clang::import_macro]] is just asking for:

#ifdef from_A
import B;
#endif

which is unbuildable in general, I’m hesitant to support such a thing.

I guess you misunderstand it. This only works for the consumer side. That means the compiler may load the module in preprocessor time. It has nothing to do with the build system.

I doubt the “technically we can solve it” without dropping support for static build graph tools (namely ninja ).

I saw ‘static build graph’ as a limitation here. But this is not the targetted topic of thread.

[cor3ntin]

I’m not in favor of a general solution to exporting/importing macro because it’s pretty antithetical to the goals of modules, and macro isolations do solve important problems and are generally an important property to preserve long term, despite adoption issues in the next few years.

(if we wanted to export/import macros, I think it would have to be on a per-macro basis on the exporter side)

However, I think the problems you describe are real and should be addressed.
So, I think we should find a solution that addresses exactly the header guard use case.

In particular, clang is pretty good at recognizing header guards, so a solution that would export
a list of header guards that would maybe be an improvement.
Hopefully, we could find ways to limit where these are available so as to limit the risk of overriding an identifier declared by the importing module.

But that brings additional questions: what about #pragma once?
Should we try to address that?
And if so, should we instead try to serialize the list of included files somehow?

Exploring a solution in that direction would hopefully be easier to adopt than what you propose, as we
would not need an attribute (a flag to enable/disable that behavior should be sufficient)

[AaronBallman]

@cor3ntin

I’m not in favor of a general solution to exporting/importing macro because it’s pretty antithetical to the goals of modules,

I’m on the opposite side of the spectrum here. In order for modules to be adoptable by system headers (the one use case you would imagine would be best-suited for using modules!), there needs to be a way to export macros from a module.

The current approach here is to export all macros and import all macros, which is a good start IMO. That said, it may be nice to have more granularity like the pragma you mention, so that you don’t export macros like header guards, visibility or other function signature helpers, etc. But that seems like an extension to the proposed extension which can wait for later.

[Bigcheese]

I strongly oppose exporting macros from named modules. We had years of discussion about this in committee, and the design we ended up with for named modules make it very hard to do. The biggest issue for me is that it breaks scanning. If we had adopted Richard’s proposal to have name modules be referenced by file path this would be doable, but them not being tied to the path makes this either not be scannable in general, or have very weird behavior where macros are ignored sometimes.

Header units are the intended solution here. They solve 1, 2, 3, and 4. The opposition to them is leading to solutions like this that throw out a fundamental design principle of named modules.

@ChuanqiXu

And with modules, the problem is, the same header included by different module units, have different entry values in different module module, then when we imported these module files, the source locations for the same header will occupy N times than before (N is the number of imported modules includes the header). Then it is much easier than before to reach the hard limitations.

I think there are reasonable ways to handle this issue without growing the the size of SourceLocation, particularly for headers included in the GMF. We can sparsely represent the source location space for headers that have most of their AST content pruned.

@AaronBallman

I’m on the opposite side of the spectrum here. In order for modules to be adoptable by system headers (the one use case you would imagine would be best-suited for using modules!), there needs to be a way to export macros from a module.

Why would we want or expect to be able to turn a system header written in C into a C++ name module? I think there’s a disconnect here, as I view system headers (Lib C + POSIX + etc.) as kind of the worst case to turn into named modules. They use macros all over the place, including as input, and C doesn’t have the features needed to get rid of most of the macros.

We’ve modularized our entire set of system headers using Clang modules (which are equivalent to header units), and they are a significantly better fit.

[ChuanqiXu]

@Bigcheese

I strongly oppose exporting macros from named modules. We had years of discussion about this in committee

But we’re not discussing this in the committee. We’re trying to make it an extension from REAL and HEAVY users.

BTW, not a complaint, I know all people replied here are part of the committee and I can understand you guys to visit things from the committee’s perspective. BUT, I do think we need different LEVEL strictness for the design places. For an extension, I think it is fine if:

• There are real needs.
• The user interface design is clear enough that users can understand it.
• The implementation is fine.
• The maintainance cost is acceptable.

Then I think this is a fine proposal. It makes me unhappy that when I am providing the feedbacks and solution from REAL and LARGE SCALE using experience, people simply told me a decision made 5 years ago without enough implementation and user experience.

I am not saying something must be accepted since somebody is a user. But it is indeed frustrating that some suggestions got rejected simply by “the committee decided” while there lacks complex example using standard modules in the real world. Even if I am not proposing a paper to the committee.

@Bigcheese

The biggest issue for me is that it breaks scanning.

Yes. But as I mentioned, I don’t see this as a blocking issue. And our practice shows it doesn’t matter a lot actually.

@Bigcheese

Header units are the intended solution here.

But nobody is implementing it. I don’t want to show papers to end users and tell them to wait.

@Bigcheese

I think there are reasonable ways to handle this issue without growing the the size of SourceLocation, particularly for headers included in the GMF. We can sparsely represent the source location space for headers that have most of their AST content pruned.

Yes, this is actually a different issue. It just get triggered by the use of modules. I believe there are solutions. But I don’t think this would block the proposal.

@Bigcheese

Why would we want or expect to be able to turn a system header written in C into a C++ name module? I think there’s a disconnect here, as I view system headers (Lib C + POSIX + etc.) as kind of the worst case to turn into named modules. They use macros all over the place, including as input, and C doesn’t have the features needed to get rid of most of the macros.

While C decls are simple, technically the redeclrations in system header may affect the performance as well.

@Bigcheese

We’ve modularized our entire set of system headers using Clang modules (which are equivalent to header units), and they are a significantly better fit.

I believe it is farer story than the proposal to export macro/preprocessor state.

[ChuanqiXu]

@cor3ntin

I’m not in favor of a general solution to exporting/importing macro because it’s pretty antithetical to the goals of modules, and macro isolations do solve important problems and are generally an important property to preserve long term, despite adoption issues in the next few years.

I can understand your point of view. But I don’t feel the proposal is so devil. I am not proposing it to be a standard feature.

@cor3ntin

However, I think the problems you describe are real and should be addressed.
So, I think we should find a solution that addresses exactly the header guard use case.

In particular, clang is pretty good at recognizing header guards, so a solution that would export
a list of header guards that would maybe be an improvement.
Hopefully, we could find ways to limit where these are available so as to limit the risk of overriding an identifier declared by the importing module.

It is easy to export the header guard macro only. But I still want to emphasize the need for Macro generally are real.

@cor3ntin

But that brings additional questions: what about #pragma once?
Should we try to address that?
And if so, should we instead try to serialize the list of included files somehow?

I addressed #pragma once actually. I mentioned it as preprocessor state. We don’t need to serialize the list of included files but the preprocessed states. Luckily this is already handled by PCH/Clang Header Modules.

[ChuanqiXu]

@AaronBallman

The current approach here is to export all macros and import all macros, which is a good start IMO. That said, it may be nice to have more granularity like the pragma you mention, so that you don’t export macros like header guards, visibility or other function signature helpers, etc. But that seems like an extension to the proposed extension which can wait for later.

Yes! An important assumption in my mind is, we (as tool vendors) should believe our users are able to understand what we offers. The user interface is clear and I believe users are able to have sense for it. After all, this must be opt-in. The this may only be used by users who can understand what they are doing.

[ChuanqiXu]

BTW, I want to clarify the design a little more so that readers don’t need to be scared. I called it a two-way design. The (potential) macro provided by other modules won’t affect your code at all if you don’t enable it explicitly. For example, now you need to import a module provided by other groups and you don’t want to be affected by the non standard behavior. Then nothing will happen if you don’t compile your code with -try-load-modules-when-preprocessing no matter if macros are exported or not from that module.

So I believe this is not devil.

[Bigcheese]

I’ll expand on this once I’ve had a bit more sleep, but I have an idea that resolves my concerns and I think gives you what you want. It’s also completely standard conforming, not even an extension.

Header search is implementation defined, thus I propose the following:

import <|mod>;

This has the effect of synthesizing a header containing:

import mod;
#define blah 1
#define other_blah 2

And imports it as if it were a header unit. Note that it doesn’t actually need to be a header unit from Clang’s perspective, your existing implementation is probably fine.

This has some properties that I really like:

• It’s just implementation defined, there’s no extension here.
• It keeps the syntax we already have for “import that brings in macros”.
• Scanners can now be aware of it, and either reject if they can’t find it, or just ignore it and potentially warn if some other dependency is macro dependent.
• Users now know which imports bring in macros and which don’t. They also get to choose if macros are brought in on a per import basis using existing syntax.
• We can use the same syntax for Clang modules to import by name.
• This is implementable in other compilers with minimal compiler changes (just needs to support the syntax) by using real header units. So significantly less concern about dialects.

Some potential downsides:

• The syntax is a little odd, maybe some other chars are better. h-char and q-char can be anything in the basic character set except for \n, and the closing > or ".
• You may need to know and think about which modules need to be imported this way, but I actually think that’s good.

[ChuanqiXu]

This is interesting. I didn’t know if it is standard conforming. Before discussing this, I still feel my original proposal is good. It is simple. It is simple to use, to understand, to implement and to maintain. I don’t feel it is bad for vendors and for users.

For your proposed solution, I don’t know what’s the different between mine. Your concern was about scanning. But I am not sure how it solves the following problem:

import <|mod>;

#ifdef A
import ...;
#endif

Are you saying if the scanner saw a conditional import/include after import <|mod;>, the scanner can emit a warning? If yes, I think I got your point. But I don’t think it is better. I do feel the extension is more clear and neater.

And beyond the details, I do feel the syntax doesn’t look good… You said it is standard conforming. I didn’t check. Let’s assume it. I still feel the so-called standard conforming method is more like an weird “extension”.

• Users now know which imports bring in macros and which don’t. They also get to choose if macros are brought in on a per import basis using existing syntax.

Users can have the choice to receive macros or not with the current proposal. For choosing some specific module to load to get the macros, it is doable by the proposed attribute too.

[AaronBallman]

@ChuanqiXu

@Bigcheese

I strongly oppose exporting macros from named modules. We had years of discussion about this in committee

But we’re not discussing this in the committee. We’re trying to make it an extension from REAL and HEAVY users.

+1, this is a proposal about extending our implementation based on user needs. The committee is free to do as they will. If our extension is going to step on a design the committee is considering, that’s something we should definitely try to avoid, but I don’t believe that’s the case with this proposal.

WG21 has no actual users, we do. That’s why we are free to solve user issues the committee has not addressed as a matter of QoI. This appears to be a conforming extension.

@ChuanqiXu

While C decls are simple, technically the redeclrations in system header may affect the performance as well.

Exactly this! System headers are included incredibly often (the STL depends on system headers, but so does most user code: Windows headers, Mac OS headers, etc). Having the ability to modularize these headers can lead to significant compile time performance improvements. Some of these headers are huge once you bring in transitive includes (consider Windows.h), so even being filled with C declarations, they are not a trivial amount of parse time.

@Bigcheese

We’ve modularized our entire set of system headers using Clang modules (which are equivalent to header units), and they are a significantly better fit.

That’s circular logic, IMO. They’re a better fit because C++ modules don’t support the use cases outside of header units (theoretically – AFAIK, nobody has managed to implement them to the standard’s experimental ideal). This proposal is a step forward towards supporting those use cases, which moves C++ modules closer to being a better fit.

@Bigcheese

I’ll expand on this once I’ve had a bit more sleep, but I have an idea that resolves my concerns and I think gives you what you want. It’s also completely standard conforming, not even an extension.

Thanks, I think this is an interesting idea worth exploring (implementation-defined is certainly an improvement over an extension if it solves our needs).

@Bigcheese

Users now know which imports bring in macros and which don’t. They also get to choose if macros are brought in on a per import basis using existing syntax.

I especially like this aspect of the design.

[cor3ntin]

@Bigcheese

Header units are the intended solution here. They solve 1, 2, 3, and 4. The opposition to them is leading to solutions like this that throw out a fundamental design principle of named modules.

What is the status of importable header units in clang?
(As an aside, it would be nice to have a status report on the current state of modules somewhere users can find, I’m not sure folks have a lot of visibility into what is or isn’t supported)

[tahonermann]

@cor3ntin

What is the status of importable header units in clang?

I think it would be helpful to state somewhere how C++20 header units and Clang modules differ. @Bigcheese stated that Clang modules are equivalent to header units. My understanding is that there are some minor differences, but that the goal is for those differences to go away. Unfortunately, I don’t have a list of such differences.

I’m confused by statements that effectively state that nobody is working on C++20 header units when Clang clearly supports Clang modules. I suspect what is meant is that there hasn’t been a lot of progress in making module scanning based build systems work with C++20 header units. Thus, the situation is that implicitly built Clang modules work just fine as C++20 header units (disregarding concerns related to multiple implicitly built variants), but that work is still needed to enable deployment of build systems that use explicitly built C++20 header units. Is that a fair summary?

[jyknight]

I also have some trouble understanding the rationale here.

The standard already contemplates the requirement to use macros from modules, which is why it supports header units. So, the obvious answer if you wanted to import arbitrary macros is simply to use header units. We don’t need an extension for that.

Then, moving on to the question of duplicated declarations, and exporting header-guard macros, explicitly, to solve that. But I can’t see how this proposal really solves it. It can solve it in the case, only, where the #include appears after an import that exports the macros. But, that’s partial – the import std could very well occur after the #include, and we cannot have a module built differently depending on the preprocessor state at import.

Regarding header-units, Clang does appear to have support for them. (I haven’t used them seriously, so there could be something wrong with the implementation of course.) The example in the documentation shows that it will also translate a #include into an import when you’ve created a header-unit for a given header.

That seems like it already fully solves this issue – it ensures that all uses of the module-unit header are imported modularly. Thus, it eliminates the duplicate declaration problem no matter whether they’re spelled import <a.h> or #include <a.h>,

[Bigcheese]

I want to step back for a bit and give a higher level view of how I think about extensions. There are a few different kinds of extensions that have very different impacts on the C++ ecosystem.

The first kind are very local extensions often used within single functions. For example compiler intrinsics used to access specific hardware features. These are so common that we don’t really even treat them as extensions, people add whatever new vector intrinsics the new hardware has to LLVM and then hook them up in Clang. These kinds of extensions are extremely easy to isolate in your code and generally don’t cause any portability issues in practice.

The next kind are enhancements. They provide some benefit if used, but don’t intend to change the meaning of code. Examples here would be the sanitizers, in source diagnostic controls, Thread Safety Analysis, and Clang header modules. These all take code that is easy to make compilable with any conforming compiler and add something to it when you enable the feature in Clang. They have no impact to people using that code but not using those features.

The last are the extensions that are fundamental to the code using them. These would require writing the equivalent code twice to be portable, but without much if any resulting benefits to doing so. Some examples here are CUDA (although sometimes worth writing twice), Blocks, and proposed features not yet standardized. These all create a dialect. They can often be very useful, but they make the code fundamentally incompatible with compilers that don’t support that feature. These have the potential to impact the entire ecosystem as they are not optional. Obviously nobody is forcing you to use it, but the more it’s adopted the more likely it will impact you. There’s a reason so many C++ developers have to care about -fno-exceptions.

This extension fits into the last category. Code that needs this extension necessarily will not compile without it (otherwise you wouldn’t need the extension), and there’s no point in maintaining two copies of your entire codebase. For these kinds of extensions it’s not enough to say “don’t use it if you don’t like it”, as they can easily impact the entire ecosystem if used widely. If there’s a chance that happens, I would much rather it be in a way that’s maximally compatible.

@AaronBallman

+1, this is a proposal about extending our implementation based on user needs. The committee is free to do as they will. If our extension is going to step on a design the committee is considering, that’s something we should definitely try to avoid, but I don’t believe that’s the case with this proposal.

WG21 has no actual users, we do. That’s why we are free to solve user issues the committee has not addressed as a matter of QoI. This appears to be a conforming extension.

My intent in bringing up the committee here isn’t to say that we can’t do anything the committee didn’t tell us to, or that they are always right. It’s that this is not some new or unknown problem. It was heavily considered, particularly by those with experience with Clang header modules. The committee specifically provided a solution to these problems, so yes, the committee has addressed these with a solution that now has been in use for more than a decade. We have already encountered problems 1, 2 and 4 many years ago, all due to mixing textual headers with modular headers, which is the exact same cause here.

However, I do recognize that integrating header units into a build system in a high performance way isn’t trivial. So far Apple, Google, and Meta have all done it (I believe Microsoft has also done it, and probably a few others), but those are all in the top 10 largest companies in the world. We do not have a solution yet for commonly used build systems. It’s not that it’s impossible, it’s just that everyone working on the problem has done it in proprietary build systems.

It’s because of this lack of build system support that I do think a solution without header units is a reasonable request, but I strongly believe it is not the correct long term solution. You’re going to get the best build perf with the least changes to your code by adopting header units, and then incrementally adopting named modules. Thus why I proposed my solution which fits into the intended model.

@AaronBallman

That’s circular logic, IMO. They’re a better fit because C++ modules don’t support the use cases outside of header units (theoretically – AFAIK, nobody has managed to implement them to the standard’s experimental ideal). This proposal is a step forward towards supporting those use cases, which moves C++ modules closer to being a better fit.

It’s circular in that the feature specifically designed for this works well and the feature specifically not designed for this doesn’t work well. Note that Clang modules as implemented today are already an implementation of header units. You can write:

import <header.h>;
#include <header.h>

With -std=c++20 -fmodules and a module.modulemap file and this builds just fine and does include translation. I really want to push back on this idea that header units are some experimental or unknown thing. They were specifically designed such that Clang header modules were a valid implementation.

Also many system headers will just never work as named modules. stddef.h for example needs a textual header to use macros to select which actual declarations are made visible. In our system this textual header ends up selecting which module to import to get the requested declarations. And this isn’t the only header that does this, many system headers take macros as input.

@AaronBallman

Exactly this! System headers are included incredibly often (the STL depends on system headers, but so does most user code: Windows headers, Mac OS headers, etc). Having the ability to modularize these headers can lead to significant compile time performance improvements. Some of these headers are huge once you bring in transitive includes (consider Windows.h), so even being filled with C declarations, they are not a trivial amount of parse time.

This brings up one of my major concerns. Is this code ok?

import windows;

int max(int a, int b);

This was a common example during the discussion of how macros work. As a developer the syntax import windows; guarantees to me that this is fine to do. I really don’t want the C++ ecosystem to lose that guarantee.

@ChuanqiXu

For your proposed solution, I don’t know what’s the different between mine. Your concern was about scanning. But I am not sure how it solves the following problem:

import <|mod>;

#ifdef A
import ...;
#endif

Are you saying if the scanner saw a conditional import/include after import <|mod;>, the scanner can emit a warning? If yes, I think I got your point. But I don’t think it is better. I do feel the extension is more clear and neater.

The difference centers around the scanner and developers. We now have information in the source code that is not dependent on command line options and fits developers expectations. It is also significantly easier to port to compilers that do not support this extension.

The scanner can handle this case in various ways as we now know from the source that this may bring in new macros. Importantly it also means that if we see:

import mod;

#ifdef A
import ...;
#endif

Both the scanner and developer know that it can’t.

@ChuanqiXu

Users can have the choice to receive macros or not with the current proposal. For choosing some specific module to load to get the macros, it is doable by the proposed attribute too.

The first way is via the command line, and the 2nd is via a very semantics having and non-ignorable attribute. This is somewhat personal preference, but I much prefer specific syntax over attributes for anything that is non-ignorable.

[Bigcheese]

@tahonermann

I think it would be helpful to state somewhere how C++20 header units and Clang modules differ.

The biggest difference is in the handling of local linkage entities. The standard says they aren’t allowed, but that doesn’t work in practice, so Clang just acts nearly the same way it would with textual inclusion. See P2003R0: Fixing Internal and External Linkage Entities in Header Units, which still hasn’t been fixed because it turns out specifying this is hard.

The next biggest difference is that Clang makes use of the implementation definedness of header inclusion to in some cases actually include a different header that does some special stuff. This allows not having a separate PCM for every single header, which just doesn’t scale in practice. My understanding is Microsoft has now done the same thing too. The common case of this is taking a bunch of headers and building them together, and then when you #include <header1.h> it only makes the entities reachable from that header visible, but does end up depending from a build system perspective on all of them.

I’m at the point now where I think something very similar to Clang module maps is how we should handle header units in general, and it would be great if the compilers could all agree on a format. It makes it very easy to answer the questions of how do I make a header a header unit and how does include translation work reliably.

@tahonermann

I’m confused by statements that effectively state that nobody is working on C++20 header units when Clang clearly supports Clang modules. I suspect what is meant is that there hasn’t been a lot of progress in making module scanning based build systems work with C++20 header units. Thus, the situation is that implicitly built Clang modules work just fine as C++20 header units (disregarding concerns related to multiple implicitly built variants), but that work is still needed to enable deployment of build systems that use explicitly built C++20 header units. Is that a fair summary?

I think that’s fairly accurate. swift-build was recently open sourced, which does explicit builds of Clang header modules, but does not support C++20 named modules. It’s also an approach I don’t think a lot of build systems would be interested in, as it dynamically loads libclang to do the scan.

So we have people working on named modules, and people working on Clang header modules, but nobody focusing on using both together right now.

[ChuanqiXu]

Regarding header units and clang header modules

@cor3ntin

What is the status of importable header units in clang?

@tahonermann

I’m confused by statements that effectively state that nobody is working on C++20 header units when Clang clearly supports Clang modules. I suspect what is meant is that there hasn’t been a lot of progress in making module scanning based build systems work with C++20 header units. Thus, the situation is that implicitly built Clang modules work just fine as C++20 header units (disregarding concerns related to multiple implicitly built variants), but that work is still needed to enable deployment of build systems that use explicitly built C++20 header units. Is that a fair summary?

Header Units and Clang Header Modules are very semantically similar (there might be some difference in some macro handling). And they do share the same implementation. However, the current problem is, for the ecosystem, we do lack a user interface. End users don’t know how to use header units now. The de facto standard build system CMake don’t buy it. And I didn’t see any use of header units with clang in open source community.

The modulemap file is yet another user interface. Users always think this is a clang modules extension. I didn’t see this get used by project beyond google and apple. And this is not suggested for users to use header units. And as we know, generally there might be gaps between with and practice. That is, a lot of time, I think, “it should solve this problem !” but when I apply it in practice, I generally (or always) meet other problems.

@jyknight

The standard already contemplates the requirement to use macros from modules, which is why it supports header units. So, the obvious answer if you wanted to import arbitrary macros is simply to use header units. We don’t need an extension for that.

The problem with header units is implementation related. Let me state it with examples. CURRENTLY when we import it, we MAY still fall in redeclaration between TU traps. E.g.,

// my_header
#include <vector>

...

#define MY_MACRO

// a.cc
import <my_header>; // or #include <my_header>
import std;
...

use of MY_MACRO

Then the declaration in MAY be in std.pcm and my_header.pcm at the same time. Then we MAY meet the inefficient redeclaration problem. This is somewhat not ideal.

Note that this is implementation related. See the following example.

@jyknight

The example in the documentation shows that it will also translate a #include into an import when you’ve created a header-unit for a given header.

Yes, yes, this is the problem. Look at the example,

module;
#include <iostream>
export module M;
export void Hello() {
    std::cout << "Hello.\n";
}

Currently, the compiler may only replace the include to iostream, if the invocation provide a BMI for iostream. Like the example said:

clang++ -std=c++20 -xc++-system-header --precompile iostream -o iostream.pcm
clang++ -std=c++20 -fmodule-file=iostream.pcm --precompile M.cppm -o M.cpp

Now the problem comes, for the above example:

// my_header
#include <vector>

...

#define MY_MACRO

// a.cc
import <my_header>; // or #include <my_header>
import std;
...

use of MY_MACRO

How can the build system (no matter if it is CMake or the Clang itself or anything else) build a BMI for for vector ahead of time and pass it to the compilation of a.cc and my_header to avoid the redeclarations?

I believe this is not easy to do.

@jyknight

But I can’t see how this proposal really solves it. It can solve it in the case, only, where the #include appears after an import that exports the macros

But the case is simple for people to follow, to use in practice.

---

My summary is, in the early story of header units, it had a dream to solve all these problems. But that is at least 5 years ago. It is not achieved. And I didn’t see it is being pushed effectively. Now someone said some problems, and people say “go to use header units”. It doesn’t work in practice. And from what I saw, it only makes people frustrating.

And now I am proposing a simple and proved solution to address the problem. The solution is simple to use, to implement and maintain. It solves a common issue for modules. It is opt-in so it won’t affect users don’t want it. It is already implemented and used at scale. If we land this, users can benefit from it after a few months.

[ChuanqiXu]

@Bigcheese

I want to step back for a bit and give a higher level view of how I think about extensions. There are a few different kinds of extensions that have very different impacts on the C++ ecosystem.

The first kind are very local extensions often used within single functions. For example compiler intrinsics used to access specific hardware features. These are so common that we don’t really even treat them as extensions, people add whatever new vector intrinsics the new hardware has to LLVM and then hook them up in Clang. These kinds of extensions are extremely easy to isolate in your code and generally don’t cause any portability issues in practice.

The next kind are enhancements. They provide some benefit if used, but don’t intend to change the meaning of code. Examples here would be the sanitizers, in source diagnostic controls, Thread Safety Analysis, and Clang header modules. These all take code that is easy to make compilable with any conforming compiler and add something to it when you enable the feature in Clang. They have no impact to people using that code but not using those features.

The last are the extensions that are fundamental to the code using them. These would require writing the equivalent code twice to be portable, but without much if any resulting benefits to doing so. Some examples here are CUDA (although sometimes worth writing twice), Blocks, and proposed features not yet standardized. These all create a dialect. They can often be very useful, but they make the code fundamentally incompatible with compilers that don’t support that feature. These have the potential to impact the entire ecosystem as they are not optional. Obviously nobody is forcing you to use it, but the more it’s adopted the more likely it will impact you. There’s a reason so many C++ developers have to care about -fno-exceptions.

Nice analysis.

@Bigcheese

This extension fits into the last category. Code that needs this extension necessarily will not compile without it (otherwise you wouldn’t need the extension), and there’s no point in maintaining two copies of your entire codebase. For these kinds of extensions it’s not enough to say “don’t use it if you don’t like it”, as they can easily impact the entire ecosystem if used widely. If there’s a chance that happens, I would much rather it be in a way that’s maximally compatible.

Let’s try to discuss this.

From current projects related modules, I think there are 2 kinds of them. One only provides modules as wrappers but not used it. E.g., libc++, libstdc++ and some other traditional libraries. These libraries don’t use modules due to compatibility reasons now. Otherwise their users have to update their dependency and not able to use these libraries below C++20. For these libraries, it doesn’t matter if use the extension or not.

Some other projects will consume modules. Currently all such projects I saw are end-project. I mean a project building an executable but not a library.

And let’s assume, there are libraries using modules actually and they use this extension. And if people don’t want it, the library won’t get used. But if these libraries become so popular that other compilers have to support this extension. I assume this is what you called
“maximally compatible”.

Then it is actually funny. Since if users love the extension and compilers support it, why is it a problem? Since the community doesn’t follow the guidance from the committee? I feel this doesn’t make sense.

@Bigcheese

This brings up one of my major concerns. Is this code ok?

import windows;

int max(int a, int b);

This was a common example during the discussion of how macros work. As a developer the syntax import windows; guarantees to me that this is fine to do. I really don’t want the C++ ecosystem to lose that guarantee.

But as the consumer, we have the choice to introduce the macro or not. And as the provider, I feel the provider of such modules have the chance to not export macros like ‘max’.