[RFC] Allow recursive macros as extension

Clang Frontend
1

I propose to add preprocessor directive. All differences between #define and new directive - extension
do not prohibits recursive macro expansion.

Behavior:

  • extension do not changes any behavior of #define directive
  • recursion depth is limited, but limit is so huge it will be exceeded only with infinite recursion or with code which must never exist, because of that i do not add option for it
  • it is conditional to what must #define2 A A do, infinite recursion or replace macro with just A(because other usage is useless), in current implementation its diagnosed as infinite recursion

Itā€™s very small and fits well into the existing macro expansion implementation, so it should be easy to maintain

At this point the implementation is ready, but for now the spelling of the directive needs to be discussed

Motivation and implementation are in the pull request

2 Likes
2

Iā€™ve been thinking about how we could make this work this better as an extension.
#define2 is not a great name and having multiple ways to define macro functions seems a bit invasive and confusing.

And the only thing #define2 does is not expand its name.

So I think there is another design option that is a bit easier to justify as an extension and is probably easier to implement.
We can introduce a magic preprocessor identifier, eg __CURRENT_MACRO__ (name subject to bikeshedding of course) that, in a macro, would denote that macro

Adapting your example:


#define FOLD_RIGHT(op, head, ...) ( head __VA_OPT__(op __CURRENT_MACRO__(op, __VA_ARGS__)) )
static_assert(FOLD_RIGHT(+, 1, 2, 3) == 6);

that way we could model __CURRENT_MACRO__ on __VA_ARGS__ and __VA_OPT__ in that it would only be meaningful in function-like macros.
Making it testable with __has_extension would improve portability for now.

Itā€™s a design that is also probably to standardize in WG14/WG21 too (because it has a smaller surface area)

Iā€™d like other vendors opinion too!

But overall, I think this is a problem worth solving!

3 Likes
3

@reinterpretcast @AaronBallman @shafik

4

But if there are second layer or recursion(two macro at onceā€¦), then __CURRENT_MACRO__ will be not expanded? And it will break recursion, but in many cases it might work.
I like idea, also i want behavior of # __CURRENT_MACRO__(args) and ## __CURRENT_MACRO__ (args) to be expanded always, behavior of the C preprocessor in these places is annoying and forces to create many helper macros
I hope it wonā€™t be impossible to implement

5

Thank you for the RFC! This is an interesting idea

I have some high-level concerns:

  • The name define2 conveys nothing to the user as to how the feature works; weā€™ll have to pick a more descriptive name at some point.
  • I think macro expansion behavior should be predictable for the user, and I think we lose that property with this specific design. The user using a macro now has to know whether that macro was defined with #define or with #define2 to understand the expansion properties of the macro, and thatā€™s a pretty high cognitive burden for users. The suggestion from @cor3ntin helps in this regard, but another approach that might work (or might be a terrible idea) would be to wrap the macro name with a directive at the point you want to force recursive expansion. e.g., recursively_expand(MACRO) (where recursively_expand is a preprocessor operator).

Also, we have a list of criteria for adding extensions to Clang. The items I have concern with are:

  • Evidence of a significant user community. Macros have existed in C for a long time and you can achieve non-infinite recursion with macros using the existing preprocessor functionality (and there are libraries which help you with this, such as P99). Iā€™d appreciate more details on why these libraries are insufficient and this requires a language feature; infinite recursion is not possible (thatā€™s why we have to add a recursion depth limit) and I believe it is rarely necessary, so this seems like a very specific feature for a pretty uncommon problem.
  • A specification. The preprocessor has some very curious properties that have led to implementation divergence over the years; we should nail down the behavior of any new preprocessor extension so that itā€™s clear how it behaves. This also helps with the next partā€¦
  • Representation within the appropriate governing organization. We expect the preprocessor to remain broadly the same between C and C++ and the standards committees typically ask for that as well. So this feature will need some sort of proposal to both WG14 and WG21. Thatā€™s a tall order, but I think itā€™s critical if this extension is to be adopted by users ā€“ preprocessor differences between compilers can be a source of pain for users, and standardization helps to avoid that pain. Iā€™ve not seen a proposal like this in my time on the committees, and I canā€™t find evidence that someone else has already proposed this idea. Getting feedback from the committees can be tricky though ā€“ WG14 (the C committee) wants to see implementation experience, but as implementers, we do not want to implement an extension to the preprocessor and have the committee(s) ā€œtweakā€ the design such that we break users, so we want some sign from the committee that the design approach is correct. I think starting a high-level discussion on both the committee reflectors could at least kick-start getting that design feedback. But even that is tricky because ISO has been far more strictly enforcing their rules about who can participate in standardization. I think we should circle back to figure out how best to interface with the standards committees once we think weā€™ve got a roughly final design for the feature.
6

Iā€™d like to push back on that.
If we assume recursion in macro is useful - and i think the original PR had some examples, along with the mere existence of boost PP, P99, and similar facilitiesā€¦ - I had needed that a few times even though i try to limit my use of the preprocessor - then I think ā€œto call a macro recursively you need a libraryā€, is a tough sale.
But assuming you find a library with a suitable license and it gets blessed to be included in your company code base, or you reimplement it yourself, you end up with 2 issues:

  • The best interface you can get is APPLY(F, args) which kind of works but it would be more natural for F(args) to work
  • The libraries that exist work by generating (either manually or through a script in python, perl, cmake, etc) a list of N macros, N being how much nested calls they want to support.
    That produces unnecessarily large files that need to be pre-processed, unnecessarily causing the
    compiler to do more work and leading to bad diagnostics if there is an error in your pile of macros.

So the feature does sound well motivated to me, but i agree the design needs more explorations and starting a conversation with the different vendors and committee seems like the best next step.

4 Likes
7

Okay, I can see that logic, thank you! Thereā€™s a natural tension between ā€œyou can do this already and in fact people have provided libraries to do itā€ and ā€œwhy not make this part of the language?ā€ and the line is a bit fuzzy. I guess I see folks wanting to do less macro programming in C++ (e.g., C++'s treatment of macros in modules), so I tend to think the bar is higher here because this is in the preprocessor. However, existence of those libraries is a sign of a need in practice.

8

I donā€™t think user will be required to know more about define2/ define than now, he can think ā€˜its just somehow works, may be they generated 100ā€™000 lines by Python scriptā€™

I will list the problems with the current state of recursion in preprocessor:

  • complicates implementation, this forces user to create worse solution, which will be less readable or less usable
  • if you use a library like boost PP or P99 first you need to learn how to use this library, and this can be very difficult, because such an implementation complicates the interface too
  • generated recursion usually very limited in numbers,10 to 100 or something like
  • it greatly increases the volume of the source code and increases the likelihood of errors.
    Imagine, you forgot to remove ā€˜,ā€™ in one line, or the script generator inserts an extra comma in one edge case, when and how you will find this error?
  • if you use script to generate code, then this script is part of your project now, because if you want to change implementation in future you need to change this script
9

I agree with @AaronBallman that having two kinds of function-like macros could be a source of surprise and confusion. That being said, there are languages that support multiple kinds of macros with success; e.g., GNU make and its simply expanded and recursively expanded variables.

Iā€™m more inclined towards solutions like those suggested by @cor3ntin and @AaronBallman, assuming they suffice for the desired use cases.

10

It might be useful to look at the features offered by the various Preprocessor libraries like PP99 and boost PP and survey how the proposed recursive macro would simplify/replace these features, it might help informing the design.

3 Likes
11

I think a better approach is to redefine macros with the _Pragma operator, my code isnā€™t complete, but feel free to peruse it, itā€™s in the _Pragma(redefine_macro) branch, and itā€™s written in the spirit of _Pragma(push_macro/pop_macro)

As for the recursive expansion aspect, Iā€™m working on a directive #repeat that allows expansion a specified number of times, itā€™s in the #Repeat branch of my LLVM fork.

My primary use case is in registering test cases and test suites in C, and also to get around the issues with ā€˜ COUNTERā€™

Opinions? @tahonermann @cor3ntin @AaronBallman

Github fork here: GitHub - MarcusJohnson91/llvm-project: The LLVM Project is a collection of modular and reusable compiler and toolchain technologies. Note: the repository does not accept github pull requests at this moment. Please submit your patches at http://reviews.llvm.org.

12

Design changed from new preprocessor directive to special token __THIS_MACRO__, which means recursive macro call.
Its enabled only in function-like macros, its ready, all changes in linked pull request

13

I understand there are complications with a vendor extending the preprocessor.

However, the proposed changes are fairly minimal, can help remove a lot of cruft for people who only use clang, and do not take any design space beside a reserved identifier
(we could reduce that concern furger by picking a name such as __CLANG_THIS_MACRO__ or something along those lines - but that would reduce the chance of an adoption by other vendors, so not ideal).

Given that we are not going to solve the chicken / egg problem without laying something first, Iā€™m personally in favor of pursuing that change.

I see two outcomes:

  • Either this proves to be a success, gets adopted by more vendor, at which point WG14 might adopt it, or something similar
  • Either itā€™s not and we can either keep around a small amount of code, or deprecate the magic identifiers over a few release.

It is surprising that i would be in favor of compiler extensions, but Iā€™m unconvinced that WG14 would be able to do targeted improvements to the preprocessor without implementation experience. This change is somewhat similar in scope to __VA_OPT__ which i think was well received.

On the recursion limits concerns, Iā€™m not convince itā€™s new, itā€™s possible to use, eg boost PP in a way that creates an unreasonable amount of expansion.
We could and probably should consider an (user configurable) limit to how many expansions can be done by a top level macro, independently of this change.

1 Like
14

I reached out to the author of P99 to ask his opinions on the proposal here. My paraphrasing of his response is:

  • Making recursive macros less complicated is a good move, the code in P99 to handle this is pretty fragile.
  • __VA_OPT__ is good precedent.
  • What design to go with is much less clear (whether itā€™s a new directive or a new preprocessor keyword, etc).
  • The current design is more restrictive than general recursion because it does not allow for complicated patterns with implicit recursion through several layers of function-like macros.
  • We should be sure to keep the broader context in mind: how much of this do we need? Would a feature that does iteration over a finite set be more appropriate?

My take is:

  • I agree thereā€™s a chicken and egg problem with WG14 and implementations likely have to move first to get WG14 to standardize anything. So far, the discussion on the reflectors has been underwhelming, potentially because weā€™re still in the middle of balloting and so people are not supposed to discuss changes to the standard during that time. We might get more feedback after the C23 ballot closes, but I think weā€™d need an actual paper in front of the committee to get significant feedback. One thing that would help here is some coordination with GCC developers on the extension; they donā€™t have to be willing to adopt it on our timeline, but it would be good to know up front whether they have some amount of buy-in with the design or are opposed to it. That makes the standards paper more compelling as well because thereā€™s more prior art to point to (even if GCC doesnā€™t implement it, some public show of support helps).
  • I agree that we need to keep the broader context in mind. Some of this boils down to motivating examples where we can take existing code and change it to use the new model to demonstrate what cases work and what cases wonā€™t work. But some of this is also design-level: given that we know we cannot do truly infinite recursion (we have to have some recursion limitations as a compiler limit), would it instead be a more portable and useful feature to let the user specify the recursion limits as part of the preprocessor feature so that the behavior is then portable across compilers without needing to rely on command line switches? (Maybe thatā€™s a bad idea for other reasons, however.)
  • I like the __THIS_MACRO__ form better than the #define2 form because itā€™s a syntactic marker in code at the point of expansion (you donā€™t have to look at how the macro was defined to understand its expansion behavior). I do wonder how well this design works with mutual recursion though.
1 Like
15

Mutual recursion was first thing i thinking about before change define2 ā†’ __THIS_MACRO__, but I couldnā€™t think of a scenario in which this problem would occur.

would it instead be a more portable and useful feature to let the user specify the recursion limits as part of the preprocessor

Because of that, i think best user strategy to compile all possible code - set recursion depth to max possible, but if so, why we dont set it to max without new command-line option?

16

I definitely agree with that, but I think naming still needs some work. __THIS_MACRO__ appears to me to conflate two features; 1) the ability to name the current macro without knowing its name (often desired for generic code), and 2) the ability to recursively expand a macro.

Iā€™m not convinced that this approach solves a substantial set of use cases for recursively expanded macros since it doesnā€™t support recursive expansion of a macro other than the current one. In other words, it canā€™t handle structural nesting operations. Consider the following example (which I hope I got right).

#define PROCESS_ELEM(X) X
#define PROCESS_LIST(...) __VA_OPT__(PROCESS(__VA_ARGS__))
#define PROCESS_NEXT(X, ...) PROCESS_##X __VA_OPT__(, PROCESS_NEXT(__VA_ARGS__))
#define PROCESS(X, ...) { PROCESS_##X __VA_OPT__(, PROCESS_NEXT(__VA_ARGS__)) }
struct S {
  int dm1;
  struct {
    int dm1, dm2;
  } dm2;
  int dm3;
};
S s = PROCESS(ELEM(0), LIST(ELEM(1), ELEM(2)), ELEM(3));

The intent is that the last line expand to (ignoring white space concerns):

S s = { 0, { 1, 2 }, 3 };

The proposed __THIS_MACRO__ would suffice to allow recursion for the case where PROCESS_NEXT recursively invokes itself, but it doesnā€™t suffice to address the case where PROCESS is recursively invoked during the expansion of PROCESS_LIST. A preprocessor operator that enables unconditional (recursive or non-recursive) expansion of the macro name it is applied to would suffice to address this use case.

Using @ as a placeholder for operator syntax, the macros above could then be defined as follows and would produce the intended output for the example above.

#define PROCESS_ELEM(X) X
#define PROCESS_LIST(...) __VA_OPT__(@PROCESS(__VA_ARGS__))
#define PROCESS_NEXT(X, ...) @PROCESS_##X __VA_OPT__(, @PROCESS_NEXT(__VA_ARGS__))
#define PROCESS(X, ...) { @PROCESS_##X __VA_OPT__(, @PROCESS_NEXT(__VA_ARGS__)) }
2 Likes
17

Thats a good example, but simplest possible solution - just mark all macros with token __THIS_MACRO__ as recursive (allow expand always), it also will simplify implementation

18

I would prioritize providing a general solution that addresses more use cases over ease of implementation; particularly when the effort required might not be much greater (which I suspect is the case).

I might be wrong, but I donā€™t think the example I provided can be implemented using the __THIS_MACRO__ approach. If you believe differently, please show how you would write it.

1 Like
19

To support such cases, there is nothing left to do but allow the macro to expand recursively whenever it is mentioned. After this, all that remains is to decide by what criterion we should select macros for which this is allowed
In the original version, this tag was a new preprocessor directive, but this had its drawbacks

But the special token __THIS_MACRO__ fits this role quite well. With this we simultaneously solve several problems:

  • it is quite obvious to the user what is happening (__THIS_MACRO__ accurately reflects the intention to do something recursive)
  • we do not introduce new directives and completely reuse the macro mechanism, without adding any complexity to support this in other parts (such as clangd)

But we need to decide what to do with such declarations

#define A(...) __THIS_MACRO__() A()

I would diagnose using name of macro in macro which uses __THIS_MACRO__ as a error for removing ambiguity

20

I donā€™t agree. As previously indicated, we have the option of adding a preprocessor operator to opt-in to a recursive expansion; e.g., the use of @ exhibited at the end of my earlier comment.

Perhaps it is helpful to think of this operator like eval in some other languages. In fact, perhaps it would be more useful to use an operator that supports delimiters. Today, we sometimes have to introduce additional macros just to force an expansion:

#define DO_STRINGIFY(X) #X
#define STRINGIFY(X) DO_STRINGIFY(X)
#define TEXT blah
STRINGIFY(TEXT) // expands to "blah", not "TEXT"

With an eval operator, perhaps STRINGIFY could instead be written:

#define STRINGIFY(X) #__EVAL__(X)

Going back to my previous example, the PROCESS related macros could be defined as:

#define PROCESS_ELEM(X) X
#define PROCESS_LIST(...) __VA_OPT__(__EVAL__(PROCESS(__VA_ARGS__)))
#define PROCESS_NEXT(X, ...) __EVAL__(PROCESS_##X) __VA_OPT__(, __EVAL__(PROCESS_NEXT(__VA_ARGS__)))
#define PROCESS(X, ...) { __EVAL__(PROCESS_##X) __VA_OPT__(, __EVAL__(PROCESS_NEXT(__VA_ARGS__))) }
1 Like