Hello,
SVN revision 218614 documented that C++ modules works in clang.
Using the llvm apt repo on Ubuntu 14.04, I get this result:
$ clang++-3.6 --version
Ubuntu clang version 3.6.0-svn218873-1~exp1 (trunk) (based on LLVM 3.6.0)
Target: x86_64-pc-linux-gnu
Thread model: posix
$ cat modules.cpp
@import std;
$ clang++-3.6 -std=c++1z -fmodules modules.cpp
modules.cpp:2:1: error: expected unqualified-id
@import std;
^
1 error generated.
How do I try out clang modules support?
Thanks,
Steve.
http://clang.llvm.org/docs/Modules.html should have all the answers, and if it doesn’t please complain so that we can fix it 
Currently there is no “dotted path” (e.g. import std.algorithm, or whatever) import in C++; #include’s are translated into it: try looking at -E with modules enabled.
You currently have to do everything through module maps; listing your header in a module map basically tells clang “I promise you this header meets certain assumptions of modularity” (which are really fairly reasonable assumptions; e.g. you aren’t expecting a certain macro to be defined somewhere else in the code). Module maps also describe a “dotted import path”, but that probably isn’t going to be relevant for C++ for a long time. Currently Clang’s diagnostics in the case where your header fails to meet those assumptions are not very good (by that I mean: really quite poor), but it is something that was recently discussed at the social and is recognized as an issue; you will primarily have to face this when actually writing your module map for the first time; after that, it should just transparently speed up your build.
I’ve attached a “minimal example” that should illustrate how it works.
And really, what is the difference between “#include <foo.h>” and “import mymodule.foo”? The real difference is that one can be used in your existing codebase (in conjunction with a module map) and give you real build performance wins for your real production C++ code right now (even if you have to support other compilers), and the other one a) isn’t enabled now b) will probably not be what is actually standardized, and so will never work. How to organize the “dotted import paths” is a big bikeshed to paint (e.g. is std.string <string.h> or ? is it std.c.io or std.c.stdio?) and ultimately the standard committee will be the one to decide for the standard library; in your own module maps for C++, I recommend focusing on making sure your headers are hygienic.
Right now, it is important to make sure your headers are “hygienic” so that they down the road, they can be composed into modules with nice “dotted import paths”. Btw, you can get the build time improvements as soon as your headers are “hygienic”.
– Sean Silva
modulestry.zip (958 Bytes)
Sean Silva wrote:
Currently there is no "dotted path" (e.g. import std.algorithm, or
whatever) import in C++; #include's are translated into it:
Right, this is what I was missing from the doc page, thanks!
The starting point you attached allows me to try to make sense of the rest
of the page.
try looking at -E with modules enabled.
That's interesting. Usually, the output from -E can be saved to a .i file
and can be compiled. With modules enabled though, a file containing
'@import' is generated, which has no further usefulness. Not that using -E
to create output for further usage, but I'm only making a general remark.
You currently have to do everything through module maps; listing your
header in a module map basically tells clang "I promise you this header
meets certain assumptions of modularity" (which are really fairly
reasonable assumptions; e.g. you aren't expecting a certain macro to be
defined somewhere else in the code).
This is exactly something I want to experiment with, to get a good feeling
for what the limitations are, with examples.
in your own module maps for
C++, I recommend focusing on making sure your headers are hygienic.
My interest/curiosity is whether module maps can be created for
Qt/KDE/Boost. It seems that module maps for dependencies (such as the std
library) are a prerequisite before that kind of playing around anyway. Is
something floating around already for libc++/libstdc++? Even a trivial one,
not providing separate std.io etc, but just std?
With my 'buildsystem guy/CMake' hat on, I'm also curious how this modules
stuff affects the buildsystems out there. Possibly not very much, because
clang (and hopefully all drivers) will manage the dependency tracking etc.
I'm interested in user frustration and porting pain.
1) The link directive could be problematic because it seems to assume
library names only, and that paths are specified with -L as needed by the
buildsystem. That's the opposite of CMake philosophy which uses full paths
instead. Some background information here:
CMP0003 — CMake 3.0.2 Documentation
I guess that's not something for clang to worry about, but possibly
something for a 'how to use modules with cmake' document instructing users
to use -fno-autolink.
2) All module.modulemap files have the same name.
[Aside: in the doc, the sentence "Module map files are typically named
module.modulemap", but elsewhere on the page it is specified to be a naming
requirement - 'typically' is wrong here]
The 'compiled' module.modulemap is not a distributable. The module.modulemap
file is a distributable. It should be installed with the headers.
That is, if I build/install the foo package to the /usr/local prefix, I will
get files like:
/usr/local/include/foo.h
/usr/local/include/module.modulemap
If I build and install bar, then I will get:
/usr/local/include/bar.h
/usr/local/include/module.modulemap
How do I install foo and bar at the same time to the same prefix? Each
provides files in the same location which are required to be called
"module.modulemap".
This potentially affects the installation rules written in buildsystems for
foo and bar.
3) cplusplus11 as a 'requires declaration' use is unclear.
If a user uses static_assert in their header, then they say they require
cplusplus11. MSVC 2010 supports static_assert.
Similarly, if a user uses constexpr in their header, then they say they
require cplusplus11. MSVC does not support that at all. I can also point to
other 'c++11' features that even the next MSVC release will not support.
So, is this something that is proposed for standardization? How is MSVC
expected to handle such a 'requires declaration'? Will there be a
cplusplus17 requires declaration? When clang implements c++17 partially,
what behavior will be used for the cplusplus17 requires declaration?
Or do I misunderstand? Is cplusplus11 or similar only suitable for binary
header is available/not available such as the type_traits example, and it
doesn't relate to features required by the header? Does cplusplus11 exist
only for the benefit of std library implementations?
Thanks,
Steve.
Sean Silva wrote:
> Currently there is no "dotted path" (e.g. import std.algorithm, or
> whatever) import in C++; #include's are translated into it:Right, this is what I was missing from the doc page, thanks!
The starting point you attached allows me to try to make sense of the rest
of the page.
We say this, but it's buried in a section about Objective-C; perhaps it
should be moved somewhere more prominent:
"At present, there is no C or C++ syntax for import declarations. Clang
will track the modules proposal in the C++ committee. See the section
Includes as imports to see how modules get imported today."
try looking at -E with modules enabled.
That's interesting. Usually, the output from -E can be saved to a .i file
and can be compiled. With modules enabled though, a file containing
'@import' is generated, which has no further usefulness. Not that using -E
to create output for further usage, but I'm only making a general remark.
I think we should add a #pragma clang module_import (or similar) to support
this.
You currently have to do everything through module maps; listing your
> header in a module map basically tells clang "I promise you this header
> meets certain assumptions of modularity" (which are really fairly
> reasonable assumptions; e.g. you aren't expecting a certain macro to be
> defined somewhere else in the code).This is exactly something I want to experiment with, to get a good feeling
for what the limitations are, with examples.> in your own module maps for
> C++, I recommend focusing on making sure your headers are hygienic.My interest/curiosity is whether module maps can be created for
Qt/KDE/Boost. It seems that module maps for dependencies (such as the std
library) are a prerequisite before that kind of playing around anyway. Is
something floating around already for libc++/libstdc++? Even a trivial one,
not providing separate std.io etc, but just std?
libc++ already ships with a module map.
With my 'buildsystem guy/CMake' hat on, I'm also curious how this modules
stuff affects the buildsystems out there. Possibly not very much, because
clang (and hopefully all drivers) will manage the dependency tracking etc.
I'm interested in user frustration and porting pain.1) The link directive could be problematic because it seems to assume
library names only, and that paths are specified with -L as needed by the
buildsystem. That's the opposite of CMake philosophy which uses full paths
instead. Some background information here:CMP0003 — CMake 3.0.2 Documentation
I guess that's not something for clang to worry about, but possibly
something for a 'how to use modules with cmake' document instructing users
to use -fno-autolink.2) All module.modulemap files have the same name.
[Aside: in the doc, the sentence "Module map files are typically named
module.modulemap", but elsewhere on the page it is specified to be a naming
requirement - 'typically' is wrong here]
'typically' is correct; while only 'module.modulemap' is searched for in -I
paths, additional module map files can be specified via -fmodule-map-file=
and 'extern module' declarations, and these files do not need to be named
module.modulemap.
The 'compiled' module.modulemap is not a distributable. The module.modulemap
file is a distributable. It should be installed with the headers.
That is, if I build/install the foo package to the /usr/local prefix, I
will
get files like:/usr/local/include/foo.h
/usr/local/include/module.modulemapIf I build and install bar, then I will get:
/usr/local/include/bar.h
/usr/local/include/module.modulemapHow do I install foo and bar at the same time to the same prefix? Each
provides files in the same location which are required to be called
"module.modulemap".This potentially affects the installation rules written in buildsystems for
foo and bar.
One option would be for each library to provide its own module map file,
which you specify on the clang command line with -fmodule-map-file=. I
don't think we necessarily have the right answer for this case yet.
3) cplusplus11 as a 'requires declaration' use is unclear.
If a user uses static_assert in their header, then they say they require
cplusplus11. MSVC 2010 supports static_assert.Similarly, if a user uses constexpr in their header, then they say they
require cplusplus11. MSVC does not support that at all. I can also point to
other 'c++11' features that even the next MSVC release will not support.So, is this something that is proposed for standardization?
No. Nothing about module maps is proposed for standardization, nor even
guaranteed to be compatible across Clang revisions (yet). So be aware that
if you go and write a bunch of module maps now, you may need to change them
all to keep them working with future versions of Clang (or you may not; we
don't know).
How is MSVC
expected to handle such a 'requires declaration'? Will there be a
cplusplus17 requires declaration? When clang implements c++17 partially,
what behavior will be used for the cplusplus17 requires declaration?Or do I misunderstand? Is cplusplus11 or similar only suitable for binary
header is available/not available such as the type_traits example, and it
doesn't relate to features required by the header? Does cplusplus11 exist
only for the benefit of std library implementations?
"requires" flags in general only indicate whether a module is available in
a particular build configuration. The header still needs to do whatever
feature checking it would normally do, to avoid using features the compiler
doesn't support.
Richard Smith wrote:
"At present, there is no C or C++ syntax for import declarations. Clang
will track the modules proposal in the C++ committee. See the section
Includes as imports to see how modules get imported today."
Yes, to be more-specific than I was before, this ^ is what I was missing
before this thread.
> in your own module maps for
> C++, I recommend focusing on making sure your headers are hygienic.My interest/curiosity is whether module maps can be created for
Qt/KDE/Boost. It seems that module maps for dependencies (such as the std
library) are a prerequisite before that kind of playing around anyway. Is
something floating around already for libc++/libstdc++? Even a trivial
one, not providing separate std.io etc, but just std?libc++ already ships with a module map.
Ok, the version of libcxx on my system doesn't. Maybe I can point clang to
the version in the git repo...
[Aside: in the doc, the sentence "Module map files are typically named
module.modulemap", but elsewhere on the page it is specified to be a
naming requirement - 'typically' is wrong here]'typically' is correct; while only 'module.modulemap' is searched for in
-I paths, additional module map files can be specified via
-fmodule-map-file= and 'extern module' declarations, and these files do
not need to be named module.modulemap.
I see. Perhaps "(each named module.modulemap)" could be rephrased in the
docs too then.
However, I tried this with the example from Sean after moving the
module.modulemap file to a foo subdirectory and renaming it.
It did not work:
$ ls $PWD/foo/module.modulemap
/home/stephen/dev/src/playground/modules/modulestry/foo/map
$ clang++-3.6 -E -fmodules -fmodule-map-file=$PWD/foo/map test.cpp
<snip>
static inline int foo() { return 1; }
# 5 "test.cpp" 2
int main(int, char **) {
return foo();
}
Instead of the expected @import declaration.
I also tried -fmodule-map-file=$PWD/foo/does_not_exist and no warning or
error was reported.
I then moved the map file out of the subdirectory, and ran
$ clang++-3.6 -E -fmodules -fmodule-map-file=map -Weverything test.cpp
and the first time I ran it (but not any subsequent time), I got the output:
<module-includes>:1:1: warning: header 'map' is included in module 'test'
but not listed in module map [-Wincomplete-module]
#include "/home/stephen/dev/src/playground/modules/modulestry/./test.h"
^
and the expected
@import test;
line.
So, I guess I can specify the name, but the file must still be with the
headers. I do have other things I wonder about along these lines, but I'll
experiment more later.
One option would be for each library to provide its own module map file,
which you specify on the clang command line with -fmodule-map-file=. I
don't think we necessarily have the right answer for this case yet.
Right. At least in CMake, we can add an abstraction (eventually, when the
feature is stable) to specify the -fmodule-map-file as a usage requirement.
So, is this something that is proposed for standardization?
No. Nothing about module maps is proposed for standardization, nor even
guaranteed to be compatible across Clang revisions (yet).
I see. I remember hearsay that there are 3 different modules proposals in
the works currently. I found
http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2014/n4047.pdf
Can you point me to other modules related N-documents?
So be aware that
if you go and write a bunch of module maps now, you may need to change
them all to keep them working with future versions of Clang (or you may
not; we don't know).
Understood, thanks.
Steve.
Stephen Kelly wrote:
It did not work:
$ ls $PWD/foo/module.modulemap
/home/stephen/dev/src/playground/modules/modulestry/foo/map
$ clang++-3.6 -E -fmodules -fmodule-map-file=$PWD/foo/map test.cpp
This worked when I changed the content of map after moving to list the
'../test.h'
header instead of just
'test.h'
Thanks,
Steve.
Stephen Kelly wrote:
Stephen Kelly wrote:
It did not work:
$ ls $PWD/foo/module.modulemap
/home/stephen/dev/src/playground/modules/modulestry/foo/map
$ clang++-3.6 -E -fmodules -fmodule-map-file=$PWD/foo/map test.cpp
I created a module-map-file for some of QtCore, and I noticed that if I
specify -fmodule-map-file multiple times, only the last one has any effect.
Thanks,
Steve.
Stephen Kelly wrote:
Stephen Kelly wrote:
Stephen Kelly wrote:
It did not work:
$ ls $PWD/foo/module.modulemap
/home/stephen/dev/src/playground/modules/modulestry/foo/map
$ clang++-3.6 -E -fmodules -fmodule-map-file=$PWD/foo/map test.cppI created a module-map-file for some of QtCore, and I noticed that if I
specify -fmodule-map-file multiple times, only the last one has any
effect.
I filed a bug for this and other bugs for some other issues.
21214 – clang needs to accept -fmodule-map-file multiple times
Thanks,
Steve.
Sean Silva wrote:
> Currently there is no "dotted path" (e.g. import std.algorithm, or
> whatever) import in C++; #include's are translated into it:Right, this is what I was missing from the doc page, thanks!
The starting point you attached allows me to try to make sense of the rest
of the page.> try looking at -E with modules enabled.
That's interesting. Usually, the output from -E can be saved to a .i file
and can be compiled. With modules enabled though, a file containing
'@import' is generated, which has no further usefulness. Not that using -E
to create output for further usage, but I'm only making a general remark.
Yeah, it's not clear if this is actually a "bug" since this is what clang
is actually doing, but it's still weird to end up with objective C inside
your preprocessed C++ file.
> You currently have to do everything through module maps; listing your
> header in a module map basically tells clang "I promise you this header
> meets certain assumptions of modularity" (which are really fairly
> reasonable assumptions; e.g. you aren't expecting a certain macro to be
> defined somewhere else in the code).This is exactly something I want to experiment with, to get a good feeling
for what the limitations are, with examples.> in your own module maps for
> C++, I recommend focusing on making sure your headers are hygienic.My interest/curiosity is whether module maps can be created for
Qt/KDE/Boost. It seems that module maps for dependencies (such as the std
library) are a prerequisite before that kind of playing around anyway. Is
something floating around already for libc++/libstdc++? Even a trivial one,
not providing separate std.io etc, but just std?
My experience has been that the easiest thing is to start out by just doing
a "unity module" like
module foo {
header "bar"
header "baz"
...
}
Just list every file in the directory (possibly recursively). Then just
exclude any headers that are causing problems. That should get you up and
running pretty fast.
-- Sean Silva
Richard Smith <richard@...> writes:
libc++ already ships with a module map.
Today I built llvm, clang and libcxx on a Mac (OSX 10.9 I assume), and when
I try to use -fmodules, it can't build the module.modulemap shipped with
libcxx. Any idea what's going wrong?
$ clang++ -E -fmodules test.cpp
# 1 "test.cpp"
# 1 "<built-in>" 1
# 1 "<built-in>" 3
# 325 "<built-in>" 3
# 1 "<command line>" 1
# 1 "<built-in>" 2
# 1 "test.cpp" 2
While building module 'std' imported from test.cpp:6:
In file included from <module-includes>:4:
In file included from /Users/ske/dev/prefix/bin/../include/c++/v1/ccomplex:21:
In file included from /Users/ske/dev/prefix/bin/../include/c++/v1/complex:247:
In file included from /Users/ske/dev/prefix/bin/../include/c++/v1/sstream:174:
In file included from /Users/ske/dev/prefix/bin/../include/c++/v1/ostream:140:
/Users/ske/dev/prefix/bin/../include/c++/v1/locale:873:26: error: use of
undeclared identifier 'strtoll_l'; did you mean 'wcstoll_l'?
long long __ll = strtoll_l(__a, &__p2, __base, _LIBCPP_GET_C_LOCALE);
^
/usr/include/xlocale/_wchar.h:98:2: note: 'wcstoll_l' declared here
wcstoll_l(const wchar_t * __restrict, wchar_t ** __restrict, int,
^
While building module 'std' imported from test.cpp:6:
In file included from <module-includes>:4:
In file included from /Users/ske/dev/prefix/bin/../include/c++/v1/ccomplex:21:
In file included from /Users/ske/dev/prefix/bin/../include/c++/v1/complex:247:
In file included from /Users/ske/dev/prefix/bin/../include/c++/v1/sstream:174:
In file included from /Users/ske/dev/prefix/bin/../include/c++/v1/ostream:140:
/Users/ske/dev/prefix/bin/../include/c++/v1/locale:873:36: error: cannot
initialize a parameter of type 'const wchar_t *' with an lvalue of type
'const char *'
long long __ll = strtoll_l(__a, &__p2, __base, _LIBCPP_GET_C_LOCALE);
^~~
/usr/include/xlocale/_wchar.h:98:38: note: passing argument to parameter here
wcstoll_l(const wchar_t * __restrict, wchar_t ** __restrict, int,
^
While building module 'std' imported from test.cpp:6:
In file included from <module-includes>:4:
In file included from /Users/ske/dev/prefix/bin/../include/c++/v1/ccomplex:21:
In file included from /Users/ske/dev/prefix/bin/../include/c++/v1/complex:247:
In file included from /Users/ske/dev/prefix/bin/../include/c++/v1/sstream:174:
In file included from /Users/ske/dev/prefix/bin/../include/c++/v1/ostream:140:
/Users/ske/dev/prefix/bin/../include/c++/v1/locale:913:35: error: use of
undeclared identifier 'strtoull_l'; did you mean 'wcstoull_l'?
unsigned long long __ll = strtoull_l(__a, &__p2, __base,
_LIBCPP_GET_C_LOCALE);
^
/usr/include/xlocale/_wchar.h:101:2: note: 'wcstoull_l' declared here
wcstoull_l(const wchar_t * __restrict, wchar_t ** __restrict, int,
^
While building module 'std' imported from test.cpp:6:
In file included from <module-includes>:4:
In file included from /Users/ske/dev/prefix/bin/../include/c++/v1/ccomplex:21:
In file included from /Users/ske/dev/prefix/bin/../include/c++/v1/complex:247:
In file included from /Users/ske/dev/prefix/bin/../include/c++/v1/sstream:174:
In file included from /Users/ske/dev/prefix/bin/../include/c++/v1/ostream:140:
/Users/ske/dev/prefix/bin/../include/c++/v1/locale:913:46: error: cannot
initialize a parameter of type 'const wchar_t *' with an lvalue of type
'const char *'
unsigned long long __ll = strtoull_l(__a, &__p2, __base,
_LIBCPP_GET_C_LOCALE);
^~~
/usr/include/xlocale/_wchar.h:101:39: note: passing argument to parameter here
wcstoull_l(const wchar_t * __restrict, wchar_t ** __restrict, int,
^
While building module 'std' imported from test.cpp:6:
In file included from <module-includes>:4:
In file included from /Users/ske/dev/prefix/bin/../include/c++/v1/ccomplex:21:
In file included from /Users/ske/dev/prefix/bin/../include/c++/v1/complex:247:
In file included from /Users/ske/dev/prefix/bin/../include/c++/v1/sstream:174:
In file included from /Users/ske/dev/prefix/bin/../include/c++/v1/ostream:140:
/Users/ske/dev/prefix/bin/../include/c++/v1/locale:943:28: error: use of
undeclared identifier 'strtold_l'
long double __ld = strtold_l(__a, &__p2, _LIBCPP_GET_C_LOCALE);
^
test.cpp:6:10: fatal error: could not build module 'std'
#include <utility>
~~~~~~~~^
int main(int argc, char** argv)
{
return 0;
}
6 errors generated.
$ cat test.cpp
//#include "foo.h"
//#include <QtCore/qxmlstream.h>
#include <utility>
int main(int argc, char** argv)
{
return 0;
}
$ clang++ --version
clang version 3.6.0
Target: x86_64-apple-darwin13.4.0
Thread model: posix
After a bit of looking around on my Mac, I think I figured out what is going on. The declaration for those functions is in <xlocale/_stdlib.h>. That header is only included from two places:
I’ll assume you don’t have USE_EXTENDED_LOCALES defined on the command line. Looking at libc++'s and the headers it includes, it looks like xlocale.h gets included without stdlib.h having been included ( → <__locale> → <xlocale.h>). If this hypothesis is correct, then these errors should go away if you change libc++'s by moving #include <__locale> below #include <cstdlib>.
The trick that the Apple headers seem to use to work around the “must be included after” issue is that xlocale.h defines USE_EXTENDED_LOCALES so that a subsequent include of stdlib.h will bring them in. However, stdlib.h is brought in from the module, where it was built without USE_EXTENDED_LOCALES defined. Thus, <xlocale/_stdlib.h> is never seen by the compiler, and those functions are missing.
This trick breaks one of the promises the module map file makes about stdlib.h, which roughly boils down to “this header isn’t meant to change its behaviors depending on the set of macros defined when it is included”; thus it is not surprising that this should break. I made a reduced test case (attached) which suggests that even clang’s config_macros doesn’t deal with this.
Richard, any ideas on what the “right” solution is here?
Stephen, thanks for being a brave guinea pig here
– Sean Silva
testdoubleinclude.zip (1.34 KB)
After a bit of looking around on my Mac, I think I figured out what is
going on. The declaration for those functions is in <xlocale/_stdlib.h>.
That header is only included from two places:1. From stdlib.h if _USE_EXTENDED_LOCALES_ is defined.
2. From xlocale.h if stdlib.h has been previously included (this is sort
of nasty, but documented
xlocale(3) "These
functions are exposed by including <xlocale.h> after including the
relevant headers for the standard variant."; Apple doesn't document this
because it looks like they use a non-modular trick described below to work
around this).I'll assume you don't have _USE_EXTENDED_LOCALES_ defined on the command
line. Looking at libc++'s <locale> and the headers it includes, it looks
like xlocale.h gets included without stdlib.h having been included
(<locale> --> <__locale> --> <xlocale.h>). If this hypothesis is correct,
then these errors should go away if you change libc++'s <locale> by moving
`#include <__locale>` below `#include <cstdlib>`.The trick that the Apple headers seem to use to work around the "must be
included after" issue is that xlocale.h defines _USE_EXTENDED_LOCALES_ so
that a subsequent include of stdlib.h will bring them in. However, stdlib.h
is brought in from the module, where it was built without
_USE_EXTENDED_LOCALES_ defined. Thus, <xlocale/_stdlib.h> is never seen by
the compiler, and those functions are missing.This trick breaks one of the promises the module map file makes about
stdlib.h, which roughly boils down to "this header isn't meant to change
its behaviors depending on the set of macros defined when it is included";
thus it is not surprising that this should break. I made a reduced test
case (attached) which suggests that even clang's config_macros doesn't deal
with this.
Did you try adding -D_USE_ to the command line in your reduced testcase?
Richard, any ideas on what the "right" solution is here?
One possibility would be to split stdlib.h up into three:
1) A module providing most of the stdlib.h contents
2) A module providing xlocale/_stdlib.h (it seems this part is already done)
3) A file that imports (1) and (2)
... and to treat (3) as a textually-included header rather than a module. I
think that's my preferred approach, since it allows retaining the existing
horrible mess of _*_SOURCE configuration macros controlling which symbols
come from which includes.
Another would be for stdlib.h to always provide everything when built with
modules enabled.
Another would be to add -D_USE_EXTENDED_LOCALES_ to your clang command line.
Stephen, thanks for being a brave guinea pig here
After a bit of looking around on my Mac, I think I figured out what is
going on. The declaration for those functions is in <xlocale/_stdlib.h>.
That header is only included from two places:1. From stdlib.h if _USE_EXTENDED_LOCALES_ is defined.
2. From xlocale.h if stdlib.h has been previously included (this is sort
of nasty, but documented
xlocale(3) "These
functions are exposed by including <xlocale.h> after including the
relevant headers for the standard variant."; Apple doesn't document this
because it looks like they use a non-modular trick described below to work
around this).I'll assume you don't have _USE_EXTENDED_LOCALES_ defined on the command
line. Looking at libc++'s <locale> and the headers it includes, it looks
like xlocale.h gets included without stdlib.h having been included
(<locale> --> <__locale> --> <xlocale.h>). If this hypothesis is correct,
then these errors should go away if you change libc++'s <locale> by moving
`#include <__locale>` below `#include <cstdlib>`.The trick that the Apple headers seem to use to work around the "must be
included after" issue is that xlocale.h defines _USE_EXTENDED_LOCALES_ so
that a subsequent include of stdlib.h will bring them in. However, stdlib.h
is brought in from the module, where it was built without
_USE_EXTENDED_LOCALES_ defined. Thus, <xlocale/_stdlib.h> is never seen by
the compiler, and those functions are missing.This trick breaks one of the promises the module map file makes about
stdlib.h, which roughly boils down to "this header isn't meant to change
its behaviors depending on the set of macros defined when it is included";
thus it is not surprising that this should break. I made a reduced test
case (attached) which suggests that even clang's config_macros doesn't deal
with this.Did you try adding -D_USE_ to the command line in your reduced testcase?
Richard, any ideas on what the "right" solution is here?
One possibility would be to split stdlib.h up into three:
1) A module providing most of the stdlib.h contents
2) A module providing xlocale/_stdlib.h (it seems this part is already
done)
3) A file that imports (1) and (2)... and to treat (3) as a textually-included header rather than a module.
I think that's my preferred approach, since it allows retaining the
existing horrible mess of _*_SOURCE configuration macros controlling which
symbols come from which includes.Another would be for stdlib.h to always provide everything when built with
modules enabled.Another would be to add -D_USE_EXTENDED_LOCALES_ to your clang command
line.
I think in this case it's not meant to be something for the end-user to
configure. It seems like _USE_EXTENDED_LOCALES_ is just a cookie that
xlocale.h sets so that stdlib.h et al. remember to include their relevant
xlocale/_stdlib.h file (I was mostly thinking about config_macros as a hack
to work around it).
-- Sean Silva
Sean Silva <chisophugis@...> writes:
Stephen, thanks for being a brave guinea pig here
No problem!
With the define I get more errors of the form:
$ clang++ -E -fmodules -fmodules-cache-path=cache -D_USE_EXTENDED_LOCALES_
test.cpp
# 1 "test.cpp"
# 1 "<built-in>" 1
# 1 "<built-in>" 3
# 325 "<built-in>" 3
# 1 "<command line>" 1
# 1 "<built-in>" 2
# 1 "test.cpp" 2
While building module 'std' imported from test.cpp:6:
While building module 'Darwin' imported from
/Users/ske/dev/prefix/bin/../include/c++/v1/__config:23:
In file included from <module-includes>:3:
In file included from /usr/include/ctype.h:384:
/usr/include/xlocale/_ctype.h:35:50: error: unknown type name 'locale_t'
unsigned long ___runetype_l(__darwin_ct_rune_t, locale_t);
^
/usr/include/xlocale/_ctype.h:36:53: error: unknown type name 'locale_t'
__darwin_ct_rune_t ___tolower_l(__darwin_ct_rune_t, locale_t);
^
Actually I think I hit similar errors from files in /usr/include when I
tried this a few weeks ago on linux, and I think my solution was to add a
modulemap in /usr/include for a few things. I'll check that in a few days.
This patch allows me to build the libcxx module:
diff --git a/include/ios b/include/ios
index ff79998..e5dc654 100644
--- a/include/ios
+++ b/include/ios
@@ -213,6 +213,7 @@ storage-class-specifier const error_category&
iostream_category() noexcept;
#include <__config>
#include <iosfwd>
+#include <cstdlib>
#include <__locale>
#include <system_error>
Though this stuff is very fiddly, and I have not yet grokked the whole
connection/problem between modules and defines (I haven't passed the 'it
basically works' barrier yet in order to grok that ), so I don't know if
the above patch is the right approach.
I am surprised that this problem has survived as I thought it would be an
obvious "step 1" for anyone trying modules on mac. I guess either it's been
tested so far on a different kind of system, or whoever added the modulemap
for libcxx only tested it on linux. A unit test could be added in libcxx
which attempts to build a module for it, right?
Thanks,
Steve.
Sean Silva <chisophugis@...> writes:
> Stephen, thanks for being a brave guinea pig here
No problem!
With the define I get more errors of the form:
$ clang++ -E -fmodules -fmodules-cache-path=cache -D_USE_EXTENDED_LOCALES_
test.cpp
# 1 "test.cpp"
# 1 "<built-in>" 1
# 1 "<built-in>" 3
# 325 "<built-in>" 3
# 1 "<command line>" 1
# 1 "<built-in>" 2
# 1 "test.cpp" 2
While building module 'std' imported from test.cpp:6:
While building module 'Darwin' imported from
/Users/ske/dev/prefix/bin/../include/c++/v1/__config:23:
In file included from <module-includes>:3:
In file included from /usr/include/ctype.h:384:
/usr/include/xlocale/_ctype.h:35:50: error: unknown type name 'locale_t'
unsigned long ___runetype_l(__darwin_ct_rune_t, locale_t);
^
/usr/include/xlocale/_ctype.h:36:53: error: unknown type name 'locale_t'
__darwin_ct_rune_t ___tolower_l(__darwin_ct_rune_t, locale_t);
^Actually I think I hit similar errors from files in /usr/include when I
tried this a few weeks ago on linux, and I think my solution was to add a
modulemap in /usr/include for a few things.
If you don't modularize bottom-up, the problems you're most likely to run
into are submodule visibility problems. Essentially, when a top-level
module has multiple submodules which try to textually include the same file
(and that file has include guards), only the first submodule to include
that file actually includes its contents (for the rest of them, the include
guards prevent repeated inclusion), so only that first submodule ends up
owning the header's contents. The result is that you can no longer rely on
indirect textual #includes to bring in file contents.
One way to avoid the problem is to avoid using submodules (Sean did this in
some of his experiments, and got some good results). Another way is to
properly modularize bottom-up (which means starting by fixing the module
maps and modularization of your system headers). We've also been discussing
ways that Clang could make this transparent by changing its submodules
model (so that multiple submodules of the same module *could*, in effect,
textually include the same source file).
I'll check that in a few days.
This patch allows me to build the libcxx module:
diff --git a/include/ios b/include/ios
index ff79998..e5dc654 100644
--- a/include/ios
+++ b/include/ios
@@ -213,6 +213,7 @@ storage-class-specifier const error_category&
iostream_category() noexcept;#include <__config>
#include <iosfwd>
+#include <cstdlib>
#include <__locale>
#include <system_error>Though this stuff is very fiddly, and I have not yet grokked the whole
connection/problem between modules and defines (I haven't passed the 'it
basically works' barrier yet in order to grok that
the above patch is the right approach.I am surprised that this problem has survived as I thought it would be an
obvious "step 1" for anyone trying modules on mac. I guess either it's been
tested so far on a different kind of system, or whoever added the modulemap
for libcxx only tested it on linux. A unit test could be added in libcxx
which attempts to build a module for it, right?
As far as I know, the libc++ (and LLVM and Clang) module maps have only
been successfully used on Linux. I don't know whether the problems on Mac
are from a combination of subtly-incorrect module maps and non-modular
system headers, or are caused by Clang bugs (or both).
Sean Silva <chisophugis@...> writes:
> Stephen, thanks for being a brave guinea pig here
No problem!
With the define I get more errors of the form:
$ clang++ -E -fmodules -fmodules-cache-path=cache -D_USE_EXTENDED_LOCALES_
test.cpp
# 1 "test.cpp"
# 1 "<built-in>" 1
# 1 "<built-in>" 3
# 325 "<built-in>" 3
# 1 "<command line>" 1
# 1 "<built-in>" 2
# 1 "test.cpp" 2
While building module 'std' imported from test.cpp:6:
While building module 'Darwin' imported from
/Users/ske/dev/prefix/bin/../include/c++/v1/__config:23:
In file included from <module-includes>:3:
In file included from /usr/include/ctype.h:384:
/usr/include/xlocale/_ctype.h:35:50: error: unknown type name 'locale_t'
unsigned long ___runetype_l(__darwin_ct_rune_t, locale_t);
^
/usr/include/xlocale/_ctype.h:36:53: error: unknown type name 'locale_t'
__darwin_ct_rune_t ___tolower_l(__darwin_ct_rune_t, locale_t);
^
This is probably being caused because you aren't actually supposed to
define _USE_EXTENDED_LOCALES_ on the command line. It's an internal macro,
and defining it on the command line causes the system headers module to be
built with that defined. When the system headers module is being built,
that define is causing ctype.h to behave as though xlocale.h has already
been included, even though it hasn't (locale_t is defined in xlocale.h and
ctype.h ends up needing locale_t when _USE_EXTENDED_LOCALES_ is defined).
-- Sean Silva
I saw this error when I tried out -fmodules on OS X earlier as well, but it disappeared and I took it to be transient and due to something in my build of llvm/clang/libcxx. Now of course I'm seeing it again and I have no idea what allowed -fmodules to work for me before.
I tried Sean's suggestion of "moving `#include <__locale>` below `#include <cstdlib>`". To get things to build I had to put both of the following:
#include <cstdlib>
#include <xlocale/_stdlib.h>
before `#include <__locale>` in the `<locale>` header. Since `<xlocale/_stdlib.h>` is not in a module map (AFAICT, MacOSX10.10.sdk/usr/include/module.map does not mention the xlocale headers in the 10.10 SDK) I believe the included declarations become part of the `<locale>` module...
I saw this error when I tried out -fmodules on OS X earlier as well, but
it disappeared and I took it to be transient and due to something in my
build of llvm/clang/libcxx. Now of course I'm seeing it again and I have no
idea what allowed -fmodules to work for me before.I tried Sean's suggestion of "moving `#include <__locale>` below `#include
<cstdlib>`". To get things to build I had to put both of the following:#include <cstdlib>
#include <xlocale/_stdlib.h>before `#include <__locale>` in the `<locale>` header. Since
`<xlocale/_stdlib.h>` is not in a module map (AFAICT,
MacOSX10.10.sdk/usr/include/module.map does not mention the xlocale headers
in the 10.10 SDK) I believe the included declarations become part of the
`<locale>` module...
You may want to file a bug with Apple about this issue with the system
headers. This issue seems like it would affect C/ObjC (which are officially
supported) just the same as C++.
-- Sean Silva
Done.