Visibility in libc++

Clang Frontend
#1

Hi,

libc++ applies visibility default to all public types and functions, including templates. On Windows these visibility macros expand to dllexport/import which makes no sense for templates (If someone uses a vector<int> it is then expected that something else is exporting such a specialization).

Before I send a patch I have two questions:

1. AFAICS the visibility attributes were added for cases when the
    libc++ headers are included inside of visibility scopes. Could
    someone clarify why visibility is needed on header-only templates?

2. If these are indeed needed, then templates require a different macro
    that only expands to visibility. I currently use
    _LIBCPP_(TYPE|FUNC)_PUBLIC for both, and _LIBCPP_(TYPE|FUNC)_VIS for
    only visibility. Thoughts?

-Nico

#2

Hi,

libc++ applies visibility default to all public types and functions, including templates. On Windows these visibility macros expand to dllexport/import which makes no sense for templates (If someone uses a vector<int> it is then expected that something else is exporting such a specialization).

Before I send a patch I have two questions:

1. AFAICS the visibility attributes were added for cases when the
  libc++ headers are included inside of visibility scopes. Could
  someone clarify why visibility is needed on header-only templates?

On OS X, there is a flag -fvisibility=hidden that can be put on the command line. This makes everything not marked otherwise, private to a dynamic library.

Also on this platform exception catching/throwing, and dynamic_cast uses the type_info of each type in its implementation.

Furthermore, it is legal C++ to throw/catch *any* object.

So when -fvisibility=hidden is given on the command line, we want clients to still be able to catch / throw std::string and std::vector<int> across dylib boundaries. The _LIBCPP_TYPE_VIS marks each public type/template to make this work. It marks the type_info of these objects as "default" even though the client has asked us not to.

_LIBCPP_TYPE_VIS is used only for non-templated functions which are part of the public libc++ API. This ensures that this API is visible even when the client specifies -fvisibility=hidden.

_LIBCPP_INLINE_VISIBILITY is used to augment inline functions, templated or not. It is currently set to give hidden visibility and __always_inline__. The aim is to make inline functions never be outlined into the libc++.dylib and become part of its ABI. The aim is also to reduce the number of symbols with default visibility. But I'm having second thoughts on the __visibility__("hidden") part of this.

_LIBCPP_ALWAYS_INLINE and _LIBCPP_INLINE_VISIBILITY are currently synonyms, and this needs to be cleaned up (pick one and stick with it).

All of these macros are set up in such a way that the client can override their definition with a -D on the command line (at least on OS X, I notice now this is not true on Windows).

2. If these are indeed needed, then templates require a different macro
  that only expands to visibility. I currently use
  _LIBCPP_(TYPE|FUNC)_PUBLIC for both, and _LIBCPP_(TYPE|FUNC)_VIS for
  only visibility. Thoughts?

I don't know how Windows works, but others here do. I don't think exceptions are working yet with clang on Windows, but I could be wrong. It would be good to know how clang/exceptions are going to work on Windows before too much libc++ design on Windows happens. If we need some macro redesign work in libc++ in order to get libc++ working on Windows with clang, exceptions and visibility, I'm open to that. Naturally we can't break other platforms in the process. Someone who can test on Windows will have to drive it, and that isn't me.

Howard

#3

[... explanation for class template visibility ...]

So if I understand this correctly, the reason is that the Itanium ABI requires typeinfos to compare via their addresses, which breaks if a class template might be hidden in case libc++ headers get non-default visibility applied. And a user would have to explicitly specify the visibility for template instantiations (at least I think that this is possible).

I don't know how Windows works, but others here do. I don't thinkexceptions
are working yet with clang on Windows, but I could be wrong. It would be good
to know how clang/exceptions are going to work on Windows before too much libc++
design on Windows happens. If we need some macro redesign work in libc++ in
order to get libc++ working on Windows with clang, exceptions and visibility,
I'm open to that. Naturally we can't break other platforms in the process.
Someone who can test on Windows will have to drive it, and that isn't me.

I don't think exceptions affect libc++ much in this regard. If the exception is in a shared library, and not header-only, it must be properly exported and imported. And due to how DLLs work, equal typeinfos can have different addresses and must use strcmp for comparison.

I currently have libc++ running as a DLL, and it passes all but 361 tests. Most of these fail due to missing exception support or broken vtables, which are C++ ABI issues. For the rest I haven't implemented the necessary libc functions yet.

I encountered the following problems:

1. As said before, public templates have visibility applied which
    expands to import/export. Such templates must never be decorated
    with import/export because libc++ does obviously not export all
    instantiations (example: vector<T>).

2. A lot of internal globals, types, functions and specializations
    (internal as in those prefixed with "__") are undecorated, but
    must be imported/exported. And I think these must also have
    visibility applied (a few like class __thread_id already do).

    Examples:
    - extern __ph<1> _1;
    - __rs_default __rs_get()
    - template<> __codecvt_utf8<wchar_t>;

3. Extern template instantiations are undecorated but must be
    imported/exported.

    Examples:
    - extern template class __vector_base_common<true>;
    - extern template class class basic_istream<char>;

4. A few public types and functions are undecorated.

    I'm listing all I found to ensure it's ok to apply visibility
    to them. They are:

    exception:
    - exception_ptr current_exception()
    - void rethrow_exception(exception_ptr)
    future:
    - template<> class promise<void>
    memory:
    - void declare_reachable(void*)
    - void declare_no_pointers(char*, size_t)
    - void undeclare_no_pointers(char*, size_t)
    - pointer_safety get_pointer_safety() noexcept
    string:
    - void* align(size_t, size_t, void*&, size_t&)
    - narrow/wide versions of stoi, stol, stoul, stoll, stoull, stof,
      stod, stold
    - to_string, to_wstring
    system_error:
    - const error_category& generic_category() noexcept
    - const error_category& system_category() noexcept
    thread:
    - void sleep_for(const chrono::nanoseconds&)
    iostream.cpp:
    - cin, cout, cerr, clog and their wide counterparts (their
      declarations in <iostream> are decorated, but iostream.cpp
      does not include <iostream>, and thus the definitions are
      not imported/exported)

5. Exception classes/functions

    What exactly does _LIBCPP_EXCEPTION_ABI mean? At first I thought it
    applies to things defined in libc++abi, but it's also found
    on other types like regex_error, bad_weak_ptr, future_error etc.

    Also, the main exception classes like logic_error, runtime_error
    etc. are split between libc++ and libc++abi which requires importing
    one half and exporting the other half. This can be done by decorating
    the members explicitly.

    In any case there must be a distinction whether a header is used to
    compile libc++abi or if it's used to compile something using
    libc++abi.

Because a distinction between public and internal entities seems unnecessary, just one additional macro is needed that only ever expands to type visibility (case A):

A. Default type visibility (for templates)
B. Default type visibility + import/export (for public classes and
    public extern template instantiations)
C. Default visibility + import/export (for other public/internal
    entities)

I've named A. "_LIBCPP_TYPE_VIS_ONLY" so far, but am not exactly happy with that name.

Depending on how case 5 will be resolved, additional macros might be needed, but they don't affect cases A-C.

-Nico

#4

> [... explanation for class template visibility ...]

So if I understand this correctly, the reason is that the Itanium ABI requires typeinfos to compare via their addresses, which breaks if a class template might be hidden in case libc++ headers get non-default visibility applied. And a user would have to explicitly specify the visibility for template instantiations (at least I think that this is possible).

I don't know how Windows works, but others here do. I don't thinkexceptions
are working yet with clang on Windows, but I could be wrong. It would be good
to know how clang/exceptions are going to work on Windows before too much libc++
design on Windows happens. If we need some macro redesign work in libc++ in
order to get libc++ working on Windows with clang, exceptions and visibility,
I'm open to that. Naturally we can't break other platforms in the process.
Someone who can test on Windows will have to drive it, and that isn't me.

I don't think exceptions affect libc++ much in this regard. If the exception is in a shared library, and not header-only, it must be properly exported and imported. And due to how DLLs work, equal typeinfos can have different addresses and must use strcmp for comparison.

I currently have libc++ running as a DLL, and it passes all but 361 tests. Most of these fail due to missing exception support or broken vtables, which are C++ ABI issues. For the rest I haven't implemented the necessary libc functions yet.

I encountered the following problems:

1. As said before, public templates have visibility applied which
  expands to import/export. Such templates must never be decorated
  with import/export because libc++ does obviously not export all
  instantiations (example: vector<T>).

2. A lot of internal globals, types, functions and specializations
  (internal as in those prefixed with "__") are undecorated, but
  must be imported/exported. And I think these must also have
  visibility applied (a few like class __thread_id already do).

  Examples:
  - extern __ph<1> _1;
  - __rs_default __rs_get()
  - template<> __codecvt_utf8<wchar_t>;

3. Extern template instantiations are undecorated but must be
  imported/exported.

  Examples:
  - extern template class __vector_base_common<true>;
  - extern template class class basic_istream<char>;

4. A few public types and functions are undecorated.

  I'm listing all I found to ensure it's ok to apply visibility
  to them. They are:

  exception:
  - exception_ptr current_exception()
  - void rethrow_exception(exception_ptr)
  future:
  - template<> class promise<void>
  memory:
  - void declare_reachable(void*)
  - void declare_no_pointers(char*, size_t)
  - void undeclare_no_pointers(char*, size_t)
  - pointer_safety get_pointer_safety() noexcept
  string:
  - void* align(size_t, size_t, void*&, size_t&)
  - narrow/wide versions of stoi, stol, stoul, stoll, stoull, stof,
    stod, stold
  - to_string, to_wstring
  system_error:
  - const error_category& generic_category() noexcept
  - const error_category& system_category() noexcept
  thread:
  - void sleep_for(const chrono::nanoseconds&)
  iostream.cpp:
  - cin, cout, cerr, clog and their wide counterparts (their
    declarations in <iostream> are decorated, but iostream.cpp
    does not include <iostream>, and thus the definitions are
    not imported/exported)

5. Exception classes/functions

  What exactly does _LIBCPP_EXCEPTION_ABI mean? At first I thought it
  applies to things defined in libc++abi, but it's also found
  on other types like regex_error, bad_weak_ptr, future_error etc.

It is roughly equivalent to _LIBCPP_TYPE_VIS, except that it applies only to those types that are std-defined exception types. The intended use case is that you might want to treat the visibility of the exception types differently than other types. By default they are treated the same.

  Also, the main exception classes like logic_error, runtime_error
  etc. are split between libc++ and libc++abi which requires importing
  one half and exporting the other half. This can be done by decorating
  the members explicitly.

  In any case there must be a distinction whether a header is used to
  compile libc++abi or if it's used to compile something using
  libc++abi.

We do this in at least one place. Search for _LIBCPP_BUILDING_MEMORY.

Because a distinction between public and internal entities seems unnecessary, just one additional macro is needed that only ever expands to type visibility (case A):

A. Default type visibility (for templates)
B. Default type visibility + import/export (for public classes and
  public extern template instantiations)
C. Default visibility + import/export (for other public/internal
  entities)

I've named A. "_LIBCPP_TYPE_VIS_ONLY" so far, but am not exactly happy with that name.

Depending on how case 5 will be resolved, additional macros might be needed, but they don't affect cases A-C.

I believe the way to proceed is for you to post a patch for review.

Howard