Feedback on feature (and status of lld linker script support?)

Hi all,

I emailed the llvm mailing list but have not heard any response yet,
but maybe I was sent to the wrong list. Hopefully this is the right
one.

TLDR:
1) Is anyone available to provide some quick feedback on a feature [1]
before I submit it?
2) What is the status of linker script support for LLD on macOS?

I'm a bit of a newcomer to clang / llvm development and I'm working on
a feature that I hope will improve the experience on macOS for
applications that rely on custom section placement. Specifically, it
addresses this type of error:

error: argument to 'section' attribute is not valid for this target:
mach-o section specifier requires a section whose length is between 1
and 16 characters

The solution I've put together in [1] simply hashes long section names
to fit into N (16) characters. The approach is SHA256 -> base64 ->
truncate to N which is already somewhat common in various runtimes on
macOS.

I've added two options:
* -fhash-section-names=N
  * hashes section names when they are > N characters
* -fhashed-section-names=outputfile
  * outputs a mapping of all section names to the hash value in tabbed
CSV format
  * each line is of the form: (section)\t(hash)
  * would be nice to reuse any artifact file names if "outputfile" is ""
  * e.g. build/foo.o => build/foo.sn, build/foo.exe => foo.sn,
build/foo => foo.sn

The above changes allow one to use linker-generated symbols to iterate
over all elements placed into a particular section. It is working
quite well for local "native_posix_64" builds of Zephyr so far. Zephyr
uses (long) custom section names extensively. Otherwise the section
name size limitation of macOS is a showstopper.

Does anyone have any quick feedback before I submit this patch to the
mailing list?

The second question I had was about LLD support for GNU style linker
scripts on macOS. I'm currently getting this error:

ld64.lld: error: zephyr/linker_zephyr_pre0.cmd: unhandled file type

Is there an option I can provide to LLD that would provide more
information about the error? Currently I'm using "-Wl,-flat_namespace,
-Wl,-undefined,warning" just to make it link, but there are several
symbols defined in the linker script that are required for the
application to run properly. Supporting GNU style linker scripts would
be immensely helpful in this case, so I'd like to make that work if at
all possible. Would be happy to patch LLD as well.

Thanks,

C

[1] https://github.com/llvm/llvm-project/compare/main...cfriedt:fhash-long-section-names

+Fangrui Song for general LLD stuff
+Jez Ng for ld64.lld stuff
+Reid Kleckner for discussion around section/symbol reduction

I’m guessing it’ll be best to keep each issue in a separate thread - and the previous thread on llvm-dev ( https://groups.google.com/g/llvm-dev/c/YzHbrfTxLCY/m/ssYtlxIdAwAJ )

The section name hashing might be more of a clang feature than an LLVM one - if it’s specifically for user-specified sections, etc. It could have some overlap with some symbol name length issues I’ve been considering for a while with somewhat similar strategies (hash, but then have a lookup table). I’m not sure who’s going to be best to actually make the final call on whether what you’re proposing is suitable for Clang/LLVM

+Fangrui Song <maskray@google.com> for general LLD stuff
+Jez Ng <jezng@fb.com> for ld64.lld stuff
+Reid Kleckner <rnk@google.com> for discussion around section/symbol
reduction

I'm guessing it'll be best to keep each issue in a separate thread - and
the previous thread on llvm-dev (
https://groups.google.com/g/llvm-dev/c/YzHbrfTxLCY/m/ssYtlxIdAwAJ )

The section name hashing might be more of a clang feature than an LLVM one
- if it's specifically for user-specified sections, etc. It could have some
overlap with some symbol name length issues I've been considering for a
while with somewhat similar strategies (hash, but then have a lookup
table). I'm not sure who's going to be best to actually make the final call
on whether what you're proposing is suitable for Clang/LLVM

(I use this subthread for the 16-byte section name length limitation.)

It would be nice to fix the 16-byte section name limitation for Mach-O.
For many instrumentations, for binary format portability, we have to restrict the section name length even
when the length size doesn't cost that much.
I think this probably should be fixed at the binary format level.
For example, PE/COFF had a 7-char limitation and extended it with

("For longer names, this field contains a slash (/) that is followed by
an ASCII representation of a decimal number that is an offset into the
string table.")

The rationale is that this is a limitation that the binary format maker
would likely have an opinion on and needs to change many things in the
ecosystem, e.g.

* does the assembler need to do anything special?
* how do otool/llvm-objdump/llvm-readobj render the section name?
* how do ld64/ld64.lld render the section name, especially in diagnostics
* whether the custom section can be stripped by binary manipulation tools
* how section$start$__SEG$__section symbols work with the section representation
* ...

If the section name length is a problem, the binary format maker can use
compression on the section name part (see __llvm_prf_nm).

(I use this subthread for the 16-byte section name length limitation.)

It would be nice to fix the 16-byte section name limitation for Mach-O.
For many instrumentations, for binary format portability, we have to restrict the section name length even
when the length size doesn't cost that much.
I think this probably should be fixed at the binary format level.

I agree - it really should be fixed at Mach-O spec level.

For example, PE/COFF had a 7-char limitation and extended it with
PE Format - Win32 apps | Microsoft Docs
("For longer names, this field contains a slash (/) that is followed by
an ASCII representation of a decimal number that is an offset into the
string table.")

I thought of something similar - providing a key in the MSByte (0xff)
- could be used to indicate a binary offset to the string table and it
would work equally well for either 64 or 32-bit headers. However, that
solution really only works if Apple were to adopt it and amend the
Mach-O spec. Does something like that already exist on Macs? Is there
anyone at Apple who would be willing to do that?

The rationale is that this is a limitation that the binary format maker
would likely have an opinion on and needs to change many things in the
ecosystem, e.g.

* does the assembler need to do anything special?
* how do otool/llvm-objdump/llvm-readobj render the section name?
* how do ld64/ld64.lld render the section name, especially in diagnostics
* whether the custom section can be stripped by binary manipulation tools
* how section$start$__SEG$__section symbols work with the section representation
* ...

I agree, and that's good long-term thinking to push Apple to amend
Mach-O. I can see it taking a while to get Apple on board with
something like that. Other tooling would just see hashed strings,
which isn't ideal.

However, the proposed solution works today and is quite handy for C
and C++. It's a nice, high-level source to source transform that
produces fully legal section names with Mach-O or really any other
binary format. Good to hear that others are thinking about that
problem as well.

Good chatting and I hope we can keep the conversation going.

Will the LLD conversation take place on llvm-dev?

Thanks,

C

Good to know that I'm not the only one thinking about this stuff,
hehe. I was planning on using it for user-specified sections, yea.

Hopefully this or something like it could be upstreamed. It's not
ideal because it's only hiding the root problem (the Mach-O spec) but
at least it's a short term solution.

Almost as critical, imho, is supporting GNU linker scripts with LLD
for macOS. What are the main limiting factors right now preventing
ld64.lld from borrowing most of the GNU linker script support from
ld.lld? Or is that a conversation for a different mailing list?

Thanks for your response,

C

For ld64.lld, we're aiming to be compatible with ld64 (the standard platform linker for Apple). ld64 doesn't support linker scripts, and we had no plans of supporting them either.

I'd be super reluctant to add support for linker scripts unless there's a very compelling use case for them. They add a lot of complication to the linker, and my understanding is that many aspects of them are ill-defined (bfd, gold, and LLD might do different things for the same input linker script). There's also no precedent of linker script support for Apple platforms that I'm aware of.

(llvm-dev is the right mailing list for LLD discussions, so I moved this to llvm-dev and BCC'd cfe-dev.)

    > The section name hashing might be more of a clang feature than an LLVM one - if it's specifically for user-specified sections, etc. It could have some overlap with some symbol name length issues I've been considering for a while with somewhat similar strategies (hash, but then have a lookup table). I'm not sure who's going to be best to actually make the final call on whether what you're proposing is suitable for Clang/LLVM

    Good to know that I'm not the only one thinking about this stuff,
    hehe. I was planning on using it for user-specified sections, yea.

    Hopefully this or something like it could be upstreamed. It's not
    ideal because it's only hiding the root problem (the Mach-O spec) but
    at least it's a short term solution.

    Almost as critical, imho, is supporting GNU linker scripts with LLD
    for macOS. What are the main limiting factors right now preventing
    ld64.lld from borrowing most of the GNU linker script support from
    ld.lld? Or is that a conversation for a different mailing list?

    Thanks for your response,

    C

    Almost as critical, imho, is supporting GNU linker scripts with LLD
    for macOS. What are the main limiting factors right now preventing
    ld64.lld from borrowing most of the GNU linker script support from
    ld.lld? Or is that a conversation for a different mailing list?

The code to parse linker scripts could be borrowed from the ELF ld.lld but that would only be one small piece of the puzzle. AFAIK the GNU linker doesn't support Mach-O so there would have to be a mapping worked out from GNU linker scripts to Mach-O. This is possible but a lot of work and experimentation will be needed to work out what makes sense. For example something like PHDRS is unlikely to cross over from ELF. Then there's the complexity of what linker scripts can do to the linker code. A lot of corner case code has to be written to account for what can be written in a linker script.

Given the lack of precedent in Mach-O for linker scripts it may be better to look at what specific needs there are and try to look at alternative ways of supporting them rather than adopting GNU linker scripts. For example are scripts needed for linker defined symbols? section/atom ordering within a section?

Peter

The code to parse linker scripts could be borrowed from the ELF ld.lld but that would only be one small piece of the puzzle. AFAIK the GNU linker doesn't support Mach-O

I think you're right - even though GNU binutils bfd, gas, etc supports
Mach-O, IIRC, it calls out to Apple's ld to actually do the linking.

so there would have to be a mapping worked out from GNU linker scripts to Mach-O. This is possible but a lot of work and experimentation will be needed to work out what makes sense. For example something like PHDRS is unlikely to cross over from ELF. Then there's the complexity of what linker scripts can do to the linker code. A lot of corner case code has to be written to account for what can be written in a linker script.

Agreed... there are many corner cases with GNU linker scripts, in part
because they are fairly powerful.

Given the lack of precedent in Mach-O for linker scripts it may be better to look at what specific needs there are and try to look at alternative ways of supporting them rather than adopting GNU linker scripts.

I did look at the "--order-file" option briefly, but I'm not entirely
sure if it's exactly the same concept or if it has any of the same
features of GNU linker scripts. Any idea?

For example are scripts needed for linker defined symbols? section/atom ordering within a section?

As long as the section name is a valid C identifier, linker scripts
are not needed for linker-defined symbols with GCC (presumably also
Clang + ld.lld). A corresponding __start_SECTION and __stop_SECTION
symbol would automatically be generated (discarded if unused).

That's not the case with ld64.lld, which requires a
__asm("section$start$SEGMENT$SECTION") or
__asm("section$end$SEGMENT$SECTION"). I'm not sure if that has always
been the case with macOS or if it's more of a recent thing since the
switch to Clang.

However, there are other cases where it is either not desired or
impossible to use default linker generated start and stop symbols.
E.g.

devices :
{
  __device_start = .;
  __device_PRE_KERNEL_1_start = .;
  KEEP(*(SORT(.z_device_PRE_KERNEL_1[0-9]_*)));
  KEEP(*(SORT(.z_device_PRE_KERNEL_1[1-9][0-9]_*)));
  __device_PRE_KERNEL_2_start = .;
  KEEP(*(SORT(.z_device_PRE_KERNEL_2[0-9]_*)));
  KEEP(*(SORT(.z_device_PRE_KERNEL_2[1-9][0-9]_*)));
  __device_POST_KERNEL_start = .;
  KEEP(*(SORT(.z_device_POST_KERNEL[0-9]_*)));
  KEEP(*(SORT(.z_device_POST_KERNEL[1-9][0-9]_*)));
  __device_APPLICATION_start = .;
  KEEP(*(SORT(.z_device_APPLICATION[0-9]_*)));
  KEEP(*(SORT(.z_device_APPLICATION[1-9][0-9]_*)));
  __device_SMP_start = .;
  KEEP(*(SORT(.z_device_SMP[0-9]_*)));
  KEEP(*(SORT(.z_device_SMP[1-9][0-9]_*)));
  __device_end = .;
}

It sounds as though adding linker script support would almost mean
writing an entirely new linker. Could be an interesting experiment,
but I have to be careful to not go too far down this rabbit hole.