Hi @mingmingl-llvm, really exciting work!
We have been also working on similar memory profile guided data symbols ordering on MachO binaries, which mainly focuses on improving mobile app startup performance by reducing page faults with hot/code splitting.
Specifically,
(1) we added memprof support for Darwin platform, and a new functionality to dump addresses belonging to loaded binaries/modules whose counter is not 0. Note that we change the MEM_GRANULARITY to 8 bytes, because according to our analysis, most data symbols sizes fall into 8 - 64 bytes in mobile apps.
(2) by running the memprof instrumented mobile apps, we can get the binary access profile for app startup.
(3) Then we need to translate/map the runtime addresses to symbols, so we know what the hot data symbols that are accessed at app startup are. We have also considered using dwarf debug info, but we are using linker maps for simplicity for now.
(4) we then order the regular mobile apps’ data symbols in lld based on the symbol order file generated in step 3. We use the existing -order_file to order the symbolizable data with stable names, mostly in __DATA_CONST,__const and __TEXT,__const in MachO. We added a similar -order_file_cstring for string literals ordering (will upstream soon) in lld and use the hash of the string literal to represent and match them in the order file. This mostly orders __TEXT,__cstring in MachO.
Our experiments on a large social app show that, the work can reduce the pre-main startup time by 11-12%, and page faults by 5%.
We face the same issue of matching un-symbolizable data, such as the linker local symbol starting with l_; the symbolizable global variables with non-unique names. For the latter, we are also thinking identifying them with the aid of their source location, but we are leaning to the idea of appending the hashes of their location info, e.g. file name, class name etc to their symbol names.
I also chatted about this work with @teresajohnson at the LLVM Dev meeting earlier this year.
Are you planning to upstream the code any time soon? We are interested to see how the new symbolized data access profiles are integrated into the existing memprof profile format.
Thanks!
Sharon