[RFC] Add "hot" function attribute to LLVM IR and use hot/cold attribute in function section prefix

[RFC] Add “hot” function attribute to LLVM IR and use hot/cold attribute
in function section prefix

attribute((hot)) and attribute((cold)) are supported in Clang.
Only the “cold” attribute is passed down to LLVM IR. “hot” attribute is
silently dropped by Clang, as currently only have cold function attribute
in LLVM IR.

These two attributes are not documented in clang reference manual:
https://clang.llvm.org/docs/AttributeReference.html
They are here mainly for gcc-compatibility:
https://gcc.gnu.org/onlinedocs/gcc-4.7.2/gcc/Function-Attributes.html

We propose to add a “hot” function attribute to LLVM IR. Like the “cold”
function attribute, “hot” attribute can be used in optimization passes.

The immediate usage we are having is to set the function section prefix/suffix.
Currently “.hot” and “.unlikely” suffix only are added in PGO (or Sample PGO)
compilation (through isFunctionHotInCallGraph and isFunctionColdInCallGraph).

With the hot/cold function attribute, the new behavior is:
(1) If the user annotates a function as hot or isFunctionHotInCallGraph
is true, this function will be marked as hot.
(2) If the user annotates a function as cold or isFunctionColdInCallGraph
is true, this function will be marked as cold.

The changes are:
(1) user annotation will be used in setting function section prefix/suffix.
(2) hot attribute overwrites profile count based hotness.
(3) profile count based hotness overwrite user annotated cold attribute.

The intention for these changes is to provide the user a way to mark
certain functions as hot in cases where training input is hard to cover all
the hot functions.

Note that with this change, the attributes will have a different semantic
from GCC, where PGO profile hotness always has precedence over the function
attribute annotation.

Currently the cold function attribute is also set in PGO use pass. The code is before isFunctionColdInCallGraph and does not use PSI. We might want to change that code
to make this attribute as user-annotation only.

Comments and feedback are welcome. The patch for this RFC is:
https://reviews.llvm.org/D92493

Thanks,

-Rong