[RFC] Avoid Functions Like <foo>.llvm.<...>() For Kernel Live Patch

At thin-lto mode, compilers often promote some static
functions from <foo>() to <foo>.llvm.<...>() to avoid
name conflict. But these <foo>.llvm.<...>() functions
makes live patch difficult and complicated.

Song Liu and I presented what we found in LPC2025.
https://lpc.events/event/19/contributions/2212/

Basically, live patch might cause generating new
functions like <foo>.llvm.<...> and such new functions
do not exist in the kernel without live patch.
This makes things difficult and developers
have to work hard to workaround this to avoid
these new functions.

We did an experiment with a recent kernel with thin-LTO.
There are total 2117 functions like <foo>.llvm.<...>.
All of them are unique without suffix .llvm.<...>.
That is, all these .llvm.<...> suffix can be removed.

In summary, we want to do

  <foo>.llvm.<...>() ==> <foo>()

whenever possible.

Of course, it is possible that some of .llvm.<...> suffix
cannot be removed. But based on the above kernel
experiments, the case with needed .llvm.<...> suffix
should be really rare and then live patch will have large
chance to work in most cases at thin-lto mode.

The above LPC2025 link has more details. Can we have a pass
in llvm to check those <foo>.llvm.<...>() functions if it can
be renamed to <foo>() without any conflict? If the rename is
allowed, llvm can do renaming to <foo>() and this will make
live patch easier (best will be similar to non-lto mode).

Will a consistent object renaming for promotion (across builds) work? @mtrofin has some idea on making that happen.

@orodley picked it up in this RFC which he’s making good progress on - in fact, I think it should be landing shortly.

TL;DR; we compute GUIDs for each function early, and associate it via metadata. Subsequent ThinLTO imports or linkage changes don’t affect this early-assigned GUID.

But I think the OP is rather interested in dropping suffixes, which maybe the GUID stuff can help with (if the justification for adding suffixes was symbol name disambiguation for e.g. sample based profiling), but I think there are other reasons the suffixes are currently needed. @teresajohnson

@xur to comment on this as he has experience with Kernel + ThinLTO + live patching issues

The GUID work that @mtrofin mentioned isn’t really going to affect the suffix issue.

We could presumably strip the suffixes in many cases, when there is a single function per name in the target, and we are LTO linking the whole program, but there are certainly going to be cases where they cannot be stripped. I.e. any time there are same named local functions this isn’t necessarily going to work.

I thought the new GUID proposal was also to introduce stability across different builds. If not, it would solve the livepatch issue.

No, it is to help track the GUID across phases of a single build. The GUID is not used as the suffix. While using the GUID instead would lead to more (but not perfect) name stability across builds, that has issues as in some cases there can be an alias between the GUID between different local copies, depending on the path used for the initial compiles.

For this one:

We could presumably strip the suffixes in many cases, when there is a single
function per name in the target, and we are LTO linking the whole program, but
there are certainly going to be cases where they cannot be stripped. I.e. any
time there are same named local functions this isn’t necessarily going to work.

I understand that some functions (.llvm.<…>) cannot strip suffix “.llvm.<…>”. But in most cases, majority of functions like .llvm.<…> can strip the suffix. This will already be a huge improvement for live patching with thin-lto mode.

I am not sure I fully understood the problem. Is it that the .{suffix} in itself poses a problem to live patching?

I am not sure I fully understood the problem. Is it that the .{suffix} in itself poses a problem to live patching?

For example, the original application has foo1(), foo2() and foo3() in different files. Say in the file with foo1(),
there is a static function bar1().
After thin-lto optimization, three functions still exist() including bar1() and let us say this is base binary.

The live patch process is to

  • add some changes in the application, e.g., foo1() is modified.
  • rebuild the binary, now a new function appears bar1.llvm.<…>().

The live patch process is to compare old and new binaries. In this case, the live
patch find a new function bar1.llvm.<…>() which does not exist in the old binary.
This will make things hard.

If in the new binary, bar1.llvm.<…>() can be renamed back to bar1(), then things
will be much easier for live patch.

So this is mainly used in the offline step to minimize the diff/delta? There might other ways to solve this issue (e.g. recording the original name).

So this is mainly used in the offline step to minimize the diff/delta?

Yes, this is an offline step. But I am not sure we should say the goal is to minimize changes. It is more like “highlight what is actually changed”.

Let me repeat some background so we are on the same page.

kernel live patching at runtime basically works as “replace buggy functions with fixed version”. In the offline step, we need to find which functions are buggy, and get fixed version of them. Usually, people propose the fix as a patch to the source code, but what we really need is changed functions in the binary. To find which functions in the binary are changed by a patch, the kpatch-build toolchain does binary diff: compile the code without the fix, then compile the code with the fix, then compare before and after ELF files and see which functions are changed.

With LTO, we have those .llvm. suffix. In some cases, probably more likely with newer LLVM, we see an object file got a new hash. Then the before ELF file has foo.llvm.<old_hash>, and the after ELF file has foo.llvm.<new_hash>. The binary diff tool is totally confused by this, and thinks foo.llvm.<new_hash> is a new function, and previous foo.llvm.<old_hash> is somehow disappeared.

Because the binary diff is done for the ELF files, I am not sure whether recording original name can help here. OTOH, I think removing .llvm. suffix is an easier approach, and should work better.

Note that, we don’t need a perfect solution that removes all the .llvm. suffixes. If there are foo.llvm.<hash_1> and foo.llvm.<hash_2> from two different files, we can just keep the suffixes for both. Data shows that this is a really rare case anyway.

Also, the .llvm. suffix has caused problems for kernel tracing. If we can remove most of them, it will also make things easier for these users.

Sounds like we don’t need to remove the suffix, we just need the hash to be stable across multiple compiles when nothing changes. So the inputs of the hashing would need to change.

On Fri, Jan 9, 2026, 7:37 PM Song Liu <notifications@llvm.discoursemail.com> wrote:

Someone replied to a topic you are Watching.

liu-song-6
January 10

So this is mainly used in the offline step to minimize the diff/delta?

Yes, this is an offline step. But I am not sure we should say the goal is to minimize changes. It is more like “highlight what is actually changed”.

Let me repeat some background so we are on the same page.

kernel live patching at runtime basically works as “replace buggy functions with fixed version”. In the offline step, we need to find which functions are buggy, and get fixed version of them. Usually, people propose the fix as a patch to the source code, but what we really need is changed functions in the binary. To find which functions in the binary are changed by a patch, the kpatch-build toolchain does binary diff: compile the code without the fix, then compile the code with the fix, then compare before and after ELF files and see which functions are changed.

With LTO, we have those .llvm. suffix. In some cases, probably more likely with newer LLVM, we see an object file got a new hash. Then the before ELF file has foo.llvm.<old_hash>, and the after ELF file has foo.llvm.<new_hash>. The binary diff tool is totally confused by this, and thinks foo.llvm.<new_hash> is a new function, and previous foo.llvm.<old_hash> is somehow disappeared.

Because the binary diff is done for the ELF files, I am not sure whether recording original name can help here. OTOH, I think removing .llvm. suffix is an easier approach, and should work better.

Note that, we don’t need a perfect solution that removes all the .llvm. suffixes. If there are foo.llvm.<hash_1> and foo.llvm.<hash_2> from two different files, we can just keep the suffixes for both. Data shows that this is a really rare case anyway.

Also, the .llvm. suffix has caused problems for kernel tracing. If we can remove most of them, it will also make things easier for these users.


Visit Topic or reply to this email to respond.

To unsubscribe from these emails, click here.

we just need the hash to be stable across multiple compiles when nothing changes

I think we already have this, but it is not sufficient.

For example, we have foo and bar in the same file. Before the patch, they are named foo.llvm.<old_hash> and bar.llvm.<old_hash>. If the patch fixes foo, but do not touch bar, we want both foo and bar to keep the same <old_hash> after the patch. If either of them got a new hash, the kpatch-build toolchain will be confused.

Also, we have seen cases where a function/variable does not have a suffix before a patch, but gets a suffix after the patch. The kpatch-build is also confused in these cases.

Given most of these symbols are unique without suffix anyway, I think it is still better to just remove these suffixes.

1 Like

Using content hash can lead to the problem. Path based hashing is stable (cross build), but it may have its own problems.

Overall, stripping unneeded suffixes can improve debuggability of thinlto binaries, which is an additional benefit.

1 Like

Overall, stripping unneeded suffixes can improve debuggability of thinlto binaries, which is an additional benefit.

Any recommendation how to proceed to do implementation if the community agrees?

A general solution independent of thinLTO is to annotate those suffixed functions with their original names. At link time, linker or some offline too can do the renaming for any original function name that do not have conflicting suffixed names.

Could you be more specific about the following:

At link time, linker or some offline too can do the renaming for any original function name that do not have conflicting suffixed names.

It would be great if you can help identify some places in llvm code base to implement the above suggestions.

@davidxl @xur Ping. Any suggestions/guidelines on how to remove unnecessary suffixes with thin-lto?

There seem to be an option to use path based suffix naming and work for kernel build. Is that a solution?

If we want to go down this path of avoiding suffix. One way is mark local functions with unique names during thin_link and skip renaming when they are promoted. The other one is record the original names for promoted symbols in an separate ELF section which needs LLD involvement. To limit the scope of the change, a thinlto specific solution (first one) may be preferred.

This is just my opinion. You may want to hear feedbacks from others @teresajohnson .

There seem to be an option to use path based suffix naming and work for kernel build. Is that a solution?

Technically, yes, it resolves this issue. But it makes the kernel symbol table really ugly. And it doesn’t really address potential duplicated kernel symbols. Someone proposed a similar suffix solution to fully avoid duplicated kernel symbols [1]. But people really don’t like this idea.

[1] https://lpc.events/event/18/contributions/1730/