[RFC] Implementing the sanitizer runtimes for heterogeneous devices

Hi all,

We are thinking of extending the LLVM sanitizers, starting with asan, to heterogeneous situations

such as those found in OpenCL and HIP. We have already started thinking about the way

instrumentation will look like for different address spaces and posted a small RFC

(http://lists.llvm.org/pipermail/llvm-dev/2020-July/143310.html) about removing the generated

inttoptr and ptrtoint pair from instrumented IR. We have started to look into the runtime

implementation, and we will be happy to get feedback and suggestion from the community so that we

can incorporate them in the design itself. We are looking at implementing a sanitizer runtime that

would support heterogeneous devices such as AMD GPUs and we present some directions that we can take

to make that happen:

  1. Add support for compilation of the compiler-rt sanitizer runtime by a device compiler

We propose to update the sanitizer parts of compiler-rt so they can additionally be compiled

with the HIP compiler and generate device code. This would be our first choice. It provides

obvious advantages over keeping separate sanitizer runtimes for heterogeneous devices and

easier maintenance while ensuring feature parity. This can be broadly summarized in the

following activities:

a. Supporting device code generation

We propose to add suitably protected changes which allow the code to be compiled for the

HIP language and generate device code.

b. Supporting device and host communication

Sanitizer features like violation reporting requires communication from device to host.

We propose to add support for the device code to communicate with the host for

reporting. We see this going into a target specific sub-folder.

c. Adding new interceptors

We foresee a need for new interceptors, for example, device specific memory allocators.

We propose to add these interceptors.

d. Updating the build system

We propose to update the build system to build device code when requested, as well as

host code.

  1. Implement the device side sanitizer runtime separately

In this scenario, the device side of the sanitizer runtime would be implemented in a

separate folder, possibly in a separate repository. For example, this could be added to our

existing device side runtime. The changes mentioned in 1b and 1c to the existing host

runtime would still be required.

Combinations of different approaches listed above are also possible. We will be happy to hear about

other approaches and ideas to move forward with them as well. Adding direct support for

heterogeneous device sanitizer runtime in compiler-rt will also be helpful for other heterogeneous

devices in future.

Many thanks,

Reshabh and Brian

Hi all,

Following approach #2 from the previous RFC, we are preparing some updates to the address sanitizer

runtime and we would like to get some feedback on how to approach these changes.

In order to isolate the impact of our changes on others, we are thinking to guard our changes inside a

distinguishable macro, for example, SANITIZER_AMDGPU, and to localize our changes as much as possible

within new files whose name includes AMDGPU. We would your feedback on this, and suggestions

about how to trigger the definition of the macro. The host runtime will always be targeted to the host

which implies that we will not be able to check the target architecture and set the macro, like other

architectures. We propose two ways to achieve this: