New to LLVM: Seeking Guidance on Adding a Custom Built-in Function for RISC-V"

Beginners
1

Hello LLVM Community,

I’m very new to LLVM and have just started learning it.

I’m currently working on an LLVM compiler project for the RISC-V architecture and could use some advice on adding a new built-in function to Clang/LLVM that involves configuring a hardware accelerator. Here’s the prototype:

void __builtin_riscv_custom(__uint64_t q, __uint32_t *pa, __uint32_t *pb, __uint32_t *pc);

When called, this function should generate the following assembly code:

sd a0, -24(s0)
sd a1, -32(s0)
sd a2, -40(s0)
sd a3, -48(s0)
accelerator.add

In this context, a0 to a3 are RISC-V argument registers, and accelerator.add is a custom assembly instruction designed to configure the “add” module of a hardware accelerator.

I’m very new to LLVM and have just started learning it. While I understand how to add this instruction, I’m struggling with the necessary front-end and middle-end modifications, specifically:

  1. Defining a new IR intrinsic corresponding to the built-in function in Clang’s front end.
  2. Mapping the IR to the assembly instruction (SelectionDAG-related work).

I came across a relevant suggestion pointing to this particular commit:

Commit

However, I’m having trouble understanding how this commit relates to my specific issue. Apologies if I’m not describing the problem accurately—it’s due to my newness to LLVM. I’d greatly appreciate any guidance, tips, or relevant documentation/examples on adding new built-ins and defining new IR intrinsics.

Thanks so much for your time!

2

How does s0 relate to the arguments to the intrinsic? What precisely does accelerator.add do?

4

@jrtc27 , Thank you for the reply.

How does s0 relate to the arguments to the intrinsic?

In the RISC-V calling convention, s0 is typically used as a frame pointer register. When calling the __builtin_riscv_custom function, the arguments q , pa , pb , and pc are passed through the argument registers a0 , a1 , a2 , and a3 , respectively.

What precisely does accelerator.add do?
Assume that a0, a1 are input vector pointers and a2 is an output vector pointer. a3 can be the length of the vectors. “accelerator.add” might do vector addition based on a0, a1 input vectors and then put back the result on a2.

5

So why does your intrinsic need to generate stores of a0 - a3 to s0 at those specific offsets? It sounds to me like it doesn’t, and instead you just need LLVM to know that registers are being used and clobbered (and thus might spill it to the stack, but if, how and when that’s done isn’t the intrinsic’s concern), but that’s what Uses and Defs on the instruction in TableGen are for.

6

Hi @jrtc27 , sorry for the delayed reply

I simplified my C Builtin to not take arguments. Therefore, I no longer need to worry about the a0-a3 registers. When the intrinsic calls, the following instruction is executed:

accelerator.add

void __builtin_riscv_accelerator_add();

The changes I made are somewhat extensive. You can view them as a git diff in the following file on my Google Drive:

(Sorry, I have to share the changes this way.)

(This changes include adding C Builtin and adding new assembly instruction accelerator.add, which I want to execute using C Builtin)
I can build LLVM without any errors.

However, when I compile a program using the __builtin_riscv_accelerator_add() Builtin, I encounter an error. As I am new to LLVM, I don’t understand much. My guess is that something is wrong with my SelectionDAG part, how I handle intrinsic in the back-end

The error message appears as follows:

PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace, preprocessed source, and associated run script.
Stack dump:
0.	Program arguments: riscv64-unknown-linux-gnu-clang -v -O0 -S -o vadd_cpu_builtins.s vadd_mproc.c -DMPROC
1.	<eof> parser at end of file
2.	Code generation
3.	Running pass 'Function Pass Manager' on module 'vadd_mproc.c'.
4.	Running pass 'RISC-V DAG->DAG Pattern Instruction Selection' on function '@accelerator_vadd'
 #0 0x000055aa4f7571a0 llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) (/home/omega/Work/Office/riscv-llvm/bin/clang-17+0x1a951a0)
 #1 0x000055aa4f755064 llvm::sys::CleanupOnSignal(unsigned long) (/home/omega/Work/Office/=riscv-llvm/bin/clang-17+0x1a93064)
 #2 0x000055aa4f6a3cf8 CrashRecoverySignalHandler(int) CrashRecoveryContext.cpp:0:0
 #3 0x00007fcd22b2f420 __restore_rt (/lib/x86_64-linux-gnu/libpthread.so.0+0x14420)
 #4 0x000055aa506e91b6 llvm::SelectionDAG::getNode(unsigned int, llvm::SDLoc const&, llvm::SDVTList, llvm::ArrayRef<llvm::SDValue>, llvm::SDNodeFlags) (/home/omega/Work/Office/riscv-llvm/bin/clang-17+0x2a271b6)
 #5 0x000055aa5067754f llvm::SelectionDAGBuilder::visitTargetIntrinsic(llvm::CallInst const&, unsigned int) (/home/omega/Work/Office/riscv-llvm/bin/clang-17+0x29b554f)
 #6 0x000055aa50655fa4 llvm::SelectionDAGBuilder::visitIntrinsicCall(llvm::CallInst const&, unsigned int) (/home/omega/Work/Office/riscv-llvm/bin/clang-17+0x2993fa4)
 #7 0x000055aa50697c10 llvm::SelectionDAGBuilder::visit(llvm::Instruction const&) (/home/omega/Work/Office/riscv-llvm/bin/clang-17+0x29d5c10)
 #8 0x000055aa50706686 llvm::SelectionDAGISel::SelectBasicBlock(llvm::ilist_iterator<llvm::ilist_detail::node_options<llvm::Instruction, false, false, void>, false, true>, llvm::ilist_iterator<llvm::ilist_detail::node_options<llvm::Instruction, false, false, void>, false, true>, bool&) (/home/omega/Work/Office/riscv-llvm/bin/clang-17+0x2a44686)
 #9 0x000055aa50707b8c llvm::SelectionDAGISel::SelectAllBasicBlocks(llvm::Function const&) (/home/omega/Work/Office/riscv-llvm/bin/clang-17+0x2a45b8c)
#10 0x000055aa50709e8e llvm::SelectionDAGISel::runOnMachineFunction(llvm::MachineFunction&) (.part.0) SelectionDAGISel.cpp:0:0
#11 0x000055aa4eca9bb4 llvm::MachineFunctionPass::runOnFunction(llvm::Function&) (.part.0) MachineFunctionPass.cpp:0:0
#12 0x000055aa4f2067ca llvm::FPPassManager::runOnFunction(llvm::Function&) (/home/omega/Work/Office/riscv-llvm/bin/clang-17+0x15447ca)
#13 0x000055aa4f206954 llvm::FPPassManager::runOnModule(llvm::Module&) (/home/omega/Work/Office/riscv-llvm/bin/clang-17+0x1544954)
#14 0x000055aa4f207354 llvm::legacy::PassManagerImpl::run(llvm::Module&) (/home/omega/Work/Office/riscv-llvm/bin/clang-17+0x1545354)
#15 0x000055aa4f998410 clang::EmitBackendOutput(clang::DiagnosticsEngine&, clang::HeaderSearchOptions const&, clang::CodeGenOptions const&, clang::TargetOptions const&, clang::LangOptions const&, llvm::StringRef, llvm::Module*, clang::BackendAction, llvm::IntrusiveRefCntPtr<llvm::vfs::FileSystem>, std::unique_ptr<llvm::raw_pwrite_stream, std::default_delete<llvm::raw_pwrite_stream>>) (/home/omega/Work/Office/riscv-llvm/bin/clang-17+0x1cd6410)
#16 0x000055aa50869c54 clang::BackendConsumer::HandleTranslationUnit(clang::ASTContext&) (/home/omega/Work/Office/riscv-llvm/bin/clang-17+0x2ba7c54)
#17 0x000055aa5175fa79 clang::ParseAST(clang::Sema&, bool, bool) (/home/omega/Work/Office/riscv-llvm/bin/clang-17+0x3a9da79)
#18 0x000055aa501a09e9 clang::FrontendAction::Execute() (/home/omega/Work/Office/riscv-llvm/bin/clang-17+0x24de9e9)
#19 0x000055aa50129121 clang::CompilerInstance::ExecuteAction(clang::FrontendAction&) (/home/omega/Work/Office/riscv-llvm/bin/clang-17+0x2467121)
#20 0x000055aa50264443 clang::ExecuteCompilerInvocation(clang::CompilerInstance*) (/home/omega/Work/Office/riscv-llvm/bin/clang-17+0x25a2443)
#21 0x000055aa4e815b6b cc1_main(llvm::ArrayRef<char const*>, char const*, void*) (/home/omega/Work/Office/riscv-llvm/bin/clang-17+0xb53b6b)
#22 0x000055aa4e80eced ExecuteCC1Tool(llvm::SmallVectorImpl<char const*>&, llvm::ToolContext const&) driver.cpp:0:0
#23 0x000055aa4ff6842d void llvm::function_ref<void ()>::callback_fn<clang::driver::CC1Command::Execute(llvm::ArrayRef<std::optional<llvm::StringRef>>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>*, bool*) const::'lambda'()>(long) Job.cpp:0:0
#24 0x000055aa4f6a41c7 llvm::CrashRecoveryContext::RunSafely(llvm::function_ref<void ()>) (/home/omega/Work/Office/riscv-llvm/bin/clang-17+0x19e21c7)
#25 0x000055aa4ff687c7 clang::driver::CC1Command::Execute(llvm::ArrayRef<std::optional<llvm::StringRef>>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>*, bool*) const (.part.0) Job.cpp:0:0
#26 0x000055aa4ff31351 clang::driver::Compilation::ExecuteCommand(clang::driver::Command const&, clang::driver::Command const*&, bool) const (/home/omega/Work/Office/riscv-llvm/bin/clang-17+0x226f351)
#27 0x000055aa4ff31e0d clang::driver::Compilation::ExecuteJobs(clang::driver::JobList const&, llvm::SmallVectorImpl<std::pair<int, clang::driver::Command const*>>&, bool) const (/home/omega/Work/Office/riscv-llvm/bin/clang-17+0x226fe0d)
#28 0x000055aa4ff4324c clang::driver::Driver::ExecuteCompilation(clang::driver::Compilation&, llvm::SmallVectorImpl<std::pair<int, clang::driver::Command const*>>&) (/home/omega/Work/Office/riscv-llvm/bin/clang-17+0x228124c)
#29 0x000055aa4e81379f clang_main(int, char**, llvm::ToolContext const&) (/home/omega/Work/Office/riscv-llvm/bin/clang-17+0xb5179f)
#30 0x000055aa4e763ac3 main (/home/omega/Work/Office/riscv-llvm/bin/clang-17+0xaa1ac3)
#31 0x00007fcd22517083 __libc_start_main /build/glibc-e2p3jK/glibc-2.31/csu/../csu/libc-start.c:342:3
#32 0x000055aa4e80e16e _start (/home/omega/Work/Office/riscv-llvm/bin/clang-17+0xb4c16e)
riscv64-unknown-linux-gnu-clang: error: clang frontend command failed with exit code 139 (use -v to see invocation)
clang version 17.0.2 (https://github.com/llvm/llvm-project.git b2417f51dbbd7435eb3aaf203de24de6754da50e)
Target: riscv64-unknown-linux-gnu
Thread model: posix
InstalledDir: /home/omega/Work/Office/riscv-llvm/bin
riscv64-unknown-linux-gnu-clang: note: diagnostic msg: 
********************

PLEASE ATTACH THE FOLLOWING FILES TO THE BUG REPORT:
Preprocessed source(s) and associated run script(s) are located at:
riscv64-unknown-linux-gnu-clang: note: diagnostic msg: /tmp/vadd_mproc-8d5b66.c
riscv64-unknown-linux-gnu-clang: note: diagnostic msg: /tmp/vadd_mproc-8d5b66.sh
riscv64-unknown-linux-gnu-clang: note: diagnostic msg: 

********************
7

That makes it impossible to use properly. The builtin needs to capture the inputs and outputs.

8

Oh :confused:
In my case, the current requirement, I need to execute this particular assembly instruction through a builtin function. For that I do not need any input.

Do you mean that there should be a return type (other than void) for a builtin?

9

A void (void) builtin has no way of knowing what its inputs or outputs should be. Even with inline assembly right next to it the compiler has no knowledge that whatever registers you touched are needed by the builtin, and can freely reorder things in between. If you want more than inline assembly for your custom instruction, your builtin’s arguments needs to include all the input registers to your custom instruction (implicit or otherwise) and your builtin’s return value needs to include all the outputs (implicit or otherwise). Whether you have the output be via a pointer argument that gets written to or be via a normal return value is up to you.