What does this error mean: psuedo instructions should be removed before code emission?

Also there is a typo: it probably should read pseudo.

Yuri

A pseudo instruction is used internally without llvm, but it does not correspond directly to a real instruction in the target architecture. Before emitting the final compiled code, all the pseudo instructions must be expanded to real instructions. If you're seeing that error, it means that something was not expanded.

I fixed all the "psuedo" misspellings in svn 112202. I didn't see the error message, so that must have been changed already.

...and speaking of typos, that was supposed to be "used internally _within_ llvm"

So what is likely to be wrong? This error only happens on amd64 and not on i386, variable Opcode is 2493 when the error occurs.
It is triggered from under the call that I make: JIT::runFunction.

Yuri

Could you get it to print out the instruction when it happens? (just change the line above the error message to print it out to errs()).

It basically means that a pseudo wasn't lowered to something that the jit can output before the jit was run. Is this on ToT?

-eric

Insn before the error: TCRETURNri64 %RAX<kill>, 0, %RDI<kill>, %RAX<imp-def,dead>, %RDI<imp-def,dead>, %RSP<imp-use>, ...

Yuri

I don't know what is ToT. 42 meanings listed here http://acronyms.thefreedictionary.com/TOT don't seem to apply.

Yuri

ToT = "top of trunk"

I'm pretty sure the answer is "no", because the misspelled error message is not present in the trunk sources.

But now that I updated I am almost on ToT (rev 112200 from today) and the problem is still there.

Yuri

Odd. I thought TCReturn was being lowered. At any rate can you file a bug with the .ll file that causes this?

-eric

It should be getting lowered in emitEpilogue. Must be a bug somewhere.

For some reason I am getting this error even when I only have an empty 'main' function. So I couldn't create .ll file reproducing it and I have to debug myself.

The function causing the problem is stub created in JIT::runFunction: Function *Stub = Function::Create(STy, Function::InternalLinkage, "" ...

Here is the log of machine instructions before and after emitEpilogue for this function:
* PEI::insertPrologEpilogCode: === >> before emitEpilogue
   - insn: PUSH64r %RBP<kill>, %RSP<imp-def>, %RSP<imp-use>
   - insn: PROLOG_LABEL <MCSym=.Ltmp2>
   - insn: %RBP<def> = MOV64rr %RSP
   - insn: PROLOG_LABEL <MCSym=.Ltmp3>
   - insn: %RDI<def> = MOV64ri64i32 60910096
   - insn: %RAX<def> = MOV64ri <ga:@_ZN010HelloWorld4mainEPN13ContainerSVecE>
   - insn: TCRETURNri64 %RAX<kill>, 0, %RDI<kill>, %RAX<imp-def,dead>, %RDI<imp-def,dead>, %RSP<imp-use>, ...
   - insn: RET
* PEI::insertPrologEpilogCode: === << before emitEpilogue
* PEI::insertPrologEpilogCode: === >> after emitEpilogue
   - insn: PUSH64r %RBP<kill>, %RSP<imp-def>, %RSP<imp-use>
   - insn: PROLOG_LABEL <MCSym=.Ltmp2>
   - insn: %RBP<def> = MOV64rr %RSP
   - insn: PROLOG_LABEL <MCSym=.Ltmp3>
   - insn: %RDI<def> = MOV64ri64i32 60910096
   - insn: %RAX<def> = MOV64ri <ga:@_ZN010HelloWorld4mainEPN13ContainerSVecE>
   - insn: TCRETURNri64 %RAX<kill>, 0, %RDI<kill>, %RAX<imp-def,dead>, %RDI<imp-def,dead>, %RSP<imp-use>, ...
   - insn: %RBP<def> = POP64r %RSP<imp-def>, %RSP<imp-use>
   - insn: RET
* PEI::insertPrologEpilogCode: === << after emitEpilogue

Function only has on BB. Is this wrong that it has both TCRETURNri64 and RET in one BB?
You can see that instruction TCRETURNri64 is left by emitEpilogue and and this later causes problem in code emitter.
Where in X86RegisterInfo::emitEpilogue TCRETURNri64 is supposed to disappear (be lowered)?

Yuri

Yes, that is wrong. The reason emitEpilogue isn't lowering the TCRETURN is that it doesn't see it, it only sees the RET. The real problem will be where that RET is generated. Normally that's LowerCall (where it is spelled TC_RETURN), but reading through it I can't see any way to generate both a TC_RETURN and a RET.

Bug somewhere for sure, but without a testcase or way to reproduce it then it'd be hard to find.

Could have a machine verify pass that makes sure each MBB only had one exit instruction.

-eric

Yes, Yuri's going to have to debug it, or provide a way for others to reproduce it.

It could be his host compiler is miscompiling llvm, too.

I am debugging it.
I use gcc-4.5.0 that I built myself to compile llvm.

Yuri

Function only has on BB. Is this wrong that it has both TCRETURNri64 and RET in one BB?

Yes, that is wrong. The reason emitEpilogue isn't lowering the TCRETURN is that it doesn't see it, it only sees the RET. The real problem will be where that RET is generated. Normally that's LowerCall (where it is spelled TC_RETURN), but reading through it I can't see any way to generate both a TC_RETURN and a RET.

Bug somewhere for sure, but without a testcase or way to reproduce it then it'd be hard to find.

Yes, Yuri's going to have to debug it, or provide a way for others to reproduce it.

It could be his host compiler is miscompiling llvm, too.

As I understand only one of TCRETURNri64 and RET should be created.
I have sources of rev.112200.

Here is the stack when TCRETURNri64 instruction is created:
#1 0x0000000802c8b4e2 in llvm::MachineFunction::CreateMachineInstr (this=0x30eb000, TID=@0x803a78940, DL={LineCol = 0, ScopeIdx = 0}, NoImp=false) at /tmp/llvm-svn/llvm/lib/CodeGen/MachineFunction.cpp:153
#2 0x00000008028ea45b in llvm::BuildMI (BB=@0x4b69378, I={<std::iterator<std::bidirectional_iterator_tag, llvm::MachineInstr, long int, llvm::MachineInstr*, llvm::MachineInstr&>> = {<No data

}, NodePtr = 0x4b69440}, DL={LineCol = 0, ScopeIdx = 0},

TID=@0x803a78940) at MachineInstrBuilder.h:183
#3 0x0000000803451825 in llvm::X86RegisterInfo::emitPrologue (this=0x1a96220, MF=@0x30eb000) at /tmp/llvm-svn/llvm/lib/Target/X86/X86RegisterInfo.cpp:1037
#4 0x0000000802cdf3e5 in llvm::PEI::insertPrologEpilogCode (this=0x3231900, Fn=@0x30eb000) at /tmp/llvm-svn/llvm/lib/CodeGen/PrologEpilogInserter.cpp:680
#5 0x0000000802cdcf04 in llvm::PEI::runOnMachineFunction (this=0x3231900, Fn=@0x30eb000) at /tmp/llvm-svn/llvm/lib/CodeGen/PrologEpilogInserter.cpp:106
#6 0x0000000802c92abd in llvm::MachineFunctionPass::runOnFunction (this=0x3231900, F=@0x4530c60) at /tmp/llvm-svn/llvm/lib/CodeGen/MachineFunctionPass.cpp:33
#7 0x0000000802e76e77 in llvm::FPPassManager::runOnFunction (this=0x235e540, F=@0x4530c60) at /tmp/llvm-svn/llvm/lib/VMCore/PassManager.cpp:1443
#8 0x0000000802e76b3f in llvm::FunctionPassManagerImpl::run (this=0x1e1a300, F=@0x4530c60) at /tmp/llvm-svn/llvm/lib/VMCore/PassManager.cpp:1394
#9 0x0000000802e767ef in llvm::FunctionPassManager::run (this=0x4b566c0, F=@0x4530c60) at /tmp/llvm-svn/llvm/lib/VMCore/PassManager.cpp:1324
#10 0x0000000802f50500 in llvm::JIT::jitTheFunction (this=0x327a7e0, F=0x4530c60, locked=@0x7fffffff9840) at /tmp/llvm-svn/llvm/lib/ExecutionEngine/JIT/JIT.cpp:655
#11 0x0000000802f5037b in llvm::JIT::runJITOnFunctionUnlocked (this=0x327a7e0, F=0x4530c60, locked=@0x7fffffff9840) at /tmp/llvm-svn/llvm/lib/ExecutionEngine/JIT/JIT.cpp:633
#12 0x0000000802f507e4 in llvm::JIT::getPointerToFunction (this=0x327a7e0, F=0x4530c60) at /tmp/llvm-svn/llvm/lib/ExecutionEngine/JIT/JIT.cpp:719

Here is the stack when RET instruction is created:
#1 0x0000000802c8b4e2 in llvm::MachineFunction::CreateMachineInstr (this=0x30eb000, TID=@0x803a99240, DL={LineCol = 0, ScopeIdx = 0}, NoImp=false) at /tmp/llvm-svn/llvm/lib/CodeGen/MachineFunction.cpp:153
#2 0x00000008028ea45b in llvm::BuildMI (BB=@0x4b69378, I={<std::iterator<std::bidirectional_iterator_tag, llvm::MachineInstr, long int, llvm::MachineInstr*, llvm::MachineInstr&>> = {<No data

}, NodePtr = 0x4b69388}, DL={LineCol = 0, ScopeIdx = 0},

TID=@0x803a99240) at MachineInstrBuilder.h:183
#3 0x00000008033c7637 in X86SelectRet (this=0x4530bd0, I=0x493fae0) at /tmp/llvm-svn/llvm/lib/Target/X86/X86FastISel.cpp:771
#4 0x00000008033cc4fc in TargetSelectInstruction (this=0x4530bd0, I=0x493fae0) at /tmp/llvm-svn/llvm/lib/Target/X86/X86FastISel.cpp:1783
#5 0x000000080315a64f in llvm::FastISel::SelectInstruction (this=0x4530bd0, I=0x493fae0) at /tmp/llvm-svn/llvm/lib/CodeGen/SelectionDAG/FastISel.cpp:706
#6 0x0000000803275ee0 in llvm::SelectionDAGISel::SelectAllBasicBlocks (this=0x45174d0, Fn=@0x4530c60) at /tmp/llvm-svn/llvm/lib/CodeGen/SelectionDAG/SelectionDAGISel.cpp:726
#7 0x00000008032737f7 in llvm::SelectionDAGISel::runOnMachineFunction (this=0x45174d0, mf=@0x30eb000) at /tmp/llvm-svn/llvm/lib/CodeGen/SelectionDAG/SelectionDAGISel.cpp:252
#8 0x0000000802c92abd in llvm::MachineFunctionPass::runOnFunction (this=0x45174d0, F=@0x4530c60) at /tmp/llvm-svn/llvm/lib/CodeGen/MachineFunctionPass.cpp:33
#9 0x0000000802e76e77 in llvm::FPPassManager::runOnFunction (this=0x235e540, F=@0x4530c60) at /tmp/llvm-svn/llvm/lib/VMCore/PassManager.cpp:1443
#10 0x0000000802e76b3f in llvm::FunctionPassManagerImpl::run (this=0x1e1a300, F=@0x4530c60) at /tmp/llvm-svn/llvm/lib/VMCore/PassManager.cpp:1394
#11 0x0000000802e767ef in llvm::FunctionPassManager::run (this=0x4b566c0, F=@0x4530c60) at /tmp/llvm-svn/llvm/lib/VMCore/PassManager.cpp:1324
#12 0x0000000802f50500 in llvm::JIT::jitTheFunction (this=0x327a7e0, F=0x4530c60, locked=@0x7fffffff9840) at /tmp/llvm-svn/llvm/lib/ExecutionEngine/JIT/JIT.cpp:655
#13 0x0000000802f5037b in llvm::JIT::runJITOnFunctionUnlocked (this=0x327a7e0, F=0x4530c60, locked=@0x7fffffff9840) at /tmp/llvm-svn/llvm/lib/ExecutionEngine/JIT/JIT.cpp:633
#14 0x0000000802f507e4 in llvm::JIT::getPointerToFunction (this=0x327a7e0, F=0x4530c60) at /tmp/llvm-svn/llvm/lib/ExecutionEngine/JIT/JIT.cpp:719
....my code.....

Yuri

It seems wrong that TCRETURNri64 is created in X86RegisterInfo::emitPrologue even when RET instruction is already there. This seems to be a bug.

Yuri

I don't think I believe this; emitPrologue should not be generating a TCRETURN at all, and line 1037 is generating a PROLOG_LABEL. Why do you say it's a TCRETURN?

The way this is supposed to work:
SelectBasicBlock calls visit on the Call instruction, leading to visitCall:
  which winds up in X86TargetLowering::LowerCall:
which generates the TCRETURN and sets the isTailCall argument to True
which causes the loop in SelectBasicBlock to exit early, causing it to skip the RET.

It looks like the incoming RET (the Instruction, not the MachineInstr) is not getting skipped for some reason. I suggest stepping through the path above to figure out why.

I don't think I believe this; emitPrologue should not be generating a TCRETURN at all, and line 1037 is generating a PROLOG_LABEL. Why do you say it's a TCRETURN?

Sorry, my bad. I have set breakpoints by MI addresses. But it turns out that these addresses were reused and the second MIs created at these particular addressed are of interest, not the first ones.

RET creation stack (created first):
#1 0x0000000802c8b4e2 in llvm::MachineFunction::CreateMachineInstr (this=0x30eb000, TID=@0x803a99240, DL={LineCol = 0, ScopeIdx = 0}, NoImp=false) at /tmp/llvm-svn/llvm/lib/CodeGen/MachineFunction.cpp:153
#2 0x00000008028ea45b in llvm::BuildMI (BB=@0x4b69378, I={<std::iterator<std::bidirectional_iterator_tag, llvm::MachineInstr, long int, llvm::MachineInstr*, llvm::MachineInstr&>> = {<No data

}, NodePtr = 0x4b69388}, DL={LineCol = 0, ScopeIdx = 0},

TID=@0x803a99240) at MachineInstrBuilder.h:183
#3 0x00000008033c7637 in X86SelectRet (this=0x4530f30, I=0x493f240) at /tmp/llvm-svn/llvm/lib/Target/X86/X86FastISel.cpp:771
#4 0x00000008033cc4fc in TargetSelectInstruction (this=0x4530f30, I=0x493f240) at /tmp/llvm-svn/llvm/lib/Target/X86/X86FastISel.cpp:1783
#5 0x000000080315a64f in llvm::FastISel::SelectInstruction (this=0x4530f30, I=0x493f240) at /tmp/llvm-svn/llvm/lib/CodeGen/SelectionDAG/FastISel.cpp:706
#6 0x0000000803275ee0 in llvm::SelectionDAGISel::SelectAllBasicBlocks (this=0x45174d0, Fn=@0x4530bd0) at /tmp/llvm-svn/llvm/lib/CodeGen/SelectionDAG/SelectionDAGISel.cpp:726
#7 0x00000008032737f7 in llvm::SelectionDAGISel::runOnMachineFunction (this=0x45174d0, mf=@0x30eb000) at /tmp/llvm-svn/llvm/lib/CodeGen/SelectionDAG/SelectionDAGISel.cpp:252V
#8 0x0000000802c92abd in llvm::MachineFunctionPass::runOnFunction (this=0x45174d0, F=@0x4530bd0) at /tmp/llvm-svn/llvm/lib/CodeGen/MachineFunctionPass.cpp:33
#9 0x0000000802e76e77 in llvm::FPPassManager::runOnFunction (this=0x235e540, F=@0x4530bd0) at /tmp/llvm-svn/llvm/lib/VMCore/PassManager.cpp:1443
#10 0x0000000802e76b3f in llvm::FunctionPassManagerImpl::run (this=0x1e1a300, F=@0x4530bd0) at /tmp/llvm-svn/llvm/lib/VMCore/PassManager.cpp:1394
#11 0x0000000802e767ef in llvm::FunctionPassManager::run (this=0x4b566c0, F=@0x4530bd0) at /tmp/llvm-svn/llvm/lib/VMCore/PassManager.cpp:1324
#12 0x0000000802f50500 in llvm::JIT::jitTheFunction (this=0x327a7e0, F=0x4530bd0, locked=@0x7fffffff8b50) at /tmp/llvm-svn/llvm/lib/ExecutionEngine/JIT/JIT.cpp:655
#13 0x0000000802f5037b in llvm::JIT::runJITOnFunctionUnlocked (this=0x327a7e0, F=0x4530bd0, locked=@0x7fffffff8b50) at /tmp/llvm-svn/llvm/lib/ExecutionEngine/JIT/JIT.cpp:633
#14 0x0000000802f507e4 in llvm::JIT::getPointerToFunction (this=0x327a7e0, F=0x4530bd0) at /tmp/llvm-svn/llvm/lib/ExecutionEngine/JIT/JIT.cpp:719
#15 0x0000000802f4f194 in llvm::JIT::runFunction (this=0x327a7e0, F=0x4530bd0, ArgValues=@0x7fffffff9230) at /tmp/llvm-svn/llvm/lib/ExecutionEngine/JIT/JIT.cpp:405
#16 0x0000000802f4fe4f in llvm::JIT::runFunction (this=0x327a7e0, F=0x4530c60, ArgValues=@0x7fffffff9350) at /tmp/llvm-svn/llvm/lib/ExecutionEngine/JIT/JIT.cpp:561

TCRETURNri64 creation stack (created second):
#1 0x0000000802c8b4e2 in llvm::MachineFunction::CreateMachineInstr (this=0x30eb000, TID=@0x803a9f840, DL={LineCol = 0, ScopeIdx = 0}, NoImp=false) at /tmp/llvm-svn/llvm/lib/CodeGen/MachineFunction.cpp:153
#2 0x00000008028ea302 in llvm::BuildMI (MF=@0x30eb000, DL={LineCol = 0, ScopeIdx = 0}, TID=@0x803a9f840) at MachineInstrBuilder.h:147
#3 0x0000000803164513 in llvm::InstrEmitter::EmitMachineNode (this=0x7fffffff7f80, Node=0x4b6c510, IsClone=false, IsCloned=false, VRBaseMap=@0x7fffffff8050) at /tmp/llvm-svn/llvm/lib/CodeGen/SelectionDAG/InstrEmitter.cpp:671
#4 0x00000008031f6bdf in llvm::InstrEmitter::EmitNode (this=0x7fffffff7f80, Node=0x4b6c510, IsClone=false, IsCloned=false, VRBaseMap=@0x7fffffff8050) at InstrEmitter.h:118
#5 0x00000008031f5781 in llvm::ScheduleDAGSDNodes::EmitSchedule (this=0x4b68400) at /tmp/llvm-svn/llvm/lib/CodeGen/SelectionDAG/ScheduleDAGSDNodes.cpp:602
#6 0x0000000803275459 in llvm::SelectionDAGISel::CodeGenAndEmitDAG (this=0x45174d0) at /tmp/llvm-svn/llvm/lib/CodeGen/SelectionDAG/SelectionDAGISel.cpp:547
#7 0x00000008032740fc in llvm::SelectionDAGISel::SelectBasicBlock (this=0x45174d0, Begin={<std::iterator<std::bidirectional_iterator_tag, const llvm::Instruction, long int, const llvm::Instruction*, const llvm::Instruction&>> = {<No data fields>}, NodePtr = 0x4530ed0}, End={<std::iterator<std::bidirectional_iterator_tag, const llvm::Instruction, long int, const llvm::Instruction*, const llvm::Instruction&>> = {<No data fields>}, NodePtr = 0x493f240}, HadTailCall=@0x7fffffff853f) at /tmp/llvm-svn/llvm/lib/CodeGen/SelectionDAG/SelectionDAGISel.cpp:364
#8 0x0000000803276048 in llvm::SelectionDAGISel::SelectAllBasicBlocks (this=0x45174d0, Fn=@0x4530bd0) at /tmp/llvm-svn/llvm/lib/CodeGen/SelectionDAG/SelectionDAGISel.cpp:744
#9 0x00000008032737f7 in llvm::SelectionDAGISel::runOnMachineFunction (this=0x45174d0, mf=@0x30eb000) at /tmp/llvm-svn/llvm/lib/CodeGen/SelectionDAG/SelectionDAGISel.cpp:252
#10 0x0000000802c92abd in llvm::MachineFunctionPass::runOnFunction (this=0x45174d0, F=@0x4530bd0) at /tmp/llvm-svn/llvm/lib/CodeGen/MachineFunctionPass.cpp:33
#11 0x0000000802e76e77 in llvm::FPPassManager::runOnFunction (this=0x235e540, F=@0x4530bd0) at /tmp/llvm-svn/llvm/lib/VMCore/PassManager.cpp:1443
#12 0x0000000802e76b3f in llvm::FunctionPassManagerImpl::run (this=0x1e1a300, F=@0x4530bd0) at /tmp/llvm-svn/llvm/lib/VMCore/PassManager.cpp:1394
#13 0x0000000802e767ef in llvm::FunctionPassManager::run (this=0x4b566c0, F=@0x4530bd0) at /tmp/llvm-svn/llvm/lib/VMCore/PassManager.cpp:1324
#14 0x0000000802f50500 in llvm::JIT::jitTheFunction (this=0x327a7e0, F=0x4530bd0, locked=@0x7fffffff8b50) at /tmp/llvm-svn/llvm/lib/ExecutionEngine/JIT/JIT.cpp:655
#15 0x0000000802f5037b in llvm::JIT::runJITOnFunctionUnlocked (this=0x327a7e0, F=0x4530bd0, locked=@0x7fffffff8b50) at /tmp/llvm-svn/llvm/lib/ExecutionEngine/JIT/JIT.cpp:633
#16 0x0000000802f507e4 in llvm::JIT::getPointerToFunction (this=0x327a7e0, F=0x4530bd0) at /tmp/llvm-svn/llvm/lib/ExecutionEngine/JIT/JIT.cpp:719
#17 0x0000000802f4f194 in llvm::JIT::runFunction (this=0x327a7e0, F=0x4530bd0, ArgValues=@0x7fffffff9230) at /tmp/llvm-svn/llvm/lib/ExecutionEngine/JIT/JIT.cpp:405
#18 0x0000000802f4fe4f in llvm::JIT::runFunction (this=0x327a7e0, F=0x4530c60, ArgValues=@0x7fffffff9350) at /tmp/llvm-svn/llvm/lib/ExecutionEngine/JIT/JIT.cpp:561

The way this is supposed to work:
SelectBasicBlock calls visit on the Call instruction, leading to visitCall:
which winds up in X86TargetLowering::LowerCall:
which generates the TCRETURN and sets the isTailCall argument to True
which causes the loop in SelectBasicBlock to exit early, causing it to skip the RET.

It looks like the incoming RET (the Instruction, not the MachineInstr) is not getting skipped for some reason. I suggest stepping through the path above to figure out why.

Here's what happens:
The first instruction created is RET.
SelectBasicBlock is called.
TCRETURNri64 is created from within it.
HasTailCall is set to true as you mentioned.
Cycle in SelectionDAGISel::SelectBasicBlock skips the rest.

All like you described. But the RET instruction still stays in the end.
RET is skipped by visit but I guess stays in the list. Where should it be destroyed?

Yuri