I am writing an new backend. And I don’t have vsge(vector set when larger than or equal to) instruction in the instruction set.
How could I implement the instruction? I have vslt(vector set when less than) instruction, vseq(vector set when equal to), vneg(vector negate), vand(vector and), vor(vector or) and vxor(vector xor) in the instruction set.
Should I consider the situation when one of the operand of vsge is nan(not a number)?
Where should I implement the vsge operation? Should it be implemented in the SelectionDAG lowering step or the pseudo instruction expansion step?

And I don't have vsge(vector set when larger than or equal to) instruction in the instruction set. How could I implement the instruction? I have vslt(vector set when less than) instruction, vseq(vector set when equal to), vneg(vector negate), vand(vector and), vor(vector or) and vxor(vector xor) in the instruction set.

It's pretty common for instruction sets to only have comparisons in
one direction because a >= b if and only if b <= a (even in IEEE with
NaNs) so you can just reverse the operands. It's slightly rarer to
completely lack one of the canonical relational operators but it does
happen.

With those operations (and guessing that the dest is the first
register in your assembly syntax), for vsge dst, a, b" you'd want to
produce something like:

vslt tmp, b, a
vseq dst, b, a
vor dst, dst, tmp

Basically directly implementing "less than or equal to" with reversed operands.

Should I consider the situation when one of the operand of vsge is nan(not a number)?

Generally yes, unless you know you're in some kind of fast-math mode
(e.g. if there's "no-nans-fp-math"="true" in the function attributes).
But in this case I think the obvious translation does it anyway.

Where should I implement the vsge operation? Should it be implemented in the SelectionDAG lowering step or the pseudo instruction expansion step?

Similar situations come up in most backends because LLVM has 14-16
different kinds of comparison but instruction sets tend to have fewer.
In my experience people solve it in XYZISelLowering.cpp so that later
passes can still optimize the sequence if they want. But I think
either could be made to work.