You need to finalize the type conversion process by using the -reconcile-unrealized-casts pass. See PSA: run -reconcile-unrealized-casts after all -convert-*-to-llvm from now on and LLVM IR Target - MLIR for more details.
Also, I suspect that you may also need -convert-arith-to-llvm and -convert-std-to-llvm somewhere in pass pipeline before pass reconciliation, it is quite rare that these dialects are not in the mix after lowering from affine.