I finally got something working. The key for me was to separate it into multiple builds and stop trying to get it all into one giant cmake invocation. First I built llvm and clang, without libcxx. I set LLVM_TARGETS_TO_BUILD to X86;ARM;AArch64.
Next, I built libcxx using the new compiler, for x86. Then a second build of libcxx for arm64. So three different build directories total. Building libcxx only takes about a minute (compared to about 1 hour for the first step) so tweaking and rerunning them is not so painful.
Then I was able to compile and run a “hello world” on my Mac and iPhone using this compiler and libc++. Never touched Xcode.