I’m not sure if this is the right place for this question as I’m not exactly a beginner but I also don’t see anywhere else to ask questions and hopefully get help with clang. If I’m in the wrong place, please tell me and (hopefully!) where I should be asking this question.
I work on an Android app and I’m in charge of the underlying C++ code. As I compiled everything with the NDK, it relies on clang++ for compilation. We’ve been using NDK 25.0.8775105 but I have been attempting to update to NDK 26.1.10909125.
This bumps the underling Clang version from 14.0.6 to 17.0.2. This came with some code issues, but I quickly resolved them – mostly bad behaviour on the part of our dependencies or our own source. I went to make a full dependency build as a final check, only to discover that it build everything a full ten times slower!
I added build timing to our builds so I could compare, and this was the result:
I swapped the LLVM folders between the two NDK directory trees, and when I build with NDK v26 (but use clang 14.0.6) the slowdown goes away. So I’m 98% sure it’s a clang issue. So you can basically read ‘NDK v25’ as ‘Clang 14.0.6’ and ‘NDK v26’ as ‘Clang 17.0.2’ above.
That said, I’m guessing Clang 17.x isn’t 10x slower than 14.x, so presumably there’s some flag or something that needs to be changed? My only clue so far is that it looks like my build isn’t using very many (or any?) parallel processes to build the source. Though that doesn’t fully explain the problem, as the CMake generation step is also very slow (incredibly slow for curl!) and I don’t think that every uses multiple processes when testing capability (I could be wrong).
I’m hoping this is a know problem and its simply a matter of updating a flag whose default has changed, or adding some new flag that was added between 14 & 17.
Any help would be greatly appreciated! I obviously can’t switch to the new NDK if its going to take 10x longer to build, but I’d also love to start using C++20 so I’m in a bind.
This is surprising. We also build for Android (with a Clang we built ourselves instead of from the NDK though), and we didn’t observe any slowdown with Clang 17. (We went in steps, from 13 to 15 to 16 to 17, whereas you went straight from 14 to 17, so it’s possible each step had incremental slowdowns that we didn’t catch because they were below a threshold, but I’d be surprised.)
Issues · android/ndk · GitHub might be a better place to ask for help about this, in case they’re aware of any issues specific to the NDK’s Clang. What platform are you seeing this on though? Figuring out particular source files whose compilation times have increased dramatically and getting a time trace might also be helpful, though less so for a systematic slowdown like this.
I have a post in the NDK’s discussion forum but I hadn’t gotten any bites so I thought given what I was seeing maybe LLVM was a better place to post. Naturally, as soon as I did that, someone replied on the NDK forum, but it was basically suggesting the same thing you are – to file a ticket. I’ll probably end up doing that after a bit more legwork, but I wasn’t sure if they’d be interested since what I’ve learned in the last few hours seems to point away from the NDK itself.
I also doubt there was some small performance drop that you missed – 10x is a hard total to get to, even incrementally, without noticing it! I’m sure it’ll turn out to be a flag or something but I haven’t the foggiest idea. I’ll give time trace a look too, maybe it’ll reveal something.
Oh, and I’m running this on Windows via Git bash, which is admittedly a bit of an odd duck. It’s not impossible that’s the problem, though it wasn’t until the jump to NDK v26.x. Our team is spread across Linux, Windows & OSX, so if it turns out to be Git bash, that’ll be a problem, but a solvable one.
Yeah, it’d be interesting if any of your colleagues on Linux or macOS are also experiencing the issue. We don’t build on Windows ourselves. I’d be kinda surprised if it was a git bash issue specifically vs a general Windows problem, though it’s certainly possible.
I’ve lined up a couple of colleagues to try it out tomorrow morning so I’ll have some information on that hopefully by then. I have seen weird build issues with Git Bash on Windows before (GDAL’s pre-CMake for example) but I’d also be surprised if that was the problem here.
I’ll try your idea of using the official binaries. It hadn’t occurred to me that it could be the NDK’s build that was at fault. That’s worth pursuing, thanks!
As it turns out, your idea was excellent! I dropped in the clang*.exe from the 17.0.2 build package (figured it was better to be exact) and the problem went away completely. In fact, its faster than NDK v25 (slightly) with the official executables.
So clearly however it’s being build by the NDK team is not great. I’ve got a couple of colleagues testing it on OSX (and hopefully Linux) but they’re not seeing the same degree of problem. Slower, but not nearly as bad. So I think I have enough information to go back and file a ticket to the NDK team – it does look like this is their issue.
