[clangd] Worker threads stack size


clangd uses std::thread for spawning worker threads to parse files (see AsyncTaskRunner::runAsync in tools/clang/tools/extra/clangd/Threading.cpp). Unfortunately, std::thread lacks the ability to customize the stack size, and the default one might be insufficient for parsing complex source files. For example, on macOS, the default stack size for non-main threads is 512K, so the following code causes a stack overflow (on my machine) with clangd, while being way below the default constexpr evaluation depth threshold:

  constexpr int foo(int x) {
      return x == 1 ? 1 : foo(x - 1) + 1;
  static_assert(foo(50) > 0, "");

This also can be observed on some real-world projects.

Do I understand correctly that the only way to fix it would be to abandon std::thread and reimplement the platform-specific thread spawning (using pthreads/WinAPI), or is there another solution available in LLVM/clang codebase?

Best regards,
Dmitry Kozhevnikov.

+clangd-dev@lists.llvm.org +Ilya Biryukov

Would be nice to get around this limitation in the standard library. I don’t think we can do it in a standard-compliant way, though.
There’s llvm::llvm_execute_on_thread, but it is an OS-specific implementation.

Are there any ways to set default stack sizes for the newly created threads? Even if it’s platform-specific code, it should be pretty localized and we could reuse the STL implementation of threads.
I don’t know of reliable APIs to do that, unfortunately.

In case MacOS LibC++ is implemented on top of pthreads, than you can can use pthreads API to set default stack size. This will probably be miminal possible platform-specific workaround.

пт, 17 авг. 2018 г. в 6:27, Ilya Biryukov via cfe-dev <cfe-dev@lists.llvm.org>:

I think the best solution is to use llvm::llvm_execute_on_thread instead of std::thread.


llvm::execute_on_thread waits for completion of the operation, clangd spawns long-living threads managing requests to the AST and short-lived threads for async processing of requests.

Both are asynchronous and we only join them when clangd exits.
We might have something similar to llvm:: execute_on_thread, but that would mean reimplementing a big chunk of threading lib from STL, I think it’s exactly what Dmitry proposed as an option.

I've submitted a patch trying to do so: ⚙ D50993 [clangd] Increase stack size of the new threads on macOS (I haven't found a way to set the default value on macOS and continue using the default std::thread implementation, though).

Hm, so we do currently have a single entry point for creating threads. Thanks, I’ll leave some comments in the patch.