VS Code Clangd and CMake integration

Hello! I’m not sure if this is the best place to ask, but it’s better to get a discussion going rather than just being confused.
I tried using Clangd in Visual Studio Code, but noticed that it doesn’t integrate well with the CMake Tools extension, which makes it hard to use together with a CMake workflow. I would like a way for Clangd to automatically get the correct info when CMake runs in the editor.
Specifically, I want seamless and automatic integration between these two extensions without the user having to do any additional config:

  1. Work with the author of CMake Tools to make the extension generate the appropriate info for vscode-clangd
  2. Make vscode-clangd retrieve the info from CMake Tools somehow
  3. Create a new extension that acts as a bridge between the two
    Or alternatively, some combination of the different approaches.
    I know that the CMake Tools extension integrates well with Microsoft’s C/C++ extension, so maybe it would be a good idea to do this in a similar way that they did it. If my understanding is correct, a separate bridge extension was previously used for this, but I think they now use a combination of something similar to approach 1 and 2 for the extensions to communicate with each other.
    I would be interested to hear what you all think would be the best way to go about this, and maybe I can get in touch with vector-of-bools (the author of the CMake Tools extension) about it too.

Another question I had regarding this: how does Clangd find the correct include paths and other info in the first place? Does it use some kind of compilation commands file for this? Is it possible to tell Clangd where to look for such a file? I couldn’t really find any documentation about this when I looked around.

Cheers,
Anton Södergren

Hello! I’m not sure if this is the best place to ask, but it’s better to get a discussion going rather than just being confused.

Right place! Welcome :slight_smile:

I tried using Clangd in Visual Studio Code, but noticed that it doesn’t integrate well with the CMake Tools extension, which makes it hard to use together with a CMake workflow. I would like a way for Clangd to automatically get the correct info when CMake runs in the editor.
Specifically, I want seamless and automatic integration between these two extensions without the user having to do any additional config:

I’m not familiar with CMake Tools, I guess it helps with editing CMakeLists, CMake configs, and running configure/generate?
(Sorry, I would dig into it, about to head out on vacation)

I see three possible approaches for this:

  1. Work with the author of CMake Tools to make the extension generate the appropriate info for vscode-clangd
  2. Make vscode-clangd retrieve the info from CMake Tools somehow
  3. Create a new extension that acts as a bridge between the two
    Or alternatively, some combination of the different approaches.
    I know that the CMake Tools extension integrates well with Microsoft’s C/C++ extension, so maybe it would be a good idea to do this in a similar way that they did it. If my understanding is correct, a separate bridge extension was previously used for this, but I think they now use a combination of something similar to approach 1 and 2 for the extensions to communicate with each other.
    I would be interested to hear what you all think would be the best way to go about this, and maybe I can get in touch with vector-of-bools (the author of the CMake Tools extension) about it too.

Another question I had regarding this: how does Clangd find the correct include paths and other info in the first place? Does it use some kind of compilation commands file for this? Is it possible to tell Clangd where to look for such a file? I couldn’t really find any documentation about this when I looked around.

So if my understanding of CMake Tools is correct, these questions are very much related :slight_smile:

Clangd works like other clang-based tools. Starting at the source file in question, it walks up directories until it finds compile_commands.json. This lists the flags required to compile each file in its scope.

So I think there are three relevant parts here:

  • this file has to be created somehow. CMake can do this, it’s controlled by CMAKE_EXPORT_COMPILE_COMMANDS, which I believe is on by default now. It writes the files into the root of the build tree.
  • the file has to be discovered somehow (if the CMake build tree != the source tree). The current best-practice is to manually create a symlink src/compile_commands.json → build/compile_commands.json. I think symlinking src/.build to build/ is a slightly more principled approach, but something still has to create a symlink. (Maybe we could get CMake or CMake Tools to do it?). Or maybe there’s some other clever way to find the build tree.
  • clangd has to reload the data when appropriate. Today, it caches forever :frowning: Ilya is working on fixing it!
    Gotta run, hope this helps!

Patches and ideas welcome :slight_smile:

I’m not familiar with CMake Tools, I guess it helps with editing CMakeLists, CMake configs, and running configure/generate?

Except for the file editing part (that’s handled by another extension), you’re pretty much right. It also allows for building and launching debuggers. But it’s the configure/generate part that’s the most interesting in our case.

Clangd works like other clang-based tools. Starting at the source file in question, it walks up directories until it finds compile_commands.json.

So it turned out that it indeed seemed to generate a compile_commands.json file in the build folder (at least yesterday, today it doesn’t seem to work well at all). I tried copying that file over to my project root folder and Clangd found it and started working correctly indeed!

The current best-practice is to manually create a symlink src/compile_commands.json → build/compile_commands.json. I think symlinking src/.build to build/ is a slightly more principled approach, but something still has to create a symlink. (Maybe we could get CMake or CMake Tools to do it?)

While this is a possible approach, I think is sounds a little unwieldy (though that might just be personal preference). When I looked around a bit more, I saw that the Clang tooling docs mentioned that it’s possible to pass the build path directly to Clang tools with a -p parameter, is this possible in Clangd too? For example, other IDE’s that utilize clang-tidy (e.g. CLion) seem to do so without any visible symlinks. To me, that sounds more robust and clean than having to pollute the source tree with symlinks. Of course, that would require some work probably on both of the plugins involved. Also, having a general and flexible approach for this would be nice, as I’m also interested in getting work on a build2 extension going, which would require the same functionality.

clangd has to reload the data when appropriate. Today, it caches forever :frowning: Ilya is working on fixing it!

I did notice this problem when testing it out yesterday. But I trust that it will get fixed then!

Cheers,
Anton

I’m not familiar with CMake Tools, I guess it helps with editing CMakeLists, CMake configs, and running configure/generate?

Except for the file editing part (that’s handled by another extension), you’re pretty much right. It also allows for building and launching debuggers. But it’s the configure/generate part that’s the most interesting in our case.

Clangd works like other clang-based tools. Starting at the source file in question, it walks up directories until it finds compile_commands.json.

So it turned out that it indeed seemed to generate a compile_commands.json file in the build folder (at least yesterday, today it doesn’t seem to work well at all). I tried copying that file over to my project root folder and Clangd found it and started working correctly indeed!

The current best-practice is to manually create a symlink src/compile_commands.json → build/compile_commands.json. I think symlinking src/.build to build/ is a slightly more principled approach, but something still has to create a symlink. (Maybe we could get CMake or CMake Tools to do it?)

While this is a possible approach, I think is sounds a little unwieldy (though that might just be personal preference). When I looked around a bit more, I saw that the Clang tooling docs mentioned that it’s possible to pass the build path directly to Clang tools with a -p parameter, is this possible in Clangd too?

It is in fact possible as a command flag (it’s spelled -compile-commands-dir). It works in a pinch, but it’s a subtly broken idea and I don’t think it’s the right long-term direction.

  1. a clangd instance is typically 1:1 with an editor session. But an editor can open files from multiple projects, so what should the flag value be?
  2. how does the editor know which dir to pass to clangd? If it’s an editor preference, how does this work for users who work in different projects at different times?
  3. each tool (clangd, clang-tidy etc) needs to get this configuration in some appropriate way, it’s not shared (whereas all clang-based tools share discovery logic for compile_commands.json)

Requiring the symlink (and adding it to gitignore etc) is a little fiddly and certainly unconventional, but it’s the best thing we’ve found so far.

For example, other IDE’s that utilize clang-tidy (e.g. CLion) seem to do so without any visible symlinks.

Yes, if you’re an IDE that already knows how to build the code (has opinions/integrations for the build system) then directly injecting the configuration is feasible. (Clangd has LSP extensions for this). But in VSCode, the build system is just another plugin that may or may not be installed, so a decoupled design seems most likely to work in many configurations.

To me, that sounds more robust and clean than having to pollute the source tree with symlinks. Of course, that would require some work probably on both of the plugins involved. Also, having a general and flexible approach for this would be nice, as I’m also interested in getting work on a build2 extension going, which would require the same functionality.

Definitely open to new ideas here, but I’m not sure exactly what you’re proposing.

Clangd needs to be able to determine the compile command given a filename, at any time, within the clangd process (i.e. without exchanging LSP messages with the editor).

Maybe one idea is to stick to compile_commands.json but extend the way they are discovered, e.g. let the Cmake Tools plugin notify Clangd that a certain compile_commands file is relevant and should be considered, even if it’s not in an ancestor dir of the source file.

clangd has to reload the data when appropriate. Today, it caches forever :frowning: Ilya is working on fixing it!

I did notice this problem when testing it out yesterday. But I trust that it will get fixed then!

It’s… not trivial. (Other clang tools don’t handle this as they tend to be batch)
So we’ll fix it, but it might take a little time :slight_smile:

But in VSCode, the build system is just another plugin that may or may not be installed, so a decoupled design seems most likely to work in many configurations.

If we could find a solution, it would of course be completely optional. Clangd would use its normal discovery mechanism of compile_commands.json, unless another plugin tells it to use a specific file or build dir. Maybe the build system extension could even give Clangd separate info for each C++ file if we want to be really flexible, but again, I don’t know exactly how that info would be sent to Clangd. I agree that it’s not reasonable that this info would be shared between different projects.

Something like this would be ideal:

  1. User opens a folder

  2. CMake runs

  3. A compile_commands.json is generated

  4. CMake Tools somehow tells Clangd where to find the file (this one is where the problem lies)

  5. Repeat steps 2-4 whenever CMake needs to generate again

Clangd needs to be able to determine the compile command given a filename, at any time, within the clangd process (i.e. without exchanging LSP messages with the editor).

I’ll admit that this sounds rather tricky. But the build plugin would only need to notify Clangd when it actually generates its compile_commands.json, after which Clangd could keep and reuse that state.

It is in fact possible as a command flag (it’s spelled -compile-commands-dir).

Would this only be passed once when the process starts and then never touched again? If so, then yeah this obviously doesn’t work.

each tool (clangd, clang-tidy etc) needs to get this configuration in some appropriate way, it’s not shared (whereas all clang-based tools share discovery logic for compile_commands.json)

Yeah, this does sound like a scalability issue if a user wants to use multiple Clang tools, as the build system extension would have to communicate with every one of them.

Maybe one idea is to stick to compile_commands.json but extend the way they are discovered, e.g. let the Cmake Tools plugin notify Clangd that a certain compile_commands file is relevant and should be considered, even if it’s not in an ancestor dir of the source file.

This sounds like a fairly reasonable compromise. But then again, we would still need a way for the build system extension to communicate this to Clangd.

Given the following circumstances:

  • There is always only one Clangd instance which is shared between different projects/folders
  • Command line arguments can only be passed to Clangd when launching it
  • Command line arguments passed to Clangd will affect all different projects/folders
  • The only ways for VSCode to communicate with Clangd after launching it is via LSP or files that both VSCode and Clangd already know about
  • LSP can not be used for this purpose
    then this problem might not actually be possible to solve. If all of this would mean completely redesigning the way that Clangd is invoked, then yeah maybe generating a symlink is the only way to go about this. I’m still not fond of polluting the source tree with something that is not super obvious that it was generated by an IDE/editor or what generated it, but maybe this is just an inherent downside of using LSP rather than an integrated C++ library (not trying to discredit your work, it’s still a great project!)…

Then the question is how the build system extension would decide where to put the symlink. Would it always end up in the root folder? Or something like CMAKE_SOURCE_DIR (these two are often the same, but I guess it’s a good idea to be specific)? What did you mean by symlinking src/.build to build/? Sorry, I never really use symlinks so I’m not very familiar with them. It seems like CMake tool actually has a config option to place a copy (not a symlink) of compile_commands.json in a path specified by the user in the config. Though I think this has to be done automatically without the user having to go into the config and set it up, maybe after clicking a “yes” button that pops up in VSCode or something. Otherwise most users will probably just assume that the two plugins simply don’t work well together (like I did). Could even have a notification to warn the user to include the file in their VCS ignore file. Of course, this is in the hands of CMake Tools and other build system extensions, just thought it was relevant to the discussion.

It’s… not trivial. (Other clang tools don’t handle this as they tend to be batch)
So we’ll fix it, but it might take a little time :slight_smile:

I’m not too worried about time, just good to hear that someone is working on it. :slight_smile:

Howdy,

the “domain” of the two tools compiler and language server are completely different, where the former usually lives in the build-tree, the latter usually lives with the user inside the source tree… So I would argue that the default lookup rather becomes the unusual case…

That being said, after long hard times the CMakeTools developer together with the Microsoft C++ extension team came up with an approach exactly like you depict somewhere around mid-year. But since the MS Extensions don’t support “find references”, it was still useless - which is why I’m currently using cquery. I bet it would be an easy thing to click into that interfacing for the clangd extension as well.

Regards,
Michael

So I would argue that the default lookup rather becomes the unusual case…

I’m sorry, I don’t quite understand what you mean by this. Could you clarify?

I bet it would be an easy thing to click into that interfacing for the clangd extension as well.

I would agree that if Clangd could use the same mechanism as Microsoft’s C/C++ extension, then it would probably be very easy to get CMake Tools to support it. However, Microsoft’s extension clearly works very differently from Clangd.
One difference is that their extension seems to handle a lot more things in TypeScript, while Clangd pretty much does the minimal amount in the vscode extension and delegates as much as possible to the Clangd program.
Another difference is how the language server system processes are run. As stated before, Clangd launches only one process when it is first activated and keeps that process around until the application is closed.
The Microsoft extension has multiple processes running at once. It seems to have one or two base processes or something and then also launches a new process for every code file that is opened. This may give the language server more freedom to isolate data, though I guess it also makes it harder to reuse AST’s and such?
The extension is open source, so one could of course look at the source code to get a better understanding of how they tackle the problem, but it’s not a trivial codebase to dive into. I guess maybe even just asking them could work too… :stuck_out_tongue:

Cheers,
Anton

So I would argue that the default lookup rather becomes the unusual case…

I’m sorry, I don’t quite understand what you mean by this. Could you clarify?

I agree with Michael here. CMake generates the compile_commands.json file in the build directory which could be anywhere. I think it would make sense for the vscode extension to have an API to allow various build tools to configure clangd properly. We need to make sure this isn’t CMake specific since all build tools in the end need to be able to generate this file. Mind you the compilation database is shared with other clang tools but ideally this all just works in an IDE setting without the user having to do additional setup.

BTW, I’m still wondering if it makes sense for the vscode extension to be in the clangd repository. We clearly won’t be adding all the client interfaces there. CDT has it’s own to hook up to the Eclipse IDE. As the CDT project is creating a debug adapter with client interfaces for both Eclipse and VS Code (yes, we’re doing both!), we occasionally have thoughts of doing VS Code client interfaces for other services. But we’re still trying to get a feel for this new world.

Doug.

The idea of using an LSP extension to communicate compile_commands.json produced by cmake to clangd should work, especially if we could also include the root of the source tree, “covered” by this compile_commands.json file (essentially the root of the project where CMakeLists.txt lives). This should fit into the overall design pretty nicely.

This way we would avoid polluting the source directory with symlinks and will be able to communicate with clangd from VSCode using our conventional LSP channel.

The downside is having yet another mechanism for configuring compilations, but I would expect it to be unified with the current -compile-commands-dir flag, so this should be acceptable.

The idea of using an LSP extension to communicate compile_commands.json produced by cmake to clangd should work

I’m not well versed in how LSP works, but if this is actually possible, it certainly sounds like the best option so far.

Considering that multiple different compile_commands.json can already be found with the current mechanism, I’d assume that this would work well for workspaces with multiple projects open as well?
In that case, I suppose that the compile_commands.json path would actually be sent to Clangd for every source file, or is there a better way to do it?
It sounds like this would also allow us to use an integration mechanism in VSCode compatible to the one that the MS C/C++ plugin uses, which may be desirable.

Anyway, I think seamless integration with build systems would greatly benefit Clangd, so it sounds like a good thing to pursue.
Would also be interesting to hear Sam’s opinion on the idea of using an LSP extension (whenever he comes back from vacation, if he hasn’t already).

Cheers,
Anton

I was thinking a signle compile_commands.json or per-project compile_commands.json. Per-file probably won’t work, Sam has mentioned it previously.

That said, a more direct approach of supporting cmake in clangd sounds even better in any case. It would allow to support any editor, not just VSCode. It will definitely take more time, though.