ClangD

Hi fellow clang devs,

we wanted to let you know that we’re (finally) starting up work on ClangD, which you might know from email threads such as [1] (June 2012!).

In the meantime, YCM had done a good enough job at packaging up a libclang interface to our favorite editors, and other priorities (like modules) have eaten up a lot of folks priority lunches.

What has changed?

  1. YCM is starting to develop more and more into a language multiplexer, with other languages (Go, Typescript, etc) providing their own servers to talk to
  2. MS has created a language server protocol [2], which already has both a bunch of client and server implementations
  3. Debugging through python into libclang crashers is a pain and eating our support resources
  4. While libclang is a good abstraction if you want to have something run in your process with close coupling, a standard protocol like the language server protocol seems like a better way to allow fast iterations on the server implementation without the need to keep backward-compatibility hacks through a restrictive C API.

One of the cool things about the MS language server protocol is that it seems to not actually do any networking, which means that we do not need to introduce any new dependencies into clang-tools-extra, which is where we want to start developing this.

If you have any thoughts / concerns let me know; otherwise look forward to code reviews on initial ClangD dropping by :smiley:

Cheers,
/Manuel

[1] http://lists.llvm.org/pipermail/cfe-dev/2012-June/022028.html
[2] https://github.com/Microsoft/language-server-protocol

Good news, good luck!

Thanks Manuel, I’m quite excited to hear that about ClangD!

I have a couple of questions:
Do you think ClangD would be able to replace libclang in the future? Would it be possible to deprecate libclang for IDE use after ClangD catches up to it, or should we keep libclang as it is even after that point?

Thanks Manuel, I’m quite excited to hear that about ClangD!

I have a couple of questions:
Do you think ClangD would be able to replace libclang in the future? Would it be possible to deprecate libclang for IDE use after ClangD catches up to it, or should we keep libclang as it is even after that point?

In the foreseeable future, I don’t see libclang going away. Not everybody is able to switch to a new development workflow easily, and there is a lot of investment in the current libclang based workflows.

Hi,

I think this is an excellent idea which should be pursued as far as possible as it will enable us to seamlessly integrate multiple clang-based services into the development environment. Possibly it will enable us to integrate non-clang-based kind of services as well. I might say that I’ve developed a (prototype) solution which somewhat resembles this idea (implementation goes up to extent of my current needs & time resources). See [1] for a short overview but in general it is a generic, message-queue based, client-server architecture which acts as a thin proxy layer between the client requests and concrete service implementation (i.e. syntax-highlighting, clang-format, etc.). Server transparently makes sure that each service is run in its own context (thread/process) and provides full-control of services (startup-(all), shutdown-(all), run, etc.): see [2] for server implementation. This way a service developer can completely put focus on service implementation details itself without caring about other details. See [3] for base service implementation and [4] & [5] for examples of concrete service implementations (clang-format service & syntax-highlight service).

What would be a way to get involved with ClangD?

Cheers,
Adi

[1] https://github.com/JBakamovic/yavide/blob/master/docs/services_framework.md#framework
[2] https://github.com/JBakamovic/yavide/blob/master/core/server/yavide_server.py
[3] https://github.com/JBakamovic/yavide/blob/master/core/services/yavide_service.py
[4] https://github.com/JBakamovic/yavide/blob/master/core/services/clang_formatter_service.py

[5] https://github.com/JBakamovic/yavide/blob/master/core/services/syntax_highlighter_service.py

Hi,

I think this is an excellent idea which should be pursued as far as possible as it will enable us to seamlessly integrate multiple clang-based services into the development environment. Possibly it will enable us to integrate non-clang-based kind of services as well. I might say that I’ve developed a (prototype) solution which somewhat resembles this idea (implementation goes up to extent of my current needs & time resources). See [1] for a short overview but in general it is a generic, message-queue based, client-server architecture which acts as a thin proxy layer between the client requests and concrete service implementation (i.e. syntax-highlighting, clang-format, etc.). Server transparently makes sure that each service is run in its own context (thread/process) and provides full-control of services (startup-(all), shutdown-(all), run, etc.): see [2] for server implementation. This way a service developer can completely put focus on service implementation details itself without caring about other details. See [3] for base service implementation and [4] & [5] for examples of concrete service implementations (clang-format service & syntax-highlight service).

What would be a way to get involved with ClangD?

Lurk on the cfe-commits mailing list for clangd based code reviews to come in (Benjamin is working on getting an initial prototype up), and contribute there.

Once we have more of the basic infrastructure in place, we’ll accept patches like we do for the rest of clang and have a bug list.

From the YCM side: I think this is a fantastic idea! We were thinking about pulling out the C++ completion out of ycmd and into a separate server that ycmd would talk to (much like it works for several other languages already) but hearing that clang devs are taking this on saves us a bunch of work!

And it’s great to hear it will follow the common completion API MS is pushing; we’ll be supporting that API on the back-end of ycmd some time in the future (and clangd might accelerate that). We’re looking into possibly supporting it on the front-end as well.

Hi fellow clang devs,

we wanted to let you know that we're (finally) starting up work on
ClangD, which you might know from email threads such as [1] (June 2012!).

In the meantime, YCM had done a good enough job at packaging up a
libclang interface to our favorite editors, and other priorities (like
modules) have eaten up a lot of folks priority lunches.

What has changed?
1. YCM is starting to develop more and more into a language multiplexer,
with other languages (Go, Typescript, etc) providing their own servers
to talk to
2. MS has created a language server protocol [2], which already has both
a bunch of client and server implementations
3. Debugging through python into libclang crashers is a pain and eating
our support resources
4. While libclang is a good abstraction if you want to have something
run in your process with close coupling, a standard protocol like the
language server protocol seems like a better way to allow fast
iterations on the server implementation without the need to keep
backward-compatibility hacks through a restrictive C API.

One of the cool things about the MS language server protocol is that it
seems to not actually do any networking, which means that we do not need
to introduce any new dependencies into clang-tools-extra, which is where
we want to start developing this.

If you have any thoughts / concerns let me know; otherwise look forward
to code reviews on initial ClangD dropping by :smiley:

[Huch, my first association with "ClangD" was support for the D language]

Hi!

We from the Qt Creator IDE team are also interested in this. :slight_smile:

Some notes:

1. You've indirectly referenced the initial design document at [1]. Will this get an update or is it already updated somewhere else? An updated document clarifying the scope and goals would be probably useful to all interested parties, also in regard to contribution and how to align own current development that uses libclang/libtooling/clang tools.

E.g. Qt Creator also already launches a separate process ("clangbackend") that makes use of libclang to provide the basic functionality like updated diagnostics, highlighting information and results for completion requests.

2. How is clangd related to "clang-refactor" [2]?

If it's not related at all, what's the state of clang-refactor?

3. I take this opportunity to share our biggest problems with libclang from the Qt Creator perspective since clangd users will probably face them also (as far as I understand, clangd will be based upon libclang - please correct me if I'm wrong). I share this in hope that they will be addressed or taken into consideration during the development of clangd for the benefit of all interested parties now that the development of clangd is announced. Please keep in mind that the actual problems might be in other components that libclang makes use of.

The problems we are facing with libclang, in this order, are:

3.1 Windows-specific issues

The whole performance of parse/reparse in general is worser than on Mac/Linux (probably also because of slow IO on Windows). At the end of the day, this makes it less usable on Windows for real-world projects/files. There are also other Windows-specific issues.

My gut feeling is that there aren't that many libclang users on Windows.

@Manual: To which extend do you or your team use and develop on Windows? Are any Windows developers working on clangd?

Concrete issues are:

  * Preamble is also generated for the second reparse on Windows
    https://llvm.org/bugs/show_bug.cgi?id=28535

  * Problems with FileManager caches leading to leaking file descriptors
    https://reviews.llvm.org/D27810

  * Very slow reparsing for big header and even slower with
    preamble-generation enabled (might be related to the one above)
    https://llvm.org/bugs/show_bug.cgi?id=31353

  * Windows: Crash with absolute path with forward slashes
    https://llvm.org/bugs/show_bug.cgi?id=28381

3.2 Synchronous/blocking API
Low-latency completion results at all times are impossible with a single CXTranslationUnit that is just being parsed or reparsed. This gets worse the longer the parse/reparse takes (hello Windows).

One can mitigate this by triggering the parse/reparse after a significant timeout of user-typing-inactivity, but this leads to a sluggish experience regarding new diagnostics and highlighting informations (you don't want to wait more than 2s for new diagnostics to see if your code is valid now).

Another approach is to use two CXTranslationUnits per file, so at least there is always a non-blocking CXTranslationUnits ready for completion requests. Downside is doubled memory consumption and potentially threading issues (@Manual: do they still exist or are they already resolved?).

3.3 libclang requires unsaved files to exist on disk.

If the unsaved file does not exist a preamble generation will happen for every reparse which simply takes to long.

For the concrete use case, refer to

  https://llvm.org/bugs/show_bug.cgi?id=31112

This one might be also a problem if you do version control checkouts and files suddenly vanish.

Best regards,
    Nikolai

[1] https://github.com/chandlerc/llvm-designs/blob/master/ClangService.rst
[2] http://lists.llvm.org/pipermail/cfe-dev/2016-September/050641.html

    Thanks Manuel, I'm quite excited to hear that about ClangD!

    I have a couple of questions:
    Do you think ClangD would be able to replace libclang in the future?
    Would it be possible to deprecate libclang for IDE use after ClangD
    catches up to it, or should we keep libclang as it is even after
    that point?

In the foreseeable future, I don't see libclang going away. Not
everybody is able to switch to a new development workflow easily, and
there is a lot of investment in the current libclang based workflows.

Just for clarity, please elaborate on "lot of investment in the current libclang based workflows".

Hi fellow clang devs,

we wanted to let you know that we’re (finally) starting up work on
ClangD, which you might know from email threads such as [1] (June 2012!).

In the meantime, YCM had done a good enough job at packaging up a
libclang interface to our favorite editors, and other priorities (like
modules) have eaten up a lot of folks priority lunches.

What has changed?

  1. YCM is starting to develop more and more into a language multiplexer,
    with other languages (Go, Typescript, etc) providing their own servers
    to talk to
  2. MS has created a language server protocol [2], which already has both
    a bunch of client and server implementations
  3. Debugging through python into libclang crashers is a pain and eating
    our support resources
  4. While libclang is a good abstraction if you want to have something
    run in your process with close coupling, a standard protocol like the
    language server protocol seems like a better way to allow fast
    iterations on the server implementation without the need to keep
    backward-compatibility hacks through a restrictive C API.

One of the cool things about the MS language server protocol is that it
seems to not actually do any networking, which means that we do not need
to introduce any new dependencies into clang-tools-extra, which is where
we want to start developing this.

If you have any thoughts / concerns let me know; otherwise look forward
to code reviews on initial ClangD dropping by :smiley:

[Huch, my first association with “ClangD” was support for the D language]

Hi!

We from the Qt Creator IDE team are also interested in this. :slight_smile:

Some notes:

  1. You’ve indirectly referenced the initial design document at [1]. Will
    this get an update or is it already updated somewhere else? An updated
    document clarifying the scope and goals would be probably useful to all
    interested parties, also in regard to contribution and how to align own
    current development that uses libclang/libtooling/clang tools.

So far the scope is replacing the C++ parts of YCM (that is, code completion, code browsing, “fast” syntax checking and fixits)

E.g. Qt Creator also already launches a separate process
(“clangbackend”) that makes use of libclang to provide the basic
functionality like updated diagnostics, highlighting information and
results for completion requests.

  1. How is clangd related to “clang-refactor” [2]?

If it’s not related at all, what’s the state of clang-refactor?

clang-refactor is still in the design stage. I think at some point we’ll probably want to integrate clang-refactor, but we first want a prototype and see how well that goes.

  1. I take this opportunity to share our biggest problems with libclang
    from the Qt Creator perspective since clangd users will probably face
    them also (as far as I understand, clangd will be based upon libclang -
    please correct me if I’m wrong). I share this in hope that they will be
    addressed or taken into consideration during the development of clangd
    for the benefit of all interested parties now that the development of
    clangd is announced. Please keep in mind that the actual problems might
    be in other components that libclang makes use of.

The problems we are facing with libclang, in this order, are:

3.1 Windows-specific issues

The whole performance of parse/reparse in general is worser than on
Mac/Linux (probably also because of slow IO on Windows). At the end of
the day, this makes it less usable on Windows for real-world
projects/files. There are also other Windows-specific issues.

My gut feeling is that there aren’t that many libclang users on Windows.

@Manual: To which extend do you or your team use and develop on Windows?

We are not Windows developers :slight_smile: We hope that by integrating with VS Code / MS language protocol we can get some interested parties on Windows, but so far we rely on help from Windows wizards.

Are any Windows developers working on clangd?

Well, it’ll be an open source project :wink:

Concrete issues are:

3.2 Synchronous/blocking API
Low-latency completion results at all times are impossible with a single
CXTranslationUnit that is just being parsed or reparsed. This gets worse
the longer the parse/reparse takes (hello Windows).

One can mitigate this by triggering the parse/reparse after a
significant timeout of user-typing-inactivity, but this leads to a
sluggish experience regarding new diagnostics and highlighting
informations (you don’t want to wait more than 2s for new diagnostics to
see if your code is valid now).

Another approach is to use two CXTranslationUnits per file, so at least
there is always a non-blocking CXTranslationUnits ready for completion
requests. Downside is doubled memory consumption and potentially
threading issues (@Manual: do they still exist or are they already
resolved?).

I work on a very large codebase and don’t have problems with code completion. I think all YCM integrations I know take care to not block the UI for completion requests (or in general).

3.3 libclang requires unsaved files to exist on disk.

If the unsaved file does not exist a preamble generation will happen for
every reparse which simply takes to long.

I would be surprised if that was true - I often work on unsaved files without seeing these problems.

Thanks Manuel, I’m quite excited to hear that about ClangD!

I have a couple of questions:
Do you think ClangD would be able to replace libclang in the future?
Would it be possible to deprecate libclang for IDE use after ClangD
catches up to it, or should we keep libclang as it is even after
that point?

In the foreseeable future, I don’t see libclang going away. Not
everybody is able to switch to a new development workflow easily, and
there is a lot of investment in the current libclang based workflows.

Just for clarity, please elaborate on “lot of investment in the current
libclang based workflows”.

YCM exists, and people use it. Xcode exists and people use it. There is a python integration based on libclang. Probably more users :slight_smile:

Hi Manuel,

If you have any thoughts / concerns let me know; otherwise look forward to
code reviews on initial ClangD dropping by :smiley:

One minor point, which I could not spot on any of the replies: is it
worth looking at RTags[1]? Its a libclang-based project with very good
emacs integration. I've been using it daily on a large-ish code base
and its pretty stable. Adding the MS protocol to it would probably be
easier than starting from scratch. (I am in no way affiliated with the
project, other than as a user).

Also, the name ClangD seems to evoke D support for Clang :slight_smile:

My 0.02 Angolan Kwanzas.

Cheers

Hi Manuel,

If you have any thoughts / concerns let me know; otherwise look forward to
code reviews on initial ClangD dropping by :smiley:

One minor point, which I could not spot on any of the replies: is it
worth looking at RTags[1]? Its a libclang-based project with very good
emacs integration. I’ve been using it daily on a large-ish code base
and its pretty stable. Adding the MS protocol to it would probably be
easier than starting from scratch. (I am in no way affiliated with the
project, other than as a user).

YCM is a libclang based editor integration that many people use - we specifically want to get away from libclang (for various reasons).
For what it’s worth, I also don’t think adding the editor protocol to a different project will be easier than starting from scratch.

<rtags>

YCM is a libclang based editor integration that many people use - we
specifically want to get away from libclang (for various reasons).
For what it's worth, I also don't think adding the editor protocol to a
different project will be easier than starting from scratch.

Ah, I see - missed that part. Well, eagerly awaiting for CLangD then. :slight_smile:

Cheers

Hi Manuel,

If you have any thoughts / concerns let me know; otherwise look forward to
code reviews on initial ClangD dropping by :smiley:

One minor point, which I could not spot on any of the replies: is it
worth looking at RTags[1]? Its a libclang-based project with very good
emacs integration. I've been using it daily on a large-ish code base
and its pretty stable. Adding the MS protocol to it would probably be
easier than starting from scratch. (I am in no way affiliated with the
project, other than as a user).

Also, the name ClangD seems to evoke D support for Clang :slight_smile:

I've heard this from multiple people, so the official spelling should
be clangd or Clangd in the traditional unix way :wink:

Trying to prevent future naming collision issue: clangd seems too generic to me and likely to collide with a general purpose clang compilation server, unless you plan to have a single clangd that can service multiple type of service (indexing, compiling, linking, …).

Hi Manuel,

If you have any thoughts / concerns let me know; otherwise look forward to
code reviews on initial ClangD dropping by :smiley:

One minor point, which I could not spot on any of the replies: is it
worth looking at RTags[1]? Its a libclang-based project with very good
emacs integration. I’ve been using it daily on a large-ish code base
and its pretty stable. Adding the MS protocol to it would probably be
easier than starting from scratch. (I am in no way affiliated with the
project, other than as a user).

Also, the name ClangD seems to evoke D support for Clang :slight_smile:

I’ve heard this from multiple people, so the official spelling should
be clangd or Clangd in the traditional unix way :wink:

Trying to prevent future naming collision issue: clangd seems too generic to me and likely to collide with a general purpose clang compilation server, unless you plan to have a single clangd that can service multiple type of service (indexing, compiling, linking, …).

Compiling and linking is within scope. It is not yet clear in what form.