What is the status of clang-based service architecture?

Hi,

What is the status of this idea: http://lists.cs.uiuc.edu/pipermail/cfe-dev/2012-June/022028.html

Regards,
Kocka

Nothing concrete has been checked in yet.

-- Sean Silva

:frowning:

Best idea ever! I would like to contribute to it. Who else is interested?

Best idea ever! I would like to contribute to it. Who else is interested?

At this point, I believe that most of what needs to be done is
non-glamorous work creating the infrastructure needed to support it.
For example I believe one thing that was identified as being
needed/desired was an in-tree portable networking library.

-- Sean Silva

:frowning:

Best idea ever! I would like to contribute to it. Who else is interested?

The first step is to make clang (more) thread-safe:
For example, the file interface is not thread-safe; this requires changes
to the file system / path interface to allow keeping state of the working
directory, and doing all relative file operations based on that state (on
unix fd* versions of the unix file api). This requires an objectification
of the file interface first, which requires some more design work.

Chandler & Matt will be able to chime in with more.

Cheers,
/Manuel

In your opinion, could libclang be used as the backend of a clang-based service or is just too constrained functionality-wise?

See the original proposal:
http://lists.cs.uiuc.edu/pipermail/cfe-dev/2012-June/022028.html

Cheers,
/Manuel

We (FreeBSD/Cambridge/some bits of Google) are also interested in clang using openat() and friends in the cc1 invocation and being passed all header search paths in as directory descriptors and the input / output files as file descriptors, from the perspective of sandboxing the entire compilation step.

David

Right. We need a virtual file system that can be shared among the various Clang instances executing within the service. All of Clang’s filesystem operations should go through that virtual file system, which (obviously) needs to provide thread-safe access.

  • Doug

Right. We need a virtual file system that can be shared among the various
Clang instances executing within the service. All of Clang's filesystem
operations should go through that virtual file system, which (obviously)
needs to provide thread-safe access.

Out of curiosity, is it feasible for clangd to just shepherd separate
processes for each instance that needs a different view of the
filesystem? Implementing a whole VFS just to run independent instances
inside the same process doesn't seem to make sense. I mean, the
separate instances aren't communicating, right?

Actually, presumably every "request" to clangd that is dependent on
the current working directory is going to include a parameter "use
this as the current working directory", so couldn't we just open
relative to that path (by just appending the paths)? Why is it
necessary to (virtually or not) chdir() into a directory to open
relative paths, compared to just appending them?

-- Sean Silva

Right. We need a virtual file system that can be shared among the various
Clang instances executing within the service. All of Clang's filesystem
operations should go through that virtual file system, which (obviously)
needs to provide thread-safe access.

Out of curiosity, is it feasible for clangd to just shepherd separate
processes for each instance that needs a different view of the
filesystem? Implementing a whole VFS just to run independent instances
inside the same process doesn't seem to make sense. I mean, the
separate instances aren't communicating, right?

Actually, presumably every "request" to clangd that is dependent on
the current working directory is going to include a parameter "use
this as the current working directory", so couldn't we just open
relative to that path (by just appending the paths)? Why is it
necessary to (virtually or not) chdir() into a directory to open
relative paths, compared to just appending them?

I'd assume a VFS is necessary for the IDE scenario where you're
life-compiling code in an editor buffer that's not necessarily
reflected on the concrete filesystem yet.

- David

It's feasible, but it's necessarily useful. We want all of the compilation/syntax checking/etc. jobs to go into the same clangd server process so that we can start sharing resources, e.g., the underlying buffers for header files that are read by many different translation units. And we want to know when those buffers no longer reflect what's in disk or in the user's IDE, which may invalidate certain results that are cached in the service.

clangd isn't about compiling a bunch of different translation units to object code through a server rather than separate processes, it's about having a stateful server that can answer questions like "what code completions are valid at foo.cpp:55:4?" efficiently from relatively dumb clients.

  - Doug

Is this all information we have now https://github.com/chandlerc/llvm-designs/blob/master/ClangService.rst?
Or are there other places where we can find more info?

I’m just trying to figure out where can I help.

Is this all information we have now https://github.com/chandlerc/llvm-designs/blob/master/ClangService.rst?
Or are there other places where we can find more info?

Unfortunately, that’s basically it for now.