[RFC][clangd] Discover C++ module interfaces missing from the compilation database

Problem

clangd currently has two backends for discovering project module information:

  • CompileCommandsProjectModules recovers module-to-BMI-to-source mappings from compile commands.
  • ScanningAllProjectModules scans source files to discover provided and required modules.

CompoundProjectModules prefers compile-command information and falls back to scanning.

Both backends ultimately depend on files enumerated by CompilationDatabase::getAllFiles(). A newly created module interface that is not yet in compile_commands.json is therefore invisible to provider discovery.

For a specific unknown file, InterpolatingCompilationDatabase can still synthesize a compile command:

getCompileCommands("test.cppm") // inferred command
getAllFiles()                   // test.cppm is absent

As a result, clangd can parse an opened test.cppm, but it cannot discover that the file provides module test when resolving an import from another translation unit.

This is the cause of llvm/llvm-project#199564.

The problem is not limited to open files. A provider may exist on disk without being open, be created after its consumer was parsed or predate the current clangd session.

Possible Solutions

Observe Open Documents

clangd could record files passed to ClangdServer::addDocument() and include them in module scanning.

This provides immediate feedback when the module interface is open but misses unopened files and providers that predate the session.

Observe Filesystem Events

workspace/didChangeWatchedFiles can report created, changed and deleted sources. ClangdServer::onFileEvent() currently receives these events but does not use them.

This covers unopened files created during the session, but clients may omit events and clangd may start after the provider was created.

Walk the Project Source Root

When an import cannot be resolved, clangd could search the governing project root for C++ source files.

This covers unopened providers and missed events but can be expensive in large projects. The walk must be lazy, cancellable, filtered and performed at most once for a given project state.

A source-root walk is suitable as a fallback but not as the primary discovery mechanism.

Change CompilationDatabase::getAllFiles()

The interpolation layer could remember queried files and include them in enumeration.

This would make generic CDB enumeration dependent on point-query history and could affect background indexing and other consumers. It would also still miss files that had never been queried.

Module-specific discovery should not change the generic CDB contract.

Query the Build System

A build system can provide an authoritative mapping from a consumer and module name to the selected provider.

This is preferable for large projects and duplicate module names but no general clangd interface currently provides such queries. Newly created files may also be unknown until the build system is regenerated.

Build-system mappings should remain preferred where available with source discovery as a fallback.

Proposed Solution

Introduce a project-scoped module source registry combining:

  1. Files opened in the editor.
  2. Files reported by filesystem events.
  3. Files found by a lazy project walk after provider resolution fails.

Existing CDB entries remain available through the current backends and do not need to be duplicated in the registry.

The resolution flow becomes:

consumer imports M
  |
  +-> query compile-command mappings
  |
  +-> scan CDB files and registered candidates
  |
  +-> if unresolved, inventory the project once
  |
  +-> retry provider resolution
  |
  +-> build or reuse the BMI

This provides fast incremental discovery during normal editing while retaining a fallback for startup and missed filesystem events.

The initial implementation will operate on saved files. Unsaved module interfaces require draft-aware scanning and separate BMI lifetime rules.

High-Level Implementation

ModuleSourceRegistry

Add:

clang-tools-extra/clangd/ModuleSourceRegistry.h
clang-tools-extra/clangd/ModuleSourceRegistry.cpp

With an interface similar to:

class ModuleSourceRegistry {
public:
  ModuleSourceRegistry(
      const GlobalCompilationDatabase &CDB);

  void observe(PathRef File);
  void remove(PathRef File);

  std::vector<Path> candidatesFor(PathRef Consumer) const;

  bool ensureProjectInventory(PathRef Consumer,
                              const ThreadsafeFS &TFS);
};

The registry is responsible for project partitioning, path normalization, synchronization and inventory state. For directory-based CDBs, ProjectInfo::SourceRoot can initially identify a project.

ModulesBuilder

ModuleSourceRegistry will be owned by the session long ModulesBuilder:

class ModulesBuilder::ModulesBuilderImpl {
  ModuleFileCache Cache;
  ModuleNameToSourceCache ProjectModulesCache;
  ModuleSourceRegistry Sources;
};

Add transport-independent forwarding methods:

void observeSource(PathRef File);
void removeSource(PathRef File);

ClangdServer::addDocument() and ClangdServer::onFileEvent() update the registry through these methods.

ProjectModules

Extend ProjectModules with an optional scanning hook:

virtual void
setAdditionalFilesForScanning(std::vector<Path> Files) {}

CompoundProjectModules forwards the candidates to ScanningAllProjectModules.

The scanning backend then scans the deduplicated union of:

CDB->getAllFiles()
+ ModuleSourceRegistry candidates

Additional sources continue to use the existing CDB point lookup, preserving command interpolation and clangd command mangling.

Lazy Retry

If provider lookup fails specifically because no source is known, ModulesBuilder inventories the project and retries once:

auto Result = buildWithKnownSources(File, TFS);

if (Result.failedBecauseProviderIsMissing() &&
    Sources.ensureProjectInventory(File, TFS))
  Result = buildWithKnownSources(File, TFS);

The project walk should be cancellable, deduplicate concurrent requests, avoid VCS and cache directories and run only once for the current project state.

Questions

  • Is a project-scoped registry the right place to combine document, filesystem, and fallback discovery?
  • Which existing clangd path filters should the lazy project walk reuse?
  • Should custom CDB implementations expose an opaque project identity instead of relying on ProjectInfo::SourceRoot?

cc @ChuanqiXu

I think it is not a question whether or not to do it. The problem you pointed is a real problem. Now we rely on CDB too much. But it is not a nice experience if users have to build their projects first and then writing codes. So it is always good to fix the problem.

Some questions:

  • Does Project Inventory means scanning every file in the project? If yes, that may be slow. It will be a blocking issue for large code base.
  • The structure may be better to add another ProjectModules to CompoundProjectModules. The structure looks better in this way. And it can be owned by Modules builder like CacheProjectModules.

For your question:

  • Is a project-scoped registry the right place to combine document, filesystem, and fallback discovery?

I think document and filesystem should be different project modules provider. I don’t like the idea to scan all files.

1 Like

Thanks, that makes sense.

By project inventory I meant walking the source files under the
project’s source root.

I agree that this can be expensive for large codebases. I will rework the proposal so that each source of additional module candidates is represented by a separate ProjectModules implementation and composed through CompoundProjectModules.

class ProjectModules {
public:
  enum class ModuleNameState {
    Unknown,
    Unique,
    Multiple,
  };

  using CommandMangler =
      llvm::unique_function<void(tooling::CompileCommand &, PathRef) const>;

  virtual std::vector<std::string> getRequiredModules(PathRef File) = 0;
  virtual std::string getModuleNameForSource(PathRef File) = 0;
  virtual std::string getSourceForModuleName(llvm::StringRef ModuleName,
                                             PathRef RequiredSrcFile) = 0;
  virtual ModuleNameState getModuleNameState(llvm::StringRef ModuleName) {
    return ModuleNameState::Unknown;
  }

  virtual void setCommandMangler(CommandMangler Mangler) {}

  virtual ~ProjectModules() = default;
};

One question:

Do we only care about getSourceForModuleName or do we prefer to provide concerete implementation for other methods as well ?

Do we only care about getSourceForModuleName or do we prefer to provide concerete implementation for other methods as well ?

For other information, I believe scanning will be the best. Because the information may be outdated. If we use other mechanism to get the information, we have to use scanning to verify if the information is still up-to-date.


Wait, if we can cache the other information and the mtime of the corresponding file, we can get the module name of the source faster.

But if we want to get the imported module of the source, we may have to cache the mtime of its included headers as well, which will be much more complex.


So we only care about getSourceForModuleName.

1 Like

@Bigcheese

1 Like