clang headers - can they be included from a memory location?

Graham_Wakefield2 · October 17, 2010, 8:12pm

It's great that an embedded clang can compile source files supplied as string buffers (rather than files). We wondered if there any way to coerce clang to do something similar for headers? That is, to resolve #include <xxx.h> to use a pre-loaded string buffer of the contents of xxx.h, instead of the file itself? This would make our embedded system much easier to distribute...

I tried using CompilerInstance.getPreprocessorOpts().addRemappedFile(name, buffer), but it doesn't appear that the header search resolves it. Would a PCH help?

Thanks in advance

Doug_Gregor · October 18, 2010, 2:14pm

It's great that an embedded clang can compile source files supplied as string buffers (rather than files). We wondered if there any way to coerce clang to do something similar for headers? That is, to resolve #include <xxx.h> to use a pre-loaded string buffer of the contents of xxx.h, instead of the file itself? This would make our embedded system much easier to distribute...

I tried using CompilerInstance.getPreprocessorOpts().addRemappedFile(name, buffer), but it doesn't appear that the header search resolves it.

Are you giving it a full path name, and is that path in the search paths? This should work, IIRC.

Would a PCH help?

PCH could be an optimization; it generally doesn't change behavior.

- Doug

Graham_Wakefield2 · October 21, 2010, 6:27am

Good to hear it confirmed that it *should* work! I must be doing something daft...

What would the full path name be? I want a directive like #include "remapped.h" to resolve to the contents of a preloaded buffer.

Tried 'faking it' with a path like /usr/local/remapped.h? -> error src line 10 col 10: 'remapped.h' file not found

Here's the relevant code (ModuleImpl is just a wrapper around llvm::Module) - if you see anything that looks awry, or out of order (literally), let me know!

bool compile(ModuleImpl * mImpl, std::string code) {
  const char * src = code.data();
  llvm::StringRef input_data(src);
  llvm::StringRef buffer_name("src");
  llvm::MemoryBuffer *buffer = llvm::MemoryBuffer::getMemBufferCopy(input_data, buffer_name);
  if(!buffer) {
    printf("couldn't create buffer\n");
  }

  CompilerInstance CI;
  CI.createDiagnostics(0, NULL);
  Diagnostic & Diags = CI.getDiagnostics();
  TextDiagnosticBuffer * client = new TextDiagnosticBuffer;
  Diags.setClient(client);
  CompilerInvocation::CreateFromArgs(CI.getInvocation(), NULL, NULL, Diags);

  LangOptions& lang = CI.getInvocation().getLangOpts();
  lang.CPlusPlus = 1;
  lang.Bool = 1;
  lang.BCPLComment = 1;
  lang.RTTI = 0;
  lang.PICLevel = 1;

  CI.createSourceManager();
  CI.getSourceManager().createMainFileIDForMemBuffer(buffer);
  CI.createFileManager();

  // Create the target instance.
  CI.setTarget(TargetInfo::CreateTargetInfo(CI.getDiagnostics(), CI.getTargetOpts()));

  CI.createPreprocessor();
  Preprocessor &PP = CI.getPreprocessor();

  // Header paths:
  HeaderSearchOptions& headeropts = CI.getHeaderSearchOpts();
  for (unsigned int i=0; i<options.system_includes.size(); i++) {
    headeropts.AddPath(options.system_includes[i], clang::frontend::Angled, true, false, false/* true ? */);
  }
  for (unsigned int i=0; i<options.user_includes.size(); i++) {
    headeropts.AddPath(options.user_includes[i], clang::frontend::Quoted, true, false, false/* true ? */);
  }
  ApplyHeaderSearchOptions(PP.getHeaderSearchInfo(), headeropts, lang, CI.getTarget().getTriple());

/*
  The goal is for clang to be able to resolve #include "remapped.h" (or <remapped.h> to the contents of the StringRef remapped_data.
*/
  // add file remapping:
// (doesn't work)
  llvm::StringRef remapped_name("remapped.h");
  //llvm::StringRef remapped_name("/usr/include/remapped.h");
  // some test code:
  llvm::StringRef remapped_data("#include <stdio.h> \n void remapped() { printf(\"remapped!\\n\");");
  llvm::MemoryBuffer * remapped_buffer = llvm::MemoryBuffer::getMemBufferCopy(remapped_data, remapped_name);
  CI.getPreprocessorOpts().addRemappedFile(remapped_name, remapped_buffer);

  PP.getBuiltinInfo().InitializeBuiltins(PP.getIdentifierTable(), PP.getLangOptions().NoBuiltin);

  CI.createASTContext();

  CodeGenOptions CGO;
  CodeGenerator * codegen = CreateLLVMCodeGen(Diags, "mymodule", CGO, mImpl->llvmContext );

  ParseAST(CI.getPreprocessor(),
      codegen,
      CI.getASTContext(),
      /* PrintState= */ false,
      true,
      0);

  llvm::Module* module = codegen->ReleaseModule();
  delete codegen;

/// etc.
}

Doug_Gregor · October 21, 2010, 3:49pm

It's great that an embedded clang can compile source files supplied as string buffers (rather than files). We wondered if there any way to coerce clang to do something similar for headers? That is, to resolve #include <xxx.h> to use a pre-loaded string buffer of the contents of xxx.h, instead of the file itself? This would make our embedded system much easier to distribute...

I tried using CompilerInstance.getPreprocessorOpts().addRemappedFile(name, buffer), but it doesn't appear that the header search resolves it.

Are you giving it a full path name, and is that path in the search paths? This should work, IIRC.

Good to hear it confirmed that it *should* work! I must be doing something daft...

What would the full path name be? I want a directive like #include "remapped.h" to resolve to the contents of a preloaded buffer.

Tried 'faking it' with a path like /usr/local/remapped.h? -> error src line 10 col 10: 'remapped.h' file not found

You mean /usr/local/include/remapped.h, I assume? I think that should work. I think I see the issue, below...

Here's the relevant code (ModuleImpl is just a wrapper around llvm::Module) - if you see anything that looks awry, or out of order (literally), let me know!

bool compile(ModuleImpl * mImpl, std::string code) {
  const char * src = code.data();
  llvm::StringRef input_data(src);
  llvm::StringRef buffer_name("src");
  llvm::MemoryBuffer *buffer = llvm::MemoryBuffer::getMemBufferCopy(input_data, buffer_name);
  if(!buffer) {
    printf("couldn't create buffer\n");
  }

  CompilerInstance CI;
  CI.createDiagnostics(0, NULL);
  Diagnostic & Diags = CI.getDiagnostics();
  TextDiagnosticBuffer * client = new TextDiagnosticBuffer;
  Diags.setClient(client);
  CompilerInvocation::CreateFromArgs(CI.getInvocation(), NULL, NULL, Diags);

  LangOptions& lang = CI.getInvocation().getLangOpts();
  lang.CPlusPlus = 1;
  lang.Bool = 1;
  lang.BCPLComment = 1;
  lang.RTTI = 0;
  lang.PICLevel = 1;

  CI.createSourceManager();
  CI.getSourceManager().createMainFileIDForMemBuffer(buffer);
  CI.createFileManager();

  // Create the target instance.
  CI.setTarget(TargetInfo::CreateTargetInfo(CI.getDiagnostics(), CI.getTargetOpts()));

  CI.createPreprocessor();
  Preprocessor &PP = CI.getPreprocessor();

  // Header paths:
  HeaderSearchOptions& headeropts = CI.getHeaderSearchOpts();
  for (unsigned int i=0; i<options.system_includes.size(); i++) {
    headeropts.AddPath(options.system_includes[i], clang::frontend::Angled, true, false, false/* true ? */);
  }
  for (unsigned int i=0; i<options.user_includes.size(); i++) {
    headeropts.AddPath(options.user_includes[i], clang::frontend::Quoted, true, false, false/* true ? */);
  }
  ApplyHeaderSearchOptions(PP.getHeaderSearchInfo(), headeropts, lang, CI.getTarget().getTriple());

/*
  The goal is for clang to be able to resolve #include "remapped.h" (or <remapped.h> to the contents of the StringRef remapped_data.
*/
  // add file remapping:
// (doesn't work)
  llvm::StringRef remapped_name("remapped.h");
  //llvm::StringRef remapped_name("/usr/include/remapped.h");
  // some test code:
  llvm::StringRef remapped_data("#include <stdio.h> \n void remapped() { printf(\"remapped!\\n\");");
  llvm::MemoryBuffer * remapped_buffer = llvm::MemoryBuffer::getMemBufferCopy(remapped_data, remapped_name);
  CI.getPreprocessorOpts().addRemappedFile(remapped_name, remapped_buffer);

You need to add the remapped file before you create the Preprocessor object. Otherwise, the remapping doesn't migrate from the options in CompilerInvocation over to the Preprocessor object itself.

- Doug

Graham_Wakefield2 · October 22, 2010, 3:29pm

What would the full path name be? I want a directive like #include "remapped.h" to resolve to the contents of a preloaded buffer.

Tried 'faking it' with a path like /usr/local/remapped.h? -> error src line 10 col 10: 'remapped.h' file not found

You mean /usr/local/include/remapped.h, I assume? I think that should work. I think I see the issue, below...

Email typo!

You need to add the remapped file before you create the Preprocessor object. Otherwise, the remapping doesn't migrate from the options in CompilerInvocation over to the Preprocessor object itself.

Brilliant, that works! Thanks!