clang refactoring/auto-completion support

Clang Frontend
#1

We've hit a couple issues while using clang for refactoring/auto-completion tools

1. clang code completion - There are two virtual methods within clang::CodeCompletionConsumer class. First is 'ProcessCodeCompleteResults'. This methods is working quite well. But second - 'ProcessOverloadCandidates', which is providing functions overload candidates within function call context, is working very strange. This method is called /only /within context of calling free function. Example:\

void Foo();

void Bar()
{
    Foo(>|<); // Overload candidates are successfully provided and contain one item ('Foo()')
}

But:
class Clazz
{
public:
    Clazz(int, int);
    void Foo();
    void TestFn();
};

void Bar(Clazz& cls)
{
    cls.Foo(>|<); // Overload candidates aren't provided
    Class other(>|<); // The same.
}

void TestFn()
{
    Foo(>|<); // The same. Overload candidates aren't provided
}

Here ">|<" mark shows a current text caret position for which code completion is requested.
Is there missing functionality here? (If so anyone interested to help us with this?)

2. Here is a sample class:

class BaseClass : public SuperBaseClass
{
public:
     BaseClass(int a);
     BaseClass(int a, int b);
     void SomeVirtualMethod(int a, int b = 10);
     void foo(int a, int b);
     void foo(double a);
     void foo(BaseClass *cls);
};

BaseClass::>|<

In the point marked with ">|<" suggestion list doesn't contain constructors of the 'BaseClass'. It's very strange.

3. AST building. We're looking to implement:
  - Create method declaration from usage
  - Create method declaration from definition

Both this features assume what source code (in initial state) contains an error and feature implementation fix this error in the corresponded way. The problem is clang can't help us to analyze the line with such error.

For example:

void Bar()
{
    int a, b;
    Foo(a, b); // There is no 'Foo' function within code yet
}

clang parser throws out the line with 'Foo' call from AST. So we can't analyze nor type of symbol or type of arguments within this context.
My primary question is: can we add some special AST nodes into clang parser in order to handle such cases? In normal situation AST consumers/visitors will ignore such nodes but in case of refactoring we could extract some additional info from source code and have special handling for it.

#2

We’ve hit a couple issues while using clang for refactoring/auto-completion tools

  1. clang code completion - There are two virtual methods within clang::CodeCompletionConsumer class. First is ‘ProcessCodeCompleteResults’. This methods is working quite well. But second - ‘ProcessOverloadCandidates’, which is providing functions overload candidates within function call context, is working very strange. This method is called /only /within context of calling free function. Example:\

void Foo();

void Bar()
{
Foo(>|<); // Overload candidates are successfully provided and contain one item (‘Foo()’)
}

But:
class Clazz
{
public:
Clazz(int, int);
void Foo();
void TestFn();
};

void Bar(Clazz& cls)
{
cls.Foo(>|<); // Overload candidates aren’t provided
Class other(>|<); // The same.
}

void TestFn()
{
Foo(>|<); // The same. Overload candidates aren’t provided
}

Here “>|<” mark shows a current text caret position for which code completion is requested.
Is there missing functionality here? (If so anyone interested to help us with this?)

  1. Here is a sample class:

class BaseClass : public SuperBaseClass
{
public:
BaseClass(int a);
BaseClass(int a, int b);
void SomeVirtualMethod(int a, int b = 10);
void foo(int a, int b);
void foo(double a);
void foo(BaseClass *cls);
};

BaseClass::>|<

In the point marked with “>|<” suggestion list doesn’t contain constructors of the ‘BaseClass’. It’s very strange.

It might have to do with the fact that one cannot invoke the constructor of a class with a qualifier in front of it: BaseClass::StaticMethod is a valid method call but BaseClass::BaseClass is not. The C++ syntax is slightly ambiguous since providing an out of line definition and calling the method have the same form, it could probably be disambiguated (more or less) by looking up the context: at namespace level and outside an expression, it’s obviously a method definition for example; however I don’t know if differentiating between the two is always so clear cut (I guess there are some weird cases).

– Matthieu

#3

We've hit a couple issues while using clang for refactoring/auto-completion tools

1. clang code completion - There are two virtual methods within clang::CodeCompletionConsumer class. First is 'ProcessCodeCompleteResults'. This methods is working quite well. But second - 'ProcessOverloadCandidates', which is providing functions overload candidates within function call context, is working very strange. This method is called /only /within context of calling free function. Example:\

void Foo();

void Bar()
{
  Foo(>|<); // Overload candidates are successfully provided and contain one item ('Foo()')
}

But:
class Clazz
{
public:
  Clazz(int, int);
  void Foo();
  void TestFn();
};

void Bar(Clazz& cls)
{
  cls.Foo(>|<); // Overload candidates aren't provided
  Class other(>|<); // The same.
}

void TestFn()
{
  Foo(>|<); // The same. Overload candidates aren't provided
}

Here ">|<" mark shows a current text caret position for which code completion is requested.
Is there missing functionality here? (If so anyone interested to help us with this?)

Yes, there is missing functionality here. I implemented only proof-of-concept functionality for providing the user with guidance about what overloads could be selected at the given point in the source code (and which parameter she is on), but I never added support for all of the various different kinds of calls.

2. Here is a sample class:

class BaseClass : public SuperBaseClass
{
public:
   BaseClass(int a);
   BaseClass(int a, int b);
   void SomeVirtualMethod(int a, int b = 10);
   void foo(int a, int b);
   void foo(double a);
   void foo(BaseClass *cls);
};

BaseClass::>|<

In the point marked with ">|<" suggestion list doesn't contain constructors of the 'BaseClass'. It's very strange.

This is because the handling of completion after '::' is really broken. It all goes through one code completion function in Sema, which assumes that you want to name the thing after the '::' in an expression. The right implementation here would be to pass the nested-name-specifier off to each of the existing code completion functions in Sema, and for them to tweak their search for results based on that nested-name-specifier.

3. AST building. We're looking to implement:
- Create method declaration from usage
- Create method declaration from definition

Both this features assume what source code (in initial state) contains an error and feature implementation fix this error in the corresponded way. The problem is clang can't help us to analyze the line with such error.

For example:

void Bar()
{
  int a, b;
  Foo(a, b); // There is no 'Foo' function within code yet
}

clang parser throws out the line with 'Foo' call from AST. So we can't analyze nor type of symbol or type of arguments within this context.

Right.

My primary question is: can we add some special AST nodes into clang parser in order to handle such cases? In normal situation AST consumers/visitors will ignore such nodes but in case of refactoring we could extract some additional info from source code and have special handling for it.

This would be a significant undertaking, although it would be beneficial to a number of libclang-based clients. I think we'd want to extend the existing Expr/Stmt nodes with an "invalid" bit, so that we can re-use the Expr/Stmt nodes we have to capture the syntactic structure of the ill-formed code. When one of the inputs to Sema's semantic-analysis routines has the invalid bit set, we exit out early, building either a new invalid Expr/Stmt node or simply returning some singleton "error expression" node that has the invalid bit. The former (building a new, invalid node) is for cases where we've parsed something, while the latter is for cases where we've hit an error during template instantiation. Pushing this through all of Clang is going to take a lot of work.

  - Doug