Segfault in ASTContext.cpp

Clang Frontend
#1

Hello everyone,

As part of the SMT encoding of constraints done in the CSA, we are triggering a segfault when trying to get the integer type order between an enumType and an integer.

The code from ASTContext.cpp, starting in line 5814:

static const Type *getIntegerTypeForEnum(const EnumType *ET) {
// Incomplete enum types are not treated as integer types.
// FIXME: In C++, enum types are never integer types.
if (ET->getDecl()->isComplete() && !ET->getDecl()->isScoped())
return ET->getDecl()->getIntegerType().getTypePtr();
return nullptr; // [1]
}

/// getIntegerTypeOrder - Returns the highest ranked integer type:
/// C99 6.3.1.8p1. If LHS > RHS, return 1. If LHS == RHS, return 0. If
/// LHS < RHS, return -1.
int ASTContext::getIntegerTypeOrder(QualType LHS, QualType RHS) const {
const Type *LHSC = getCanonicalType(LHS).getTypePtr();
const Type *RHSC = getCanonicalType(RHS).getTypePtr();

// Unwrap enums to their underlying type.
if (const auto *ET = dyn_cast(LHSC))
LHSC = getIntegerTypeForEnum(ET); // [0]
if (const auto *ET = dyn_cast(RHSC))
RHSC = getIntegerTypeForEnum(ET);

if (LHSC == RHSC) return 0;

bool LHSUnsigned = LHSC->isUnsignedIntegerType(); // [2]
bool RHSUnsigned = RHSC->isUnsignedIntegerType();

The problem happens when LHS (or RHS) is a C++11 enum: getIntegerTypeForEnum [0] is called but returns a nullptr [1], and later we try to call a method from the nullptr [2].

#2

The code definitely does look like it’s missing some useful assertions, but i’m also wondering how come that we have to do something during symbolic execution that doesn’t ever happen during “concrete” execution. That’s usually a good indication that we’re doing something wrong - not even in SMT API but above it, in the code that calls it.