[RFC] Implementing Randstruct in Clang

This RFC accompanies the patchset on Phabricator: https://reviews.llvm.org/D59254

The Randstruct feature is a compile-time hardening technique that randomizes the field layout for designated structures of a code base. Admittedly, this is mostly useful for closed-source releases of code (since the randomization seed would be available for public and open source applications).

Why implement it? This patch set enhances Clang’s feature parity with that of GCC which already has the Randstruct feature. There's a growing interest in compiling projects such as the Linux kernel with Clang. The Linux kernel is a good example of a large code base which uses randomized field order in certain structures to help thwart attacks that depend on structure layouts in memory.

This patch set is a from-scratch reimplementation of the Randstruct feature that was originally ported to GCC. The patches for this implementation in GCC can be found here:

 https://www.openwall.com/lists/kernel-hardening/2017/04/06/14.

This feature identifies structures for randomization in two ways. The first method targets structures that are manually marked with the new “randomize_layout” attribute. The second is an optional feature that will automatically select and randomize structures that are found to consist entirely of function pointers. This automatic selection feature can be extended to include other vulnerable structure types that are safe to randomize as they are identified. You can also opt a specific structure out of this feature with the “no_randomize_layout” attribute. Automatic structure selection is enabled with the “-randstruct-auto” compiler flag. By default, Randstruct seeds on the empty string, but a seed can be supplied with the “-randstruct-seed=” command line argument.

Clang Randstruct is the sum total of an undergraduate computer science capstone team’s effort.

Portland State University Clang Randstruct Capstone Team (Fall 2018-Winter 2019):

Co-authored-by: Cole Nixon <nixontcole@gmail.com>
Co-authored-by: Connor Kuehl <cipkuehl@gmail.com>
Co-authored-by: James Foster <jafosterja@gmail.com>
Co-authored-by: Jeff Takahashi <jeffrey.takahashi@gmail.com>
Co-authored-by: Jordan Cantrell <jordan.cantrell@mail.com>
Co-authored-by: Nikk Forbus <nicholas.forbus@gmail.com>
Co-authored-by: Tim Pugh <nwtpugh@gmail.com>