I forgot to mention, this is a build with -fsanitize=address. Removing this (or using -O0) makes it work.
Here are some timings for things that work and the max RSS at the end:
clang 3.9, -O1, -fsanitize=address: 3m:14.53, 922MB
clang 4.0, -O1, -fno-sanitize=address: 3m:17.31, 697 MB
clang 4.0, -O0, -fsanitize=address: 3m:22.78, 17.9 GB
I'm bisecting now and will also try my luck with creduce later.