Hey everyone, you may have noticed that LLVM’s Phabricator instance has been replaced with a static archive. As has been pointed out, this was not consistent with the last update to the turndown plan. Phabricator was taken offline because someone has compromised the Sendgrid API key used to send code review email. This API key was used to send phishing emails, and Sendgrid disabled the account, resulting in a failure to send emails from Phabricator. Sendgrid first notified us about the phishing on Nov 23, and @nikic reported that Phabricator failed to send email starting around Nov 23.
We at Google suspected that the Phabricator instance was compromised, and decided to expedite the turndown. @maskray drove the process of creating a static archive, and he documented the technical details in a blog post. We have yet to confirm that the VM was compromised, and it is possible that an attacker acquired the API key from somewhere else.
This incident occurred over the holidays, resulting in a complete lack of communication about the status of Phabricator, which I, as somebody in the loop, apologize for. Fangui Song deserves a big thanks for doing the actual work of creating the static archive.
Going forward, I hope we can do some forensic analysis of the VM image to determine if it has been compromised and if so what access was gained. To set expectations, however, I am not a security engineer, and unless we find more resources to look into this, I’m not expecting to get clear answers. My hope is that we find no evidence of a compromise, but I will do my best to communicate what I can going forward.
Regarding the status of the archive, I recommend that everyone review any of their remaining pending patches in Phabricator. Some users have reported that the archive is incomplete, and any efforts to validate the backup are appreciated. @maskray has compiled a complete list of unclosed Differentials in this CSV file, which you can search for your Phabricator username. It is possible to improve the archive in the future as the data still exists, but I don’t want to overpromise.