The Drupal Association fosters and supports the Drupal software project, the community, and its growth. That’s our mission statement, and it covers a lot of territory. We support the community through Community Cultivation Grants, Global Training Days, fiscal sponsorship of Camps, and, of course, the Cons. We support the Drupal software project by making sure everyone involved in the project has an effective place to work together as a community: Drupal.org. It’s our responsibility, and we take it seriously. We recently faced one of the toughest situations any site owner could face, and we want to share the steps taken and lessons learned along the way.
As a Watchdog reader, you are undoubtedly aware that in May, the Drupal Association reset all Drupal.org passwords due to unauthorized access to user accounts. This was not a Drupal software issue – an unauthorized user gained access via a third party application. Due to legal, liability, and security concerns, that’s all we can share about the breach itself, but what we can share is what we did about it.
The Immediate Aftermath
Once the issue was discovered, we were very fortunate that members of the Drupal Security Team were on hand, along with the Drupal.org Infrastructure Team and Oregon State University Open Source Lab, to diagnose the severity of the problem, help mitigate the issues technically, and work with DA staff to formulate and implement a public response. Working together, our first job was to understand what happened – and fix it.
The team went to work immediately to determine the scope and severity of the issue. They took down non-essential sites which had been lagging on maintenance, such as module updates. They combed logs for more information, searched file uploads for anything suspicious, and checked the integrity of project downloads. The infrastructure team started rebuilding servers that had been affected, with increased security hardening. The team also deployed phpass password hashing for more secure password hash storage, and removed admin access for non-active users.
With the scope of the breach identified and the issue addressed, the next step was to execute the password reset. Fortunately, Drupal.org and its sub-sites generally do not store sensitive user data such as birth dates, credit card information, or social security numbers or equivalents. However, a password reset was the only way to ensure that access to projects and accounts stayed safe. Working quickly, we emailed every Drupal.org account holder, posted advisories on the login pages and on the home page, and set up a help desk account and team to respond to inquiries, of which there were hundreds.
When the End is Not the End
Though we solved the problem technically and protected our users, this is not where the story ends. The breach provided us with the opportunity to assess how we got here in the first place, and how we can prevent it from happening again. To that end, we are currently working with a security firm to review our systems and practices and provide recommendations. Though the audit is not yet complete (as of July), we have already begun implementing some new process, procedures, and staffing considerations.
We are currently working to implement privilege separation between Drupal.org subsites and to implement more comprehensive, better monitored, and more fine-grained backups, among other technical improvements. We are also documenting these changes to our systems, so staff and project volunteers can more easily get up to speed with our services. Finally, we’re exploring options for funding a staff or consultant role to focus on security.
What Did We Learn?
In short, we’ve learned a lot – probably too much to list here. But there are a few key concepts that we are taking with us as we consider what we do next to secure Drupal.org. First, responding quickly and accurately is essential, and a lot easier, when one has logs and backups that show what is needed in order to pinpoint the problem and investigate the extent of the incident. Reviewing our logs and backups periodically and asking how we would track down a potential breach is essential: In other words, running fire drills as preparation for a real emergency.
Second, technology safeguards are obviously important, but in the end, it’s generally people and processes that play the biggest role in security issues. Especially for a volunteer-run project like Drupal, it’s essential that responsibilities are well defined and communicated, and processes are put in place and documented, so that nothing falls through the cracks. We will continue this work for the foreseeable future.
Lastly, we learned that our community has boundless talent and endless passion. Throughout this entire ordeal, we were amazed at the speed and accuracy of the work being done to protect the project – and at the record number of hours our volunteers could go without sleep! Now it’s our job at the DA to make sure they never have to do that again.
We wanted to share some of what happened – and our next steps – as quickly as possible, but this is not the end of the community conversation. When we receive the final report from the auditor, you will find that on the Association blog at https://association.drupal.org/blog. We look forward to continuing the conversation there.