fromAugust 2015

You Know It Don't Come Easy

Drupal 8 Usability Study

House Of Blues Boston, MA
Drupal has long been described as a content management system for developers. It’s been criticized for its Drupal-centric jargon and characterized as unfriendly to inexperienced and experienced web site creators alike. In the DrupalCon Barcelona 2007 State of Drupal address, project creator Dries Buytaert stressed the need to focus on Drupal’s usability.

Not long afterward, the first formal usability study took place at the University of Minnesota, just after the release of Drupal 6 in February, 2008. Several studies of Drupal 7 were conducted in subsequent years. In June, 2015, community members returned to the university for Drupal 8’s first formal evaluation.

These formal usability tests are just one metric about Drupal’s user experience. Anyone who has introduced a new site builder to Drupal, or tried to help a Dreamweaver-savvy friend get started, has a pretty good idea where existing major challenges lie. has methodology suggestions to empower anyone to conduct their own studies, which can take place any time. New features in Drupal 8 are evaluated as they’re introduced, as well. For example, the Drupal User Experience team has conducted more than 70 informal sessions on Drupal 8-specific changes. The formal studies, however, lend a certain gravitas to recommendations for improvements; as we return to Barcelona for DrupalCon 2015, the history from formal evaluations provides a valuable metric to reflect on how far the project has come.

When I was invited to attend Drupal 8’s study, I was eager and hesitant. Eager, because who doesn't want to geek out on eye tracking feedback and all the experience-capturing equipment while spending focused time with key players who are working toward sorely needed improvements? Hesitant, because four years into the development of Drupal 8 seemed like a difficult time in the cycle to introduce meaningful change.

The 2015 Study

The usability team observed study participants through one-way glass as they approached a series of scenario-based tasks. In addition, audio of the participants’ think-aloud monologue played in the observation room along with dual-screen output. One screen showed camera footage of the participant working through the tasks. The other screen displayed the participant’s computer screen, overlaid with red lines from eye-tracking software.

Each session followed a single person for seventy-five minutes. In all, seven experienced website creators participated. Some had prior Drupal experience, others were experienced with different tools. All were new to Drupal 8.

Usability scenarios deliberately avoided using product-specific language (like content type, for example), but the ones for this study were intended to prompt testers to:

  • Create a basic page;
  • Add it to the main menu;
  • Create a content type with fields, likely referencing users and taxonomy terms;
  • Author and edit content on a mobile device;
  • Place the preconfigured “Who’s New” block on the home page.

There were, of course, other valid ways to accomplish the scenarios, as the participants quickly demonstrated.

What Was it Like?

The sessions corroborated the perspective I've gained over the years as a teacher and trainer: Drupal's user interface supplies inadequate support for site builders and in places feels like it’s actively working against them. People can learn to use it, but for many, initial self-discovery is tough and even with one-on-one guidance and all of the internet at their disposal, it can still be incredibly exasperating.

I’m used to working with frustrated Drupal users, but I never expected exactly how difficult it would be to watch users try to do something and not be able to help them. By the end, I felt like I was in a milder software variation of a Milgram experiment where the real test was how much frustration we, the observers, could tolerate before we were willing to end the exercise.

It’s hard to take comfort in the realization that the intensity of their annoyance was mainly because, being participants in a study, they persevered beyond the point where they would have chosen a different approach: an external form creation tool, another CMS, or the long-hand creation of HTML pages and navigation.

What’s Different About a Formal Study

Thorough and nuanced descriptions of what participants encountered are available online. Odds are, if you’ve built a Drupal 6, 7, or 8 site with the admin interface, you’ll see very familiar concerns. You might wonder, as I used to, if it really requires a formal usability study to identify most issues. It does not. By being in the room, though, I learned that uncovering issues is just one part of the value.

There were two major benefits to the formal study I’d not considered:

First, observing people as they encountered Drupal 8 for the first time with other people from the Drupal project – people empowered to lead change – mattered. I learned how differently we can interpret the same observations. Sometimes, those interpretations were just different, but often, one or the other of us made assumptions that weren’t warranted or missed something important that happened. In those cases, someone else would question the assumption, point out a detail, or offer a counter interpretation, and adjustment was easy. What could take years in an issue queue and still not be clarified often amounted to a ten second blip in the conversation.

Second, having skilled people (with distance from the project, and using an established methodology) in charge (recruiting participants, scheduling the sessions, facilitating attendance – both physical and remote – providing and tending to equipment, and dealing with the unexpected), allowed the Drupal community observers to stay focused.

In the observation room, having a highly skilled, neutral facilitator to keep conversation on track, and cut off premature discussions of solutions or long digressions about past history, was likewise incredibly helpful. Our facilitator was also masterful at interacting with the testers to reduce bias, and at eliciting their feedback and expectations.

You can watch the sessions online. If you do, I recommend you make it a group activity, with other Drupal people in the same room. It changes everything.

So How Did Drupal 8 Test?

Mobile Was a Highlight

Participants tested mobile functionality on either an iOS tablet or an iPod at the lab. Minor issues cropped up, but overall they created and edited content on the mobile devices with only occasional inconvenience.

Creating a Basic Page for the First Time? Less So.

Over half the participants struggled to find where to add a page – even when they followed the main Content link – because they were looking for the word “page” somewhere in the interface. But page was also more than a label. The idea of a web page as the entire thing you're viewing in your browser, or even as something whose underlying markup and code you can edit directly (for both content and navigation) – permeated users' struggles with Drupal 8.

Other Trouble Spots

The Field UI is one particularly troublesome area that illustrates tension between the mental models of a developer and a site builder. Even as an experienced user, I find it unfriendly, and it certainly was for participants, who were searching for “Dropdown” and had generally unhappy remarks about a list that starts with “Boolean.”

Overall, participants focused on their users and how those users would interact with the site, not with the database and how data would be stored. Compare the widget-first interface from Typeform with the Drupal interface in Image A.

Figure 1, Comparing Typeform and Drupal.

Adding fields was tough in Drupal 7, too, but at least the widget select list was nearby to give faster confirmation, if you were on the right track. We can talk about whether it would be better to restore that dependent Widget field, put the most-used choices at the top, remove the data-type sub-headings, better label Text (formatted) and Text (formatted, long), but that won’t adequately address the difference between an outside/in and an inside/out perspective.

If the interface isn’t for site builders who don’t write code, then I’m not sure who it is for. Site builders who do write code? They seem to avoid the interface at every turn, favoring tools like Features, Drush, Git, Drush Make files, installation profiles, and next, CMI. Without agreement on who Drupal’s admin interface serves and respect for those users, it seems hard to bridge the usability gaps.

What Impact Will This Study Have?

As with many things in Drupal core development, it's hard to say what will change. We can look to the impact of the Drupal 7 research for an idea. Conducted in 2011, the Drupal 7 usability study prompted creation of 80 issues.. Four years later, over half of them remain unresolved.[1]

Many changes are frozen in the beta phase. UX changes, however, are prioritized during this period and, according to product manager Angela Byron, nearly all of the 145 issues from this year's study could be addressed in some fashion prior to Drupal 8’s release.

What Gets in the Way?

A number of things can hamper change:

  • Some people blame site builders for a lack of education, skill, or intelligence, and see no need to change.
  • Others disregard difficult user experiences as anecdotal or dismiss formal participants as outliers.
  • Some believe that Unix command line skills, debugging with PHPStorm, and confidence with Git are requisite site building skills.
  • Varying perspectives can make it difficult to agree on whether an issue is actually a problem.
  • An issue may lack a champion to lead it through the community discussion in order to get agreement on the nature of the problem.
  • Even with agreement about the problem, people cannot always agree on the appropriate course of action.
  • When both problem and solution have enough support, there still may be no one willing to do the work or to shepherd the code from its initial patch through the requirements that make it committable.
  • A subsystem maintainer may still decline to maintain the code and reject it.

What’s Next, Then?

It’s traditional at this point to invoke the do-ocracy and tell folks to roll up their sleeves, get in there, and give back!

It’s hard, though, to confront fundamental usability issues that were identified seven years and a major release ago. Recent core governance clarifications may provide a more effective path for addressing usability, but without a cultural commitment to site builders, it seems that the disconnect between developers and site builders will likely remain unresolved.

[1] Issues may be be closed as time goes on. Closed issues also include duplicate, won't fix, works as designed and other states that could indicate that usability issues weren't sufficiently addressed.

The Elephant

While usability has improved in non-trivial ways since Drupal 6, I don’t see much progress on the foundational problems.

Here are some conceptual barriers in Drupal 6, from the first formal study in 2008:

  • Where do I start? Missing are step-by-step, task-based, conceptual help, tutorials, and example content.
  • Where did my page go? Users often lose all sense of context.
  • What is “content”? The word “content” is used ambiguously throughout the user interface: content type, content management, create content, etc.
  • How do I add a form to my page? Drupal doesn't communicate its mental models well. Users thought content types were fields, content types were content, etc.
  • Where's that key word? Words like"form" and "field" are hardly used in the interface, so users resort to guesswork.
  • What do my users see? There’s no clear distinction between admin and user-level views, and no way to preview things like node add forms as you're creating them.

The University of Minnesota, Usability, and Drupal

The University of Minnesota selected Drupal 7 in November, 2012 as a replacement for their end-of-life Oracle-based content management system and began rolling out sites in March, 2014. According to Steve Nguyen, Service Director for Collaboration & Web Content Services in the Office of Information Technology, accessibility and usability were two of many factors in the university's choice of Drupal. The project's commitment to accessibility in the form of WCAG 2.0 AA compliance means programs using Drupal are assured they are adhering to federal guidelines. In addition, they have been developing a tool to tailor site and content creation to the university’s needs and reduce the steep site builder’s learning curve.

Hosting the Drupal 8 Usability session, Nguyen says, is one of their ways of giving back to the Drupal community. The university’s centrally-funded usability lab plays a key role in evaluating their customizations and site designs. Outside organizations using the lab are rare, said User Experience Analyst Nick Rosencrans, because of the high internal demand for its services.

The Drupal 8 usability team's desire to live stream the sessions required two significant changes to their usual process. First, Rosencrans said, they worked closely with the Office of the General Counsel to develop a talent release form that allowed participants to consent to the remote presentation of the lab’s composite video of them, rather than just the video of their screens.

Live streaming required technical adaptations as well. Joe Finnegan, Technology Support Staff, said the most significant challenge was integrating with the existing lab technology. The key was to treat the audio and visual input from the existing system as USB devices so they were selectable as “presenters” in the conferencing software in order to display video full-screen to remote observers. He says that with the success of this session, they plan to begin design of a more permanent approach.

The Drupal project may continue to see long-lasting benefits from these changes. Bringing people together physically is logistically and financially challenging. Remote access to sessions makes it more feasible to identify issues in user experience before major version releases.

Image: "House of Blues by Stephen Dwyer is Public Domain


Great article.I wonder though *lights fuse and runs* whether Drupal's collaborative development model, based around, is the best way to design UX. Excellent UX design often comes from very small high performance teams who have complete control over the design, rather than from very large open collaborations following the open source development model. Reading some of the issue trackers for UX on is painful. Another way is needed.

Have you seen the issue discussing how to tackle big UX changes? It came out of DrupalCon Barcelona and the Usability session there. It looks like a good place to bring up your concerns. At least, participants in the thread seem to be echoing your sentiment that adjustments to the existing process are in order.