After many years of managing and performing research, I’ve noticed some similar misnomers about usability research among colleagues and friends. While I agree that research may result in creating additional cycles of design iteration or some beta programming, it is an upfront investment that almost always yields longer-term success.

I know it is difficult for industry experts to go into a dark observation room and listen to participants criticize their product, but it is one of the most important processes an expert can go through prior to the launch of their next “big thing.” Consumer testing provides the necessary feedback to aid in product design, development and consumer acceptance.

Below are some of the myths I’ve run across when it comes to conducting usability studies:

#1 Traffic data can tell us all we need to know, besides, 6 people can’t yield useful insights

• Why This is False: The best projects are those that combine the how and why. We can make anyone click on a big, flashy button, but you don’t know the user’s intention. When it comes to qualitative testing, reliable trends are typically seen after 6 similar individuals are interviewed.
• Example: A client had asked the research team to investigate a recent drop in revenue on one of their most trafficked pages after a recent minor redesign. If a user searched on a term, the results page displayed both sponsored links and search results. The new search results page appeared to follow industry best practices. However, performance data showed that users were less attracted to the sponsored links on the new design when compared to the previous design. We conducted usability testing to find out why. Users were shown both versions in random order (old vs. new). The only difference between the versions was that there was a “button” graphic around the older design of the sponsored link, which made it look like an option for navigation. We watched as one user after another clicked on the button and expressed a good amount of confusion. As it turns out, most users believed that the button on the old design would further filter the search results. The old design drove more click-thru revenue simply because users misunderstood what the button would do. The new design eliminated this confusion, but unfortunately it also reduced revenue in the process.

#2 Anyone can conduct usability

• Why This is False: Moderation requires someone who can remain an unbiased party. Moderators typically receive training on test structure, flow of information, reading and interpreting non-verbal cues, the ability to handle recruiting and screening for the appropriate individuals, and constructing a report of findings to meet the needs of the key stakeholders.
• Example: One research project was being led primarily by a team of designers. They worked closely with the product and engineering teams to develop various versions of their prototypes. When it came to conducting the interviews, the designers decided to lead the interviews so they could make quick changes to the prototypes. Unfortunately, the research proved to be a self-fulfilling prophecy. To no one’s surprise, users always favored the design that the moderator was responsible for creating.

#3 You need a fully operational prototype

• Why This is False: There are costs associated with building out a design and/or prototype. However, testing concepts and new product ideas with simple wireframes or screen shots very early in the design process with respondents can contain cost. Any testing is better than no testing.
• Example: A client was torn between using a top navigation scheme vs. a left-side navigation scheme. There were merits to both approaches and differing opinions among the internal stakeholders. Because extensive engineering time would be required to build out even semi-functional prototypes of both, the research team suggested testing flat HTML mock-ups. We gave respondents scenarios and asked them where they would go, what they would expect to get if they clicked, etc. We also gave the participants two examples of competitor’s websites, one with a top navigation and one with side navigation. To the client’s surprise, the respondents gave extremely rich feedback with just basic stimuli, ultimately helping the client decide between the two navigation schemes with little cost.

#4 Results can’t be trusted because they’re collected in an unreal environment

• Why This is False: While users’ behavior in a lab can be different from their behavior in a customary environment, there are inherent behaviors that can be observed in the lab and translated to real life. In addition, it provides an opportunity for a deeper discussion. Benefits of lab research include the ability to:
  • See initial reaction from a subject
  • Probe more on statements or reactions
  • See reactions to products firsthand
  • Test sensitive stimuli
• Example: A recent online advertising campaign received high marks in an online survey. The company’s management team originally felt it would be more reliable to measure the ad with a large group of individuals via an online survey rather than bringing respondents into the lab. When asked about their impression of the campaign, such as their likes or dislikes, respondents claimed to be extremely interested in the campaign. However, after launch, there was little to no engagement with the ad unit on the live site. The team did not understand why “reported” interest did not reflect the true interest in the real world. To understand “why” respondents had reported interest, we took the ads to the lab. Using an eye-tracking machine, we presented respondents with various websites that contained the test campaign as well as other random advertising campaigns. While respondents claimed the test campaign was of most interest, the eye-tracking data showed that the design was less attractive compared to other campaigns on the page. The ad campaign simply did not draw users’ attention when surrounded by a website vs. the stand alone view that was tested in the survey. It became clear to the team that while the survey provided a large sample, the results did not provide the context needed to measure the respondents impressions nor did it provide an opportunity to understand the “why” of respondents ratings.

#5 Usability testing takes too long

• Why This is False: From kick off to final presentation, usability studies can be turned around in 4 weeks. In some cases this is much shorter, depending on the product, availability of the participants, number of testing days needed, travel time, etc. Some clients establish rolling testing weeks that allow the flexibility to test various products in various stages.
• Example: A large site redesign required the product development cycles to be broken out into several tracks with varying sprint lengths. There were parallel tracks on some development cycles and staggered development on others. The research team dedicated “User Testing Fridays” to accommodate all product cycles and teams. Throughout the entire development cycle, the team brought anything from sketches to fully-designed prototypes to get reactions from potential customers. This allowed teammates to take advantage of usability at various stages of development. As a result, more areas were tested and done at various stages of the development process – very quickly.

I hope that you can use the insights gleaned from these debunked myths for a successful future project.

Tags: myths, prototype, research, usability

This entry was posted on Thursday, September 10th, 2009 at 7:42 pm and is filed under Uncategorized. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.