Build a UX Benchmarking Program: Two Approaches

We get a lot of questions from our clients about benchmarking research. What is it? How do we do it? Will it help us meet our goals? It’s easy to get lost in the storm and feel stuck not knowing how to proceed. Over the years, we’ve refined our benchmarking best practices to address all of these questions and create a streamlined, effective approach for our clients.

At its core, UX benchmarking is a comprehensive test of usability, pairing hard metrics with qualitative insights to get a high-resolution perspective on the effectiveness of your products. Unlike small scale usability tests, benchmarking aims to understand how the components of a system contribute to overall usability, and how a product as a whole meets the needs of its users.

Benchmarking any product, digital or hardware, starts by asking: What is this product intended to do, and how well is it doing it? 

What can benchmarking accomplish?

Benchmarking provides both comprehensive “big picture” frameworks for overall product health and detailed usability information in observational and participant feedback data. The combination gives you available knowledge and a holistic, prioritized picture of product health.

The pairing of observational and participant feedback provides data for several analytical approaches, giving you a range of benefits and outcomes. Here are some benefits of benchmarking programs we see with our clients:

• Understand overall product health
  Benchmarking is an excellent approach for gathering a baseline dataset to understand which parts of your product work well and which need improvement. For instance, a financial services client who recently started to engage in UX research sought to evaluate their whole app through UX benchmarking. Their goal was to make sure the product team had clear data on where they were succeeding and where they wanted to improve, with the intention of repeating the benchmarking in the future to gauge how effective their changes were.

• Build a competitive analysis
  Benchmarking’s focus on metrics allows for comparison between similar competitor products to compare scores and see where one product outperforms another. A media and entertainment client recently chose benchmarking to understand where they out- or under-perform their key competitors in the user experience.

• Diagnose critical usability issues
  Explanations of benchmarking metrics provide usability information to target both the major and minor usability issues in a system. For example, one of our technology clients wanted to assess potential pitfalls in their usability for those using assistive technologies. We helped them design a benchmarking project to draw out critical usability issues.

• Prioritize improvements to solve usability issues
  After diagnosing usability issues within success/failure metrics, necessary improvements to a product can be prioritized based on the scale of the impact of the issue. A social media client who had done qualitative research to understand pain points in their interface wanted to know how big of an impact the pain points had on the experience. We used benchmarking to determine which improvements would be highest-value to implement first.

• Inform your overall product roadmap
  Benchmarking can also provide critical information about which tools in your product are most critical to invest time in to improve your UX. For instance, some of our clients with a regular benchmarking cadence prefer to do benchmarking at the beginning or end of every year. This data collection is the basis for their upcoming product roadmap and research plan. By comparing benchmarking metrics to previous years, they understand where they’ve made gains or losses in the experience and can prioritize their efforts over the next 12 months.

• Establish a proof-of-concept
  Prior to launching a new product or feature, benchmarking can validate how successful your design will be in the hands of the user. For instance, we worked with a media streaming client who was launching a redesign of their app. To validate their new design, they benchmarked both the old and new designs, comparing the measurements of both to identify the extent of the improvement.

Two Approaches to Benchmarking

Benchmarking is most valuable when repeated, to maximize the value of data comparisons over time and to capture the impact of changes in the interface. While both are repeatable approaches, depending on what you hope to accomplish with your benchmarking program you may want to pursue periodic comprehensive benchmarking or take a progressive benchmarking approach.

Comprehensive Benchmarking

You’ve likely heard of comprehensive benchmarking before, as it’s the most common approach in UX research. This form of benchmarking tests all aspects of a product at periodic intervals such as quarterly, biannually, or annually. This approach is a complete assessment of usability, testing everything your product is “meant to do” in order to gauge overall product health and identify high and low performing areas.

When to Use It

Comprehensive benchmarking is best suited when your need is for a complete or comprehensive dataset at one point in time. It may also be the best research design if your product’s components are dependent on one another and testing individual components would isolate them in a way that was highly unrealistic to the live experience.

Additionally, periodic comprehensive benchmarking is a useful approach when you want to gauge impact over time, such as year-over-year changes or when you would like to make comparisons with competitors. It’s good to use when launching or relaunching, or seeking foundational data on which to build product roadmaps.

Things to Know:

• In most cases when using this approach, we recommend sample sizes no smaller than n=20 per segment. When you cut your data by segment, this helps maintain accuracy.
• Comprehensive benchmarking maximizes time over investment as you can gather a full set of benchmarking data on your system at one time.
• For planning purposes, it’s good to remember most benchmarks of this approach take 6-8 weeks to complete from kickoff to completion.

Progressive Benchmarking

Progressive Benchmarking is a more paced approach than comprehensive benchmarking, intended to provide a consistent flow of benchmarking data to aggregate into a whole while making consistent usability improvements over time. In this approach, individual components of a system are tested at a regular pace, such as biweekly, in order to continuously assess parts of the system, building into a complete body of data about your product. This is a more “bite-sized” approach, intended to allow teams to focus on the health and usability of individual flows.

When to use it

Progressive benchmarking is best suited when you’re iterating on or releasing components of a product or product suite independently of one another and want to test them individually. This approach can be especially useful when you want to integrate benchmarking into your regular research program and build a body of data to guide your product roadmap over time.

Things to Know

• In this approach, a smaller sample size of n=10-12 can be used for each individual test because of the more qualitative usability focus; though it can be increased for more robust quantitative needs. As you test multiple times with each component the accuracy of the metric measurements will increase with the large body of aggregate data.
• Progressive benchmarking maximizes investment over time, since it provides opportunities to hyper focus on individual parts of a system.
• Most benchmarks in this style take 3-4 weeks to complete, and rounds can often be completed on a rolling basis.

These two approaches can be thought of on a spectrum of sample size, qualitative vs. quantitative, time, and investment, and either approach can be customized to be smaller or larger in scale and scope. In either approach the key to successful benchmarking is creating a custom research design that is focused on highly valid measurements for your particular product.

Want to learn more? Check out Three Common Benchmarking Metrics to Ditch and What to Use Instead.