Understanding the Experience Gap: The Next Step in Creating Inclusive User Experiences

An older woman is catfished on a popular dating app. She feels so much shame she tells no one, even though there were no safeguards in place in the app to protect her. Black guests of a hospitality company are refused lodging because of racism. Young women hide their gender when gaming so they don’t experience harassment by the men on the platform. Black students are not “seen” by facial recognition AI in their classroom’s meeting software.

Many of us have either experienced or witnessed scenarios like this. While designers and product teams of these products and services didn’t set out to exclude users or cause them harm, these situations are unfortunately very common once products are out in the world. For many product teams, inclusive research is seen as a “nice to have.” But too often, if there had been an inclusive research approach to these products, the product teams would have seen and heard this feedback prior to launch. With the help of research with historically underrepresented groups, they could have not only stopped the harm from being done, but also prevented negative coverage in the press. 

The Importance of Inclusive User Research

At AnswerLab, we believe inclusive research is a critical piece of every research program. We’ve conducted over 60 research studies with underrepresented groups, gathering feedback and perspectives from participants who felt unsafe and often unseen by product teams. Not only have we heard pain points that prevented exclusion; we’ve also uncovered areas of opportunity for clients to expand on and improve their products to promote inclusion. For example, in a study on a new grocery delivery service, we heard from a participant on the autism spectrum who used the service to help cope with some of her own challenges. This inspired the product team to create benefits and features that fit with this new previously unknown use case.

Across all of our research projects, AnswerLab practices inclusive recruiting to ensure we’re including a diverse group of participants in every research study. But this is only the first step in creating truly inclusive experiences. Inclusive research is not a checkbox on a path to launch, but rather a fundamental change to product development lifecycle. To take the next step, you need to research the Experience Gap.

Diving into the Experience Gap

An Experience Gap is a gap between a product experience for underrepresented users and dominant culture users. To study the experience gap, the team explicitly gathers data to understand or benchmark how a specific under-represented group experiences the product and whether or not it’s different than their current assumptions. Teams can use this research to identify any potential harms or uncover helpful areas that can be improved or expanded on.

We recommend product teams use research to identify and measure if concepts or prototypes meet the needs of specific populations. Products are typically designed from a dominant culture point of view, which in the U.S. is a white, male, cis-gendered, able-bodied point of view. Even with a diverse team, underrepresented voices are often drowned out in favor of this dominant culture. It’s imperative to conduct Experience Gap research so the needs, goals, challenges, concerns, and harms of underrepresented populations are seen and heard by the product team. 

How to Conduct Experience Gap User Research

Step 1: Do your homework on their experience

Your research session should not be your only source of data to understand a group’s experience and background. Remember, your participants are not there to educate you on who they are or how the population experiences the dominant culture. Do your homework by following experts and reading about their experiences, lifestyles, wants, needs, and relationships before conducting the research so you have a firm foundation of their needs and challenges.

Step 2:  Formulate clear research questions

Ensure your research is successful by putting in clear and purposeful preparation time. To start, identify the experience the team would like to research. An experience is an explicit set of steps with a clear start and stop point that a user must go through to complete a task. Examples of experiences are sign-up/registration or onboarding; core tasks of playing, posting, searching, and finding; seeking help; and canceling.

Write explicit research questions to understand the population’s interaction with that experience and how it makes them feel, the underlying reasons for their experience, the impact on the product’s actions, their trust in the brand or company assisting them, and an ideal solution or next step. If you’re showing a prototype, spend time asking for examples of when the solution would be helpful or when it may be harmful to them. 

Some examples include:

1. Learning the circumstances where participants felt uncomfortable when using the product.
2. Understanding whether prototypes meant to address their concerns resonate with the participants
3. Exploring gaps between the prototypes and the participants’ own experiences.

Step 3: Fully understand and define the underrepresented population to recruit.

The deliberate choice to conduct Experience Gap research means that all participants fit into an identity group, so the screener should reflect this definition and allow a participant to choose how they identify. 

When we started to re-evaluate our screeners to promote more inclusive recruiting, our first version of the new screener mentioned “Black” as one of the ethnicity choices. After recruiting our pilot study, we discovered that the participants were primarily first-generation immigrants from Africa and our answer choice failed to include Foundational Black Americans whose family has experienced many generations of racism. We applied that learning to our next iteration of the screener and published our learnings in The Many Shades of Black. It’s important to clearly define the identity criteria used in a screener to make sure there’s an accurate representation of the population because their attitudes, beliefs, and perceptions were different between two groups. 

Step 4: Select the moderator with intention.

Do not automatically assume that your moderator or researcher running the project has to be the same identity as the population. This may cause them undue harm or tokenize the researcher. If the topic requires a participant to be vulnerable or share deeply personal details, make sure the moderator is trained in reflective listening so the participant feels heard. If the participants feel more comfortable sharing in their native language or ASL, then use a bilingual moderator.

Step 5: Moderate with awareness of your unconscious bias.

Everyone has unconscious bias and brings attitudes and perspectives to every interaction. Research moderation is a conversation with the goal of collecting impactful data and making the participant feel welcomed and comfortable. A good moderator inspires trust, improvises, and exhibits flexibility throughout the session. However, in their discussion guide and in the moment, the moderator may bring in their unconscious bias. Never, ever treat a population as “less than” you or members of the dominant culture. Always show respect for each participant and their views. 

As you moderate, watch for these types of bias:

1. Affinity bias: Make sure that you’re not agreeing with or showcasing a participant's feedback because they are like you or you like them. 
2. Confirmation bias: Focus on asking the same questions of everyone in the same tone and manner so you collect data regardless of your opinion of them. Use caution with improvisation and don't share any harmful assumptions with the participant.
3. Attribution bias: Don’t discount a participant’s response if they express themselves in a way you don’t like or prefer.
4. Halo bias: Do not promote or hold any participant’s feedback and observations as “better” than others, especially those that provide soundbites in the manner of the dominant culture, simply because you think they may be more acceptable to the team. 
5. Horns effect: Do not downplay or dismiss feedback by participants who you may not like or may not be like you.

Step 6: Report insights fearlessly and fully.

The purpose of Experience Gap research is to find what’s missing in an experience for a specific population. This research can sometimes surface surprising and far-reaching findings. To showcase all areas of concern and offer potential solutions, your reporting must be fearless and intentional.

Highlight participant stories of how the experience gap affects them and makes them feel. Don’t be afraid to share blue-sky solutions or suggestions. Ensure your team can act on the feedback by providing tangible suggestions and results. 

When using quotes and stories in your reporting, aim to use quotes from a mix of participants. Don’t quote the same participants repeatedly. Provide short-term, actionable insights to enable teams to make small, but meaningful changes. Finding small steps along the way to greater progress enables teams to act quickly to resolve the experience gap. When sharing your report, ask for accountability on who will take on resolving larger issues, track the recommended changes and next steps, and follow up with the team regularly.

...

Want to learn more about experience gap research? Watch our recent webinar on the topic: Next Steps in Inclusive Research: How to Bridge the Experience Gap

AnswerLab's Kristin Zibell, Principal UX Researcher, and Shakima Jackson-Martinez, DEI Manager, will share advice for navigating stakeholder resistance, determining the inclusivity of your product, and crucial considerations for designing, moderating, and reporting on Experience Gap research.