Cracking the Code: Lessons Learned Moderating UX Research for Generative AI Products

In the ever evolving landscape of emerging tech and innovation, AI has the potential to reshape industries, support creativity, and change the way products and services are developed with the end-user in mind. We haven’t seen a technology space that’s felt this nascent in a long time, and UX research is critical to getting your products right.

When conducting research for AI-powered products, you can still use the same tried-and-tested UX research methodologies you know and love. Diary studies, IDIs, ethnography, and journey mapping, among others, are great methods to start with. But when it comes to understanding your research objectives for this modality and successfully moderating a session, the rules that usually apply may not always work. 

Moderating research for generative AI requires a higher level of flexibility, improvisation, and participant engagement, and thus, a deeper level of understanding and preparation. In this article, we identify six lessons learned from moderating hundreds of AI user research projects to help you get the insights you need. 

Embrace a dynamic discussion guide

A good discussion guide centers the research objectives, addresses stakeholder questions, and structures your time with the participant. Building a discussion guide requires considering your pacing, timing, and the order of questions, leaving room for follow-ups and exploration. But while many typical discussion guides act as a “script,” when conducting research for GenAI, you have to think of it more as an outline.

When testing generative AI tools, UX researchers need to expect the unexpected, leaving room for further exploration of unexpected paths and journeys with spontaneous follow-up questions. Participants are often reacting to early prototypes or concepts, and as a result, your questions and structure will have to be flexible. Additionally, you should be prepared to provide more context or explain advanced concepts since users will have varying levels of understanding of AI. 

For a lot of GenAI testing, you must leave room for organic interaction with the participant, allowing them to go wherever their reactions take them.

Not to mention, you can usually expect constantly shifting and changing research questions as new insights surface and new iterations of your prototype are released.

AI research projects often uncover unforeseen insights, and a flexible guide allows moderators to adapt on the fly and delve deeper into unanticipated issues in order to yield valuable insights, ultimately enhancing the product's quality and user experience.

Building great discussion guides: how to rethink the most important part of your research prep process.

Adaptability: not just another buzzword

AI development is fast-paced in nature and the products or concepts you’re testing are constantly shifting–for example, you might find yourself with an updated prototype or new features in the middle of your session schedule.

Expect the unexpected, acknowledging the potential for language model updates or personality tweaks, rushed timelines, new code, and changing stakeholder questions.

As your launch dates loom close, the pace only quickens to turn insights into action. Researchers must be prepared to update their questions, methodologies, and priorities from session to session, even on the same day.

What was relevant yesterday might be obsolete tomorrow as the marketplace changes or as your engineering team releases new iterations of the prototypes you’re testing. Iterative research ensures that your insights remain fresh and relevant, ultimately leading to a more refined and successful product.

The power and importance of stakeholder engagement

Higher-ups wield significant influence in AI projects, and as a researcher, you may find yourself working with not only a range of roles (product, engineering, design), but also stakeholders all the way up the chain of command.

You might even have several high-profile stakeholders watching your sessions live or reviewing recordings. Engaging with them effectively and considering the best way to report on your insights is key to the success of a research project.

More than likely, you will be working with stakeholders who have a big impact on the project and the company’s AI direction as a whole. Researchers need to be confident and self-assured in their work and findings, no matter the line of questioning.

Advocating for user needs is a critical part of this process and being able to stand behind your findings is a surprisingly important part of working with high-profile stakeholders who may ask tough questions. Building strong relationships with your stakeholders not only ensures that your research is valued, but also enhances collaboration to keep the timeline moving forward.

Experienced and passionate researchers will take the project far

Seasoned UX researchers and subject matter experts are essential for navigating the complexities of AI-driven projects. GenAI research demands a deep understanding of the technology, user behavior, and industry-specific nuances, as well as a passion for emerging technology.

And, as we’ve mentioned above, researchers need to be flexible with their discussion guide and be able to hold their own when challenged or questioned by executives. Experienced researchers bring expertise to the table, ensuring that research efforts are insightful and aligned with the project goals.

Consider in-person settings for added flexibility and security

Generative AI prototypes and concepts are often top-secret and highly confidential, meaning you may want to consider conducting research in-person for added privacy and security measures.

Depending on what you’re testing, it’s important to see how the participant is interacting with the prototype live, (e.g. what their hands are doing, how the motions play out), which can be difficult to capture remotely. After years of remote-only research, consider what it looks like to do this kind of research in-person and how you can create a space for participants to interact with these prototypes comfortably.

The final puzzle piece: Creative, open-minded participants and intentional moderation skills

AI tools are new to most people - even early adopters are just starting to dabble with generative AI platforms. AI user research studies often tread into uncharted territory, requiring participants who are open to AI and can envision the unknown. 

Typically, with new product research, we don’t want the participant to make too many leaps of abstraction. Best practice often says to give participants something concrete and solid to react to in order to get high-quality insights. But with new AI technology, there isn’t always something solid to react to. To get quality insights for these types of projects, you need creative, open-minded participants.

Look for individuals who can think outside the box
and value innovation during the screening process.

They should be able to independently interact with the prototype, helping to uncover new use cases, potential pain points, and solutions that might otherwise go unnoticed. UX researchers must account for this and have open session time to really guide participants through the process. 

Lastly, as a moderator, you need to be able to draw a vibrant picture and guide participants to think of all the possibilities, including things that don’t exist yet or that they may not have experienced. Even if you’re enthusiastic about this type of technology, it can be hard to conceive of all the unknowns.

People may feel afraid of seeming foolish or throwing out crazy ideas–it’s your job as the researcher to create a space where they feel they can share without inhibition. Inventive and open-minded participants will ultimately provide valuable insights into the user experience of cutting-edge AI applications. 

***

AnswerLab has conducted over 100 UX research studies focused on AI experiences, ranging in methodologies, product areas, and research topics. Learn more about UX research for AI-powered products