How to Conduct Research to Understand Your Customers’ Logged-In Experiences

Studying the logged-in experience of your product: How to make a long-term impact with your research

Researching and improving your customers’ experience while they’re shopping for and purchasing your products can help make a short-term impact on your sales funnel.

But to understand how to improve customer loyalty and retention, you need to prioritize understanding and addressing gaps in the logged-in experience. One way to do this is by measuring the usability of your product in a natural environment and evaluating points of friction for your customers when they are logged in.

From personally identifiable information (PII) concerns to session logistics, getting insights into people’s logged-in experience is challenging and complex. People are often uneasy showing their account information and personal data during a research session. 

If you’re unsure where to start, here are tips and strategies from our team for creating a successful session experience.

Recruiting for your logged-in experience study

You are asking people to share personal information beyond a typical study, so building trust with your research participants is the key to successful logged-in experience studies. First and foremost, you have to make sure they’re comfortable and understand the expectations. This starts with the recruiting process. 

Our Research Operations team recommends addressing specific points and expectations to ensure your recruit is successful:

Communicate with participants about the ‘what’ and ‘why’

To build trust and start a project with transparency, ensure your participants fully understand and are comfortable with what you ask them to do in the session. It’s helpful to explain why you’re asking them to do these tasks. Understanding how you’ll use the information and why it’s helpful to the research process can help participants feel more comfortable participating, rather than leaving things up for interpretation.

Provide participants with examples

Many participants will feel less comfortable screen sharing their logged-in experience given the level of personal information shown, whether that’s account balances on an online banking product or their street address and saved credit card number on a shopping app.

Clarify exactly what you’re looking to study by providing participants with examples (and screenshots, if possible) of what they would and would not need to share during a research session. Examples set proper expectations and allow participants to prepare accordingly.

Plan accordingly to ensure you find great participants

Sometimes recruiting for these kinds of studies takes extra effort and time due to the more significant participation requirements. Ensure you build in spare time for the recruit to make sure you find participants who are comfortable doing a live screen share. Building extra time also grants you greater flexibility in pivoting session dates and scheduling needs if the recruit is more difficult than expected.

Preparing and conducting sessions

Now that you’ve recruited them, it’s time to make your study a seamless and comfortable experience for your participants while prioritizing getting the insights you need. Our research experts recommend following these steps to ensure study success.

Have a backup plan

If a participant can’t share their screen for some reason due to discomfort or technical issues, provide at least one alternative way to get their feedback. Here are some creative options and workarounds we’ve used: 

• Requesting participants complete a short homework assignment a few days or a full week before the session that includes screenshots of their logged-in experience

• Asking participants to log in and think aloud about their experience without screen sharing. It won’t get you screenshots or a visual reference, but you’ll gather their thoughts and feedback in a more comfortable environment for the participant

• Showing a concept or prototype similar in functionality to the research topic so the participant can compare and contrast it with their logged-in experience

• Extending session dates and prioritizing flexibility to recruit additional participants who are willing to screen share

Before sessions, set proper expectations with observers 

For any research session, we ask observers to mute their microphones and keep their video off. We also set the following expectations around logged-in studies:

• When sending internal Slack messages or emails about the session, observers need to respect participants’ privacy and avoid sharing any PII. For example, observers should not reference any participants’ email addresses or other information even if they appear on the screen.

• Similarly, observers should respect the participants as people, regardless of the data they share. So, for example, if you’re interviewing business owners and how they use financial products, observers shouldn’t comment on their balances seeming low or high if that isn’t the topic of the study.

During sessions, address any participant questions or concerns

Remind participants of the format of the session and provide examples of what they will and will not be asked to share. Let participants know they are free to stop screen sharing at any point for any reason, and assure them that all information is kept confidential and anonymous. You can also set expectations by letting participants know they won’t need to begin their screen share until after they’ve logged in to avoid showing their username and password information.

Take note of what to blur out from session recordings

At the start of the session, ask participants to mention anything they want blurred, especially if anything unexpected appears during the screen share. As the researcher, write down specific timestamps of when information should be blurred, and let the participants know you are doing so to increase their comfort level. Write down these timestamps if any sensitive information is shown (e.g., account numbers, username or password, social security number, or home address), even if the participant doesn’t mention or notice it. If you call attention to it, it may make them feel less comfortable, but this information should always be blurred due to privacy reasons.

Regardless of your product, whether an investment account or shopping app, studying your customers' logged-in experiences will help you make significant long-term improvements to customer loyalty, retention, and the usability of your product.