Years of mobile testing have led to a number of items that we, as user experience researchers, consider whenever we set out to conduct a test that takes place on a mobile device such as a personal phone or iPad.
In somewhat more typical test, participants will arrive at a lab and use a generic PC for the task at hand. With mobile testing, however, often the participants are asked to bring their own phone with them so that we can see how they interact with the subject using their own device. Of course, we’ve conducted plenty of studies using proprietary subjects, and in those circumstances, the participants use a ‘control’ device. Those tests are easier since you eliminate the issues that inevitably appear when participants are asked to bring in their own device.
With that in mind, and in no particular order, here are ten tips to consider when testing a website/app on a mobile device.
1. Have a backup plan – This includes a backup plan for participants, devices, and location. We’ve seen it all. The participant forgets to bring her phone. Or, he dropped his phone on the way to the study and the screen cracked. Maybe reception is bad for a certain carrier in the particular location that you are conducting sessions. All of these things can, and will, thwart your study. Trust me, I’ve seen them all happen. By assuming things will go wrong, and creating a backup plan, you can salvage almost any situation.
2. Test reception ahead of time – Before conducting the study, you should have an idea of which devices you want to test. Based on this information, do yourself a favor and sit in the room that you plan to have sessions, pull out your phone, and see what kind of reception you get. Now, get a hold of someone with a phone for each type you intend to test and repeat this process.
As a backup plan, make sure that a strong wifi network is accessible. Cell reception is notorious for dropping bars at the exact moment your session starts. Being able to quickly switch to wifi in order to carry on is important.
3. Do your homework – Some of our clients have an idea of which devices they would like to test but there are so many variations that sometimes it is difficult to be totally prepared. Doing your homework can help. For instance, one study we conducted focused on smartphones like iPhones and Android. However, our client wanted to test a few phones that fall somewhere between a smartphone and a feature phone. By working closely with our recruiter, we were able research certain models to see if the participant’s phone would be a good match.
Another good option for doing your homework is to use a virtual testing environment. Products like MITE (Mobile Internet Testing Environment) by Keynote (Disclaimer: Keynote is an AnswerLab client) provide you with an array of virtual devices which make it easy to pull up a website and see exactly how it will appear on that device.
4. Pilot test – At least a day before your first session, and hopefully sooner, conduct a pilot session. It’s best if you treat the pilot exactly like a real session so that you can find the problem areas before you actually bring in real people for testing. If you neglect all the other tips in the article, this is the one to keep. Conduct a pilot and you won’t regret it.
5. Record make/model – Even if our client doesn’t ask, we always record the make and model of the participant’s device. Either have your recruiter do it or take a moment at the beginning of each session to capture this information. It comes in handy, especially for design teams, when something unexpected happens during the test.
6. Bring phone/charger – Instruct your participants, in big, bold letters, to bring their phone and charger to the session. And then tell them again. And once more for good measure. Despite this, at least one of your participants will arrive without their phone. Or their phone will have 10% battery left. I guarantee this will happen during your mobile study.
7. Plan for screenshots – How are you going to capture screenshots for your report? On desktops, that’s simple. For mobile devices, it can be quite a challenge. iPhones, for instance, make it easy to grab screenshots. Androids, on the other hand, do not.
8. Ask questions about site/mobile site/apps in pre-interview – I’ll often include some line of questioning in the pre-interview to understand the participant’s comprehension of full website, mobile websites, and mobile apps. I have found that many participants struggle to articulate the difference between them. Though it may not be pertinent for your study, I think it is easy information to capture and gives my clients a better understanding of consumers.
9. Turn off ringer. Don’t answer texts, etc. – Just like they do at the movie theater, tell your participants that you require their undivided attention during the study. Have them turn off their ringer and encourage them to dismiss texts. Remember, mobile devices are extremely personal and people are used to reacting to any alert, in the moment.
10. Figure out how you are going to record the session and allow clients to observe. This can be a complicated and sometimes costly affair. Simple solutions are often best but every study requires it’s own solution. In another blog post, Image Capture Devices, we discuss some options that might help you sort out the best solution for your needs.
