This unique and entertaining SXSW Interactive panel consisted of debates pinning the top industry user testing techniques against each other. The execution was content rich, organized, and surprisingly fun.
Round 1: Focus Groups vs. Field Studies
– Context is key. By immersing yourself in the user’s world via field studies, you can uncover tacit findings not consciously available.
– “What people say, what people do, and what they say they do are entirely different.”
– Focus group risk factors: human memory fail, suggestibility, social norms, self-enhancement, self-serving facilitators
– When it comes to the all-important factors of time and money, field studies have an Achilles heel. It takes significantly longer to interview participants individually than in a group.
– Think outside the focus group. Focus groups can be remote and you can even give a pre-group homework assignment to ensure a full data set.
Winner: Field Studies
Round 2: High Fidelity Prototypes vs. Rapid Iterative Testing
– Those on the side of high fidelity prototypes argue that users need the full experience to understand a design.
– Rapid iterative testing can be as simple as a sketch and works in a cyclical process (test, revise, test, revise).
– Exhibit A (from the prototypes team): Angry Birds wireframe
Do you think that by looking at this a user could understand the experience that has made Angry Birds the most successful app of all time?
– Getting end user feedback earlier in the design process is an important part of iterative testing because the longer you spend working on something, the harder it is to change it.
– Participants are less apt to give negative feedback on designs that have a “finished” appearance (because they see how much work has gone into it).
Winner: Rapid Iterative Testing (although a combination of both is recommended)
Round 3: Eye-Tracking vs. Un-Moderated Remote Usability Testing
– With remote usability testing, you can test hundreds of people simultaneously while they’re in their natural environment.
– Time and money again rear their heads taking a significant hit on eye-tracking studies, which require resources like a facility and moderator.
– Big brands that use eye tracking include Levis, Mattel and PBS.
– When using an automated, remote system, there are fewer logistics to manage, such as no-shows and last minute replacements.
– Critical success factors for conducting eye-tracking studies: the right recruiting, right methodology, right analysis, right point in the process
Winner: Un-Moderated Remote Usability Testing