It always happens: Someone asks me about screwing up as an amateur facilitator/moderator for user research and usability testing sessions. This time, I had just given a pep talk to a bunch of user experience professionals about sharing responsibility with the whole team for doing research. “But what if the (amateur) designer does a bad job of moderating the session?”
What not to do
There are numerous ways in which a moderator can foul things up. Here are just a few possibilities that might render the data gathered useless:
- Leading the participant
- Interrupting or intervening at the wrong time
- Teaching or training rather than observing and listening
- Not following a script or checklist
- Arguing with the participant
Rolf Molich and Chauncey Wilson put together an extensive list of the many wrong things moderators could do. There are dozens of behaviors on the list. I have committed many of these sins myself at some point. It’s embarrassing, but it is not the end of the world. So, here, let’s talk about what to do to be the best possible moderator in your first session.
Your role as a moderator
To be the best moderator you can be, remember that there are three basic roles of the moderator in user research and usability testing. When Carolyn Snyder worked for User Interface Engineering, she codified these:
Flight attendant. Though you might think that your priority is collecting data, your number one job during the session is to see to the comfort and safety of the participant. Make sure this person is comfortable, is appreciated, and knows she can stop at any time. Set up a relaxed situation that is still focused on the goal of learning from the person.
Sportscaster. The line of sight and the acoustics of the session situation aren’t always ideal for the observers. Because the observers from your team will be helping you take notes and analyze the data, you can help them by talking just enough so they can keep their places in the session. For example, if the participant is vague about a UI element in pointing out goods and bads, simply echo the last couple of words the participant said to get them to clarify or expand.
Scientist. The moderator is usually the person who designed the study and will be responsible for analyzing the data that comes out of it. This means managing any recordings to ensure the privacy of the participants, tracking notes and data gathering from observers, and pulling observations and data together so the team can come to a design direction based on the evidence gathered.
(Hat tip to Carolyn Snyder and Jared Spool for the moderator roles.)
Who should moderate UX sessions?
Who makes a good moderator? Anyone who is a quick learner, is a good listener, can build rapport with a participant, and has a good memory. Typically, there isn’t a lot of time to know all the nuances of a UI before going into a usability test. Likewise, if you’re in the field doing basic ethnographic research, you may learn characteristics of the participants or the environment that inform the rest of the interview direction. Handling those on-the-fly perceptions will help everyone get value out of the session.
The listening is important for asking insightful follow-up questions as participants think aloud. Getting clarification on comments made, drilling in a bit to get to specifics, and always keeping in mind “why is this behavior happening” will come to you from listening and (gently) questioning.
Rapport with the participant is key to creating trust. The participant is always trying to get a reading from you about whether what he’s doing is correct and whether what he’s giving you is what you want. Even a newbie to moderating can be friendly, objective, and neutral at the same time. (It may take some practice.)
Remembering what happened early in the session will help you ask useful follow up questions later in the session. Remembering the main, interesting behaviors will help you work with the observers after the session is over and you’re all telling the story of what happened, especially if you have assigned someone else to take notes while you concentrate on running the session.
If there’s someone on the design team besides you who has these characteristics, that’s who you want to moderate sessions, no matter what their regular job is.
How to be a great UX moderator
Keeping those roles and attributes in mind, this is what I tell clients and workshop attendees about how to be a good moderator. You can pass the list below to your team’s candidate.
Be willing to let go of ownership of the design. When you’re doing field research, you may enter the session with design ideas in mind. Try not to. Instead, let the heft of the data over sessions help build the ideas. If you’re testing a design, as soon as you put a design in front of another person, you no longer own it, the person you’re showing it to or who is using it owns it. In that act, you have specifically asked for reactions and interactions. Open yourself up to the possibilities.
Shut up. After you’ve explained the purpose of the session, and explained your role, and the roles of the other people in the room, stop talking. Even when there are silences, don’t be too quick to fill them. Wait. Count to 20 slowly and silently before you say anything. Chances are, something interesting has happened by then and you won’t have to open your mouth.
Listen. This is not the same as shutting up. Listening is about being present. Be fully attentive so you can not only hear the words, but process their meaning. Be empathetic to the participant and what she’s trying to do.
Suspend judgment. This is one of the hardest things to do, but it is also the most important. You have invited the participant to help you learn about what you’re designing. If you have shut up, and listened well, and the participant is appropriate for the study, then let go of assessing what is happening in that moment. Give yourself time to process later. This will also prevent you from asking inappropriate questions during the session that may betray your feelings about a participant or what she has to say about the design.
Plan ahead. Script, create checklists, and read Beth Loring and Joe Dumas’s book, Moderating Usability Tests. You may feel silly using a script, but really, there’s absolutely nothing wrong with doing that. Having a script to follow means you say the same things the same way to every participant. You can also ensure that you’ve hit everything on the list that the team wanted to learn about. Finally, scripting out what to say and thinking through the checklist of focus questions will give everyone a better feeling for how much can reasonably be covered during a given session.
Rehearse. If you don’t set up a pilot session, then your first “real” session will end up being the rehearsal. Practice in a dry run by yourself, out loud. Record it. Change the script if you need to, making sure that the words you say make sense and feel authentic. Then find someone down the hall or in the next cube to play your participant and try the script out again. Make changes if you need to.
Do enough sessions. As you moderate each session, you will get better at it. Remember, it’s not about you. Though you may feel awkward doing this in front of your team, and reading from a script, they’re not paying attention to you, they’re paying attention to the participant. If you feel like you have made a mistake – you said the wrong thing, or asked a question the wrong way, or you led the participant somehow, keep going. And then, go do another session. You don’t have to throw everything out from the session you made a mistake in. Salvage what you can and move on.
It’s a chance for team members to get closer to the participants
The whole object of doing user research is for the team to learn about current experiences. More data, even data gathered sloppily, is better than a tiny bit of data gathered expertly. And the new moderator will get better at it. No one is born to the role; moderating well is a set of learned skills. And I think that anyone can learn them with time, practice, and coaching. Find your next moderator on your design team. And keep up the good work.
Other resources you might find useful:
Common Problems in Usability Test Facilitation by Rolf Molich and Chauncey Wilson
Moderating Usability Tests: Principles and Practices for Interacting (Interactive Technologies), by Joseph S. Dumas and Beth A. Loring
Remote Research: Real Users, Real Time, Real Research (Rosenfeld Media), by Nate Bolt and Tony Tulathimutte
Seven Common Usability Testing Mistakes, by Jared Spool
Moderating with Multiple Personalities: 3 Roles for Facilitating Usability Tests, by Jared Spool
Usability Tools Podcast: Moderating Usability Tests, Part 1
Jared Spool and Brian Christiansen
Effectively Moderating Usability Tests, with Beth Loring
Recorded virtual seminar from UIE ($149)