(This article was originally published on May 30, 2008. This is a refresh.)
Research that you do alone ends up in only your head. No matter how good the report, slide deck, or highlights video, not all the knowledge gets transferred to your teammates. This isn’t your fault. It just is.
So what to do? Enlist as many people on your team as possible to help you by observing your usability testing sessions. You can even give your observers jobs, such as time-keeper if you’re measuring time on task. Or, if you are recording sessions, it could be an observer’s job to start and stop the recordings and to label and store them properly.
The key is to involve the other people on the team – even managers – so they can
- help you
- learn from participants
- share insights with you and other observers
- buy in
- reach consensus on what the issues are and how to solve them
Who should observe: Everyone
Ideally, everyone on the design and development team should observe sessions. Every designer, every programmer, every manager on the project should watch as real people use their designs. People on the wider team who are making design decisions should also observe sessions. I’m talking about QA testers, project managers, product managers, product owners, legal people, compliance people, operations people — everyone.
Each of the observers should watch as many sessions as possible. (Okay, you could settle for two sessions.) Seeing only one session is just a snapshot; what happens in that session may not be similar to what happens in the other sessions. If the observer who saw only one participant is in an influential position in the organization, that one set of observations may outweigh others when it comes time to get some consensus from observers about what the issues are. (Hint: You don’t really want that.)
Training observers: Ground rules are essential
Observers can watch from a separate space – most labs are set up this way – or they can be present in the testing room with you and the participant during each session. If you’re doing field work, take 2 with you at a time, and stream the session back to HQ, if you have enough connectivity and bandwidth.
Either way, you should brief them on
- what to look for
- how they might see it
- how to behave with participants
- what to do (and not do) with their observations
It probably isn’t enough just to hand out a list of rules if this is the first usability test these people will watch. Consider holding a short training session to explain how the usability test will be conducted and what they can expect in watching sessions. Be plain (but diplomatic, of course) about your expectations of your observers.
Here’s one of my favorite sets of guidelines for observers who will be in the testing room with you and the participant, borrowed from UIE.
Guidelines for observers
Running a successful usability test requires observers to adhere to strict guidelines so participants feel comfortable and willing to share information. To make sure your presence as an observer does not cause discomfort to participants or otherwise affect the quality of the data we collect, please observe the following rules.
Arrive before the session is scheduled to start. It is important for you to be present when participants arrive so you can choose your seat and get settled and ready to begin taking notes. Participants may interpret your lateness as disregard for what they have to say, and a stranger entering the room is distracting and disruptive.
You must stay for the duration. We would like participants to forget that anyone else is in the room. Having people constantly coming in and going out is very distracting. Therefore, after we start the session, you must stay until the session is complete.
Turn off your phone or leave it at your desk. If you bring your computer, turn off email and instant messaging. It’s a short session. Please be fully present and ready to pay attention to the moderator and the participant.
No laughing, grunting, aha-ing, or distracting body language. Participants may think you are laughing at them. Please do your best to keep as quiet as possible. It is important that observers do not make facial expressions or utter comments during the session. There is space in the script for you to take notes. Please turn the pages of the script quietly.
Ask open-ended questions about what happened in the session when invited by the moderator to do so. Do not offer design or feature alternatives. Avoid asking about preferences or opinions.
Keep the participant’s identity confidential. We have promised participants that their identity will be kept confidential. Please help us maintain this privacy and confidentiality.
If observers will not be in the same space with you and the participant, but rather will be watching from somewhere else, ask observers to resist redesigning or finding solutions until the end of the study (or until the day’s debrief, whichever works best in your situation).
This is another good reason to give each observer a little job – it keeps them occupied at different things that demand the attention that might otherwise be given to premature redesign. For example, you and your team can identify lenses to observe through ahead of time, such as pain points and barriers, breakthroughs, or myths and assumptions.
Working with observers: Reaching consensus
No small part of working with observers is understanding what their separate stakes in the test are. When you know this, you can facilitate discussion among observers effectively. If you don’t know, you’re probably working harder than you need to to gain consensus.
Consensus is important. Reaching consensus among observers means that
- There’s a shared vision of where the design should go. This is good for the team and the design.
- Everyone uses the same descriptions for issues. Having a common language for talking about the usability issues among departments or groups will help the diverse group visualize the user experience together.
- Nuances and exceptions have been discussed and agreed on. Anything that isn’t optimally feasible to fix can be negotiated among design and development team members in a daily debrief or final design direction meeting.
- There should be no surprises when you publish the report. Because team members observed sessions and discussed the issues together, insights and outcomes in the report should reflect the agreed direction from those observations and group discussions.
Feeling lonely? Gather up some observers for your next user research or usability test from your design and development teams. Make them feel welcome, give them duties, and work with them to understand the issues. They’ll thank you for it.