Listen up, people. This is why — quantity is not quality — you are not your user.
The lesson for today on participant sampling is Google Buzz. Google has been working on Buzz for some time. And it’s a cool idea. Integrating the sharing of photos, status updates, conversations, and email is a thing a lot of us have been looking for. Buzz makes lots of automatic connections. That’s what integrating applications means.
BUT. One of the features of Buzz was that it would automatically connect you to people whom you have emailed in Gmail. On the surface, a great idea. A slick idea, which worked really well with 20,000 Google employees.
Large samples do not always generate quality data
Twenty thousand. Feedback from 20,000 people is a lot of data. How many of us would kill to have access to 20,000 people? So. How can such a large sample be bad? Large samples can definitely generate excellent data on which to make superfine design decisions. Amazon and Netflix use very large samples for very specialized tests. There’s discussion everywhere, including at the recent Interaction10 conference in Savannah, about cheap methods for doing remote, unmoderated usability testing with thousands of people. More data seems like a good idea.
If you have access to 20,000 people and you can handle the amount of data that could come out of well designed research from that sample, go for it. But it has to be the right sample.
Look outside yourself (and your company)
Google employees are special. They’re very carefully selected by the company. They have skills, abilities, and lives that are very different from most people outside Google. So, there’s the bias of being selected to be a Googler. And then there’s indoctrination as you assimilate into the corporate culture. It’s a rarified environment.
But Google isn’t special in this way. Every organization selects its employees carefully. Every organization has a culture that new people undergo indoctrination and assimilation for, or they leave. In aggregate, the people in an organization begin to behave similarly and think similarly. They aspire to the same things, like wanting products to work.
But what about 37 Signals and/or Apple? They don’t do testing at all. (We don’t actually know this for sure. They may not call it testing.) They design for themselves and their products are very successful in the marketplace. I think that those companies do know a lot about their customers. They’ve observed. They’ve studied. And, over time, they do adjust their designs (look at the difference in interaction design in the iPod from first release in 2001 to now). Apple has also had its failures (Newton, anyone?).
The control thing
By not using an outside sample, Google ran into a major interaction design problem. About as big as it gets. This is a control issue, not a privacy issue, though the complaints were about over sharing. One of the cardinal rules of interaction design is to always let the user feel she’s in control. By taking control of users’ data, Buzz invaded users’ privacy. That’s the unfortunate outcome in this case, and now, users will trust Google less. It’s difficult to regain trust. But I digress.
The moral of today’s lesson: Real users always surprise us
Google miscalculated when it assumed that everyone you email is someone you want to share things with, and that you might want those people connected to one another. In a work setting, this might be true. In a closed community like a corporation, this might be true. But the outside world is much messier.
For example, I have an ex. He emails me. Sometimes, I even email him back. But I don’t want to share things with him anymore. We’re not really friends. I don’t want to connect him to my new family.
Even testing with friends and family might have exposed the problem. Google has a Trusted Tester program. Though there are probably some biases in that sample because of the association with Google employees, they are not Google employees. This makes friends and family who use Gmail one step closer to typical users. But Google didn’t use Trusted Testers for Buzz.
You get to choose your friends in real life. Google could have seen this usage pattern pretty quickly just by testing with a small sample who live beyond the Google garden walls.