“If it disagrees with experiment it is wrong. In that simple statement is the key to science. It does not make any difference how beautiful your guess is. It does not make any difference how smart you are, who made the guess, or what his name is — if it disagrees with experiment it is wrong. That is all there is to it.”
– Richard Feynman, theoretical physicist and winner of the Nobel Prize in Physics
A real usability “expert” will be the first to tell you that they don’t have all the answers (beware of anyone who says they do). We know best practices, and previous experience has taught us a lot about user behavior, but there is simply no substitute for user testing. Research will only tell you so much. User interviews, while helpful, are also not enough. The only way to create a user experience that truly serves your target audience and that best drives the results you want is to test the people who will actually be using your product, website, or application. Here are just a few reasons why user testing is so important.
1. User behavior is hard to predict
People are irrational and unpredictable. They will always surprise you. They’ll use products in completely unexpected ways, they’ll struggle with what you perceive as the simplest tasks, and they will breeze through what you expected them to find difficult. The only way to know what your users will do is to observe them in real-world situations.
2. Your users are NOT like you
One of the biggest usability mistakes people make is saying, “This makes the most sense to me, so that’s probably what will make the most sense to our users.” Unfortunately, 99% of the time, you are not your target user. Don’t assume your users think or behave like you. By nature of being an advertiser or designer or developer, you’re already atypical.
3. People can’t TELL you what they like or have liked
Interviewing people can’t replace user testing. That’s because people can’t tell you what they’ll like. They can tell you what they think they’ll like, but they’re probably wrong. People’s ability to actually understand the factors that influence their behavior is surprisingly poor. For example, studies have shown that people rank products differently when asked to explain why they do it, than when simply asked to rank based on preference. Take this 1991 psychology study for example: “Thinking Too Much: Introspection Can Reduce the Quality of Preferences and Decisions”. In part 1 of the study, students were asked to taste different brands of strawberry jam and rate them. Those students who were also asked to analyze their ratings agreed less with experts than students who didn’t have to explain themselves. In part 2 of the study, the same was found for students who were asked to rate their college course preferences. The point? People’s rankings were a lot more “correct” when they didn’t have to explain them.
4. Test results help convince stakeholders
You can’t argue with your users. That would simply be counterproductive. If 90% of your target users didn’t notice a button on your website, you can’t just call them “dumb” and move on. You have to listen to your users if you’re going to build a successful business or product — and your superiors and clients know that. It can also help to test competitor sites, because it will help stakeholders be less defensive about their own website if they see others making similar mistakes.
5. Usability testing can reveal much more than how usable a site is
There are many different types of user testing, and they are all extremely useful in their own way. You can gauge your audience’s emotional response to your design, figure out the optimal structure for your website, pinpoint the best-performing email headline, discover major usability flaws, and much more.
The Different Types of User Testing
Now I’ll talk about some of the different types of testing that exist out there, why each one is useful, and how each one can be done (including some technology suggestions).
What it answers: How do users feel about _______?
What it is: Opinion polls can take many different forms — from questionnaires about the technologies your target audience uses, to questions about which of two possible designs they prefer. It can be very helpful when you’re conducting initial audience research or when you’re trying to decide between different creative directions.
How to do it: There are myriad online polling platforms you can use. You’re probably familiar with SurveyMonkey, but if you’re looking for a platform that will also source your testers for you, there’s SurveyMonkey Audience, Google Consumer Surveys, AYTM.com, and many others. Just be careful how you ask your questions if you’re doing an opinion poll about a product design, for example. Don’t just ask which product someone is more likely to buy (they won’t actually know); ask testers to rate each design on how “fun” or “professional” it looks, and then choose the design that most closely aligns with your brand and the purpose of your product.
What it answers: What’s the most logical way to group the topics/pages on this site?
What it is: Card sorting is an exercise that asks users to take a set of cards with words written on them and to group them into “buckets” — and then to label each “bucket” with a title that makes sense to them. It’s very helpful when you’re organizing pages on your website (creating a site map), and you’re not sure how to group them or which category to put them under in your menu.
How to do it: Card sorting can be done in person (using regular index cards), or online (using a platform such as OptimalSort by Optimal Workshop). Both types have their value. In-person card sorting allows you to ask questions and have a conversation with your users about why they’re grouping items in a certain way. With very niche or technical topics — which is very common in B2B — it also allows you to ask your testers what they think some of the terms on your cards mean (you’d be surprised how often our testers don’t really understand terms that we simply assumed they’re familiar with). However, if you have a lot of cards and are looking for statistically significant quantitative results, you’ll need to go digital and run non-moderated online card sorting. And there’s no reason you can’t do both, of course.
Site Map Testing
What it answers: Can users find what they’re looking for with the site map we created?
What it is: Site map testing (also called taxonomy testing or information architecture testing) asks users to find specific pages/topics within your current infrastructure. It’s a great tool for validating the site map you created (with input from card sorting) is actually “correct.” It can also replace card sorting if you already have a good idea about how to structure your website for the most part, but there’s just one or two topics you’re unsure about and want to test.
How to do it: For site map testing, you’ll need a tool like Treejack by Optimal Workshop. You build out your site map within Treejack and create a set of tasks that ask users to find specific topics within that site map, and then Treejack provides you with a comprehensive report of success rates and user paths for every task.
A/B and Multivariate Testing (MVT)
What it answers: Which of these interface elements produces optimal results?
What it is: A/B and MVT testing pits different design or content elements against each other. For example: email subject lines, landing page headlines, button colors, etc. Of course, you have to have a user action that you’re using to measure success, such as email open rates or form submissions.
How to do it: You should be able to easily test emails with whichever email marketing platform you’re using — be it MailChimp, Oracle Eloqua, Marketo, or something else. With landing pages, these can either be developed internally, or there are a couple of good online platforms that allow you to do this with a visual editor (such as Unbounce, Instapage, and Visual Website Optimizer).
What it answers: Pretty much anything about your website or app.
What it is: This is what most people think of when they think of user testing. You essentially lead users through a series of questions or tasks on your website or application, and you record their thoughts and actions. This kind of testing can be extremely helpful, because it can help you pinpoint usability issues on your website, discover where people are looking for certain topics, hear how people feel about your design, and even ask for improvement suggestions.
How to do it: For this kind of user testing, you’ll definitely need a UX expert to help you set it up correctly and analyze the results. If you’re sourcing your own users, you can use screen-sharing and recording software such as GoToMeeting. Otherwise, you will probably want to work with someone like UserTesting.com, which sources target users for you and which we use a lot.