Usability Testing: The 7 Key Research Design Decisions

Usability testing is a critical asset for brands. But a successful usability testing study requires some solid upfront thought. Here are the seven key questions that really demand consideration (and decisions!) at the beginning of a solid website usability testing project.

1. When should we do usability testing?

In an ideal world, usability testing is deployed early and often. This is because testing delivers maximum benefit when it’s part of an iterative process. So, if you can find a way to test at every stage from prototype to live, that’s best – even if it means that some of the testing rounds are “guerrilla” testing.

But, if you are limited by time and resources and can only do a single round of usability testing, you’ll get the best value by testing an early prototype (rather than waiting for a more finished design). Early feedback will help prevent your design team from going too far in the wrong direction.

2. Which usability testing method should we choose?  Moderated or Unmoderated?

In moderated usability testing, a moderator is “with” the respondent – either in-person or online – observing their progress with the test tasks in real-time, asking questions, and probing for clarity and understanding as to what the respondent is noticing and being confused by.

In contrast, unmoderated usability testing uses an online software program to record the screen and voices of users who are attempting to complete a series of pre-determined tasks with no moderator present. Unmoderated testing allows for larger sample sizes and the calculation of statistically reliable measures of usability. But there’s no opportunity to provide guidance, probe for the reasons for behaviour, or have any “back and forth” discussion for test participants.

Moderated usability testing is the best-practice approach when your need is to identify usability issues and give designers the diagnostic information they will need to determine the best solution. Moderated testing is also pretty much the only way to go when you are dealing with early-stage prototypes that require some explanation, or very complex interfaces.

Unmoderated usability testing is the best practice approach when your need is to understand how your site’s usability compares to the competition’s, or when you want to establish quantitative benchmarks about your own site’s usability.

3. For Moderated Testing, should we do in-person or remote?

If your approach is to do moderated usability testing, your next decision is whether to have the testing be in-person (in a lab or research viewing facility) or remote (where the moderator interviews people using a screen sharing application like Skype).

Since the whole point of usability testing is to get a strong empathetic understanding of the user experience, we find that in-person testing tends to deliver better results. It’s just easier for a moderator to connect with someone that they are in the same room with, and in-person testing allows a moderator to read body-language, which sometimes communicates a lot more than words do! You don’t get that with remote testing, and – what’s worse – we find that communication with remote testing is often somewhat stilted as a result of audio lags.

Also, contrary to common expectations, in most cases remote testing does not tend to be significantly cheaper than in-person tests.

But! Remote testing has its advantages, including the fact that it allows users to test with their own equipment and in their normal environment – rather than the artificial environment of a lab. As well, remote testing makes testing users who are difficult to get to a lab or dispersed across broad geographies easy.

So – the decision as to whether to test in-person or remote is really a judgment call which is best made in the context of a particular project’s particularities.

4. How many users do we need to test?

One of the great things about usability testing is that it is totally focused on behaviour. And since behaviour doesn’t vary as broadly as opinions, we can do robust testing with a small number of users. Most people, when handed a tool, will use it in the same way – and run into the same problems.

With moderated usability testing, you can typically uncover about 80% of the usability issues by testing with just 5. users. The exception is when you have a website that will be used by several very distinct groups of users. If, for example, you have a site that will be used by teenagers and senior citizens, you may want to have 8 users total (four from each group).

Sidebar: don’t forget that roughly 1 in 7 people has some kind of disability. Usability testing is an invaluable tool for ensuring that your site delivers on legally-required measures of accessibility!

If what you’re doing is unmoderated usability testing, then the question of how many users to test becomes a statistical question that really relates to the following 3 questions:

• In our benchmarking, what are we comparing to? How small a difference do we need to be able to reliably detect between our site and the comparator?
• How much variation do we expect among users?
• What accuracy and reliability do we need in the statistical calculations that can be made? For example, if you want to be able to say that your findings are accurate +/- five percentage points 95% of the time, you’ll need to test with 384 users.

NOW! – if you detest statistics (as most people do) – do not fret. A good research partner will guide you through this issue!

5. What tasks should we test?

In setting up your usability testing, consider what tasks you’ll be asking people to complete. You want to ask people to do the tasks that MOST users will do MOST of the time, and which are important to the site owner. The point is to test the frequent and important things, and not spend too much time on the things people almost never do, and which are inconsequential to the business. Google Analytics can be a great resource for figuring out what people are doing on a site, so mining your data is a good start.

Once you’ve figured out which tasks to include, you’ll need to turn those tasks into “scenarios” for users. Asking users to “add a product to a cart” always feels unnatural for a user, because users don’t “do” tasks – they find themselves in a scenario – like needing new shoes – and end up doing a bunch of things (including adding a product to a cart!) as part of that. Giving a user a scenario (like “Imagine you need a new pair of basketball shoes. How might you use this site to get some”) is more naturalistic (and realistic!) and leads to more robust findings.

6. What metrics should we collect?

Whether you’re doing qualitative (moderated) or quantitative (unmoderated) testing, you’ll want to determine what – if any – metrics you will collect.

With moderated testing, the metrics might be as modest as “how many people can successfully do (x)?” and, perhaps, the satisfaction ratings provided by users. But with unmoderated testing, where you can test a lot more users, you can also think about metrics like the length of time it takes users to complete a task. It’s always a good idea to decide what to measure in advance – rather than on the fly!

7. What do we need to do to ensure that test findings are used?

It should go without saying that any testing where the findings don’t get used is a waste of time and money! So – in designing your research, you need to think about what you need to do to ensure that there’s IMPACT. From the outset, research should be designed to ensure that the final results will be persuasive! This means thinking about questions like:

– What biases do stakeholders have that we can plan for in the research?
– Do the designers need to be present during testing?
– Who will be seen as a credible practitioner for doing the testing?
– Who is will be seen as credible in conducting the analysis? Can the moderator do it and present the findings? Or do the findings need to be “co-crafted” with the designers in order to be accepted?

Need a bit of help with your website usability testing? Reach us here.