Competitive Usability Testing

We’re redesigning our site, but there’s internal debate about whether the new design is any better than the current site. What’s the best way to determine which one is better?

Contact us

Competitive Usability Testing

We’re redesigning our site, but there’s internal debate about whether the new design is any better than the current site. What’s the best way to determine which one is better?

Fill out form to continue
All fields required.
Enter your info once to access all resources.
By submitting this form, you agree to Expero’s Privacy Policy.
Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.

Q: We’re redesigning our site, but there’s internal debate about whether the new design is any better than the current site. What’s the best way to determine which one is better?

Rather than continue to debate internally, you should get data from your end-users so you can make an informed decision. The guy down the hall has an opinion, and so do you, but your users’ responses matter a lot more.

Many techniques can help you figure out which design is better, but we’ll discuss two here. Both are types of “competitive benchmarking”, also known as “competitive usability testing” or “A/B testing”.

First, you could run a competitive test based on the think-aloud methodology. In this test, a user performs tasks with Version A of the design (or site, etc.) while thinking out loud. You make note of what the user liked and didn’t like, why they failed to complete certain tasks, where they got lost or confused, etc. Then the user does the same thing with Version B of the design while you observe. At the end, you ask the user which version they preferred and why. Do this with other representative users and look for patterns in the responses. The winning design usually emerges (“8 out of 10 users preferred Version A, and Version A was easier to use”), along with plenty of ideas on how to improve it.

Second, if the new design is already implemented, you could run a live test in which some percentage of visitors to your site get the new design while the rest see the usual design. You look at web analytics data and metrics you care about (for example, conversion) to decide whether the new design is “better”.

We recommend using both of these techniques together (the first during the design prototype phase and the second after implementation). But either technique is better than none.

Tips for competitive testing include: using the same tasks and the same metrics for both versions; asking users “why?” when possible; and alternating the presentation order across test participants if they see both versions (half will see Version A first, and the other half will see Version B first). It’s also good to have users compare designs at the same level of fidelity and interactivity—don’t test static wireframes against a live site and expect to get valid results. In this case, you could create static wireframe pages from your live site for the test.

Expero offers a training course on Competitive Usability Testing for people who want to learn the techniques in depth.

User Audience

Services & capabilities

Project Details

Technologies

Expero Staff

October 28, 2009

Competitive Usability Testing

We’re redesigning our site, but there’s internal debate about whether the new design is any better than the current site. What’s the best way to determine which one is better?

Tags:

Q: We’re redesigning our site, but there’s internal debate about whether the new design is any better than the current site. What’s the best way to determine which one is better?

Rather than continue to debate internally, you should get data from your end-users so you can make an informed decision. The guy down the hall has an opinion, and so do you, but your users’ responses matter a lot more.

Many techniques can help you figure out which design is better, but we’ll discuss two here. Both are types of “competitive benchmarking”, also known as “competitive usability testing” or “A/B testing”.

First, you could run a competitive test based on the think-aloud methodology. In this test, a user performs tasks with Version A of the design (or site, etc.) while thinking out loud. You make note of what the user liked and didn’t like, why they failed to complete certain tasks, where they got lost or confused, etc. Then the user does the same thing with Version B of the design while you observe. At the end, you ask the user which version they preferred and why. Do this with other representative users and look for patterns in the responses. The winning design usually emerges (“8 out of 10 users preferred Version A, and Version A was easier to use”), along with plenty of ideas on how to improve it.

Second, if the new design is already implemented, you could run a live test in which some percentage of visitors to your site get the new design while the rest see the usual design. You look at web analytics data and metrics you care about (for example, conversion) to decide whether the new design is “better”.

We recommend using both of these techniques together (the first during the design prototype phase and the second after implementation). But either technique is better than none.

Tips for competitive testing include: using the same tasks and the same metrics for both versions; asking users “why?” when possible; and alternating the presentation order across test participants if they see both versions (half will see Version A first, and the other half will see Version B first). It’s also good to have users compare designs at the same level of fidelity and interactivity—don’t test static wireframes against a live site and expect to get valid results. In this case, you could create static wireframe pages from your live site for the test.

Expero offers a training course on Competitive Usability Testing for people who want to learn the techniques in depth.

User Audience

Services

Project Details

Similar Resources

Hey TigerGraph! Your 2020 Forward Looking Thoughts Are Already Here!

At Expero, TigerGraph’s 2020 projections are already in place or well underway.

Watch Demo

Chatting with Data: Interrogative UIs for Exploratory Data Analytics

Natural language processing is a powerful tool in building investigatory user interfaces for data exploration.

Watch Demo

5 Things Developers Should Know About UX

Software and web developers often wear many hats, but some lack the knowledge to design or collaborate effectively with UX designers and researchers.

Watch Demo

When to Test What: Validating Standard Features & Game-Changers

As a user researcher, I’m always inclined to say, “Test everything, all the time!” when people ask, “What/when/how should we validate with users?”

Watch Demo