Benefits, Advantages and Disadvantages

Benefits

Benchmarking a product or website against competitors is useful for identifying features that enhance usability and, when used to compare one version of a product/website with a later one, can provide proof of usability improvement. If the benchmark study shows that your product/site is deficient, management can be motivated to fund design improvements and additional usability research. If the benchmark study shows that your product is superior, your organization's product marketing can use that information in marketing campaigns.

Advantages

Benchmark studies are similar to summative usability studies in that they test usability of a functioning product, application, or website. The goal is to collect data along specific measures such as error rate, number of clicks, success/failure, and satisfaction ratings. If you conduct summative studies, you already know how to design a benchmark study.

Disadvantages

The challenge with benchmark studies is in their size. The number of sessions you need to run must be sufficient for performing a meaningful statistical analysis. This number can vary from 24 to 60 sessions (or more) per product/site, depending on the difference in performance you expect (and ultimately find), the level of confidence desired in the results, and the margin of error that is acceptable. See [the statistics section] for more information on power analysis, confidence intervals, and margin of error.

In addition, the preparation required to design a benchmark study is greater than that for a "one-off" summative test. The study must be designed for repeatability without change. The user profiles, tasks performed, and measures used (and the definition of those measures) must be consistent for each round of research.

Cost-Effectiveness

Benchmark studies are more expensive to conduct than formative or summative usability studies because of the additional preparation and the greater number of sessions required. The return is worth the investment when it justifies usability improvements or provides favorably comparative statements that product marketing can use.

To minimize the expense of a benchmark study and increase its cost effectiveness, one can consider conducting it as an automated study [see the automated study section]. Assuming one is not observing recordings after the fact, this approach supports the greater number of sessions required by the method at a lower cost per participant (incentive). The downside is the lack of first-hand observation of the participant's behavior, which is critical for descriptive error coding.

Otherwise, the best way to minimize the expensive of a benchmark study, as with any study, is to identify the focus of the study clearly and conduct only the tasks for that focus, minimizing session time and collecting only the data that answers the key issues of the study.

Appropriate Uses

Benchmark testing that includes error rates, click counts, and success/failure data is most appropriately used with a product/website that supports full user interactivity and natural user behavior. A partially functional prototype is not appropriate, as it introduces variables whose effect cannot be measured. For example, a user who clicks on user interface elements and receives no response because the product/site is a mockup may start hesitating to clicking in general.

For a benchmark study of user attitudes and opinions, a less than fully functional prototype can be used, with a moderator who drives the user through the experience and asks questions.

The moderator chosen to run a benchmark study must be well-trained in facilitation techniques to be 100% consistent from one session to the next, through all sessions conducted.