This description highlights issues that are important for summative usability testing. A more detailed description of How to Do It can be found in Usability testing.

Procedure

Prerequisites/Constraints

• Equipment: The requirements for a usability lab to run a summative test vary greatly. A mobile lab, meaning a laptop computer with recording software and a webcam can be sufficient, however big laboratories which include observation rooms for usability experts and developers can have advantages. The most important requirement however is to have a controlled environment in which the experiment takes place. For recording purposes there exist different software products, such as Techsmith Morae or Noldus Observer which record audio, video and screen for detailed post-analysis.
• Participants: It is important to select participants from the expected target group. In many cases, university researches will rely on students as participants because of cost issues. However in many cases this is not sufficient, especially if the expected end-user is a specialist in his work area. The number of participant depends on the experimental design (see below).
• Knowledge requirements: Knowledge requirements are manifold. First of all, the evaluator should have basic knowledge of different experimental designs and statistical analysis (see for example: David Martin's "Doing Psychology Experiments" for an introduction). The reliability of a study relies to a large extent on the quality of the planning, the execution, and the analysis process. Furthermore the evaluator should not be involved in the development process of a product in order to avoid any bias. Experience is also a key issue in order to get valid and reliable results.

Planning

• Define variables, hypothesis and measurements
  • Independent variable: could be different products or interfaces that will be compared
  • Dependent variable: everything that can be measured
  • Hypothesis: pre-defined assumptions about the relation between independent and dependent variables, needed for statistical analysis.
• Define the experimental design. If several conditions will be compared (e.g. different products), a more complex design has to be chosen (e.g. within-subjects or between-subjects design, mixed-designs). The latter also has an influence on the number of participants - twelve participants per condition are often considered as the minimum to allow for doing statistical analysis afterwards.
  • In most of the cases a within subjects design will be appropriate. In this case, subjects work with all different occurrences of the independent variable, e.g. meaning all different types of interfaces/products. To avoid learning effects, interfaces and tasks have to be counterbalanced, meaning all kinds of possible conditions have to be applied in the experiment. If asymmetric learning effects occur, a within subjects design is not appropriate and the evaluator has to switch to a between subjects design, which means different groups for each occurrence of the independent variable. The drawback is that more participants are needed and subjective questions where users should compare both systems are not possible.
• Define Tasks to be carried out by the user during the test. Tasks should be derived from the user-centred development process, e.g. out of scenarios or use-cases.
  • Make sure that when comparing two or more products, each task goal can be achieved with all products
• Define abort-criteria for each task (e.g. max time)
• Select or create pre- and post-test questionnaire. E.g. use a satisfaction questionnaire as post-test questionnaire to assess the subjective users' satisfaction. Pre-Test questionnaires are used to collect demographic data and pre-knowledge.

• Select users who are representative of each user group

Running

The standard procedure can be divided into three parts.

• First subjects are asked to fill in a pre-test questionnaire for demographic data, prior knowledge and other interesting information regarding the study. All variables collected can be useful for the statistical analysis.
• In the second phase, subjects work with the system(s) (independent variable) and try to solve realistic tasks. Common dependent variables measured in this process are task time and task accuracy. Interaction steps or other system specific measures are also possible.
• The third phase is used for subjective questionnaires, mostly standardized satisfaction questionnaires like QUIS, Sumi, SUS, Attrakdiff, etc, often supplemented with questions regarding the specific system.
• Duration is typically one hour.

Participants and Other Stakeholders

Participants should be representative of the user population for whom the application is being designed.

Materials Needed

• Running product/prototype
• Observer/Logging Software makes measuring task time, etc. easier
• Task descriptions for the user to carry out
• Interview guidelines and questionnaires for participants
• form of consent
• if required: non-disclosure agreement to be signed by participants

Common Problems

• Mistakes regarding the experimental design as well as the statistical analysis are quite common. It is useful to refer to literature from psychology to avoid mistakes.

Data Analysis Approach

• Statistical procedures such as analysis of variance (ANOVA) or Chi-Square testing are commonly used to test for statistical significance of the reported results (e.g. to compare whether the difference in task-completion time between two products is due to chance or due to the different product).
• The results shoud be reported with confidence intervals.

Next Steps

• Results can be used to decide whether a product is ready for deployment (usability goals met) or should be redesigned or improved in some aspects. They can also be used for marketing purposes (e.g. our product is better than product X).