User-based evaluation of unimplemented technology where, generally unknown to the user, a human or team is simulating some or all the responses of the system.
The technique has often been used to explore design and usability with speech systems, natural language applications, command languages, imaging systems, and pervasive computing applications.
The originator, J.F. Kelley explains: "The term Wizard of Oz (originally Oz Paradigm) has come into common usage in the fields of Experimental Psychology, Human Factors, Ergonomics and Usability Engineering to describe a testing or iterative design methodology wherein an experimenter (the "Wizard"), in a laboratory setting, simulates the behavior of a theoretical intelligent computer application (often by going into another room and intercepting all communications between participant and system). Sometimes this is done with the participant's a-priori knowledge and sometimes it is a low-level deceit employed to manage the participant's expectations and encourage natural behaviors (though always, I would hope, with appropriate disclosure during the debriefing part of the experiments)."
J. F. Kelley coined the terms "Wizard of Oz" and "Oz Paradigm" when he was working on his dissertation in the early 1980s. These articles describe the origins of the method:
Benefits, Advantages and Disadvantages
The Wizard of Oz technique can provide valuable information on which to base future designs. It can be used to:
Wizard of Oz testing is a highly cost-effective way to compare multiple designs.
This technique can be used to test device concepts and techniques and suggested functionality before it is implemented. For example, this technique can be used to simulate a caller-system interaction. The user experience is similar to interacting with a functioning interactive voice response (IVR) system.
The Wizard of Oz technique can provide valuable information on which to base future designs. It can:
The wizard sits in a back room, observes the user's actions, and simulates the system's responses in real-time. For input device testing the wizard will typically watch live video feeds from cameras trained on the participant's hand(s), and simulate the effects of the observed manipulations. Often users are unaware (until after the experiment) that the system was not real.
The wizard has to be able to quickly and accurately discern the user's input, which is easiest for simple for voice input or hand movements. The output must also be sufficiently simple that the "wizard" can simulate or create it in real time.
The basic wizard of Oz procedure involves the following steps:
Ethical and Legal Considerations
The Wizard of Oz method can involve a low level of deception - the participants are lead to believe that they are using a working system rather than a simulation controlled by an expert, the wizard. According to ASA Code of Ethics: "When deception is an integral feature of the design and conduct of research, (researchers) attempt to correct any misconception that research participants may have no later than at the conclusion of the research."
This concept of ethics on experiment is not universal in the field of usability testing, but this should be take more seriously, especially the performed usability testing could induce the participant to have wrongful expectation about the technological status.