Card Sorting

Procedure

Planning

1. Decide which type of card sorting will help you meet your goals. Types of card sorting include: open (where participants give you the names for the groups they create), closed (where they sort into groups you provide), and Q-sorts (where they sort related items into an order or ranking).
2. Choose a tool for the card sorting exercise. You can use paper cards, individual card sorting software, or online software that allows remote card sorting.
3. Develop a test plan that contains your sorting procedures,the analysis plan (whether formal or informal), a schedule, and your deliverables.
4. Recruit your participants. Tullis and Wood (2004) and Weller (1998) suggest that 20-30 participants provide a cost-effective sample for card sorts assuming that you recruit good representatives of your target user groups.
5. Generate the items for the card sort. The items can come from many sources including user research, brainstorming, competitive evaluations, and task analysis.
6. Write each statement on a separate card.
7. Number each card with a unique code.
8. Consider providing a definition because some participants may not know what a particular item means because of their background, culture, or country.
9. If you are conducting a closed card sort with pre-defined categories, prepare 'place mats' on which your informants may place the cards. Otherwise ensure there is a large empty table for the informant to place their piles of cards on.
10. Conduct a pilot test and refine your procedures.

Running

1. Brief the participants on the procedures for the specific type of card sort.
2. Shuffle the cards and place them randomly on a physical surface. If you are doing virtual card sorting, most software will randomize the order of the cards to prevent an order bias.
3. Give the informants a common understanding of the expected grain of analysis and the expected maximum number of piles. For example, you might tell the participants that they must have "at least 2 piles of cards and you might have as many 8-12 piles. They should be told that piles with many cards should be split up. Participants can be told that piles with a single item are permissible. * Informants should be aware, that there may well be an 'unsortable' pile, but that they should attempt to place as many cards together into piles as they reasonably can.
4. Then informants sort cards into piles on the table in front of them, using place mats if they are doing a closed sort.
5. Ask the participants to give a name to each pile of cards ("how would you describe the cards in this pile?"). You might also ask the participant to describe the rationale for putting cards in particular piles.
6. At the end, note which cards have been put together by the respondent by noting the numbers on the back of the cards.

Variations

There are three common types of card sorting for examining how users organize information: Open, closed, and inverse:

Open Card Sort. The purpose of this card sort is to explore organizational models for new information or information which is difficult to classify. In an open card sort, participants are provided with a set of items to sort and asked to generate groups, arrange groups into a hierarchical structure, and provide the labels for the groups. Some open card sorts also allow the participants to suggest additional content or to eliminate content from the sorted results. This type of card sort is generally used in the earliest phases of a design project.

Closed Card Sort. The purpose of this card sort is to explore how new content may fit into an existing structure, or how new content relates to existing content. In a closed card sort, participants are provided a set of items to sort and pre defined categories and are asked to place the items into the predefined categories. The primary focus of this type of card sort is to evaluate the contents of the groups and their labels. This type of card sort is generally used in the later phases of a design project after an open card sort or to restructure existing architectures.

Inverse Card Sort. Open and closed card sorting are a “bottom up” processes since participants determine a logical structure or evaluate a structure after viewing all of the items that will make up the structure. However, the activity of discovering the location of items within a structure is a top down process where the entire content set of the structure is often unknown to the user. For this reason, an Inverse (also known as a Reverse Card Sort) activity may be used to evaluate a proposed structure. In an Inverse Card Sort activity, participants are asked to try to locate a specific item within a completed structure. This is, essentially, a usability evaluation of a completed card sort but focuses exclusively on the structure and labeling. The primary goal of this type of card sort activity is to evaluate a structure in a format more closely aligned with its actual use.

Number of Participants

Most card sort activities are of a much smaller scale and often include some discussion between the participant and a facilitator, between multiple participants, or both. In fact, some practitioners recommend only doing card sorting as a group activity even if individual card sorting is included. Kaufman [9] recommends “at least ten participants” for a card sort exercise but cites no data for this recommendation and even states that “you can achieve reasonable results with fewer”. Tullis and Wood [17] concluded that ” reasonable structures are obtained from 20-30 participants” based on analysis of data from a project involving 168 participants in a card sort study. Neilson [12] recommends performing cards sorts with 15 based on an analysis of the same data used by Tullis and Wood. Paul [13] suggests that a reasonable structure can be generated using as few as 5 participants if the card sort exercise is part of a Delphi-based research approach. However, none of the works cited discuss the effect of multiple potential user groups providing input, how to address potentially different but equally viable architectures, or how the natural similarities or differences in the data affect the number of participants needed. As a result, no clear evidence exists on an appropriate number of participants to use. Since the most common use of this technique is to generate data to help designers understand the user’s perspective on organization but not to actually generate a proposed architecture, expert assessment is needed to determine when sufficient data has been collected.

Informants should be representative of the user population for whom the application is being designed.

Materials Needed

For a physical card sort:

• cards containing the items to be sorted, an identifier, and perhaps a definition
• for a closed sort, a "page" or "placemat" for each group
• for an open sort, extra cards for the participants to label their groups
• rubber bands
• if you cannot transcribe the results immediately, you will need a set of cards for each participant you plan to include in your day's run.

For an online card sort:

• your list of items to be sorted
• for a closed sort, your list of group identifiers

A real advantage of online card sorting software is the ability for the software to tally and post results for each participant. A disadvantage of online card sorting software is that many of the tools don't allow participants to create subcategories.

Who Can Facilitate

For on-site card sorting, an experienced facilitator with knowledge of card sorting procedures is recommended. The facilitator is responsible for briefing the participant, explaining the procedures, and answering questions without affecting the participants' responses.

Common Problems

• Granularity The items in a card sort should generally be of the same granularity. If you mix items that range from "tiny features" to "home pages on the web", your results may be hard to interpret.

• Divergent groups of participants If you have several user groups in your sorting study, you may end up with confusing results because the groups structure things differently. For example, you might have both employees and their managers in your sample, but you would be better off to have them sort the data separately and look at the results of each group.

Data Analysis Approach

• The use of automated tools and the web provides the potential to do a card sort exercise with a large number (or even all) users serving as participants; however, most card sort exercises are conducted with a small set of participants. Large-scale card sorts can provide large data sets for analysis and lends itself to automated analysis (e.g., cluster analysis).
• Several software packages are available to perform on-line card sort activities and usually provide data analysis tools. Though a large amount of raw data is generated, the scale of these activities make them limited in their ability to provide clarification to the participants or to gain additional insight into the rationale, assumptions, or decision making process used when performing the card sort. These limitations can significantly affect the validity of the data. For example, multiple viable structures may be represented in the data sets (a common occurrence when multiple user profiles or personas are included as participants).
• If clusters are relatively clear and straightforward, it is enough to write names for the clusters on cards and place those name cards near the piles.
• Otherwise some additional analysis is required driving to final clustering of cards. Usually, this can be achieved with a statistical method called cluster analysis, which may help to discover connections between cards and groups.
• After the final groupings are determined, the groups need to be named. Often, the participants won't use the same terminology as each other, so some analysis needs to be done to determine what the best name should be.

Next Steps

• After a card sort is complete, the groupings are used to define the structure of the pages within the site.