Contact  |  Login  Volunteer

Card Sorting

The card sorting method is used to generate information about the associations and grouping of specific data items. Participants in a card sort are asked to organize individual, unsorted items into groups and may, depending on the technique, also provide labels for these groups.

In a user-centered design process, it is commonly used when developing a site architecture but has also been applied to developing workflows, menus, toolbars, and other elements of system design.

  • Card sorting may be conducted as a low tech method using index cards or post-it notes, or may be automated using one of several software packages
  • Card sorting may be conducted as a series of individual exercises, as a concurrent activity of a small group, or as a hybrid approach where individual activity is followed by group discussion of individual differences
  • Card sorting is usually conducted as a specific activity in the early design phase of a project for defining an architecture, but can similarly be used during a product evaluation to determine if usability issues are due to problems with grouping or group labels

Sorting and grouping have long been studied within psychology and the research dates back at least to the 1950s. Numerous, non-peer reviewed descriptions, case studies, and blogs have been written in the last several years on the technique and its use in the user-centered design process, but only a few peer reviewed articles on the technique have been published and little is known of its validity or reliability as a means of directly producing a useful and usable architecture. Instead, card sorts are generally used to provide insight that is used by a practitioner to generate an architecture.

 

Related Links

The Card sorting technique originates in cognitive psychology techniques. One of most significant of the methods is known as the Wisconsin Card Sorting Task (WCST), which was introduced in 1946. The WCST is quick and easy to administer, requiring no specialized equipment but the deck of cards itself and a book to compare scores. It's used to determine competence with abstract reasoning, and ability to change problem solving strategies when needed.

Detailed description

Benefits, Advantages and Disadvantages

Advantages

  • Simple – Card sorts are easy for the organizer and the participants.
  • Cheap – Typically the cost is a stack of index cards, sticky notes, a pen or printing labels, and some time.
  • Quick to execute – it is possible to perform several sorts in a short period of time, which provides significant amount of data.
  • Established – The technique has been used for over 10 years, by many designers.
  • Involves users – Because the information structure suggested by a card sort is based on real user input
  • Provides a good foundation for the structure of a site or product.

Disadvantages

  • Does not consider users’ tasks – Card sorting is an inherently content-centric technique. If used without considering users’ tasks, it may lead to an information structure that is not usable when users are attempting real tasks.
  • Results may vary –The card sort may provide fairly consistent results between participants, or may vary widely.
  • Analysis can be time consuming – The sorting is quick, but the analysis of the data can be difficult and time consuming, particularly if there is little consistency between participants.
  • May capture “surface” characteristics only – Participants may not consider what the content is about or how they would use it to complete a task and may just sort it by surface characteristics

Appropriate Uses

Card sorting can be used to:

  • Identify themes or patterns from qualitative data
  • Develop the information and navigational architecture for a Web site or application
  • Design or redesign a site or application
  • Organize icons, images, menu items, and other objects into related groups
  • Determine how a specific individual classifies items from a particular domain
  • Examine how different groups (users versus developers, for example) view the same subject matter
  • Rank or rate items on specific dimensions.

How To

Procedure

Planning

  1. Decide which type of card sorting will help you meet your goals. Types of card sorting include: open (where participants give you the names for the groups they create), closed (where they sort into groups you provide), and Q-sorts (where they sort related items into an order or ranking).
  2. Choose a tool for the card sorting exercise. You can use paper cards, individual card sorting software, or online software that allows remote card sorting.
  3. Develop a test plan that contains your sorting procedures,the analysis plan (whether formal or informal), a schedule, and your deliverables.
  4. Recruit your participants. Tullis and Wood (2004) and Weller (1998) suggest that 20-30 participants provide a cost-effective sample for card sorts assuming that you recruit good representatives of your target user groups.
  5. Generate the items for the card sort. The items can come from many sources including user research, brainstorming, competitive evaluations, and task analysis.
  6. Write each statement on a separate card.
  7. Number each card with a unique code.
  8. Consider providing a definition because some participants may not know what a particular item means because of their background, culture, or country.
  9. If you are conducting a closed card sort with pre-defined categories, prepare 'place mats' on which your informants may place the cards. Otherwise ensure there is a large empty table for the informant to place their piles of cards on.
  10. Conduct a pilot test and refine your procedures.

Running

  1. Brief the participants on the procedures for the specific type of card sort.
  2. Shuffle the cards and place them randomly on a physical surface. If you are doing virtual card sorting, most software will randomize the order of the cards to prevent an order bias.
  3. Give the informants a common understanding of the expected grain of analysis and the expected maximum number of piles. For example, you might tell the participants that they must have "at least 2 piles of cards and you might have as many 8-12 piles. They should be told that piles with many cards should be split up. Participants can be told that piles with a single item are permissible. * Informants should be aware, that there may well be an 'unsortable' pile, but that they should attempt to place as many cards together into piles as they reasonably can.
  4. Then informants sort cards into piles on the table in front of them, using place mats if they are doing a closed sort.
  5. Ask the participants to give a name to each pile of cards ("how would you describe the cards in this pile?"). You might also ask the participant to describe the rationale for putting cards in particular piles.
  6. At the end, note which cards have been put together by the respondent by noting the numbers on the back of the cards.

Variations

There are three common types of card sorting for examining how users organize information: Open, closed, and inverse:

Open Card Sort. The purpose of this card sort is to explore organizational models for new information or information which is difficult to classify. In an open card sort, participants are provided with a set of items to sort and asked to generate groups, arrange groups into a hierarchical structure, and provide the labels for the groups. Some open card sorts also allow the participants to suggest additional content or to eliminate content from the sorted results. This type of card sort is generally used in the earliest phases of a design project.

Closed Card Sort. The purpose of this card sort is to explore how new content may fit into an existing structure, or how new content relates to existing content. In a closed card sort, participants are provided a set of items to sort and pre defined categories and are asked to place the items into the predefined categories. The primary focus of this type of card sort is to evaluate the contents of the groups and their labels. This type of card sort is generally used in the later phases of a design project after an open card sort or to restructure existing architectures.

Inverse Card Sort. Open and closed card sorting are a “bottom up” processes since participants determine a logical structure or evaluate a structure after viewing all of the items that will make up the structure. However, the activity of discovering the location of items within a structure is a top down process where the entire content set of the structure is often unknown to the user. For this reason, an Inverse (also known as a Reverse Card Sort) activity may be used to evaluate a proposed structure. In an Inverse Card Sort activity, participants are asked to try to locate a specific item within a completed structure. This is, essentially, a usability evaluation of a completed card sort but focuses exclusively on the structure and labeling. The primary goal of this type of card sort activity is to evaluate a structure in a format more closely aligned with its actual use.

Number of Participants

Most card sort activities are of a much smaller scale and often include some discussion between the participant and a facilitator, between multiple participants, or both. In fact, some practitioners recommend only doing card sorting as a group activity even if individual card sorting is included. Kaufman [9] recommends “at least ten participants” for a card sort exercise but cites no data for this recommendation and even states that “you can achieve reasonable results with fewer”. Tullis and Wood [17] concluded that ” reasonable structures are obtained from 20-30 participants” based on analysis of data from a project involving 168 participants in a card sort study. Neilson [12] recommends performing cards sorts with 15 based on an analysis of the same data used by Tullis and Wood. Paul [13] suggests that a reasonable structure can be generated using as few as 5 participants if the card sort exercise is part of a Delphi-based research approach. However, none of the works cited discuss the effect of multiple potential user groups providing input, how to address potentially different but equally viable architectures, or how the natural similarities or differences in the data affect the number of participants needed. As a result, no clear evidence exists on an appropriate number of participants to use. Since the most common use of this technique is to generate data to help designers understand the user’s perspective on organization but not to actually generate a proposed architecture, expert assessment is needed to determine when sufficient data has been collected.

Informants should be representative of the user population for whom the application is being designed.

Materials Needed

For a physical card sort:

  • cards containing the items to be sorted, an identifier, and perhaps a definition
  • for a closed sort, a "page" or "placemat" for each group
  • for an open sort, extra cards for the participants to label their groups
  • rubber bands
  • if you cannot transcribe the results immediately, you will need a set of cards for each participant you plan to include in your day's run.

For an online card sort:

  • your list of items to be sorted
  • for a closed sort, your list of group identifiers

A real advantage of online card sorting software is the ability for the software to tally and post results for each participant. A disadvantage of online card sorting software is that many of the tools don't allow participants to create subcategories.

Who Can Facilitate

For on-site card sorting, an experienced facilitator with knowledge of card sorting procedures is recommended. The facilitator is responsible for briefing the participant, explaining the procedures, and answering questions without affecting the participants' responses.

Common Problems

  • Granularity The items in a card sort should generally be of the same granularity. If you mix items that range from "tiny features" to "home pages on the web", your results may be hard to interpret.
  • Divergent groups of participants If you have several user groups in your sorting study, you may end up with confusing results because the groups structure things differently. For example, you might have both employees and their managers in your sample, but you would be better off to have them sort the data separately and look at the results of each group.

Data Analysis Approach

  • The use of automated tools and the web provides the potential to do a card sort exercise with a large number (or even all) users serving as participants; however, most card sort exercises are conducted with a small set of participants. Large-scale card sorts can provide large data sets for analysis and lends itself to automated analysis (e.g., cluster analysis).
  • Several software packages are available to perform on-line card sort activities and usually provide data analysis tools. Though a large amount of raw data is generated, the scale of these activities make them limited in their ability to provide clarification to the participants or to gain additional insight into the rationale, assumptions, or decision making process used when performing the card sort. These limitations can significantly affect the validity of the data. For example, multiple viable structures may be represented in the data sets (a common occurrence when multiple user profiles or personas are included as participants).
  • If clusters are relatively clear and straightforward, it is enough to write names for the clusters on cards and place those name cards near the piles.
  • Otherwise some additional analysis is required driving to final clustering of cards. Usually, this can be achieved with a statistical method called cluster analysis, which may help to discover connections between cards and groups.
  • After the final groupings are determined, the groups need to be named. Often, the participants won't use the same terminology as each other, so some analysis needs to be done to determine what the best name should be.

Next Steps

  • After a card sort is complete, the groupings are used to define the structure of the pages within the site.

Special Considerations

Costs and Scalability

Most card sorts are relatively inexpensive and online card sorting software allow large samples of data to be collected efficiently. The major issue with scalability involves the number of cards that participants must sort in what is usually one to two hours (unless the participants are very dedicated). Recommendations on how many cards people can sort range from about 20 to 200. If there are more than 200 cards, then some type of sampling might be effective (Nakhimovsky, Schusteritsch, & Rodden, 2006). Some of the online card sorting tools become unwieldy around 100 items so that might affect scalability.

Participants who are familiar with the items on the cards can probably sort a larger sample than people less familiar with the content.

Facts

Sources and contributors: 
Bill Killam, Alice Preston, Shannon McHarg, Chauncey Wilson. Based on Jacek Wachowicz contributions in COST Action 294: MAUSE.
Released: 2009-06
© 2010 Usability Professionals Association