Card Sorting 101

In a closed card sort, you give people categories to sort the cards into. This time, instead of trying to find out how people conceptualize your information, you want to know where people think information belongs within your conceptual framework. You can really think outside the box with this technique — use it to find out a bunch of interesting things.

You know you need closed card sorting if you want to:

• find out if people agree on where your information is best placed within existing categories
• pinpoint unclear or misleading category labels based on mixed results and fix them
• reduce the number of categories you have based on which categories are ignored the most

Aside from how you structure your website content, you can also use closed card sorting to prioritize and rank features, products, search filters, and so on based on criteria such as ‘Important’ to ‘Unimportant’, or ‘Use often’ to ‘Use never.’

In a hybrid card sort, you give people categories to sort your cards into and enable them to create their own categories as well.

Though it is distinct enough from open and closed card sorting to warrant its own approach, a hybrid card sort will be ‘more open’ or ‘more closed’, depending on the number and type of categories you create.

When to use hybrid card sorting instead of open

When you set much fewer categories than people need to sort all the cards, your hybrid card sort will lean towards open. This means people will be less likely to use your categories and more likely to create new categories to complete the card sort.

You’ll run a hybrid card sort like this if you:

• want to generate ideas for grouping your information and want to give people a category pattern to take inspiration from
• see high agreement on some categories after an open card sort, but need clarity on some less certain groupings

When to use hybrid card sorting instead of closed

When you set enough categories for people to sort all the cards into, your hybrid category will lean towards closed. This means people will be more likely to sort the cards into your categories only, and less likely to create new categories.

You’ll run a hybrid card sort like this if you:

• are happy with the groupings and labels you have, but want people to have the option to suggest their own just in case
• want to find out if participants come up with category labels that are better than the ones you have
• want to avoid the risk of people ‘making do’ with categories they may not wholly agree with.

Hybrid card sorting results are the same as open because you’re allowing people to create and name their own categories, but your set categories will be standardized to replicate how they’re analyzed in a closed card sort.

You’ll approach hybrid card sorting results with questions like:

• How many people agree on where cards belong in your set categories?
• What new categories did people create, and are they better than yours?
• Did people create categories similar to the ones we gave them, or did they take a different approach

Card sorting is a conceptual activity, not a usability test or game of Snap. Your goal is to discover how people think about and make sense of your information, not whether or not they can find it quickly on a homepage. So when you’re deciding what cards to include, look underneath the language for the concept they represent.

Card sorting tests concepts, not usability — so the cards themselves don’t need to be written in the most ‘usable’ format, or exactly as they are on your website. Jakob Nielsen points out that “It’s OK to actually reduce the usability of the cards, because people don’t actually use them in the UI [user interface]”.

The cards need to be on the same conceptual level and similar enough for participants to actually be able to sort them into groups. By conceptual level, we mean that if you want people to sort grocery items, you won’t include the higher level category ‘Vegetables’ as a card at the same time as the lower-level ‘Carrots’:

In her book on card sorting, Donna is firm on the importance of making sure all cards are actually groupable. She cites the example of a 100-card card sort she tested with a colleague before taking it to a client. Her colleague was unable to create coherent groups because the cards were inconsistent and often unrelated, and therefore the card sort couldn’t “provide much insight into how the content could be grouped on the site”.

The solution? She recommends reviewing your card sort with this in mind to make sure “each item…could have a potential partner (or many partners)”. Ask someone to test it out for you (which you can do by sharing a study preview link with people in your team).

And if you find a card that is difficult to partner with any others, but that you think is valuable to your study, follow her lead: “On a recent sort, I deliberately included three cards I didn’t really need…so participants would have some cards that were easy to group…[to give] participants the confidence to proceed to more difficult groupings.”

When you ask people to complete a card sort, you’re asking them to look for and create patterns with your cards. The human mind is so fond of pattern-finding that we use it regularly as a shortcut when making decisions, especially on intellectually-taxing tasks.

For user research, this is one of the great strengths of card sorting, and one of its biggest pitfalls. If you include enough cards with the same opening phrases, casings, and sentence structures, it’s likely that most people will group them together — but instead of them approaching it conceptively, they’ve quickly and without realizing it played a simple game of Snap.

Jakob Nielsen illustrates the issue these example cards, that are written the way they would be on a website, but that offer too-obvious pairings:

Card Set A	Card Set B
Strawberry Planting	Planting Strawberry
Strawberry Growing	Growing Strawberry
Strawberry Harvesting	Harvesting Strawberry
Wheat Planting	Planting Wheat
Wheat Growing	Growing Wheat
Wheat Harvesting	Harvesting Wheat

To solve this, he recommends editing your labels using synonyms and non-parallel structures, an approach that doesn’t need to involve extensive rewrites: instead of ‘Harvesting strawberries’, we could say ‘Picking strawberries’; and instead of two cards that begin with planting, you could make their structures different by labelling one ‘Planting corn’ and another ‘Wheat planting’.

Images can be as effective as text for representing concepts and items, and in some cases more so. You can include images to illustrate or clarify the text on your cards, or you can include images on their own.

You might choose to add images to your cards if you:

• have products your customers will expect to see as images, like clothes, appliances, and furniture
• design portfolio or photography websites and want ideas for grouping images
• want to find out what people think of your sketches or designs.

In OptimalSort, you can upload JPEG or PNG files of any size, and each image will be resized to a maximum width of 200px. Resize all your images before you upload them if you want them to be the same size, and preview your card sort to make sure it looks how you want it to. Giving each image a descriptive label will make your analysis easier.

When you’re creating a closed or hybrid card sort, take care to craft categories that help you achieve your objectives.

For closed card sorts, you need to create enough categories that people can find a home for most of your cards, but not too many that only include categories that match your intentions for your website or your research questions. The more categories you create, the more options participants will have, and the more likely it’ll be that you find out which categories are preferred over others.

When you run closed and hybrid card sorts, the categories you set will lead people to think about your cards in a particular way, whether it’s on purpose or not.

For example, if you run an open card sort with 40 cards containing grocery items, you might find that some people group the items by type (vegetables, fruit, dairy) and some by meal (breakfast, lunch, dinner). If you run a hybrid card sort with even just one category, most people will take your lead: Set the category ‘Vegetables’ and most people will create the category ‘Fruit’.

The number and type of categories you set for a hybrid card sort will determine whether the card sort is more open or more closed.

• When you set fewer categories, participants are more likely to create their own, and thus it will tend towards open.
• When you set more categories, enough for people to find a home for every card, they’re less likely to create their own categories, and thus it will tend towards closed.

Running an in-person, moderated card sort with 5 to 15 participants will give you invaluable qualitative insights to go with the quantitative numbers you get by running the card sort online.

After you’ve created your card sort in OptimalSort, download and print your cards onto thick paper or card stock, and then use Donna’s guidance to set up and run your sessions. Then, you can upload the card sorts for each participant in the Participant’s tab of your OptimalSort results. Their results will be included with the unmoderated participants, and you’ll have the extra insight from observing and listening to your participants. If you lack the time or resources to invite participants to in-person sessions, your next best option is to use a service such as UserTesting.com — you can order 5+ videos of people completing your card sort online and speaking aloud as they do so (which means you won’t have to print the cards out at all).

Running an open card sort with OptimalSort is a generative exercise: the results give you lots of ideas of how you could label and organize your website content. As such, the quantitative numbers you need for techniques like tree testing and first-click testing may not be your objective.Instead, you’ll want enough completed card sorts to get ideas and see consensus forming, and not too many that you’re overwhelmed with the data.

Also, keep in mind that the more participants you have completing your card sort, the potential for more complexity in your analysis increases as well. This is simply because narrowing down the most effective structure from 40 different suggested categorizations will probably be an easier task than from 200 different suggestions.

If you want to gather as many suggested categorizations as you can, though, don’t be afraid to recruit more than 50.

You’ll explore your open and hybrid card sorting results to get ideas for labelling and grouping your information, so kick off your analysis with these questions in your head:

• What logic do participants follow in the categories they’ve created?
• What cards do people put together in the same group all the time?
• What cards are never put together, and are thus considered conceptually different by all participants?
• What kinds of labels do people suggest for representing your information?

The Categories results for open and hybrid card sorts

The Categories tab is a great doorway into your open and hybrid card sorting results. Spend just a few minutes scanning the categories people came up with and you’ll quickly form an impression their ‘mental models’ (how they perceived the overall theme or concept of your cards).

A useful way to explore the categories results and refine your data is to standardize categories that have similar labels (with different wording, spelling, capitalizations, and so on) and contain similar cards. In a hybrid card sort, your set categories are already standardized.

But before you eye up a bunch of similar-looking labels and standardize straight away, it’s important to look at the similarities between the categories in more depth. You can use both exploratory and statistical analysis approaches to do this.

Get familiar with any similar category labels and card groupings

When you first look at the Categories table, take some time to get familiar with the labels your participants have suggested — you’ll probably start seeing patterns pretty quickly.

Look more closely at the categories and cards to make sure people had the same conceptual approach

It’s tempting to throw categories with the same or similar labels together straight away, but before you do, cast your eye over both the labels and the cards in each category to ensure the participants are all thinking in the same way.

Of the 15 groups with the word ‘Animal’ in the label, 13 had a similar set of cards, but two participants had labelled their categories slightly differently (Animals and Environment’ and ‘Animals and Nature’) and had thus included extra cards the others didn’t have (‘Glacier’s melting faster than previously thought’, for example).

Standardize to explore how high the agreement actually is

When we place every category that uses the same words or phrases into a standardized category, we can assess the effectiveness of the category by looking at the agreement score.

The agreement score tells you the agreement level between the cards that participants placed in each category. Beside the agreement score, you can see the number of participants included in that score.

You’ll see a dash in place of an agreement score before standardizing, as each category has only been created by one person.

Once you standardize a category, check the agreement score to get an objective assessment of how similar the groupings are. An agreement score of .8 means that 80% of participants in the new category agree with the grouping. A perfect agreement score is 1.0 (100%).

Review your new categories in the standardization grid

As well as looking at the cards and assessing the agreement score, head over to the Standardization grid to see your standardized categories in more detail. The grid simply shows you how the number of times a particular card appears in a standardized category.

Similarity matrix

The similarity matrix shows you the percentage of participants who grouped two cards together, and clusters the most closely related pairings along the right edge. The darker the blue and the larger the cluster, the higher the agreement between participants on which cards go together.

You could use the Similarity Matrix to:

• draft a potential website structure based on the right-hand clusters
• quickly see which card pairings are the most common and thus probably belong together on your website
• quickly see which cards are very rarely paired together so you don’t need to waste time thinking they might.

Participant-Centric Analysis (PCA)

The PCA aims to find the most popular grouping strategy, and then to find two more popular alternatives among those who disagreed with the first strategy. It works best when you have a lot of results. Besides the first option hopefully being useful to you, you might get useful ideas for footer and sidebar facets from options two and three.

It’s called a ‘Participant-centric analysis’ because every participant response is treated as a potential solution and ranked for similarity with other entire responses. If you see a sort with a 11/43 agreement score, it’s telling you that 10 other participants sorted their cards into groups similar to these ones.

Dendrograms

The dendrograms give you another way to quickly spot popular groups of cards, and to get a general sense of how similar (or different) your participants’ card sorts were to each other.

You have two dendrograms to explore, which you’ll find more or less useful depending on the number of completed card sorts you have:

• Got more than 30 participants? The Actual Agreement Method (AAM) dendrogram gives you the data straight: “X% of participants agree with this exact grouping”.
• Got fewer than 30 participants? The Best Merge Method (BMM) tells you “X% of participants agree with parts of this grouping”, and so enables you to extract as much as you can from the data.

You’ll approach closed card sorting results with questions like:

• how many participants sorted the same cards into each category
• the cards with the highest agreement on where they belong, and the cards with the lowest
• which categories meant different things to different people (ie. if every card was sorted into one category at least once, then it’s obviously ambiguous…)
• what were the most popular groups?

Categories table for closed card sorts

The Categories table for a closed card sort tells you how many times a card was sorted into your each given category. You’ll also find out how many unique cards were sorted into each category, with fewer unique cards meaning higher agreement among participants.

With the closed card sort categories results, you could:

• use the top three cards that appear most often in a given category as examples for what belongs there on your actual website
• pinpoint the categories with the least agreement among participants and reword the labels
• get quick answers on ranked items if people sorted the cards according to your ranking criteria.

The Cards table for closed card sorts

The Cards table for a closed card sort tells you which category each card was placed in the most often. You’ll also see the number of unique categories for each card, with fewer categorizations meaning higher agreement among participants.

You can use the cards results to:

• pinpoint the cards with the fewest unique categorizations and thus the clearest matches with their categories
• find out which cards were placed in every category, and could thus be ambiguous
• quickly pinpoint the cards that were placed in all categories at least once.

Results Matrix

The Results Matrix shows you the number of times each card was sorted into your pre-set categories. The darker the blue, the more times a card appears in the corresponding category. The number of participants is also displayed (not the percentages).

The Results Matrix tells you pretty quickly which cards had the highest and lowest agreement between participants on where the cards were best placed.

Popular Placements Matrix

The popular placements matrix shows the percentage participants who sorted each card into the corresponding category, and attempts to propose the most popular groups based on each individual card’s highest placement score.