Recently we ran a card sorting study on CA.gov providing us with some interesting insights into how our customers view and think regarding the organization of our content. The tool we utilized for the study was chosen because it is free, for a limited time. As of the publishing of this post, it is still being offered for free.
ConceptCodify.com allows you to create a simple card sort study and to brand it in a manner so that it looks more “officially sanctioned” by your agency. The following is what it would look like if your were in the middle of doing the study via a tablet (yes, it is responsive). Note: some cards from the “deck” I have already moved over into one of the “groups”.
Note: You can still view and perform this same card sort study here: CA.gov Card Sort Study
The ConceptCodify tool provides some interesting analytical views of your study results. The following is called an “Hierarchical Clustering Analysis”
I tried “Googling” the term: “hierarchical clustering analysis” and after viewing a few articles written by college universities, I decided it was best to just view the graph and know that the items clustered together at a lower level were identified as more closely related by the study participants. ConceptCodify provides a good “knowledge base” with explanations. I prefer their explanation for “hierarchical clustering analysis”:
A clustering analysis is a research analysis tool that is used to create groupings of similar items. A hierarchical clustering analysis links pairs of items from the bottom up, forming a tree. The hierarchical clustering analysis is the primary analysis tool for any card sorting study, and is a serious win for any information architecture work.
Similar information to the graph above is a “Similarity Matrix” and the “Relationship Ranking Analysis.” The Relationship Ranking Analysis is a that table gives a percentage measure to the highest ranking similarities between the cards of the card sort study. (Note: the Similarity Matrix seems to show the same information, but it is actually harder to use and understand.)
They also is provide an “Ordering Analysis” view to your study data.
ConceptCodify gives the following explanation of the value of this analysis method:
The idea behind this sort of analysis is it can provide ranking information of the cards that otherwise is not considered in other methods, such as the clustering or similarity analysis. It can be used to determine the optimal order within groups defined by the clustering analysis for example.
Finally last, but certainly not least, ConceptCodify provides a “Group Names Analysis” which can be used to basically see the groups that your study participants tended to place cards into the most.
How We Set Up Our Card Sort Study
We placed a very visible link to the study on the CA.gov home page and on every page of the site, in the top right hand side column.
As you already have seen by now, we branded our card sort study with the official CA.gov logo and color. We also modified some of the “directional” wording on the study page to make it a little easier for the users to quickly understand what to do.
Note: it took about month to gather results from 70 volunteer participants.
Lessons Learned:
- Participants, for the most part, will not create a new group to sort into, even if you encourage them to do so.
- Create different group names for the same thing if you are looking for insights into the best group name for your highest level of site organization and navigation.
- Examine the Google search analytics and Webmaster tools to understand the types of terms people are mostly using to find content on your site and consider those “names” for groups and cards for your study.
- Don’t make your “card deck” to big. You will discourage people from completing the study.
Some “Bugs” in the ConceptCodify product we discovered:
1. It appears that if the study participant is doing the study with possibly Google automatic translate on, that the translated name of the card gets captured. The evidence of something like this happening can be found in some of the results I am showing above.
2. Don’t put any spaces into the “group” names in your study. What happens is ConceptCodify will treat that group as two groups with replicated data, each one with one of the words from the original two word name. For example: we originally had a group named “CA Government”. ConceptCodify created two groups in the “Group Names Analysis”, one named “CA” and the other named “Government”.