The short answer: aim for around 50 participants for each group you want to report on. You'll start to see patterns from about 30, and the picture becomes steadier towards 50. Beyond that, extra people change the numbers less and less.
The longer answer depends on what you're trying to learn.
Why more than a usability test?
You may have heard that five users are enough to find most usability problems. That advice is about moderated, qualitative testing, where you watch each person and hear why they struggle. A handful of sessions reveals the big problems because you see them happen.
A tree test is quantitative. You're measuring things like "62% of people found the right place", and a percentage from a small group is very uncertain. With 10 people, 6 successes is consistent with a true rate anywhere from about 31% to 83%. With 50 people, the same 60% narrows to roughly 46% to 72%. That's the difference between a hunch and something you can act on.
Picking a number
| Participants per group | What you can say |
|---|---|
| 10 to 20 | Spot obvious failures. Treat percentages as rough. |
| 30 | Patterns start to appear. Good for an early check. |
| 50 | Solid for most decisions. A sensible default. |
| 100+ | Useful for comparing two trees, or small differences. |
ClickTree shows a likely range next to each success rate, so you can see how certain a number is at your current sample size.
Several user groups
If you'll compare groups, such as customers and staff, or new and returning visitors, you need enough people in each group. Two groups of 50 means 100 participants in total. Decide this before you recruit, because it's hard to top up one group later.
Long task lists
Every participant usually does every task. If you have so many tasks that each person only does some of them, you'll need more people to get enough answers per task. For example, with 20 tasks and 10 per person, you need about twice as many participants for the same number of answers per task.
It's usually better to keep studies to 5 to 10 tasks and run a second study if needed.
Comparing two versions
When you test a new tree against the old one, you're looking for a difference between two percentages. Small differences need large samples to be convincing. As a rough guide, 50 people per version will usually detect a big improvement (say, from 50% to 75%), but a small one (from 70% to 75%) would need many hundreds per version to show up reliably. ClickTree's Compare view tells you whether a change between rounds is likely real or could be chance.
Quality matters as much as quantity
- Recruit people like your real users. Fifty of the wrong people is worse than thirty of the right ones.
- Remove obviously careless responses, such as someone who skipped every task in seconds.
- Watch for tasks everyone skipped. That's often a badly worded task, not a navigation problem.
Ready to start? Here's how to run a tree test, step by step.