Beta testing is a crucial part of every new feature release.
Even if you’ve done a good job building and designing it, finding every issue on your own is hard.
There’s also a good chance you don’t dogfood your own product as much as you think you do.
Having an extra pair (or a few) of eyes helps cover an and all various use cases.
This is why it’s important to let a selected few people play around with a feature before releasing it into the wild.
Why selection is crucial in beta testing
If you’re an established business with existing customers, you have a goldmine of beta testers right at your fingertips.
Your existing users are more invested in improving it. They use your product or service for their own benefit, and therefore genuinely want it to be better.
However, the specific group of testers should be very carefully selected.
You would think an open beta test would create much more feedback.
However:
- Open betas are harder to manage (they tend to include lots of people), especially for small teams.
- People often join out of curiosity and don’t offer much value (but that doesn’t mean you don’t have to manage them).
- People outside of your target group can join (their feedback isn’t the best for actually improving).
- If anyone can join, anyone can see a less-than-perfect version of the new feature.
So, even if an open beta creates “more feedback” in general, it doesn’t mean it’s useful feedback.
The feedback you want from beta testing is actionable and targeted.
There are customers that are easy to get feedback from and who are qualified and able to do it. And then there are customers who aren’t.
So, how do you choose the best beta testers?
Choose beta testers only from your target audience
This is probably the most important thing to keep in mind.
Your beta testers should definitely be a part of your core target user base. If your target audience is “everyone”, then you have bigger problems than beta testing.
Your target audience are the people and/or companies you’re building the product for to begin with. This is why getting their feedback first is important.
For example, Canny is primarily built for SMB-s and SaaS companies. They are our target market. They’re the companies we try to benefit most with our product (and, let’s be honest, the ones who generate most of our RR).
If we asked for beta feedback from our corporate clients, the results would be very different. Their needs and wants are very different. It would skew our results on what matters most.
This doesn’t mean we don’t care about them. We just don’t prioritize their feedback when asking for feedback on a feature meant for our target market.
Every company has edge case customers or users. They’re fine to have if they’re happy with their arrangement and not making demands.
However, they’re not the greatest people to ask for advice for when testing a new feature. They’re not the ones you’re building the product for.
Choose people who have asked for the feature
If you track which users have requested which features, you’re in luck.
Who’s a better candidate for testing a feature than the people who have specifically asked for it?
If they asked for it, it means it’s supposed to solve a problem for them.
With their help, you can see if you’ve solved the problem right for the most important people behind it, or not.
Not only will they be stoked that the feature is coming, but they’ll be super happy to give you their input. Since they needed it and will definitely be using it, they’ll be invested in improving it.
If you’re using Canny, you can just get a list of people who have upvoted or commented on a feature request.
You’re one comment or message away from letting them know they can hop into beta if they’d like.
Choose people who have given good constructive feedback before
Some people genuinely love giving feedback.
We’re not talking about bug reports—bugs are annoyances that disrupt people’s work. They have to let you know about it so you can fix it.
Reaching out about a bug doesn’t mean that someone is interested in getting involved. It’s a chore.
We’re talking about people who have reached out with ideas, requests, or general feedback before. The people whose thoughts you have found to be genuinely valuable and interesting.
This also includes people who are generally a big fan of your company, team or product. They care, so they’ll be more attentive.
You can start flagging certain customers as good future beta testers, or adding them to a list.
This way, you have a good starting point already when choosing testers for your next feature.
Choose beta testers from people who use your product a lot
Most SaaS products have “zombie” customers. These are the users who are signed up, but use the product very little…
…if at all.
Even if they’re willing to beta test a new feature, their feedback isn’t exactly the most constructive you can get.
The problem is that if they don’t use the product a lot to begin with, they’re not aware of its full capabilities. They most likely don’t even need the new feature.
They’re clearly not particularly invested in your company or product. This means that their feedback (if there even is any) will most likely be superficial and not very useful.
It’ll also most likely take way longer and be way harder to engage with them. If they don’t engage with your product a lot, why would they engage with you a lot?
It’s more crucial to get it right for people who rely on your success and frequently interact with your product
Make sure they have time for beta testing
Now that you have a list of users based on these “requirements”, make sure that they actually have time for it.
Even if they’re the perfect candidate for being a beta tester and are willing to do it, it doesn’t mean they can.
Beta testing (especially the first rounds of it) can be pretty time consuming.
People have stuff going on that might mean they’re too busy or unable to give you constructive feedback.
So, before you commit to your list of testers, make sure to let them know exactly the “scope” of what you expect from them.
If they don’t think they have the resources or time to do it currently, they can opt out.
This way, you won’t be disappointed if you don’t get anything useful from them, and they won’t feel bad about it.
Not all users are created equal for beta testing
As a baseline, you should always try to care for your customers equally.
However, beta testing is a lot different from your everyday operations. You’re building something new, and you want it to be awesome for everyone who will be using it in the future.
And, you don’t want to take eons to get it to that point. This means that your goal with beta testing should be efficiency.
Whereas most users are efficient at being your customers, they might not be efficient at giving feedback.
Pick and choose the people who are most qualified, excited, able, and constructive.
You can move your polishing process on faster, and get your new feature to its optimal capacity ASAP.
The best way to gather & manage user feedback is to use Canny. Try it for free!
Nice article.