This is the fifth in a series of articles on the topic of test plans and test case management. You will find the fourth article of the series, How to write a test plan on Lean Testing’s blog.
Playtests, or user testing surveys, differ from regular test plans since they require interpretation. You do not ask the tester an objective question (whether or not something works) but a subjective one (what he thinks of something). Therefore, these tests increase the risk of biased results (data containing errors caused by unintentionally encouraging one outcome over others).
When dealing with human beings, there is no way to completely eliminate bias. However, there are ways to mitigate it by asking the right questions and using proper phrasing. Although all surveys yield biased results to some extent, you should only focus on limiting bias which directly impacts your study. To do this, you need to understand what you are looking to learn, what data is important to you, and what might stand in your way or affect the study.
Here are some things that you can do to limit bias in your playtests.
Include an introduction
Create a short introduction to explain the procedure and what you expect from participants. In this introduction, it is good to remind the testers that their feedback will remain entirely anonymous and that you welcome all feedback, both good and bad.
Be careful not to give participants any indication as to what you would like to hear, such as “We want to make sure that we provide a great service” or “We want our clients to be 100% satisfied” etc. Instead, let them know that you are interested in hearing everything they have to say in order to make your product better, even if it means receiving negative feedback.
Use clear language
Remember that your readers may not understand terms that are too technical. Some of them may not be native English speakers. They may come from different countries or regions and have different backgrounds. Try to avoid phrases and figures of speech that leave room for interpretation. Be careful to write short, clear and concise sentences that can be easily understood by all with simple vocabulary.
Be careful to stay neutral when phrasing your questions. Make sure that all possible answers are similar (that none of them stands out) and that the questions do not orient readers towards a certain answer. Avoid words that have positive or negative connotations.
Keep in mind that people have a much harder time choosing an option which differs from the one they are presented with. For example, you should never ask a tester:
Did you find this tutorial useful?
When presented with a statement, a user will interpret it as the status quo, and will be much less likely to disagree with it in the future. In this case, the fact that the question contained the word “useful” is subconsciously leading the respondent.
It is better to phrase the question in a neutral way by presenting different options, but not leaning toward any, as such:
Did you find this tutorial:
- Interesting but not necessary
A good example of this bias is the rate of organ donation per country: countries who ask people to opt out of donation have a 99% donor rate, whereas countries who ask people to opt in have a 4% – 20% donor rate.
This isn’t due to the fact that certain countries are much more generous than others, but rather to the fact people rarely choose the option that differs from the status quo they are presented with (Dan Ariely, Main Lessons of Psychology).
Styles and colors
Make sure the style and colors of the survey are neutral. You do not want the survey to look like it is intended for a specific age group or population as it may influence peoples’ answers.
Make sure your survey is representative
One of the biggest sources of bias are unrepresentative samples. Before organizing your playtests, you must understand your target users in order to ask for a representative sample of testers. For example, if you are developing a casual trivia game, using a testing group of hardcore gamers will yield very meaningful results.
You should be aware of your target audience’s:
- Commonly used devices (ex: children will not use the same devices as older, affluent businessmen)
- Gamer profile
- Familiarity with the game (do you want testers who are already fans or ones who have never heard of it?)
Playtests can be extremely useful when looking to understand your users better and their reactions when playing your game. They have become more and more important with the rise of casual games and in-app purchases, as gaming companies look to maximize their returns.
Are you looking for help organizing your playtests? Don’t hesitate to contact Crowdsourced Testing‘s team of experts in playtests and survey analysis, we would love to talk to you about how we can help you understand your users better!
For more information on test case management, please stay tuned for the next article in this series: Lean Testing for Test Case Management.