Thousands of online surveys have already been conducted with LamaPoll. We have provided support to thousands of customers and helped them create, run and analyse online surveys. Over time, this has given us many useful insights that we are happy to share on our website.
Here, we focus on the mistakes that occur most frequently and are also among the easiest to avoid. These points apply to all types of online surveys and generally to offline or paper surveys as well. Paying attention to them can significantly improve your response rate, meaning the number of completed responses. Every additional response also contributes to the reliability of your results and the confidence with which you can make decisions.
Table of Contents
Unclear questions
Avoid leaving room for interpretation unless this is intentional. A pre-test with colleagues who are not familiar with your questionnaire can help: ask them how they understood each question. Where appropriate, also provide a comment field. Comments can reveal issues that predefined response options do not capture.
It is often only during analysis that very different responses show that participants interpreted a question in different contexts. Run a test survey first, send it to a few colleagues and ask them to answer seriously. Then review the results, especially free-text and comment fields.
Examples of ambiguity:
- Ambiguous wording: “sporty” could mean “fair” or “athletic”. Is the intended meaning clear from the context?
- “… received the product broken” could mean the customer received a damaged product or broke it themselves.
- Punctuation can change meaning. Make sure the sentence structure leaves no room for two different interpretations.
- Even apparently obvious wording can be ambiguous. Read questions from the respondent’s point of view, not only from your own.
These examples may seem obvious when you read them in isolation. When you are working on question 30 on page 18, however, it is easy to lose the outside perspective. Test the survey at least once before launching it widely.
Imprecise questions
When you create a survey, nobody is as immersed in the topic as you are. Much of the subject matter may feel clear and obvious to you, but respondents can have a very different level of knowledge and context. When they see your online survey, they should be able to understand the questions correctly without having to reconstruct the background first.
Make every question concrete and precise, and deal with one subject at a time.
- Wrong: If there were an election, who would you vote for?
- Better: If a federal election were held next Saturday [time reference], which party would you vote for with your second vote [definition of the vote]?
A question should contain only one substantive issue.
- Wrong: How do you rate the performance of your colleagues and managers?
- Better: How do you rate the performance of your colleagues, for example on a grading scale?
Terms and abbreviations are also often used even though not all respondents know them. People may guess, choose an option at random or abandon the survey.
- Wrong: Should quarterly SWOT analysis be mandatory for DAX companies?
- Better: first explain what SWOT and DAX mean.
Missing response options
For closed questions, response options need to be exhaustive. If respondents cannot find an option that fits, they may either leave the question unanswered or choose an inaccurate response. This can lead to misleading results and frustration.
Free-text questions and comment fields can help by allowing respondents to add an option that is missing. In our experience, it is difficult to anticipate every possible answer for every respondent. Even apparently simple yes/no questions do not always fit neatly into a binary choice. Questions about gender should likewise provide options beyond “female” and “male”, or an open text field where appropriate. Reliable results depend on giving respondents suitable ways to answer.
Imprecise response options
Besides missing options, response categories are often vague or overlap. Respondents then do not know which option is the right one because several may apply, or none may fit.
Response options should meet three basic classification requirements:
- completeness,
- mutual exclusivity,
- clarity.
Common problems include missing categories and overlapping ranges:
- Wrong: overlapping ranges such as €0 to €2, €2 to €5 and €5 to €10.
- Wrong: incomplete age categories such as 20–30, 30–50 and 50–60.
Take particular care with preselected responses, for example in scale questions or dropdown lists. They can influence respondents to accept the default answer.
Leading questions
The way a question is phrased affects response behaviour. In most cases, you want honest, subjective and individual responses. Participants should not answer what they think you consider correct, but give their own view. Only then can survey results provide a useful basis for decisions.
Example: What did you like least?
This is negatively framed and may encourage a more critical overall response. In some cases criticism may be exactly what you want, but the effect of the wording should be deliberate.
Example: Preselected responses
A default selection can encourage respondents to accept it or, conversely, put them off answering. If the question “How would you rate us?” already has the top score selected, this can appear manipulative.
Example: Unbalanced design and layout
Design and layout can also be suggestive, for example when certain response fields are highlighted or made larger and more prominent. Avoid this kind of influence if you want responses to be as unbiased as possible.
Surveys that are too long
The optimum online survey has 15–25 questions (Bosnjak & Batinic). More questions can overwhelm respondents. However, this statement should be treated with caution because every survey and every target group is different. The acceptable length depends on the questionnaire, respondents’ level of involvement and their interest in the topic. Employees may be more willing to complete longer and more complex surveys when they are directly affected by the results. Customers, by contrast, often place greater value on clear design and shorter questionnaires and may be less willing to work through complex questions. In any case, focus on what is essential.
Note: a question is not simply one line of question text followed by many response options. Every prompt that asks respondents to take a position requires cognitive effort.
Matrix question example: a matrix is not really just one question. Each row effectively asks another question, and the cognitive load is often underestimated.
Be honest about the expected completion time. If your invitation or first questionnaire page describes the survey as “short” and says it takes “2–3 minutes”, that should reflect reality. Otherwise you risk losing credibility and data quality.
Long surveys are also more likely to be abandoned. Customers in particular are unlikely to spend unlimited free time working through a lengthy questionnaire.
Asking too many questions
Some survey creators seem to think: “Now that the participant is here, I might as well ask everything.” For a successful survey, however, fewer relevant questions are often better than an exhaustive list. Response categories should be complete, but that does not mean every conceivable topic has to be covered.
Limit the questionnaire and focus on what matters most. Questions that are particularly relevant or controversial should take priority.
- Wrong: repeatedly asking about the relevance of many similar website attributes such as layout, usability, design, content, entertainment, ease of use, loading time, recency, readability and navigation.
- Better: focus on the essentials.
Tip: Start by asking what matters most, for example: “What is most important to you?” You can then ask respondents to rate only the options they marked as important. This shortens the questionnaire for each person while maintaining, and potentially improving, data quality.
Too many questions can lead to disorientation, higher dropout rates and boredom. This increases the risk of respondents clicking through without reading carefully.
Texts that are too long
Usability research confirms that users skip parts of a text they do not consider important and focus instead on highlighted passages. Reading web text is 25% slower than reading on paper, so experts recommend writing 50% less.
Focus on the essentials. If a question or scenario requires a very long explanation, there is a risk that respondents will misunderstand it or not read it at all. If longer explanations are unavoidable, use paragraphs, appropriate visual elements and bullet points rather than long, nested sentences. This breaks up the “wall of text” and makes the content easier to follow.
Question text should therefore be short, simple and easy to scan.
Wrong (an unnecessarily complicated frame of reference):
- Which of the following topics would interest you, taking into account the stated transmission medium, in the form of a completed course?
This wording requires a great deal of concentration and remains difficult to understand even then.
Also problematic:
- long, nested sentences,
- too many specialist terms,
- unnecessary Anglicisms,
- unexplained abbreviations.
Boring questionnaires
A boring questionnaire can lead to high dropout rates and poorer results because respondents’ concentration declines. Experienced internet users in particular may have a short attention span.
Questionnaires feel monotonous when consecutive questions repeatedly cover the same content in the same way. Varying the question types and keeping them appropriate to the flow of the questionnaire can help maintain attention.
Purely statistical questions are often less interesting to respondents. They provide information without necessarily learning anything themselves. If these sections drag on, respondents may feel that their time is being wasted and abandon the survey.
Tip: Consider variety not only in the content but also in presentation. Use different question types where appropriate.
Poor web typography
Text on a screen is perceived differently from text in print. Do not underestimate web typography and its effect on respondents.
The most important rules:
- Line length: lines averaging 40 to 50 characters produce the best reading times.
- Font size: text in web browsers should be at least 7 pixels high. Smaller text noticeably slows reading. Increasing it to 8 pixels does not further accelerate reading, at least on flat-panel displays.
- Interaction: there is also a strong interaction between font size and typeface. Reading times for Times improve as font size increases, while Verdana is read more slowly at the largest measured size than at the next smaller size.
- Typeface: Times New Roman, Verdana and Arial are common typefaces on the internet. Research has found no significant difference in their effect on reading speed, although they are perceived differently.
- Choosing a typeface: Arial and Verdana are rated as more attractive and subjectively easier to read than Times New Roman.
Tip: If you use an unusual corporate font, respondents may not have it installed and may see a different font instead. Avoid rare or highly unusual fonts.
Matrix questions
Matrix questions are popular, but we recommend avoiding them altogether. They present many questions with similar or identical response categories in a table.
Methodological research has identified a tendency for respondents to fall into response patterns. This is cognitively easier than reconsidering each row independently. The effect can occur across market research methods and may be particularly relevant in anonymous online surveys.
Long questionnaires, boredom, uninteresting questions and complex wording can further reinforce these response patterns.
The visual relationship between individual questions and their responses may also become less clear.
Survey creators often use matrix questions to collect a large amount of information quickly or in the hope of making the questionnaire appear shorter. In practice, matrix-question results can be harder to use and may require additional analysis or cleaning.
What is LamaPoll?
LamaPoll is an online survey tool for creating employee surveys and measuring employee satisfaction. With us, you can create, conduct, and analyse all types of employee surveys.
Set up your employee survey in just a few steps, customise it with your company's corporate design, and invite your employees conveniently by email or via your intranet. You'll gather insights to boost employee satisfaction in no time. Sign up for free with our online survey tool. You can then run surveys for up to 50 employees immediately and with no obligation. If you need more responses for your online employee survey, simply use our monthly plans. You can also make use of our templates and examples for employee survey questionnaires.
