Optimise your questionnaire: expert tips

Optimising your questionnaire

A well-designed questionnaire helps to build and maintain respondents' interest and motivation to take part. Two major risks to data quality are random responses and respondents dropping out before completing the survey. To reduce these risks, potential sources of error should be identified and, where possible, eliminated.

Writing good questions

A good question is as precise and concise as possible. Always assume that some respondents may be distracted by other tasks and will not give your survey their full attention. The following general tips can help you formulate clear questions.

Keep your questions simple so that everyone in your target group understands them in the same way:

1. Use language that is familiar to your target group.
2. Keep questions as short and precise as possible.
3. Use neutral, factual wording (“malfunction” rather than “disaster”).
4. Use common vocabulary and explain technical terms where necessary.
5. Avoid hypothetical questions.
6. Avoid negatively worded questions (“This software is not complicated”).
7. Avoid leading questions and assumptions (“How would you rate our excellent software?”).

Tip: Ask about only one issue at a time. Double-barrelled questions such as “How would you rate the functionality and customisability of the software?” produce ambiguous results.

Tip: If a question refers to frequency or a time interval, define it as precisely as possible. For example, replace “often” with a specific formulation such as “at least once a day”.

Tip: Avoid questions that assume knowledge most respondents are unlikely to have. If you need to explain a complex topic, you can use the LamaPoll info button to provide targeted additional information.

Question types

Questions can generally be divided into two groups: closed and open-ended questions.

• The main advantage of closed questions is that the data is easy to analyse and answering requires less effort from respondents.
• Open-ended questions give respondents room to express themselves in their own words and can provide detailed insights and additional information that you might not otherwise obtain.

Closed questions

LamaPoll offers a wide range of question types and plenty of flexibility when designing a questionnaire. You can also use collections of frequently used questions, such as demographic questions, contact details and questions for customer or employee surveys, and adapt them to your needs.

Tip: For clarity, provide as many response options as necessary and as few as possible. If you cannot anticipate every possible response, it is usually better to list the most likely options and add an “Other” option, with or without a text field, rather than presenting an unwieldy list.

Tip: Avoid situations in which respondents cannot find a suitable response option or cannot decide which option applies to them. Response options should therefore be complete and should not overlap.

Tip: Use the advanced editing options to tailor a question to your information requirements. Depending on the question type, these include options for defining permitted input formats and value ranges, display settings, scoring and more.

Matrix questions

A matrix question is useful when several aspects of a topic need to be assessed using the same criteria. This compact format can make a questionnaire easier to scan while allowing you to collect detailed information without making the questionnaire appear unnecessarily long.

Many matrix questions use a rating scale, for example:

• 0 to 10
• 0 to 100
• never to always
• negative to positive
• strongly disagree to strongly agree

LamaPoll gives you extensive options for designing matrix questions, including single-choice and multiple-choice matrices, visual variants using stars or smileys, trend questions, semantic differential scales, and two- or three-part matrices in which each item is assessed against several criteria.

Tip: Do not overload respondents. A matrix is not really a single question: each row is an item that must be read and answered. Too many items can contribute to survey fatigue and encourage respondents to select answers without carefully reading the question text, which can reduce data quality. As a rule of thumb, avoid matrix questions with more than six items where possible. Double and triple matrices should also be used sparingly.

Tip: The individual items in a matrix should collectively cover the relevant aspects of the topic and should not overlap, so that you do not ask the same thing twice.

Tip: A common approach is to use a scale with an odd number of points and a neutral midpoint such as “neither/nor”. You can also use an even-numbered scale, sometimes referred to as “forced choice”, where respondents cannot select a neutral middle category.

Tip: The appropriate number of scale points depends largely on the topic and the level of differentiation you need. Five- and seven-point scales are common. A highly differentiated scale may add little value for a small respondent group if the results will later be combined into broad categories such as “agree” and “disagree”.

Tip: To reduce avoidable bias, use a balanced scale. For example, a scale can contain two negative points, one neutral midpoint and two positive points.

Tip: Clear labels such as “very negative”, “rather negative”, “neither positive nor negative”, “rather positive” and “very positive” make the meaning of each scale point explicit. For a highly granular scale, such as 0 to 100, it may be sufficient to label only the endpoints and midpoint.

Tip: If your survey contains several matrix questions, keep the number of scale points as consistent as possible. The direction of the scales should also remain consistent, for example always negative on the left and positive on the right.

Open-ended questions

With an open-ended question, respondents answer in their own words. Typical examples include:

1. General opinion or assessment questions, such as “What is important to you in relation to topic XYZ?”, or a final question such as “Is there anything else you would like to tell us?”
2. Exploratory questions, for example: “Which problems with XYZ software annoy you most? Please name the three most troublesome.”
3. Requests for additional information or reasons following a closed question, for example: “You indicated above that you are less satisfied with XYZ software. Could you briefly explain why?”

The main advantage of open-ended questions is that they do not restrict what respondents can tell you. They can be particularly useful because they allow you to:

1. Discover unexpected responses and learn something new.
2. Collect appropriate answers to complex questions. A simple yes/no or multiple-choice response is often insufficient when a detailed description or explanation is needed.
3. Explore a topic without extensive prior knowledge. Closed questions require you to anticipate at least some possible responses. If you know little about a topic, an open-ended question can help you identify relevant aspects first.

Open-ended questions should nevertheless be used with care:

• Depending on the size of the respondent group, preparing and coding free-text responses can involve substantial additional work.
• Open-ended questions require more effort from respondents than closed formats. If used too frequently, they can increase the risk of respondents abandoning the questionnaire. Each open-ended question should therefore be important enough to justify the time and effort required to answer it.
• The quality and level of detail in free-text responses can vary considerably. Some respondents provide extensive explanations, while others enter only brief or difficult-to-interpret keywords. Many respondents may also leave open-ended questions unanswered.

Tip: Not everyone in your target group will be willing to provide a detailed explanation at a given moment. Open text fields should therefore be optional.

Tip: Be as precise as possible. If the wording is vague, respondents may interpret the question differently from what you intended, focus on a secondary issue or provide only vague answers.

Tip: Respondents answer open-ended questions in their own words. Their wording or descriptions of specific situations can sometimes reveal information that makes an otherwise anonymous response identifiable. If this is relevant to your survey, inform respondents of the risk and ask them not to include information that could identify them personally.

Tip: LamaPoll questionnaires can include file-upload questions, allowing respondents to provide their own files. This can produce information comparable to open-ended responses. If you use this question type, decide in advance how you plan to evaluate the submitted material.

Questionnaire flow

A questionnaire should provide respondents with as smooth and positive an experience as possible. The challenge is to make individual questions self-explanatory and the overall flow easy to follow. A typical questionnaire has three parts: (1) an introduction and the relevant data protection information or consent, (2) the questions, which may be divided into sections, and (3) a thank-you page.

The first page should maintain existing interest and encourage undecided invitees to continue. To keep early drop-out as low as possible, present the key information clearly and concisely at the start:

• the topic
• the benefit for respondents
• the expected completion time
• the participation deadline
• information about the responsible organisation and contact persons

The first page should also provide information about measures to protect anonymity, how data is stored and used, retention periods, and whether cookies are used to prevent multiple submissions.

Tip: Make responsibilities transparent and provide an easy way for respondents to get in touch. For example, you can include legal information and an email link in the survey footer for direct enquiries.

A good questionnaire follows a logical sequence of topics that respondents can understand.

Tip: Particularly for difficult topics, consider starting with a simple icebreaker question that relates to respondents' experience and is easy for everyone to answer.

Tip: Where useful, place a short set of completion instructions, possibly with an example, before the main section.

Tip: Use attributes, skips, filters and visibility conditions so that respondents see only the questions that are relevant to them.

Tip: Use mandatory questions sparingly. They may be necessary when questions are linked to filters or visibility conditions, but otherwise they can reduce motivation to continue.

Author of the guest article “Expert Tips”

Elisabeth Kerkhoff, graduate social scientist
kerkhoff@extern.lamapoll.de