Attention, students! Please complete the following brief questionnaire:
On a scale of 1-5, with 1 representing “no experience” and 5 representing “high proficiency,” how would you rate your skills for writing constituent surveys and reporting on survey feedback?
If you’ve already been reading ahead this semester in your Research Methods textbook, you probably recognized that my survey question included a survey no-no: a double-barreled question. Double-barreled questions are survey items that ask respondents about something and something else. For example, you might have read my survey question and thought, “I’m great at writing surveys! But I really don’t know how to report on my data.” If you thought that, you’d probably respond with a “3,” thinking it was a neutral answer. Unfortunately, I would have thought that your “3” meant that you had basic skills in both writing and reporting.
This is just one example of why you need to make sure that you are following survey-writing best practices. To ensure clean, actionable data, people need to know what they are being asked. Look out for questions with “and” in them, and turn them into two separate questions instead. Below, we share some other best practices for writing strong surveys.
Provide respondents with opportunities to share their thoughts candidly and directly in open-ended questions
Open-ended questions are useful for providing you with direct constituent quotations, in their own words. The answers you receive to open-ended questions can guide your survey revisions for future iterations, because a respondent may bring up a point that you may not have considered. Furthermore, qualitative data (i.e., non-numerical data, such as open-ended question responses) can be coded for relevant themes, enabling quantitative statistical analyses of rich constituent responses.
Build data categorization into your survey structure
Open-ended questions can be useful, but require extra effort when you want to present the findings in terms of a measurable outcome, such as “95% of people said X.” When you write your survey, if you include both open-ended questions and categorical questions (e.g., rating scales, choices, check boxes), you are providing yourself with more flexibility for reporting on your data in different ways. For example, you could pair an open-ended question, such as “Why do you donate?” with a pick list (i.e., categorical data), such as “Which of the following reasons best describe why you donate?”
Provide room for response variance
Many surveys include a question that asks, “How much do you agree with…?” and provides three response options: Not at all, Somewhat, or Completely. A three point scale like this one does not provide a wide range of options for respondents, making it unlikely that much variance will emerge. In other words, respondents might all respond in a uniform way – and that’s not necessarily helpful for you to see. Using a five or seven point scale with “how often,” “how much,” or “how likely” questions is recommended to provide more options for constituents who would like to respond somewhere between the extremes.
Make the survey identifiable
If you want to track constituents’ data over time, make sure that surveys are clearly identified by the respondent. A first and last name alone, most likely will not suffice. Use unique ID numbers instead, or an email address may be sufficient in certain scenarios. ID numbers are also preferable in cases where other personal information is provided throughout the survey to ensure constituents’ confidentiality and honest reporting.
Group respondents
Collect adequate demographic and other grouping data so that a professional data analyst, if hired, could examine patterns among and between groups. Collect both demographic information (e.g., zip code, number of people in the household) and organization-specific information (e.g., membership level, intake date). If you already have this data, perhaps from a previous intake form, you do not need to collect this type of information on every survey. This is especially true if you have a tool like Salesforce, in which your survey data can be easily integrated with your constituent demographic and participation data. Remember to include items that should be collected again because they may change over time. One example of this is dietary restrictions.
Avoid grids and matrices for response entry
According to a leading online survey tool, Qualtrics, grids are not mobile-friendly, and respondents are less likely to fill them in correctly or accurately.
Consider participant reading level
You cannot collect good data if people are unable to understand your survey. Be cognizant of who your respondents are, and use tools like the “Show Readability Statistics” feature in Word, which measures reading ease.
Offer an incentive for survey completion
Raffling off a gift card or providing extra credit to get people to respond to surveys is popular among for-profits, nonprofits, and researchers alike. Studies have shown that incentives work to increase survey response rates – although the type of survey and incentive matters. If it is an option for your organization, then offering an incentive may be worthwhile. Be mission-specific if you can. For example, if you’re a mentoring organization, consider a raffle that includes two tickets for a baseball game. The winning volunteer can make a mentor/mentee night out of it, furthering your organization’s mission. If you are surveying donors, perhaps you can offer a chance to win a gift provided by one of your board members – especially if you have any famous board members that can offer up something signed.
Use standardized questionnaires
Using existing surveys enables you to compare your constituents to other groups of survey respondents, providing a relevant benchmark. You may want to modify the survey (e.g., change the reading level, use only relevant subscales). However, be cautious when modifying standardized questionnaires, as making changes reduces your ability to compare your findings to other findings. If you’re using, and especially if you’re modifying, a standardized questionnaire, you may want to bring in a professional data analyst to help you identify the best survey for your needs and modify it appropriately. MeasuringU, a quantitative research firm in Denver, has provided helpful tips on customizing standardized surveys to help you avoid some of the common pitfalls.
Ensure that your survey is clearly related to the outcome(s) you want to track
While it is absolutely okay to have a variety of items on your survey, it’s easy to get caught up with extraneous information that may be interesting but is unrelated to your ultimate question. Always prioritize mission-focused or outcome-focused questions. You don’t want to realize that you’ve collected 200 surveys, and while you now know that your donors love the updated look of your website (yay!), you still don’t know what motivates their giving.
Now that you know how to write a strong survey, you can collect accurate and actionable constituent data – and you can get an A in Research Methods this semester. Although, you are not quite ready to deliver your survey just yet. You will need to identify and configure a tool for distributing the surveys and collecting the results. There are quite a few options available through the AppExchange, that will integrate with your Salesforce instance. Now, it’s time to start writing that survey to load into the survey tool of your choice.
You may be interested in reading the following:
