You can't fix by analysis what you've spoiled by design. Rickards G, Magee C, Artino AR. JGME. 2012; 4(4): 407-410. Available online from the Baystate Health Sciences Library and at your institution.
Tracing the steps of survey design: A graduate medical education research example. Magee C, Rickards G, Byars LA, Artino AR. JGME. 2013; 5(1): 1-5. Available online from the Baystate Health Sciences Library and at your institution.
What do our respondents think we're asking? Using cognitive interviewing to improve medical education surveys. Willis GB, Artino AR. JGME. 2013; 5(3): 353-356. Available online from the Baystate Health Sciences Library and at your institution.
If you’re anything like me, you’ve completed more questionnaires than you care to count. Whether it’s an end-of-course evaluation of a workshop you attended or a satisfaction survey from a recent visit to the clinic, questionnaires are ubiquitous in education and health care. There’s one problem though, and it’s a problem you’ve surely encountered – many questionnaires are poorly designed and often times they fail to capture the very thing they’re attempting to measure. Some common problems with questionnaires include confusing or biased language, baffling visual layout and design, and unclear instructions. Unfortunately, in the age of email, the Internet, and online tools such as SurveyMonkey, the number of survey requests grows exponentially with each passing day.
Despite the plethora of bad questionnaires that exist in education and health care, there is a wealth of evidence-based knowledge regarding the “best practices” in survey design. Much of this knowledge is detailed in the highlighted articles and briefly summarized below as three principles:
1. You can’t fix by analysis what you’ve spoiled by design. Even though this principle is true for all types of research and evaluation, it is especially true in questionnaire design for one simple reason – when creating a survey, we’re often trying to assess things that are traditionally hard to measure (so-called “fuzzy” or non-observable constructs). These fuzzy constructs include things like student anxiety, resident confidence, and faculty job satisfaction. As such, it’s critically important that survey designers take the time to carefully design and pretest their questionnaires prior to implementation.
One way to pretest a questionnaire is to have a group of experts review the items and then have a group of potential respondents complete the survey while you observe. Having experts review your draft ensures, among other things, that the content of your survey is relevant and clear; whereas having potential respondents complete your survey verifies that the way they interpret your items aligns with what you had in mind when you designed the questionnaire.
2. The questions guide the answers. People often underestimate the degree to which the precise wording of a question plays a critical role in determining the answers provided by respondents. Take, for example, the following two questions about health insurance: “Are you fairly treated by your health insurance company?” versus “Does your health insurance company resort to deception in order to cheat you of covered benefits?” These two questions would likely elicit very different responses, and it probably wouldn’t surprise you to find that an advocate for health insurance reform asked the second question. Clearly, words like “deception” and “cheat” are strong indications that the author of the question doesn’t have a high opinion of the health insurance industry. Thus, as this principle implies, the wording of a question largely determines the answers people provide.
And while this principle is true in everyday life, when it comes to questionnaires, the effect is even more pronounced; most surveys don’t give respondents the chance to provide feedback about misunderstandings and ambiguities. Therefore, when it comes to questionnaire design, small wording changes can often make big differences, which is another reason to pretest your survey before sending it out to 3,000 respondents.
3. Think of it as a conversation. At the end of the day, a questionnaire is really just a conversation between you (the skillful survey designer) and your respondents. As such, you should consider the implicit assumptions that underlie the conduct of conversations in everyday life. These conversational “rules” include the idea that speakers should try to be informative, truthful, relevant, and clear. If you break these rules as a survey designer, you shouldn’t be surprised (or upset) if your respondents, in turn, provide you with poor-quality answers.
An important implication of this principle is that you should ask questions when you want to learn something from your respondents, as opposed to asking them to agree or disagree with a list of statements. Asking people to rate a bunch of statements is not very conversational – when’s the last time you went up to a friend in the hallway and asked her to “rate the following statements on a scale of 1 to 10”? At the end of the day, people are more familiar and adept at answering questions – not rating statements – so, as an informed survey designer, you should ask well-thought-out questions and pretest those questions on experts and potential respondents prior to implementing your survey.
Notwithstanding the temptation to think of survey design as “more art than science,” there’s actually quite a bit of scientific evidence to guide you through the survey design process. Following these evidence-based best practices will not only save you time and effort during data analysis and interpretation, but they will also improve the chances that your survey will actually measure what you intend it to measure. Bottom Line:
A. Put some effort and thought into the development of your questionnaire and pretest your survey items before implementation.
B. Use these articles as a way to develop good practice.
C. All of the above!
C. All of the above!
