Thriving in long-term care facilities: instrument development, correspondence between proxy and residents' self ratings and internal consistency in the Norwegian version. Bergland A, Kirkevold M, Sandman PO, Hofoss D, Vassbo T, Edvardsson D. J Adv Nursing. 2013; Early Online. Available online from the Baystate Health Sciences Library, or from PubMed at your institution.
In educational research, we measure things that Paul Visintainer might call "soft." These things are constructs (a term which might seem familiar to you because Tony Artino gave us some insight to writing surveys not too long ago). Constructs help us describe our learners and our patients. Well, this article demonstrates a comprehensive look at developing instruments to measure constructs (no, no - stay with me!).
There are many ways to develop and validate your instrument (yes, I know, the instrument itself is not valid, it produces valid DATA. I know this. I preach this. But in the essence of shortening the sentence, I said "validate your instrument." As long as we're all on board that an instrument which produces valid scores for one group doesn't automatically produce valid scores for another, you'll please bear with me on the details).
So, there are many ways to develop an instrument with rigor that set it up to produce valid scores. These ways all have some similarities, which I think are identified well in this article. They boil down to:
First, the "thing" being measured is defined. This is simple, yet critical, and often skipped. (Bad. No skipping.) In this article, the authors measure "thriving" which they define by what it is and what it is not. In fact, they have a nice background on how it has been defined previously. Even if we think that everyone knows what the construct is that we're measuring, we still define it. (Quick example: Try to measure "success." From whose perspective? Based on financial security? Based on happiness? Wait - how do we measure "happiness"? Exactly.)
Second, question items are developed in a meaningful and thoughtful way. Think theoretical or conceptual framework. Perhaps you conducted focus groups or in-depth interviews and then analyzed them for themes which became the backbone of your instrument. Or, like in this article, you used interviews and a structured lit review. Note that "talked it over with your friend" does not appear here.
Third, you report on some measures of consistency/reliability and validity. Side note here about validity and reliability: my two-year-old sleeps with a stuffed monkey. He needs this monkey to survive. From what I can tell, the monkey does not feel the same way. This is sort of the deal with validity (2-year-old) and reliability (monkey). Validity needs reliability to exist. But reliability can exist just fine without validity. So to demonstrate validity, you must accumulate evidence that it is valid. An accumulation of reliability metrics (see this article) and your theoretically and conceptually sound process for item development (see above) can help here.
Fourth, and Tony Artino wrote about this, instruments need to be piloted. In the highlighted article, a pilot instrument was designed for three different groups of respondents. Statistical analysis helped to guide item reduction.
Even with their comprehensive approach to instrument design in this article, the authors still end with a tool that they claim needs "further psychometric evaluation and refinement." In other words, even after all this work, it's still not done! And, of course, that's the rub with developing an instrument from scratch - it's a lot tougher than using what already exists (thwarting the wheel reinvention). And, if it's not tough, check to see if you missed something, or start a blog to explain it to us.
There are many ways to develop and validate your instrument (yes, I know, the instrument itself is not valid, it produces valid DATA. I know this. I preach this. But in the essence of shortening the sentence, I said "validate your instrument." As long as we're all on board that an instrument which produces valid scores for one group doesn't automatically produce valid scores for another, you'll please bear with me on the details).
So, there are many ways to develop an instrument with rigor that set it up to produce valid scores. These ways all have some similarities, which I think are identified well in this article. They boil down to:
First, the "thing" being measured is defined. This is simple, yet critical, and often skipped. (Bad. No skipping.) In this article, the authors measure "thriving" which they define by what it is and what it is not. In fact, they have a nice background on how it has been defined previously. Even if we think that everyone knows what the construct is that we're measuring, we still define it. (Quick example: Try to measure "success." From whose perspective? Based on financial security? Based on happiness? Wait - how do we measure "happiness"? Exactly.)
Second, question items are developed in a meaningful and thoughtful way. Think theoretical or conceptual framework. Perhaps you conducted focus groups or in-depth interviews and then analyzed them for themes which became the backbone of your instrument. Or, like in this article, you used interviews and a structured lit review. Note that "talked it over with your friend" does not appear here.
Third, you report on some measures of consistency/reliability and validity. Side note here about validity and reliability: my two-year-old sleeps with a stuffed monkey. He needs this monkey to survive. From what I can tell, the monkey does not feel the same way. This is sort of the deal with validity (2-year-old) and reliability (monkey). Validity needs reliability to exist. But reliability can exist just fine without validity. So to demonstrate validity, you must accumulate evidence that it is valid. An accumulation of reliability metrics (see this article) and your theoretically and conceptually sound process for item development (see above) can help here.
Fourth, and Tony Artino wrote about this, instruments need to be piloted. In the highlighted article, a pilot instrument was designed for three different groups of respondents. Statistical analysis helped to guide item reduction.
Even with their comprehensive approach to instrument design in this article, the authors still end with a tool that they claim needs "further psychometric evaluation and refinement." In other words, even after all this work, it's still not done! And, of course, that's the rub with developing an instrument from scratch - it's a lot tougher than using what already exists (thwarting the wheel reinvention). And, if it's not tough, check to see if you missed something, or start a blog to explain it to us.
No comments:
Post a Comment