September 2012: Dose of Education Research Basics

AM Last Page: Reliability and Validity in Educational Measurement, Artino AR. Durning SJ, Creel AH. Academic Medicine, 2010; 85(9): 1545. 

AM Last Page: Understanding Qualitative and Quantitative Research Paradigms in Academic Medicine, Castillo-Page L, Bodilly S, Bunton SA. Academic Medicine, 2012; 87(3): 386.

The development and dissemination of educational milestones from the ACGME has helped infuse many educational research concepts into our vernacular. We have “qualitative” and “quantitative” approaches towards evaluating residents with milestones, and we hope to have “valid” and “reliable” evaluation tools. As our conversations progress and our evaluations mature, it’s helpful to pause and ensure that we’re all on the same page-or, pages, in this case.

The “Last Page” of each Academic Medicine issue presents a peer-reviewed snapshot of key educational ideas, projects, or concepts relevant to medical education.  The benefit of this approach is that seemingly cumbersome topics are made accessible. The two Last Pages referenced above present an approachable contrast in validity and reliability and in research paradigms, both wrapped up in 1-page digestible bites.

While reliability is increasingly being consumed under the title of validity, it still represents a key consideration for measurement and assessing it is a task worthy of consideration, especially as we put increasing emphasis on resident and student performance evaluations and knowledge or skill tests. Assessing validity is not as straightforward and, I would argue, achieved with a collection of evidence rather than a single number.

As we develop new educational innovations and ask ourselves and each other why, how and to what extent these innovations work, some comfort with research paradigms would serve us well. Thanks to Castillo-Page and her co-authors, this one-pager presents the difference in methodological approaches to education research across critical aspects including research design, data analysis, and, of course, reliability and validity.

Bottom Line:

These two 1-page articles are a concentrated dose of educational research basics. Ten minutes of your day is well served by ensuring that your definitions of these critical terms are fully developed and ready for use.