Assessment is a huge topic and it’s super important to teachers, students, and other stakeholders. One of the best books I’ve seen for explaining the fundamental concepts of testing in the context of language teaching is Testing for Language Teachers (India) by Arthur Hughes. The book clearly explains the need for reliability and validity in testing. Once you understand those concepts, you can apply them to one of the most challenging and complex areas of assessment – item writing.
Item writing – actually creating a test item is something every teacher should strive to do well. We owe it to our students. One useful way I’ve found to look at item writing is to recognize it as both an art and a science.
Ebel goes on to tell us that item writing “requires an uncommon combination of special abilities. It is mastered only through extensive and critically supervised practice. It demands, and tends to develop, high standards of quality and a sense of pride in craftsmanship. Item writing is essentially creative.”
All of this explains why item writing is so difficult. So, if there are no “set formulas,” how is it done? Well, there are some suggested principles for writing good items, especially for objective type items, like multiple-choice questions, but, often, they are just that, suggestions, or, at best, agreed upon standards, often untested standards.
In studies reporting comparisons between intuitive and statistical predictions, the score between algorithms and humans hasn’t changed in over 60 years: “60% of the studies have shown significantly better accuracy for the algorithms. The other comparisons scored a draw in accuracy, but a tie is tantamount to a win for the statistical rules, which are normally much less expensive to use than expert judgment. No exception has been convincingly documented” (Kahneman).
In other words, we can use statistical analysis to our advantage. For instance, the instinct, and often the suggestion, for writing multiple choice items says that more choices, meaning more distracters, is “better.”
And yet writing more distracters makes the job of item writing even more difficult. We are familiar, as students, with the multiple choice question where one choice is just silly and can be eliminated easily.
We can all see that four choices, where one is silly, is really three choices, so why not write just three? It turns out we can.
Kahneman, D. (2011). Thinking, fast and slow (India). New York: Farrar, Straus and Giroux. Kahneman’s fascinating book on how we think is a great buy in the Kindle version. Remember you don’t need a Kindle device to read Kindle books. Get the Kindle reader app for free.
Grove, W.M., & Lloyd, M. (2006). Meehl’s contribution to clinical versus statistical prediction. Journal of Abnormal Psychology, 115, 192–194
Rodriguez, M.C. (2005). Three options are optimal for multiple-choice items: A meta-analysis of 80 years of research. Educational Measurement: Issues and Practice, 24(2), 3-13. Read Rodriguez’s entire article on why three options are optimal for MCQs here.
HT to VidyaBLOG.com. An extended version of this article was also published in ELT Quarterly.