science-206x300

Creating MCQs? Three Choices is Enough!

Assessment is a huge topic and it’s super important to teachers, students, and other stakeholders. One of the best books I’ve seen for explaining the fundamental concepts of testing in the context of language teaching is Testing for Language Teachers (India) by Arthur Hughes. The book clearly explains the need for reliability and validity in testing. Once you understand those concepts, you can apply them to one of the most challenging and complex areas of assessment – item writing.

Item writing – actually creating a test item is something every teacher should strive to do well. We owe it to our students. One useful way I’ve found to look at item writing is to recognize it as both an art and a science.

“Item writing is an art. … Just as there can be no set formulas for producing a good story or a good painting, so there can be no set of rules that…guarantee… good test items.” (Ebel, 1951, as quoted in Rodriquez)

Ebel goes on to tell us that item writing “requires an uncommon combination of special abilities. It is mastered only through extensive and critically supervised practice. It demands, and tends to develop, high standards of quality and a sense of pride in craftsmanship. Item writing is essentially creative.”

All of this explains why item writing is so difficult. So, if there are no “set formulas,” how is it done? Well, there are some suggested principles for writing good items, especially for objective type items, like multiple-choice questions, but, often, they are just that, suggestions, or, at best, agreed upon standards, often untested standards.

artist-206x300

If test writing were truly art, it would be largely instinctual. Most teachers would probably not say that they are conducting testing in the same way they’d create a painting, but they probably would say they know their students and that they have the experience, or the instinct, to provide valid testing.
Unfortunately, if we want validity and reliability (and, as we see in works like Testing for Language Teachers, we certainly do), well, it turns out, our instincts are notoriously wrong.

In studies reporting comparisons between intuitive and statistical predictions, the score between algorithms and humans hasn’t changed in over 60 years: “60% of the studies have shown significantly better accuracy for the algorithms. The other comparisons scored a draw in accuracy, but a tie is tantamount to a win for the statistical rules, which are normally much less expensive to use than expert judgment. No exception has been convincingly documented” (Kahneman).

The takeaway from this should be threefold: a true appreciation for the complexity of the task, an understanding that careful craft is a necessity, a crucial appreciation for the limits of that craft.

In other words, we can use statistical analysis to our advantage. For instance, the instinct, and often the suggestion, for writing multiple choice items says that more choices, meaning more distracters, is “better.”

“Measurement specialists have been writing about the construction of multiple-choice items since the early 1900s…. However, even with this long tradition and attention to item writing, guidelines remain largely anecdotal….How many options should a multiple-choice item have? The advice as stated by most measurement textbook authors is to write as many options as feasible…” (Rodriquez).

And yet writing more distracters makes the job of item writing even more difficult. We are familiar, as students, with the multiple choice question where one choice is just silly and can be eliminated easily.

We can all see that four choices, where one is silly, is really three choices, so why not write just three? It turns out we can.

“Researchers have endorsed 3-option items for over 80 years with empirical evidence” (Rodriquez). Rodriquez shows that writing three is not only accurate, but it allows test makers to write more individual items which cover more content. More items over more content increases the validity and reliability in itself.

Kahneman, D. (2011). Thinking, fast and slow (India). New York: Farrar, Straus and Giroux. Kahneman’s fascinating book on how we think is a great buy in the Kindle version. Remember you don’t need a Kindle device to read Kindle books. Get the Kindle reader app for free.

Do you want to access U.S. content from outside the U.S.? A VPN can protect your privacy as you access the sites you need.

Grove, W.M., & Lloyd, M. (2006). Meehl’s contribution to clinical versus statistical prediction. Journal of Abnormal Psychology, 115, 192–194

Rodriguez, M.C. (2005). Three options are optimal for multiple-choice items: A meta-analysis of 80 years of research. Educational Measurement: Issues and Practice, 24(2), 3-13. Read Rodriguez’s entire article on why three options are optimal for MCQs here.

HT to VidyaBLOG.com. An extended version of this article was also published in ELT Quarterly.

Share on FacebookShare on Google+Tweet about this on TwitterShare on LinkedIn