By Greg Chung

Tests. Ugh. Do I really need to test my learners? Isn’t it obvious that my instruction will lead to learning?  Isn’t testing a waste of time? These are all too common reactions to the idea of testing.  

Believe it or not, how to develop a good test is not obvious: How many times in your life have you taken a test and felt the multiple choice questions were terrible? Were there questions about content or concepts that weren’t covered in the training? Did the test have trick questions? Did it just test for rote memory? Were the instructions unclear? Or maybe the test was just plain stupid? Yeah, you know what I’m talking about.

In this article, I’ll be giving you a few tips on how to write better test items. I’ll focus on multiple-choice (MC) questions (or items) because MC items are one of the most common formats used by instructors. 

But before we go there, I want you to keep in the back of your mind the following questions. These questions can serve as the rationale for getting your boss to send you to conferences and training on instructional design: 

• If a program of instruction claims to be effective, but the test used is shoddy, what can you actually say about effectiveness?
• If you’re the stuckee instructional designer tasked to create a test, how are you going to know whether you are doing a good job? 

To get warmed up, let’s start with a test. If you received this item, what would you think of it? I hope you would be upset because this is a terrible item. 

But why is this item bad? The question is legitimate but the options are bad. This item violates established principles of good item design. Let’s take a look at each option.

(A) This option is legitimate.

(B) This is a “you problem” and reflects a misuse of tests. Tests intended to measure knowledge should focus on knowledge outcomes rather than classroom management. 

(C) All too common. The “A and B” option increases the chances of getting the item correct if you can eliminate (A) or (B). Suppose you knew (B) was not an option, then you can eliminate both (B) and (C) as options. 

(D) and (E). These are crowd-pleasers for novice item writers. Similar to (C), these option types should be avoided because if you know that any other option is true (or not true), then you’ve eliminated that option as well as the (D) or (E) option.

By simply avoiding the use of options like (C), (D), or (E), you have reduced the chances of someone who does not know much about the subject matter getting the item correct. The more you can reduce people guessing through your test, the more your test results will be meaningful, and the closer you will get to knowing the effectiveness of your instruction.

And this example item illustrates just one guideline. Fortunately, there are researchers who study test item design. Thomas Haladyna at Arizona State University has studied the design, development, and validation of tests and test items. Haladyna (2018) presents 20 guidelines related to item development and is very accessible. Haladyna and Rodriguez (2013) present a more comprehensive treatment of test item design.

References

Haladyna, T. M. (2018, June). Developing test items for course examinations (IDEA Paper #70). IDEA. https://www.ideaedu.org/idea_papers/developing-test-items-for-course-examinations/

Haladyna, T. M., & Rodriguez, M. C. (2013). Developing and validating test items. Routledge.

About the Writer

Greg is the Associate Director of Technology and Research Innovation of CRESST, a UCLA Adjunct Professor in the School of Education and Information Studies, and teaches undergraduate classes on field research at UCLA and a graduate class on assessment and evaluation at USC.

He has led research funded by the U.S. Department of Education, National Science Foundation, Office of Naval Research, USMC, DARPA, PBS KIDS, and a range of foundation and industry funders on the development and validation of computer-based games and simulations, assessments of learning processes and outcomes using sensors, web-based measures of problem-solving, concept mapping, and development and application of AI-based computational methods. He has a PhD in Educational Psychology from UCLA.

Greg is interested in measuring human learning and performance using technology. He focuses on validating inferences about learning from behavioral data, and is interested in process data–data that will tell him how people learn and solve problems. Greg used to be a software engineer and worked on satellites in a former life.