Step One: Before Test Construction Begins
- Define learning outcomes to be assessed by the test questions
- Construct a test matrix or blueprint to track learning outcomes and multiple choice questions
Once you know the learning outcomes and item types you want to include in your test you should create a test matrix or blueprint. This consists of a matrix, or chart, representing the number of questions you want in your test within each topic and level of learning outcome. The blueprint identifies the outcomes and skills that are to be tested and the relative weight on the test given to each. The blueprint can help you ensure that you are obtaining the desired coverage of topics and level of outcome. Once you create your test blueprint you can begin writing your items.Example: 40 item exam (Adapted from Writing good multiple choice exams by Dawn Zimmaro, PhD.)
Learning Outcome Level Topic A Topic B Topic C Topic D TOTAL Remembering 1 2 1 1 5 (12.5%) Understanding 2 1 2 2 7 (17.5%) Applying 4 4 3 4 15 (37.5%) Analyzing 3 2 3 2 10 (25%) Evaluationg 1 1 2 (5%) Creating 1 1 (2.5%) TOTAL 10 (25%) 10 (25%) 10 (25%) 10 (25%) 40
- Plan for questions to address each learning outcome. The more questions the more reliable the information about student performance is likely to be.
Step Two: Prepare Item Stem.
Always end the item stem with a question. Do not ask a student to find an answer that completes an unfinished sentence.
- Be succinct by providing only those details that pertain directly to the question being asked.
- Clearly state the problem or question. Make sure that differing interpretations of the same question are not likely.
- Use positive wording. Avoid questions like “Which of the following treatments for condition X is likely not to be helpful in alleviating symptom Y?”
Step Three: Construct Alternatives.
- Make all alternatives roughly similar in length.
- Make sure there is correct grammar and spelling of each alternative choice.
- Make sure there is only one correct answer. Pilot test among a few colleagues to ensure this.
- Avoid extremes like “never, always, and only.”
- Avoid “all of the above” as an alternative.
- Make sure alternatives are mutually exclusive of one another.
- Do not adhere to a strict rule on the number of alternatives. Sometimes three or four are enough because coming up with more results in non-plausible distractors. However, the number of alternatives is positively correlated with the difficulty level of a test question. If you want to encourage students in a relatively low-stakes testing situation, make it easier by providing fewer alternatives from which to choose. If you want to minimize the possibility of guessing in a high-stakes situation, use five or even more alternatives.
Step Four: Avoid these mistakes.
- Don’t give the answer to a question in another question.
- Make sure to distribute keyed responses over all possible answer choices, i.e, strive to have approximately equal numbers of a’s, b’s, c’s and d’s as the keyed response.
- Try not to use distractors that are clearly not plausible, e.g, Bill Clinton as a distractor to the question “Who was the first president of the United States?”
Step Five: Follow-up after test is administered and scored
- Use item analysis to revise and weed out bad questions.