excerpt from This Is a Test
A Handbook for Writing Good Tests
by Jan Gleiter

Introduction

There are other ways to assess students, employees, job applicants, and candidates for medical research besides giving them a traditional pencil-and-paper or computer-and-mouse test. In a school setting, these other ways are generally termed “alternative assessment” and include portfolios, interviews, journals, and learning logs, as well as that time-honored method known as “observation.” I'm going to assume you know about them. Despite the usefulness of these other methods, many people rely, for a variety of reasons, on pencil-and-paper tests. (I'm going to drop the “pencil-and-paper” modifier from here on out. For the rest of this handbook, when I use the word test , I don't mean performance test, laboratory test, lie detector test, or “If your mother and I both fell overboard, whom would you save first?” test. I mean pencil-and-paper test.)

In the real life that whirls around us, tests are almost nonexistent except in one's attempts to acquire various kinds of licenses. What people know and how well they know it is rarely determined through testing. It is determined by discussion and observation. Testing is an imperfect substitute for discussion and observation. All tests are imperfect substitutes.

How many times, in a conversation, have you asked a question, listened to a person begin to answer you, and then interrupted with, “No, I meant…”? This is not possible during a test.

How many times, in a conversation, have you asked a question, listened to the answer, and realized that you needed to follow up with another, more specific, question to get the information you were really after? This is not possible during a test.

How many times, in a conversation, have you asked a question based on an assumption, listened to the reply, and realized that your assumption was wrong? This is not possible during a test.

How many times have you wondered how good someone is at something and found out by watching? This is not possible during a test.

If teachers had time to carry on searching conversations with each student and observe those things that can't be discussed, they probably wouldn't use tests any more than parents, friends, and shift supervisors do. Time restrictions make this impossible and make testing necessary. The purpose of this handbook is to make tests less imperfect substitutes—to help those people, particularly teachers, who want or need to write tests to write better ones.

Why Give Tests at All?

There are many reasons to give tests. They may not be the best way to determine what, or how much, someone knows or can do, but they are often the best practical way. They are useful for answering questions you have about your students. How much do they know? Or, what skills have they mastered? Or, how closely have they listened to all those brilliant things you've been saying? Or, do they stand a chance of passing the state-mandated test you'll be administering in a few weeks? A good test will help you answer your question. A bad test may help you answer some other question, but you may never know what question that was.

In addition to helping you answer questions you have about your students, good tests actually help students learn. They encourage students to study the material they've been assigned. They provide a way for students to measure their own progress. And, if a test is reviewed after being returned to the class, with discussions about why the keyed (correct) answers are actually correct and the distractors (incorrect answers) are actually incorrect, it can serve as a solid review. Reviewing essay tests can also be helpful. Without identifying the students who wrote them, you could read aloud the best answers or several very good answers you received, identifying what was insightful or meaningful about them. (If you didn't receive any very good answers, perhaps you were asking for information or interpretations that your students didn't have the necessary basis for answering.)

Why Bad Tests Matter

If Hippocrates had been concerned with teachers instead of doctors, his oath would almost surely still have begun with “First, do no harm.” Bad tests do harm.

If the alternative to good tests were no tests at all, education would survive. Teachers would find other ways of measuring progress and mastery. Unfortunately, the alternative to good tests is, almost always, bad tests. And bad tests have bad consequences. The consequences range from the inconvenient and troublesome to the downright disastrous. For one thing, they tend to teach poor learning habits and make students feel that there is no point in studying or learning.

Bad tests differ from each other in innumerable ways, but they share one characteristic: they are invalid. They do not measure what they are intended to measure. As a result, they lead to erroneous conclusions.

Teachers who know that they don't know whether a student has mastered a skill or learned a subject will pay attention to what the student says and does. They will notice indications of mastery or lack of it. Teachers who think they know but are wrong are far less likely to notice such indications .

Teachers are likely to receive ready-made tests as part of program materials. If you know a particular test that you've been given to administer is a bad one, you could choose not to administer it unless it is a mandated test. Even in those situations in which you believe a test is flawed but must administer it, you can avoid basing your own teaching on the scores your students receive. If you don't know you're administering a bad test, face it, you're going to reach erroneous conclusions. And you will believe your conclusions are accurate.

In addition to leading teachers to erroneous conclusions, bad tests are confusing to those who take them. They can create panic, frustration, and cynicism in students, and these reactions can have significant and long-lasting effects. Students who have negative associations with testing tend to do poorly, even on good tests.

Children are not born with test anxiety. Tell any four-year-old that you want to find out how much he or she knows about something, and the child is almost certain to react with curiosity and eagerness. It's all just another game, and children tend to want to prove just how knowledgeable they are. However, if we give children enough bad tests, they are almost guaranteed to develop test anxiety, and test anxiety can have terrible consequences: lowered scores, more self-image problems, and less accurate expectations of how much they can learn.

In test development, there is no substitute for common sense. The rules for test writing provide a jumping off place; any test written in accordance with the rules will be better than any test not written in accordance with them. That does not mean that the result will be a good test.

For a test to be good, the writer must have engaged his or her brain. Fully. And he or she must have consistently paid attention to what is possible, reasonable, and fair. Are you testing what you think you're testing? Are you testing what you claim you're testing? Most important, if there is another, better way to determine what you want to know, forget the test and use that other method! If not, follow the fifteen rules in this book and write a good test.

Paperback

eBook