Can Computers Give Good Grades?
If you've been a student or teacher in the past 20 years or so, chances are you have experience with tests that are 'graded' by computers. We trust computers to give proper grades on tests like this because of the inherent objectivity when there is always only one correct answer to a question. But would you trust a computer to give grades on essays and other test components that have more subjective answers?
Computer-graded multiple-choice tests are the norm at all levels of education these days. The concern with these tests is usually with using the right writing utensil (No. 2 pencils only!) and 'bubbling in' the answer choices. Students and teachers typically don't worry about the computer making mistakes like marking correct answers as wrong. Some of the most important tests we take, like the SAT and even driver's license tests, use computers to determine what answers are right and wrong.
Tests with both multiple choice and written essay portions aren't completely evaluated by machine. The essay sections are read and scored by people, usually teachers or other educated individuals who have been specifically trained in the metrics of a certain test. Correct responses for essay exams are seen as more flexible, because while the grader may not agree with the substance of a student's argument, it's usually how they make that argument - or the quality of their writing - that is the focus.
A new computerized system from the Educational Testing Service (ETS) seeks to change this. ETS developed a new program to evaluate writing, and tested it on freshman writing placement exams at the New Jersey Institute of Technology. According to the company, the results produced by its grading program produced accurate results. The grades produced by the computerized system were compared to those given by human evaluators. Based on the similarity of the evaluations, ETS declared the program a success. Plans to use the program in other writing evaluations are already being rolled out.
What's At Stake?
While ETS is confident that its technology works, critics aren't so sure. One researcher at the Massachusetts Institute of Technology (MIT) criticized the computerized system for valuing wordiness more than creativity. This is a critique that many seasoned essay graders are likely to sympathize with - students may rely too heavily on 'five-dollar words' to spice up their writing, rather than actually producing thoughtful, original content. It stands to reason that a computer would only be able to use something measurable, like the complexity of words, to determine writing quality. Judging more subjective, difficult-to-define concepts like ingenuity and style aren't really a computer's strong suit.
However, supporters of computerized essay grading can easily argue that it is precisely this subjective quality to 'good' writing that makes the new program so necessary. An ETS researcher noted that human graders can be inconsistent with their grading, even if there is a rubric, and that machine grading is more likely to produce reliable, fair results. Supporters of the system also note that students may feel more comfortable taking writing critiques from a computer. After all, if a computer tells you that you need to improve, it's difficult to write those comments off as a personal attack.
Whether students will prefer to be graded by a computer remains to be seen. It seems possible that opinions on this matter will be divided, depending on personal preference. However, it seems likely that more computerized essay grading will become the norm. Technology is considered reliable to evaluate other areas of standardized testing, and it doesn't seem too far-fetched that discomfort with ETS' system will be written off as growing pains. Is it possible critics of machine-graded writing will come up with a compelling argument to halt this process?