Beth Fertig is the contributing editor for education, covering the New York City public school system for WNYC on air and online at SchoolBook.org. She has covered education in the city for more than 15 years. Beth is the author of Why cant u teach me 2 read? Three Students and a Mayor Put Our Schools to the Test (FSG Books) which grew out of a radio series on the low graduation rate for special education students. Follow her @bethfertig.
Experts Question Evaluating Teachers with Student Test Scores
Saturday, October 23, 2010
With the teachers union contesting the city's plan to release the names of 12,000 teachers evaluated with student test scores, some academics question the value of such reports.
Educational economist Sean Corcoran, at New York University, studied the city's ratings last year when they were released without naming teachers. He says these "value-added" measurements, which seek to isolate a teacher's impact on student performance by comparing their students' scores on state exams, weren't very precise and had huge margins of error.
"A teacher's value-added measures changes dramatically from one year to the next, simply due to the composition of students that he or she may get in her class in a particular year," says Corcoran. "So if someone wants to learn something about a teacher's true effectiveness one has to look at that teacher over many years of classroom performance to really get a reliable estimate."
Corcoran noted that because the measurement for the value a teacher adds to student achievement is statistically estimated, there's always a margin of error. On New York City's teacher data reports, this is reported as a range of possible percentiles associated with the teacher's score. So a teacher at the 43rd percentile (meaning he's more effective than 43 percent of other teachers with similar kids) might have a margin of error that ranges from the 15th percentile to the 71st.
The scoring system also leaves out kids who weren't tested the previous year. There are also issues when kids move around from one school to another. That's why Corcoran says the system becomes more reliable over time, when the teacher has worked with many classes of kids.
Corcoran also notes that in any system that relies on percentiles, 50 percent of teachers will always fall in the "average" rating category. Another 40 percent will be "above average" or "below average." The remaining 10 percent are either "exceptional" (top 5 percent) or "failing" (bottom 5 percent).
But the city's Department of Education says it's mostly looking for which teachers are consistently rated at the top and bottom, when compared to others with similiar levels of experience and classes. Deputy Chancellor John White says the ranking is helpful especially for teachers "who year after year after year are in the 90th, 95th percentile. Or year after year after year are in the tenth, fifth percentile."
"It's not to say that the distinction of being in the 50th and 55th percentile tells us an extraordinary amount about that individual," he adds. "But where we see people year after year after year performing at a certain level, that tells us something very significant."
White's assertion, though, is challenged by others who have looked at value-added measurements of teachers. They've also been tried in Houston, where Corcoran says they fluctuated a great deal from one year to the next. A report released in August by the Economic Policy Institute in Washington also noted that while value-added measurements have become more sophisticated, they're still inaccurate. It cited one study of five large urban districts where most teachers who ranked in the top 20 percent of effectiveness in the first year fell the next year, with many of them hitting the bottom 40 percent. The EPI report concluded that test scores shouldn't dominate the information used by school officials in making high-stakes decisions about evaluating, discipining or compensating teachers.
Douglas Ready, an assistant professor at Teachers College-Columbia University, has looked at methodologies used in measuring teachers. Whie these value-added models and the tests on which they rely are getting better, he says, "the science behind these approaches is not yet to a point where I'm comfortable releasing teacher names. I think value-added approaches are very, very valuable as one element of a broader approach to evaluating teachers. But just as I wouldn't want a child to be evaluated by a single measure, I wouldn't want a teacher to be judged based on a single measure."
Although New York City says its system aims to include more than one single measure of a teacher's effectiveness, Ready worries that message will get lost in the recent media coverage. Which means that if the city's teacher evaluations are ultimately published with names, as the Los Angeles Times did in its own study of 6,000 teachers, we can expect a lot of of debate about their usefulness.
In the case of LA, data on teacher effectiveness was compiled by the newspaper based on records from the LA Unified School District. This time New York City is offering to release the names of teachers following requests from various media outlets. But when the New York Times, Daily News and researchers requested the reports last year, they were provided without the names of any teachers. The teachers union's argument for not releasing the names hinges in large part on an agreement it had with the city to fight any requests for them by the media.
Jeffrey Henig, a political science professor at Teachers College-Columbia University, says he's surprised by the city's sudden willingness to fully comply with Freedom of Information Act requests. "I'm struck by the fact that, if they do, they (at least the principals) will be placed in an incredibly difficult position when parents start waving these scores around and asking their child to be assigned to a different teacher," he said.
"It does seem to me, though, that this might be a way for the Department of Education to win some support among parent activists who have complained that the administration is indifferent to their concerns and at the same time create a bit of a wedge between those parents and the union, which itself has tried to enlist parents and community-based groups in support of its perspective."
Meanwhile, the teachers union might also be looking to score political points by fighting the release of the reports. It's been without a contract for almost a year, and its members are angry that the Bloomberg Administration wants to freeze their wages.
A judge has scheduled a hearing for November 24 on whether releasing the names of teachers violates the union's agreement with the city.