Mayor Michael R. Bloomberg and the schools chancellor, Dennis M. Walcott, often boast that student performance is improving in New York City, as evidenced by the percentage of students passing state exams and graduating from high school. But a new analysis finds that most city students are holding steady, getting very similar test scores between third and sixth grades.
The study by the city's Independent Budget Office looked at 46,400 students who were third graders in 2006 and tracked their performance on the state's English Language Arts exams through sixth grade. Nearly 62 percent ended up at the same proficiency level three years later.
"The primary finding is one of consistency," said Raymond Domanico, director of testing research for the budget office. "Generally, kids stayed at the same performance level relative to their grade over the three years of the study."
Mr. Domanico said 30 percent of the students improved, with half of them moving from Level 2 on the state exam to Level 3, which the state defines as being proficient. Another 8 percent dropped a level.
But the study does not provide a slam-dunk answer to the question of whether city students are doing better. For one thing, state tests only measure what students are supposed to know in each grade.
"The proficiency level cutoffs are arbitrary, and there's no way to compare a third-grade score to a fourth-grade score to assess growth in student knowledge over time," said Aaron Pallas, a professor of sociology and education at Teachers College, Columbia University.
The Education Department called the study "flawed."
"Testing experts know that performance levels on New York State tests cannot be compared from grade to grade without additional analysis, which this study failed to complete or consider," said a spokeswoman, Jessica Scaperotti.
The budget office's analysis differed from the city's evaluation of test scores in that it was a longitudinal study, meaning it followed the same group of students over time. The city normally looks at how third graders in one year, for example, performed compared with third graders in a previous year.
Test scores in the city increased steadily until 2010, when the state raised the cutoff needed to pass. In 2011, 43.9 percent of all students in third through eighth grades met or exceeded the state's standard for proficiency (Level 3 or Level 4) in English, compared with 42.4 percent in 2010. The state is expected to release this year's test scores next week.
The students included in the budget office's report are not representative of all children who started third grade in the 2005-2006 school year. The study only followed students who continued taking exams on grade level each year, leaving out those who were held back a year, along with some special education students and English learners who could not take the exams.
Still, Mr. Domanico said that performance levels were valuable because the city used them to rate its schools, and that the report offered a useful look at the 30 percent of students who did make progress. "It's better than having them go backwards," he said, noting that half of those students had moved from Level 2 to Level 3 by sixth grade. The percentage of students in the study who scored at the lowest level shrank dramatically during the years studied.
Sanabria Fleming, principal of Public School 13 in Brooklyn, said the findings were in line with many schools' goals.
"A lot of the schools put their emphasis on the Level 1s and 2s," Ms. Fleming said, referring to the lowest-performing students. She said her former school, P.S. 260, provided academic intervention services for these struggling students, along with enrichment services for students performing on grade level.
But the report found that students at the higher end of the performance scale did not make significant gains. Mr. Domanico said that 82 percent of the students who scored at Level 3 in third grade also scored at Level 3 in sixth grade. Almost 10 percent of them improved to Level 4, while almost 8 percent slid backward to Level 2.
The teachers' union president, Michael Mulgrew, said the study showed that the city's continued reliance on test preparation was not working.
"In the long run, it doesn’t even give you better test scores," Mr. Mulgrew said in a statement. "If the D.O.E. had given teachers a solid curriculum and the ability to really educate our kids, we would have seen a lot more growth in student learning.”
The report also found no evidence that the achievement gap between white and black students had narrowed, though it found a small decline in the gap between white and Hispanic students.
Dr. Pallas, the Teachers College professor, suggested that the National Assessment of Educational Progress was a better way of judging the performance of New York City students. Scores on the N.A.E.P. English exam increased somewhat last year for eighth graders but were flat for fourth graders.