If there's one over-arching principle Mayor Michael Bloomberg has brought to city government, it's accountability through data. If you can measure something you can manage it better.
In 2007, the Bloomberg administration launched a new computer system for the city schools called ARIS, the Achievement Reporting Innovation System. The goal was to put information about test scores, attendance and student histories all in one place to help principals and teachers do a better job of reaching their students and improving performance. But the $80 million system hasn't yet achieved its full potential — even at schools that use it heavily.
At PS 10 in Brooklyn's Park Slope, 26 fifth graders sit at different tables in a classroom, likely grouped according to test scores on last year's state math exams.
"At the table closest to her (the teacher) there is a four sitting next to a three and across from a two and a one," said Principal Laura Scott, using numbers that represent the students' scores the exams. Level one is the lowest and four is the highest.
"So it seems like [the teacher] purposefully set up the table so that there's strength at one side and the strugglers at the other, and that they could refer to their peers whenever they needed assistance," Scott said.
Good teachers are constantly switching around their groups to give their struggling students more assistance while challenging those who are more advanced. This requires them to stay on top of their students' progress with great vigilance, and for generations they've done so with paper notebooks.
To help them digitize that information and use it more efficiently, the computer system ARIS combines different data sources in one Web-based computer network.
Deborah VanDoren, the fifth grade teachers who seated her children according to their math scores, demonstrated the program on a computer in the principal's office. She clicked on tab that read My Students, which then showed columns of names next to the individuals' color-coded performance on state math and reading tests.
"Red would indicate that they were far below standards, an orange would indicate that they’re approaching standards," VanDoren said, adding that green shows students who are meeting the standards and therefore on target.
Another column shows how many days each student was absent — pink indicates a child who was out too often in the previous school year. VanDoren can see her English Language Learners and special education students, too, and sort everyone by their test scores if she wants to group students according to who needs more attention in math or English.
In some ways, VanDoren said ARIS is like a fancy highlighter pen because any good teacher knows which students have trouble and which ones don’t. But VanDoren said the program is innovative in the way she can share ideas with her colleagues at PS 10, which has almost 800 students.
"So for example 'The Ticky Tacky Doll' by Cynthia Rylant," she said, referring to a children's book. "One of our teachers finds [it] useful for alliteration, similes, metaphors, so we can then click on it and create a link."
Whoever links on the book can then buy it immediately online or look for it in the school library.
The creators of ARIS envisioned this when they said their goal was to let data drive instruction. They wanted a teacher in Brooklyn who found a successful reading intervention to share that strategy with teachers all over the city. But while the teachers at PS 10 are pretty active, Chris Casal, the school's computer teacher and resident data expert, said ARIS still isn't very user friendly.
"I think it's great, I mean I’m a tech guy, I'm a data guy, I enjoy it," said Casal, who previously worked in education publishing. "But it also does sort of give me pause at multiple times from the teacher perspective."
Teachers complained from the start that they were left out of ARIS, even though the education department said it conducted focus groups. Training was rolled out slowly. And even today, the mere mention of ARIS can cause some teachers to roll their eyes over its $80 million price tag.
There's still a divide between the super-users, like Casal and VanDoren, and those who rarely check into ARIS. Casal suspects that's because some teachers find it difficult to post and read each other's recommendations.
"It has functionality and it has functionality in multiple places," Casal said, "meaning you go to say a resource and you can click on the resource to download it. But there's also a tab that's the resource tab. Teachers in general don't need to have that all going on. Have a button that says, 'Here's your resource, here's how to download it.'"
And it's even harder to share information outside your own school. That's why Casal and VanDoren call ARIS a skeleton. Principals and data geeks can use it to sort out information for an entire school or grade, but it doesn't have the details teachers want on a daily basis. Classroom quizzes, for example, never make it to ARIS.
To compensate, some schools have used their own money to buy separate data systems than can track daily and weekly updates of student learning, as well as periodic instead of daily attendance. One program called DataCation is very popular with city schools; other principals use a program called Edline. Schools, including PS 10, have also built their own websites for giving parents access to homework assignments, PTA meetings and other updates they want but don’t get onto ARIS.
Shael Polakow-Suransky, the school system's chief academic officer, said there are limitations so far.
"We've done a very big set of work to create an environment where lots and lots of data can flow," he said. "And we've created an initial set of applications that a lot of our teachers are using. We’ve seen a steady increase by both teachers, parents ever since we created ARIS. And so there’s evidence that it’s each month getting more and more usage."
Suransky said almost 165,000 parents have logged-in this year, and 51,000 teachers out of a total of about 75,000. He also said ARIS is now adding in more features. Like an iPhone, he said, it will soon accept new applications in respond to the demands of its customers.
"You can look at it as, as technology develops things that were developed two or three years ago begin to feel clunky," said Suransky. "Or you can look at it as how do you develop a strategy so that the tool can continually rise to the challenges and the needs that people are asking of it."
Suransky defended the $80 million price tag of ARIS because the city had to upgrade its aging data systems anyway. States and cities around the country have spent the past decade building data warehouses to comply with the No Child Left Behind law, which required them to track student achievement and break it down into separate groups such as blacks, Hispanics, and English Language Learners.
President Obama is now continuing that trend by giving grants to states, including New York, to build better data systems. New York State's system is modeled on ARIS.
If ARIS is struggling to keep up with changing demands of teachers, that's partly because it had a difficult roll-out. IBM had to turn over most of the work when it fell behind schedule. The project was handed over to a subcontractor, the Brooklyn-based Wireless Generation (now owned by Rupert Murdoch's News Corp, which also hired former Chancellor Joel Klein as a vice president for educational technology).
Sources said this wasted time and money. Plans to make the system more user-friendly for teachers to share information with one another got put on the back burner and so were plans to allow more regular updates of school-by-school data. But Suransky said ARIS should be able to add some of those features in by the fall with a feature called ARIS Local.
That's why as some teachers want more details from ARIS, the system is also still struggling with what you might call "customer buy-in."
Even at PS 10, where many teachers do use ARIS, fourth grade science teacher Theresa D'Ambrosi said she doesn't give it too much thought. She sees signs of her students' progress wherever she turns. The classroom is filled with charts from science experiments. D'Ambrosi has been teaching for 35 years and says she knows how to track her students.
"I can see the children that are responding and those who just sit back," she said. "And everyone participates because I will call on those children who aren’t responding, that are sitting there very meek and mild quietly. So everyone does participate."
Still, computer teacher Chris Casal said ARIS is gradually changing the culture of teaching because there's now a place to share information online.
"That's where it helps to drive students forward is that you're more easily able to remember all the mass amount of things that come our way," he said. "It's much easier to do it digitally than it is in different notebooks, or having to go to that other teacher and say, 'By the way what did you say the other day? I forgot.' You have the meeting, you post it on ARIS in the meetings space, boom it's there. You can go back to it any time. I think that is invaluable."
As ARIS becomes easier to use, the city expects teachers will become more comfortable with using data. Officials said the fact that many of them are already using websites other than ARIS for sorting details such as which periods each student attends and how they performed on weekly quizzes shows teachers do want the information.
Meanwhile, ARIS is used by the Department of Education to crank out annual report cards for schools based on how much progress their students make. And the city has begun using it for grading teachers. But it's taken longer than expected to give teachers that same degree of power with a system that’s all about student achievement.