Sometimes the most powerful and transformative technologies emerge by accident, an unintended consequence of other developments. When this happens, the scope and power of the new technology can't be fully appreciated until after we have embedded it in our culture.
Big Data is all that and much, much more.
By now we all recognize the many revolutions of the Internet. But who could have known back in the 1960s that getting a few hulking mainframe computers to swap digital spit could lead to the Facebooked/Googled/e-banking/YouTubed world we inhabit today? But as head spinning as it all has been, simmering within the Internet upheaval bubbles another possibility that has slowly been taking shape. For now, it's called Big Data. If it lives up to its promise (or peril), it will rework the architecture of human experience in ways we simply cannot imagine.
And because our urban centers have always been engines of information, there is likely no nexus of human culture more susceptible to Big Data's hurricane winds than Big Cities.
By now, of course, you may be wondering if there's really something going on or if Big Data is just this year's overheated hype. The answer to that question is a definitive NOT HYPE and the reason can be summed up in two words: Digital Breadcrumbs.
For years now we have all been dropping digital breadcrumbs — electronic markers in 1s and 0s — spread across the wired world. From cell-phone locations to grocery store shopping choices to Facebook posts, we are leaving a record of our life that is out there to be followed by anyone with the resources and the time.
And it's not just us.
Every function of our culture is generating reams of numbers that flow into the data sphere: from the monthly billing records of public utilities to the traffic data recorded by municipal street sensors, it's all getting recorded and most of it is getting electronically archived.
If you want a physical representation of Big Data, consider this: to store all the information humanity created in just one year you'd need 80 billion 16-GB iPhones. That's enough iPhones to create a ring circling the Earth 100 times.
The premise of Big Data is deceptively simple: hidden in all that information lies a hyper-resolution map of the world's behavior in space and time. It's a representation of human life and the natural world with a fidelity we have never had before. Think of those movies where a character can stop time and then walk around poking at people and objects frozen in action. Now give that character X-ray microscopic eyes and you begin to get a feel for what Big Data allows.
The science behind Big Data lies in learning how to dive into its digital oceans to find patterns in the real world. Those patterns are the key. From the movie-renting habits of 28-year-old factory workers to the daily flow of stock trades in companies processing salmon, those patterns represent the contours of real life captured in numbers. Once you see the patterns you can understand the world's behavior. Once you understand behavior you can predict it. Once you can predict behavior you can control it. That is the true promise — and danger — of Big Data.
Cities are created human environments. They are ecosystems of energy and matter imagined into existence through human effort. Because cities are essentially ideas transformed into action, they are creatures of information and a Big Data problem. By breathing in the torrents of data cities generate every second, Big Data scientists and engineers believe they can make cities efficient, effective and responsive to human needs in ways that will reshape their very nature.
In the most ambitious vision, the Big Data of Big Cities will mean these dense hubs of human habitation, where 85 percent of all people will live by 2050, might become adaptive, almost self-aware. Given the need to create a sustainable global human culture on a finite planet with finite resources, some say the Big Data revolution can't come fast enough for Big Cities.
Examples of Big Data/Big Cities projects are everywhere as researchers, engineers and municipal planners struggle to put the rivers of data being generated to good use on issues like sustainability, security and public health.
Consider that most basic aspect of urban life for a moment: traffic. These days a trip in a car from one side of town to another would not be complete without a quick peek at Google Maps and its traffic data. Google (and a host of other platforms, like Waze) gives you a map of the city and a representation of the traffic flow (green for good, red for "you're screwed"). But how does Google get this data. To get a better real time picture of traffic flow, Google recently began using a crowd-sourcing model where people's smart phones become sensors. Smart phone locations are tracked by cell-phone companies and that gives a measure of how traffic is (or is not) flowing. So the same people who are using Google's map application may be sending Google traffic data to inform the map. Any city — or in this case Google — can buy that data and use it to monitor traffic.
But you can do more than just track traffic. You might take that traffic data and use it to find patterns that are shaping the life of your city. You could, for example, combine the traffic data with other municipal datasets to see how traffic patterns correlate with, say, the electricity use of households in neighborhoods, or the rates of heart attacks or the shopping patterns of the 18-28 year olds. All that data is out there to be mined. Within it lies the secret life of our cities.
If you're looking for the unexpected in Big Data/Big City projects you need go no further than social media. At my own school, the University of Rochester, computer scientist Henry Kuatz and his student Adam Sadilek are "mining" twitter data to track the spread of disease. Normally, the way a city gathers data about who is sick — say, how many people have the flu during flu season — is to wait for folks to come in to a hospital or to their doctor's office. So people get sick, they report it, and the numbers eventually get tallied up. But that analysis is pretty backward looking, revealing only how many people have been sick last week or last month.
Kuatz and his collaborators can search through tweets containing the words "I feel sick" in specific urban areas as they appear now. By intelligently manipulating the constraints on their data-mining, Kautz's team are trying to see the spread of the flu from one neighborhood to another by watching the degree that digital social networks map out real networks of human contact and its contagious consequences.
These kinds of Big Data uses of social media are just beginning and there will be lots of mistakes made. But, taken as a whole, they hold enormous promise. Perhaps in the future you'll get an automatic tweet telling you that yesterday you were in a room with someone who had the flu. "Drink extra fluids and get some more sleep just in case."
For all the promise of Big Data and Big Cities it's not hard to imagine the dark side. All these digital breadcrumbs we are leaving may very well leave us without a shred of privacy. And while Big Data may allow a kind hyper-subtle control over a city's functions for benign uses, like developing sustainability, that control can also be used to reign in the functions that make a vital democracy work.
Regardless, we and our big cities are entering the era of Big Data. What comes next will be what we make of it and take from it.