The Age of Big Data

Last night I finally watched the movie Moneyball which I thought was terrific and succeeded in capturing the spirit of the 2003 book by the same name written by Michael Lewis.  Both describe the efforts of the 2002 Oakland Athletics baseball team to select players not by the traditional methods of old-time scouts ("he looks like a ballplayer") but through detailed analysis of statistics.  To me, the message of the book transcended baseball, saying that you can count and measure anything and that decisions should be made on the basis of quantitative evidence not gut instincts.  The secondary message of the book and the movie was that when you try to adopt such methods which happen to be counter to the established ways of doing things, you get tremendous push-back from those used to doing things the traditional way.  As the character playing Red Sox owner John Henry put it in the movie, "that makes these folks bat-s*** crazy because they see it as a threat to them."  How true.

A major story in the Opinion section of yesterday's New York Times picked up this theme and gave us a sort of "state of data analysis" piece ten years post Moneyball.  The profusion of data available today from the internet, social networks, sensors in vehicles and attached to packages and from an almost infinite variety of places would have been beyond the ability of someone ten years ago to comprehend.  All of this data now gives us 'a new approach to understanding the world and making decisions."  It also provides pretty good job opportunities.  The story estimates that nearly 200,000 employees with serious data-analysis skills will soon be needed and that there will be demand for another 1.5 million "data-literate managers."

The article also shares the view held by many that all of this data poses a substantial threat to our privacy.  I guess that's true, assuming the data is misused.  But there are great opportunities to become more efficient in our work and everyday lives and that's not necessarily a bad thing.