Big data: are we making a big mistake?
1–10 of 90 posts
Re: Big data: are we making a big mistake?
#2I get the impression from looking at local "big data" events that the enterprise software crowd has tuned into big data.
I fear that now that SOAP and enterprise buses have gone their way, they look a new buzzword to sell. More solutions looking for problems...
Re: Big data: are we making a big mistake?
#3"But while big data promise much to scientists, entrepreneurs and governments, they are doomed to disappoint us if we ignore some very familiar statistical lessons.
“There are a lot of small data problems that occur in big data,” says Spiegelhalter. “They don’t disappear because you’ve got lots of the stuff. They get worse.”"
This should be the main learning point. Humans can be astonishingly bad at dealing with stats and biases which can led to erroneous decisions being made. If you want an example where such decisions by very smart people can have catastrophic consequences, look up the Challenger disaster [1].
I rarely see people stating their assumptions upfront, which doesn't help the problem (I guess it's not cool to admit potential weaknesses). The more people/companies that get into 'big data' (without adequate training) the more false positives we're going to see.
[1] http://en.wikipedia.org/wiki/Space_Shuttle_Challenger_disast...
Re: Big data: are we making a big mistake?
#4Great article. I think the brightest gem here is the Multiple comparisons problem:
Re: Big data: are we making a big mistake?
#5Great article. I think the brightest gem here is the Multiple comparisons problem: http://en.wikipedia.org/wiki/Multiple_comparisons
Re: Big data: are we making a big mistake?
#6> a provocative essay published in Wired in 2008, “with enough data, the numbers speak for themselves”
I think that's indicative of Wired breathless enthusiasm for technology that turned my off buying the print version many years ago.
Scrape away some of the hyperbole and it is true that data driven management has made many companies more competitive and, if I dare mention the hobgoblin, efficient.
Hunches and ideas can only get you so far. It is important to visit the data gemba and do the genchi genbutsu.
Re: Big data: are we making a big mistake?
#7I'm much more impressed when someone can squeeze information out of small data. W.S Gosset was extracting tons of information from as little as two observations. I'm very grateful that my advisor guided my cohort to work with two-observation MLE in many contexts. This type of practice focuses the analyst on squeezing out as much information as possible. When applied to big data, this approach can be very useful. Big data comes with data wrangling challenges, but if you don't carefully squeeze out information, you'll be leaving tons and tons on the table.
Re: Big data: are we making a big mistake?
#8if i work for facebook and i want to figure out something about my users, isn't it safe to say N = All since the data im accessing is all user data from fb?
it's easy to go wrong with big data, and although the article glossed over some fairly important things (assuming the people who work on these datasets are much dumber than they are in reality), they're right on about idea that the scope and scale of what big data promises may be too grandiose for it's capabilities
Re: Big data: are we making a big mistake?
#9I find it amusing that the article talks about big mistakes in polling data, when the clear winner of the last two US elections is one Nate Silver, who aggregated polls to get predictions so close to the actual results, one wonders why people actually vote anymore.
Now, just like with every other technological solution, we only learn about the limits of its use by overuse. There's plenty of people out there storing large amounts of data and getting no valuable conclusions out of it. But the fact that many people will fail doesn't mean the concept is not worth pursuing.
Chasing what is cool is a pretty dangerous impulse. The trick is to be able to tell when it can pay off, and to quickly learn when it will not, and cut your losses. Maybe you don't need big data, just like maybe your shiny cutting edge library might not be ready for production.
Re: Big data: are we making a big mistake?
#10Great article. I think the brightest gem here is the Multiple comparisons problem: http://en.wikipedia.org/wiki/Multiple_comparisons
If we aren't careful the singularity AI will believe in God, and not necessarily us.