Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Big data: are we making a big mistake?

ft.com

Post reply on HN
Print view View original

51–60 of 90 posts

Re: Big data: are we making a big mistake?

#51

Post by dannypgh »

xroche wrote: post #34
twic wrote: post #28

Earlier quoted context omitted.

I usually follow DevOps Borat's definition [1]: "Big Data is any thing which is crash Excel." Many a true word spoken in jest. [1] https://twitter.com/DEVOPS_BORAT/status/288698056470315008

This is very inaccurate/misleading IMHO. Big Data is something which does not fit in a regular machine for a given operation. You can sort billions of records on an iPhone, for example. You can grep a string within a terabyte-file data on a single personal computer, and I am not convinced you'd go faster with a distributed system (reading the file on cold storage will be the limiting factor). People claiming to do "b…

With a distributed storage system you should be able to read said terabyte file using far more disk heads.

It would also be easier to engineer it so the terabyte file was entirely in RAM by distributing it across multiple machines (although single machines with TB ram capacity are no doubt continuing to become more common)

Sure, store it on a single tape or disk and distributing the computation won't help. You need distributed storage to properly leverage distributed computation for otherwise I/O bound processes.

Re: Big data: are we making a big mistake?

#52

Post by [deleted] »

beejiu wrote: post #30
JonLim wrote: post #14

Earlier quoted context omitted.

Out of curiosity, when does it effectively become "big data"? I ask not to be snarky, but it might be the case that it's "big data" to someone else, but not necessarily to you. I figured it was a relative term for your industry/business, but the hacker crowd definitely seems to peg that amount in the millions of data points before calling it big data at all. Seems fair, but I'd rather clarify.

When you are constrained to O(n) methods, you have big data.

[deleted]

Re: Big data: are we making a big mistake?

#53

Post by akadien »

This is my favorite line and the one that damns so many "big data" efforts:

"They cared about ­correlation rather than causation."

Analytics are a tool to help find correlations and patterns so that humans can do the hard work of determining and testing for causation. Computers are doing their jobs; humans aren't.

Re: Big data: are we making a big mistake?

#54

Post by linuxhansl »

Any either-or discussion is doomed to fail. Saying that BigData is the end of theory is clearly nonsense.

BigData vs. Theory, Java vs. C++, Capitalism vs. Socialism, Industry vs. Nature, Good vs. Bad, etc.

BigData allows to store a lot of data and provides a means run some computation on that data. Not more, and not less.

Re: Big data: are we making a big mistake?

#56

Post by joe_the_user »

shas3 wrote: post #43
sitkack wrote: post #13

This article reminds me of the argument [0] between Noam Chomsky [1] and Peter Norvig [2]. TL;DR (paraphrased with hyperbole) Chomsky claims the statistical AI of Norvig is a fancy sideshow that doesn't understand _why_ it is doing a thing. It just throws gigabytes of data at an ensemble and comes out with an answer. [0] - http://www.theatlantic.com/technology/archive/2012/11/noam-c... [1] - http://en.wikipedia.org/w…

Also relevant to this discussion is Douglas Hofstadter's solitary pursuit of 'thinking machines', outlined recently in this Atlantic profile: http://www.theatlantic.com/magazine/archive/2013/11/the-man-... This analogy is particularly illuminating, "“The quest for ‘artificial flight’ succeeded when the Wright brothers and others stopped imitating birds and started … learning about aerodynamics,” Stuart Russell and Pe…

The problem with the analogical comparison between the tupples [birds, airplanes, flight] and [humans, AI-machines, intelligence] is that flight is a clear and ambiguous achievement whereas intelligence is something we haven't fully defined and for-which humans, we ourselves are our only accepted model (and self-interrogation is an activity that can feel easy but in-which we found many subtle and obvious problem).

Re: Big data: are we making a big mistake?

#57

Post by ronaldx »

JonLim wrote: post #14
nemesisj wrote: post #12

Another conclusion to draw from this article (which I really enjoyed, by the way) is that Big Data has been turned into one of the most abstract buzzwords ever. You thought "cloud" was bad? "Big Data" is far worse in its specificity. I can't count the number of times I'll be talking to some sales rep and they'll describe how they scan the data within whatever application they're demoing and "suggest" items using "big…

Out of curiosity, when does it effectively become "big data"? I ask not to be snarky, but it might be the case that it's "big data" to someone else, but not necessarily to you. I figured it was a relative term for your industry/business, but the hacker crowd definitely seems to peg that amount in the millions of data points before calling it big data at all. Seems fair, but I'd rather clarify.

I appreciate the definition of Big Data as requiring >1 machine.

However... Small Data, that which traditional researchers handle, is normally much, much smaller than that: perhaps 10-10000 data points (and most often on the small end of that). An experienced researcher can essentially can know everything about this data set, including its outliers and quirky points, and get a good sense of it by drawing out simple graphs.

There is clearly some disconnect between these two ideas: is that "Medium Data"?

I would accept a concept of "Big Data" as data that cannot easily be eyeballed to get a sense of what's going on, so 10000+ points would count (under some circumstances). Maybe the concept of "six sigma" is useful - enough data that you would reasonably expect a six sigma outlier.

Mathematically/statistically, the storage limit is not a particularly important milestone: the ideas and methods don't change once you reach this scale (except for potential parallelisation).

Re: Big data: are we making a big mistake?

#59

Post by RogerL »

Malarkey73 wrote: post #26
JonLim wrote: post #14

Earlier quoted context omitted.

Out of curiosity, when does it effectively become "big data"? I ask not to be snarky, but it might be the case that it's "big data" to someone else, but not necessarily to you. I figured it was a relative term for your industry/business, but the hacker crowd definitely seems to peg that amount in the millions of data points before calling it big data at all. Seems fair, but I'd rather clarify.

I think Hadley Wickham has a decent description of big data in terms of the analytical process... to expand slightly on his description: On normal data you can iteratively explore and visualise it hitting return and seeing plots or model results instantaneously or at most a few seconds. When you have time to grab a coffee after hitting return then you have bigger data. If you carefully think through what you are abou…

So my complicated algorithm that processes 200,000 data points is big data because it takes 1/2 hour to run, but someone else's petabyte algorithm that takes 1 second on a cluster is small to "bigger" data? I don't think this makes sense.

It's a measure of the size of the problem to be sure, but it is not a measure of the size of the data, or an indication of what techniques might be required to solve the problem.

Re: Big data: are we making a big mistake?

#60

Post by greenyoda »

nemetroid wrote: post #47
nemesisj wrote: post #12

Another conclusion to draw from this article (which I really enjoyed, by the way) is that Big Data has been turned into one of the most abstract buzzwords ever. You thought "cloud" was bad? "Big Data" is far worse in its specificity. I can't count the number of times I'll be talking to some sales rep and they'll describe how they scan the data within whatever application they're demoing and "suggest" items using "big…

Re: copying from FT, if you're using Firefox you can set dom.event.clipboardevents.enabled to false to get around that. Will probably break copying in some web apps.

NoScript also solves the problem, since they need to run JavaScript code to disable copying. (Their site displays just fine without JavaScript.)
Post reply on HN