Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Big data: are we making a big mistake?

ft.com

Post reply on HN
Print view View original

61–70 of 90 posts

Re: Big data: are we making a big mistake?

#61

Post by greenyoda »

mxfh wrote: post #35
hawkharris wrote: post #22

The misconceptions about big data are similar to those surrounding the word science. Many people associate "science" with things: cells, microscopes, the inner workings of the body. But science isn't a set of things; it's a process, a method of thinking, that can be applied to any facet of life. Big data is similar, in my opinion. It's not so much about the stuff — the size or diversity of a company's datasets. It ha…

It's the misconception that measurable observations equal the real distribution of the underlying events. Even professional data people often get that wrong, and it's not strictly limited to big data. One of the most obvious examples was this one: A data set of all known meteorite landings[1] turns into "Every meteorite fall on earth mapped" [2] with looks like a world population maps sprinkled with some deserts know…

You have the same problem with historical global temperature data: weather stations tend to be in or near populated areas, which excludes oceans (70% of the earth's surface area) and huge, sparsely populated regions like the Arctic, the Antarctic, deserts, rain forests, remote mountain ranges like the Himalayas and Andes, etc.

Re: Big data: are we making a big mistake?

#63

Post by [deleted] »

RogerL wrote: post #59
Malarkey73 wrote: post #26

Earlier quoted context omitted.

I think Hadley Wickham has a decent description of big data in terms of the analytical process... to expand slightly on his description: On normal data you can iteratively explore and visualise it hitting return and seeing plots or model results instantaneously or at most a few seconds. When you have time to grab a coffee after hitting return then you have bigger data. If you carefully think through what you are abou…

So my complicated algorithm that processes 200,000 data points is big data because it takes 1/2 hour to run, but someone else's petabyte algorithm that takes 1 second on a cluster is small to "bigger" data? I don't think this makes sense. It's a measure of the size of the problem to be sure, but it is not a measure of the size of the data, or an indication of what techniques might be required to solve the problem.

[deleted]

Re: Big data: are we making a big mistake?

#64

Post by Malarkey73 »

RogerL wrote: post #59
Malarkey73 wrote: post #26

Earlier quoted context omitted.

I think Hadley Wickham has a decent description of big data in terms of the analytical process... to expand slightly on his description: On normal data you can iteratively explore and visualise it hitting return and seeing plots or model results instantaneously or at most a few seconds. When you have time to grab a coffee after hitting return then you have bigger data. If you carefully think through what you are abou…

So my complicated algorithm that processes 200,000 data points is big data because it takes 1/2 hour to run, but someone else's petabyte algorithm that takes 1 second on a cluster is small to "bigger" data? I don't think this makes sense. It's a measure of the size of the problem to be sure, but it is not a measure of the size of the data, or an indication of what techniques might be required to solve the problem.

Yes to a data analyst that is big data. If you are doing some MCMC and that is really what it takes on that size of data then you have a big data problem.

The more sophisticated a statistic, the more high dimensional the data, the more sampling required, or the more of the dataset it requires to memorise at once - then the smaller your big data threshold will be.

It depends a lot on your point of view too. If I google something now it may bounce across lots of crazy server farms but to me I don't feel like I'm doing big data.. the person who built it all probably feels differently.

Re: Big data: are we making a big mistake?

#65

Post by SixSigma »

sireat wrote: post #16
SixSigma wrote: post #6

> a provocative essay published in Wired in 2008, “with enough data, the numbers speak for themselves” I think that's indicative of Wired breathless enthusiasm for technology that turned my off buying the print version many years ago. Scrape away some of the hyperbole and it is true that data driven management has made many companies more competitive and, if I dare mention the hobgoblin, efficient. Hunches and ideas…

I have some Wired issues from mid 90s in the bathroom and the tone is the same. It seems pretty much everything they write about is supposed to change the world in a major paradigm shift.

It delights in the techno-utopia envisaged by Nicholas Negroponte, personally I just can't be doing with it.

http://en.wikipedia.org/wiki/Nicholas_Negroponte

Re: Big data: are we making a big mistake?

#66

Post by emiliobumachar »

sitkack wrote: post #10
emiliobumachar wrote: post #4

Great article. I think the brightest gem here is the Multiple comparisons problem: http://en.wikipedia.org/wiki/Multiple_comparisons

If we aren't careful the singularity AI will believe in God, and not necessarily us.

I didn't get it. Would you please elaborate?

Re: Big data: are we making a big mistake?

#67

Post by owenversteeg »

nemesisj wrote: post #12

Another conclusion to draw from this article (which I really enjoyed, by the way) is that Big Data has been turned into one of the most abstract buzzwords ever. You thought "cloud" was bad? "Big Data" is far worse in its specificity. I can't count the number of times I'll be talking to some sales rep and they'll describe how they scan the data within whatever application they're demoing and "suggest" items using "big…

They only add their preface if the text is greater or equal to 185 characters, it seems.

Re: Big data: are we making a big mistake?

#68

Post by frozenport »

JonLim wrote: post #14
nemesisj wrote: post #12

Another conclusion to draw from this article (which I really enjoyed, by the way) is that Big Data has been turned into one of the most abstract buzzwords ever. You thought "cloud" was bad? "Big Data" is far worse in its specificity. I can't count the number of times I'll be talking to some sales rep and they'll describe how they scan the data within whatever application they're demoing and "suggest" items using "big…

Out of curiosity, when does it effectively become "big data"? I ask not to be snarky, but it might be the case that it's "big data" to someone else, but not necessarily to you. I figured it was a relative term for your industry/business, but the hacker crowd definitely seems to peg that amount in the millions of data points before calling it big data at all. Seems fair, but I'd rather clarify.

When you can't read all the data.

~(From a math professor I worked with)

Re: Big data: are we making a big mistake?

#69

Post by aaron695 »

Houshalter wrote: post #29
ElDiablo666 wrote: post #21

Earlier quoted context omitted.

Noam Chomsky had the best response to "big data": it's basically a nonsense concept (which I agree with) because "thinking is hard."

Who said anything about thinking, and how do you know it's hard? EDIT: I'm getting downvoted, but your statement is incredibly vague and I believe wrong. "Big Data" might be overused as a buzzword, but it's not a "nonsense concept". "Thinking is hard", I assume you are talking about strong AI, and it's not related to this at all. Saying it's "hard" adds nothing of value, and we don't even know if it's true (in the se…

Downvoted for mentioning downvoting but totally agree.

I see no connection between big data and AI. Like everything you can of course apply AI to it but I think step one is getting the analytic side down pat.

And also agree thinking may not be hard. It is hard to create a thinking machine (Other than using DNA) but I don't necessary think there is anything special to it actually thinking.

I'd be disappointed if Chomsky actually thought this way, would need context.

Re: Big data: are we making a big mistake?

#70

Post by gtrubetskoy »

ronaldx wrote: post #57
JonLim wrote: post #14

Earlier quoted context omitted.

Out of curiosity, when does it effectively become "big data"? I ask not to be snarky, but it might be the case that it's "big data" to someone else, but not necessarily to you. I figured it was a relative term for your industry/business, but the hacker crowd definitely seems to peg that amount in the millions of data points before calling it big data at all. Seems fair, but I'd rather clarify.

I appreciate the definition of Big Data as requiring >1 machine. However... Small Data, that which traditional researchers handle, is normally much, much smaller than that: perhaps 10-10000 data points (and most often on the small end of that). An experienced researcher can essentially can know everything about this data set, including its outliers and quirky points, and get a good sense of it by drawing out simple g…

I was just going to post the ">1 machine" definition when I saw your comment.

I think there are also at least two meanings of "Big Data". The more popular one is simply a trendy name for good old and boring "statistics", but with a twist that the data comes by way of the Internet, social media, all that.

The second one (and a little closer to my heart) is what ">1 machine" means from a developer/sysadmin perspective. This is where the hadoops, hives, cassandras, etc. come into play, and it's A LOT to learn, even for seasoned developers.

I think it's also a little intimidating for people who have become very comfortable with the typical rdbms stack. Parallel processing can be hard to understand, it's not something you can tinker with on your laptop over the weekend, and it's not surprising to hear all the "your big data thing is stupid" comments.

Post reply on HN