Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Big data: are we making a big mistake?

ft.com

Post reply on HN
Print view View original

21–30 of 90 posts

Re: Big data: are we making a big mistake?

#21

Post by ElDiablo666 »

nemesisj wrote: post #12

Another conclusion to draw from this article (which I really enjoyed, by the way) is that Big Data has been turned into one of the most abstract buzzwords ever. You thought "cloud" was bad? "Big Data" is far worse in its specificity. I can't count the number of times I'll be talking to some sales rep and they'll describe how they scan the data within whatever application they're demoing and "suggest" items using "big…

Noam Chomsky had the best response to "big data": it's basically a nonsense concept (which I agree with) because "thinking is hard."

Re: Big data: are we making a big mistake?

#22

Post by hawkharris »

The misconceptions about big data are similar to those surrounding the word science.

Many people associate "science" with things: cells, microscopes, the inner workings of the body. But science isn't a set of things; it's a process, a method of thinking, that can be applied to any facet of life.

Big data is similar, in my opinion. It's not so much about the stuff —  the size or diversity of a company's datasets. It has more to do with the types of observations you're making and the statistical methods involved.

This distinction is important for two reasons:

1. If Big Data is recognized as a process rather than a circumstance, businesses will be more deliberate in deciding whether to use the methods. They will weigh the benefits of, say, MapReduce against other approaches.

2. The idea that "Big Data" techniques have everything to do with size is somewhat misleading. A comprehensive query of a 50,000 user dataset can be more computationally expensive than a simple operation on a 100,000-record dataset.

Re: Big data: are we making a big mistake?

#23

Post by ntoshev »

JonLim wrote: post #14
nemesisj wrote: post #12

Another conclusion to draw from this article (which I really enjoyed, by the way) is that Big Data has been turned into one of the most abstract buzzwords ever. You thought "cloud" was bad? "Big Data" is far worse in its specificity. I can't count the number of times I'll be talking to some sales rep and they'll describe how they scan the data within whatever application they're demoing and "suggest" items using "big…

Out of curiosity, when does it effectively become "big data"? I ask not to be snarky, but it might be the case that it's "big data" to someone else, but not necessarily to you. I figured it was a relative term for your industry/business, but the hacker crowd definitely seems to peg that amount in the millions of data points before calling it big data at all. Seems fair, but I'd rather clarify.

I believe the accepted definition for big data is data you can't handle on a single machine and need a cluster to process. So Moore's law makes it a moving threshold.

Re: Big data: are we making a big mistake?

#24

Post by stillsut »

What executives say it does...

"Facebook’s mission is to give people the power to share and make the world more open and connected."

What it actually does... (that will be left to the reader.)

"Big Data" is often sold as one thing by Enterprise software folks. But what value the data, or processing of it actually has is usually much more dependent on the user and his context (like FB!) and usually doesn't fit as nicely onto a PPT slide.

Articles like this usually confuse the PR definition and the analyst definition.

Re: Big data: are we making a big mistake?

#25

Post by rapala »

SworDsy wrote: post #8

if i work for facebook and i want to figure out something about my users, isn't it safe to say N = All since the data im accessing is all user data from fb? it's easy to go wrong with big data, and although the article glossed over some fairly important things (assuming the people who work on these datasets are much dumber than they are in reality), they're right on about idea that the scope and scale of what big dat…

At first I thought so too. But it's actually easy to come up with cases where N != all. As a radical example, Facebook preserves the accounts of dead users.

Re: Big data: are we making a big mistake?

#26

Post by Malarkey73 »

JonLim wrote: post #14
nemesisj wrote: post #12

Another conclusion to draw from this article (which I really enjoyed, by the way) is that Big Data has been turned into one of the most abstract buzzwords ever. You thought "cloud" was bad? "Big Data" is far worse in its specificity. I can't count the number of times I'll be talking to some sales rep and they'll describe how they scan the data within whatever application they're demoing and "suggest" items using "big…

Out of curiosity, when does it effectively become "big data"? I ask not to be snarky, but it might be the case that it's "big data" to someone else, but not necessarily to you. I figured it was a relative term for your industry/business, but the hacker crowd definitely seems to peg that amount in the millions of data points before calling it big data at all. Seems fair, but I'd rather clarify.

I think Hadley Wickham has a decent description of big data in terms of the analytical process... to expand slightly on his description:

On normal data you can iteratively explore and visualise it hitting return and seeing plots or model results instantaneously or at most a few seconds.

When you have time to grab a coffee after hitting return then you have bigger data.

If you carefully think through what you are about to ask the computer to do before pressing return then maybe you have big data.

I actually think this is a better description than just size of files or data distributed across many computers as an algorithm that just streams over a massive dataset maybe in parallel can be less challenging than one that has to hold a much smaller e.g many Gb dataset fully in memory.

Re: Big data: are we making a big mistake?

#27

Post by _delirium »

SworDsy wrote: post #8

if i work for facebook and i want to figure out something about my users, isn't it safe to say N = All since the data im accessing is all user data from fb? it's easy to go wrong with big data, and although the article glossed over some fairly important things (assuming the people who work on these datasets are much dumber than they are in reality), they're right on about idea that the scope and scale of what big dat…

Even if you have the full population in question and thereby avoid sampling issues, you still have a lot of pitfalls. For example if you just start correlating every variable against every other one and picking out ones that hit some test of statistical significances as "findings", you run into a range of familiar problems generally grouped under the pejorative term "data dredging".

Re: Big data: are we making a big mistake?

#28

Post by twic »

JonLim wrote: post #14
nemesisj wrote: post #12

Another conclusion to draw from this article (which I really enjoyed, by the way) is that Big Data has been turned into one of the most abstract buzzwords ever. You thought "cloud" was bad? "Big Data" is far worse in its specificity. I can't count the number of times I'll be talking to some sales rep and they'll describe how they scan the data within whatever application they're demoing and "suggest" items using "big…

Out of curiosity, when does it effectively become "big data"? I ask not to be snarky, but it might be the case that it's "big data" to someone else, but not necessarily to you. I figured it was a relative term for your industry/business, but the hacker crowd definitely seems to peg that amount in the millions of data points before calling it big data at all. Seems fair, but I'd rather clarify.

I usually follow DevOps Borat's definition [1]:

"Big Data is any thing which is crash Excel."

Many a true word spoken in jest.

[1] https://twitter.com/DEVOPS_BORAT/status/288698056470315008

Re: Big data: are we making a big mistake?

#29

Post by Houshalter »

ElDiablo666 wrote: post #21
nemesisj wrote: post #12

Another conclusion to draw from this article (which I really enjoyed, by the way) is that Big Data has been turned into one of the most abstract buzzwords ever. You thought "cloud" was bad? "Big Data" is far worse in its specificity. I can't count the number of times I'll be talking to some sales rep and they'll describe how they scan the data within whatever application they're demoing and "suggest" items using "big…

Noam Chomsky had the best response to "big data": it's basically a nonsense concept (which I agree with) because "thinking is hard."

Who said anything about thinking, and how do you know it's hard?

EDIT: I'm getting downvoted, but your statement is incredibly vague and I believe wrong. "Big Data" might be overused as a buzzword, but it's not a "nonsense concept". "Thinking is hard", I assume you are talking about strong AI, and it's not related to this at all. Saying it's "hard" adds nothing of value, and we don't even know if it's true (in the sense that when someone does figure it out, it might seem simple and obvious in retrospect.)

Re: Big data: are we making a big mistake?

#30

Post by beejiu »

JonLim wrote: post #14
nemesisj wrote: post #12

Another conclusion to draw from this article (which I really enjoyed, by the way) is that Big Data has been turned into one of the most abstract buzzwords ever. You thought "cloud" was bad? "Big Data" is far worse in its specificity. I can't count the number of times I'll be talking to some sales rep and they'll describe how they scan the data within whatever application they're demoing and "suggest" items using "big…

Out of curiosity, when does it effectively become "big data"? I ask not to be snarky, but it might be the case that it's "big data" to someone else, but not necessarily to you. I figured it was a relative term for your industry/business, but the hacker crowd definitely seems to peg that amount in the millions of data points before calling it big data at all. Seems fair, but I'd rather clarify.

When you are constrained to O(n) methods, you have big data.
Post reply on HN