Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Big data: are we making a big mistake?

ft.com

Post reply on HN
Print view View original

81–90 of 90 posts

Re: Big data: are we making a big mistake?

#81

Post by zmmmmm »

JonLim wrote: post #14
nemesisj wrote: post #12

Another conclusion to draw from this article (which I really enjoyed, by the way) is that Big Data has been turned into one of the most abstract buzzwords ever. You thought "cloud" was bad? "Big Data" is far worse in its specificity. I can't count the number of times I'll be talking to some sales rep and they'll describe how they scan the data within whatever application they're demoing and "suggest" items using "big…

Out of curiosity, when does it effectively become "big data"? I ask not to be snarky, but it might be the case that it's "big data" to someone else, but not necessarily to you. I figured it was a relative term for your industry/business, but the hacker crowd definitely seems to peg that amount in the millions of data points before calling it big data at all. Seems fair, but I'd rather clarify.

I thought the definition in the article was actually really insightful: big data is when you start to behave as if you have N=all for a non-trivial sample.

Re: Big data: are we making a big mistake?

#82

Post by einhverfr »

JonLim wrote: post #14
nemesisj wrote: post #12

Another conclusion to draw from this article (which I really enjoyed, by the way) is that Big Data has been turned into one of the most abstract buzzwords ever. You thought "cloud" was bad? "Big Data" is far worse in its specificity. I can't count the number of times I'll be talking to some sales rep and they'll describe how they scan the data within whatever application they're demoing and "suggest" items using "big…

Out of curiosity, when does it effectively become "big data"? I ask not to be snarky, but it might be the case that it's "big data" to someone else, but not necessarily to you. I figured it was a relative term for your industry/business, but the hacker crowd definitely seems to peg that amount in the millions of data points before calling it big data at all. Seems fair, but I'd rather clarify.

The typical definition is where standard data management approaches do not work due to high volume, velocity, and/or variety of data sources.

What are standard data management approaches? I don't know. Usually they mean single machine relational db's.

But the thing is that once you get to a certain point on these three you need specialized solutions. High volumes of transactional data with real-time reporting might be handled well by something like Postgres-XC, but that won't handle data of sufficient variety. High velocity data may be best handled with something like VoltDB, but it can't handle volume. Etc....

Re: Big data: are we making a big mistake?

#83

Post by radmuzom »

JonLim wrote: post #14
nemesisj wrote: post #12

Another conclusion to draw from this article (which I really enjoyed, by the way) is that Big Data has been turned into one of the most abstract buzzwords ever. You thought "cloud" was bad? "Big Data" is far worse in its specificity. I can't count the number of times I'll be talking to some sales rep and they'll describe how they scan the data within whatever application they're demoing and "suggest" items using "big…

Out of curiosity, when does it effectively become "big data"? I ask not to be snarky, but it might be the case that it's "big data" to someone else, but not necessarily to you. I figured it was a relative term for your industry/business, but the hacker crowd definitely seems to peg that amount in the millions of data points before calling it big data at all. Seems fair, but I'd rather clarify.

I think "big data" is a term characterized more by the analytical techniques you apply on them rather than the size of the data. Traditional inferential statistical techniques work on "small data", while newer Bayesian techniques work on "big data" - note that this does not imply that one cannot work on the other.

Re: Big data: are we making a big mistake?

#84

Post by LambdaAlmighty »

nemesisj wrote: post #12

Another conclusion to draw from this article (which I really enjoyed, by the way) is that Big Data has been turned into one of the most abstract buzzwords ever. You thought "cloud" was bad? "Big Data" is far worse in its specificity. I can't count the number of times I'll be talking to some sales rep and they'll describe how they scan the data within whatever application they're demoing and "suggest" items using "big…

I say, make hay while the sun shines, and make money while businesses throw them at vague hyped keywords.

Such is the nature of primates: it's about social proofs and herd mentality.

Efficient rationalism would be a dangerous chimera if you're after money.

Re: Big data: are we making a big mistake?

#85

Post by sam_sach »

radmuzom wrote: post #83
JonLim wrote: post #14

Earlier quoted context omitted.

Out of curiosity, when does it effectively become "big data"? I ask not to be snarky, but it might be the case that it's "big data" to someone else, but not necessarily to you. I figured it was a relative term for your industry/business, but the hacker crowd definitely seems to peg that amount in the millions of data points before calling it big data at all. Seems fair, but I'd rather clarify.

I think "big data" is a term characterized more by the analytical techniques you apply on them rather than the size of the data. Traditional inferential statistical techniques work on "small data", while newer Bayesian techniques work on "big data" - note that this does not imply that one cannot work on the other.

Not necessarily. Try running basic descriptive statistics on terabyte scale data.

Re: Big data: are we making a big mistake?

#86

Post by sam_sach »

hnriot wrote: post #73
JonLim wrote: post #14

Earlier quoted context omitted.

Out of curiosity, when does it effectively become "big data"? I ask not to be snarky, but it might be the case that it's "big data" to someone else, but not necessarily to you. I figured it was a relative term for your industry/business, but the hacker crowd definitely seems to peg that amount in the millions of data points before calling it big data at all. Seems fair, but I'd rather clarify.

Big Data is less about size and more of a characterization. Human generated data can never be big data - there's just not enough humans to make it all. Possibly with the exception of the biggest social networks. Big Data is machine generated by systems. Typically its logs, IoT etc.

Not sure how you came to that conclusion. Humans generate voice data, which are then translated to a time series of frequencies for analysis.

Comment data on this site alone would be a pretty big task to analyze.

Re: Big data: are we making a big mistake?

#87

Post by sam_sach »

Conclusion: "Big Data" is a stupid buzzword and it makes me cringe every time I'm forced to say it to sell some new solution or frame something in a way someone who barely knows anything about computer science can understand.

It's nebulous. I've seen it applied to machine learning, data management, data transfer, etc. These are all things that existed long before the term, but bloggers just won't STFU about it. Businesses, systems, etc. generate data. If you don't analyze that data to test your hypotheses and theories, at the end of the day, you don't understand your own business and are relying on intuition for decision making.

Re: Big data: are we making a big mistake?

#89

Post by sam_sach »

twic wrote: post #28
JonLim wrote: post #14

Earlier quoted context omitted.

Out of curiosity, when does it effectively become "big data"? I ask not to be snarky, but it might be the case that it's "big data" to someone else, but not necessarily to you. I figured it was a relative term for your industry/business, but the hacker crowd definitely seems to peg that amount in the millions of data points before calling it big data at all. Seems fair, but I'd rather clarify.

I usually follow DevOps Borat's definition [1]: "Big Data is any thing which is crash Excel." Many a true word spoken in jest. [1] https://twitter.com/DEVOPS_BORAT/status/288698056470315008

Haha, perfect.
Post reply on HN