Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Big data: are we making a big mistake?

ft.com

Post reply on HN
Print view View original

71–80 of 90 posts

Re: Big data: are we making a big mistake?

#71

Post by rbanffy »

nemesisj wrote: post #12

Another conclusion to draw from this article (which I really enjoyed, by the way) is that Big Data has been turned into one of the most abstract buzzwords ever. You thought "cloud" was bad? "Big Data" is far worse in its specificity. I can't count the number of times I'll be talking to some sales rep and they'll describe how they scan the data within whatever application they're demoing and "suggest" items using "big…

Less than a month ago I was at an event about Big Data in marketing. One of the speakers spoke about how they used user data to improve their client's brand experience. If was very effective and I agree they did something extraordinary.

I then asked what tools they used. He responded with a well-known relational database. I then asked the total size of their dataset, with a good idea of what the upper bounds would be. He responded "around 100 million events" since the product started, 6 months ago.

It's really sad because they may end up under fire despite the effectiveness of their work.

Big Data is a lot like teen sex.

Re: Big data: are we making a big mistake?

#72

Post by salemh »

owenversteeg wrote: post #67
nemesisj wrote: post #12

Another conclusion to draw from this article (which I really enjoyed, by the way) is that Big Data has been turned into one of the most abstract buzzwords ever. You thought "cloud" was bad? "Big Data" is far worse in its specificity. I can't count the number of times I'll be talking to some sales rep and they'll describe how they scan the data within whatever application they're demoing and "suggest" items using "big…

They only add their preface if the text is greater or equal to 185 characters, it seems.

I believe Tynt pioneered this copy/paste: http://www.tynt.com/product_copypaste

http://daringfireball.net/2010/05/tynt_copy_paste_jerks

Re: Big data: are we making a big mistake?

#73

Post by hnriot »

JonLim wrote: post #14
nemesisj wrote: post #12

Another conclusion to draw from this article (which I really enjoyed, by the way) is that Big Data has been turned into one of the most abstract buzzwords ever. You thought "cloud" was bad? "Big Data" is far worse in its specificity. I can't count the number of times I'll be talking to some sales rep and they'll describe how they scan the data within whatever application they're demoing and "suggest" items using "big…

Out of curiosity, when does it effectively become "big data"? I ask not to be snarky, but it might be the case that it's "big data" to someone else, but not necessarily to you. I figured it was a relative term for your industry/business, but the hacker crowd definitely seems to peg that amount in the millions of data points before calling it big data at all. Seems fair, but I'd rather clarify.

Big Data is less about size and more of a characterization. Human generated data can never be big data - there's just not enough humans to make it all. Possibly with the exception of the biggest social networks.

Big Data is machine generated by systems. Typically its logs, IoT etc.

Re: Big data: are we making a big mistake?

#74

Post by pedrosorio »

Malarkey73 wrote: post #64
RogerL wrote: post #59

Earlier quoted context omitted.

So my complicated algorithm that processes 200,000 data points is big data because it takes 1/2 hour to run, but someone else's petabyte algorithm that takes 1 second on a cluster is small to "bigger" data? I don't think this makes sense. It's a measure of the size of the problem to be sure, but it is not a measure of the size of the data, or an indication of what techniques might be required to solve the problem.

Yes to a data analyst that is big data. If you are doing some MCMC and that is really what it takes on that size of data then you have a big data problem. The more sophisticated a statistic, the more high dimensional the data, the more sampling required, or the more of the dataset it requires to memorise at once - then the smaller your big data threshold will be. It depends a lot on your point of view too. If I googl…

I think defining "big data" as something that depends on the algorithm you are applying is not very useful. In that sense, almost anything is big data when you are trying to solve an NP-complete problem (nice course about algorithmic approaches to these problems: https://www.coursera.org/course/optimization).

The problem is, if you define "big data" as something that depends on the algorithm, then it makes no sense to include the word "data" in it. The expression "big data" as it is commonly used refers to flows of data so big that you need specialized approaches even when applying simple transformations to the data.

Since the dawn of computing we've always wanted to solve problems, with small or large amounts of data, that required complex algorithms. The usage of a new expression is justified by the fact that huge flows of data are now available to many companies (mainly because of the web), not because these companies are attempting to perform extremely complicated transformations to the data.

TL;DR: Big data means big volume/flow of data, and not (as you are defining it) using large Big-O complexity algorithms on some set of data. In fact, the size of big data precludes applying large Big-O complexity algorithms to said data.

Re: Big data: are we making a big mistake?

#76

Post by mark_integerdsv »

JonLim wrote: post #14
nemesisj wrote: post #12

Another conclusion to draw from this article (which I really enjoyed, by the way) is that Big Data has been turned into one of the most abstract buzzwords ever. You thought "cloud" was bad? "Big Data" is far worse in its specificity. I can't count the number of times I'll be talking to some sales rep and they'll describe how they scan the data within whatever application they're demoing and "suggest" items using "big…

Out of curiosity, when does it effectively become "big data"? I ask not to be snarky, but it might be the case that it's "big data" to someone else, but not necessarily to you. I figured it was a relative term for your industry/business, but the hacker crowd definitely seems to peg that amount in the millions of data points before calling it big data at all. Seems fair, but I'd rather clarify.

Fourteen years in IT, ten of them in BI... My definition is as follows: data pertaining to and generated by the source systems that govern various business processes are 'data' data (internal data owned by the business.)

Data pertaining (in whatever abstract sense) to the business, generated by systems outside of the business are 'big'(external data.)

Nothing to do with rowcounts directly IMHO.

Re: Big data: are we making a big mistake?

#77

Post by com2kid »

twic wrote: post #28
JonLim wrote: post #14

Earlier quoted context omitted.

Out of curiosity, when does it effectively become "big data"? I ask not to be snarky, but it might be the case that it's "big data" to someone else, but not necessarily to you. I figured it was a relative term for your industry/business, but the hacker crowd definitely seems to peg that amount in the millions of data points before calling it big data at all. Seems fair, but I'd rather clarify.

I usually follow DevOps Borat's definition [1]: "Big Data is any thing which is crash Excel." Many a true word spoken in jest. [1] https://twitter.com/DEVOPS_BORAT/status/288698056470315008

You underestimate Excel! You can point Excel to a DB table and use pivot tables on top.

I know it can at least get up to several million, didn't have a chance to test it beyond that! :)

Re: Big data: are we making a big mistake?

#79

Post by roel_v »

STORMSPLITTER wrote: post #38
nemesisj wrote: post #12

Another conclusion to draw from this article (which I really enjoyed, by the way) is that Big Data has been turned into one of the most abstract buzzwords ever. You thought "cloud" was bad? "Big Data" is far worse in its specificity. I can't count the number of times I'll be talking to some sales rep and they'll describe how they scan the data within whatever application they're demoing and "suggest" items using "big…

I enjoyed your post, nemesisj. Within your field of longitudinal patient data, if I am correct in what you have written, your large datasets simply have a new name, paranthetically Big Data, and that you could get what you need to save money without the newfangled algorithms. Within academic bioscience, I think there is great consensus on what Big Data is - I have not seen much argument at all; however, it is still v…

" This is my first post: )"

I really don't mean to be snarky, but please try to use paragraphs. It'll make it much more likely people will read your post.

Re: Big data: are we making a big mistake?

#80

Post by danieltillett »

rbanffy wrote: post #71
nemesisj wrote: post #12

Another conclusion to draw from this article (which I really enjoyed, by the way) is that Big Data has been turned into one of the most abstract buzzwords ever. You thought "cloud" was bad? "Big Data" is far worse in its specificity. I can't count the number of times I'll be talking to some sales rep and they'll describe how they scan the data within whatever application they're demoing and "suggest" items using "big…

Less than a month ago I was at an event about Big Data in marketing. One of the speakers spoke about how they used user data to improve their client's brand experience. If was very effective and I agree they did something extraordinary. I then asked what tools they used. He responded with a well-known relational database. I then asked the total size of their dataset, with a good idea of what the upper bounds would be…

>Big Data is a lot like teen sex.

LOL.

Post reply on HN