Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Big data: are we making a big mistake?

ft.com

Post reply on HN
Print view View original

41–50 of 90 posts

Re: Big data: are we making a big mistake?

#41

Post by Varcht »

hawkharris wrote: post #22

The misconceptions about big data are similar to those surrounding the word science. Many people associate "science" with things: cells, microscopes, the inner workings of the body. But science isn't a set of things; it's a process, a method of thinking, that can be applied to any facet of life. Big data is similar, in my opinion. It's not so much about the stuff — the size or diversity of a company's datasets. It ha…

Agreed, the buzzword 'Big Data' has nothing to do with the actual size of a given dataset except that it is about gathering as much data ( really metadata) as possible and finding novel ways to extract value from that data.

Re: Big data: are we making a big mistake?

#42

Post by bsbechtel »

There is definitely value to big data, but isn't it also a form of legitimizing stereotypes, at least in some cases? I mean, the general premise of big data, is to glean conclusions and new knowledge of the world from billions of records. When humans are the source of the data that is being extracted and analyzed, are the conclusions not stereotypes of those individuals, unless the correlation is 100%? This might be ok, and even useful, when trying to optimize clicks on ads, but what about when the government uses it to make policy decisions?

Re: Big data: are we making a big mistake?

#43

Post by shas3 »

sitkack wrote: post #13

This article reminds me of the argument [0] between Noam Chomsky [1] and Peter Norvig [2]. TL;DR (paraphrased with hyperbole) Chomsky claims the statistical AI of Norvig is a fancy sideshow that doesn't understand _why_ it is doing a thing. It just throws gigabytes of data at an ensemble and comes out with an answer. [0] - http://www.theatlantic.com/technology/archive/2012/11/noam-c... [1] - http://en.wikipedia.org/w…

Also relevant to this discussion is Douglas Hofstadter's solitary pursuit of 'thinking machines', outlined recently in this Atlantic profile: http://www.theatlantic.com/magazine/archive/2013/11/the-man-...

This analogy is particularly illuminating,

"“The quest for ‘artificial flight’ succeeded when the Wright brothers and others stopped imitating birds and started … learning about aerodynamics,” Stuart Russell and Peter Norvig write in their leading textbook, Artificial Intelligence: A Modern Approach. AI started working when it ditched humans as a model, because it ditched them. That’s the thrust of the analogy: Airplanes don’t flap their wings; why should computers think?"

While the Norvig-Chomsky debate is about the philosophy of the science of AI, it has practical implications to practitioners who tend to apply statistical techniques as if they are popping a pill. Engineers applying statistical learning, etc. should understand the limitations of the techniques, as outlined by Chomsky in the debate. The outcome of the Chomsky-Norvig (or Hofstadter vs. everyone else in CS) debate is less important than the arguments put forth by both the groups.

Re: Big data: are we making a big mistake?

#44

Post by coldtea »

Houshalter wrote: post #29
ElDiablo666 wrote: post #21

Earlier quoted context omitted.

Noam Chomsky had the best response to "big data": it's basically a nonsense concept (which I agree with) because "thinking is hard."

Who said anything about thinking, and how do you know it's hard? EDIT: I'm getting downvoted, but your statement is incredibly vague and I believe wrong. "Big Data" might be overused as a buzzword, but it's not a "nonsense concept". "Thinking is hard", I assume you are talking about strong AI, and it's not related to this at all. Saying it's "hard" adds nothing of value, and we don't even know if it's true (in the se…

>Saying it's "hard" adds nothing of value, and we don't even know if it's true (in the sense that when someone does figure it out, it might seem simple and obvious in retrospect.)

If it takes decades of hard working geniuses to figure it out, then even if "it seems simple and obvious in retrospect", it IS hard.

Re: Big data: are we making a big mistake?

#45

Post by [deleted] »

nemesisj wrote: post #12

Another conclusion to draw from this article (which I really enjoyed, by the way) is that Big Data has been turned into one of the most abstract buzzwords ever. You thought "cloud" was bad? "Big Data" is far worse in its specificity. I can't count the number of times I'll be talking to some sales rep and they'll describe how they scan the data within whatever application they're demoing and "suggest" items using "big…

[deleted]

Re: Big data: are we making a big mistake?

#46

Post by geocar »

beejiu wrote: post #30
JonLim wrote: post #14

Earlier quoted context omitted.

Out of curiosity, when does it effectively become "big data"? I ask not to be snarky, but it might be the case that it's "big data" to someone else, but not necessarily to you. I figured it was a relative term for your industry/business, but the hacker crowd definitely seems to peg that amount in the millions of data points before calling it big data at all. Seems fair, but I'd rather clarify.

When you are constrained to O(n) methods, you have big data.

I prefer this definition to others involving the size of memories, or number of computers, because it underscores the data rate instead of just its' (instantaneous) volume.

Re: Big data: are we making a big mistake?

#47

Post by nemetroid »

nemesisj wrote: post #12

Another conclusion to draw from this article (which I really enjoyed, by the way) is that Big Data has been turned into one of the most abstract buzzwords ever. You thought "cloud" was bad? "Big Data" is far worse in its specificity. I can't count the number of times I'll be talking to some sales rep and they'll describe how they scan the data within whatever application they're demoing and "suggest" items using "big…

Re: copying from FT, if you're using Firefox you can set dom.event.clipboardevents.enabled to false to get around that. Will probably break copying in some web apps.

Re: Big data: are we making a big mistake?

#48

Post by netcan »

nemesisj wrote: post #12

Another conclusion to draw from this article (which I really enjoyed, by the way) is that Big Data has been turned into one of the most abstract buzzwords ever. You thought "cloud" was bad? "Big Data" is far worse in its specificity. I can't count the number of times I'll be talking to some sales rep and they'll describe how they scan the data within whatever application they're demoing and "suggest" items using "big…

But, like "cloud" or "web 2.0" or any similar buzzword there obviously is some substance to it unspecific, un-novel, abused, or not. It just break into unsatisfying mush when you look at it to closely.

Web 2.0 was some sort of a shift over web 1.0, the line between publisher and consumer melted. Cloud is etherealizing computing and data. There was a thread a few days ago about the film Her. "Where is Samantha" (the AI) is borderline nonsensical statement. It doesn't come up to a viewer. That's because people are used to cloud as an idea now It doesn't really matter that servers, replication, dumb clients, remote data or whatever were invented a long time ago.

Re: Big data: are we making a big mistake?

#50

Post by entendre »

JonLim wrote: post #14
nemesisj wrote: post #12

Another conclusion to draw from this article (which I really enjoyed, by the way) is that Big Data has been turned into one of the most abstract buzzwords ever. You thought "cloud" was bad? "Big Data" is far worse in its specificity. I can't count the number of times I'll be talking to some sales rep and they'll describe how they scan the data within whatever application they're demoing and "suggest" items using "big…

Out of curiosity, when does it effectively become "big data"? I ask not to be snarky, but it might be the case that it's "big data" to someone else, but not necessarily to you. I figured it was a relative term for your industry/business, but the hacker crowd definitely seems to peg that amount in the millions of data points before calling it big data at all. Seems fair, but I'd rather clarify.

when you need statistical models and a tool for querying beyond the ken of an average sql dba
Post reply on HN