Let’s Talk Your Data, Not Big Data • Hacker News Board

Re: Let’s Talk Your Data, Not Big Data

#21

Post by kinleyd » Thu, Nov 15, 2012, 4:13 AM UTC

tumanian wrote: post #15

A former Yahoo big data engineer here. At some point I was bombarded with enquires from different startups who were looking for a "Big Data Guy". The conversation would go as: - (Me) So, how much data are we talking about? - (them) 50GB - Per hour? - No thats our dataset so far. - (pause) - (pause) (FIN)

Nice anecdote. :)

Re: Let’s Talk Your Data, Not Big Data

#22

Post by mark_integerdsv » Thu, Nov 15, 2012, 4:31 AM UTC

dbecker wrote: post #5

The wild confusion around "big data" stems in part from the fact that many people use it to mean something unrelated to data size. I was recently in a back-and-forth on twitter about this. Some people argued that "big data" refers to the complexity of the analysis or the value of the insight, rather than the size of the data. Kaggle CEO Anthony Goldbloom advocated for a definition "too big to fit in an excel spreadsh…

For me, and from an enterprise poit of view, Big Data has always been data external to the enterprise, i.e.: not stored in the warehouse but rather harvested from the web or other sources.

Re: Let’s Talk Your Data, Not Big Data

#23

Post by rdtsc » Thu, Nov 15, 2012, 5:18 AM UTC

I have my own definition of big data or maybe this is the next "after big data" and that is "streaming data".

Data being generated non stop at a high enough rate that it doesn't make sense to store it. You can only analyse, extra relevant statistics or some features and move on.

Storing it is just putting in a huge buffer and as new data comes in the old data falls of the end.

In some situations where products are up 24/7 in multiple time zones, there is no time for offline batch processing. By the time the batch has finished there is newer possibly bigger batch and so on.