Let’s Talk Your Data, Not Big Data
wired.com
Let’s Talk Your Data, Not Big Data
1–10 of 23 posts
Re: Let’s Talk Your Data, Not Big Data
#2"The paper concludes by advising analysts to not go through the Hadoop hoops until your data size passes standard hard drive limits (currently around 1 Terabyte) or at least reasonable memory limits (512 GB)."
Is a TB really considered Big currently?
Re: Let’s Talk Your Data, Not Big Data
#3"The paper concludes by advising analysts to not go through the Hadoop hoops until your data size passes standard hard drive limits (currently around 1 Terabyte) or at least reasonable memory limits (512 GB)." Is a TB really considered Big currently?
Its hard to put a handle on what is 'Big' and what isn't. I think a decent place to draw the line is on data that can't fit on one disk, and 1 TB seems to be a pretty standard disk size right now.
Re: Let’s Talk Your Data, Not Big Data
#4"The paper concludes by advising analysts to not go through the Hadoop hoops until your data size passes standard hard drive limits (currently around 1 Terabyte) or at least reasonable memory limits (512 GB)." Is a TB really considered Big currently?
I would say big enough to start thinking about alternate solutions than a monolithic RDBMS. Of course the solutions may include sharding and not necessarily a non-relational store.
Re: Let’s Talk Your Data, Not Big Data
#5The wild confusion around "big data" stems in part from the fact that many people use it to mean something unrelated to data size.
I was recently in a back-and-forth on twitter about this. Some people argued that "big data" refers to the complexity of the analysis or the value of the insight, rather than the size of the data.
Kaggle CEO Anthony Goldbloom advocated for a definition "too big to fit in an excel spreadsheet."
I advocated for for a definition "large enough that the storage and manipulation becomes part of the challenge (in addition to the analysis).
The phrase has taken on so many definitions as to become meaningless.
Re: Let’s Talk Your Data, Not Big Data
#6"The paper concludes by advising analysts to not go through the Hadoop hoops until your data size passes standard hard drive limits (currently around 1 Terabyte) or at least reasonable memory limits (512 GB)." Is a TB really considered Big currently?
Unless you've got a ridiculous RAID array, there's plenty of times where processing anything over 100GB without a cluster is going to suck. Sequentially reading 1TB of data is going to take a few hours even with a RAID array, or be very expensive with SSDs.
I maintain a database that's over 1TB and Oracle handles it very well. The trick there is understanding that, under absolutely no circumstance should you ever need to do a full table scan, because the table isn't designed for that.
So, I'd argue that a monolithic database is okay even up to 10TB even with slower disks as long as you never need to touch more than 10% of it. If you need to touch 100% of the data 100% of the time, I'd say anything over 100GB is too big for one machine.
The reality is that it just depends. There's times where you're going to want a hadoop cluster even for only 16GB of data, and there's going to be times where a database is going to be fine with 10TB of data.
Re: Let’s Talk Your Data, Not Big Data
#7The wild confusion around "big data" stems in part from the fact that many people use it to mean something unrelated to data size. I was recently in a back-and-forth on twitter about this. Some people argued that "big data" refers to the complexity of the analysis or the value of the insight, rather than the size of the data. Kaggle CEO Anthony Goldbloom advocated for a definition "too big to fit in an excel spreadsh…
I argue for your definition:
"Big data is when the size of the data becomes part of the problem."
Mike Loukides (O'Reilly) mentioned to me that Roger Magoulas (O'Reilly) was the first he heard using that definition.
According to this definition, physicists in the 80's were doing big data.
Re: Let’s Talk Your Data, Not Big Data
#8The wild confusion around "big data" stems in part from the fact that many people use it to mean something unrelated to data size. I was recently in a back-and-forth on twitter about this. Some people argued that "big data" refers to the complexity of the analysis or the value of the insight, rather than the size of the data. Kaggle CEO Anthony Goldbloom advocated for a definition "too big to fit in an excel spreadsh…
Yes I think sheer size is actually just a small part of "big data". The number of dimensions/facets to the data is more important...don't know if that's the correct term, but I mean the ability to, given the schema of your vistor records, analyze which browser has the highest change in bounce-rate given a certain kind of design change in your site. The old-fashioned visitor record type...the number of database pings per day...would not have this much dimension for analysis.
Another part of "big data": the ability to gather, clean, and organize it in a normalized way.
Re: Let’s Talk Your Data, Not Big Data
#9The wild confusion around "big data" stems in part from the fact that many people use it to mean something unrelated to data size. I was recently in a back-and-forth on twitter about this. Some people argued that "big data" refers to the complexity of the analysis or the value of the insight, rather than the size of the data. Kaggle CEO Anthony Goldbloom advocated for a definition "too big to fit in an excel spreadsh…
"large enough that the [storage] and manipulation becomes part of the challenge"
-- Also: [acquisition] (not/just storage). the genius of Google and FB. they [create] massive, usable data sets.
Re: Let’s Talk Your Data, Not Big Data
#10There's this – potentially apocryphal – story that the etymology of the term "monkey patch" comes from someone mistaking the then-current term "guerilla patch" for the similar-sounding "gorilla patch".
Big data seems like a very similar story to me. It has nothing to do with the size of the data, not even with the complexity of the analysis, it simply means, "a movement that wants to do more with more kinds of data." Nowhere near what big data used to mean, but outside of technical circles, we're beyond the point where that even matters anymore.