Don't use Hadoop when your data isn't that big
chrisstucchio.com
Don't use Hadoop when your data isn't that big
1–10 of 235 posts
Re: Don't use Hadoop when your data isn't that big
#2Hooray! Some sense at last.
I have worked for at least 3 different employers that claimed to be using "Big Data". Only one of them was really telling the truth.
All of them wanted to feel like they were doing something special.
The sad thing is, they were all special, each in their own particular way, but none of what made each company magic and special had anything to do with the size of the data that they were handling.
Hadoop was required in exactly zero of these cases.
Funnily enough, after I left one of them, they started to build lots of Hadoop-based systems for reasons which, as far as I could fathom, had more to do with the resumes of the engineers involved than the actual technical merits of the case.
Sad, but 'tis the way of the world.
Re: Don't use Hadoop when your data isn't that big
#3"A 2 terabyte hard drive costs $94.99, 4 terabytes is $169.99. Buy one and stick it in a desktop computer or server. Then install Postgres on it."
Done! Although with more drives and a backup server. Right now, we're pushing 15Tb with no loss in performance.
Re: Don't use Hadoop when your data isn't that big
#4I understand and agree with the author's main point that many companies that use big data do not need to use these technologies.
I do not agree that the tools are inferior to Sql. Hive is really close to sql and Pig is extremely powerful. I would take a look at a few of the recent updates to these tools before declaring them inferior to Sql.
Re: Don't use Hadoop when your data isn't that big
#5So true!
I would 'up' it twice if I could.
Too many startups go over to Hadoop/no-sql solutions before the overhead is indeed justified. SQL for most of the data with a bit of Redis and numpy for background processing will take you much further than most people assume.
It's fun to think you must have DynamoDB, Hadoop or a Cassandra backend, but in real life -- you better invest in more features (or analytics!)
Re: Don't use Hadoop when your data isn't that big
#6Indeed.
A rule of the thumb that I've inferred from many installations is: Just introducing Hadoop makes everything 10 times slower AND more expensive than an efficient tool set (e.g. pandas).
So it only makes sense to start hadooping when you are getting close to the limit of what you can pandas - everything you do before that is a horrible waste of resources.
And when you do get there - often, a slightly smarter distribution among servers and staying with e.g. pandas, will let you keep scaling up without introducing the /10 factor in productivity. Although, it might be unavoidable at some point.
Re: Don't use Hadoop when your data isn't that big
#7I sort of agree. However when you think you will grow into the TB range, you just as well do it in hadoop right away.
We are using Hive and HiveQL and have SQL like queries which generate the correct output. The result is: we dont have to hassle with the hadoop mappers and reducers. And we can write our "queries" in a human readable fashion.
Re: Don't use Hadoop when your data isn't that big
#8Hooray! Some sense at last. I have worked for at least 3 different employers that claimed to be using "Big Data". Only one of them was really telling the truth. All of them wanted to feel like they were doing something special. The sad thing is, they were all special, each in their own particular way, but none of what made each company magic and special had anything to do with the size of the data that they were hand…
Tricky catch 22. You want the kind of employees that are interested in what they do enough to want to learn new skills and try new exciting technologies. But, you want them to actually use something boring.
Re: Don't use Hadoop when your data isn't that big
#9Hooray! Some sense at last. I have worked for at least 3 different employers that claimed to be using "Big Data". Only one of them was really telling the truth. All of them wanted to feel like they were doing something special. The sad thing is, they were all special, each in their own particular way, but none of what made each company magic and special had anything to do with the size of the data that they were hand…
Tricky catch 22. You want the kind of employees that are interested in what they do enough to want to learn new skills and try new exciting technologies. But, you want them to actually use something boring.
There's nothing boring about pandas
Re: Don't use Hadoop when your data isn't that big
#10Hooray! Some sense at last. I have worked for at least 3 different employers that claimed to be using "Big Data". Only one of them was really telling the truth. All of them wanted to feel like they were doing something special. The sad thing is, they were all special, each in their own particular way, but none of what made each company magic and special had anything to do with the size of the data that they were hand…
Agreed, but I'm not optimistic that the view will become wide-spread.
There was a phrase I learned long ago: "if it fits into cache it's not a supercomputing problem."
This essay promotes the related view: "If it fits on a desktop, it's not Big Data."
BTW, you might like Greg Wilson's 2008 essay "HPC Considered Harmful". I can't find the slides anymore, but there's a lovely interview with him about this at http://itc.conversationsnetwork.org/shows/detail3682.html .