Don't use Hadoop when your data isn't that big • Hacker News Board

Re: Don't use Hadoop when your data isn't that big

#71

Post by jacquesm » Tue, Sep 17, 2013, 2:12 PM UTC

Spot on. I recently audited a project that was using an over the top technical solution for a problem that would - with only minor nuts and bolts work - have fit easily on a single machine instead of on a cluster, and it would have run much faster too. Demonstrating this made the case and they've since happily converted. You can buy off-the-shelf machines with 256G of ram at reasonable (for large values of reasonable) cost with IO speeds to match if you equip them with SSDs.

Big Data to me means at a minimum 10's of T, and what big data means changes over time, so todays big data will fit the laptop of the day after tomorrow.

Re: Don't use Hadoop when your data isn't that big

#72

Post by agibsonccc » Tue, Sep 17, 2013, 2:19 PM UTC

Very good points. I liked learning how to use hadoop academically, but it's not an end all be all tool.

If you want to do map reduce, it's perfectly reasonable just to use something smaller scale on multiple cores to do some kind of data processing.

Another thing is real time processing, using something like storm (http://storm-project.net/) or even the parallelism based systems like Akka on the JVM or Go will allow you to have adequate performance. Hadoop has a lot of overhead in not only operations but also job startup.

Re: Don't use Hadoop when your data isn't that big

#73

Post by beagle3 » Tue, Sep 17, 2013, 2:22 PM UTC

dbecker wrote: post #53

meshko wrote: post #28

Earlier quoted context omitted.

I was with you until "and I can't recall anything like this for any other software."

Just for my curiosity, what other software had the same level of unwarranted demand and hype?

Java around 1998. All bugs a thing of the past! Write once run anywhere! Applets taking over the world!

Re: Don't use Hadoop when your data isn't that big

#74

Post by samspenc » Tue, Sep 17, 2013, 2:23 PM UTC

Great article. I would say 1 TB is really the absolute minimum at which you should start considering Hadoop - for anything less, use Python or variants (for flat files) or MySQL or PostGRES (for relational data.)

Re: Don't use Hadoop when your data isn't that big

#75

Post by mattjaynes » Tue, Sep 17, 2013, 2:34 PM UTC

StavrosK wrote: post #51

mattjaynes wrote: post #37

Novelty Driven Development (NDD) Chris points out a great example of NDD here with Hadoop. I do a lot of client work and I see this mistake CONSTANTLY. So often in fact, that I recently wrote up a story to illustrate the problem. Rather than use a tech example, I use a restaurant and plumbing to drive the point home. When the same scenario is put into the context of something more concrete like physical plumbing, it…

That blog's background is 2.6 MB, perhaps you should compress it a bit. Also, I think it's a bit disingenuous to compare a complex system built to various specific needs versus a simpler one that doesn't address those needs at all and just say "their only difference is that one worked and the other didn't". Obviously, if the only criterion was that it should just work, everyone would go with the simpler system. The m…

"Obviously, if the only criterion was that it should just work, everyone would go with the simpler system."

It's obvious to you and me, but to many people it's not obvious at all. You only have to look at the very large deadpool of companies that made these mistakes and died because of it.

People optimize prematurely, scale prematurely, make ego based decisions, care more about interesting tech than making the business grow, etc etc. That's really the whole point of the parent article - people make bad decisions because of the allure of some "cool" but complex and unnecessary tool.

Also, thanks for the note on the background - I'll fix that :)

Re: Don't use Hadoop when your data isn't that big

#76

Post by akanet » Tue, Sep 17, 2013, 2:34 PM UTC

Perhaps this is the beginning of the resurgence of the "Small Data Expert"

Re: Don't use Hadoop when your data isn't that big

#77

Post by t1m » Tue, Sep 17, 2013, 2:37 PM UTC

mrcactu5 wrote: post #38

I remember starting to grasp what "big data" meant when I had a phone interview with Twitter. @ Imagine you have some numbers spread over some computers -- too many to fit in one computer find the median. ▪ Uhh, sort them? @ Can you find the median on a single computer without sorting them. ▪ :-( @ We'll call you back tomorrow. I was promptly rejected, but it set the tone for my later studies. The criterion for Big D…

Twitter often gets singled out for it's big dataness, though the latest numbers I've seen are only about 400 million tweets per day. Even allowing 1K/tweet this is a rather manageable 400GB uncompressed.

15 minutes of US tweets would fit on your phone :-)

Re: Don't use Hadoop when your data isn't that big

#78

Post by bitL » Tue, Sep 17, 2013, 2:38 PM UTC

I couldn't disagree more with some of the statements in the article.

Hadoop is not a database! It's a parallel computing platform for MapReduce-style problems that could preserve locality. If your problem fits this, Hadoop absolutely rocks. If your problem is different then please use another tool. If your problem deals for example with high-resolution geographic or LiDAR data that can be easily processed independently (and that would easily give you a few petabytes each scan/flight so you can't stuff it into GPU), Hadoop is about the only open thing you can use to process them reliably (imagine having Earth surface data with the resolution of 1cm and the need to prepare multiple levels of detail, simplify geometry, perform object recognition, identify roads etc.). Even when your data is smaller if your problem fits in the MapReduce model well, Hadoop is a pretty convenient way to be future-proof while enjoying already mature application infrastructure. Why would you even bother working with toys that put everything into memory and then fail miserably in production (HW failure, need to reseed data after crash etc.)?

I worked in a company that routinely processed these kinds of data; usually people accepting the thinking from this article hit a wall someday in production, couldn't guarantee reliability and ended up writing endless hacks for their algorithms that didn't scale when it was needed and became frustrating bottlenecks for everyone.

Yes, I saw also some ridiculous uses of Hadoop (database that didn't have a chance ever growing over 20M records, problems not fitting MapReduce that needed custom messaging between jobs etc). Just use your reason properly, whatever has the potential for the future to handle large data do it with Hadoop or any appropriate system that supports your algorithmic model (S4, Kafka, OrientDB, Storm etc.) straight away.

Make your software future-proof now or you'll have to rewrite it from the scratch when you will be under huge pressure. Don't become complacent with what you "know" now.