Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Don't use Hadoop when your data isn't that big

chrisstucchio.com

Post reply on HN
Print view View original

201–210 of 235 posts

Re: Don't use Hadoop when your data isn't that big

#201

Post by jake_morrison »

superuser2 wrote: post #191
bhauer wrote: post #171

Earlier quoted context omitted.

Just about anything on the JVM (Dropwizard, Play, Finagle, Scalatra, Rest-Express, Rest-Easy, Compojure, Unfiltered, Jersey, Vert.x, Spark), Go (Gorilla, Beego, Revel), Lua (Lapis), Haskell, Erlang...

Are any of these environments actually proven in more than a handful of real-world production environments? I'd certainly like to get into alternatives but my impression was that none of these battle-tested the way Rails and Django are.

There is a reason that the largest Internet companies are still running their infrastructure on Java and C++. It scales and operations people know how to manage it.

Rails is the classic example of optimizing for developer ease of use instead of performance. Which probably makes sense for startups, but scaling is a pain in the ass.

Openresty is great for infrastructure level solutions, e.g. request routing and authentication. It is used by some of the largest Internet companies in China.

The benchmarks at http://www.techempower.com/benchmarks/ are a reminder of how poorly some popular frameworks perform. And how poorly cloud performs vs relatively cheap dedicated hardware.

Re: Don't use Hadoop when your data isn't that big

#202

Post by travisoliphant »

sitkack wrote: post #172

Hadoop is the problem, MapReduce is not the problem. Having used both Hadoop and Disco, I can say that Disco is by far a net positive on all projects I used it on. And the overhead to coding it in Disco vs single node is about an extra 30 minutes. You can start with working single node and go multinode w/o much effort. http://discoproject.org Hadoop on the other hand is a huge, massive pain in the ass. And I am a Had…

This is 100% true. Hadoop gets way too much attention given the other useful solutions that exist out there. I have known people to use Disco successfully on several hundred node cluster. You can also interact Disco with IPython parallel much more easily.

This is why we include it in Linux versions of Anaconda.

Re: Don't use Hadoop when your data isn't that big

#203

Post by [deleted] »

superuser2 wrote: post #191
bhauer wrote: post #171

Earlier quoted context omitted.

Just about anything on the JVM (Dropwizard, Play, Finagle, Scalatra, Rest-Express, Rest-Easy, Compojure, Unfiltered, Jersey, Vert.x, Spark), Go (Gorilla, Beego, Revel), Lua (Lapis), Haskell, Erlang...

Are any of these environments actually proven in more than a handful of real-world production environments? I'd certainly like to get into alternatives but my impression was that none of these battle-tested the way Rails and Django are.

[deleted]

Re: Don't use Hadoop when your data isn't that big

#204

Post by e12e »

vidarh wrote: post #180
davidmr wrote: post #57

While I couldn't agree more with the general point of the article, I have some small additional comments. Just as a bit of background, I think that Chris would very much agree that I am not the intended recipient of this advice, and so my comments probably aren't keeping in the spirit of the article. I've spent the last 10+ years exclusively in very large HPC environments where the average size of the problem set is…

> We'll be generous and say that 4TB can do 150MB/s. A single run through the data at maximum efficiency will cost you ~8 hours. Since we've restricted ourselves to a single box, we're also not going to be able to keep the data in memory for subsequent calculations/simulations/whatever. I agree with your overall point, but of course the actual limits for a fairly cost effective single box are fairly high these days.…

Do you mean leasing, as in renting the box, and having it on-site? 2k/month sounds awfully cheap for a 64k box? Even if you rent it for 3 years without upgrading, that hardly seems worth it to the leasing company? Or maybe there is something going on with the margins here that I'm missing?

Re: Don't use Hadoop when your data isn't that big

#205

Post by e12e »

rdtsc wrote: post #13

"We don't have big data" or "our data is rather small" -- said no dev team ever. "Big data" is like "cloud" it is a cool label everyone applying to their system. Just like OO was in its time. Well once they applied the label they feel they need to live up to it so well "we gotta use what big data companies use" and they pick Hadoop. I've heard hadoop used when MySQL, SQLite or even flat files would have worked.

I did say that my former employer. Former.

Re: Don't use Hadoop when your data isn't that big

#206

Post by xradionut »

asabjorn wrote: post #137
raverbashing wrote: post #17

Earlier quoted context omitted.

You would probably want some RAID + Backups, (and a very generous amount of RAM there) but this is ok. In fact, I wouldn't doubt that for several "big data" users even SQLite would be enough.

From my experience SQLite is great when only one process with no parallelism needs to write to the DB and where no other process will frequently overlap reads with writes. However, for CPU-heavy computations that needs multiple cores to compute and write to the DB PostgreSQL has a much better fit for me even for fairly small data-sets. And the ease of use and setup of PostgreSQL is not that much more than SQLite.

As date guy that uses both, there is a higher level of expertise required to properly maintain PostgreSQL. SQLite is something that I can call instantly from Python without much forethought.

Re: Don't use Hadoop when your data isn't that big

#207

Post by CurtMonash »

While the point in the headline is fine, the supporting reasoning is in places dubious.

Don't use SQL for anything over 5 TB? Huh? You can put a lot more data than that on a node with a nice open source columnar analytic DBMS, and of course there are a lot of MPP relational analytic DBMS as well.

SQL on Hadoop requiring full table scans? Well, that's what Impala is for. Hadapt is more mature than Impala. Stinger is coming on, and is open source.

Nice marketing line, however.

Re: Don't use Hadoop when your data isn't that big

#208

Post by dalke »

edraferi wrote: post #185
dalke wrote: post #14

Earlier quoted context omitted.

It's even more exciting mixing pandas with the IPython notebook.

heh. Until you actually have to repeat something! It's quite useful for iterative refinements like tweaking pyplot graphics, but it's all to easy to lose track of what versions of which blocks executed to create the current state. Maybe I just need more discipline..

Walking on a highwire is also exciting. :)

Re: Don't use Hadoop when your data isn't that big

#209

Post by e12e »

e12e wrote: post #205
rdtsc wrote: post #13

"We don't have big data" or "our data is rather small" -- said no dev team ever. "Big data" is like "cloud" it is a cool label everyone applying to their system. Just like OO was in its time. Well once they applied the label they feel they need to live up to it so well "we gotta use what big data companies use" and they pick Hadoop. I've heard hadoop used when MySQL, SQLite or even flat files would have worked.

I did say that my former employer. Former.

*at my former employer.
Post reply on HN