Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Don't use Hadoop when your data isn't that big

chrisstucchio.com

Post reply on HN
Print view View original

191–200 of 235 posts

Re: Don't use Hadoop when your data isn't that big

#191

Post by superuser2 »

bhauer wrote: post #171
superuser2 wrote: post #151

Earlier quoted context omitted.

> the same developers who believe it's reasonable for a web application with modest functionality to require dozens of application nodes to service concurrent request load measured in the mere thousands I'm assuming this is a dig at Rails and Django? What are you suggesting instead?

Just about anything on the JVM (Dropwizard, Play, Finagle, Scalatra, Rest-Express, Rest-Easy, Compojure, Unfiltered, Jersey, Vert.x, Spark), Go (Gorilla, Beego, Revel), Lua (Lapis), Haskell, Erlang...

Are any of these environments actually proven in more than a handful of real-world production environments?

I'd certainly like to get into alternatives but my impression was that none of these battle-tested the way Rails and Django are.

Re: Don't use Hadoop when your data isn't that big

#192

Post by cmccabe »

corresation wrote: post #26
jdk wrote: post #21

Back in college in '97 or so, in our databases class on the first day, the professor asked, "Who's worked with databases before?" A bunch of hands went up. "Oh, sorry, let me rephrase, who's worked with databases larger than a few dozen gigs?" Only one or two hands remained up. "If it's smaller than that, just save yourself the effort and use a flat-file instead." 15 years later and it's the same thing, plus a few or…

Interesting, but it isn't really similar advice. Indeed, it sounds like absolutely terrible advice (unless there is some context that is missing).

Yes, it is terrible advice. Just like this article.

Repeats all the usual myths about Hadoop... that it's just about MapReduce, that it doesn't support indexes (hint: Hive has a CREATE INDEX command, guess what it does?) and then adds some of its own.

After seeing this, plus an article advocating web frontend programming in C, I'm starting to think this place is going downhill fast.

Re: Don't use Hadoop when your data isn't that big

#193

Post by ohashi »

toomuchtodo wrote: post #163
dekhn wrote: post #154

Earlier quoted context omitted.

CERN and its associated teams (CMS and ATLAS) are awesome. They are the only outfit outside of Google that seems to be handling data management and computing properly.

Not to disappoint you, but I could tell you horror stories about my time working on the CMS team at Fermilab. Horrible, horrible stories. I hope the CMS and ATLAS teams onsite at CERN were much better.

I would be curious to hear some of these stories.

Re: Don't use Hadoop when your data isn't that big

#194

Post by porges »

dalke wrote: post #120
batbomb wrote: post #105

Earlier quoted context omitted.

I work in particle astrophysics data processing and management, mostly with gamma ray events for Fermi gamma ray space-telescope. Our total data size after five years is somewhere around 2 PB, but the data set usually used for analysis runs about 200 TB, not really that large. AFAIK, none of the 100+ collaborators use Hadoop. Mostly they use software I develop to run batch jobs in parallel on one of several clusters.…

CERN collects about 25 PB/year.

In the future, the Square Kilometre Array is forecast to require 300-1500 PB/year of archiving, somewhere between 10 PB/hr and 1 EB/day of raw data. (That's 3-12 TB/s...)

Re: Don't use Hadoop when your data isn't that big

#195

Post by gfodor »

mindcrime wrote: post #184
dekhn wrote: post #158

Earlier quoted context omitted.

Some of us have backgrounds in MPI, BSP and other distributed computing paradigms. I switched to high throughput computing and I can tell you that I would never go back to MPI. People who use MPI almost always use it as a crutch to avoid solving the hard problem that would make their computation a lot easier. As an example I used to write MPI code for molecular dynamics simulators. The goal was to use a large superco…

Sure, and I'm not saying "MPI > Hadoop" or anything. Just pointing out that Hadoop is not the only game in town, and is hardly the only way to deal with "big data". I'm a Hadoop fan myself, but I also did a lot of MPI stuff in the past, and I believe there are still scenarios where MPI makes a lot of sense. I have less experience with OpenMP, but I think anybody planning a "big data" project would be well served to a…

Often times choosing the framework du jour is the best choice just because it's the framework du jour. Support, training, books, an active ecosystem, a rich base of developers to hire from with experience, corporations incentivized to fund further development, etc, etc all act as a hedge against "slight technical mismatch between our requirements and the technology vs more esoteric ones."

Re: Don't use Hadoop when your data isn't that big

#196

Post by toomuchtodo »

ohashi wrote: post #193
toomuchtodo wrote: post #163

Earlier quoted context omitted.

Not to disappoint you, but I could tell you horror stories about my time working on the CMS team at Fermilab. Horrible, horrible stories. I hope the CMS and ATLAS teams onsite at CERN were much better.

I would be curious to hear some of these stories.

I'll have to put a blog post together about it. Extremely poor planning, no accountability, a boss who was verbally and emotionally abusive to a co-worker (the reason I quit after only a year there and went back to the private sector).

Re: Don't use Hadoop when your data isn't that big

#197

Post by bhauer »

superuser2 wrote: post #191
bhauer wrote: post #171

Earlier quoted context omitted.

Just about anything on the JVM (Dropwizard, Play, Finagle, Scalatra, Rest-Express, Rest-Easy, Compojure, Unfiltered, Jersey, Vert.x, Spark), Go (Gorilla, Beego, Revel), Lua (Lapis), Haskell, Erlang...

Are any of these environments actually proven in more than a handful of real-world production environments? I'd certainly like to get into alternatives but my impression was that none of these battle-tested the way Rails and Django are.

I assume you mean battle-tested from a security standpoint.

I suppose you'd need to make that judgment call yourself. In my opinion, many of the options I listed have seen sufficient production usage that I consider them roughly as secure as any mainstream framework. For example, Play and Scalatra are used at Netflix, BBC, LinkedIn, etc.

Many expressly avoid notoriously dangerous (and frankly alarming) deserialize-user-input-and-execute patterns, which is in my experience the most common form of framework vulnerability.

As for performance, scalability, and reliability under pressure, that is precisely the point I was making earlier: higher-performance platforms combined with frameworks designed for performance and scalability are targeting grace under pressure moreso than legacy frameworks. Higher performance gives you greater peace of mind with availability, responsiveness, and user experience.

Re: Don't use Hadoop when your data isn't that big

#198

Post by Fomite »

lmm wrote: post #83

So the OP claimed Hadoop skills , the interviewer asked him to use Hadoop , gave a him small example problem. He then didn't use Hadoop , and thinks there's something wrong with the interviewer for objecting to this? Interview problems are sometimes kinda artificial, no shit. Given the impracticality of giving every candidate the kind of dataset Hadoop would be needed for, how would the OP suggest an employer test fo…

The author suggested that this was all their data - and thus the entire scope of the problem, rather than a test sample that, while inefficient, would show the interviewee's skill.

Re: Don't use Hadoop when your data isn't that big

#199

Post by bjelkeman-again »

lemmsjid wrote: post #126

While there is a point to be made here, this article does not make it. Or perhaps it goes too far in attempting to make it, to the point where I feel like it might tip people in the wrong direction. The point of the article is taken if: A) Your data is not large. B) You aren't creating large intermediary datasets with the data. C) You aren't running an increasingly large number of analysis jobs on the data. D) Your c…

Thanks. That put my thoughts on screen rather succinctly for me.

Re: Don't use Hadoop when your data isn't that big

#200

Post by lars512 »

mightybyte wrote: post #27

I agree with the general thrust of this article. But hadoop isn't just for scaling up the absolute size of the data set. It is also useful for scaling up the absolute amount of CPU power you can throw at a problem. If I have 1 GB data set, but the computations that I need to do on that data set are complex enough that it would take a single machine a long time to do them, then hadoop is still useful. I gain tremendou…

We use Hadoop/Hive/EMR for a sub-TB dataset for exactly this reason. The flip side is that I've found the ecosystem to be pretty flakey (specifically Hive). Debugging code you wrote yourself is pretty different to working your way through failures on a distributed system.
Post reply on HN