Live data from Hacker News

Don't use Hadoop when your data isn't that big

chrisstucchio.com

121–130 of 235 posts

Re: Don't use Hadoop when your data isn't that big

#121
post #21

Back in college in '97 or so, in our databases class on the first day, the professor asked, "Who's worked with databases before?" A bunch of hands went up. "Oh, sorry, let me rephrase, who's worked with databases larger than a few dozen gigs?" Only one or two hands remained up. "If it's smaller than that, just save yourself the effort and use a flat-file instead." 15 years later and it's the same thing, plus a few or…

That is perhaps the worst advice I've ever heard. I spent a summer some years ago writing what, in abstract, were hand-coded SELECT and INSERT statements for CSVs. It was a waste of time; the number of bugs was ridiculous and the data integrity was attained by brute force. Today I'd use sqlite and/or postgres.

Re: Don't use Hadoop when your data isn't that big

#122
post #7

I sort of agree. However when you think you will grow into the TB range, you just as well do it in hadoop right away. We are using Hive and HiveQL and have SQL like queries which generate the correct output. The result is: we dont have to hassle with the hadoop mappers and reducers. And we can write our "queries" in a human readable fashion.

Why would Hadoop automatically be the right tool for the job in this case? In fact, when is any tool automatically the right tool for the job? I understand the desire not to do unnecessary work and to plan ahead. However, something like Hadoop is very heavy weight and requires a significant change to the way you write your software. It isn't something easy to undo if Hadoop turns out to be the wrong direction. Thus,…

I do agree with you. I prefer too keep it simple too, and not plan ahead too much According to the KISS principle. I'm not saying hadoop is the solution all the time. I'm just saying that sometimes it makes sense.

Re: Don't use Hadoop when your data isn't that big

#123
This isn't directly related, but is hadoop the only java-based solution to parallel computing? I've seen some examples of people attempting to do work in java mpi[0] again. It seems like dealing with the gc and memory management in general has been an issue when trying to do high performance computing in java, especially at a distributed scale.

0:http://blogs.cisco.com/performance/mpi-and-java-redux/

Re: Don't use Hadoop when your data isn't that big

#124
post #7

I sort of agree. However when you think you will grow into the TB range, you just as well do it in hadoop right away. We are using Hive and HiveQL and have SQL like queries which generate the correct output. The result is: we dont have to hassle with the hadoop mappers and reducers. And we can write our "queries" in a human readable fashion.

Why would Hadoop automatically be the right tool for the job in this case? In fact, when is any tool automatically the right tool for the job? I understand the desire not to do unnecessary work and to plan ahead. However, something like Hadoop is very heavy weight and requires a significant change to the way you write your software. It isn't something easy to undo if Hadoop turns out to be the wrong direction. Thus,…

I do agree with you. I prefer too keep it simple too, and not plan ahead too much According to the KISS principle. I'm not saying hadoop is the solution all the time. I'm just saying that sometimes it makes sense.

Re: Don't use Hadoop when your data isn't that big

#125
I guess I haven't been around enough small startups (fewer than 100 people) because I hardly get the sense that people are haphazardly spinning hadoop clusters. People generally pick the right tools for the right jobs. This is particularly true in the datawarehousing world.

That being said I can see problems where people pick hadoop without knowing how it's going to integrate into their systems 1-3 years down the road. Particularly with cloud computing these days, you can easily bring large complex systems online with little effort. It's cool and scary at the same time.

Re: Don't use Hadoop when your data isn't that big

#126
While there is a point to be made here, this article does not make it. Or perhaps it goes too far in attempting to make it, to the point where I feel like it might tip people in the wrong direction.

The point of the article is taken if:

A) Your data is not large. B) You aren't creating large intermediary datasets with the data. C) You aren't running an increasingly large number of analysis jobs on the data. D) Your computational overhead is small. E) Your memory overhead is small (this requires an asterix, because some tasks that require extreme amounts of memory will not work well in hadoop and should be brought outside) F) You don't need or want a system to track the increasingly large number of analysis jobs you're running. G) You can guarantee you won't outgrow A, B, C which will force you to rewrite all your code.

G is especially difficult because it's hard to predict. F is always underestimated at the beginning of a project and bites you later. Yes you can write analysis scripts--what happens when there are a hundred of them, written by different developers? Time to write a job tracking system, with counters, retries, notification, etc. Like Hadoop.

To further D and E, there are workloads that are relatively straightforward across terrabytes of data, and there are workloads that are expensive over gigabytes of data (especially those involving the creation of intermediate indices, which is where MR itself speeds things up considerably, esp if done in parallel).

Also, in a critique of Hadoop the article obsesses over MapReduce (in a way, conflating Hadoop and MapReduce, just as it conflates 'SQL' with a 'SQL database'), ignoring the increasingly powerful tools that can be used, such as Hive, Pig, Cascading, etc. Do those tools beat a SQL database in flexibility? The answer is that the question is not really relevant. If you already understand the nature of your data, and you've gone through the very difficult act of designing a normalized schema that fits what you need, then you're in a good place. If you have a chunk of data in which the potential has not yet been unlocked, or in which the act of writing to it happens to quickly to justify the live indexing implied by a database, then Hadoop is an essential tool. They really sit next to one another.

None of this is to knock writing analysis scripts against local data. I do that all the time. In fact often I'll ship data from HDFS to the local system so I can write and run a script. I just think it's important at a company to make sure your people have access to good tools so there aren't hurdles in front of them, and when it comes to data analysis I've come to the opinion that you really want to have a Hadoop cluster set up next to your SQL databases and your other tooling, because it will become useful in sometimes unpredictable ways.

Yes, if there are a few hundred megabytes in front of you and you need to analyze them, then write a script--and were I interviewing someone for a job I would not hesitate to accept a script that solves a data analysis task, so clearly the people being interacted with by the author were somewhat myopic. But most companies require that an ecosystem be built to handle the increasing complexity that will ensue over the years. And Hadoop is a huge bootstrap to that ecosystem, regardless of data size.

Re: Don't use Hadoop when your data isn't that big

#127

I agree with the general thrust of this article. But hadoop isn't just for scaling up the absolute size of the data set. It is also useful for scaling up the absolute amount of CPU power you can throw at a problem. If I have 1 GB data set, but the computations that I need to do on that data set are complex enough that it would take a single machine a long time to do them, then hadoop is still useful. I gain tremendou…

I can understand this use case. e.g. you have some 100 parameter financial instrument that can only be valued with monte-carlo methods. You could have 1 GB of data, but many core years of computation.

In this case, is Hadoop actually a good tool? When you have such a small amount of data, it is unlikely you will ever be faced with a "reduce" challenge, and thus the problem is better solved with a simple queue rather than a hadoop cluster.

Re: Don't use Hadoop when your data isn't that big

#128

Does anyone use Hadoop for job management? We have millions of XML documents in a document database. Many of the questions we want to ask about those documents can be answered through the native database querying capabilities. But there are always questions falling outside the scope of the query capabilities, that could be answered by a simple map function applied to each document, with a reduce to combine the result…

The company I work for creates a product which may help you : http://www.syncsort.com/en/Data-Integration/Products/DMX-h/D...

Re: Don't use Hadoop when your data isn't that big

#129

Earlier quoted context omitted.

Do you use standard off-the-shelf consumer hard drives at those prices? Most companies I've worked for have shelled out quite a bit more for "enterprise" class hard drives. I've always struggled to understand what these bring to the table, and my understanding is that it's some combination of greater reliability and a service agreement. It's always seemed to me, in my software-centric naivete, that it would be more c…

One of the most important things is that enterprise drives are much more likely to be telling you the truth when they tell you your data is written to disk. Even if it hasn't actually hit the platter, they'll have a battery or cap backup power supply that allows them to write anything still pending to disk before they run out of power. Consumer drives often lie horribly about whether data is truly durable.

Not really an issue for building a reporting/aggregation database.

But certainly something to think about when processing realtime financial transactions.

Re: Don't use Hadoop when your data isn't that big

#130
post #66

99% of the cases I have seen where people have been working with tables that are in the 5+ TB range for analysis, there is some obvious way to compress the data that they have overlooked. Most analysts find some way to aggregate a dataset once, then do actual work on that aggregated dataset, rather than the raw data. In geospatial analytics, for example, a trillion records can be aggregated down to census blocks/bloc…

In addition to compression, let's not forget sampling. For a lot of problems a random subset of the data will give you a statistically meaningful answer with sufficient precision. It seems like the rise of "big data" has led to the assumption that all queries have to be run against the entire dataset.

True. The issue with "Big Data" is that sometimes, especially when you need to produce personalised recommendations, the sampling doesn't cut it (or at least produces sub-optimal results).
Post reply on HN