My company's data is that big, thank you very much.
But if yours isn't, sure, don't use a system designed for handling astronomical data. Should be common sense, probably isn't.
Don't use Hadoop when your data isn't that big
81–90 of 235 posts
Re: Don't use Hadoop when your data isn't that big
#82Earlier quoted context omitted.
I was with you until "and I can't recall anything like this for any other software."
Just for my curiosity, what other software had the same level of unwarranted demand and hype?
I don't think it's about software per se, but more about the idea of Big Data. We have seen many ideas that have come and go most of the times a good idea but a good idea is not something you should use for everything.
From the top of my head
Ajax (very handy but lot of times misused and misunderstood or just DHTML)
No SQL (can be handy, but for some people it is like a religion)
XML (why use CSV when you are able to use XML)
Cloud (Oh by cloud you mean the internet?)
and many more :)
Re: Don't use Hadoop when your data isn't that big
#83So the OP claimed Hadoop skills, the interviewer asked him to use Hadoop, gave a him small example problem. He then didn't use Hadoop, and thinks there's something wrong with the interviewer for objecting to this?
Interview problems are sometimes kinda artificial, no shit. Given the impracticality of giving every candidate the kind of dataset Hadoop would be needed for, how would the OP suggest an employer test for Hadoop skills?
Re: Don't use Hadoop when your data isn't that big
#84While I couldn't agree more with the general point of the article, I have some small additional comments. Just as a bit of background, I think that Chris would very much agree that I am not the intended recipient of this advice, and so my comments probably aren't keeping in the spirit of the article. I've spent the last 10+ years exclusively in very large HPC environments where the average size of the problem set is…
"Big Data starts at 1.5 PB [because that's what fits in memory on Blue Waters]" -- Bill Gropp
On today's top HPC installations, it currently takes about an hour to read or write the contents of memory from/to global storage. If the workflow has broad dependencies (PDEs are especially bad, but lots of network analysis also fits the bill), it's much better to use more parallelism to run for a shorter period of time with the working set in memory. If you can't fit your working set in memory on the largest machine available, chances are you should either find time on a bigger machine or you can't afford to do the analysis. (The largest scientific allocations are in the 10s of millions of core hours per year, which would be burned in a few reads and writes on a million-core machine.)
Also note that MapReduce is not very expressive. When IBM's Watson team was getting started, some people suggested using MapReduce/Hadoop, but the team quickly concluded it was way too slow/constraining and instead opted for an in-memory database with MPI for communication.
Re: Don't use Hadoop when your data isn't that big
#85You would be surprised to find out how much processing 400GB of geodata needs sometimes. 7 days in a normal machine to do some non-trivial analysis of OpenStreetMap data. Hadoop can reduce that to hours instead.
Re: Don't use Hadoop when your data isn't that big
#86You are right that it should be used the proper tool for each particular problem. And Hadoop world is harder than single machine systems (like pandas). So, you shouldn't user Hadoop if you can do the job with simpler systems.
But I have something to add. Hadoop is not only introducing new techniques for distributed storage and computation. Hadoop is also proposing a methodological change in the way a data project is approached.
I'm not talking only about doing some analytic over the data, but building an entire data driven system. A good example would be the case of building a vertical search engine, for example for classified ads around the world. You can try to build the system just using a database and some workers dealing with the data. But soon you'll find a lot of problems for managing the system.
Hadoop provides you all the storage and processing power that you want (it is matter of money). Why if you build your system in a way where you recompute always everything from the raw input data? That can be seen as something stupid: Why doing that if you can run the system with less resources?
The answer is that with this approach you can:
- Being human fail-tolerant. If somebody introduces a bug in the system, you just have to fix the code, and relaunch the computation. That is not possible with stateful systems, like those based in doing updates over a database. - Being very agile in developing evolutions. Change the whole system is not traumatic, as you just have to change the code and relaunch the process with the new code without much impact in the system. That is not something simple in database backed systems.
The following page shows how a vertical search engine would be built using Hadoop and what would it be its advantages: http://www.datasalt.com/2011/10/scalable-vertical-search-eng...
Re: Don't use Hadoop when your data isn't that big
#87Back in college in '97 or so, in our databases class on the first day, the professor asked, "Who's worked with databases before?" A bunch of hands went up. "Oh, sorry, let me rephrase, who's worked with databases larger than a few dozen gigs?" Only one or two hands remained up. "If it's smaller than that, just save yourself the effort and use a flat-file instead." 15 years later and it's the same thing, plus a few or…
Aside from the ACID properties others mentioned, if you're using a relational database (and most people are), there may be non-performance related benefits. Some data is inherently relational, and it can be easier to manage it using relational abstractions such as SQL.
Re: Don't use Hadoop when your data isn't that big
#88I remember starting to grasp what "big data" meant when I had a phone interview with Twitter. @ Imagine you have some numbers spread over some computers -- too many to fit in one computer find the median. ▪ Uhh, sort them? @ Can you find the median on a single computer without sorting them. ▪ :-( @ We'll call you back tomorrow. I was promptly rejected, but it set the tone for my later studies. The criterion for Big D…
That second question is kind of dirty. Pretty much all algorithms to find the median will perform a partial sort. Without any sorting at all, the only thing I can think of is some kind of statistical approximation based on sampling.
Re: Don't use Hadoop when your data isn't that big
#89I have no experience with hadoop at all and it may be slightly off topic but this reminds me of a post titled "Taco bell programming"[1], after reading which I started learning and using Unix tools and commands much more than before instead of writing silly Python scripts for almost anything that needed automation.
[1]: http://web.archive.org/web/20110220110013/http://teddziuba.c...
Re: Don't use Hadoop when your data isn't that big
#90Does anyone use Hadoop for job management?
We have millions of XML documents in a document database. Many of the questions we want to ask about those documents can be answered through the native database querying capabilities.
But there are always questions falling outside the scope of the query capabilities, that could be answered by a simple map function applied to each document, with a reduce to combine the results.
Seems like a pain to always query for the documents you want to process, find some place to store them on disk, then run a program locally to get the result, vs. writing a map and reduce job then pointing it at the documents in the database to run against (this document store has a Hadoop integration API). Hadoop also seems to have a lot of nice job frameworks and monitoring tools and APIs to track job progress.
Anyone have similar situation where you used Hadoop just to get job management and tracking and flexibility in performing data analysis tasks? Are there easier ways to accomplish this goal?