on a tangential note, sometimes I use a slower methods for UI reasons. For example avoiding blocking the UI, or allowing for canceling the computation, or displaying partial results during the computation (that last one might completely trash the cache).
Command-line tools can be faster than your Hadoop cluster
191–200 of 315 posts
Re: Command-line tools can be faster than your Hadoop cluster
#192Earlier quoted context omitted.
Keep reading, he removes the cat and grep in the final solution.
Yes, but he still keeps the awkward Awk code with the substr and such. I haven't benchmarked, maybe that's faster than the pretty regex matches.
I believe this is to be a bit more educative about how to build a pipeline. Also, iteratively building such solutions quickly often leads to such "inefficiencies" but makes things easier to reason with. Besides, the awk step may have been factored out in the end so it wouldn't make sense to optimise early. Also, by the time the author reaches the end, he gets IO-bound so there's not much need to optimise further (in the context of the exercise).
Re: Command-line tools can be faster than your Hadoop cluster
#193Earlier quoted context omitted.
The problem with shell scripting is that nearly nobody is very, very good at it. The Steam bug doing an rm -rf / is an example, but it's very common for shell scripts to have horrible error handling and checks for important things. The shell is just not suitable for extremely robust programs. I would bet that 80%+ of people who think they're good at shell scripting... aren't.
> The shell is just not suitable for extremely robust programs. Absolute statements like this are usually wrong. This one does not escape the rule. When Linux distros init is mostly bash scripting, there is very little need to further prove that robust systems can be written in bash scripting without the language fighting the developer.
Wait, is it really a good argument for shell-based approach when all major distros are switching to the systemd due to the configuration/maintainability/boilerplate issues with bash init scripting?
Re: Command-line tools can be faster than your Hadoop cluster
#194One common misconception about using Hadoop is that use Hadoop if your data is large. Usage of Hadoop should be more driven based on the growth of data rather than size.
I agree that for the given use case, the solution is appropriate and works fine. Problem mentioned in the given post is not a Big Data problem.
Hadoop will be helpful in case if there are millions of games are played everyday and we need to update the statistics daily e.t.c. For this case, the given solution will hit bottleneck and there will be some optimisation/code change needed to keep running the code.
Hadoop and its ecosystem are not a silver bullet and hence should not be used for everything. The problem has to be a Big Data problem
Re: Command-line tools can be faster than your Hadoop cluster
#195Everyone with basic knowledge of CS could realize that Hadoop is a waste. Unfortunately, it isn't about efficiency at all. It just memeization. Bigdata? Hadoop! Runs everywhere. Same BS like Webscale? MongoDB! meme.
Well sorry but you don't have a clue what you're talking about. I very much work in "big data" with about 2 terabytes of new data coming in every day that has to be ingested and processed with hundreds of jobs running against them. The data needs to be queryable via an SQL like language and analyzed by a dozen data scientists using R or Map Reduce. There isn't anything on the market today that has been proven to work…
The point here is that the same functionality could be implemented way more efficiently even with shell scripts.
The idea of using standard UNIX tools for the showcase is good one. Basically, it tells you that a modern FS is very good at storing chunks of read-only data (one don't need Java for that) with efficient caching and in-kernel procedures. That using pthreads for jobs is a waste, because context-switching has its costs, etc.
To put it simple - by mere rewriting basic functionality in, say, Erlang, one could get orders of magnitude more efficient implementation.
The only selling point of Hadoop is that it exist (mature, stable, blah-blah). It also has one problem - Java. But as long as hardware is cheap and credit is easy - who cares?
Re: Command-line tools can be faster than your Hadoop cluster
#196Earlier quoted context omitted.
Functions, mostly - the big `awk` command in the example goes into something like # @param $1 whatever chess_extract_scores() { awk blah blah blah } and then your whole pipeline simplifies to cat foo | grep bar | chess_extract_scores which is pretty readable. You can even do most of this in a live bash session with ^X ^E.
You can actually do without cat: grep bar foo | chess_extract_scores http://en.wikipedia.org/wiki/Cat_%28Unix%29#Useless_use_of_c...
Sure you can, but premature optimization is also a real thing http://en.wikipedia.org/wiki/Program_optimization#When_to_op...
Re: Command-line tools can be faster than your Hadoop cluster
#197To quote the memorable Ted Dziuba[0]: "Here's a concrete example: suppose you have millions of web pages that you want to download and save to disk for later processing. How do you do it? The cool-kids answer is to write a distributed crawler in Clojure and run it on EC2, handing out jobs with a message queue like SQS or ZeroMQ. The Taco Bell answer? xargs and wget. In the rare case that you saturate the network conn…
Oh right the "cool kids" approach. Here's what the "sensible adults" think about when they see problems like this. Operational Supportability: How do you monitor the operation ? Restart Recovery: Do you have the ability to restart the operation mid way through if something fails ? Maintainability: Can we run the same application on our desktop as on our production servers ? Extensibility: Can we extend the platform e…
And I can't stand developers who overengineer things. We have a couple of them at my company and something that should take a few hours always take several weeks just because of all the reasons you mention. Most things don't need that kind of features and maintainability and if they do in the future we can just rewrite them from scratch. The overall expected return on investment is still better since we seldom need to.
Re: Command-line tools can be faster than your Hadoop cluster
#198Earlier quoted context omitted.
Oh right the "cool kids" approach. Here's what the "sensible adults" think about when they see problems like this. Operational Supportability: How do you monitor the operation ? Restart Recovery: Do you have the ability to restart the operation mid way through if something fails ? Maintainability: Can we run the same application on our desktop as on our production servers ? Extensibility: Can we extend the platform e…
And I can't stand developers who overengineer things. We have a couple of them at my company and something that should take a few hours always take several weeks just because of all the reasons you mention. Most things don't need that kind of features and maintainability and if they do in the future we can just rewrite them from scratch. The overall expected return on investment is still better since we seldom need t…
Quite the opposite, and, quite simple: engineers over-engineer thing in order to make things generic. and generic make solutions robust. that's basic science. Unless the problem and solution are well understood, your investment won't guarantee a return at all.
Re: Command-line tools can be faster than your Hadoop cluster
#199Earlier quoted context omitted.
And I can't stand developers who overengineer things. We have a couple of them at my company and something that should take a few hours always take several weeks just because of all the reasons you mention. Most things don't need that kind of features and maintainability and if they do in the future we can just rewrite them from scratch. The overall expected return on investment is still better since we seldom need t…
Quite the opposite, and, quite simple: engineers over-engineer thing in order to make things generic. and generic make solutions robust. that's basic science. Unless the problem and solution are well understood, your investment won't guarantee a return at all.
No, developers over-engineer because setting up a 20-node Hadoop cluster is fun, whereas doing the same task in an hour in Excel means you have to move onto some other boring task.
Generic doesn't mean robust either, I don't know where you got that from,the two concepts are entirely unrelated.
Re: Command-line tools can be faster than your Hadoop cluster
#200One common misconception about using Hadoop is that use Hadoop if your data is large. Usage of Hadoop should be more driven based on the growth of data rather than size. I agree that for the given use case, the solution is appropriate and works fine. Problem mentioned in the given post is not a Big Data problem. Hadoop will be helpful in case if there are millions of games are played everyday and we need to update th…
It is that buzz surrounding Hadoop that makes people misunderstood its use and capability. I have met non-technical analysts who want RDBMS performance on Hadoop. They expect seconds to minutes scale queries on hundreds of GB of data.
I always throw this analogy to people who misunderstood Hadoop: A stone to crack an egg or a spoon?
Hadoop and RDBMS only have a thin overlapping region in the Venn diagram that describes their capabilities and use cases.
Ultimately, it is cost vs efficiency. Hadoop can solve all data problems. Likewise for RDBMS. This is an engineering tradeoff that people have to make.