Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Command-line tools can be faster than your Hadoop cluster

aadrake.com

Post reply on HN
Print view View original

201–210 of 315 posts

Re: Command-line tools can be faster than your Hadoop cluster

#201

Post by collyw »

kansface wrote: post #154
freehunter wrote: post #149

Earlier quoted context omitted.

Yeah but 80% of the people writing Java and think they're good aren't as well. And plenty of companies support Java. The answer isn't "don't use it", it's "train your programmers in the languages they use".

The sorts of bugs people experience with Java mostly result in a crashed/stalled/hung process. Bash bugs erase your entire file system. The thing about Bash is that it is trivially easy to make these sorts of mistakes- the language just isn't suitable to general purpose scripting.

It shouldn't be able to erase your filesystem unless you are running as root or doing something equally stupid. That's pretty much common sense stuff for anyone that isn't a beginner.

Re: Command-line tools can be faster than your Hadoop cluster

#202

Post by vidarh »

kansface wrote: post #154
freehunter wrote: post #149

Earlier quoted context omitted.

Yeah but 80% of the people writing Java and think they're good aren't as well. And plenty of companies support Java. The answer isn't "don't use it", it's "train your programmers in the languages they use".

The sorts of bugs people experience with Java mostly result in a crashed/stalled/hung process. Bash bugs erase your entire file system. The thing about Bash is that it is trivially easy to make these sorts of mistakes- the language just isn't suitable to general purpose scripting.

If you're worried about that, don't give the script permissions to access your entire filesystem. Easily handled with separate users, cgroups, assorted containerisation, and more.

Re: Command-line tools can be faster than your Hadoop cluster

#203

Post by gaius »

angrybits wrote: post #126
pjc50 wrote: post #101

Earlier quoted context omitted.

Windows Powershell is actually quite nice and powerful, while also avoiding some of the shell legacy traps around escaping. It suffers from not being very discoverable and not having a community.

I wish they had just been less stubborn and made something that would run bash and standard unix commands. I've used it for production jobs and it's worked as advertised, but I would have rather have just had bash.

A lot of bash is using awk, sed, cut, tr and others to munge the output of one command into the input of another. The killer feature of Powershell is that it does away with all of that.

Re: Command-line tools can be faster than your Hadoop cluster

#204

Post by nilbot »

angrybits wrote: post #125
threeseed wrote: post #104

Earlier quoted context omitted.

Anybody can throw some crap together and make it stick. And it's a perfectly valid solution. My issue is when there is criticism laid against those solutions which are actually engineered in a way that allows for supportability and extensibility. They are arguably far more important than execution time.

I can't figure out why you're arguing against standard unix tools/idioms in the name of supportability and extensibility? It defies logic.

Simple: because std utils are programs that do what they supposed to do. if problems bound are well within the definition domain of a std util then its all good. Supportability and Extensibility is way too generic for you to draw a line saying std utils can handle them all. After all, they are programs, not programming languages.

Re: Command-line tools can be faster than your Hadoop cluster

#205

Post by microtonal »

gpapilion wrote: post #110
threeseed wrote: post #90

Earlier quoted context omitted.

Oh right the "cool kids" approach. Here's what the "sensible adults" think about when they see problems like this. Operational Supportability: How do you monitor the operation ? Restart Recovery: Do you have the ability to restart the operation mid way through if something fails ? Maintainability: Can we run the same application on our desktop as on our production servers ? Extensibility: Can we extend the platform e…

I look at this article as a criticism of the hadoop being the wrong tool for small data sets. This starts to become a question of data locality, and size. 1.75 GB isn't enough data to justify a hadoop solution. That data size fits easily in memory, and without doubt on a single system. From that point you only need some degree of parallelism to maximize the performance. That being said when its 35TB of data, the answ…

1.75 GB isn't enough data to justify a hadoop solution. That data size fits easily in memory, and without doubt on a single system.

It depends on what you do with the data. If you are processing the data in 512KB chunks and each chunk takes a day to process (because expensive computation), you probably do want to spread the work over some cluster.

Re: Command-line tools can be faster than your Hadoop cluster

#206

Post by nilbot »

gaius wrote: post #199
nilbot wrote: post #198

Earlier quoted context omitted.

Quite the opposite, and, quite simple: engineers over-engineer thing in order to make things generic. and generic make solutions robust. that's basic science. Unless the problem and solution are well understood, your investment won't guarantee a return at all.

No, developers over-engineer because setting up a 20-node Hadoop cluster is fun, whereas doing the same task in an hour in Excel means you have to move onto some other boring task. Generic doesn't mean robust either, I don't know where you got that from,the two concepts are entirely unrelated.

Generic -> robust. i... i dont know how to explain that. honestly i haven't thought about the necessity of explaining things like this. its... basic mathematics.

Re: Command-line tools can be faster than your Hadoop cluster

#207

Post by sergiosgc »

vdaniuk wrote: post #193
sergiosgc wrote: post #188

Earlier quoted context omitted.

> The shell is just not suitable for extremely robust programs. Absolute statements like this are usually wrong. This one does not escape the rule. When Linux distros init is mostly bash scripting, there is very little need to further prove that robust systems can be written in bash scripting without the language fighting the developer.

Wait, is it really a good argument for shell-based approach when all major distros are switching to the systemd due to the configuration/maintainability/boilerplate issues with bash init scripting?

I'm not going into the systemd VS sysvinit discussion. For my argumentation, it is enough to recognize bash based sysvinit has been with us for circa 20 years with no stability problems.

Re: Command-line tools can be faster than your Hadoop cluster

#208

Post by sleepythread »

yuanchuan wrote: post #200
sleepythread wrote: post #194

One common misconception about using Hadoop is that use Hadoop if your data is large. Usage of Hadoop should be more driven based on the growth of data rather than size. I agree that for the given use case, the solution is appropriate and works fine. Problem mentioned in the given post is not a Big Data problem. Hadoop will be helpful in case if there are millions of games are played everyday and we need to update th…

It is that buzz surrounding Hadoop that makes people misunderstood its use and capability. I have met non-technical analysts who want RDBMS performance on Hadoop. They expect seconds to minutes scale queries on hundreds of GB of data. I always throw this analogy to people who misunderstood Hadoop: A stone to crack an egg or a spoon? Hadoop and RDBMS only have a thin overlapping region in the Venn diagram that describ…

I totally agree with you. Capability "LIKE" will drive Hadoop adoption, Hadoop should not be seen as replacement of R.D.B.M.S. These are two different tools for made for different purpose.

Re: Command-line tools can be faster than your Hadoop cluster

#209

Post by gaius »

nilbot wrote: post #206
gaius wrote: post #199

Earlier quoted context omitted.

No, developers over-engineer because setting up a 20-node Hadoop cluster is fun, whereas doing the same task in an hour in Excel means you have to move onto some other boring task. Generic doesn't mean robust either, I don't know where you got that from,the two concepts are entirely unrelated.

Generic -> robust. i... i dont know how to explain that. honestly i haven't thought about the necessity of explaining things like this. its... basic mathematics.

I don't think these words mean what you think they mean. Like "science" and "mathematics".

Re: Command-line tools can be faster than your Hadoop cluster

#210

Post by roel_v »

angrybits wrote: post #126
pjc50 wrote: post #101

Earlier quoted context omitted.

Windows Powershell is actually quite nice and powerful, while also avoiding some of the shell legacy traps around escaping. It suffers from not being very discoverable and not having a community.

I wish they had just been less stubborn and made something that would run bash and standard unix commands. I've used it for production jobs and it's worked as advertised, but I would have rather have just had bash.

Because there are a lot of shitty things about bash, too, that anybody with half a brain would think should be blindingly simple from the command line. For example, add a virtual host to apache with an Allow for localhost on /var/www/localtest. And do it in a portable way, i.e. no 'put every vhost in a separate file and Include those. Or fetch a configuration parameter from another machine and that other machine might run any of 5 different distros. The list goes on and on - look I'm no PowerShell fan, or 'real' user even, but we have to admit that the old Unix approach is reaching EOL (well I should say 'it should reach EOL', unfortunately it seems like it's not going to go away soon, with there not even being an alternative).
Post reply on HN