Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Command-line tools can be faster than your Hadoop cluster

aadrake.com

Post reply on HN
Print view View original

211–220 of 315 posts

Re: Command-line tools can be faster than your Hadoop cluster

#211

Post by roel_v »

UrMomReadsHN wrote: post #93
Swizec wrote: post #41

Maybe I come from a weird world, or even a weird generation. But when I was in high school, Linux fanboyism was at its peak and just like people get all wound up on bands and such, us geeks got wound up on open-source and linux and fck Micro$oft etc. etc. This was early-ish 2000's. As a result. Every serious programmer I know, especially those who are about my age, lives their life in the CLI. It always comes a surpr…

15 years ago when i first started programming i learned on the *nix CLI. I worked that way for years, thats just how it was done. Well, I started using an IDE in windows and now.... I really like it. I don't want to go back to the command line. That being said, I still use the Windows command line from time to time.

Yeah me too. I do quite a bit of 'bit bigger than average' data processing, and I write the tools in cmd.exe or as command line C++ applications first - and then when they work and I need to run them more often than a few times, I made a small GUI in front of it so that I don't have to spend half an hour every time to set up the parameters right.

Re: Command-line tools can be faster than your Hadoop cluster

#212

Post by charltones »

threeseed wrote: post #90
MrBuddyCasino wrote: post #61

To quote the memorable Ted Dziuba[0]: "Here's a concrete example: suppose you have millions of web pages that you want to download and save to disk for later processing. How do you do it? The cool-kids answer is to write a distributed crawler in Clojure and run it on EC2, handing out jobs with a message queue like SQS or ZeroMQ. The Taco Bell answer? xargs and wget. In the rare case that you saturate the network conn…

Oh right the "cool kids" approach. Here's what the "sensible adults" think about when they see problems like this. Operational Supportability: How do you monitor the operation ? Restart Recovery: Do you have the ability to restart the operation mid way through if something fails ? Maintainability: Can we run the same application on our desktop as on our production servers ? Extensibility: Can we extend the platform e…

There are command line tools available to help the transition from 'hack' one liner to a more maintainable / supportable solution. For instance drake (https://github.com/Factual/drake) a 'Make for data' which does dependency checking would allow for sensible restarts of the pipeline.

The O'Reilly Data Science at the Command Line book (linked elsewhere in the comments) has a good deal to say on the subject: turning one liners into extensible shell scripts, using drake, using Gnu Parallel.

Re: Command-line tools can be faster than your Hadoop cluster

#213

Post by mrweasel »

crcsmnky wrote: post #7

Perhaps I'm missing something. It appears that the author is recommending against using Hadoop (and related tools) for processing 3.5GB of data. Who in the world thought that would be a good idea to begin with? The underlying problem here isn't unique to Hadoop. People who are minimally familiar with how technology works and who are very much into BuzzWords™ will always throw around the wrong tool for the job so they…

People love the idea of being big and having "big problems". Wanting to use Hadoop isn't that different from wanting to use all sorts of fancy "web-scale" databases.

Most of us don't have scaling issues or big data, but that sort of excludes us from using all the fancy new tools that we want to play with. I'm still convinced that most of the stuff I work on at work could be run on SQLite, with designed a bit more careful.

The truth is that most of us will never do anything that couldn't be solved with 10 year old technology. And honestly we should happy, there's a certain comfort in being able to use simple and generally understood tools.

Re: Command-line tools can be faster than your Hadoop cluster

#214

Post by DougBTX »

yuubi wrote: post #176
yourad_io wrote: post #169

Earlier quoted context omitted.

> The problem with shell scripting is that nearly nobody is very, very good at it. The Steam bug doing an rm -rf / is an example The steam bug is an example of of utter incompetence; not of someone not being very, very good at it. Whoever is happy with shipping `rm -rf $VAR/` without extreme checking around it should get their computer driving license revoked. > The shell is just not suitable for extremely robust pro…

> rm -rf $VAR/ > / facepaw.jpg Without the trailing slash, null or undefined $VAR would cause an error instead of a request to delete all the things .

The Steam line was more like ` rm -rf $VAR/* `, eg, they didn't want to delete the $VAR directory. Still, ` rm -rf /* ` is no fun.

Re: Command-line tools can be faster than your Hadoop cluster

#215

Post by reitanqild »

mech4bg wrote: post #128
kyllo wrote: post #65

Earlier quoted context omitted.

Windows is just inherently GUI-centric, if you force developers to use Windows they are naturally going to gravitate toward using GUI based tools because Windows command line is so crippled. Tools like Cygwin and Chocolatey are nice in a pinch but they just don't compare to being in a real UNIX environment.

Why use a loaded word like 'force'? Developing on Windows was excellent. Microsoft provides great tools and support; it was incredibly productive. The only real negative is the licensing requirements. As of 2011 (I couldn't find more recent data publicly available) Windows was far and away the most popular development environment. http://www.cnet.com/news/coders-choosing-mac-os-over-linux-e... It's interesting to see…

> The only real negative is the licensing requirements

Fix: Some of the real negatives are:

* the licensing requirements,

* lack of virtual desktops,

* having to manually update everything except the office suite and the OS itself

* having to use a different platform than what is used in production

* many standard tools works slower / is less tested / etc

(PS: I'm working happily from a Windows workstation now.)

Re: Command-line tools can be faster than your Hadoop cluster

#216

Post by rakoo »

eru wrote: post #159
nine_k wrote: post #151

Earlier quoted context omitted.

Unix tools are composable. Functional languages (e.g. Clojure) are all about composability. While bash might be a reasonable glue language, I wonder why Clojure wouldn't be — and it could probably be as compact, if not terser. The problem of the Hadoop approach is that the overhead of parallelization over multiple hosts is serious, and the task fits one machine neatly. A few GBs of data can and should be processed on…

> [...] I wonder why Clojure wouldn't be — and it could probably be as compact, if not terser. Because Clojure is a goo language, that question depends mostly on the libraries available for Clojure. (Whereas some other languages are worse at gluing, so libraries will only help you so far.)

That's if you want to do everything in Clojure, but if Clojure were to be used as a glue language, it seems to me it has a clear syntax to do it :

http://clojuredocs.org/clojure.java.shell/sh

Someone even went further to make it more useful:

https://github.com/Raynes/conch

Re: Command-line tools can be faster than your Hadoop cluster

#217

Post by ajuc »

LeoPanthera wrote: post #147
nextos wrote: post #127

Earlier quoted context omitted.

I love Unix, but it's just a local minima in the design space. For example, it's typical text processing pipelines are hard to branch. I have hacked up some solutions, but never found them very elegant. I would love to hear some solutions to this. Ended up switching to Clojure (Prismatic's) Graph.

> For example, it's typical text processing pipelines are hard to branch. I'm not entirely sure what you mean by this, but it sounds like you should use "tee" pointing at a fifo.

The problem - you have file, you want to do one thing for lines matching REGEX and other thing for lines not-matching REGEX.

How to do it without iterating the file 2 times? You can do while of course, but it defeats the reason to use shell.

I would love to have two-way grep that writes matching lines to stdout and nonmatching to stderr. I wonder if grep maintainers would accept new option for grep "--two-way".

Re: Command-line tools can be faster than your Hadoop cluster

#218

Post by vidarh »

mech4bg wrote: post #128
kyllo wrote: post #65

Earlier quoted context omitted.

Windows is just inherently GUI-centric, if you force developers to use Windows they are naturally going to gravitate toward using GUI based tools because Windows command line is so crippled. Tools like Cygwin and Chocolatey are nice in a pinch but they just don't compare to being in a real UNIX environment.

Why use a loaded word like 'force'? Developing on Windows was excellent. Microsoft provides great tools and support; it was incredibly productive. The only real negative is the licensing requirements. As of 2011 (I couldn't find more recent data publicly available) Windows was far and away the most popular development environment. http://www.cnet.com/news/coders-choosing-mac-os-over-linux-e... It's interesting to see…

> Why use a loaded word like 'force'?

Because a lot of us would always pick alternatives when there's a choice. In fact, some of us won't take jobs where we have to develop on Windows.

Re: Command-line tools can be faster than your Hadoop cluster

#219

Post by gambiting »

collyw wrote: post #201
kansface wrote: post #154

Earlier quoted context omitted.

The sorts of bugs people experience with Java mostly result in a crashed/stalled/hung process. Bash bugs erase your entire file system. The thing about Bash is that it is trivially easy to make these sorts of mistakes- the language just isn't suitable to general purpose scripting.

It shouldn't be able to erase your filesystem unless you are running as root or doing something equally stupid. That's pretty much common sense stuff for anyone that isn't a beginner.

Yeah the "common sense stuff for anyone that isn't a beginner" argument is repeated ad nauseam, and even the largest companies make this mistake in their largest products. Take Valve - they should know how to write good code, right? And yet, last week an article was on top of HN, outlining how they put:

"rm -rf '$STEAMROOT'/*" in their code, used to remove the library. But hey, no one checked if $STEAMROOT is not empty, so when it was for one user, Steam deleted all of his personal files, whole /home and /media directories next time it started.

I'm not saying that command line tools shouldn't be used,but sometimes they are just too powerful for some users,and stupid mistakes like this happen.

Re: Command-line tools can be faster than your Hadoop cluster

#220

Post by _delirium »

reitanqild wrote: post #215
mech4bg wrote: post #128

Earlier quoted context omitted.

Why use a loaded word like 'force'? Developing on Windows was excellent. Microsoft provides great tools and support; it was incredibly productive. The only real negative is the licensing requirements. As of 2011 (I couldn't find more recent data publicly available) Windows was far and away the most popular development environment. http://www.cnet.com/news/coders-choosing-mac-os-over-linux-e... It's interesting to see…

> The only real negative is the licensing requirements Fix: Some of the real negative s are : * the licensing requirements, * lack of virtual desktops, * having to manually update everything except the office suite and the OS itself * having to use a different platform than what is used in production * many standard tools works slower / is less tested / etc (PS: I'm working happily from a Windows workstation now.)

> lack of virtual desktops

They're not built in, but a few of my colleagues use Dexpot and like it: http://www.dexpot.de/index.php?lang=en

Post reply on HN