Command-line tools can be faster than your Hadoop cluster • Hacker News Board

Re: Command-line tools can be faster than your Hadoop cluster

#241

Post by kyllo » Mon, Jan 19, 2015, 2:11 PM UTC

mech4bg wrote: post #128

kyllo wrote: post #65

Earlier quoted context omitted.

Windows is just inherently GUI-centric, if you force developers to use Windows they are naturally going to gravitate toward using GUI based tools because Windows command line is so crippled. Tools like Cygwin and Chocolatey are nice in a pinch but they just don't compare to being in a real UNIX environment.

Why use a loaded word like 'force'? Developing on Windows was excellent. Microsoft provides great tools and support; it was incredibly productive. The only real negative is the licensing requirements. As of 2011 (I couldn't find more recent data publicly available) Windows was far and away the most popular development environment. http://www.cnet.com/news/coders-choosing-mac-os-over-linux-e... It's interesting to see…

I say "force" because a lot of employers simply do not allow developers to choose their workstation OS. Windows is very popular with the non-programmers responsible for making IT purchasing deals within most major corporations. Among programmers, it's not quite so popular.

Re: Command-line tools can be faster than your Hadoop cluster

#242

Post by JustSomeNobody » Mon, Jan 19, 2015, 2:12 PM UTC

threeseed wrote: post #83

EpicEng wrote: post #10

Earlier quoted context omitted.

Exactly this just happened where I work. The CIO was recommending Hadoop on AWS for our image processing/analysis jobs. We process a single set of images at a time which come in around ~1.5GB. The output data size is about 1.2GB. Not a good candidate for Hadoop but, you know... "big data", right?

Another explanation is that your CIO is not an idiot but rather they know about future projects that you don't. CIOs want to build capabilities (skills and technologies) not just one off implementations every time. Not saying this is the case but CIO bashing is all too easy when you're an engineer.

The last people you want to keep secrets from are your engineers.

Re: Command-line tools can be faster than your Hadoop cluster

#243

Post by impostervt » Mon, Jan 19, 2015, 2:31 PM UTC

Any decent tutorials out there to get me up to speed on CL tools? I use grep and a few others regularly, but have avoided sed and awk as they seem difficult to jump into.

Re: Command-line tools can be faster than your Hadoop cluster

#244

Post by fulafel » Mon, Jan 19, 2015, 2:32 PM UTC

ajdecon wrote: post #57

kunstmord wrote: post #22

Earlier quoted context omitted.

I think I read this somewhere here a few months ago (paraphrasing, obviously): "When the indices for your DB don't fit into a single machines RAM, then you're dealing with Big Data, not before."

And following up: Your laptop does not count as a "single machine" for purposes of RAM size. If you can fit the index of your DB in memory on anything you can get through EC2, it's still not Big Data.

There's still 40x difference to biggest EC2 instance to a maxed out Dell server (244 GB EC2 vs 6 TB for a R920). Not to mention non-PC hardware like SPARC, POWER and SGI UV systems that fit even more.

Re: Command-line tools can be faster than your Hadoop cluster

#245

Post by TallGuyShort » Mon, Jan 19, 2015, 2:51 PM UTC

dundun wrote: post #135

smegel wrote: post #47

What about if you are processing 100 Petabytes? And you are comparing to a 1000-node Hadoop cluster with each node running 64 cores and 1TB of main memory?

Then you're hardly using commodity hardware anymore. While jobs like that probably actually work on Hadoop, I'd imagine a problem like that might be better suited for specialized systems.

IME, most installations where Hadoop is "successfully" used it's running on pretty high-end machines. "Commodity hardware" really means standard hardware, not cheap hardware (as opposed to buying proprietary appliances and mainframes).

Re: Command-line tools can be faster than your Hadoop cluster

#246

Post by barrkel » Mon, Jan 19, 2015, 2:52 PM UTC

ajuc wrote: post #217

LeoPanthera wrote: post #147

Earlier quoted context omitted.

> For example, it's typical text processing pipelines are hard to branch. I'm not entirely sure what you mean by this, but it sounds like you should use "tee" pointing at a fifo.

The problem - you have file, you want to do one thing for lines matching REGEX and other thing for lines not-matching REGEX. How to do it without iterating the file 2 times? You can do while of course, but it defeats the reason to use shell. I would love to have two-way grep that writes matching lines to stdout and nonmatching to stderr. I wonder if grep maintainers would accept new option for grep "--two-way".

Write to more than one fifo from awk. If you're composing a dag rather than a pipeline, fifos are one way to go.

Personally though, I'd output to temporary files. The extra cost in disk usage and lack of pipelining is made up for by the easier debugging, and most shell pipelines aren't so slow that they need that level of optimization.

Re: Command-line tools can be faster than your Hadoop cluster

#247

Post by JabavuAdams » Mon, Jan 19, 2015, 2:58 PM UTC

nilbot wrote: post #198

Kiro wrote: post #197

Earlier quoted context omitted.

And I can't stand developers who overengineer things. We have a couple of them at my company and something that should take a few hours always take several weeks just because of all the reasons you mention. Most things don't need that kind of features and maintainability and if they do in the future we can just rewrite them from scratch. The overall expected return on investment is still better since we seldom need t…

Quite the opposite, and, quite simple: engineers over-engineer thing in order to make things generic. and generic make solutions robust. that's basic science. Unless the problem and solution are well understood, your investment won't guarantee a return at all.

No. Look to safety-critical software for intuition on why.

Simpler is more reliable. Also, it's hard to know enough about a problem to make a generic solution until you've solved the problem 2-3 times already. But ... having solved a problem multiple times increases the risk that you will be biased towards seeing new problems as some instance of the old problem and therefore applying unsuitable "generic" solutions.

Re: Command-line tools can be faster than your Hadoop cluster

#248

Post by lucaspottersky » Mon, Jan 19, 2015, 3:00 PM UTC

linuxhansl wrote: post #138

So don't use Hadoop to crunch data that fits on a memory stick, or that a single disk spindle can read in few seconds. Why is this first on the HN front-page? Reminds me of the C++ is better than Java, Go is better than C++, etc, pieces. Yes, the right tool for the right job. That's what makes a good engineer. Somebody who thinks there is _no_ valid use case for Hadoop is a fool. (The author did not say that, but man…

> Why is this first on the HN front-page?

Because controversial topics are always fun! d:-)

Re: Command-line tools can be faster than your Hadoop cluster

#249

Post by JabavuAdams » Mon, Jan 19, 2015, 3:01 PM UTC

nilbot wrote: post #206

gaius wrote: post #199

Earlier quoted context omitted.

No, developers over-engineer because setting up a 20-node Hadoop cluster is fun, whereas doing the same task in an hour in Excel means you have to move onto some other boring task. Generic doesn't mean robust either, I don't know where you got that from,the two concepts are entirely unrelated.

Generic -> robust. i... i dont know how to explain that. honestly i haven't thought about the necessity of explaining things like this. its... basic mathematics.

This smacks of an unexamined bias. Or maybe we're not using the words to mean the same things?

Re: Command-line tools can be faster than your Hadoop cluster

#250

Post by JabavuAdams » Mon, Jan 19, 2015, 3:04 PM UTC

sedachv wrote: post #70

x0x0 wrote: post #23

Earlier quoted context omitted.

I've used hadoop at petabyte scale (2+pb input; 10+pb sorted for the job) for machine learning tasks. If you have such a thing on your resume, you will be inundated with employers who have "big data", and at least half will be under 50g with a good chunk of those under 10g. You'll also see multiple (shitty) 16 machine clusters, any of which -- for any task -- could be destroyed by code running on a single decent serv…

> Also, hadoop is so painfully slow to develop in it's practically a full employment act for software engineers. It's comical how bad Hadoop is compared even to the CM Lisp described in Daniel Hillis' PhD dissertation. How do you devolve all the way from that down to "It's like map/reduce. You get one map and one reduce!"

Programming is very faddish. It's amazing how bad commonly used technologies are. I'm so happy I'm mostly a native developer and don't have to use the shitty web stack and its shitty replacements.