Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Command-line tools can be faster than your Hadoop cluster

aadrake.com

Post reply on HN
Print view View original

301–310 of 315 posts

Re: Command-line tools can be faster than your Hadoop cluster

#301

Post by TallGuyShort »

MichaelGG wrote: post #262
TallGuyShort wrote: post #245

Earlier quoted context omitted.

IME, most installations where Hadoop is "successfully" used it's running on pretty high-end machines. "Commodity hardware" really means standard hardware, not cheap hardware (as opposed to buying proprietary appliances and mainframes).

Or it could be a company with 30M records a month that buys 100 x $200 servers off eBay and is still unable to query their data.

I'm not sure what your point is with a hypothetical situation. Why wouldn't the be able to query their data? All I'm saying is from my actual experience with real users, it's best to build a Hadoop cluster with high quality hardware if you can.

Re: Command-line tools can be faster than your Hadoop cluster

#302

Post by pacala »

yuanchuan wrote: post #300
pacala wrote: post #256

Earlier quoted context omitted.

> They expect seconds to minutes scale queries on hundreds of GB of data. Use BigQuery from Google.

On-premise cluster. Cloud solution are totally out due to the nature of the data. Not everything can be done in cloud. If you have such huge amount of data, the total amount of time it takes to transfer there and compute is not as competitive as an on-premise solution, unless all your data live in the cloud.

I would look into https://spark.apache.org/ then. You can get quite good performance out of it, but you need to spend more effort in babysitting your data.

Re: Command-line tools can be faster than your Hadoop cluster

#303

Post by dynode »

tokenrove wrote: post #287
sebastianavina wrote: post #234

Earlier quoted context omitted.

to be fair, you could have achieved all that with a simple python script. sometimes i feel python is the new bash.

Note that you lose some of the parallelism you get effortlessly from the Unix pipeline if you do it as a single (simple) Python script.

I use the multiprocessing module in python all the time for quick parallelism " rel="nofollow">https://medium.com/@thechriskiehl/parallelism-in-one-line-40... Let's you easily map to multiple cores. I use it a lot for image processing tasks. Quickly crawl through directories to find all the files, then spin up all the cores on my machine to crunch through them. Wish there was an easy way to enlist multiple machines.

Re: Command-line tools can be faster than your Hadoop cluster

#305

Post by jodrellblank »

impostervt wrote: post #243

Any decent tutorials out there to get me up to speed on CL tools? I use grep and a few others regularly, but have avoided sed and awk as they seem difficult to jump into.

A couple of days ago this clear, brief introduction to AWK was submitted: http://ferd.ca/awk-in-20-minutes.html

Re: Command-line tools can be faster than your Hadoop cluster

#306

Post by jbergens »

kyllo wrote: post #241
mech4bg wrote: post #128

Earlier quoted context omitted.

Why use a loaded word like 'force'? Developing on Windows was excellent. Microsoft provides great tools and support; it was incredibly productive. The only real negative is the licensing requirements. As of 2011 (I couldn't find more recent data publicly available) Windows was far and away the most popular development environment. http://www.cnet.com/news/coders-choosing-mac-os-over-linux-e... It's interesting to see…

I say "force" because a lot of employers simply do not allow developers to choose their workstation OS. Windows is very popular with the non-programmers responsible for making IT purchasing deals within most major corporations. Among programmers, it's not quite so popular.

It still depends on what programmers you ask. Windows works fine for me, Linux did not work well when I tried it (but it was many years ago). Unix through XWindows from a pc was actually quite nice. I think developers like users use feelings more than thoughts when choosing an environment. Yes, you can do some things on Linux that are harder on Windows but saying that missing "grep" is severly limiting you is strange since it is easy to install. And you have find in most editors. You can run Vim or Emacs if you want to on Windows. Powershell is very good but with a strange syntax. If that is someones problem they sound like they don't want to learn something new. Listing thousands of files in a folder is probably still faster on Linux but I rarely do that (I can't see the point in collecting many years of logs in one folder, for example). The main problem I have with Windows is that I have to restart the computer every now and then.

Re: Command-line tools can be faster than your Hadoop cluster

#307

Post by jbergens »

x0x0 wrote: post #23
crcsmnky wrote: post #7

Perhaps I'm missing something. It appears that the author is recommending against using Hadoop (and related tools) for processing 3.5GB of data. Who in the world thought that would be a good idea to begin with? The underlying problem here isn't unique to Hadoop. People who are minimally familiar with how technology works and who are very much into BuzzWords™ will always throw around the wrong tool for the job so they…

I've used hadoop at petabyte scale (2+pb input; 10+pb sorted for the job) for machine learning tasks. If you have such a thing on your resume, you will be inundated with employers who have "big data", and at least half will be under 50g with a good chunk of those under 10g. You'll also see multiple (shitty) 16 machine clusters, any of which -- for any task -- could be destroyed by code running on a single decent serv…

It sounds a little bit like BizTalk :-)

Re: Command-line tools can be faster than your Hadoop cluster

#308

Post by shitgoose »

angrybits wrote: post #125
threeseed wrote: post #104

Earlier quoted context omitted.

Anybody can throw some crap together and make it stick. And it's a perfectly valid solution. My issue is when there is criticism laid against those solutions which are actually engineered in a way that allows for supportability and extensibility. They are arguably far more important than execution time.

I can't figure out why you're arguing against standard unix tools/idioms in the name of supportability and extensibility? It defies logic.

I imagine what would have happened to Linux if Linus designed it with supportability and extensibility in mind.

Re: Command-line tools can be faster than your Hadoop cluster

#309

Post by ec664 »

This is blown out of proportion... actually increase is probably a factor 10-20X, not 100s. The fact that EMR is used is a problem, provisioning, bootstrapping the cluster alone accounts for probably half the time.

The fact that shell commands were run repeatedly means that the data ends up in the OS buffer cache and basically in memory.

I'm not discounting that CLI is faster than Hadoop by an order of magnitude on small datasets. Nor will I dive into Hadoop vs CLI. The answer to all that IMO is that it depends. And in this case, it's not well warranted.

What I do take exception to is the Fox News style headlines that are disproportional to the truth. EMR != Hadoop.

Re: Command-line tools can be faster than your Hadoop cluster

#310

Post by yourad_io »

PhasmaFelis wrote: post #298
yourad_io wrote: post #169

Earlier quoted context omitted.

> The problem with shell scripting is that nearly nobody is very, very good at it. The Steam bug doing an rm -rf / is an example The steam bug is an example of of utter incompetence; not of someone not being very, very good at it. Whoever is happy with shipping `rm -rf $VAR/` without extreme checking around it should get their computer driving license revoked. > The shell is just not suitable for extremely robust pro…

> The same probably goes for driving[1], this doesn't make cars any less robust. I don't think I can imagine anything less robust than cars, in terms of the frequency and severity of operational failure. They're pretty much the deadliest thing we've ever invented that wasn't actually designed to kill people. It's actually a good example of the point developer1 was making: cars and shell scripts are perfectly safe if…

> I don't think I can imagine anything less robust than cars, in terms of the frequency and severity of operational failure.

Maybe US cars :P

> They're pretty much the deadliest thing we've ever invented that wasn't actually designed to kill people.

It's a box weighing 1-2 tons that travels at 100kmh+. Millions (billions?) of km are driven every year. There will be accidents for both good drivers and bad. This won't change.

> cars and shell scripts are perfectly safe if operated by highly competent people, and only become (extremely) dangerous when operated by incompetents, but in practice most operators are incompetent, in denial, and refuse to learn from others' mistakes.

That's simply untrue - both points. Highly competent drivers will have accidents. I highly doubt you feel extreme danger when you get behind the wheel/in a car. The way you phrase it, one expects millions of fatalities daily.

Post reply on HN