Command-line tools can be faster than your Hadoop cluster • Hacker News Board

Re: Command-line tools can be faster than your Hadoop cluster

#291

Post by alrs » Tue, Jan 20, 2015, 1:44 AM UTC

recursive wrote: post #152

alrs wrote: post #105

Earlier quoted context omitted.

The example was a multi-gigabyte CSV file. You just sucked the whole thing off the disk into RAM so that you could shave off the first line. If you're unlucky, you started swapping out to disk about halfway through.

That code you're replying about was carefully and correctly written. You just replied as if you know how it works just so you could look like you know what you're talking about. If you're unlucky, someone who actually knows how File.ReadLines() works will show up in an hour or two and explain that it's lazily evaluated.

:) touche

Re: Command-line tools can be faster than your Hadoop cluster

#292

Post by saym » Tue, Jan 20, 2015, 1:56 AM UTC

knodi123 wrote: post #284

We have a proprietary algorithm for assigning foods a "suitability score" based on a user's personal health conditions and body data. It used to be a fairly slow algorithm, so we ran it in a hadoop cluster and it cached the scores for every user vs. every food in a massive table on a distributed database. Another developer, who is quite clever, rewrote our algorithm in C, and compiled it as a database function, which…

May I ask: Who is we?

Re: Command-line tools can be faster than your Hadoop cluster

#293

Post by gpapilion » Tue, Jan 20, 2015, 2:57 AM UTC

lloeki wrote: post #189

gpapilion wrote: post #110

Earlier quoted context omitted.

I look at this article as a criticism of the hadoop being the wrong tool for small data sets. This starts to become a question of data locality, and size. 1.75 GB isn't enough data to justify a hadoop solution. That data size fits easily in memory, and without doubt on a single system. From that point you only need some degree of parallelism to maximize the performance. That being said when its 35TB of data, the answ…

> That data size fits easily in memory, and without doubt on a single system. From that point you only need some degree of parallelism to maximize the performance. That being said when its 35TB of data, the answer starts to change. Not at all, because data is being streamed . It could just as easily be 35TB and only use a few MB of RAM.

The IO bandwidth of the system will limit you more loading 35TB of data in ram on a single system, even if it is streamed. You'll need more than one disk, and network card to do this in a timely fashion.

Re: Command-line tools can be faster than your Hadoop cluster

#294

Post by gpapilion » Tue, Jan 20, 2015, 3:19 AM UTC

microtonal wrote: post #205

gpapilion wrote: post #110

Earlier quoted context omitted.

I look at this article as a criticism of the hadoop being the wrong tool for small data sets. This starts to become a question of data locality, and size. 1.75 GB isn't enough data to justify a hadoop solution. That data size fits easily in memory, and without doubt on a single system. From that point you only need some degree of parallelism to maximize the performance. That being said when its 35TB of data, the answ…

1.75 GB isn't enough data to justify a hadoop solution. That data size fits easily in memory, and without doubt on a single system. It depends on what you do with the data. If you are processing the data in 512KB chunks and each chunk takes a day to process (because expensive computation), you probably do want to spread the work over some cluster.

I don't think of hadoop being built for high complexity computation, but high IO throughput.

When you describe this kind of setup, I imagine things that involve proof through exhaustion. For example prime number search is something with a small input and large calculation time. However, these solution don't really benefit from hadoop since you don't really need the data management facilities, and a simpler MPI solution could handle this better.

Search indexing could fit this description(url -> results), but generally you want the additional network cards for throughput, and the disks to store the results. Then again the aggregate space on disk starts looking closer to TB instead of GB. Plus in the end you need to do something with all those pages.

Re: Command-line tools can be faster than your Hadoop cluster

#295

Post by snambi » Tue, Jan 20, 2015, 6:29 AM UTC

Roboprog wrote: post #275

saidajigumi wrote: post #103

Earlier quoted context omitted.

> Dreadful for the long term. Here comes a bubble-bursting: I've lead a team that built data processing tools exactly like this, and the performance and ease of manipulating vast amounts of text using classic shell tools is hard to beat. We had no problems with any of: operational supportability, restart recovery, or maintainability. Highly testable, even. No, it's not just cowboy-coded crappy shell scripts and pipel…

Unix/POSIX backgrounds are pretty common among the Hacker News crowd. Not so in "Enterprise" development. (Beam me up Scottie, there's no intelligent life here, only risk avoidance) Enterprise development is predominated by 2 or 3 trusted operating systems: Windows (/ .NET), and the JVM. POSIX systems are only useful in-so-far-as they are a cheaper (or sometimes more reliable) place to host Java virtual machines. Ent…

I agree with @roboprog. Most software shops employ engineers who don't have exposure into UNIX tools. Only few hardcore engineers have exposure or interest learning UNIX tools. For majority of engineers it is just a job. They simply use the same tool for everything. And they tend to use the tools that seem to get them into well paying jobs. If hadoop can get them good paying jobs, they would like us "hadoop" for something in their current job, even if that job can performed by a set of CLI utils. I have seen 100s of resume builder projects in the my past experience.

Re: Command-line tools can be faster than your Hadoop cluster

#296

Post by jdmichal » Tue, Jan 20, 2015, 9:29 AM UTC

Roboprog wrote: post #275

saidajigumi wrote: post #103

Earlier quoted context omitted.

> Dreadful for the long term. Here comes a bubble-bursting: I've lead a team that built data processing tools exactly like this, and the performance and ease of manipulating vast amounts of text using classic shell tools is hard to beat. We had no problems with any of: operational supportability, restart recovery, or maintainability. Highly testable, even. No, it's not just cowboy-coded crappy shell scripts and pipel…

Unix/POSIX backgrounds are pretty common among the Hacker News crowd. Not so in "Enterprise" development. (Beam me up Scottie, there's no intelligent life here, only risk avoidance) Enterprise development is predominated by 2 or 3 trusted operating systems: Windows (/ .NET), and the JVM. POSIX systems are only useful in-so-far-as they are a cheaper (or sometimes more reliable) place to host Java virtual machines. Ent…

Windows has PowerShell, which can be even more powerful than Unix shell depending on the data and processing required.

Re: Command-line tools can be faster than your Hadoop cluster

#297

Post by sgt101 » Tue, Jan 20, 2015, 10:17 AM UTC

There's a perception that hadoop commands are terribly complex. If you run

$spark-shell

you can execute (interactively)

val file = spark.textFile("hdfs://...") val errors = file.filter(line => line.contains("ERROR")) errors.count()

And wordcount a file - ok the wget is not there, but this is really not complex!

Re: Command-line tools can be faster than your Hadoop cluster

#298

Post by PhasmaFelis » Tue, Jan 20, 2015, 11:10 AM UTC

yourad_io wrote: post #169

developer1 wrote: post #140

Earlier quoted context omitted.

The problem with shell scripting is that nearly nobody is very, very good at it. The Steam bug doing an rm -rf / is an example, but it's very common for shell scripts to have horrible error handling and checks for important things. The shell is just not suitable for extremely robust programs. I would bet that 80%+ of people who think they're good at shell scripting... aren't.

> The problem with shell scripting is that nearly nobody is very, very good at it. The Steam bug doing an rm -rf / is an example The steam bug is an example of of utter incompetence; not of someone not being very, very good at it. Whoever is happy with shipping `rm -rf $VAR/` without extreme checking around it should get their computer driving license revoked. > The shell is just not suitable for extremely robust pro…

> The same probably goes for driving[1], this doesn't make cars any less robust.

I don't think I can imagine anything less robust than cars, in terms of the frequency and severity of operational failure. They're pretty much the deadliest thing we've ever invented that wasn't actually designed to kill people.

It's actually a good example of the point developer1 was making: cars and shell scripts are perfectly safe if operated by highly competent people, and only become (extremely) dangerous when operated by incompetents, but in practice most operators are incompetent, in denial, and refuse to learn from others' mistakes.

Re: Command-line tools can be faster than your Hadoop cluster

#299

Post by mercurial » Tue, Jan 20, 2015, 11:18 AM UTC

MrBuddyCasino wrote: post #61

To quote the memorable Ted Dziuba[0]: "Here's a concrete example: suppose you have millions of web pages that you want to download and save to disk for later processing. How do you do it? The cool-kids answer is to write a distributed crawler in Clojure and run it on EC2, handing out jobs with a message queue like SQS or ZeroMQ. The Taco Bell answer? xargs and wget. In the rare case that you saturate the network conn…

You could just use Scrapy [1]. Easy to setup, and plenty of options you can activate if needed. Likely more robust than shell scripts as well. No Hadoop involved.

1: http://doc.scrapy.org/en/0.24/intro/tutorial.html

Re: Command-line tools can be faster than your Hadoop cluster

#300

Post by yuanchuan » Tue, Jan 20, 2015, 2:39 PM UTC

pacala wrote: post #256

yuanchuan wrote: post #200

Earlier quoted context omitted.

It is that buzz surrounding Hadoop that makes people misunderstood its use and capability. I have met non-technical analysts who want RDBMS performance on Hadoop. They expect seconds to minutes scale queries on hundreds of GB of data. I always throw this analogy to people who misunderstood Hadoop: A stone to crack an egg or a spoon? Hadoop and RDBMS only have a thin overlapping region in the Venn diagram that describ…

> They expect seconds to minutes scale queries on hundreds of GB of data. Use BigQuery from Google.

On-premise cluster.

Cloud solution are totally out due to the nature of the data. Not everything can be done in cloud.

If you have such huge amount of data, the total amount of time it takes to transfer there and compute is not as competitive as an on-premise solution, unless all your data live in the cloud.