Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Command-line tools can be faster than your Hadoop cluster

aadrake.com

Post reply on HN
Print view View original

281–290 of 315 posts

Re: Command-line tools can be faster than your Hadoop cluster

#281

Post by devonkim »

rhizome wrote: post #46
underpants wrote: post #25

Earlier quoted context omitted.

In a similar situation. In fact stupider. We have a 120Gb baseline of data inside a relational store. The vendor has a file stream option that allows blobs to be stored on disk instead of the transaction log and be pushed through the DB server rather than using our current CIFS DFS cluster. So lets stick our 950Gb static document load in there too (while i was on holiday typically) and off we go. Do thing starts goin…

Resume-Driven Development

No, it's called enterprise software, and it's driven by "appeal to the enormous egos and matching budgets of leaders of companies that are held back more by their own bloat than by anything resembling competition." This is just standard practice for the enterprise software sector in general, and after everyone's spent their billions and the next recession is upon us, they'll probably axe most of these utter failures to produce business value. But all the consultants will just blame someone other than the client being dysfunctional because that's just the fastest way to get kicked off a contract.

Consulting for enterprise customers tends to be a lot like marriages - you can be right, or you can be happy (or paid). It takes a unique customer to have gotten past their cultural dysfunctions to accept responsibilities for their problems and to take legitimate, serious action. But like marriage, there can be great, great upsides when everyone gets on the same page and works towards mutual goals with the spirit of selflessness and growth. Yeah....

Re: Command-line tools can be faster than your Hadoop cluster

#282

Post by ajdecon »

fulafel wrote: post #244
ajdecon wrote: post #57

Earlier quoted context omitted.

And following up: Your laptop does not count as a "single machine" for purposes of RAM size. If you can fit the index of your DB in memory on anything you can get through EC2, it's still not Big Data.

There's still 40x difference to biggest EC2 instance to a maxed out Dell server (244 GB EC2 vs 6 TB for a R920). Not to mention non-PC hardware like SPARC, POWER and SGI UV systems that fit even more.

This is true, but at the upper end the "it isn't Big Data if it fits in a single system's memory" rule starts to get fuzzy. If you're using an SGI UV 2000 with 64 TB of memory to do your processing, I'm not going to argue with you about using the words "Big Data". ;-) I figured using an EC2 instance was a decent compromise.

Re: Command-line tools can be faster than your Hadoop cluster

#283

Post by guardiangod »

Here is an analysis from a developer who looked at Hadoop- http://ossectools.blogspot.ca/2012/03/why-elsa-doesnt-use-ha...

(ELSA is a logger that claims to be able to handle 100000 entries/sec (!!))

When to Use Hadoop

This is a description of why Hadoop isn't always the right solution to Big Data problems, but that certainly doesn't mean that it's not a valuable project or that it isn't the best solution for a lot challenges. It's important to use the right tool for the job, and thinking critically about what features each tool provides is paramount to a project's success. In general, you should use Hadoop when:

    Data access patterns will be very basic but analytics will be very complicated.
    Your data needs absolutely guaranteed availability for both reading and writing.
    There are inadequate traditional database-oriented tools which currently exist for your problem.
Do not use Hadoop if:

    You're don't know exactly why you're using it.
    You want to maximize hardware efficiency.
    Your data fits on a single "beefy" server.
    You don't have full-time staff to dedicate to it.
The easiest alternative to using Hadoop for Big Data is to use multiple traditional databases and architect your read and write patterns such that the data in one database does not rely on the data in another. Once that is established, it is much easier than you'd think to write basic aggregation routines in languages you're already invested in and familiar with. This means you need to think very critically about your app architecture before you throw more hardware at it.

Re: Command-line tools can be faster than your Hadoop cluster

#284

Post by knodi123 »

We have a proprietary algorithm for assigning foods a "suitability score" based on a user's personal health conditions and body data.

It used to be a fairly slow algorithm, so we ran it in a hadoop cluster and it cached the scores for every user vs. every food in a massive table on a distributed database.

Another developer, who is quite clever, rewrote our algorithm in C, and compiled it as a database function, which was about 100x faster. He also did some algebra work and found a way to change our calculations, yielding a measly 4-5x improvement.

It was so, so, so much faster that in one swoop we eliminated our entire Hadoop cluster, and the massive scores table, and were actually able sort your food search results by score, calculating scores on the fly.

Re: Command-line tools can be faster than your Hadoop cluster

#285

Post by juliangregorian »

earino wrote: post #177
rkwasny wrote: post #97

Bottom line is - you do not need hadoop until you cross 2TB of data to be processed (uncompressed). Modern servers ( bare metal ones, not what AWS sells you ) are REALLY FAST and can crunch massive amounts of data. Just use a proper tools, well optimized code written in C/C++/Go/etc - not all the crappy JAVA framework-in-a-framework^N architecture that abstracts thinking about the CPU speed. Bottom line, the popular…

Dell sells a server with 6TB of ram (I believe.) I think the limit is way over 2TB. If you want to be able to query it quickly for analytical workloads, MPPs like Vertica scale up to 150+TB (at Facebook.) I honestly don't know what the scale is where you need Hadoop, but it's gotten to be a large number very quickly.

They do, I checked. It comes in at a cool half million (Helloooo, investors!)

Re: Command-line tools can be faster than your Hadoop cluster

#286

Post by jodrellblank »

angrybits wrote: post #126
pjc50 wrote: post #101

Earlier quoted context omitted.

Windows Powershell is actually quite nice and powerful, while also avoiding some of the shell legacy traps around escaping. It suffers from not being very discoverable and not having a community.

I wish they had just been less stubborn and made something that would run bash and standard unix commands. I've used it for production jobs and it's worked as advertised, but I would have rather have just had bash.

They weren't stubborn, they set out to "give windows a Unix command line" but then they discovered how unsuitable a Unix command line is, and had to rethink.

See the original lead architects comments here: http://stackoverflow.com/a/573861

Re: Command-line tools can be faster than your Hadoop cluster

#287

Post by tokenrove »

sebastianavina wrote: post #234
lloeki wrote: post #186

Earlier quoted context omitted.

Alternative, real life scenario: navigate through 6 months of daily MySQL dumps, assorted YAML files and Rails production.log, looking for some cross product between tables, requests and serialised entities, for analysis and/or data recovery (pinpoint or retrieval). zcat/cut/sed/grep/awk/perl crawled through it in a couple of minutes and required less than half an hour to craft a reliable enough implementation (inclu…

to be fair, you could have achieved all that with a simple python script. sometimes i feel python is the new bash.

Note that you lose some of the parallelism you get effortlessly from the Unix pipeline if you do it as a single (simple) Python script.

Re: Command-line tools can be faster than your Hadoop cluster

#288

Post by tokenrove »

mech4bg wrote: post #128
kyllo wrote: post #65

Earlier quoted context omitted.

Windows is just inherently GUI-centric, if you force developers to use Windows they are naturally going to gravitate toward using GUI based tools because Windows command line is so crippled. Tools like Cygwin and Chocolatey are nice in a pinch but they just don't compare to being in a real UNIX environment.

Why use a loaded word like 'force'? Developing on Windows was excellent. Microsoft provides great tools and support; it was incredibly productive. The only real negative is the licensing requirements. As of 2011 (I couldn't find more recent data publicly available) Windows was far and away the most popular development environment. http://www.cnet.com/news/coders-choosing-mac-os-over-linux-e... It's interesting to see…

> Developing on Windows was excellent. Microsoft provides great tools and support; it was incredibly productive.

I don't know about that. I've made several good faith efforts to really see what people like about the Windows development ecosystem, and I consistently come away dismayed. "Great tools and support" could never be used to describe msbuild, for example. Or any of the MSDN documentation with incredible antipattern code that people like to blithely copy and paste into their programs. Visual Studio is slow, brittle, and makes it difficult to do version control right.

I could list examples for days but that's not the point. Nowhere is perfect, but there's no way anyone could consider Microsoft a clear leader here.

Re: Command-line tools can be faster than your Hadoop cluster

#289

Post by tokenrove »

pjc50 wrote: post #101
kyllo wrote: post #65

Earlier quoted context omitted.

Windows is just inherently GUI-centric, if you force developers to use Windows they are naturally going to gravitate toward using GUI based tools because Windows command line is so crippled. Tools like Cygwin and Chocolatey are nice in a pinch but they just don't compare to being in a real UNIX environment.

Windows Powershell is actually quite nice and powerful, while also avoiding some of the shell legacy traps around escaping. It suffers from not being very discoverable and not having a community.

It also has a surprising number of WTFs, which was a great disappointment to me because it really seemed like MS had done scripting better. Of course, later versions fix some of the pitfalls if you use them right, but then you need to make sure you're always running machines that will have the latest Powershell on them.

(And, of course, it would have been nice if they could improve its interactive use to even be on par with the Unix shells of the 80s)

Post reply on HN