Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Command-line tools can be faster than your Hadoop cluster

aadrake.com

Post reply on HN
Print view View original

271–280 of 315 posts

Re: Command-line tools can be faster than your Hadoop cluster

#271

Post by sukilot »

threeseed wrote: post #86
dschiptsov wrote: post #55

Everyone with basic knowledge of CS could realize that Hadoop is a waste. Unfortunately, it isn't about efficiency at all. It just memeization. Bigdata? Hadoop! Runs everywhere. Same BS like Webscale? MongoDB! meme.

Well sorry but you don't have a clue what you're talking about. I very much work in "big data" with about 2 terabytes of new data coming in every day that has to be ingested and processed with hundreds of jobs running against them. The data needs to be queryable via an SQL like language and analyzed by a dozen data scientists using R or Map Reduce. There isn't anything on the market today that has been proven to work…

A few large organizations and microscpic scientists have big data. 99.9% of users do not.

Re: Command-line tools can be faster than your Hadoop cluster

#272

Post by EpicEng »

threeseed wrote: post #83
EpicEng wrote: post #10

Earlier quoted context omitted.

Exactly this just happened where I work. The CIO was recommending Hadoop on AWS for our image processing/analysis jobs. We process a single set of images at a time which come in around ~1.5GB. The output data size is about 1.2GB. Not a good candidate for Hadoop but, you know... "big data", right?

Another explanation is that your CIO is not an idiot but rather they know about future projects that you don't. CIOs want to build capabilities (skills and technologies) not just one off implementations every time. Not saying this is the case but CIO bashing is all too easy when you're an engineer.

I think that hiding requirements from the people building the system would, in fact, be an idiotic move.

Re: Command-line tools can be faster than your Hadoop cluster

#273

Post by mech4bg »

vidarh wrote: post #218
mech4bg wrote: post #128

Earlier quoted context omitted.

Why use a loaded word like 'force'? Developing on Windows was excellent. Microsoft provides great tools and support; it was incredibly productive. The only real negative is the licensing requirements. As of 2011 (I couldn't find more recent data publicly available) Windows was far and away the most popular development environment. http://www.cnet.com/news/coders-choosing-mac-os-over-linux-e... It's interesting to see…

> Why use a loaded word like 'force'? Because a lot of us would always pick alternatives when there's a choice. In fact, some of us won't take jobs where we have to develop on Windows.

Yes, but my point was that there is no evidence that is the majority, nor are they really 'forced' to use it - if it's that much of an issue, just don't take the job (as you say).

Many, many developers have no issue developing on Windows or even enjoy it. There seems to be a mindset amongst certain people that Windows developers are not 'real' developers, which is what I'm arguing against.

Re: Command-line tools can be faster than your Hadoop cluster

#274

Post by sillysaurus3 »

pyre wrote: post #261
sillysaurus3 wrote: post #173

Earlier quoted context omitted.

Open file on disk. See that it's 404. Delete file. Re-run crawler. You'd turn that into code by doing grep -R 404 . or whatever the actual unique error string is and deleting any file containing the error message. (You'd be careful not to run that recursive delete on any unexpected data.) Really, these problems are pretty easy. It's easy to overthink it.

> grep -R 404 This isn't 1995 anymore. When you hit a 404 error, you no longer get Apache's default 404 page. You really can't count on there being any consistency between 404 pages on different sites. If wget somehow stored the header response info to disk (e.g. "FILENAME.header-info") you could whip something up to do what you are suggesting though.

Yeah, wget stores response info to disk. Besides, even if it didn't, you could still visit a 404 page of the website and figure out a unique string of text to search for.

Re: Command-line tools can be faster than your Hadoop cluster

#275

Post by Roboprog »

saidajigumi wrote: post #103
threeseed wrote: post #90

Earlier quoted context omitted.

Oh right the "cool kids" approach. Here's what the "sensible adults" think about when they see problems like this. Operational Supportability: How do you monitor the operation ? Restart Recovery: Do you have the ability to restart the operation mid way through if something fails ? Maintainability: Can we run the same application on our desktop as on our production servers ? Extensibility: Can we extend the platform e…

> Dreadful for the long term. Here comes a bubble-bursting: I've lead a team that built data processing tools exactly like this, and the performance and ease of manipulating vast amounts of text using classic shell tools is hard to beat. We had no problems with any of: operational supportability, restart recovery, or maintainability. Highly testable, even. No, it's not just cowboy-coded crappy shell scripts and pipel…

Unix/POSIX backgrounds are pretty common among the Hacker News crowd. Not so in "Enterprise" development. (Beam me up Scottie, there's no intelligent life here, only risk avoidance)

Enterprise development is predominated by 2 or 3 trusted operating systems: Windows (/ .NET), and the JVM. POSIX systems are only useful in-so-far-as they are a cheaper (or sometimes more reliable) place to host Java virtual machines. Enterprise dev groups generally have very limited exposure to, and a lot of fear of, things like Borne shell, AWK, Perl, Python. These languages don't have Visual Studio or Eclipse to hold your hand while you make far reaching refactorings like renaming a variable.

Sure, you and I would crawl log/data files trivially with a few piped commands, but that's a rare skill in most shops, at least since the turn of the century.

Ugh, that sounds cliche, but it's hard not to feel that way after being drowned in "Java or nothing" for so long at work.

http://tvtropes.org/pmwiki/pmwiki.php/Main/ElegantWeaponForA...

Re: Command-line tools can be faster than your Hadoop cluster

#276

Post by sedachv »

JabavuAdams wrote: post #250
sedachv wrote: post #70

Earlier quoted context omitted.

> Also, hadoop is so painfully slow to develop in it's practically a full employment act for software engineers. It's comical how bad Hadoop is compared even to the CM Lisp described in Daniel Hillis' PhD dissertation. How do you devolve all the way from that down to "It's like map/reduce. You get one map and one reduce!"

Programming is very faddish. It's amazing how bad commonly used technologies are. I'm so happy I'm mostly a native developer and don't have to use the shitty web stack and its shitty replacements.

What really puzzles me is that Doug Cutting worked at Xerox PARC and Mike Cafarella has two (!) CS Masters degrees, a PhD degree, and is a professor at the University of Michigan. It's not like they were unaware of the previous work in the field (Connection Machine languages, Paralations, NESL).

Re: Command-line tools can be faster than your Hadoop cluster

#278

Post by falcor84 »

sebastianavina wrote: post #234
lloeki wrote: post #186

Earlier quoted context omitted.

Alternative, real life scenario: navigate through 6 months of daily MySQL dumps, assorted YAML files and Rails production.log, looking for some cross product between tables, requests and serialised entities, for analysis and/or data recovery (pinpoint or retrieval). zcat/cut/sed/grep/awk/perl crawled through it in a couple of minutes and required less than half an hour to craft a reliable enough implementation (inclu…

to be fair, you could have achieved all that with a simple python script. sometimes i feel python is the new bash.

Python is a great tool for this, and is even better when used with the pythonpy tool, which allows for convenient integration of python commands inside unix pipelines - https://github.com/Russell91/pythonpy

Re: Command-line tools can be faster than your Hadoop cluster

#279

Post by taltman1 »

This is a great exercise of how to take a Unix command line and iteratively optimize it with advanced use of awk.

In that spirit, one can optimize the xargs mawk invocation by 1) Getting rid of string-manipulation function calls (which are slow in awk), 2) using regular expressions in the pattern expression (which allows awk to short-circuit the evaluation of lines), and 3) avoiding use of field variables like $1, and $2, which allows the mawk virtual machine to avoid implicit field splitting. A bonus is that you end up with an awk script which is more idiomatic:

  mawk '
  /^\[Result "1\/2-1\/2"\]/ { draw++ }
  /^\[Result "1-0"\]/ { white++ }
  /^\[Result "0-1"\]/ { black++ }

  END { print white, black, draw }'
Notice that I got rid of the printing out of the intermediate totals per file. Since we are only tabulating the final total, we can modify the 'reduce' mawk invocation to be as follows:

  mawk '
  {games += ($1+$2+$3); white += $1; black += $2; draw += $3}
  END { print games, white, black, draw }'
Making the bottle-neck data stream thinner always helps with overall throughput.

Re: Command-line tools can be faster than your Hadoop cluster

#280

Post by djeikyb »

reitanqild wrote: post #215
mech4bg wrote: post #128

Earlier quoted context omitted.

Why use a loaded word like 'force'? Developing on Windows was excellent. Microsoft provides great tools and support; it was incredibly productive. The only real negative is the licensing requirements. As of 2011 (I couldn't find more recent data publicly available) Windows was far and away the most popular development environment. http://www.cnet.com/news/coders-choosing-mac-os-over-linux-e... It's interesting to see…

> The only real negative is the licensing requirements Fix: Some of the real negative s are : * the licensing requirements, * lack of virtual desktops, * having to manually update everything except the office suite and the OS itself * having to use a different platform than what is used in production * many standard tools works slower / is less tested / etc (PS: I'm working happily from a Windows workstation now.)

Windows does have virtual desktops...just not the way X allows. The main thing that bugs me about Windows's desktops is that it is given a logical display surface compromised of all the display units. Whereas with X (going by my use of xmonad) it's easy to separate the display and desktop abstractions and have desktop 2 on display 1 and desktop 5 on display 2.

There's a nice overview of the limitations and intentions of the Windows model here: http://msdn.microsoft.com/en-us/library/windows/desktop/ms68...

When on windows, I've been using this (it crashes sometimes, but only itself, never takes other programs with it): https://github.com/Tzbob/python-windows-tiler

Post reply on HN