Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Command-line tools can be faster than your Hadoop cluster

aadrake.com

Post reply on HN
Print view View original

251–260 of 315 posts

Re: Command-line tools can be faster than your Hadoop cluster

#251

Post by JabavuAdams »

wglb wrote: post #94
Swizec wrote: post #41

Maybe I come from a weird world, or even a weird generation. But when I was in high school, Linux fanboyism was at its peak and just like people get all wound up on bands and such, us geeks got wound up on open-source and linux and fck Micro$oft etc. etc. This was early-ish 2000's. As a result. Every serious programmer I know, especially those who are about my age, lives their life in the CLI. It always comes a surpr…

Every serious programmer I know, especially those who are about my age, lives their life in the CLI. This is not even close to true. Of the developers that I know, many of them quite serious, live in the Windows world, and are very happy with for example VS or eclipse. This is very likely to be the case in most BFEs.

> Every serious programmer I know

So, you know all the same programmers the parent poster knows? The parent explicitly stated that this was anecdotal, and from (not your) personal experience.

Re: Command-line tools can be faster than your Hadoop cluster

#252

Post by ajuc »

_delirium wrote: post #111
sabalaba wrote: post #96

I've had the pleasure and displeasure of working with small datasets (~7.5GB of images) in shell. One often needs to send SIGINT to the shell when it starts to glob expand or tab complete a folder with millions of files. But besides minor issues like that, command line tools get the job done.

Until semi-recently, millions of files in a directory would not only choke up the shell, but the filesystem too. ext4 is a huge improvement over ext3 in that regard; with 10m files in an ext3 directory you ended up with long hangs on various operations. And even with ext4, make sure not to NFS-export the volume that directory is on!

I've encountered this (or similar) issue on production.

We had C++ system that wrote temporary files to /tmp when printing, /tmp was cleared on system startup, it worked ok for years, but the files accumulated. At some point it started to randomly throw file access errors when trying to create these temporary files. Not for each file - only for some of them.

Disk wasn't full, some files could be created in /tmp, others couldn't, it turned out after a few days of tracking it, that filesystem can be overwhelmed by too many similary named files in one directory - and it can't create file XXXX99999 even if there's is no such file in this directory, but it can create files like YYYYY99999 :)

I just love such bugs where your basic assumptions turn out to be wrong.

Re: Command-line tools can be faster than your Hadoop cluster

#253

Post by spiffytech »

AnthonyMouse wrote: post #106
threeseed wrote: post #90

Earlier quoted context omitted.

Oh right the "cool kids" approach. Here's what the "sensible adults" think about when they see problems like this. Operational Supportability: How do you monitor the operation ? Restart Recovery: Do you have the ability to restart the operation mid way through if something fails ? Maintainability: Can we run the same application on our desktop as on our production servers ? Extensibility: Can we extend the platform e…

The Unix people have thought of these things. You can easily do them with command line tools. > Operational Supportability: How do you monitor the operation ? Downloading files with wget will create files and directories as it proceeds. You can observe and count them to determine progress, or pass a shell script to xargs that writes whatever progress data you like to a file before/after calling wget. > Restart Recove…

wget isn't the only part of the puzzle you may need Restart Recovery for - the CPU-bound map/reduce portion may also need to recover from partial progress. Unix tools aren't well-designed for that.

Re: Command-line tools can be faster than your Hadoop cluster

#254

Post by jefftk »

Spooky23 wrote: post #80
danso wrote: post #11

I'm becoming a stronger and stronger advocate of teaching command-line interfaces to even programmers at the novice level...it's easier in many ways to think of how data is being worked on by "filters" and "pipes"...and more importantly, every time you try a step, something happens ...making it much easier to interactively iterate through a process. That it also happens to very fast and powerful (when memory isn't a…

A few years ago between projects, my coworkers cooked up some satirical amazing Web 2.0 data science tools. They used git, did a screencast and distributed it internally. It was basically a few compiled perl scripts and some obfuscated shell scripts with a layer of glitz. People actually used it and LOVED it... It was supposedly better than the real tools some groups were using. It was one of the more epic work troll…

Maybe I'm misreading you, but it sounds like you're saying "my coworkers made something with a really great UI and people loved it!"

Re: Command-line tools can be faster than your Hadoop cluster

#255

Post by anthony_d »

gambiting wrote: post #219
collyw wrote: post #201

Earlier quoted context omitted.

It shouldn't be able to erase your filesystem unless you are running as root or doing something equally stupid. That's pretty much common sense stuff for anyone that isn't a beginner.

Yeah the "common sense stuff for anyone that isn't a beginner" argument is repeated ad nauseam, and even the largest companies make this mistake in their largest products. Take Valve - they should know how to write good code, right? And yet, last week an article was on top of HN, outlining how they put: "rm -rf '$STEAMROOT'/*" in their code, used to remove the library. But hey, no one checked if $STEAMROOT is not emp…

You're right to an extent, but this isn't relevant to the Java vs Bash discussion. The largest companies make this kind of mistake in whatever language they happen to use.

People delete data and screw things up in MapReduce jobs for Hadoop. A lot.

Re: Command-line tools can be faster than your Hadoop cluster

#256

Post by pacala »

yuanchuan wrote: post #200
sleepythread wrote: post #194

One common misconception about using Hadoop is that use Hadoop if your data is large. Usage of Hadoop should be more driven based on the growth of data rather than size. I agree that for the given use case, the solution is appropriate and works fine. Problem mentioned in the given post is not a Big Data problem. Hadoop will be helpful in case if there are millions of games are played everyday and we need to update th…

It is that buzz surrounding Hadoop that makes people misunderstood its use and capability. I have met non-technical analysts who want RDBMS performance on Hadoop. They expect seconds to minutes scale queries on hundreds of GB of data. I always throw this analogy to people who misunderstood Hadoop: A stone to crack an egg or a spoon? Hadoop and RDBMS only have a thin overlapping region in the Venn diagram that describ…

> They expect seconds to minutes scale queries on hundreds of GB of data.

Use BigQuery from Google.

Re: Command-line tools can be faster than your Hadoop cluster

#257

Post by gd1 »

threeseed wrote: post #86
dschiptsov wrote: post #55

Everyone with basic knowledge of CS could realize that Hadoop is a waste. Unfortunately, it isn't about efficiency at all. It just memeization. Bigdata? Hadoop! Runs everywhere. Same BS like Webscale? MongoDB! meme.

Well sorry but you don't have a clue what you're talking about. I very much work in "big data" with about 2 terabytes of new data coming in every day that has to be ingested and processed with hundreds of jobs running against them. The data needs to be queryable via an SQL like language and analyzed by a dozen data scientists using R or Map Reduce. There isn't anything on the market today that has been proven to work…

>There isn't anything on the market today that has been proven to work in environments like this and has the tooling to back it up.

Here you go:

http://kx.com/

Re: Command-line tools can be faster than your Hadoop cluster

#258

Post by FreezerburnV »

_delirium wrote: post #220
reitanqild wrote: post #215

Earlier quoted context omitted.

> The only real negative is the licensing requirements Fix: Some of the real negative s are : * the licensing requirements, * lack of virtual desktops, * having to manually update everything except the office suite and the OS itself * having to use a different platform than what is used in production * many standard tools works slower / is less tested / etc (PS: I'm working happily from a Windows workstation now.)

> lack of virtual desktops They're not built in, but a few of my colleagues use Dexpot and like it: http://www.dexpot.de/index.php?lang=en

Also note, if I'm remembering correctly, there are native virtual desktops slated to come with Windows 10. Kinda late to the game, but it's nice that they're finally getting them.

Re: Command-line tools can be faster than your Hadoop cluster

#259

Post by MichaelGG »

developer1 wrote: post #140
saidajigumi wrote: post #103

Earlier quoted context omitted.

> Dreadful for the long term. Here comes a bubble-bursting: I've lead a team that built data processing tools exactly like this, and the performance and ease of manipulating vast amounts of text using classic shell tools is hard to beat. We had no problems with any of: operational supportability, restart recovery, or maintainability. Highly testable, even. No, it's not just cowboy-coded crappy shell scripts and pipel…

The problem with shell scripting is that nearly nobody is very, very good at it. The Steam bug doing an rm -rf / is an example, but it's very common for shell scripts to have horrible error handling and checks for important things. The shell is just not suitable for extremely robust programs. I would bet that 80%+ of people who think they're good at shell scripting... aren't.

Yeah I learned this trying to massage some data with Elasticsearch. curl -XDELETE host/index/type/$id ... Except $id didn't exist.
Post reply on HN