Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What you need may be “pipeline +Unix commands” only

nanxiao.me

Post reply on HN
Print view View original

81–90 of 181 posts

Re: What you need may be “pipeline +Unix commands” only

#81

Post by srean »

speedplane wrote: post #30

This article's primary example is a single static text file with 5M lines. Sure, in that case, awk works great, but how often does that come up? In the real world, those 5M lines are growing by several hundred thousand every day, and after a few months, grows beyond what a single computer or awk can handle. Further, users want real-time results, not just a few times a day when your cron script runs. Unix commands are…

> But to deal with Terabytes of data quickly and efficiently, these tools totally break down

Scaling up to 400~500GB of logs with awk and parallel has not been a problem for me. I dont think TB will be particularly hard, especially if one is reasonably proficient with the tools. Of course, if one has the mental bias that one has to throw hadoop or spark at it, thats a significant obstacle right there. Upton Sinclair effect also plays a role -- It is difficult to get a man to understand something, when his salary depends upon his not understanding it.

Of course at some scale simple unix tools become impractical, but usually people reach for flashier tools even at scales where unix tools will suffice.

Re: What you need may be “pipeline +Unix commands” only

#83

Post by [deleted] »

rofo1 wrote: post #6

I feel like the art of UNIX is slowly fading into oblivion, especially with the new generation of programmers/developers. Eventually, they'll become the ones that decide the fate of software engineers (by being hiring managers, etc.) and we'll see more and more monstrosity like the article portraits, instead of cleverly using UNIX tools where applicable. There's so many things that the software world is doing wrong t…

[deleted]

Re: What you need may be “pipeline +Unix commands” only

#84

Post by mooreds »

lapnitnelav wrote: post #70
rofo1 wrote: post #6

I feel like the art of UNIX is slowly fading into oblivion, especially with the new generation of programmers/developers. Eventually, they'll become the ones that decide the fate of software engineers (by being hiring managers, etc.) and we'll see more and more monstrosity like the article portraits, instead of cleverly using UNIX tools where applicable. There's so many things that the software world is doing wrong t…

I think it's mostly a matter of chance more than anything else. If you've jumped straight into programming, you'll probably consider any of those problems as a nail to your C/JS/Java/Python Hammer. I was lucky to be initiated to the GNU / UNIX toolset by operation folks when doing tech support in a SAAS biz. We were dealing with a lot of text files and it didn't feel right to offload whatever my problem was to them,…

> Turns out, most people don't want to have anything to do with a command prompt, even if the hard part has been done for you. That's been a pretty good lesson BTW.

My experience has been that some folks are resistant to the command line (but I wouldn't say most). This is too bad because I feel like it's a crucial part of development. I even wrote a post about it: https://letterstoanewdeveloper.com/2019/02/04/learn-the-comm...

Re: What you need may be “pipeline +Unix commands” only

#85

Post by mooreds »

twic wrote: post #72

I've been a pipeline junkie for a long time, but i've only recently started to get into awk. The thing i can do with awk but not other tools is to write stateful filters, which accumulate information in associative arrays as they go. For example, if you want to do uniq without sorting the input, that's: awk '{ if (!($0 in seen)) print $0; seen[$0] = 1; }' This works best if the number of unique lines is small, either…

I did a lightning talk on awk last year and found this great article series from 2000 on all the powers of awk (including network access, but not yet email :) ).

https://www.ibm.com/developerworks/library/l-awk1/index.html

Re: What you need may be “pipeline +Unix commands” only

#86

Post by mobilemidget »

twic wrote: post #72

I've been a pipeline junkie for a long time, but i've only recently started to get into awk. The thing i can do with awk but not other tools is to write stateful filters, which accumulate information in associative arrays as they go. For example, if you want to do uniq without sorting the input, that's: awk '{ if (!($0 in seen)) print $0; seen[$0] = 1; }' This works best if the number of unique lines is small, either…

So I assume you would use some sort of ' grep XX | sort | uniq' (I still) to get a unique line output. Is this awk line now your default, or did you find yourself using both for convenience?

Do you alias these awk commands on all machines you work on, or other way put, I did not find a nice way to keep my custom aliases 'in sync' over different machines, perhaps you have some recommendation or workflow that is really sweet?

TIA!

Re: What you need may be “pipeline +Unix commands” only

#88

Post by chewz »

twic wrote: post #72

I've been a pipeline junkie for a long time, but i've only recently started to get into awk. The thing i can do with awk but not other tools is to write stateful filters, which accumulate information in associative arrays as they go. For example, if you want to do uniq without sorting the input, that's: awk '{ if (!($0 in seen)) print $0; seen[$0] = 1; }' This works best if the number of unique lines is small, either…

Thank you. I have just been working on something a this

> awk '{ if (!($0 in seen)) print $0; seen[$0] = 1; }'

fits right in.

Re: What you need may be “pipeline +Unix commands” only

#89

Post by havkom »

> BTW, if your data set can be disposed by an awk script, it should not be called “big data”.

I think this statement is wrong. The popular meaning of the hype term “big data” can not be easily changed.

Rather, awk, sed and other tools that can read from stdin and write to stdout are great tools for “big data” and often more efficient and suitable than larger and more hyped systems.

Re: What you need may be “pipeline +Unix commands” only

#90

Post by zaphar »

speedplane wrote: post #43
aidos wrote: post #36

Earlier quoted context omitted.

Well, the article does explicitly say that if you can do this then you don’t have “big data”. Maybe you see TB level processing a lot in your line of work, but most developers never will. Whenever I deal with anything a bit bigger, I break off the smallest section I need to deal with and work with that.

> Maybe you see TB level processing a lot in your line of work, but most developers never will. I doubt this is the case. If you're working on even a medium sized team, you'll see this level of data, whether it's internal server logs, publicly-sourced data, or a variety of other applications. Almost by definition, any company that runs a cluster is probably dealing with TBs of data (otherwise they could probably run…

Just a few weeks ago I was working with an 11TB dataset which I processed with just unix command line tools told on a single Linux VM.

You can get a lot of milage out of parallel xargs and other of the shelf tools.

Post reply on HN