Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What you need may be “pipeline +Unix commands” only

nanxiao.me

Post reply on HN
Print view View original

151–160 of 181 posts

Re: What you need may be “pipeline +Unix commands” only

#151

Post by twic »

mobilemidget wrote: post #86
twic wrote: post #72

I've been a pipeline junkie for a long time, but i've only recently started to get into awk. The thing i can do with awk but not other tools is to write stateful filters, which accumulate information in associative arrays as they go. For example, if you want to do uniq without sorting the input, that's: awk '{ if (!($0 in seen)) print $0; seen[$0] = 1; }' This works best if the number of unique lines is small, either…

So I assume you would use some sort of ' grep XX | sort | uniq' (I still) to get a unique line output. Is this awk line now your default, or did you find yourself using both for convenience? Do you alias these awk commands on all machines you work on, or other way put, I did not find a nice way to keep my custom aliases 'in sync' over different machines, perhaps you have some recommendation or workflow that is really…

I still default to sort -u (or sort | uniq -c if i need counts), partly from habit, but partly because it's often useful to have the output sorted anyway.

I have a script on my path called huniq ('hash uniq') that contains that awk program. I prefer scripts to aliases because they play better with xargs and so on.

I have a Mercurial repository full of little scripts like this, and other handy things, which lives on Bitbucket, and which i clone on machines i do a lot of work on. In principle, whenever i make changes to it i should commit and push them, then pull them down on other machines, but i'm pretty slack about it. It still helps more than not having anything, though.

Re: What you need may be “pipeline +Unix commands” only

#152

Post by 01100011 »

It's not just data processing. There are simple solutions to all sorts of tasks.

I run 'motion' on my linux desktop at home to serve as a security camera when no one is home. For months I've been manually starting and stopping it, figuring I needed to setup an IoT system if I wanted to automate things. i.e. IFTTT on our phones, an MQTT server in the cloud, etc. Then I realized - I just need to start the camera when all of our phones are off the LAN. It took about 15 minutes to setup, and now I never have to worry about forgetting to stop or start the camera.

Re: What you need may be “pipeline +Unix commands” only

#153

Post by twic »

ryl00 wrote: post #128
twic wrote: post #123

Earlier quoted context omitted.

Python is fantastic for little (or large!) bits of logic, but its handling of input is clunky enough to put me off for tiny things. AFAIK the boilerplate you need to get to working on the fields on each line is: import sys for line in sys.stdin: fields = line.split() # now you can do your logic If you want to use regular expressions, that's another import. Python also doesn't play well with others in a pipeline. You…

This is exactly where perl (namely, perl -ne) is so very, very useful.

Yes, that helps a lot. The fact that Perl uses braces rather than whitespaces also makes it work much better in this situation.

I still wouldn't touch Perl with a bargepole, though. Sorry not sorry.

Re: What you need may be “pipeline +Unix commands” only

#154

Post by faho »

mirceal wrote: post #132
IshKebab wrote: post #121

Earlier quoted context omitted.

The problem is there are like 5 different tools that do the same "one thing" well - awk, sed, grep, cut, etc. Kind of simpler to learn one language that does lots of things well!

i’m going to respectfully disagree. each tool you listed (with the exception of awk) does one thing and does it extremely well. my goal is not to use a language to solve all possible variations on problems i have. my goal is to solve the problem. another interesting side effect is that a lot of times this is super compact and good enough. when it’s not you can go to a programming language

Isn't sed's "one thing" a superset of grep's?

Re: What you need may be “pipeline +Unix commands” only

#155

Post by photon_lines »

dontbenebby wrote: post #122

Does anyone have suggestions on books to grow my scripting fu? (End of chapter exercises tend to be useful for me) I know bash, and know a lot of basic commands, but I'm not familiar with some more advanced things. I don't know awk or sed for example.

If you're looking for good books, I can vouch for 'A Practical Guide to Linux Commands, Editors, and Shell Programming' which has very through coverage and end of chapter exercises. For PowerShell, I'm currently reading the the free PowerShell Notes for Professionals ( https://books.goalkicker.com/PowerShellBook/ ) and it's a great resource as well.

Re: What you need may be “pipeline +Unix commands” only

#156

Post by oarabbus_ »

redsavagefiero wrote: post #130
rofo1 wrote: post #6

I feel like the art of UNIX is slowly fading into oblivion, especially with the new generation of programmers/developers. Eventually, they'll become the ones that decide the fate of software engineers (by being hiring managers, etc.) and we'll see more and more monstrosity like the article portraits, instead of cleverly using UNIX tools where applicable. There's so many things that the software world is doing wrong t…

This is something I've noticed in the last 8-10 years. The rise of the python/js/java paradigm everywhere. Some of the associated LDIF (json) I enjoy much more than XML and flat files but the misapplication of tools is becoming an epidemic. When I can write: awk -F "," '{for (x = 1 ; x <= NF ; x++) {if ($x ~ /[0-9]+/) {a[x] = a[x] + $x}}} END { for (p in a) {printf "%d = %d\n",p,a[p]}}' to sum columns in 5 seconds an…

>When I can write: awk -F "," '{for (x = 1 ; x To be fair, what the matrix libraries do is provide readability and clarity.

Show a programmer unfamiliar with awk your statement and they're going to be spending quite a while parsing it.

Show a programmer unfamiliar with numpy/pandas some matrix multiplication and they may understand it intuitively for the most part without even having to look up references.

edit: I'm actually an awk noob myself but after rereading your code for a second, it makes quite a bit of sense. "For all rows in the column, take all digits 0-9 and sum them". So perhaps not much is gained by the library

Re: What you need may be “pipeline +Unix commands” only

#157

Post by oarabbus_ »

wildmindwriting wrote: post #74
lapnitnelav wrote: post #70

Earlier quoted context omitted.

I think it's mostly a matter of chance more than anything else. If you've jumped straight into programming, you'll probably consider any of those problems as a nail to your C/JS/Java/Python Hammer. I was lucky to be initiated to the GNU / UNIX toolset by operation folks when doing tech support in a SAAS biz. We were dealing with a lot of text files and it didn't feel right to offload whatever my problem was to them,…

I have also found this to be the case too. Most people would rather have a GUI before even touching the command line. Most notably is Git; every single one of my developers use Sourcetree and if I have to help them with something, I always have to pop open the terminal. It's gotten to the point where I'm considered "odd" because I use the command line. It's become a running joke among everyone.

>It's gotten to the point where I'm considered "odd" because I use the command line. It's become a running joke among everyone.

That's... very bizarre. Amongst the engineers I work with it's always an "oh cool, you know how to use bash/terminal" rather than "weird, why are you using terminal".

Re: What you need may be “pipeline +Unix commands” only

#158

Post by joshuamorton »

redsavagefiero wrote: post #130
rofo1 wrote: post #6

I feel like the art of UNIX is slowly fading into oblivion, especially with the new generation of programmers/developers. Eventually, they'll become the ones that decide the fate of software engineers (by being hiring managers, etc.) and we'll see more and more monstrosity like the article portraits, instead of cleverly using UNIX tools where applicable. There's so many things that the software world is doing wrong t…

This is something I've noticed in the last 8-10 years. The rise of the python/js/java paradigm everywhere. Some of the associated LDIF (json) I enjoy much more than XML and flat files but the misapplication of tools is becoming an epidemic. When I can write: awk -F "," '{for (x = 1 ; x <= NF ; x++) {if ($x ~ /[0-9]+/) {a[x] = a[x] + $x}}} END { for (p in a) {printf "%d = %d\n",p,a[p]}}' to sum columns in 5 seconds an…

When I can write

    import pandas as pd
    data = pd.read_csv(filename)
    print(data.sum())
and have the same result, I'm going to do the one that is faster to write, fewer characters, and lets me understand what's going on.

And don't get me wrong, I've written some gnarly pipelined bash before, although I'm by no means an expert, but that doesn't mean its always the right tool for the job.

Re: What you need may be “pipeline +Unix commands” only

#159

Post by brogrammer2018 »

photon_lines wrote: post #155
dontbenebby wrote: post #122

Does anyone have suggestions on books to grow my scripting fu? (End of chapter exercises tend to be useful for me) I know bash, and know a lot of basic commands, but I'm not familiar with some more advanced things. I don't know awk or sed for example.

If you're looking for good books, I can vouch for 'A Practical Guide to Linux Commands, Editors, and Shell Programming' which has very through coverage and end of chapter exercises. For PowerShell, I'm currently reading the the free PowerShell Notes for Professionals ( https://books.goalkicker.com/PowerShellBook/ ) and it's a great resource as well.

Thank you for sharing

Re: What you need may be “pipeline +Unix commands” only

#160

Post by EdwardDiego »

sokoloff wrote: post #69
wodenokoto wrote: post #8

Does this add anything to the Taco Bell post linked in TFA? I suggest changing the link to: http://widgetsandshit.com/teddziuba/2010/10/taco-bell-progra...

TacoBellArticle> I could have done the whole thing Taco Bell style if I had only manned up and broken out sed, but I pussied out and wrote some Python. That’s cringe-worthy...

Despite the gendered terms, what makes me cringe is the belief that X is somehow "better" than Y - if you know Python, and have access to Python, and express yourself in Python faster, then use Python. I use sed, awk, cut, Python, as needed - whatever lets me solve my problem faster.

No flexing about how you use X needed.

Post reply on HN