Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What you need may be “pipeline +Unix commands” only

nanxiao.me

Post reply on HN
Print view View original

91–100 of 181 posts

Re: What you need may be “pipeline +Unix commands” only

#91

Post by 3pt14159 »

I really agree with aspects of this, and I think CLIs and Unix pipes are way more powerful than we treat them, but be forewarned that there are problems with doing everything with pipes.

You need to code more defensively with them. For example, it is rare, but every so often a newline will be fail to be emitted.

    kinda\n
    likethis\n
    \n 
    example\n
There are many other gotchas, but that one is a doozy because if you're using, say, tab delimited data and cut you'll miss a line. It's one of the reasons I use line delimited JSON if at all possible.

Also, this constant re-parsing of text does mean your string validation needs to be more paranoid. For example, some JSON parsers parse curly quotes as normal programming quotes. Horrible practice, I know, but it could have been avoided. Also, it's easy to accidentally do shit like this when you're in a rush. Some string matching tools handle the character matching of the different ways of creating, say, "ë" will also make matching quotes more relaxed.

Anyway, all of this to say that I 100% agree with the posted and linked articles, but each method has its own security considerations and software folk should be aware of them before starting.

Re: What you need may be “pipeline +Unix commands” only

#92

Post by 12298765 »

JoeSmithson wrote: post #49
speedplane wrote: post #30

This article's primary example is a single static text file with 5M lines. Sure, in that case, awk works great, but how often does that come up? In the real world, those 5M lines are growing by several hundred thousand every day, and after a few months, grows beyond what a single computer or awk can handle. Further, users want real-time results, not just a few times a day when your cron script runs. Unix commands are…

> how often does that come up? It's an important point to remember that a lot of things involved in human society have not exploded in size or complexity in the last 30 years. Many data sets are basically proportional to the human population (health records, criminal records, property records etc), and these have been measured in the millions for 30+ years. In the same time the compute power of a single script has mo…

I've never seen that Grace Hopper bit, but that's awesome! Thanks for sharing. Definitely going to show it to some co-workers

Re: What you need may be “pipeline +Unix commands” only

#93

Post by nudpiedo »

I don't get the motivation of this article, it links to the taco bell programming article which says exactly the same. I usually wouldn't write an article to repeat the same another article says, or if it is something that could have been just a comment in the original blog.

Re: What you need may be “pipeline +Unix commands” only

#95

Post by tyingq »

chewz wrote: post #88
twic wrote: post #72

I've been a pipeline junkie for a long time, but i've only recently started to get into awk. The thing i can do with awk but not other tools is to write stateful filters, which accumulate information in associative arrays as they go. For example, if you want to do uniq without sorting the input, that's: awk '{ if (!($0 in seen)) print $0; seen[$0] = 1; }' This works best if the number of unique lines is small, either…

Thank you. I have just been working on something a this > awk '{ if (!($0 in seen)) print $0; seen[$0] = 1; }' fits right in.

Perl equivalent is:

perl -ne 'print unless $SEEN{$_}++'

Re: What you need may be “pipeline +Unix commands” only

#96

Post by type0 »

wildmindwriting wrote: post #74
lapnitnelav wrote: post #70

Earlier quoted context omitted.

I think it's mostly a matter of chance more than anything else. If you've jumped straight into programming, you'll probably consider any of those problems as a nail to your C/JS/Java/Python Hammer. I was lucky to be initiated to the GNU / UNIX toolset by operation folks when doing tech support in a SAAS biz. We were dealing with a lot of text files and it didn't feel right to offload whatever my problem was to them,…

I have also found this to be the case too. Most people would rather have a GUI before even touching the command line. Most notably is Git; every single one of my developers use Sourcetree and if I have to help them with something, I always have to pop open the terminal. It's gotten to the point where I'm considered "odd" because I use the command line. It's become a running joke among everyone.

> It's gotten to the point where I'm considered "odd" because I use the command line. It's become a running joke among everyone.

Well, the joke is on them for choosing to only stick to the GUI.

Re: What you need may be “pipeline +Unix commands” only

#97

Post by 4thaccount »

radiator wrote: post #79
wildmindwriting wrote: post #74

Earlier quoted context omitted.

I have also found this to be the case too. Most people would rather have a GUI before even touching the command line. Most notably is Git; every single one of my developers use Sourcetree and if I have to help them with something, I always have to pop open the terminal. It's gotten to the point where I'm considered "odd" because I use the command line. It's become a running joke among everyone.

I don't see how this can be a target of their joke: they have problems with their GUI (I am assuming that's what Sourcetree is), you solve them with your CLI. If they want to laugh, they 'd better fix their own problems themselves, I guess?

They probably respect his knowledge and know he is the only competent one, but they simply don't care to learn the CLI method as they are lazy, or can't justify all the time it takes to learn it if he is there to fix their problems (more efficient). I've been on both sides of that situation before. There is always an expert in something you want to know, but can't justify the time. So don't take it too hard on them. I support a Linux based app at work and thus got pretty comfortable with the command line (vim, grep, awk, head, tail, cut, sort, ls, cp, top, cat...etc). To my knowledge I'm the only one in engineering with this knowledge (not in IT support). I also notice my fellow engineers will frequently use more complicated techniques for something that is a single piped command for me. I don't fault them for it though as they have no need to use Linux.

Re: What you need may be “pipeline +Unix commands” only

#98

Post by tyingq »

twic wrote: post #72

I've been a pipeline junkie for a long time, but i've only recently started to get into awk. The thing i can do with awk but not other tools is to write stateful filters, which accumulate information in associative arrays as they go. For example, if you want to do uniq without sorting the input, that's: awk '{ if (!($0 in seen)) print $0; seen[$0] = 1; }' This works best if the number of unique lines is small, either…

"I don't think there's another simple tool in the unix toolkit that lets you do things like this."

Perl can, since it borrowed a fair amount of awk. It's also almost as commonly already installed. The one liner equivalents to what you showed are pretty similar, for example: https://news.ycombinator.com/item?id=19294575

Though, I concede it falls outside the realm of "simple tool".

Re: What you need may be “pipeline +Unix commands” only

#99

Post by klhugo »

twic wrote: post #72

I've been a pipeline junkie for a long time, but i've only recently started to get into awk. The thing i can do with awk but not other tools is to write stateful filters, which accumulate information in associative arrays as they go. For example, if you want to do uniq without sorting the input, that's: awk '{ if (!($0 in seen)) print $0; seen[$0] = 1; }' This works best if the number of unique lines is small, either…

I admire your work. Clever usage of unix tools is very handy. But for parsing text, do you really see that awk and Unix tools as a better solution then a simple python script?

Although I admit that the key argument for Unix tools is that they don’t get updated. That sounds awful, but think about it, once it works, it works everywhere, no matters OS type, version or packages installed. That is something experienced programmers always want from their solutions.

Post reply on HN