Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What you need may be “pipeline +Unix commands” only

nanxiao.me

Post reply on HN
Print view View original

161–170 of 181 posts

Re: What you need may be “pipeline +Unix commands” only

#161

Post by DenseComet »

unhammer wrote: post #126
Svoka wrote: post #108

I had a task the other day to aggregate some logs. So I wrote a one liner, which did most of what I wanted. I took about 4 minutes to run. Then I decided to run it on larger dataset (because I needed too). Like week of logs, not a day of logs. While it was running, I wrote rust CLI, which was working like `cat /*.log | logparser` and did one day in 12 seconds, and a week in a two minutes. And I gave up waiting on awk…

If your awk script gets too long / unreadable you just put it in a file and use some whitespace and longer variable names. AWK scripts tend to be very readable (much more so than e.g. sed) as long as they stick to the "stateful filters" use-case as https://news.ycombinator.com/item?id=19294195 calls it, but yes they have their limits. If speed is a concern, you may want to try using mawk instead of GNU awk/gawk. I've…

Based on his timings, rust achieved a 20x speedup vs a 4x speedup if he used mawk.

Re: What you need may be “pipeline +Unix commands” only

#162

Post by 3pt14159 »

AnIdiotOnTheNet wrote: post #110
3pt14159 wrote: post #91

I really agree with aspects of this, and I think CLIs and Unix pipes are way more powerful than we treat them, but be forewarned that there are problems with doing everything with pipes. You need to code more defensively with them. For example, it is rare, but every so often a newline will be fail to be emitted. kinda\n likethis\n \n example\n There are many other gotchas, but that one is a doozy because if you're us…

This is one of the reasons I prefer PowerShell, it requires a lot fewer text parsing shenanigans. UNIX tools simply failed to evolve. Single io stream pipelining on raw ASCII was perfectly reasonable in the 1970s but it isn't the 1970s anymore. We should be composing tools with multiple typed io stream paths in GUIs (or TUIs I suppose), leveraging two or even three dimensional layouts. All our interfaces should be co…

You know it is so funny that you're mentioning this. I completely agree.

It's to the point where I've been toying around with creating my own shell and faking typed IO streams via Postgres+DSL. It's tricky though. Sometimes I want pub-sub, other times I want event stream. Sometimes I want crash-on-failure, other times I don't. There is this problem in software that I can't really word precisely, but the closest I can come is "do it like this, except these cases here, except-except those cases there" and these things kinda keep stacking up until you have a program that has too much knowledge baked into it.

Take, for example, emoji TLDs. Because emojis aren't consistent across platforms they can get coerced into different types. I didn't know that when I bought and used a couple emoji domains. When someone tried to click on a link in Android and was met with a 404, I was so confused. I wasn't even seeing the request come into nginx!

After I figured it out, I realized that emoji domains won't work. The underlying assumption of TLDs is that there is one, and only one, way of encoding something and that these things aren't coerced. That assumption is wrong.

Re: What you need may be “pipeline +Unix commands” only

#163

Post by 333c »

rofo1 wrote: post #6

I feel like the art of UNIX is slowly fading into oblivion, especially with the new generation of programmers/developers. Eventually, they'll become the ones that decide the fate of software engineers (by being hiring managers, etc.) and we'll see more and more monstrosity like the article portraits, instead of cleverly using UNIX tools where applicable. There's so many things that the software world is doing wrong t…

> I feel like the art of UNIX is slowly fading into oblivion, especially with the new generation of programmers/developers.

For what it's worth, I'm a first-year computer science major and the class I'm currently taking is very much focused on the "art of Unix." We've been doing shell scripting and regular expressions and the like. I quite enjoy it.

Re: What you need may be “pipeline +Unix commands” only

#164

Post by redsavagefiero »

joshuamorton wrote: post #158
redsavagefiero wrote: post #130

Earlier quoted context omitted.

This is something I've noticed in the last 8-10 years. The rise of the python/js/java paradigm everywhere. Some of the associated LDIF (json) I enjoy much more than XML and flat files but the misapplication of tools is becoming an epidemic. When I can write: awk -F "," '{for (x = 1 ; x <= NF ; x++) {if ($x ~ /[0-9]+/) {a[x] = a[x] + $x}}} END { for (p in a) {printf "%d = %d\n",p,a[p]}}' to sum columns in 5 seconds an…

When I can write import pandas as pd data = pd.read_csv(filename) print(data.sum()) and have the same result, I'm going to do the one that is faster to write, fewer characters, and lets me understand what's going on. And don't get me wrong, I've written some gnarly pipelined bash before, although I'm by no means an expert, but that doesn't mean its always the right tool for the job.

I was going to be the long haired *nix geek here but I have no hair and the world is moving on. I can't pick bones with python for data science/analysis and personal convenience. _However_ as a principal engineer if someone was to say, for a trivial dataset, that we need python and pandas for an operation like this where python + pandas was not already provisioned the answer would be no.

Re: What you need may be “pipeline +Unix commands” only

#165

Post by commandar »

redsavagefiero wrote: post #130
rofo1 wrote: post #6

I feel like the art of UNIX is slowly fading into oblivion, especially with the new generation of programmers/developers. Eventually, they'll become the ones that decide the fate of software engineers (by being hiring managers, etc.) and we'll see more and more monstrosity like the article portraits, instead of cleverly using UNIX tools where applicable. There's so many things that the software world is doing wrong t…

This is something I've noticed in the last 8-10 years. The rise of the python/js/java paradigm everywhere. Some of the associated LDIF (json) I enjoy much more than XML and flat files but the misapplication of tools is becoming an epidemic. When I can write: awk -F "," '{for (x = 1 ; x <= NF ; x++) {if ($x ~ /[0-9]+/) {a[x] = a[x] + $x}}} END { for (p in a) {printf "%d = %d\n",p,a[p]}}' to sum columns in 5 seconds an…

>The aversion to the command line is also something that bothers me but I don't run into it as much in my field.

The company I recently started with is really big on Splunk.

The fact that they're proud enough of coming up with the tagline "Taking the sh out of IT" to print it on branded t-shirts featured in their training material was a hint that I wouldn't be a huge fan of the product, personally.

Abstracting things away is great, but something about IT pros being proud of avoiding the command line rubs me entirely the wrong way.

Re: What you need may be “pipeline +Unix commands” only

#166

Post by alchemism »

empath75 wrote: post #80
rofo1 wrote: post #6

I feel like the art of UNIX is slowly fading into oblivion, especially with the new generation of programmers/developers. Eventually, they'll become the ones that decide the fate of software engineers (by being hiring managers, etc.) and we'll see more and more monstrosity like the article portraits, instead of cleverly using UNIX tools where applicable. There's so many things that the software world is doing wrong t…

These complex solutions allow one operations engineer to manage thousands and thousands of servers/containers. Guys that just knew how to bang together bash and Perl scripts got laid off all over the place in favor of people that know cloud stuff.

It turns out that banging together bash scripts which contain invocations of cloud SDK tools has as much relevance in the k8s era as any before.

Maybe it’s Python instead of Perl but that is about the most-significant change.

Re: What you need may be “pipeline +Unix commands” only

#167

Post by Svoka »

redsavagefiero wrote: post #133
Svoka wrote: post #108

I had a task the other day to aggregate some logs. So I wrote a one liner, which did most of what I wanted. I took about 4 minutes to run. Then I decided to run it on larger dataset (because I needed too). Like week of logs, not a day of logs. While it was running, I wrote rust CLI, which was working like `cat /*.log | logparser` and did one day in 12 seconds, and a week in a two minutes. And I gave up waiting on awk…

Would you mind posting your awk write-only monstrosity?

It wasn't all awk, there was a several commands connected by pipe, extracting data from JSON entries, some grep filtering, awk to aggregate data by key, etc.

I tried to look it up and couldn't find, sorry. It was more than half a year ago.

Re: What you need may be “pipeline +Unix commands” only

#168

Post by Svoka »

unhammer wrote: post #126
Svoka wrote: post #108

I had a task the other day to aggregate some logs. So I wrote a one liner, which did most of what I wanted. I took about 4 minutes to run. Then I decided to run it on larger dataset (because I needed too). Like week of logs, not a day of logs. While it was running, I wrote rust CLI, which was working like `cat /*.log | logparser` and did one day in 12 seconds, and a week in a two minutes. And I gave up waiting on awk…

If your awk script gets too long / unreadable you just put it in a file and use some whitespace and longer variable names. AWK scripts tend to be very readable (much more so than e.g. sed) as long as they stick to the "stateful filters" use-case as https://news.ycombinator.com/item?id=19294195 calls it, but yes they have their limits. If speed is a concern, you may want to try using mawk instead of GNU awk/gawk. I've…

I also find that programming with languages which has strong type system is very helpful, if you trying to make anything with logic you can't fit in one simple sentence.

For example, I was parsing logs. They had entries urlencoded JSON, one document per line, each could be invalid, I had to extract 'id' field, and count number of entries with same IDs, and number of entries with same IDs and with special marker. Then take only entries with 10000+ results.

You can totally write cat/grep/awk/sort/head. But then I wanted to add a field, and it was hard to edit. Rust solved my problem, and I had pleasant time writing code, not editing foot long line of untyped code.

Re: What you need may be “pipeline +Unix commands” only

#169

Post by mirceal »

faho wrote: post #154
mirceal wrote: post #132

Earlier quoted context omitted.

i’m going to respectfully disagree. each tool you listed (with the exception of awk) does one thing and does it extremely well. my goal is not to use a language to solve all possible variations on problems i have. my goal is to solve the problem. another interesting side effect is that a lot of times this is super compact and good enough. when it’s not you can go to a programming language

Isn't sed's "one thing" a superset of grep's?

grep is for searching for stuff

sed is for editing streams

Re: What you need may be “pipeline +Unix commands” only

#170

Post by hedora »

DenseComet wrote: post #161
unhammer wrote: post #126

Earlier quoted context omitted.

If your awk script gets too long / unreadable you just put it in a file and use some whitespace and longer variable names. AWK scripts tend to be very readable (much more so than e.g. sed) as long as they stick to the "stateful filters" use-case as https://news.ycombinator.com/item?id=19294195 calls it, but yes they have their limits. If speed is a concern, you may want to try using mawk instead of GNU awk/gawk. I've…

Based on his timings, rust achieved a 20x speedup vs a 4x speedup if he used mawk.

Yep, and adding gnu parallel or xargs to the mix would give a # of cores speedup, which would make mawk faster than the single-threaded rust on a > 6 core machine (roughly).
Post reply on HN