Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What you need may be “pipeline +Unix commands” only

nanxiao.me

Post reply on HN
Print view View original

131–140 of 181 posts

Re: What you need may be “pipeline +Unix commands” only

#131

Post by walshemj »

speedplane wrote: post #30

This article's primary example is a single static text file with 5M lines. Sure, in that case, awk works great, but how often does that come up? In the real world, those 5M lines are growing by several hundred thousand every day, and after a few months, grows beyond what a single computer or awk can handle. Further, users want real-time results, not just a few times a day when your cron script runs. Unix commands are…

Users "want" vs do users "need" is the Q.

Also most people don't deal with terabyte sized data sets

Re: What you need may be “pipeline +Unix commands” only

#132

Post by mirceal »

IshKebab wrote: post #121
mirceal wrote: post #105

Earlier quoted context omitted.

parsing text is what a lot of these scripts/mini-pipelines do. the key argument for *nix tools is that they do one thing and only one thing extremely well. at a meta level these tools are units of functionality and you’re actually doing functional programming, on the command line, without realizing it.

The problem is there are like 5 different tools that do the same "one thing" well - awk, sed, grep, cut, etc. Kind of simpler to learn one language that does lots of things well!

i’m going to respectfully disagree.

each tool you listed (with the exception of awk) does one thing and does it extremely well.

my goal is not to use a language to solve all possible variations on problems i have. my goal is to solve the problem.

another interesting side effect is that a lot of times this is super compact and good enough. when it’s not you can go to a programming language

Re: What you need may be “pipeline +Unix commands” only

#133

Post by redsavagefiero »

Svoka wrote: post #108

I had a task the other day to aggregate some logs. So I wrote a one liner, which did most of what I wanted. I took about 4 minutes to run. Then I decided to run it on larger dataset (because I needed too). Like week of logs, not a day of logs. While it was running, I wrote rust CLI, which was working like `cat /*.log | logparser` and did one day in 12 seconds, and a week in a two minutes. And I gave up waiting on awk…

Would you mind posting your awk write-only monstrosity?

Re: What you need may be “pipeline +Unix commands” only

#134

Post by schoen »

When I was in middle school, a friend and I had an internship with a physicist who wanted us to write some software to perform a simple transformation on data sets that he downloaded by FTP from some experiment.

We spent about a week writing a program in QuickBASIC that successfully parsed the files and performed the transformation.

Some years later, I realized that this would be a one-line awk script which I could now write in 20-30 seconds. (Probably someone comfortable with Excel could also perform the transformation in 20-30 seconds, although it might not scale as well to larger files.)

Re: What you need may be “pipeline +Unix commands” only

#135

Post by redsavagefiero »

speedplane wrote: post #30

This article's primary example is a single static text file with 5M lines. Sure, in that case, awk works great, but how often does that come up? In the real world, those 5M lines are growing by several hundred thousand every day, and after a few months, grows beyond what a single computer or awk can handle. Further, users want real-time results, not just a few times a day when your cron script runs. Unix commands are…

In my world your case is the corner case. Don't think I want to deal with multiple terabytes of data as it's provenance is questionable and this data services industries that the human population did without until about ten/fifteen years ago.

Re: What you need may be “pipeline +Unix commands” only

#136

Post by MisterOctober »

rofo1 wrote: post #6

I feel like the art of UNIX is slowly fading into oblivion, especially with the new generation of programmers/developers. Eventually, they'll become the ones that decide the fate of software engineers (by being hiring managers, etc.) and we'll see more and more monstrosity like the article portraits, instead of cleverly using UNIX tools where applicable. There's so many things that the software world is doing wrong t…

I dunno -- there's definitely a community of developers who prefer not to use the command-line / Unix toolset, but at least half of those I know [especially the macOS users and those with some ops experience] routinely reach for the core Unix utilities , especially when doing ad hoc quick fixes / file processing. For my part, I can't imagine doing my job without them [or I guess more accurately, my job wouldn't exist in the first place with them]

I think git has a pretty significant amount to do with the persistence of command-line tool use among the younger set -- it's just so much more efficient / easier to use from the terminal.

Re: What you need may be “pipeline +Unix commands” only

#138

Post by MisterOctober »

mirceal wrote: post #132
IshKebab wrote: post #121

Earlier quoted context omitted.

The problem is there are like 5 different tools that do the same "one thing" well - awk, sed, grep, cut, etc. Kind of simpler to learn one language that does lots of things well!

i’m going to respectfully disagree. each tool you listed (with the exception of awk) does one thing and does it extremely well. my goal is not to use a language to solve all possible variations on problems i have. my goal is to solve the problem. another interesting side effect is that a lot of times this is super compact and good enough. when it’s not you can go to a programming language

^ agree -- I've seen lots of folks [newer users mostly] turn to grep when really what they wanted was sed. It's just a matter of learning which screwdriver is for which type of screw

Re: What you need may be “pipeline +Unix commands” only

#139

Post by fwip »

monsieurbanana wrote: post #76
lstodd wrote: post #45

Earlier quoted context omitted.

You know, "billions a day" is only on the order of 10K per second. A single machine can handle that.

That would be amazing, wouldn't it? It's not true though, the problem with dealing with billion of operations a day is the spikes, most of the time you don't get a nice homogene rate for 24 hours straight.

Depends on how much latency matters. A lot of big data is batch processing, for which if the data is 3 hours old that's more than good enough.
Post reply on HN