Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What you need may be “pipeline +Unix commands” only

nanxiao.me

Post reply on HN
Print view View original

101–110 of 181 posts

Re: What you need may be “pipeline +Unix commands” only

#101

Post by sigi45 »

rofo1 wrote: post #6

I feel like the art of UNIX is slowly fading into oblivion, especially with the new generation of programmers/developers. Eventually, they'll become the ones that decide the fate of software engineers (by being hiring managers, etc.) and we'll see more and more monstrosity like the article portraits, instead of cleverly using UNIX tools where applicable. There's so many things that the software world is doing wrong t…

Leads are having the responsibility. Most of them are good and know there shit.

So I wouldn't worry to much. All other small companies who have people who don't know better, don't have too much technology inside anyway. Or at least often enough.

Re: What you need may be “pipeline +Unix commands” only

#103

Post by 4thaccount »

empath75 wrote: post #80
rofo1 wrote: post #6

I feel like the art of UNIX is slowly fading into oblivion, especially with the new generation of programmers/developers. Eventually, they'll become the ones that decide the fate of software engineers (by being hiring managers, etc.) and we'll see more and more monstrosity like the article portraits, instead of cleverly using UNIX tools where applicable. There's so many things that the software world is doing wrong t…

These complex solutions allow one operations engineer to manage thousands and thousands of servers/containers. Guys that just knew how to bang together bash and Perl scripts got laid off all over the place in favor of people that know cloud stuff.

I believe the author probably understands that fully and is instead referring to the situation where the optimal solution is a little bash script and somehow the developer designs this horribly complex solution that isn't any more performant, but seems more fancy. Sadly, I think developers sometime do this just to learn new things and keep their resume up to date. It's rough out there from what I read (glad I'm not a developer).

Re: What you need may be “pipeline +Unix commands” only

#104

Post by Aeolun »

groestl wrote: post #19
Aeolun wrote: post #15

Earlier quoted context omitted.

Most stream processing doesn’t rely too much on RAM, unless you literally need all the data in memory at the same time.

If it _needs_ to be in RAM, then either you got a big enough machine (then by definition it's not Big Data) or it's impossible. If you manage to come by with RAM using smart algorithms, although the full dataset would never fit in RAM, then it's Big Data. So I'd argue, stream processing is Big Data, exactly because it doesn't rely too much on RAM.

True, but I can stream process 20GB of data on my tiny 2GB RAM home server as well.

That’s not really ‘big data’ in my opinion.

Re: What you need may be “pipeline +Unix commands” only

#105

Post by mirceal »

klhugo wrote: post #99
twic wrote: post #72

I've been a pipeline junkie for a long time, but i've only recently started to get into awk. The thing i can do with awk but not other tools is to write stateful filters, which accumulate information in associative arrays as they go. For example, if you want to do uniq without sorting the input, that's: awk '{ if (!($0 in seen)) print $0; seen[$0] = 1; }' This works best if the number of unique lines is small, either…

I admire your work. Clever usage of unix tools is very handy. But for parsing text, do you really see that awk and Unix tools as a better solution then a simple python script? Although I admit that the key argument for Unix tools is that they don’t get updated. That sounds awful, but think about it, once it works, it works everywhere, no matters OS type, version or packages installed. That is something experienced pr…

parsing text is what a lot of these scripts/mini-pipelines do.

the key argument for *nix tools is that they do one thing and only one thing extremely well. at a meta level these tools are units of functionality and you’re actually doing functional programming, on the command line, without realizing it.

Re: What you need may be “pipeline +Unix commands” only

#106

Post by nothrabannosir »

radiator wrote: post #79
wildmindwriting wrote: post #74

Earlier quoted context omitted.

I have also found this to be the case too. Most people would rather have a GUI before even touching the command line. Most notably is Git; every single one of my developers use Sourcetree and if I have to help them with something, I always have to pop open the terminal. It's gotten to the point where I'm considered "odd" because I use the command line. It's become a running joke among everyone.

I don't see how this can be a target of their joke: they have problems with their GUI (I am assuming that's what Sourcetree is), you solve them with your CLI. If they want to laugh, they 'd better fix their own problems themselves, I guess?

Jokes can be light-hearted and inclusive.

Re: What you need may be “pipeline +Unix commands” only

#107

Post by noisy_boy »

dredmorbius wrote: post #61
pdimitar wrote: post #55

Earlier quoted context omitted.

Are you planning on open-sourcing the downloader part? I'm very interested.

Literally just a Bash while-read loop over community IDs. It's embarrassingly trivial. I'm planning on posting the data, probably to https://social.antefriguserat.de/ and will include procssing scripts. This is the fetch-script, which saves both the HTML and HEAD responses: #!/bin/bash sample_file=$1 comm_path='community-pages' base_url='https://plus.google.com/communities' i=0 time sed -e 's,^.*/,,' $sample_file | w…

This seems to be a perfect use case for GNU Parallel[0] to download and process, say 10 ids, in parallel. If you have already downloaded/processed and have no need to do it again, then probably doesn't matter now.

[0]: https://www.gnu.org/software/parallel/

Re: What you need may be “pipeline +Unix commands” only

#108

Post by Svoka »

I had a task the other day to aggregate some logs. So I wrote a one liner, which did most of what I wanted. I took about 4 minutes to run.

Then I decided to run it on larger dataset (because I needed too). Like week of logs, not a day of logs.

While it was running, I wrote rust CLI, which was working like `cat /*.log | logparser` and did one day in 12 seconds, and a week in a two minutes.

And I gave up waiting on awk, btw. It is not always better to use command line. If you have gigabytes or tens of gigabytes of data, it would be easier to write some cli tool to help you out.

Also, it was much easier to put significantly more complex logic into it because of type checking, and, you know, being actual high level programming language, not hack&slash awk script.

EDIT: Looking back on my "one liner" vs "rust cli" I would not be able to make meaningful adjustments to one liner comprehension. It is, to my sorrow, write-only thing.

Re: What you need may be “pipeline +Unix commands” only

#109

Post by plaidfuji »

havkom wrote: post #89

> BTW, if your data set can be disposed by an awk script, it should not be called “big data”. I think this statement is wrong. The popular meaning of the hype term “big data” can not be easily changed. Rather, awk, sed and other tools that can read from stdin and write to stdout are great tools for “big data” and often more efficient and suitable than larger and more hyped systems.

Among programmers I've found that the size of your "big data" is often implied to correlate with the size of something else

Re: What you need may be “pipeline +Unix commands” only

#110

Post by AnIdiotOnTheNet »

3pt14159 wrote: post #91

I really agree with aspects of this, and I think CLIs and Unix pipes are way more powerful than we treat them, but be forewarned that there are problems with doing everything with pipes. You need to code more defensively with them. For example, it is rare, but every so often a newline will be fail to be emitted. kinda\n likethis\n \n example\n There are many other gotchas, but that one is a doozy because if you're us…

This is one of the reasons I prefer PowerShell, it requires a lot fewer text parsing shenanigans. UNIX tools simply failed to evolve. Single io stream pipelining on raw ASCII was perfectly reasonable in the 1970s but it isn't the 1970s anymore.

We should be composing tools with multiple typed io stream paths in GUIs (or TUIs I suppose), leveraging two or even three dimensional layouts. All our interfaces should be composed this way, allowing us to take them apart and modify them at will to fit our workflow.

But that never happened. We never made a better hammer, we just try to squeeze all our problems into ASCII-processing nails instead.

Post reply on HN