Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What you need may be “pipeline +Unix commands” only

nanxiao.me

Post reply on HN
Print view View original

111–120 of 181 posts

Re: What you need may be “pipeline +Unix commands” only

#111

Post by edoloughlin »

wildmindwriting wrote: post #74
lapnitnelav wrote: post #70

Earlier quoted context omitted.

I think it's mostly a matter of chance more than anything else. If you've jumped straight into programming, you'll probably consider any of those problems as a nail to your C/JS/Java/Python Hammer. I was lucky to be initiated to the GNU / UNIX toolset by operation folks when doing tech support in a SAAS biz. We were dealing with a lot of text files and it didn't feel right to offload whatever my problem was to them,…

I have also found this to be the case too. Most people would rather have a GUI before even touching the command line. Most notably is Git; every single one of my developers use Sourcetree and if I have to help them with something, I always have to pop open the terminal. It's gotten to the point where I'm considered "odd" because I use the command line. It's become a running joke among everyone.

it's gotten to the point where I'm considered "odd" because I use the command line

Me too. I'm thinking of doing a presentation on the Dunning-Kruger effect to see if it sparks some introspection in my colleagues.

Re: What you need may be “pipeline +Unix commands” only

#112

Post by srean »

zaphar wrote: post #90
speedplane wrote: post #43

Earlier quoted context omitted.

> Maybe you see TB level processing a lot in your line of work, but most developers never will. I doubt this is the case. If you're working on even a medium sized team, you'll see this level of data, whether it's internal server logs, publicly-sourced data, or a variety of other applications. Almost by definition, any company that runs a cluster is probably dealing with TBs of data (otherwise they could probably run…

Just a few weeks ago I was working with an 11TB dataset which I processed with just unix command line tools told on a single Linux VM. You can get a lot of milage out of parallel xargs and other of the shelf tools.

Indeed! Gnu prallel, xargs, make -j, netcat, mawk/gawk (with C extensions if needed), jq, coreutils, textutils can get insane amount of stuff done.

Re: What you need may be “pipeline +Unix commands” only

#113

Post by gcmeplz »

twic wrote: post #72

I've been a pipeline junkie for a long time, but i've only recently started to get into awk. The thing i can do with awk but not other tools is to write stateful filters, which accumulate information in associative arrays as they go. For example, if you want to do uniq without sorting the input, that's: awk '{ if (!($0 in seen)) print $0; seen[$0] = 1; }' This works best if the number of unique lines is small, either…

  awk '{ if (!($2 in seen)) print $0; seen[$2] = 1; }'
You can even shorten this a bit! "awk '!seen[$2]++'" does the same thing -- awk will print the whole line when it's provided a truthy value. It's definitely more code-golfy than being explicit about what's actually going on though

Re: What you need may be “pipeline +Unix commands” only

#114

Post by dahart »

pjc50 wrote: post #57
angarg12 wrote: post #7

I agree with the sentiment that many solutions are over-engineered, but when you need to process billions of records a day, you do need more complex systems. Bottom line: when facing an engineering problem, start with the simplest, fastest to implement solution, and build complexity as necessary. The simple solution suffices most of the time.

> billions of records a day A good GPU can do >1bn calculations per frame .

A good GPU today can do > 1 Tn calculations per frame.

Re: What you need may be “pipeline +Unix commands” only

#115

Post by draw_down »

Svoka wrote: post #108

I had a task the other day to aggregate some logs. So I wrote a one liner, which did most of what I wanted. I took about 4 minutes to run. Then I decided to run it on larger dataset (because I needed too). Like week of logs, not a day of logs. While it was running, I wrote rust CLI, which was working like `cat /*.log | logparser` and did one day in 12 seconds, and a week in a two minutes. And I gave up waiting on awk…

Yeah. These tools are very convenient when they work, but there are lots of situations where they don’t quite cut it.

Re: What you need may be “pipeline +Unix commands” only

#116

Post by geofft »

speedplane wrote: post #30

This article's primary example is a single static text file with 5M lines. Sure, in that case, awk works great, but how often does that come up? In the real world, those 5M lines are growing by several hundred thousand every day, and after a few months, grows beyond what a single computer or awk can handle. Further, users want real-time results, not just a few times a day when your cron script runs. Unix commands are…

> In the real world, those 5M lines are growing by several hundred thousand every day, and after a few months, grows beyond what a single computer or awk can handle.

This is a frustrating viewpoint, and it feels to me like the same poisonous worldview as "If your company isn't growing 10% month-over-month, it's not worthwhile and we won't invest in it." There are plenty of meaningful and useful things you can do for the world at a static size, and doing continued interesting work with a dataset of the same size doesn't mean you're not part of the "real world."

Re: What you need may be “pipeline +Unix commands” only

#117

Post by dahart »

> Every item on the menu at Taco Bell is just a different configuration of roughly eight ingredients.

HA! This has almost been my line for years regarding Mexican food. What I like to say is: it’s amazing how every possible permutation of 8 ingredients has been named. BTW I love Mexican food, lived in Mexico.

> The post mentions a scenario which you may consider to use Hadoop to solve but actually xargs may be a simpler and better choice.

I do feel like there’s a corollary to Knuth’s “premature optimization” quote regarding web scaling; premature scaling and using tools much bigger than necessary for the job at hand is pretty common.

Re: What you need may be “pipeline +Unix commands” only

#118

Post by sizzzzlerz »

bibyte wrote: post #31

This submission seems weirdly relevant. https://news.ycombinator.com/item?id=19271135

I was going to comment on exactly this article except that I couldn't find it quickly. It isn't weirdly relevant; it's totally relevant. It demonstrates that, with sufficient knowledge of the command line, one can write the most amazing tools, quickly and succinctly. Here, knowledge doesn't necessarily meaning knowing everything immediately but also knowing what resources to reference to find out stuff.

I've been programming for 40 years and using unix/linux since the 80's and in this little one-line script, I discovered two things that one can do with the appropriate arguments that I've never known. YMMV.

Re: What you need may be “pipeline +Unix commands” only

#119

Post by ims »

Sometimes there's a middle ground: make your "map" and "reduce" steps separate scripts.

If you want to do the parsing in Python instead of awk, just make a tiny script that reads from stdin and writes to stdout - that way you can put it between xargs or parallel and whatever else is in the pipeline.

The parallelization is a separate concern, so it doesn't need to be mixed in with the parsing (or whatever) concern. The downloading is a separate concern; use wget or requests in a Python script or whatever, it doesn't need to be mingled with the parsing.

Post reply on HN