Live data from Hacker News

What you need may be “pipeline +Unix commands” only

nanxiao.me

71–80 of 181 posts

Re: What you need may be “pipeline +Unix commands” only

#71

This article's primary example is a single static text file with 5M lines. Sure, in that case, awk works great, but how often does that come up? In the real world, those 5M lines are growing by several hundred thousand every day, and after a few months, grows beyond what a single computer or awk can handle. Further, users want real-time results, not just a few times a day when your cron script runs. Unix commands are…

Why would you need to look at the entire history of the log file each time?

Isn't it logical to have historical summary info and then the full log for say the past month?

Re: What you need may be “pipeline +Unix commands” only

#72
I've been a pipeline junkie for a long time, but i've only recently started to get into awk. The thing i can do with awk but not other tools is to write stateful filters, which accumulate information in associative arrays as they go.

For example, if you want to do uniq without sorting the input, that's:

  awk '{ if (!($0 in seen)) print $0; seen[$0] = 1; }'
This works best if the number of unique lines is small, either because the input is small, or because it is highly repetitive. Made-up example, finding all the file extensions used in a directory tree:

  find /usr/lib -type f | sed -rn 's/^.*\.([^/]*)$/\1/p' | awk '{ if (!($0 in seen)) print $0; seen[$0] = 1; }'
That script is easily tweaked, eg to uniquify by a part of the string. Say you have a log file formatted like this:

  2019-03-03T12:38:16Z hob: turned to 75%
  2019-03-03T12:38:17Z frying_pan: moved to hob
  2019-03-03T12:38:19Z frying_pan: added butter
  2019-03-03T12:38:22Z batter: mixed
  2019-03-03T12:38:27Z batter: poured in pan
  2019-03-03T12:38:28Z frying_pan: tilted around
  2019-03-03T12:39:09Z frying_pan: FLIPPED
  2019-03-03T12:39:41Z frying_pan: FLIPPED
  2019-03-03T12:39:46Z frying_pan: pancake removed
If you want to see the first entry for each subsystem:

  awk '{ if (!($2 in seen)) print $0; seen[$2] = 1; }'
Or the last (although this won't preserve input order):

  awk '{ seen[$2] = $0; } END { for (k in seen) print seen[k]; }'
I don't think there's another simple tool in the unix toolkit that lets you do things like this. You could probably do it with sed, but it would involve some nightmarish abuse of the hold space as a database.

Re: What you need may be “pipeline +Unix commands” only

#73
post #11
post #6

I feel like the art of UNIX is slowly fading into oblivion, especially with the new generation of programmers/developers. Eventually, they'll become the ones that decide the fate of software engineers (by being hiring managers, etc.) and we'll see more and more monstrosity like the article portraits, instead of cleverly using UNIX tools where applicable. There's so many things that the software world is doing wrong t…

You forget that using more developers means more headcount, and more headcount means I have more responsibility as a manager. These crazy complex solutions also look a lot more difficult on the slides than the simple 3 layer architecture that’s often shown to me. With something that simple, and requiring so few people, how am I ever going to convince my clients to pay me multiple millions of dollars for it.

I guess you should sell them the choice of either the big complex thing written by a large team for $2.3 MM, or the lean highly-optimized solution written by a handful of elite "10X" developers for $1.9 MM. (Of course the 10X developers are in high demand and very expensive.)

Re: What you need may be “pipeline +Unix commands” only

#74
post #6

I feel like the art of UNIX is slowly fading into oblivion, especially with the new generation of programmers/developers. Eventually, they'll become the ones that decide the fate of software engineers (by being hiring managers, etc.) and we'll see more and more monstrosity like the article portraits, instead of cleverly using UNIX tools where applicable. There's so many things that the software world is doing wrong t…

I think it's mostly a matter of chance more than anything else. If you've jumped straight into programming, you'll probably consider any of those problems as a nail to your C/JS/Java/Python Hammer. I was lucky to be initiated to the GNU / UNIX toolset by operation folks when doing tech support in a SAAS biz. We were dealing with a lot of text files and it didn't feel right to offload whatever my problem was to them,…

I have also found this to be the case too. Most people would rather have a GUI before even touching the command line. Most notably is Git; every single one of my developers use Sourcetree and if I have to help them with something, I always have to pop open the terminal. It's gotten to the point where I'm considered "odd" because I use the command line. It's become a running joke among everyone.

Re: What you need may be “pipeline +Unix commands” only

#75
post #36

Earlier quoted context omitted.

Well, the article does explicitly say that if you can do this then you don’t have “big data”. Maybe you see TB level processing a lot in your line of work, but most developers never will. Whenever I deal with anything a bit bigger, I break off the smallest section I need to deal with and work with that.

> Maybe you see TB level processing a lot in your line of work, but most developers never will. I doubt this is the case. If you're working on even a medium sized team, you'll see this level of data, whether it's internal server logs, publicly-sourced data, or a variety of other applications. Almost by definition, any company that runs a cluster is probably dealing with TBs of data (otherwise they could probably run…

> Almost by definition, any company that runs a cluster is probably dealing with TBs of data (otherwise they could probably run it on one machine).

This doesn't follow:

1) One reason to run a cluster is to use flaky commodity hardware instead of high-reliability specialized hardware. Note the transition from specialized hardware and computer rooms of the '80s and '90s to cloud computing on preemptible instances.

2) They might just be straight-up wrong / misguided. http://www.frankmcsherry.org/graph/scalability/cost/2015/01/... As a professional developer I've seen tons of distributed systems that could be replaced by a well-designed non-distributed system, but ELK and Mongo and etcd and Kafka and more generally cloud servers are easy off-the-shelf tools.

Re: What you need may be “pipeline +Unix commands” only

#76
post #45
post #7

I agree with the sentiment that many solutions are over-engineered, but when you need to process billions of records a day, you do need more complex systems. Bottom line: when facing an engineering problem, start with the simplest, fastest to implement solution, and build complexity as necessary. The simple solution suffices most of the time.

You know, "billions a day" is only on the order of 10K per second. A single machine can handle that.

That would be amazing, wouldn't it? It's not true though, the problem with dealing with billion of operations a day is the spikes, most of the time you don't get a nice homogene rate for 24 hours straight.

Re: What you need may be “pipeline +Unix commands” only

#77

Does this add anything to the Taco Bell post linked in TFA? I suggest changing the link to: http://widgetsandshit.com/teddziuba/2010/10/taco-bell-progra...

TacoBellArticle> I could have done the whole thing Taco Bell style if I had only manned up and broken out sed, but I pussied out and wrote some Python. That’s cringe-worthy...

Yeah, I know some women that are way better at unix then me. Heck I learned to program, to the extent that it wasn't auto didactic, mostly from women.

Anyway, it's poor form to put gender into the mix of what good coding looks like. Don't man up, do it hard core or bravely. Don't pussy out, chicken out. Don't try something ballsy, try something gutsy.

Re: What you need may be “pipeline +Unix commands” only

#78
post #9
post #4

Earlier quoted context omitted.

Medium data. Substantial data. Just-enough-data.

My favorite term is "annoying sized" data, not enough to warrant clusters and HPC, but enough to make a decent laptop crawl to a halt. It's that uncomfortable in-between that makes up the bulk of the data I usually encounter.

The one nice thing about AWS and its ilk is the ability to spin up big chunky VM for few hours/days for ad-hoc data processing for pretty small amount of money. That can in many cases shift the boundary of annoying sized enough, with the added bonus that your own workstation is not fully occupied by the processing. Of course such approach is not applicable to all scenarios, but it is a useful trick to keep in your sleeve.

Re: What you need may be “pipeline +Unix commands” only

#79

Earlier quoted context omitted.

I think it's mostly a matter of chance more than anything else. If you've jumped straight into programming, you'll probably consider any of those problems as a nail to your C/JS/Java/Python Hammer. I was lucky to be initiated to the GNU / UNIX toolset by operation folks when doing tech support in a SAAS biz. We were dealing with a lot of text files and it didn't feel right to offload whatever my problem was to them,…

I have also found this to be the case too. Most people would rather have a GUI before even touching the command line. Most notably is Git; every single one of my developers use Sourcetree and if I have to help them with something, I always have to pop open the terminal. It's gotten to the point where I'm considered "odd" because I use the command line. It's become a running joke among everyone.

I don't see how this can be a target of their joke: they have problems with their GUI (I am assuming that's what Sourcetree is), you solve them with your CLI. If they want to laugh, they 'd better fix their own problems themselves, I guess?

Re: What you need may be “pipeline +Unix commands” only

#80
post #6

I feel like the art of UNIX is slowly fading into oblivion, especially with the new generation of programmers/developers. Eventually, they'll become the ones that decide the fate of software engineers (by being hiring managers, etc.) and we'll see more and more monstrosity like the article portraits, instead of cleverly using UNIX tools where applicable. There's so many things that the software world is doing wrong t…

These complex solutions allow one operations engineer to manage thousands and thousands of servers/containers. Guys that just knew how to bang together bash and Perl scripts got laid off all over the place in favor of people that know cloud stuff.
Post reply on HN