Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What you need may be “pipeline +Unix commands” only

nanxiao.me

Post reply on HN
Print view View original

41–50 of 181 posts

Re: What you need may be “pipeline +Unix commands” only

#41

Post by mrighele »

ambrop7 wrote: post #16

Unless your data contains spaces, tabs, or, god forbid, newlines. Unix pipeline tools lack any sort of useful data structuring capabilities, making them appropriate for one-off tasks at most.

Spaces, tab and newlines are not a problem.

The fact that there are some standard tools available doesn't mean you are limited to that.

If you have a CSV file with spaces and newlines use cvskit or a small python script importing the relevant library. If you have to parse JSON file you can use jq to pick the relevant fields regardless of how the document is formatted. You can even process binary data as long as the file format is understood by the tool.

Re: What you need may be “pipeline +Unix commands” only

#42

Post by ernesth »

kurczynski wrote: post #10

I hate to be that guy, but they're NOT "Unix" tools, as the name GNU literally states. The post makes a good point that I fully agree with, just doesn't explain it well enough.

awk is in the POSIX norm. It is a Unix tool.

GNU awk is one of the popular awk implementation (usually referred as gawk). I personnally prefer mawk. awk is not GNU.

Re: What you need may be “pipeline +Unix commands” only

#43

Post by speedplane »

aidos wrote: post #36
speedplane wrote: post #30

This article's primary example is a single static text file with 5M lines. Sure, in that case, awk works great, but how often does that come up? In the real world, those 5M lines are growing by several hundred thousand every day, and after a few months, grows beyond what a single computer or awk can handle. Further, users want real-time results, not just a few times a day when your cron script runs. Unix commands are…

Well, the article does explicitly say that if you can do this then you don’t have “big data”. Maybe you see TB level processing a lot in your line of work, but most developers never will. Whenever I deal with anything a bit bigger, I break off the smallest section I need to deal with and work with that.

> Maybe you see TB level processing a lot in your line of work, but most developers never will.

I doubt this is the case. If you're working on even a medium sized team, you'll see this level of data, whether it's internal server logs, publicly-sourced data, or a variety of other applications. Almost by definition, any company that runs a cluster is probably dealing with TBs of data (otherwise they could probably run it on one machine). I don't know the percentage of developers that deal with this, but I do know that it's pretty common.

Re: What you need may be “pipeline +Unix commands” only

#44

Post by zimpenfish »

mrighele wrote: post #41
ambrop7 wrote: post #16

Unless your data contains spaces, tabs, or, god forbid, newlines. Unix pipeline tools lack any sort of useful data structuring capabilities, making them appropriate for one-off tasks at most.

Spaces, tab and newlines are not a problem. The fact that there are some standard tools available doesn't mean you are limited to that. If you have a CSV file with spaces and newlines use cvskit or a small python script importing the relevant library. If you have to parse JSON file you can use jq to pick the relevant fields regardless of how the document is formatted. You can even process binary data as long as the f…

Transcoding CSV to TSV using csvkit has been a life saver for me the last few months due to an old industry's insistence on CSV as an information delivery method. jq has previously been a lifesaver too.

Re: What you need may be “pipeline +Unix commands” only

#45

Post by lstodd »

angarg12 wrote: post #7

I agree with the sentiment that many solutions are over-engineered, but when you need to process billions of records a day, you do need more complex systems. Bottom line: when facing an engineering problem, start with the simplest, fastest to implement solution, and build complexity as necessary. The simple solution suffices most of the time.

You know, "billions a day" is only on the order of 10K per second. A single machine can handle that.

Re: What you need may be “pipeline +Unix commands” only

#46

Post by agumonkey »

rofo1 wrote: post #6

I feel like the art of UNIX is slowly fading into oblivion, especially with the new generation of programmers/developers. Eventually, they'll become the ones that decide the fate of software engineers (by being hiring managers, etc.) and we'll see more and more monstrosity like the article portraits, instead of cleverly using UNIX tools where applicable. There's so many things that the software world is doing wrong t…

Things fade and shine in succession. Good bits will always come back. It's a bit like the saying about mathematics truthiness nature: doesn't matter who or when you look, they will re-emerge as is. composing tiny bits is always good, whether it's unix commands, lisp functions, or forth words..

Societies are large and full of random fluxes and waves.. right now it might be the time for Wirth 17 pages long solutions .. but McIlroy one liner will come back.

Re: What you need may be “pipeline +Unix commands” only

#47

Post by 0815test »

rofo1 wrote: post #6

I feel like the art of UNIX is slowly fading into oblivion, especially with the new generation of programmers/developers. Eventually, they'll become the ones that decide the fate of software engineers (by being hiring managers, etc.) and we'll see more and more monstrosity like the article portraits, instead of cleverly using UNIX tools where applicable. There's so many things that the software world is doing wrong t…

> I feel like the art of UNIX is slowly fading into oblivion, especially with the new generation of programmers/developers. ...

People said the exact same thing in the 1990s, too. "Enterprise"-managed projects, using tech stacks like Win32 or Java, have generally tended to produce large, unwieldly monoliths.

Re: What you need may be “pipeline +Unix commands” only

#48

Post by dredmorbius »

Case in point from my own recent work: I've been analysing characteristics of Google+ Communities, mostly looking for plausibly active good-faith instances.

There are 8.1 million communities in total, and thanks to some friendly assistance, I'd identified slihtly more than 100,000 with both 100 or more members, and visible activity within the preceeding 31 days, as of early 2019.

The task of Web scraping those 100k communities, parsing HTML to a set of characteristics of interest, and reducing that to a delimited dataset of about 16 MB, was all done via shell tools, and on very modest equipment.

Most surprising was that parsing the HTML (using the HTML-XML utilities: http://www.w3.org/Tools/HTML-XML-utils/README) took longer than downloading the data.

Creating the datafile was done with gawk, and most analysis subsequently in R, though quick-and-dirty summaries and queries can be run in gawk.

Performance: downloading (curl): 16 hours, parsing (hxextract & hxselect) 48 hours, dataset preparation (gawk): 2 minutes, analysis (gawk / R), a few seconds for simple outputs.

The parsing step is painfully long, the rest quite tractable.

Re: What you need may be “pipeline +Unix commands” only

#49

Post by JoeSmithson »

speedplane wrote: post #30

This article's primary example is a single static text file with 5M lines. Sure, in that case, awk works great, but how often does that come up? In the real world, those 5M lines are growing by several hundred thousand every day, and after a few months, grows beyond what a single computer or awk can handle. Further, users want real-time results, not just a few times a day when your cron script runs. Unix commands are…

> how often does that come up?

It's an important point to remember that a lot of things involved in human society have not exploded in size or complexity in the last 30 years.

Many data sets are basically proportional to the human population (health records, criminal records, property records etc), and these have been measured in the millions for 30+ years. In the same time the compute power of a single script has moved from the millions into the billions.

It's important, because if a government needs to, say, calculate something involving "every building in the country", or "everybody with a criminal record" they need to understand that this task, in 2019, can in fact be done by a single programmer parsing flat text files on their MBP, and does not need a new department.

This is a bit like Grace Hopper always pointing out the difference between a microsecond and a nanosecond - https://www.youtube.com/watch?v=JEpsKnWZrJ8

Re: What you need may be “pipeline +Unix commands” only

#50

Post by dredmorbius »

speedplane wrote: post #43
aidos wrote: post #36

Earlier quoted context omitted.

Well, the article does explicitly say that if you can do this then you don’t have “big data”. Maybe you see TB level processing a lot in your line of work, but most developers never will. Whenever I deal with anything a bit bigger, I break off the smallest section I need to deal with and work with that.

> Maybe you see TB level processing a lot in your line of work, but most developers never will. I doubt this is the case. If you're working on even a medium sized team, you'll see this level of data, whether it's internal server logs, publicly-sourced data, or a variety of other applications. Almost by definition, any company that runs a cluster is probably dealing with TBs of data (otherwise they could probably run…

Sampling or subetting your data is almost always the correct response, especially for analysis.

If you're literally processing TB or PB of data, you'll want to parallelise. Though shell tools do this amazingly well.

Post reply on HN