Live data from Hacker News

Next generation Unix pipe by Alex Larsson

blogs.gnome.org

51–60 of 86 posts

Re: Next generation Unix pipe by Alex Larsson

#51
post #15
post #3

If I can do a slight PG impression, "what problem does this solve?"

Among others, this problem: http://www.dwheeler.com/essays/fixing-unix-linux-filenames.h... find -print0 is a lame hack, and even filenames with spaces (not newlines) are somewhat messy to work with on the Unix shell. Or a little recurring problem I have: How do I grep the output of grep -C (matches showing multiple lines delimited with a "--" line)? I wrote a custom tool to do it, which does the job, but really it w…

[deleted]

Re: Next generation Unix pipe by Alex Larsson

#52
I've noticed that the NUL-termination problem [1] has come up a number of times in these comments. If you want a solution to this that isn't so drastic as an object system, perhaps take a look at Usul [2], non-POSIX 'tabular' Unix utilities which use an AWK-style $RS.

[1]: http://news.ycombinator.com/item?id=4369699

[2]: http://lubutu.com/soso/usul

Re: Next generation Unix pipe by Alex Larsson

#53

I've quickly jotted some thoughts here: http://damnkids.posterous.com/rich-format-unix-pipes Regarding this version, standardizing on a particular transfer format is a bad idea. If history has shown anything, it's that we like to reinvent this stuff and make it more complicated than necessary (see also XDR, ASN.1, XML, etc. :) pretty much on a 5 year cycle or thereabouts. Do the bare minimum design necessary and let…

Having to many different formats is also a problem though, as incompatible formats means you can't combine two apps in a pipepine. The negotiations in dtools is made using a F_GETLK hack with a magic value offset. That approach could easily be extended to support multiple formats.

The way I see it there's two distinct problems being coupled together, kinda like inventing HTTP but defining it only to be used with (html, gif, png) or something. The reality is that if your solution gets even some adoption, conventions will quickly emerge based on actual use rather than expected use, which almost never goes well. Additionally when some after-market use is discovered that wasn't part of the original spec, yet makes fabulous sense, existing implementations may be better positioned to deal with it (instead of suddenly finding they're being fed PNG files which are actually base64-encoded XML, or something mad like that, typical shoehorning crap).

Similarly, producing a big ecosystem of utilities to go along with it will probably result in a bunch of 1970s style compatibility commands that nobody actually uses any more (say, in 2030).

But feel free to bake a glib-specific serialization in and I'll feel free to pass it up. ;)

Love your fcntl() hack. My kind of hack!

Re: Next generation Unix pipe by Alex Larsson

#54

I've quickly jotted some thoughts here: http://damnkids.posterous.com/rich-format-unix-pipes Regarding this version, standardizing on a particular transfer format is a bad idea. If history has shown anything, it's that we like to reinvent this stuff and make it more complicated than necessary (see also XDR, ASN.1, XML, etc. :) pretty much on a 5 year cycle or thereabouts. Do the bare minimum design necessary and let…

Having to many different formats is also a problem though, as incompatible formats means you can't combine two apps in a pipepine. The negotiations in dtools is made using a F_GETLK hack with a magic value offset. That approach could easily be extended to support multiple formats.

[deleted]

Re: Next generation Unix pipe by Alex Larsson

#56
Actually I prefer Powershell's approach to transfer objects, as it is more flexible than standardize in a specific transfer format.

But I do concede that it has the downside that if the object lacks the properties you want to access, then it might be painful in some cases.

Re: Next generation Unix pipe by Alex Larsson

#57
post #55
post #11

"Even something as basic as numerical sorting on a column gets quite complicated." sort -g -k field_num

Sometimes I feel like POSIX idioms are like the bible to some here: untouchable.

Sometimes I feel like Hacker News stories are like mom's fridge to some here: everything gets two gold stars.

Re: Next generation Unix pipe by Alex Larsson

#58
post #56

Actually I prefer Powershell's approach to transfer objects, as it is more flexible than standardize in a specific transfer format. But I do concede that it has the downside that if the object lacks the properties you want to access, then it might be painful in some cases.

But it is not like you cannot do normal string processing using cmdlets like "Select-String". And an object missing a property is almost same a column missing in the returned text output right?

Re: Next generation Unix pipe by Alex Larsson

#59
post #39
post #31

Earlier quoted context omitted.

> find -print0 is a lame hack, and even filenames with spaces (not newlines) are somewhat messy to work with on the Unix shell. This problem is simply a flaw in sh (and its descendants), other shells handle it much better, see for example Tom Duff's rc shell: http://rc.cat-v.org Also note that Plan 9, the successor to Unix (and which uses the rc shell as its main shell) doesn't even have a find command, find's design…

> This problem is simply a flaw in sh (and its descendants) Indeed, although it's not just sh; if you want to, say, make a table of filenames and some attributes of each file, you're in trouble if the filenames contain spaces (awk, cut, sort don't work as easily) and screwed if they contain newlines. What does Plan 9 use instead of find? > As for your second questions, the answer might be structural regular expressio…

The Plan 9 approach is to avoid creating problems for yourself by not using spaces in file names to begin with. The file server initially disallowed spaces in file names just as nulls and slashes are disallowed. That restriction has since been relaxed,† but everyone still avoids spaces. If you cannot avoid files with spaces in their names, there exists trfs,†† a file system that transparently replaces spaces with something more convenient.

Instead of find, I run /bin/test on a list of files. For anything more complicated than what test can handle, I use Inferno's fs program.†††

http://swtch.com/cgi-bin/plan9history.cgi?f=1999/0323/port/c...

†† http://a-30.net/inferno/man/4/trfs.html

††† http://www.vitanuova.com/inferno/man/1/fs.html It has a misleading name. It is not a file server.

Re: Next generation Unix pipe by Alex Larsson

#60

I've been playing around with a similar idea - using plain JSON as the message format, you can make a set of pipeable command line utilities for manipulating data from many web APIs.

Have you seen RecordStream? It's some good cli tools based on streams of json which might be handy.

https://github.com/benbernard/RecordStream

I've used it a lot and it's a godsend for a most "record-y" manipulation.

Post reply on HN