Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Next generation Unix pipe by Alex Larsson

blogs.gnome.org

Post reply on HN
Print view View original

81–86 of 86 posts

Re: Next generation Unix pipe by Alex Larsson

#81

Post by flogic »

pka wrote: post #67

I can't believe this. Just 2 or so weeks ago I set about writing exactly something like this in Haskell [1]. It's by no means complete or even working at this point, but basically what I had in mind was something like: yls | yfilter 'mdate = yesterday && permissions.oread = true' | yformat -ls Every tool emits or consumes "typed" JSON (i.e. JSON data with an additional JSON schema). Why typed? Because then the meanin…

Personally, I'm not feeling the quotes and would prefer parens since they're nestable.

Re: Next generation Unix pipe by Alex Larsson

#82

Post by Someone »

veyron wrote: post #68
Someone wrote: post #65

Earlier quoted context omitted.

If "sometimes the app can take advantage of a constraint" is an argument here, you should be against all usage of pipes.

That's not true. So in the case of `ps`, there is a known limit to the number of processes, and it is fairly small, so the performance hit is limited. As another example in this context, if the original data source is gzip'd, it's faster to gunzip and then pipe rather than integrating the gzip logic into the app itself.

I still disagree. I think you are arguing for the inclusion of, at the least, grep, cut, head and tail in cat.

I do not claim that is a bad idea (conceptually, pipes do not require multiple processes, and those tools could be dynamically linked in) but why stop at those tools? Some people would argue that sed and awk also should be in, others would mention perl, etc.

I also do not see why it would be faster to use an external gzip tool through a pipe. If it is, the writer of the 'tool with built-in unzip' could always, in secret, start an external unzip process to do the work.

Re: Next generation Unix pipe by Alex Larsson

#83

Post by rogerbinns »

alexlarsson wrote: post #78
rogerbinns wrote: post #77

Earlier quoted context omitted.

But you still have race conditions. The sender would have to ensure that the locks are setup before the receiver calls read() for the first time. Since pipes are often setup by the shell you have no control over the startup times. Sure you could have heuristics such as the receiver waiting a few seconds just in case locks show up, but that just makes things slow and unpredictable. I stand by my assertion that this ca…

No, I avoid the race condition by: 1) Reader sets the lock before reading any data 2) Writer writes a byte to the pipe 3) Writer waits until pipe is empty (FIONREAD ioctl) 4) Writer checks for existance of lock. This should be race free.

That requires the first byte sent be compatible with whatever format is ultimately used. As an example for ps, the first byte in JSON should be a { while for plain text it should be space. (We get a little lucky since a space would also be acceptable for JSON, but I doubt there is a universal first byte.) And an initial space isn't accept for a programs like find or grep in either text or null separation mode.

I don't want to belittle what you've done, but the point remains. This can't be done robustly without an additional system call. What you have is tantalizingly close. Even a call as simple as telling the sender that the receiver has called read() would complete your solution.

Re: Next generation Unix pipe by Alex Larsson

#84

Post by agumonkey »

mongol wrote: post #49

For certain types of unix pipeing, I have found it useful to pipe from tool to CSV, and then let sqlite process the data, using SQL statements. SQL solves many of the sorting, filtering, joining things that you can do with unix pipes too, but with a syntax that is broadly known. Especially the joining I have found hard to do well with shell/piping. I think a sqlite-aware shell would be awesone, especially if common t…

very clever

Re: Next generation Unix pipe by Alex Larsson

#85

Post by uvdiv »

vidarh wrote: post #69
alexlarsson wrote: post #42

Earlier quoted context omitted.

I don't expect every user to create unix pipes 2.0, so the difficulty of that is not really what needs to be compared. It will only have to be done once. And once this is done any user can avoid having to painstakingly construct pipelines that try to cut out the right columns to treat as numbers, or avoid all the problems parsing strings that may contain spaces or other control characters. You can do an operation lik…

Your example would be about the same length with awk and sort, with the only caveat that you need to figure out the field numbers, and the upside that I can trust the tools are available pretty much everywhere.

Well, what's the awk/sort version?

Re: Next generation Unix pipe by Alex Larsson

#86

Post by alexlarsson »

rogerbinns wrote: post #83
alexlarsson wrote: post #78

Earlier quoted context omitted.

No, I avoid the race condition by: 1) Reader sets the lock before reading any data 2) Writer writes a byte to the pipe 3) Writer waits until pipe is empty (FIONREAD ioctl) 4) Writer checks for existance of lock. This should be race free.

That requires the first byte sent be compatible with whatever format is ultimately used. As an example for ps, the first byte in JSON should be a { while for plain text it should be space. (We get a little lucky since a space would also be acceptable for JSON, but I doubt there is a universal first byte.) And an initial space isn't accept for a programs like find or grep in either text or null separation mode. I don'…

Not really. All you need to be able to do is to produce the first byte of whatever would have been produced in the "fallback case", i.e. when the reader does not handle format negotiation. Then, when the writer sees that the reader supports format negotiation it will need to signal that the alternative format was chosen. I do this by sending a zero byte (which should never appear in the fallback text format). Then the two first bytes are skipped as part of the negotiation framework when a non-fallback format was chosen.
Post reply on HN