Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

For the Love of Pipes

blog.jessfraz.com

Post reply on HN
Print view View original

261–270 of 323 posts

Re: For the Love of Pipes

#261

Post by yen223 »

masklinn wrote: post #223
beachy wrote: post #212

Earlier quoted context omitted.

I would hate to see the day HN allowed any way to bold sections of text. It's way more restful purveying a page of uniformly restrained text.

> I would hate to see the day HN allowed any way to bold sections of text. HN already has shitty italics (shitty in that it commonly matches and eats things you don't want to be italicised e.g. multiplications, pointers, … in part though not only because HN doesn't have inline code). "bold" can just be styled as italics, or it can be styled as a medium or semibold. It's not an issue, and even less worth it given how…

For a site that's meant to target programmers, HN's handling of code blocks is pretty poor.

Just give me the triple-tilde code block syntax please!

Re: For the Love of Pipes

#262

Post by kradroy »

I built an entire prototype ML system/pipeline using shell scripts that glued together two python scripts that did some heavy lifting not easily reproduced.

I got the whole thing working from training to prediction in about 3 weeks. What I love about Unix shell commands is that you simply can't abstract beyond the input/output paradigm. You aren't going to create classes, types classes, tests, etc. It's not possible or not worth it.

I'd like to see more devs use this approach, because it's a really nice way to get a project going in order to poke holes in it or see a general structure. I consider it a sketchpad of sorts.

Re: For the Love of Pipes

#263

Post by hk__2 »

nojvek wrote: post #211

I’m surprised JessFraz who is employed by Microsoft doesn’t talk about powershell pipes at all. Powershell pipes are an extension over Unix pipes. Rather than just being able to pipe a stream of bytes, powershell can pipe a stream of objects. It makes working with pipes so much fun. In Unix you have to cut, awk and do all sorts of parsing to get some field out of `ls`. In poweshell, ls outputs stream of file objects…

Don’t you have to rewrite every single program to make it able to read those “objects”?

Re: For the Love of Pipes

#264

Post by noir_lord »

kradroy wrote: post #262

I built an entire prototype ML system/pipeline using shell scripts that glued together two python scripts that did some heavy lifting not easily reproduced. I got the whole thing working from training to prediction in about 3 weeks. What I love about Unix shell commands is that you simply can't abstract beyond the input/output paradigm. You aren't going to create classes, types classes, tests, etc. It's not possible…

My backup system at work is mostly bash scripts and some pipes.

If you write them cleanly they don’t suck and crucially for me, bash today works basically the same way as it did 10 years ago and likely in 10 years, that static nature is a big damn win.

I sometimes wish language vendors would just say ‘this language is complete, all future work will be bug fixes and libraries’ a static target for anything would be nice.

Elixir did say that recently except for one last major change which moved it straight up my list of things to look at in future.

Re: For the Love of Pipes

#265

Post by iamwil »

nojvek wrote: post #211

I’m surprised JessFraz who is employed by Microsoft doesn’t talk about powershell pipes at all. Powershell pipes are an extension over Unix pipes. Rather than just being able to pipe a stream of bytes, powershell can pipe a stream of objects. It makes working with pipes so much fun. In Unix you have to cut, awk and do all sorts of parsing to get some field out of `ls`. In poweshell, ls outputs stream of file objects…

What happens when there are upstream changes to the objects? Does everything downstream just need to change, and hence, the upstream objects returned by programs need to be changed with care? Or is it using something like protobuf, where fields are only additive, but never deleted, for backwards compatibility?

Or are the resulting chain of pipes so short lived, it doesn't matter?

Re: For the Love of Pipes

#266

Post by cryptonector »

__MatrixMan__ wrote: post #149
jimktrains2 wrote: post #71

Earlier quoted context omitted.

To be fair, the same critisim could be used for a socket? I think the issue is that some people want pipes to be something magical that connects their software, not a dumb connection between them.

I don't want all my pipes to be magical all the time, but occasionally I do want to write a utility that is "pipeline aware" in some sense. For example, I'd like to pipe mysql to jq and have one utility or the other realize that a conversion to json is needed in the middle for it work. Im working on a library for this kind of intra-pipeline negitiation. It's all drawing-board stuff right now but I coobbled together a…

This is interesting, yes. If the shell could infer the content type of data demanded or output by each command in a pipeline, then it could automatically insert type coercion commands or alter the options of commands to produce the desired content types.

You're right that it is in fact possible for a command to find the preceding and following commands using /proc, and figure out what content types they produce / want, and do something sensible. But there won't always be just one way to convert between content types...

Me? I don't care for this kind of magic, except as a challenge! But others might like it. You might need to make a library out of this because when you have something like curl(1) as a data source, you need to know what Content-Type it is producing, and when you can know explicitly rather than having to taste the data, that's a plus. Dealing with curl(1) as a sink and somehow telling it what the content type is would be nice as well.

Re: For the Love of Pipes

#267

Post by cryptonector »

jshen wrote: post #163
__MatrixMan__ wrote: post #149

Earlier quoted context omitted.

I don't want all my pipes to be magical all the time, but occasionally I do want to write a utility that is "pipeline aware" in some sense. For example, I'd like to pipe mysql to jq and have one utility or the other realize that a conversion to json is needed in the middle for it work. Im working on a library for this kind of intra-pipeline negitiation. It's all drawing-board stuff right now but I coobbled together a…

I wonder if something like HTTP’s content negotiation is a good model for this.

The problem with that is that each command in the pipeline would have to somehow be modified to convey content type metadata. Perhaps we could have a way to send ancillary metadata (a la Unix domain sockets SCM_*).

Re: For the Love of Pipes

#268

Post by jwilk »

tripa wrote: post #60
jgtrosh wrote: post #24

Earlier quoted context omitted.

Well you can't read either from /dev/null, and I don't think that's just a question of permissions. I'm pretty sure it's impossible to get /dev/null to behave like an executable.

/dev/null behaves like an empty file, which is (or used to be?) a valid executable. Cf http://trillian.mit.edu/~jc/humor/ATT_Copyright_true.html or https://twitter.com/rob_pike/status/966896123548872705

In most contexts empty file is indeed a valid executable. Debian folks learned this the hard way recently:

https://bugs.debian.org/919341

However, executing /dev/null doesn't seem to work on Linux:

  $ sudo chmod 777 /dev/null 
  $ /dev/null
  bash: /dev/null: Permission denied

Re: For the Love of Pipes

#269

Post by cryptonector »

__MatrixMan__ wrote: post #217
jshen wrote: post #163

Earlier quoted context omitted.

I wonder if something like HTTP’s content negotiation is a good model for this.

That sounds reasonable, I'll look into it--thanks. I was imagining an algorithm where each pipeline-aware utility can derive port numbers to use to talk/listen to its neighbors. I may be able to use http content negotiation wholesale in that context.

I was thinking something similar, buried in a library that everyone could link. It seems... awfully awkward to build, much less portably.

This reminds me of how busted Linux is for not having a SO_PEERCRED. You can actually get that information by walking /proc/net/tcp or using AF_NETLINK sockets and inet_diag, but there is a race condition such that this isn't 100% reliable. SO_PEERCRED would [have to] be.

Post reply on HN