Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

For the Love of Pipes

blog.jessfraz.com

Post reply on HN
Print view View original

311–320 of 323 posts

Re: For the Love of Pipes

#311

Post by wglb »

wglb wrote: post #186
Asooka wrote: post #156

Earlier quoted context omitted.

The section on find after pipes has also not aged well. I can see why GNU and later GNU/Linux replaced most of the old Unices (I mean imagine having a find that doesn't follow symlinks!). If I may, a bit of code golf on the problem of "print all .el files without a matching .elc" find . -name '*.el' | while read el; do [ -f "${el}c" ] || echo $el; done Of course this uses the dreaded pipes and doesn't support the ext…

So the dreaded space-in-filenames is a problem when you pass the '{}' to a script. The following works very nicely for me: * find . -name '*.el' -exec file {}c ';' 2>&1 | grep cannot

I should have said "works very nicely for me, including on file names with spaces"

Re: For the Love of Pipes

#312

Post by jancsika »

jcelerier wrote: post #259
jancsika wrote: post #213

Earlier quoted context omitted.

> Like always, simple stuff will be simple ( http://write.flossmanuals.net/pure-data/wireless-connections... ) That is not actually simple because the data is flowing across two completely different message passing paradigms . Many users of Max/MSP and Pd don't understand the rules for such dataflow, even though it is deterministic and laid out in the manual IIRC. The "silver bullet" in Max/MSP would be to only use t…

> The "silver bullet" in Max/MSP would be to only use the DSP message passing paradigm. There, all objects are guaranteed to receive their input before they compute their output. in one hand, this simplifies the semantics (and it's the approach I've been using in my visual language ( https://ossia.io) ), but in the other it tanks performances if you have large numbers of nodes... I've worked on Max patches with thous…

That is certainly true. My point is that this is a drawback to the implementation of one set of visual programming languages, not necessarily a drawback of visual programming languages.

I can't remember the name of it, but there's a Pd-based compiler that can take patches and compile them down to a binary that performs perhaps an order of magnitude faster. I can't remember if it was JIT or not. Regardless, there's no conceptual blocker to such a JIT-compiled design. In fact there's a version of [expr] that has such a JIT-compiler backing it-- the user takes a small latency hit at instantiation time, but after that there's a big performance increase.

The main blocker as you probably know is time and money. :)

Re: For the Love of Pipes

#314

Post by bjoli »

talkingtab wrote: post #119

I'm not a Clojure person, but there are transducers, https://clojure.org/reference/transducers .

Which are in a way comparable, but I'd say that pipes are more like the arrow ->.

Transducers are compostable algorithmic transformations that, in a way, generalize map, filter and friends but they can be used anywhere where you transform data. Transducers have to be invoked and handled in a way that pipes do not.

Anyone interested should check out Hickeys talks about them. They are generally a lot more efficient than chaining higher order list processing functions and since they don't build.intermediate results they have a lot better GC performance.

Re: For the Love of Pipes

#315

Post by anthk »

Crestwave wrote: post #302
anthk wrote: post #293

Earlier quoted context omitted.

awk can do all of that except sed. And I am not sure about the last. No need to wc ($NF in AWK, if I can recall), no need for grep, you have the /match/ statement, with regex too.

> except sed Doesn't gsub(/a/, "b") do the same thing as s/a/b/g?

Yes, I recall it hours ago.

Re: For the Love of Pipes

#316

Post by anthk »

leethargo wrote: post #301
anthk wrote: post #292

Earlier quoted context omitted.

Avoid TAOUP, is really bad. Most of the lore have stolen from places as LISP and VAX communities. ESR is to Pike and Ken as alien as X11 itself.

I enjoyed it when I read it many years ago, but maybe that was because I was inexperienced and naive. Could you recommend some "original sources" to learn from, instead? Ideally in book form?

Not TAOUP specially, but the Jargon file is what ESR took loads of things as wrong or Unix related. Also, at TAOUP you have Emacs, which is the Anti-UNIX by definition. https://www.dourish.com/goodies/jargon.html

Re: For the Love of Pipes

#317

Post by vharuck »

WhompingWindows wrote: post #191

Pipes have been a game-changer for me in R with the tidyverse suite of packages. Base R doesn't have pipes, requiring a bit more saving of objects or a compromise on code readability. One criticism would be that ggplot2 uses the "+" to add more graph features, whereas the rest of tidyverse uses "%>%" as its pipe, when ideally ggplot2 would also use it. One of my most common errors with ggplot2 is not utilizing the +…

I've always thought of ggplot2's process as building a plot object. Most steps only add to the input.

Of course, Hadley admitted it was because he wrote ggplot2 before adopting pipes into his packages.

Re: For the Love of Pipes

#318

Post by xelxebar »

enriquto wrote: post #308
anthk wrote: post #280

Earlier quoted context omitted.

Never parse ls.

> Never parse ls. I have heard this several times, but either I do not understand it or I disagree. Do you mean parsing the output of the ls program? Parsing ls output is not wrong, the program produces a text stream that is easy and useful to parse. There's nothing to be ashamed when doing it, even when you can do it in a different, even shorter way. I do grep over ls output daily, and I find it much more convenient…

One can certainly do fine by grepping ls output in one-off instances, but I'd be really hesitant to put that in a script.

For given paths, stat command essentially lets us directly access their inode structs, and invocations are nicely concise. The find util then lets us select files based on inode fields.

Both tools do take a bit of learning, but considerably less than grep and regexs. Anyway, I've personally found find and stat to be really nice and ergonomic after the initial learning period.

Re: For the Love of Pipes

#319

Post by laumars »

__MatrixMan__ wrote: post #298
laumars wrote: post #276

Earlier quoted context omitted.

I've been trying to solve the exact same problem with my shell too. It's pipes are typed and all the builtin commands can than automatically decode those data types via shared libraries. So commands don't need to worry about how to decode and re-encode the data. This means that JSON, YAML, TOML, CSV, Apache log files, S-Expressions and even tabulated data from `ps` (for example) can all be transparently handled the s…

> ...with my shell too... I was hoping to stick with bash or zsh, and just write processes that somehow communicate out of band, but I think we're still up against the same problem. One idea I had was that there's a service running elsewhere which maintains this directed graph (nodes = types, edges = programs which take the type of their "from" node and return the type of their "two" node). When a pipeline is execute…

> I was hoping to stick with bash or zsh, and just write processes that somehow communicate out of band, but I think we're still up against the same problem.

Yeah we have different starting points but very much similar problems.

tbh idea behind my shell wasn't originally to address typed pipelines, that was just something that evolved from it quite by accident.

Anyhow, your suggestion of overwriting / aliasing `ssh` is genius. Though I'm thinking rather than tunnelling a TCP connection, I could just spawn an instance of my shell on the remote server and then do everything through normal pipelines as I now control both ends of the pipe. It's arguably got less proverbial moving parts compared to a TCP listener (which might then require a central data type daemon et al) and I'd need my software running on the remote server for the data types to work anyway.

There is obviously a fair security concern some people might have about that but if we're open and honest about that and offer an "opt in/out" where opting out would disable support for piped types over SSH then I can't see people having an issue with it.

Coincidentally I used to do something similar in a previous job where I had a pretty feature rich .bashrc and no Puppet. So `ssh` was overwritten with a bash function to copy my .bashrc onto the remote box before starting the remote shell.

> This idea is, of course, over-engineered as hell. But then again this whole pursuit is.

Haha so true!

Thanks for your help. You may have just solved a problem I've been grappling with for over a year.

Re: For the Love of Pipes

#320

Post by chriswarbo »

sideeffffect wrote: post #229
VladimirGolovin wrote: post #110

Earlier quoted context omitted.

It has, in the form of function composition, as other replies show. However, the Unix pipe demonstrates a more interesting idea: composable programs on the level of the OS. Nowadays, most of the user-facing desktop programs have GUIs, so the 'pipe' operator that composes programs is the user himself. Users compose programs by saving files from one program and opening them in another. The data being 'piped' through su…

maybe I'm being overly pedantic, but people seem to be confused about this: the pipes in your typical functional language (`|>`) is not a form of function composition, like ``` f >> g === x -> g(f(x)) ``` but function application, like ``` f x |> g === g(f(x)) x |> f |> g // also works, has the same meaning f |> g // just doesn't work, sorry :( ```

> the pipes in your typical functional language (`|>`) is not a form of function composition

What is a "typical functional language" in this case? I don't think I've come across this `|>` notation, or anything explicitly referred to as a "pipe", in the functional languages I tend to use (Haskell, Scheme, StandardML, Idris, Coq, Agda, ...); other than the Haskell "pipes" library, which I think is more elaborate than what you're talking about.

Post reply on HN