Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Pipelining might be my favorite programming language feature

herecomesthemoon.net

Post reply on HN
Print view View original

291–300 of 360 posts

Re: Pipelining might be my favorite programming language feature

#291

Post by hiq »

3np wrote: post #243
invalidator wrote: post #205

The author keeps calling it "pipelining", but I think the right term is "method chaining". Compare with a simple pipeline in bash: grep needle Each of those components executes in parallel, with the intermediate results streaming between them. You get a similar effect with coroutines. Compare Ruby: data = File.readlines("haystack.txt") .map(&:strip) .grep(/needle/) .map { |i| i.gsub('foo', 'bar') } .map { |i| File.re…

I have to object against reusing the 'data' var. Make up a new name for each assignment in particular when types and data structures change (like the last step is switching from strings to ints). Other than that I think both styles are fine.

I agree with this comment: https://news.ycombinator.com/item?id=43759814 that this pollutes current scope, which is especially bad if scoping is not that narrow (the case in Python where if-branches do not define their own scope, I don´t know for Ruby).

Another problem of having different names for each step is that you can no longer quickly comment out a single step to try things out, which you can if you either have the pipeline or a single variable name.

Re: Pipelining might be my favorite programming language feature

#292

Post by bjourne »

In concatenative languages with an implicit stack (Factor) that expression would read:

    iter [ alive? ] filter [ id>> ] map collect
The beauty of this is that everything can be evaluated strictly left-to-right. Every single symbol. "Pipelines" in other languages are never fully left-to-right evaluated. For example, ".filter(|w| w.alive)" in the author's example requires one to switch from postfix to infix evaluation to evaluate the filter application.

The major advantage is that handling multiple streams is natural. Suppose you want to compute the dot product of two files where each line contains a float:

    fileA fileB [ lines [ str>float ] map ] bi@ [ mul ] 2map 0 [ + ] reduce

Re: Pipelining might be my favorite programming language feature

#294

Post by inkyoto »

invalidator wrote: post #205

The author keeps calling it "pipelining", but I think the right term is "method chaining". Compare with a simple pipeline in bash: grep needle Each of those components executes in parallel, with the intermediate results streaming between them. You get a similar effect with coroutines. Compare Ruby: data = File.readlines("haystack.txt") .map(&:strip) .grep(/needle/) .map { |i| i.gsub('foo', 'bar') } .map { |i| File.re…

> Each of those components executes in parallel, with the intermediate results streaming between them. You get a similar effect with coroutines.

Processes run in parallel, but they process the data in a strict sequential order: «grep» must produce a chunk of data before «sed» can proceed, and «sed» must produce another chunk of data before «xargs» can do its part. «xargs» in no way can ever pick up the output of «grep» and bypass the «sed» step. If the preceding step is busy crunching the data and is not producing the data, the subsequent step will be blocked (the process will fall asleep). So it is both, a pipeline and a chain.

It is actually a directed data flow graph.

Also, if you replace «haystack.txt» with a /dev/haystack, i.e.

  grep needle
and /dev/haystack is waiting on the device it is attached to to yield a new chunk of data, all of the three, «grep», «sed» and «xargs» will block.

Re: Pipelining might be my favorite programming language feature

#295

Post by raverbashing »

invalidator wrote: post #205

The author keeps calling it "pipelining", but I think the right term is "method chaining". Compare with a simple pipeline in bash: grep needle Each of those components executes in parallel, with the intermediate results streaming between them. You get a similar effect with coroutines. Compare Ruby: data = File.readlines("haystack.txt") .map(&:strip) .grep(/needle/) .map { |i| i.gsub('foo', 'bar') } .map { |i| File.re…

Exactly that. It looks nice but it's annoying to debug

I do it in a similar way you mentioned

Re: Pipelining might be my favorite programming language feature

#296

Post by EVa5I7bHFq9mnYK »

hnlmorg wrote: post #279
EVa5I7bHFq9mnYK wrote: post #260

Earlier quoted context omitted.

I think they are talking about nested loops, not nested function calls.

Nested loops isn’t pipelining. Some of the examples make heavy use of lambda so they do have nested loops happening as well but in those examples the pipelining logic is still the nesting of the lambda functions. Crudely put, in C-like languages, pipelining is just as way of turning fn(fn(fn())) Where the first function call is in the inner, right-most, parentheses, into this: fn | fn | fn …which can be easily read s…

Pipelining replaces several consecutive loops with a single loop, doing more complex processing.

Re: Pipelining might be my favorite programming language feature

#297

Post by aeonik »

gleenn wrote: post #183
Straw wrote: post #134

Lisp macros allow a general solution to this that doesn't just handle chained collection operators but allows you to decide the order in which you write any chain of calls. For example, we can write: (foo (bar (baz x))) as (-> x baz bar foo) If there are additional arguments, we can accommodate those too: (sin (* x pi) as (-> x (* pi) sin) Where expression so far gets inserted as the first argument to any form. If yo…

I find the threading operators in Clojure bring much joy and increase readability. I think it's interesting because it makes me actually consider function argument order much more because I want to increase opportunities to use them.

These threading macros can increase performance, the developer even has a parallelizing threading macro.

I use these with xforms transducers.

https://github.com/johnmn3/injest

Re: Pipelining might be my favorite programming language feature

#298

Post by mrkeen »

usrusr wrote: post #261
vips7L wrote: post #120

Earlier quoted context omitted.

I really wish you couldn't write extensions on nullable types. It's confusing to be able to call what look like instance functions on something clearly nullable without checking. fun main() { val s: String? = null println(s.isS()) // false } fun String?.isS() = "s" == this

The difference between .let{} and ?.let{} has great utility. You'd either have to give that up or promote let from regular code in the standard library to magic language feature. And you'd lose all those cases of extension methods where the convenience of accepting null left of the dot is their sole reason to be. Null is a valid state, not something incredibly scary best dealt with with a full reboot or better yet th…

> Null is a valid state, not something incredibly scary best dealt with with a full reboot or better yet throwing away the container.

Agreed. It should be a first-class construct in a language with its own own proper type,

  Null null;
rather than needing to hitch a ride with the Integers and Strings like a second-class construct.

Re: Pipelining might be my favorite programming language feature

#299

Post by davemp »

Computer scientists continue to pick terrible names. Pipelining is already an overloaded concept that implies some type of operation level parallelism. Picking names like this does everyone in the field a disservice. Calling it something like “composition chain” would be much clearer with respect to existing literature in the field. Maybe I’m being nitpicky, but sometimes it feels like the tower of babel parable talking to folks who use different ecosystems.

Re: Pipelining might be my favorite programming language feature

#300

Post by benrutter »

Straw wrote: post #134

Lisp macros allow a general solution to this that doesn't just handle chained collection operators but allows you to decide the order in which you write any chain of calls. For example, we can write: (foo (bar (baz x))) as (-> x baz bar foo) If there are additional arguments, we can accommodate those too: (sin (* x pi) as (-> x (* pi) sin) Where expression so far gets inserted as the first argument to any form. If yo…

Yeah, I found this when I was playing around with Hy a while back. I wanted a generic `->` style operator, and isn't wasn't too much trouble to write a macro to introduce one.

That's sort of an argument for the existence of macros as a whole, you can't really do this as neatly in something like python (although I've tried) - I can see the downside of working in a codebase with hundreds of these kind of custom language features though.

Post reply on HN