Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Pipelining might be my favorite programming language feature

herecomesthemoon.net

Post reply on HN
Print view View original

201–210 of 360 posts

Re: Pipelining might be my favorite programming language feature

#201

Post by switchbak »

eikenberry wrote: post #186
amai wrote: post #143

Pipelining looks nice until you have to debug it. And exception handling is also very difficult, because that means to add forks into your pipelines. Pipelines are only good for programming the happy path.

Pipelining simplifies debugging. Each step is obvious and it is trivial to insert logging between pipeline elements. It is easier to debug than the patterns compared in the article. Exception handing is only a problem in languages that use exceptions. Fortunately there are many modern alternatives in wide use that don't use exceptions.

This is my experience too - when the errors are encoded into the type system, this becomes easier to reason about (which is much of the work when you’re debugging).

Re: Pipelining might be my favorite programming language feature

#202

Post by matthewsinclair »

bnchrch wrote: post #24

I'm personally someone who advocates for languages to keep their feature set small and shoot to achieve a finished feature set quickly. However. I would be lying if I didn't secretly wish that all languages adopted the `|>` syntax from Elixir. ``` params |> Map.get("user") |> create_user() |> notify_admin() ```

Agree. This is absolutely my fave part of Elixir. Whenever I can get something to flow elegantly thru a pipeline like that, I feel like it’s a win against chaos.

Re: Pipelining might be my favorite programming language feature

#203

Post by Alupis »

bnchrch wrote: post #24

I'm personally someone who advocates for languages to keep their feature set small and shoot to achieve a finished feature set quickly. However. I would be lying if I didn't secretly wish that all languages adopted the `|>` syntax from Elixir. ``` params |> Map.get("user") |> create_user() |> notify_admin() ```

Pipelines are one of the greatest Gleam features[1].

[1] https://tour.gleam.run/functions/pipelines/

Re: Pipelining might be my favorite programming language feature

#205

Post by invalidator »

The author keeps calling it "pipelining", but I think the right term is "method chaining".

Compare with a simple pipeline in bash:

  grep needle
Each of those components executes in parallel, with the intermediate results streaming between them. You get a similar effect with coroutines.

Compare Ruby:

  data = File.readlines("haystack.txt")
    .map(&:strip)
    .grep(/needle/)
    .map { |i| i.gsub('foo', 'bar') }
    .map { |i| File.readlines(i).count }
In that case, each line is processed sequentially, with a complete array being created between each step. Nothing actually gets pipelined.

Despite being clean and readable, I don't tend to do it any more, because it's harder to debug. More often these days, I write things like this:

  data = File.readlines("haystack.txt")
  data = data.map(&:strip)
  data = data.grep(/needle/)
  data = data.map { |i| i.gsub('foo', 'bar') }
  data = data.map { |i| File.readlines(i).count }
It's ugly, but you know what? I can set a breakpoint anywhere and inspect the intermediate states without having to edit the script in prod. Sometimes ugly and boring is better.

Re: Pipelining might be my favorite programming language feature

#206

Post by freehorse »

MaxBarraclough wrote: post #136
elbasti wrote: post #39

Earlier quoted context omitted.

In elixir, ```Map.get("user") |> create_user |> notify_admin ``` would aso be valid, standard elixir, just not idiomatic (parens are optional, but preferred in most cases, and one-line pipes are also frowned upon except for scripting).

With the disclaimer that I don't know Elixir and haven't programmed with the pipeline operator before: I don't like that special () syntax. That syntax denotes application of the function without passing any arguments, but the whole point here is that an argument is being passed. It seems clearer to me to just put the pipeline operator and the name of the function that it's being used with. I don't see how it's uncle…

I am also confused with such syntax of "passing as first argument" pipes. Having to write `x |> foo` instead of `x |> foo()` does not solve much, because you have the same lack of clarity if you need to pass a second argument. Ie `x |> foo(y)` in this case means `foo(x,y)`, but if `foo(y)` actually gives you a function to apply to `x` prob you should write `x |> foo(y)()` or `x |> (foo(y))()` then as I understand it? If that even makes sense in a language. In any case, you have the same issue as before, in different contexts `foo(y)` is interpreted differently.

I just find this syntax too inconsistent and vague, and hence actually annoying. Which is why I prefer defining pipes as composition of functions which can then be applied to whatever data. Then eg one can write sth like `(|> foo1 foo2 (foo3) #(foo4 % y))` and know that foo1 and foo2 are references to functions, foo3 evaluates to another function, and when one needs more arguments in foo4 they have to explicitly state that. This gives another function, and there is no ambiguity here whatsoever.

Re: Pipelining might be my favorite programming language feature

#207

Post by threeseed »

rad_gruchalski wrote: post #16
jaymbo wrote: post #3

This is why I love Scala so much

Scala is by far one of the nicest programming languages I have ever worked with. Scala with no JVM dependency would a killer programming language BUT only when all async features work out of the box like they do JVM. It’s been attempted a couple of times and it never succeeded.

a) Scala with no JVM exists: https://scala-native.org or https://www.scala-js.org

b) Async works on Scala Native: https://github.com/lampepfl/gears and is coming to Scala.js.

Re: Pipelining might be my favorite programming language feature

#208

Post by wahern »

invalidator wrote: post #205

The author keeps calling it "pipelining", but I think the right term is "method chaining". Compare with a simple pipeline in bash: grep needle Each of those components executes in parallel, with the intermediate results streaming between them. You get a similar effect with coroutines. Compare Ruby: data = File.readlines("haystack.txt") .map(&:strip) .grep(/needle/) .map { |i| i.gsub('foo', 'bar') } .map { |i| File.re…

> The author keeps calling it "pipelining", but I think the right term is "method chaining". [...] You get a similar effect with coroutines.

The inventor of the shell pipeline, Douglas McIlroy, always understood the equivalency between pipelines and coroutines; it was deliberate. See https://www.cs.dartmouth.edu/~doug/sieve/sieve.pdf It goes even deeper than it appears, too. The way pipes were originally implemented in the Unix kernel was when the pipe buffer was filled[1] by the writer the kernel continued execution directly in the blocked reader process without bouncing through the scheduler. Effectively, arguably literally, coroutines; one process call the write function and execution continues with a read call returning the data.

Interestingly, Solaris Doors operate the same way by design--no bouncing through the scheduler--unlike pipes today where long ago I think most Unix kernels moved away from direct execution switching to better support multiple readers, etc.

[1] Or even on the first write? I'd have to double-check the source again.

Re: Pipelining might be my favorite programming language feature

#209

Post by axblount »

invalidator wrote: post #205

The author keeps calling it "pipelining", but I think the right term is "method chaining". Compare with a simple pipeline in bash: grep needle Each of those components executes in parallel, with the intermediate results streaming between them. You get a similar effect with coroutines. Compare Ruby: data = File.readlines("haystack.txt") .map(&:strip) .grep(/needle/) .map { |i| i.gsub('foo', 'bar') } .map { |i| File.re…

Syntactic sugar can sometimes fool us into thinking the underlying process is more efficient or streamlined. As a new programmer, I probably would have assumed that "storing" `data` at each step would be more expensive.

Re: Pipelining might be my favorite programming language feature

#210

Post by jjmarr »

senderista wrote: post #86
inetknght wrote: post #76

Earlier quoted context omitted.

This is not functionally different from operator<< which std::cout has taught us is a neat trick but generally a bad idea.

Unlike the iostreams shift operators, the ranges pipe operator isn't stateful.

There's state when you try to use the final result, though. It's not threadsafe due to caching.

https://www.youtube.com/watch?v=c1gfbbE2zts

Post reply on HN