Live data from Hacker News

Pipelining might be my favorite programming language feature

herecomesthemoon.net

351–360 of 360 posts

Re: Pipelining might be my favorite programming language feature

#351

Earlier quoted context omitted.

It kind of is something related to loops if the language supports object iterator interfaces (they bridge OOP to classical constructs like for/foreach). Or maybe even generators. It does not solve it magically, but it does give the programmer options to coalesce different paradigms into one single working implementation.

Sure, but again, you’re talking about using either lambda functions or object methods. The crux of the “magic” with pipelining is chaining functions. How those functions are composed and what they do is a separate topic entirely.

It's only magic if you don't understand how it works.

To me it is awkward to describe but simple to understand. Lucky me I have no intention of describing it.

Re: Pipelining might be my favorite programming language feature

#352

Earlier quoted context omitted.

Sure, but again, you’re talking about using either lambda functions or object methods. The crux of the “magic” with pipelining is chaining functions. How those functions are composed and what they do is a separate topic entirely.

It's only magic if you don't understand how it works. To me it is awkward to describe but simple to understand. Lucky me I have no intention of describing it.

> It's only magic if you don't understand how it works.

That’s why I used scare quotes around the term ;)

> To me it is awkward to describe but simple to understand.

It’s not awkward to describe though. It’s literally just syntactic sugar for chaining functions.

It’s probably one of the easiest programming concepts to describe.

From our conversation, I’m not sure you do understand it because you keep bringing other tangential topics into the fold. Granted I don’t think the article does a great job at explaining what pipelining is, but then I think it’s point was more to demonstrate cool syntactic tricks you can pull off when writing functions as a pipeline.

edit: just realised you aren't the same person who wrote the original comment claiming pipelining was about iteration. Apologies for getting you mixed together.

Re: Pipelining might be my favorite programming language feature

#353

Earlier quoted context omitted.

It's only magic if you don't understand how it works. To me it is awkward to describe but simple to understand. Lucky me I have no intention of describing it.

> It's only magic if you don't understand how it works. That’s why I used scare quotes around the term ;) > To me it is awkward to describe but simple to understand. It’s not awkward to describe though. It’s literally just syntactic sugar for chaining functions. It’s probably one of the easiest programming concepts to describe. From our conversation, I’m not sure you do understand it because you keep bringing other t…

Pipelining, as discussed in the linked article, is about iteration. Just look at the code example at the beginning of the article:

data.iter()

.filter(|w| w.alive)

.map(|w| w.id)

.collect()

is one loop, as opposed to

collect(map(filter(iter(data), |w| w.alive), |w| w.id),

which is three loops.

Did you notice four letters 'i', 't', 'e' and 'r' in the code, followed by two round brackets? They mean "iterator".

Re: Pipelining might be my favorite programming language feature

#354

The author keeps calling it "pipelining", but I think the right term is "method chaining". Compare with a simple pipeline in bash: grep needle Each of those components executes in parallel, with the intermediate results streaming between them. You get a similar effect with coroutines. Compare Ruby: data = File.readlines("haystack.txt") .map(&:strip) .grep(/needle/) .map { |i| i.gsub('foo', 'bar') } .map { |i| File.re…

Syntactic sugar can sometimes fool us into thinking the underlying process is more efficient or streamlined. As a new programmer, I probably would have assumed that "storing" `data` at each step would be more expensive.

It depends on the language you're using.

For my Ruby example, each of those method calls will allocate an Array on the heap, where it will persist until all references are removed and the GC runs again. The extra overhead of the named reference is somewhere between Tiny and Zero, depending on your interpreter. No extra copies are made; it's just a reference.

In most compiled languages: the overhead is exactly zero. At runtime, nothing even knows it's called "data" unless you have debug symbols.

If these are going to be large arrays and you actually care about memory usage, you wouldn't write the code the way I did. You might use lazy enumerators, or just flatten it out into a simple procedure; either of those would process one line at a time, discarding all the intermediate results as it goes.

Also, "File.readlines(i).count" is an atrocity of wasted memory. If you care about efficiency at all, that's the first part to go. :)

Re: Pipelining might be my favorite programming language feature

#355

Earlier quoted context omitted.

> It's only magic if you don't understand how it works. That’s why I used scare quotes around the term ;) > To me it is awkward to describe but simple to understand. It’s not awkward to describe though. It’s literally just syntactic sugar for chaining functions. It’s probably one of the easiest programming concepts to describe. From our conversation, I’m not sure you do understand it because you keep bringing other t…

Pipelining, as discussed in the linked article, is about iteration. Just look at the code example at the beginning of the article: data.iter() .filter(|w| w.alive) .map(|w| w.id) .collect() is one loop, as opposed to collect(map(filter(iter(data), |w| w.alive), |w| w.id), which is three loops. Did you notice four letters 'i', 't', 'e' and 'r' in the code, followed by two round brackets? They mean "iterator".

If you're going to be snarky then at least get your facts right.

`Iter` is a method of `data`. And do you know what a method is? It's a function attached to an object. A FUNCTION. Pipelining is just syntactic sugar around chaining functions.

You even proved my point when you quoted the article:

    collect(map(filter(iter(data), |w| w.alive), |w| w.id))
Literally the only thing changing is the syntax of the code. You've got all of the same functions being called, with the same parameters and in the same order.

The article itself makes no mention of this affecting how the code is executed either. Instead, it talks about code readability.

In fact the article further proves my point when it says:

> You can, of course, just assign the result of every filter and map call to a helper variable, and I will (begrudgingly) acknowledge that that works, and is significantly better than trying to do absurd levels of nesting.

What it means by this is something like the following:

  list = iter(data)
  list = map(channel, |w| w.toWingding())
  list = filter(list, |w| w.alive)
  list = map(list, |w| w.id)
  result = collect(list)
While I do have some experience in this field (having written a pipeline-orientated programming language from scratch), I'll cite some other sources too, so it's not just my word against yours:

+ Wikipedia: https://en.wikipedia.org/wiki/Pipeline_(computing) (no mention of iteration, just chaining functions and processes)

+ JavaScript proposal: https://www.geeksforgeeks.org/javascript-pipeline-operator/ (it's very clear how pipelining works in this guide)

+ Pipeline macros in LISP: https://blog.fugue88.ws/archives/2022-03/Pipelines-in-Lisp (again, literally just talking about cleaner syntax for nested functions)

The reason the article focuses on map/reduce type functions is because it's a common idiom for nesting commands. In fact you'll be familiar with this in Bash:

    cat largefile.txt | sort | uniq --count
(before you argue about "useless use of `cat`" and other optimisations that could be made, this is just an example to demonstrate my point).

In here, each command is a process but analogous to a function in general-purpose programming languages like Rust, LISP, Javascript, etc. Those UNIX processes might internally loop through the contents of STDIN as a LF-delimited list but that happens transparently to the pipeline. Bash, when piping each command to the next, doesn't know how each process will internally operate. And likewise, in general-purpose programming language world, pipelines in LISP, Rust, JavaScript (et al) don't know nor care how each function behaves internally with it's passed parameters just so long as the output data type is compatible with the data type of the next function -- and if it isn't, then that's an error in the code (ie compile-time error in Rust or runtime error in Javascript).

So to summerise, pipelining has nothing to do with iteration. It's just syntactic sugar to make nested functions easier to read. And if the examples seem to focus on map/reduce, it's just because that's a common set of functions you'd want to chain and which are particularly ugly to read in nested form. ie they're an example of functions called in a pipeline, not the reason pipelines exist nor proof that pipelines themselves have any internal logic around iteration.

Re: Pipelining might be my favorite programming language feature

#356

Earlier quoted context omitted.

Pipelining, as discussed in the linked article, is about iteration. Just look at the code example at the beginning of the article: data.iter() .filter(|w| w.alive) .map(|w| w.id) .collect() is one loop, as opposed to collect(map(filter(iter(data), |w| w.alive), |w| w.id), which is three loops. Did you notice four letters 'i', 't', 'e' and 'r' in the code, followed by two round brackets? They mean "iterator".

If you're going to be snarky then at least get your facts right. `Iter` is a method of `data`. And do you know what a method is? It's a function attached to an object. A FUNCTION . Pipelining is just syntactic sugar around chaining functions. You even proved my point when you quoted the article: collect(map(filter(iter(data), |w| w.alive), |w| w.id)) Literally the only thing changing is the syntax of the code. You've…

[deleted]

Re: Pipelining might be my favorite programming language feature

#357

Earlier quoted context omitted.

Pipelining, as discussed in the linked article, is about iteration. Just look at the code example at the beginning of the article: data.iter() .filter(|w| w.alive) .map(|w| w.id) .collect() is one loop, as opposed to collect(map(filter(iter(data), |w| w.alive), |w| w.id), which is three loops. Did you notice four letters 'i', 't', 'e' and 'r' in the code, followed by two round brackets? They mean "iterator".

If you're going to be snarky then at least get your facts right. `Iter` is a method of `data`. And do you know what a method is? It's a function attached to an object. A FUNCTION . Pipelining is just syntactic sugar around chaining functions. You even proved my point when you quoted the article: collect(map(filter(iter(data), |w| w.alive), |w| w.id)) Literally the only thing changing is the syntax of the code. You've…

Yeah, you are right, it's about syntactic sugar, I didn't read the article except first two examples.

Pipelines are about iteration, of course. And they do have internal logic around iteration.

cat largefile.txt | sort | uniq --count

is an excellent example. While cat and count iterate on each character sequentially, sort and uniq require buffering and allocating additional structures.

Re: Pipelining might be my favorite programming language feature

#358

Earlier quoted context omitted.

If you're going to be snarky then at least get your facts right. `Iter` is a method of `data`. And do you know what a method is? It's a function attached to an object. A FUNCTION . Pipelining is just syntactic sugar around chaining functions. You even proved my point when you quoted the article: collect(map(filter(iter(data), |w| w.alive), |w| w.id)) Literally the only thing changing is the syntax of the code. You've…

Yeah, you are right, it's about syntactic sugar, I didn't read the article except first two examples. Pipelines are about iteration, of course. And they do have internal logic around iteration. cat largefile.txt | sort | uniq --count is an excellent example. While cat and count iterate on each character sequentially, sort and uniq require buffering and allocating additional structures.

The pipeline isn’t doing any of that though. The commands are. The pipeline is just connecting the output of one command to the input of the other.

Iteration is about looping and if the pipeline in the above example was some kind of secret source for iteration then the commands above would fork multiple times, but they don’t.

Re: Pipelining might be my favorite programming language feature

#359

I think there's a language syntax to be invented that would make everything suffix/pipeline-based. Stack based languages are kind of there, but I don't think exactly the same thing. BTW. For people complaining about debug-ability of it: https://doc.rust-lang.org/std/iter/trait.Iterator.html#metho... etc.

What do you think stack based languages like Factor miss in this regard?

I don't want to be manipulating stack. I'd prefer something like Hoon.

Re: Pipelining might be my favorite programming language feature

#360
post #93
post #87

Earlier quoted context omitted.

I would wager within a rounding error, all humans have a lifetime of experience in following directions of the form: 1. do the first step in the process 2. then do the next thing 3. followed by a third action I struggle to think of any context outside of programming, retrosynthesis in chemistry, and some aspects of reverse-Polish notation calculators, where you conceive of the operations/arguments last-to-first. All…

Consistency is more important. If you ever wrote: a(b()) then you're already breaking your left-to-right/first-to-last rule.

[deleted]
Post reply on HN