Live data from Hacker News

Pipelining might be my favorite programming language feature

herecomesthemoon.net

341–350 of 360 posts

Re: Pipelining might be my favorite programming language feature

#341

PowerShell has the best pipeline capability of any language I have ever seen. For comparison, UNIX pipes support only trivial byte streams from output to input. PowerShell allows typed object streams where the properties of the object are automatically wired up to named parameters of the commands on the pipeline. Outputs at any stage can not only be wired directly to the next stage but also captured into named variab…

Completely agree. PowerShell is what got me hooked on scripting/programming and the pipeline is such an elegant construct.

Re: Pipelining might be my favorite programming language feature

#342

(Un)surprisingly, the author ignores this is almost already a thing in JS. What a terrible oversight. Anyway, JS wins again, give it a try if you haven't, it's one of the best languages out there.

TypeScript is a beauty to work with. It's a shame that it's most often used in web contexts though. I would love to write a game engine that uses TypeScript as the scripting language.

Indeed!

Re: Pipelining might be my favorite programming language feature

#343
post #262

Earlier quoted context omitted.

The person you are quoting already conceded that is less readable, but that the ability to set a breakpoint easily (without having to stop the process and modify the code) is more important. I myself agree, and find myself doing that too, especially in frontend code that executes in a browser. Debuggability is much more important than marginally-better readability, for production code.

> Debuggability is much more important than marginally-better readability, for production code. I find this take surprising. I guess it depends on how much weight you give to "marginally-better", but IMHO readability is the single most important factor when it comes to writing code in most code-bases. You write code once, it may need to be debugged (by yourself or others) on rare occasions. However anytime anyone nee…

Yeah, part of it is that I do find

    const foo = something()
      .hoge()
      .hige()
      .hage();
better, sure, but not actually significantly harder to read than:

    let foo = something();
    foo = foo.hoge();
    foo = foo.hige();
    foo = foo.hage();
But, while reading is more common than debugging, debugging a production app is often more important. I guess I am mostly thinking about web apps, because that is the area where I have mainly found the available debuggers lacking. Although they are getting better, I believe, I've frequently seen problems where they can't debug into some standard language feature because it's implemented in C++ native code, or they just don't expose the implicit temporary variables in a useful way.

(I also often see similar-ish problems in languages where the debuggers just aren't that advanced, due to lack of popularity, or whatever.)

Particularly with web apps, though, we often want to attach to the current production app for initial debugging instead of modifying the app and running it locally, usually because somebody has reported a bug that happens in production (but how to reproduce it locally is not yet clear).

Alternatively stated, I guess, I believe readability is important, and maybe the "second most important thing", but nevertheless we should not prefer fancy/elegant code that feels nice to us to write and read, but makes debugging more difficult (with the prevailing debuggers) in any significant way.

In an ideal world, a difference like the above wouldn't be harder to debug, in which case I would also prefer the first version.

(And probably in the real world, the problems would be with async functions less conducive to the pithy hypothetical example. I'm a stalwart opponent of libraries like RxJs for the sole reason that you pay back with interest all of the gains you realized during development, the first time you have to debug something weird.)

Re: Pipelining might be my favorite programming language feature

#344

Earlier quoted context omitted.

Nested loops isn’t pipelining. Some of the examples make heavy use of lambda so they do have nested loops happening as well but in those examples the pipelining logic is still the nesting of the lambda functions. Crudely put, in C-like languages, pipelining is just as way of turning fn(fn(fn())) Where the first function call is in the inner, right-most, parentheses, into this: fn | fn | fn …which can be easily read s…

Pipelining replaces several consecutive loops with a single loop, doing more complex processing.

Pipelining doesn’t do anything with iterating. It’s entirely about linking nested functions.

What you’re looking at is loops defined inside lambda functions. Pipelining makes it much easier to use anonymous functions and lambdas. But it doesn’t magically solve the problem of complex loops.

Re: Pipelining might be my favorite programming language feature

#345

Earlier quoted context omitted.

Pipelining replaces several consecutive loops with a single loop, doing more complex processing.

Pipelining doesn’t do anything with iterating. It’s entirely about linking nested functions. What you’re looking at is loops defined inside lambda functions. Pipelining makes it much easier to use anonymous functions and lambdas. But it doesn’t magically solve the problem of complex loops.

It kind of is something related to loops if the language supports object iterator interfaces (they bridge OOP to classical constructs like for/foreach). Or maybe even generators.

It does not solve it magically, but it does give the programmer options to coalesce different paradigms into one single working implementation.

Re: Pipelining might be my favorite programming language feature

#346
post #267

The author keeps calling it "pipelining", but I think the right term is "method chaining". Compare with a simple pipeline in bash: grep needle Each of those components executes in parallel, with the intermediate results streaming between them. You get a similar effect with coroutines. Compare Ruby: data = File.readlines("haystack.txt") .map(&:strip) .grep(/needle/) .map { |i| i.gsub('foo', 'bar') } .map { |i| File.re…

I don’t find your “seasoned developer” version ugly at all. It just looks more mature and relaxed. It also has the benefits that you can actually do error handling and have space to add comments. Maybe people don’t like it because of the repetition of “data =“ but in fact you could use descriptive new variable names making the code even more readable (auto documenting). I’ve always felt method chaining to look “cramp…

> Maybe people don’t like it because of the repetition of “data =“

Eh, at first glance it looks "amateurish" due to all the repeated stuff. Chaining explicitly eliminates redundant operations - a more minimal representation of data flow - so it looks more "professional". But I also know better than to act on that impulse. ;)

That said, it really depends on the language at play. Some will compile all the repetition of `data =` away such that the variable's memory isn't re-written until after the last operation in that list; it'll hang out in a register or on the stack somewhere. Others will run the code exactly as written, bouncing data between the heap, stack, and registers - inefficiencies and all.

IMO, a comment like "We wind up debugging this a lot, please keep this syntax" would go a long way to help the next engineer. Assuming that the actual processing dwarfs the overhead present in this section, it would be even better to add discrete exception handling and post-conditions to make it more robust.

Re: Pipelining might be my favorite programming language feature

#347
Nice post and I very much agree!

In fact I tried to make some similar points in my CMU "SQL or Death" Seminar Series talk on PRQL (https://db.cs.cmu.edu/events/sql-death-prql-pipelined-relati...) in that I would love to see PRQL (or something like it) become a universal DSL for data pipelines. Ideally this wouldn't even have to go through some query engine and could just do some (byte)codegen for your target language.

P.S. Since you mentioned the Google Pipe Syntax HYTRADBOI 2025 talk, I just want to throw out that I also have a 10 min version for the impatient: https://www.hytradboi.com/2025/deafce13-67ac-40fd-ac4b-175d5... That's just a PRQL overview though. The Universal Data Pipeline DSL ideas and comparison to LINQ, F#, ... are only in the CMU talk. I also go a bit into imperative vs declarative and point out that since "pipelining" is just function composition it should really be "functional" rather than imperative or declarative (which also came up in this thread).

Re: Pipelining might be my favorite programming language feature

#348

Earlier quoted context omitted.

Records and Tuples weren't stopped because of tc39, but rather the engine developers. Read the notes.

Aren't the engine devs all part of the TC39 committee? I know they stopped SIMD in JS because they were mire interested in shipping WASM, and then adding SIMD to it.

I would say representatives of the engine teams are involved. However not involved enough clearly, because it should have been withdrawn waaay before now due to this issue.

Re: Pipelining might be my favorite programming language feature

#349

The author keeps calling it "pipelining", but I think the right term is "method chaining". Compare with a simple pipeline in bash: grep needle Each of those components executes in parallel, with the intermediate results streaming between them. You get a similar effect with coroutines. Compare Ruby: data = File.readlines("haystack.txt") .map(&:strip) .grep(/needle/) .map { |i| i.gsub('foo', 'bar') } .map { |i| File.re…

I think the best term is "function composition", but with a particular syntax so pipelining seems alright. Method chaining is a common case, where some base object is repeatedly modified by some action and then the object reference is returned by the "method", thus allowing the "chaining", but what if you're not dealing with objects and methods? The pipelined composition pattern is more general than method chaining imho.

You make an interesting point about debugging which is something I have also encountered in practice. There is an interesting tension here which I am unsure about how to best resolve.

In PRQL we use the pipelining approach by using the output of the last step as the implicit last argument of the next step. In M Lang (MS Power BI/Power Query), which is quite similar in many ways, they use second approach in that each step has to be named. This is very useful for debugging as you point out but also a lot more verbose and can be tedious. I like both but prefer the ergonomics of PRQL for interactive work.

Update: Actually, PRQL has a decent answer to this. Say you have a query like:

    from invoices
    filter total > 1_000
    derive invoice_age = @2025-04-23 - invoice_date
    filter invoice_age > 3months
and you want to figure out why the result set is empty. You can pipe the results into an intermediate reference like so:

    from invoices
    filter total > 1_000
    into tmp
    
    from tmp
    derive invoice_age = @2025-04-23 - invoice_date
    filter invoice_age > 3months
So, good ergonomics on the happy path and a simple enough workaround when you need it. You can try these out in the PRQL Playground btw: https://prql-lang.org/playground/

Re: Pipelining might be my favorite programming language feature

#350

Earlier quoted context omitted.

Pipelining doesn’t do anything with iterating. It’s entirely about linking nested functions. What you’re looking at is loops defined inside lambda functions. Pipelining makes it much easier to use anonymous functions and lambdas. But it doesn’t magically solve the problem of complex loops.

It kind of is something related to loops if the language supports object iterator interfaces (they bridge OOP to classical constructs like for/foreach). Or maybe even generators. It does not solve it magically, but it does give the programmer options to coalesce different paradigms into one single working implementation.

Sure, but again, you’re talking about using either lambda functions or object methods.

The crux of the “magic” with pipelining is chaining functions. How those functions are composed and what they do is a separate topic entirely.

Post reply on HN