Show HN: Lfi – a lazy functional sync, async, and concurrent iteration library

Thank you for taking the time to explain! While the details seem to be well over my head, I think I‘ll stick to a worker then, as the specific thing I was talking about was taxonomy faceting of a pretty large dataset. Thank you again!

Yup, I added it in response to the feedback :)

I'm not sure I know the use case for all of this logical complexity. Is there a specific use-case you had in mind?

Split the array of elements to process. Set up a promise.all against a function that handles sync/async function calls against each chunk and accumulates an array of results. This isn't the end of the world for any of the practical use cases I can think of. The efficiency loss and memory consumption is miniscule even across hundreds of thousands of records. This is as concurrent as anything else - the event loop is still a mandatory part of the processing loop. You can even defer every 1000 records to the event loop. Javascript is syncronous so there is no concurrency without threading. I can't think of an IO or non-IO function that would create the issues described of not yielding. We used to have chunks of 1000000+ records that would freeze the event loop. We solved this with structured yields to the event loop to allow other processes to continue without dropping TCP packets etc.

If the results are objects, you're not even bloating memory too much. 800KB for 100,000 records seems fine for any practical use case.

I like the idea of an async generator to prevent over-pre-calculating the result set before it's needed but I couldn't justify pivoting to a set of new functions called pipe, map, etc that require a few too many logical leaps for your typical ECMAScript developers to understand.

What's the win here? It must be for some abuse of Javascript that should be ported to WASM. Or some other use-case that I've somehow not seen yet.

I'm somewhat familiar with fp-ts, and I think lfi is pretty different.

fp-ts seems more focused on "pure" functional programming in the style of languages like Haskell. It's much more opinionated on how you should write your code, including the data structures you should use. Plus, it's not really concerned with concurrency in the way that lfi is.

I think lfi is a lot less invasive/opinionated on how you write your code.

To clarify, the point of it isn't _just_ performance.

It's a combination for writing the iteration code in a functional style, which some people like/prefer, while retaining certain performance characteristics that you'd get from writing more imperative code.

For example, the "naive" approach to functional programming + concurrency results in some unintentional bottlenecks (see the concurrent iteration example on the home page).