21–30 of 51 posts
Call me an idiot, but nowhere on the home page does it say it's for JavaScript. The code samples looked like JS, but I don't know every language, so I wasn't sure. It's not until I got into the docs and saw "npm" was a sure we were talking about JS...
has it been added since?
great library! I just tried it out and compared it with iter-tools I like Lfi better, great documentation! Here is the example I use to compare: https://stackblitz.com/edit/stackblitz-starters-ia9ujg6m?fil...
Still, I love seeing innovation in this field as much as anyone! And it's true that I've been dragging my feet on building concurrency support for iter-tools...
That said, iter-tools still has some killer APIs like peekerate that I would find it quite hard to live without, and for iter-tools 8 I have some major tricks planned like making the sync/async divide go away completely.
Just so I understand: Could this library be used for iterating large collections without blocking the main thread? Basically as a replacement for a worker with synchronous iteration?
In other words, put on the stack at the same time but not a different thread.
Do correct me if I misunderstood though.
I don't understand (but am open to be persuaded), isn't iteration in js now trivially easy and actually quite powerful? Especially since we can now put 'await' everywhere? What are the killer use cases for this lib?
Just so I understand: Could this library be used for iterating large collections without blocking the main thread? Basically as a replacement for a worker with synchronous iteration?
It will be very efficient when the async operations are few, and slower. It will not be very efficient when the async operations are many and fast.
That's because the await keyword itself blocks the main thread every time a function call is made. It has to because the `await` keyword is defined to return to the event loop and resume processing only during the next "tick" and after other queued ticks have run.
For a large collection the overhead on the event loop can be calculated as: COLLECTION_SIZE * FUNCTION_CALLS_PER_ELEMENT. Since function calls per element goes up as you wrap layers of helper functions, even with this library you would face a strong perverse incentive to avoid using (async) function calls or nested iterators (if at all possible) when handling large collections, especially with high desired throughput, especially in situations where you want the event loop to stay unclogged so that you can, for example, redraw the UI.
Just so I understand: Could this library be used for iterating large collections without blocking the main thread? Basically as a replacement for a worker with synchronous iteration?
Yes, but I'd add a caveat. It will be very efficient when the async operations are few, and slower. It will not be very efficient when the async operations are many and fast. That's because the await keyword itself blocks the main thread every time a function call is made. It has to because the `await` keyword is defined to return to the event loop and resume processing only during the next "tick" and after other que…
Just so I understand: Could this library be used for iterating large collections without blocking the main thread? Basically as a replacement for a worker with synchronous iteration?
Yes, but I'd add a caveat. It will be very efficient when the async operations are few, and slower. It will not be very efficient when the async operations are many and fast. That's because the await keyword itself blocks the main thread every time a function call is made. It has to because the `await` keyword is defined to return to the event loop and resume processing only during the next "tick" and after other que…
I don't understand (but am open to be persuaded), isn't iteration in js now trivially easy and actually quite powerful? Especially since we can now put 'await' everywhere? What are the killer use cases for this lib?
for the record sprinkling await everywhere is not something you should take lightly. await is still probably the single most high-level high-overhead construct in JS, and putting it inside tight loops is a recipe for perf disaster
Maybe an in memory cache hit---that you explicitly code---causes a sync resolve to happen, woohoo!
But if it goes to IO and you are waiting for the event loop to schedule you back in you could be in for a shock even if the IO op is tiny. Especially doing it alot.
Only matters probably on a high CPU usage server which hopefully you avoid by scaling up on cpu and using all the cores.
On the other hand... you are being cooperative :)
But tight loop IO is an antipattern ... as is doing a lot---millions---of IO ops from node (per request or job) that you need to worry about this (use a different language for your DB server!).