itertools is a gem and has been since the 2.7 days. Glad to see people waking up to its powerful abstractions.
itertools (iterators) and collections (data structures) are both underrated modules in stdlib.
And are both written by Raymond Hettinger
More Itertools
21–30 of 51 posts
Re: More Itertools
#22This looks great. Usally, I'd cast my arrays into a pandas DF and then use the equivalent dataframe operations. To me, pandas and numpy might as well be part of the python stdlib. How should I reason about the tradeoff of using something like this vs pandas/numpy ? Esp. with Numpy 2.0 supporting the string dtype.
> Usally, I'd cast my arrays into a pandas DF
I promise I mean no offense by this but this is so comically absurd. Like you know it's not a cast right? Ie that you're constructing pandas dataframes.
> How should I reason about the tradeoff of using something like this vs pandas/numpy ?
For small sizes, operations on native types will be faster than the construction of complex objects.
Re: More Itertools
#23it has always annoyed me that flatten isn't already part of itertools
ok itertools has chain.from_iterable but that name is hard to remember
Yes, I think it might have been a slight design mistake to make the variadic version the default. I've only very rarely used it, whereas I use chain.from_iterable a lot.
Re: More Itertools
#24This looks great. Usally, I'd cast my arrays into a pandas DF and then use the equivalent dataframe operations. To me, pandas and numpy might as well be part of the python stdlib. How should I reason about the tradeoff of using something like this vs pandas/numpy ? Esp. with Numpy 2.0 supporting the string dtype.
> Usally, I'd cast my arrays into a pandas DF I promise I mean no offense by this but this is so comically absurd. Like you know it's not a cast right? Ie that you're constructing pandas dataframes. > How should I reason about the tradeoff of using something like this vs pandas/numpy ? For small sizes, operations on native types will be faster than the construction of complex objects.
Also, my grief with DF is they aren't typed (typing module) by column. Maybe that's changed though? It's been a while.
The only way to understand what's going on with DF code is to step it in a debugger. I know they can be much faster, but man you pay a maintainability price!
Re: More Itertools
#25I've implemented the "chunked" iterator a million times. Glad to see I can just import this next time.
[deleted]
Re: More Itertools
#26Shout out to JavaScript massively delaying https://github.com/tc39/proposal-async-iterator-helpers in the 23rd hour. The proposal seemed very close to getting shipped alongside https://github.com/tc39/proposal-iterator-helpers while basically accepting many of the constraints of current async iteration (one at a time consumption). But the folks really accepted that concurrency needs had evolved, decided to hold back…
> But the folks really accepted that concurrency needs had evolved, decided to hold back & keep iterating & churning for better
I'm not sure if it was this proposal or another one in a similar space, but I've recently heard about several async improvements that were woefully under-spec'd, and would likely have caused much more harm than good due to all the edge cases that were missed.
Re: More Itertools
#27Earlier quoted context omitted.
> Usally, I'd cast my arrays into a pandas DF I promise I mean no offense by this but this is so comically absurd. Like you know it's not a cast right? Ie that you're constructing pandas dataframes. > How should I reason about the tradeoff of using something like this vs pandas/numpy ? For small sizes, operations on native types will be faster than the construction of complex objects.
Also, my grief with DF is they aren't typed (typing module) by column. Maybe that's changed though? It's been a while. The only way to understand what's going on with DF code is to step it in a debugger. I know they can be much faster, but man you pay a maintainability price!
They effectively are since each column is a series, which is typed.
Re: More Itertools
#28it has always annoyed me that flatten isn't already part of itertools
Is np.flatten not a workable option in some cases?
Re: More Itertools
#29Earlier quoted context omitted.
> Usally, I'd cast my arrays into a pandas DF I promise I mean no offense by this but this is so comically absurd. Like you know it's not a cast right? Ie that you're constructing pandas dataframes. > How should I reason about the tradeoff of using something like this vs pandas/numpy ? For small sizes, operations on native types will be faster than the construction of complex objects.
Also, my grief with DF is they aren't typed (typing module) by column. Maybe that's changed though? It's been a while. The only way to understand what's going on with DF code is to step it in a debugger. I know they can be much faster, but man you pay a maintainability price!
This is incorrect: each column in a pandas DFs can have a separate type (what you're asking for is compatibility with Python's type-hinting on a per-column basis, though, which is different), and you can debug the code without needing a debugger: I use pandas regularly and I've never needed to use a debugger on pandas.
(Sure, it's easy to write obfuscated pandas, and it sometimes has version-specific bugs or deprecations which need to be hacked around in a way that compromises readability, and sometimes the API has active changes/namings that are non-trivial. But that's miles from "only way to understand is with a debugger". If you want to claim otherwise, post a counterexample on SO (or Codidact) and post the link here.)
Re: More Itertools
#30it has always annoyed me that flatten isn't already part of itertools
Is np.flatten not a workable option in some cases?
Maybe in some cases, but the performance characteristics are way different. The functions in `more_itertools` return lazy generators, but it looks like `np.flatten` materializes the results in an ndarray.