The thing I want most is a more sane and more memorable way to compose non-element-wise operations. There are so many different ways to build views and multiply arrays that I can’t remember them and never know which to use, and have to relearn them every time I use numpy… broadcasting, padding, repeating, slicing, stacking, transposing, outers, inners, dots of all sorts, and half the stack overflow answers lead to th…
NumPy 2.0
31–40 of 78 posts
Re: NumPy 2.0
#32The thing I want most is a more sane and more memorable way to compose non-element-wise operations. There are so many different ways to build views and multiply arrays that I can’t remember them and never know which to use, and have to relearn them every time I use numpy… broadcasting, padding, repeating, slicing, stacking, transposing, outers, inners, dots of all sorts, and half the stack overflow answers lead to th…
https://einops.rocks/#why-use-einops-notation
Re: NumPy 2.0
#33Earlier quoted context omitted.
The most important changes are deprecations of certain public APIs: https://numpy.org/devdocs/release/2.0.0-notes.html#deprecati... One new interesting feature, though, is the support for string routines: https://numpy.org/devdocs/reference/routines.strings.html#mo...
> One new interesting feature, though, is the support for string routines Sounds almost like they're building a language inside a language.
It's already very much a DSL, and has been for the decade-ish that I've used it.
They're not building a language. They're carefully adding a newly-in-demand feature to a mature, already-built language.
Re: NumPy 2.0
#34The thing I want most is a more sane and more memorable way to compose non-element-wise operations. There are so many different ways to build views and multiply arrays that I can’t remember them and never know which to use, and have to relearn them every time I use numpy… broadcasting, padding, repeating, slicing, stacking, transposing, outers, inners, dots of all sorts, and half the stack overflow answers lead to th…
When I started out I was basically stumbling around for code that worked. Things got a lot easier for me once I sat down and actually understood broadcasting. The rules are: 1) scalars always broadcast, 2) if one vector has fewer dimensions, left pad it with 1s and 3) starting from the right, check dimension compatibility, where compatibility means the dimensions are equal or one of them is 1. Example: np.ones((2,3,1…
The broadcasting doc is surprisingly readable and easy to follow. And the rules are surprisingly simple. The diagrams and examples are excellent. https://numpy.org/doc/stable/user/basics.broadcasting.html
After taking the time to work through that doc and ponder some real-world examples, I went from being very confused by broadcasting to employing intermediate broadcasting techniques in a matter of weeks. Writing out your array dimensions in the same style of their examples (either in a text file or on a notepad) is the key technique IMO:
Image (3d array): 256 x 256 x 3
Scale (1d array): 3
Result (3d array): 256 x 256 x 3
Re: NumPy 2.0
#35Any notable highlights for a consumer of Numpy who rarely interfaces directly with it? Most of my work is pandas+scipy, with occasionally dropping into the specific numpy algorithm when required. I am much more of an "upgrade when there is a X.1" release kind of guy, so my hat off to those who will bravely be testing the version on my behalf.
As a more or less daily user, I was surprised at how not-breaking the 2.0 changes will be for 90% of Numpy users. Unless their dependencies/environments break, I expect that casual users won't even notice the upgrade.
Even the new string dtype I expect would go unnoticed by half of users or more, because they won't be using it (because Numpy historically only had fixed-length strings and generally poor support for them) and so won't even think to try it. Pandas meanwhile has had a proper string dtype for a while, so anyone interested in doing serious work on strings in data frames / arrays would presumably be using Pandas anyway.
Most of the breaking changes are in long-deprecated oddball functions that I literally have never seen used in the wild, and in the internal parts that will be a headache for library developers.
The only change that a casual user might actually notice is the change in repr(np.float64(3.0)), from "3.0" to "np.float64(3.0)".
Re: NumPy 2.0
#36it feels like the first major release in 18 years which introduces lots of breaking changes should just be a fork rather than a version. let me do `pip install numpy2` and not have to worry about whether or not some other library in my project requires numpy<2.
From a consumer (developer consumer) point of view, I hear you. From a project point of view, there are some pretty strong contra-indicators in the last 20 years of language development that make this plan suspect, or at least pretty scary — both Perl and Python had extremely rocky transitions around major versions; Perl’s ultimately failing and Python’s ultimately taking like 10 years. At least. I think the last tim…
Could you elaborate further on this? Is it not the case that they'll still need to support numpy 1.x for at least the near-term future? My understanding was the parent comment was specifically talking about the technical difficulty of multiple versions for the same Python package, not the social problem of project and community management across breaking changes.
Re: NumPy 2.0
#37The thing I want most is a more sane and more memorable way to compose non-element-wise operations. There are so many different ways to build views and multiply arrays that I can’t remember them and never know which to use, and have to relearn them every time I use numpy… broadcasting, padding, repeating, slicing, stacking, transposing, outers, inners, dots of all sorts, and half the stack overflow answers lead to th…
When I started out I was basically stumbling around for code that worked. Things got a lot easier for me once I sat down and actually understood broadcasting. The rules are: 1) scalars always broadcast, 2) if one vector has fewer dimensions, left pad it with 1s and 3) starting from the right, check dimension compatibility, where compatibility means the dimensions are equal or one of them is 1. Example: np.ones((2,3,1…
Explicit internal broadcasting (by adding an axis with an explicit `indefinite` size instead of `1`) would be so much simpler to reason about.
Unfortunately there is far too much existing code and python is not type-safe.
Re: NumPy 2.0
#38Any notable highlights for a consumer of Numpy who rarely interfaces directly with it? Most of my work is pandas+scipy, with occasionally dropping into the specific numpy algorithm when required. I am much more of an "upgrade when there is a X.1" release kind of guy, so my hat off to those who will bravely be testing the version on my behalf.
As a more or less daily user, I was surprised at how not-breaking the 2.0 changes will be for 90% of Numpy users. Unless their dependencies/environments break, I expect that casual users won't even notice the upgrade. Even the new string dtype I expect would go unnoticed by half of users or more, because they won't be using it (because Numpy historically only had fixed-length strings and generally poor support for th…
I suspect the C ABI break to be the biggest issue, though maybe fewer packages than I imagine compile against the numpy C ABI...
Re: NumPy 2.0
#39Earlier quoted context omitted.
No. Native python ops in string suck in performance. String support is absolutely interesting and will enable abstractions for many NLP and LLM use cases without writing native C extensions.
> Native python ops in string suck in performance. That’s not true? Python string implementation is very optimized, probably have similar performance to C.
It is absolutely true that there is massive amounts of room for performance improvements for Python strings and that performance is generally subpar due to implementation decisions/restrictions.
Strings are immutable, so no efficient truncation, concatenation, or modifications of any time, you're always reallocating.
There's no native support for a view of string, so operations like iteration over windows or ranges have to allocate or throw away all the string abstractions.
By nature of how the interpreter stores objects, Strings are always going to have an extra level of indirection compared to what you can do with a language like C.
Python strings have multiple potential underlying representations, and thus have some overhead for managing and dealing with those multiple representations without exposing those details to user code
Re: NumPy 2.0
#40I'm starting to see some packages break due to not pinning 1.x in their dependencies. `pip install numpy==1.*` is a quick and hacky way to work around those issues until the ecosystem catches up.