Live data from Hacker News

The different uses of Python type hints

lukeplant.me.uk

31–40 of 70 posts

Re: The different uses of Python type hints

#31
post #7

The issue that I have with Python type hints is they they don't go nearly far enough in describing the data being manipulated. Specifically, I'm thinking of stuff like the dimensionality and cardinality of Numpy arrays or Pandas frames. Usually that's the stuff where I have most questions when I look at Python code and the type system as it's being used now offers no help there.

I like typing that with strings like '(batch,r,a,s,channel,t)'. The tooling doesn't do anything special with it, but it makes the code understandable at a glance. Adopting libraries like einops and core routines like einsum in lieu of equivalent alternatives encourages the propagation of names (rather than rolling axes or whatever) anywhere it matters. Having a coding convention about the standard order of axes helps a bit as people get more familiar with that aspect of the codebase too, only deviating where necessary.

Re: The different uses of Python type hints

#32

A few other examples for the sections given: Runtime behaviour determination: the stdlib [dataclasses]( https://docs.python.org/3/library/dataclasses.html#module-da... ) Dataclasses is notable because it's the only example (I'm aware of) of type hints effecting runtime behavior as part of the stdlib. Compiler instructions: mypyc was (one of?) the first to do this, but Cython actually supports this natively now, and i…

> Dataclasses is notable because it's the only example (I'm aware of) of type hints effecting runtime behavior as part of the stdlib. FWIW, `typing.NamedTuple` did this in Python 3.5, three years before dataclasses was introduced in 3.7. class Foo(typing.NamedTuple): a: int b: str f = Foo(a=1, b="hello") print(f.b) # "hello"

Variable annotations were only added in Python 3.6. Defining a typed namedtuple in Python 3.5 looked like this:

    Foo = typing.NamedTuple('Foo', [('a', int), ('b', str)])

Re: The different uses of Python type hints

#33
post #31
post #7

The issue that I have with Python type hints is they they don't go nearly far enough in describing the data being manipulated. Specifically, I'm thinking of stuff like the dimensionality and cardinality of Numpy arrays or Pandas frames. Usually that's the stuff where I have most questions when I look at Python code and the type system as it's being used now offers no help there.

I like typing that with strings like '(batch,r,a,s,channel,t)'. The tooling doesn't do anything special with it, but it makes the code understandable at a glance. Adopting libraries like einops and core routines like einsum in lieu of equivalent alternatives encourages the propagation of names (rather than rolling axes or whatever) anywhere it matters. Having a coding convention about the standard order of axes helps…

I like this comment because so far I don't know what you're talking about. That's the hallmark of something in this domain worth looking up but I figured you might be willing to share more on the matter. :)

Re: The different uses of Python type hints

#34
post #17
post #7

The issue that I have with Python type hints is they they don't go nearly far enough in describing the data being manipulated. Specifically, I'm thinking of stuff like the dimensionality and cardinality of Numpy arrays or Pandas frames. Usually that's the stuff where I have most questions when I look at Python code and the type system as it's being used now offers no help there.

I'm not sure how a python annotation/type system could possibly do that? If numpy/pandas had different types for different cardinalities it would work today. You just need those libraries to embrace it really, then you could theoretically have type constructors that provide well-typed NxM matrix types or whatever, allowing you to enforce that [[1,2],[3,4]] is an instance of matrix_t(2, 2). I don't see how python coul…

PEP 646 Variadic generics, https://peps.python.org/pep-0646/, was made for this specific use case but mypy is still working on implementing it. And even with it, it's expected several more peps are needed to make operations on variadic types powerful enough to handle common array operations. numpy/tensorflow/etc do broadcasting a lot and that probably would need a type level operator Broadcast just to encode that. I also expect the type definitions for numpy will go fairly complex similar to template heavy C++ code after they add shape types.

Re: The different uses of Python type hints

#35
post #31
post #7

The issue that I have with Python type hints is they they don't go nearly far enough in describing the data being manipulated. Specifically, I'm thinking of stuff like the dimensionality and cardinality of Numpy arrays or Pandas frames. Usually that's the stuff where I have most questions when I look at Python code and the type system as it's being used now offers no help there.

I like typing that with strings like '(batch,r,a,s,channel,t)'. The tooling doesn't do anything special with it, but it makes the code understandable at a glance. Adopting libraries like einops and core routines like einsum in lieu of equivalent alternatives encourages the propagation of names (rather than rolling axes or whatever) anywhere it matters. Having a coding convention about the standard order of axes helps…

Why would you do this instead of just a comment? I feel like type hints only have value if they can be used by the tooling.

Re: The different uses of Python type hints

#36
post #33
post #31

Earlier quoted context omitted.

I like typing that with strings like '(batch,r,a,s,channel,t)'. The tooling doesn't do anything special with it, but it makes the code understandable at a glance. Adopting libraries like einops and core routines like einsum in lieu of equivalent alternatives encourages the propagation of names (rather than rolling axes or whatever) anywhere it matters. Having a coding convention about the standard order of axes helps…

I like this comment because so far I don't know what you're talking about. That's the hallmark of something in this domain worth looking up but I figured you might be willing to share more on the matter. :)

The premise is that Python's type hints don't provide enough information about things like numpy arrays. Fixing that correctly is hard because things that matter in that sort of code include facts which could hypothetically be encoded in a type system:

1. What's the datatype of the array elements

2. Does this array alias other memory

3. Is the access pattern I want to do contiguous in memory

4. If you track the provenance of an array, does it include something like a "width" dimension and a "height" dimension

5. How many dimensions are there

6. What's a good semantic description (type) for each dimension

7. As an exact integer (or modulo some power of 2 or whatever), how big is each dimension

And on and on and on. An honest-to-goodness type hint capturing that sort of crap in a way that's statically analyzable is a nightmare, and it wouldn't be totally trivial to even write the code to make a type hint like that reasonable to read and write. Even if you could, it'd probably generate a lot of noise that for any particular use of an array would distract you from the aspects you care about.

A nice hybrid solution IMO is found in that Python allows arbitrary objects to be used as type hints, and a string description of the aspects you're using/providing on a particular array works as decent documentation for other developers. For a few examples:

1. The array should describe a typical 24-bit 3-channel image. You might use a type hint like 'u8:(w,h,3)' to indicate that it's a 0-255 integer field rather than a 0-1 float field, which dimensions have width/height/channels, and that it's a 3-channel image. It'd probably be good to also label those channels with a convention like 'u8:(w,h,(rgb))', like 'u8:(w,h,3):rgb', with hungarian typing, or something (no particular recommendations on my end since I'm not usually working with heterogeneous data like that, but choosing the wrong encoding or even the wrong coordinate space for RGB or whatever is a big deal, so you'd probably want to represent that somehow).

2. You have a function signature with multiple inputs, and the computation is mostly arbitrary, but it's important some dimensions align. Then label them the same. Something like matmul(left: '(n, d)', right: '(d, k)') -> '(n, k)'.

3. You're doing some ML thing on some time-series medical data, and it's common t' have giant dense tensors floating around. Label semantically what all the dimensions are with a type like `(batch, r, a, s, channel, t)', or using longer names as appropriate depending on your audience and the background knowledge you can assume.

Libraries like einops and functions like `np.einsum` take that a step further and require stringified descriptions of the operation you're trying to do. They can have a learning curve, but the crux of the idea is that instead of writing garbage like `arr[3,6,-4:,np.newaxis,...].T.reshape(4 n, -1)` or God-forbid some sort of roll/transpose logic, you have a higher-level description.

A couple examples with einsum:

1. The dot product of v and w is `np.dot(v, w)` or `np.sum(v w) # imagine there's an asterisk; HN's parser is smarter than me`, and it's also `np.einsum('d,d', v, w)`. Arguably einsum is a bit of syntactic noise for such a simple example, but if v and w have different shapes than you think then the simpler solutions will silently produce garbage (e.g., the first option will do matrix multiplications sometimes, and the second is arguably closer to correct most of the time, but if you think you're operating on 1D objects and actually do want a channeled operation like matrix multiplication when the input isn't 1D then the sum of products is wrong and not captured in the type system), but einsum will just barf if the stated dimensions don't match your expectations. Moreover, with optimize=True it'll actually fall back to whichever of the simpler solutions is fastest.

2. Imagine you have a matrix A of shape (n, n) and a matrix X of shape (n, d) and want to compute something like A @ v @ A.T for each column v of X. You can write it via standard numpy operators, but it looks like garbage and kind of hides what's actually happening. The einsum solution is just `np.einsum('vw,wd,nv->nd', A, X, A)`. You're contracting over `v` and `w` and left with `n` and `d`. It's not perfect since you just get single-letter names to work with, but it's a hell of a lot better than equivalent options, and much easier to make suitably fast (just pass optimize=True).

And then einops is even better because roll/transpose logic is incredibly fiddly and prone to off-by-one errors in your choice of dimension or needing to deeply understand how the function works to not make footgun-style mistakes. An API like `swapaxes(arr, 'batch', 'time')` is 10x easier to use than `swapaxes(arr, 0, 5)` -- like, imagine somebody adding an extra dimension in a world where positions are absolutely referenced and where if you get it wrong the program will still run and produce interesting-looking garbage because the definitions of `np.dot` and everything else in the library depend on the shape of the inputs.

Re: The different uses of Python type hints

#37
post #35
post #31

Earlier quoted context omitted.

I like typing that with strings like '(batch,r,a,s,channel,t)'. The tooling doesn't do anything special with it, but it makes the code understandable at a glance. Adopting libraries like einops and core routines like einsum in lieu of equivalent alternatives encourages the propagation of names (rather than rolling axes or whatever) anywhere it matters. Having a coding convention about the standard order of axes helps…

Why would you do this instead of just a comment? I feel like type hints only have value if they can be used by the tooling.

The comment has to go somewhere, and IME it makes the common path easy and less common paths not too hard. In particular, to use those functions correctly and understand what they're doing, you often really do just need their name and such an augmented type signature. Having all that information in one place rather than having to extract it out of a docstring (a docstring which might not be shown by default in your editor without additional keystrokes or mouse movements and scrolling), and being able to immediately glance to pieces that don't stick around in short-term memory is nice.

A comment would be fine too, especially if it's right next to the type signature, but to do that you'd need to add extra newlines, and the comment would be in roughly the same spot as the type hint, so I don't know that you gain much. Mypy doesn't really like strings used that way, but mypy isn't a great tool anyway, so c'est la vie?

If somebody just wanted to throw that in a docstring I wouldn't complain though. It's definitely more important that the information exist than that it be in a particular place.

Re: The different uses of Python type hints

#38
If you accept my -in advance- apology, why this type thing seem ugly -distraction,complex, hard to act, ...- to me ? I'm not against it , as in TS (non enforced type checking ), it is a lovely addition to python, but I'm really struggling to read, write and .. this syntax . Not sure if it is python's nature, but as most of us, C, C++, ...., TS this is a journey, evolution , but the root C is some kind of cult we attached to. Is it preventing me to love this thing ( thanks for effort )

Do not know?

Re: The different uses of Python type hints

#39

If you accept my -in advance- apology, why this type thing seem ugly -distraction,complex, hard to act, ...- to me ? I'm not against it , as in TS (non enforced type checking ), it is a lovely addition to python, but I'm really struggling to read, write and .. this syntax . Not sure if it is python's nature, but as most of us, C, C++, ...., TS this is a journey, evolution , but the root C is some kind of cult we atta…

Additions to language always confront syntax expectations. Some people can see through syntax to semantic intent, alas I am not one and the utility of a syntactic form goes (to me at least) beyond expressiveness to comprehension: if your syntax confuses then how can anyone comprehend?

Haskell dies in syntax.

Re: The different uses of Python type hints

#40
post #28
post #3

This sort of thing is why I gave up Python. I could see having strong typing. Or optional strong typing. But unchecked type hints are just silly. The way everybody else seems to be going is strong typing at function interfaces, with automatic inference of as much else as can be done easily. C++ (since "auto"), Go, Rust, etc.

With python you can have your cake and it eat it, though. Mock up something fast, no type hints. Now, take that POC and make it production ready, by using mypy and pydantic.

But why does it need additional dependencies just to get all language features?
Post reply on HN