Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Writing and linting Python at scale

engineering.fb.com

Post reply on HN
Print view View original

81–90 of 160 posts

Re: Writing and linting Python at scale

#81

Post by timeon »

corethree wrote: post #21
lagt_t wrote: post #5

But but but HN told me that python is only good for small scale and prototypes, the engineers at meta are wrong

Python not only has types but it's type system is superior to typeScript. Get this python has sum types and exhaustive pattern matching exactly like rust or haskell. The only problem with python are the libraries are sometimes written with tricks that make static typing ineffective. Other than that it is really really good at scale. Better API then typescript imo which is really it's main competitor. edit: (rate limi…

> sum types

Even PHP have those now. TypeScript is lagging here. Hopefully, they will include it too one day.

Re: Writing and linting Python at scale

#82

Post by Timon3 »

chlorion wrote: post #79
Timon3 wrote: post #49

Earlier quoted context omitted.

Your tuple example only works if both tuples have two elements. I specifically mentioned arbitrary fixed-size tuples (as in, tuples with an arbitrary non-variable length). Your generic type transformation example also doesn't come close to what Typescript does. The resulting dict will not have known keys based on the keys of the input dict. In Typescript I can write a function that takes an object with known keys, an…

The tuple thing requires variadic generics from my understanding. I don't thing variadic generics support is supported in most statically typed languages. The only one I can think of right now that supports this is C++.

Typescript supports it too (quick example[0]) :) and Python actually as well, but currently you can't unpack two TypeVarTuples in the same type expression: https://peps.python.org/pep-0646/

[0] https://www.typescriptlang.org/play?#code/C4TwDgpgBAglC8UDaA...

Re: Writing and linting Python at scale

#83

Post by dazzlefruit »

Timon3 wrote: post #78
dazzlefruit wrote: post #74

Earlier quoted context omitted.

> But I have a use case for exactly this feature. Why should the language limit me? Why should I implement x functions that take different tuple lengths, with me having to choose the correct one for each use case, when I could write one function that does all? If the PSF ran a poll for the most-wanted type checking features, I don't think this would come close to first. This sounds very niche. The people working on t…

> If the PSF ran a poll for the most-wanted type checking features, I don't think this would come close to first. This sounds very niche. The people working on typing seemed very busy in the last few versions. Sure, but while it's not possible to type basic functions like ones that concatenate tuples, I can say that Pythons typing system is not superior to Typescripts. > Returning errors as values isn't really how yo…

> Sure, but while it's not possible to type basic functions like ones that concatenate tuples, I can say that Pythons typing system is not superior to Typescripts.

That's not really fair. The uses that make sense when considering Python's convention ("Pythonic" code, nebulous but usually well-understood) are supported.

I think what confused me about these examples is that they imply multiple values that have completely different meanings, but all get processed as equals anyway. That was before you talked about refactoring old code though.

> Okay, so how am I supposed to handle non-exceptional errors? Because using exceptions for that kind of thing absolutely isn't good practice.

If an operation on a homogeneous collection can say "nope" for some values and process others, the values would be typed "T | None". If the data isn't really a collection but a structured mapping, in general, attributes would be made optional on a case-by-case basis. If all attributes happen to be optional but the mapping itself is non-optional, that sounds more like an accident of this specific case than something we should complicate a language over. If this happens over a whole codebase, I guess I feel for you. Maybe that's when it makes sense to give up a bit of static typing and treat these values a bit more like data and a bit less like separate arguments, no matter what kind of complicated typing the language can do.

> Legacy. Typescript allows me to do refactoring of things like these step by step and very easily. Python doesn't, because it's inflexible.

> Are we taking apart my code now or what?

Honestly yes, this example seems so unusual that it doesn't make sense to debate it without knowing concretely what's happening in your code that this needs to be supported.

I guess that this specific example would get easier. But I would hardly call one type system superior over that.

Edit: my final opinion on this is "this is something that's technically possible if you follow language and conventions to the letter, but with experience you see that it's a bad idea that won't fit well with the language and you should change the design to avoid it". It happens in all languages IMO.

Re: Writing and linting Python at scale

#84

Post by chlorion »

corethree wrote: post #33
seabrookmx wrote: post #22

Earlier quoted context omitted.

Yes but the if you want to do ahead-of-time type checking you need to run a tool like mypy, and none of those tools are as comprehensive or performant as Typescript. Also the ecosystem of libraries with type annotations is much smaller (Typescript has the first mover advantage, after all).

I literally said it supports exhaustive pattern matching which adds a level of safety and flexibility superior to that of type script. Many IDEs have real time type checking that highlights the errors so don't even have to run the external checker. Even if you don't use IDEs running the type checker is measured in seconds. Not far off from linters that most people will also use for TS. What you say about the librarie…

>I literally said it supports exhaustive pattern matching which adds a level of safety and flexibility superior to that of type script.

I don't have an opinion on which language is superior here, I've never written typescript or even javascript before, but I think saying that python's type system/checkers is superior because of this one feature is not correct.

I'm also skeptical of the claim that typescript doesn't support exhaustive branching. This very well could be true but it seems hard to believe.

I'm a big fan of exhaustive branching, but I think there are other things that are as or more important.

Re: Writing and linting Python at scale

#85

Post by dazzlefruit »

Timon3 wrote: post #80
dazzlefruit wrote: post #76

Earlier quoted context omitted.

What's missing is in which context the tuple's length is variable, and in which context it is fixed. You can have a tuple size fixed everywhere (because the callee sets it) or a tuple size fixed at each call (and propagated to the callee statically). "Arbitrary" doesn't really help because it could refer to the elements' values, to their types, or to the tuple's length. Also "arbitrary" and "variable-length" sound li…

> What's missing is in which context the tuple's length is variable, and in which context it is fixed. Simple example: a function has a parameter whose type is "variable-length tuple of int". You can pass any tuple in that is known to have 0..n elements, all of type int. What would you have me call that, other than the name I've seen used in discussions on this feature? > "Arbitrary" doesn't really help because it co…

> Simple example: a function has a parameter whose type is "variable-length tuple of int". You can pass any tuple in that is known to have 0..n elements, all of type int.

And n is fixed at the calling site, right? I wonder if something like "TypeVar, but for a list of type arguments" could solve your problem.

What's funny is that this is already kind of implemented in `typing.Concatenate`, but only for function parameters [1], not for type hint parameters.

Anyway, I would have written "a well-typed function that concatenates two arbitrary tuples whose size is statically known at the call site". Can't really remove "at the call site" or "statically known" without being ambiguous.

Edit: just found out about `TypeVarTuple`. So really we're only missing `concatenate`.

[1] https://docs.python.org/3/library/typing.html#typing.Concate...

Re: Writing and linting Python at scale

#86

Post by Timon3 »

dazzlefruit wrote: post #83
Timon3 wrote: post #78

Earlier quoted context omitted.

> If the PSF ran a poll for the most-wanted type checking features, I don't think this would come close to first. This sounds very niche. The people working on typing seemed very busy in the last few versions. Sure, but while it's not possible to type basic functions like ones that concatenate tuples, I can say that Pythons typing system is not superior to Typescripts. > Returning errors as values isn't really how yo…

> Sure, but while it's not possible to type basic functions like ones that concatenate tuples, I can say that Pythons typing system is not superior to Typescripts. That's not really fair. The uses that make sense when considering Python's convention ("Pythonic" code, nebulous but usually well-understood) are supported. I think what confused me about these examples is that they imply multiple values that have complete…

> That's not really fair. The uses that make sense when considering Python's convention ("Pythonic" code, nebulous but usually well-understood) are supported.

You're focussing on a very specific part of what I wrote. Just because my specific example consists of a Result type being transformed into a Tuple, it doesn't mean that the basic use case of "concatenate two tuple types" is so far out there.

> f an operation on a homogeneous collection can say "nope" for some values and process others, the values would be typed "T | None". If the data isn't really a collection but a structured mapping, in general, attributes would be made optional on a case-by-case basis. If all attributes happen to be optional but the mapping itself is non-optional, that sounds more like an accident of this specific case than something we should complicate a language over. If this happens over a whole codebase, I guess I feel for you. Maybe that's when it makes sense to give up a bit of static typing and treat these values a bit more like data and a bit less like separate arguments, no matter what kind of complicated typing the language can do.

But the static typing is extremely helpful, it prevents many kinds of errors. Not being able to use it for these kinds of things makes Python a worse language, no matter how you cut it (in the sense that it would be a better language if you could).

> Honestly yes, this example seems so unusual that it doesn't make sense to debate it without knowing concretely what's happening in your code that this needs to be supported.

Again, my specific example doesn't matter. The use case is "function takes in two tuple types and returns a concatenated version". That's something a type system should be able to handle.

> I guess that this specific example would get easier. But I would hardly call one type system superior over that.

On what metric besides expressiveness would you rate type systems?

Re: Writing and linting Python at scale

#87

Post by dragonwriter »

Timon3 wrote: post #34
corethree wrote: post #21

Earlier quoted context omitted.

Python not only has types but it's type system is superior to typeScript. Get this python has sum types and exhaustive pattern matching exactly like rust or haskell. The only problem with python are the libraries are sometimes written with tricks that make static typing ineffective. Other than that it is really really good at scale. Better API then typescript imo which is really it's main competitor. edit: (rate limi…

> Python not only has types but it's type system is superior to typeScript. I strongly disagree. It might be better at some things, but it's much worse at others. Many functions can't be accurately typed (try, for example, to make a well-typed function that concatenates two arbitrary fixed-size tuples), and as far as I know generic type transformations can't be implemented (random example "this type takes a Dict[str,…

Python can almost do this with Variadic Generics from Python 3.11; its missing an exception to the single-unpacking rule (which exists to prevent ambiguity) to allow unambiguous cases. Then you would have:

  from typing import TypeVarTuple

  Ts = TypeVarTuple("Ts”)
  Us = TypeVarTuple("Us”)

  def tconcat(
    t1: tuple[*Ts], 
    t2: tuple[*Us]
  ) -> tuple[*Ts, *Us]: ...

Re: Writing and linting Python at scale

#88

Post by Timon3 »

dazzlefruit wrote: post #85
Timon3 wrote: post #80

Earlier quoted context omitted.

> What's missing is in which context the tuple's length is variable, and in which context it is fixed. Simple example: a function has a parameter whose type is "variable-length tuple of int". You can pass any tuple in that is known to have 0..n elements, all of type int. What would you have me call that, other than the name I've seen used in discussions on this feature? > "Arbitrary" doesn't really help because it co…

> Simple example: a function has a parameter whose type is "variable-length tuple of int". You can pass any tuple in that is known to have 0..n elements, all of type int. And n is fixed at the calling site, right? I wonder if something like "TypeVar, but for a list of type arguments" could solve your problem. What's funny is that this is already kind of implemented in `typing.Concatenate`, but only for function param…

> And n is fixed at the calling site, right? I wonder if something like "TypeVar, but for a list of type arguments" could solve your problem.

Yep, and TypeVarTuple should - all the syntax etc. is in place, there is an Unpack operator for TypeVarTuples, allowing you to e.g. append or prepend individual types to a TypeVarTuple. But you can't unpack more than one TypeVarTuple in an expression, it's specifically disallowed - so I can't properly type my function.

Re: Writing and linting Python at scale

#89

Post by Timon3 »

dragonwriter wrote: post #87
Timon3 wrote: post #34

Earlier quoted context omitted.

> Python not only has types but it's type system is superior to typeScript. I strongly disagree. It might be better at some things, but it's much worse at others. Many functions can't be accurately typed (try, for example, to make a well-typed function that concatenates two arbitrary fixed-size tuples), and as far as I know generic type transformations can't be implemented (random example "this type takes a Dict[str,…

Python can almost do this with Variadic Generics from Python 3.11; its missing an exception to the single-unpacking rule (which exists to prevent ambiguity) to allow unambiguous cases. Then you would have: from typing import TypeVarTuple Ts = TypeVarTuple("Ts”) Us = TypeVarTuple("Us”) def tconcat( t1: tuple[*Ts], t2: tuple[*Us] ) -> tuple[*Ts, *Us]: ...

That's the conclusion I also arrived at. The type system is slowly getting there, but many operations like this one are still not possible.

It's a shame, because it makes features like decorators significantly harder to use with static typing.

Re: Writing and linting Python at scale

#90

Post by youerbt »

Timon3 wrote: post #80
dazzlefruit wrote: post #76

Earlier quoted context omitted.

What's missing is in which context the tuple's length is variable, and in which context it is fixed. You can have a tuple size fixed everywhere (because the callee sets it) or a tuple size fixed at each call (and propagated to the callee statically). "Arbitrary" doesn't really help because it could refer to the elements' values, to their types, or to the tuple's length. Also "arbitrary" and "variable-length" sound li…

> What's missing is in which context the tuple's length is variable, and in which context it is fixed. Simple example: a function has a parameter whose type is "variable-length tuple of int". You can pass any tuple in that is known to have 0..n elements, all of type int. What would you have me call that, other than the name I've seen used in discussions on this feature? > "Arbitrary" doesn't really help because it co…

I'd call that function: polymorphic over tuple length.
Post reply on HN