Live data from Hacker News

Writing and linting Python at scale

engineering.fb.com

101–110 of 160 posts

Re: Writing and linting Python at scale

#101

Earlier quoted context omitted.

Python can almost do this with Variadic Generics from Python 3.11; its missing an exception to the single-unpacking rule (which exists to prevent ambiguity) to allow unambiguous cases. Then you would have: from typing import TypeVarTuple Ts = TypeVarTuple("Ts”) Us = TypeVarTuple("Us”) def tconcat( t1: tuple[*Ts], t2: tuple[*Us] ) -> tuple[*Ts, *Us]: ...

No this is actually wrong. It works but it doesn't capture the meaning of the nature of a tuple. A Tuple is Typed as something as a Fixed size. That's right, it's like this at the Type level. The entire concept of a tuple is a Product type or essentially like a struct but with no names for each parameter. Tuple[int, str, float] #correct Doing what you're doing here is equivalent to creating a Struct with variadic pro…

It absolutely isn't wrong, and it's strange that you have some aversion against it. Think of Tuple[int, ...] as a union of Tuples of all lengths containing only ints. It's a specific language feature, and it's not "wrong" in any way.

And I have no idea why you claim that a list is "correct".

Re: Writing and linting Python at scale

#102
post #34

Earlier quoted context omitted.

> Python not only has types but it's type system is superior to typeScript. I strongly disagree. It might be better at some things, but it's much worse at others. Many functions can't be accurately typed (try, for example, to make a well-typed function that concatenates two arbitrary fixed-size tuples), and as far as I know generic type transformations can't be implemented (random example "this type takes a Dict[str,…

def concat_tuples(tuple1: Tuple[int, str, float], tuple2: Tuple[str, str, str]) -> Tuple[int, str, float, str, str, str]: return tuple1 + tuple2 Your second function you just broke out of the type system with Any. Give me a more exact, are you saying Any is a constrained type variable? The only possibility here is this: T = TypeVar('T') def transform(x: Dict[str, T]) -> Dict[str, Callable[[], T]]: return {key: lambda…

> concat_tuples

I specifically talked about arbitrary tuples. Your function only handles specific tuples.

> Your second function you just broke out of the type system with Any. Give me a more exact, are you saying Any is a constrained type variable?

I'm not sure what you're trying to say here. Yes, you can implement the function, I never claimed otherwise. I'm talking about the _type system_.

> The thing is because these type checkers are external, anyone can add arbitrary features to them and extend it.

The type system is a core feature of Python. Until it supports what I want, I can't really use what I want with Python.

Re: Writing and linting Python at scale

#103
post #66

Earlier quoted context omitted.

You can also use from __future__ import annotations so the quotes become unnecessary. https://peps.python.org/pep-0563/

Does that work with recursive types? I have had mixed results with `from __future__ import annotations` personally, but I haven't written much Python in ~a year or so.

[deleted]

Re: Writing and linting Python at scale

#104
post #82

Earlier quoted context omitted.

Typescript supports it too (quick example[0]) :) and Python actually as well, but currently you can't unpack two TypeVarTuples in the same type expression: https://peps.python.org/pep-0646/ [0] https://www.typescriptlang.org/play?#code/C4TwDgpgBAglC8UDaA...

Wow this is very cool thanks for sharing!

You're welcome! That's why I'm very excited about Typescript, the system is very powerful :)

Re: Writing and linting Python at scale

#105
post #49

Earlier quoted context omitted.

Your tuple example only works if both tuples have two elements. I specifically mentioned arbitrary fixed-size tuples (as in, tuples with an arbitrary non-variable length). Your generic type transformation example also doesn't come close to what Typescript does. The resulting dict will not have known keys based on the keys of the input dict. In Typescript I can write a function that takes an object with known keys, an…

> I specifically mentioned arbitrary fixed-size tuples (as in, tuples with an arbitrary non-variable length). This is wrong. Again, Arbitrary fixed-size tuples are equivalent to structs with an arbitrary amount of properties. Languages shouldn't do this, it destroys the nature of what a TUPLE is which is essentially just a struct with no names. The concept you are going for is isomorphically encapsulated by ANOTHER t…

> This is wrong. Again, Arbitrary fixed-size tuples are equivalent to structs with an arbitrary amount of properties. Languages shouldn't do this, it destroys the nature of what a TUPLE is which is essentially just a struct with no names.

Okay, that might be your personal feelings on the topic. But do you understand the concept of "generic functions"? Sometimes you have to apply generic transforms to data. Being able to correctly express your transformations in a type system isn't "wrong", it's useful.

> List[Any]

Sorry, but I really think you don't understand what I'm talking about. If I write a function that handles tuples of arbitrary length and that function returns a transformed version of that tuple, I keep the information about individual tuple elements. This is thrown away in a list.

> That being said if javascript has variadic tuples then it's not a very good type system imo. It encodes redundant concepts. Why have a tuple with Variadic arguments when I have Arrays that do the exact same thing?

Arrays don't do the same thing, so they are not redundant concepts. Tuples have elements in specified positions with specified types. Arrays have one type (possibly a union type) over many elements.

Re: Writing and linting Python at scale

#106

Earlier quoted context omitted.

Python can almost do this with Variadic Generics from Python 3.11; its missing an exception to the single-unpacking rule (which exists to prevent ambiguity) to allow unambiguous cases. Then you would have: from typing import TypeVarTuple Ts = TypeVarTuple("Ts”) Us = TypeVarTuple("Us”) def tconcat( t1: tuple[*Ts], t2: tuple[*Us] ) -> tuple[*Ts, *Us]: ...

No this is actually wrong. It works but it doesn't capture the meaning of the nature of a tuple. A Tuple is Typed as something as a Fixed size. That's right, it's like this at the Type level. The entire concept of a tuple is a Product type or essentially like a struct but with no names for each parameter. Tuple[int, str, float] #correct Doing what you're doing here is equivalent to creating a Struct with variadic pro…

> It works but it doesn't capture the meaning of the nature of a tuple.

That's backwards. It doesn't work (as noted in GP, because Python supports unpacking only a single variadic parameter in a type annotation), but it does capture the nature of a tuple.

It is (or, more precisely, would be, if the syntactic limitation was relaxed to allow it) a generic function that operates on two tuples of arbitrary tuple types and returns a tuple of a third tuple type whose shape is a simple concatenation of the shapes of the input tuples.

> Doing what you're doing here is equivalent to creating a Struct with variadic properties.

Well, a "struct" in Python is a particular low-level byte-mapped datatype, but the generic concept of a Struct differs from a tuple in two ways -- fields identified by name and that Structs are generally mutable.

> If you want some container that holds an arbitrary amount of things that is a List

But... I don't want that, and that's not what this works on.

There's a difference between a function that works on containers of changeable length and a generic function that works on immutable containers (so, fixed length and contents), but is adaptable to any length (or more specifically, shape, including not only length by order of types of elements) of arguments, producing a result of a container type whose shape is strictly determined by the shapes of the inputs, with the shapes statically verifiable.

Python is very close to allowing that, which is very different than:

  def concat(l1: list, l2: list) -> list: ...
which is a fine function, but not at all what the other one is about.

Re: Writing and linting Python at scale

#107

Earlier quoted context omitted.

> dynamic scripting languages. Why keep repeating this nonsense? "Dynamic" or "scripting" aren't features of languages. When anyone says something like this, it's like talking about square chicken... (i.e. a category error). Obviously, you had some idea in your mind, and you wanted to communicate it somehow, but your readers will not know what it was unless you make an effort to analyze what you want to say and make…

> "Dynamic" or "scripting" aren't features of languages Surely dynamic typing is a language feature? I can't imagine what else someone would refer to with "dynamic".

One could make the argument it's the opposite of a feature...

Re: Writing and linting Python at scale

#108
post #105

Earlier quoted context omitted.

> I specifically mentioned arbitrary fixed-size tuples (as in, tuples with an arbitrary non-variable length). This is wrong. Again, Arbitrary fixed-size tuples are equivalent to structs with an arbitrary amount of properties. Languages shouldn't do this, it destroys the nature of what a TUPLE is which is essentially just a struct with no names. The concept you are going for is isomorphically encapsulated by ANOTHER t…

> This is wrong. Again, Arbitrary fixed-size tuples are equivalent to structs with an arbitrary amount of properties. Languages shouldn't do this, it destroys the nature of what a TUPLE is which is essentially just a struct with no names. Okay, that might be your personal feelings on the topic. But do you understand the concept of "generic functions"? Sometimes you have to apply generic transforms to data. Being able…

>Okay, that might be your personal feelings on the topic. But do you understand the concept of "generic functions"? Sometimes you have to apply generic transforms to data. Being able to correctly express your transformations in a type system isn't "wrong", it's useful.

There's nothing like this in any type system I've seen. A struct with a generic amount of properties? Nonexistent. This isn't personal. This is the definition of a tuple. A tuple is a struct with no names. It is not a personal opinion.

You can have generic functions that operate on generic types but there's no such thing as a struct with generic amount of properties. Closest thing is a list.

>Sorry, but I really think you don't understand what I'm talking about. If I write a function that handles tuples of arbitrary length and that function returns a transformed version of that tuple, I keep the information about individual tuple elements. This is thrown away in a list.

This isn't an opinion. There's no such thing as tuple types of arbitrary length unless the implementer decides to get hand wavy with the definition of what a tuple is.

What you're talking about is only possible with dependent types. Very very few languages support this but the risk of doing this is it makes type checking undecidable. It's also extremely challenging to program this way.

Does typescript support dependent types? Probably but that's outside the realm of normal programming it's most likely exists as obscure tricks. You're getting into Idris, proof checkers and such.

Imagine this:

    func (x: Array[N], x:Array[M]) -> Array[M + N]
where M and N is the size of the array. It's called dependent types because types are getting mixed with programming level terms.

This is essentially what you need, but you want this level of type checking with structs/tuples. It's not just a "generic variable" It is much more then that:

   func(x: Tuple[*args1] y: Tuple[*args2]) -> Tuple[*(args1 + args2)]
There's nothing wrong with this but once you get into this it's beyond traditional type systems. Practical programming rarely ventures to far into this world since it's really hard to even fully prove even trivial things. You'll see it's bringing the execution of programs into the type checking level.

Maybe typescript has some shortcut that makes this level of type checking available for tuples, maybe that's what you're getting at. Unlikely that dependent types are supported generically.

If ts Does supports dependent types, this is definitely something I did not know about. It does change the equation, but I suspect it's very much outside normal usage of the language.

>Arrays don't do the same thing, so they are not redundant concepts. Tuples have elements in specified positions with specified types. Arrays have one type (possibly a union type) over many elements.

Arrays are the thing you want for variadic containers. For memory optimized languages like rust or C++ arrays are defined with a size. Array[5] is a different type then Array[3].

You can define a "interface" that accepts generic arguments to arrays:

    func(a: Array[], b: Array[]) 
but you can't define the function above where a function creates a new type that's dependent on the internal types of a and b.

Re: Writing and linting Python at scale

#109
post #100

Earlier quoted context omitted.

I haven't seen a type system that allows variadic types for Tuples. This would be equivalent to creating a struct with variadic amount of properties. The definition for tuples here is similar to a struct. They are one in the same except structs have names for each property while tuples don't. That is literally the main concept of a tuple, just a struct with no names for properties. The type system for python is alrea…

That's confusing, considering Python has them with TypeVarTuples?

Yeah they're getting flexible with the definitions. The developers are committees of people many of which don't know type theory and introduce arbitrary concepts based off of misguided intuitions.

It's the same with typescript I'm sure.

You will note that this thing doesn't exist in haskell for tuples because haskell devs tend to be well versed with concept of what a tuple is.

Re: Writing and linting Python at scale

#110

Earlier quoted context omitted.

I literally said it supports exhaustive pattern matching which adds a level of safety and flexibility superior to that of type script. Many IDEs have real time type checking that highlights the errors so don't even have to run the external checker. Even if you don't use IDEs running the type checker is measured in seconds. Not far off from linters that most people will also use for TS. What you say about the librarie…

>I literally said it supports exhaustive pattern matching which adds a level of safety and flexibility superior to that of type script. I don't have an opinion on which language is superior here, I've never written typescript or even javascript before, but I think saying that python's type system/checkers is superior because of this one feature is not correct. I'm also skeptical of the claim that typescript doesn't s…

>I don't have an opinion on which language is superior here, I've never written typescript or even javascript before, but I think saying that python's type system/checkers is superior because of this one feature is not correct.

I meant, it supports basically every common/useful type feature in addition to this one which is very powerful.

>I'm also skeptical of the claim that typescript doesn't support exhaustive branching. This very well could be true but it seems hard to believe.

It is true. Look it up. Very few languages support this feature. Python is able to do it because the type checker is external so developers can create all kinds of features and move faster then core python development. But by default other then Rust, Rust is basically the only popular language that officially has this concept.

Post reply on HN