Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Writing and linting Python at scale

engineering.fb.com

Post reply on HN
Print view View original

111–120 of 160 posts

Re: Writing and linting Python at scale

#111

Post by dragonwriter »

corethree wrote: post #97
Timon3 wrote: post #89

Earlier quoted context omitted.

That's the conclusion I also arrived at. The type system is slowly getting there, but many operations like this one are still not possible. It's a shame, because it makes features like decorators significantly harder to use with static typing.

I haven't seen a type system that allows variadic types for Tuples. This would be equivalent to creating a struct with variadic amount of properties. The definition for tuples here is similar to a struct. They are one in the same except structs have names for each property while tuples don't. That is literally the main concept of a tuple, just a struct with no names for properties. The type system for python is alrea…

> I haven't seen a type system that allows variadic types for Tuples. This would be equivalent to creating a struct with variadic amount of properties.

No, it wouldn't.

It would be equivalent to creating a generic type where the concrete types that would be compatible with it would be structs of different shapes.

But if you use the same variadic type within a particular context (e.g., a function signature and definition), then any given call to the function, that variadic type must represent the same concrete type each place.

Re: Writing and linting Python at scale

#112

Post by corethree »

Timon3 wrote: post #101
corethree wrote: post #96

Earlier quoted context omitted.

No this is actually wrong. It works but it doesn't capture the meaning of the nature of a tuple. A Tuple is Typed as something as a Fixed size. That's right, it's like this at the Type level. The entire concept of a tuple is a Product type or essentially like a struct but with no names for each parameter. Tuple[int, str, float] #correct Doing what you're doing here is equivalent to creating a Struct with variadic pro…

It absolutely isn't wrong, and it's strange that you have some aversion against it. Think of Tuple[int, ...] as a union of Tuples of all lengths containing only ints. It's a specific language feature, and it's not "wrong" in any way. And I have no idea why you claim that a list is "correct".

You're thinking fixed size arrays. It doesn't make sense to have tuples of arbitrary size. To make it arbitrary size you have to constrain it on a single generic parameter which makes it functionally equivalent to a fixed size array. You essentially just do this with an array or a list.

Read my other reply to you to see the problem of taking functions with "generic" number of parameters. What are you going to do with each parameter? What are the types of each parameter? You can't do anything other then breaking out of the type system. Basically you're filling those tuples with a bunch of "Any" types. So what's the point of the type system now? There is No point. Just like List[Any]

To get your idea to work you need dependent types.

Re: Writing and linting Python at scale

#113

Post by corethree »

Timon3 wrote: post #102
corethree wrote: post #94

Earlier quoted context omitted.

def concat_tuples(tuple1: Tuple[int, str, float], tuple2: Tuple[str, str, str]) -> Tuple[int, str, float, str, str, str]: return tuple1 + tuple2 Your second function you just broke out of the type system with Any. Give me a more exact, are you saying Any is a constrained type variable? The only possibility here is this: T = TypeVar('T') def transform(x: Dict[str, T]) -> Dict[str, Callable[[], T]]: return {key: lambda…

> concat_tuples I specifically talked about arbitrary tuples. Your function only handles specific tuples. > Your second function you just broke out of the type system with Any. Give me a more exact, are you saying Any is a constrained type variable? I'm not sure what you're trying to say here. Yes, you can implement the function, I never claimed otherwise. I'm talking about the _type system_. > The thing is because t…

>I'm not sure what you're trying to say here. Yes, you can implement the function, I never claimed otherwise. I'm talking about the _type system_.

The type constrains the definition. Look at the definition, there is ONLY one possible definition when converting this:

    Dict[str, T] -> Dict[str, Callable[[], T]]
Literally. Try to think of another way to define the lambda, you can't. The more open the type the more constrained the definition the more constrained the type the more open the definition.

> I specifically talked about arbitrary tuples. Your function only handles specific tuples.

I remarked on this in other replies. How will you define the return value of the concat and have it exactly type correct? It's not possible bro. You can't do much with tuples of arbitrary types. Not unless you have dependent types.

>The type system is a core feature of Python. Until it supports what I want, I can't really use what I want with Python.

Type checking in python is an external application. The interpreter doesn't do any type check. Type checking is achieved by running another program. All kinds of additional features and custom interpretations on types can be inserted into these type checkers.

Re: Writing and linting Python at scale

#114

Post by FreakLegion »

insanitybit wrote: post #77
npage97 wrote: post #66

Earlier quoted context omitted.

You can also use from __future__ import annotations so the quotes become unnecessary. https://peps.python.org/pep-0563/

Does that work with recursive types? I have had mixed results with `from __future__ import annotations` personally, but I haven't written much Python in ~a year or so.

It's worked for many years, but you won't often see it used outside of class definitions because all of the other tools struggle with it (Pylint, Flake8, Pylance, etc. spit out some variation of an undefined variable error).

Re: Writing and linting Python at scale

#115

Post by dragonwriter »

corethree wrote: post #112
Timon3 wrote: post #101

Earlier quoted context omitted.

It absolutely isn't wrong, and it's strange that you have some aversion against it. Think of Tuple[int, ...] as a union of Tuples of all lengths containing only ints. It's a specific language feature, and it's not "wrong" in any way. And I have no idea why you claim that a list is "correct".

You're thinking fixed size arrays. It doesn't make sense to have tuples of arbitrary size. To make it arbitrary size you have to constrain it on a single generic parameter which makes it functionally equivalent to a fixed size array. You essentially just do this with an array or a list. Read my other reply to you to see the problem of taking functions with "generic" number of parameters. What are you going to do with…

> To make it arbitrary size you have to constrain it on a single generic parameter

Your information is out of date since Python 3.11:

https://peps.python.org/pep-0646

Re: Writing and linting Python at scale

#116

Post by seabrookmx »

corethree wrote: post #33
seabrookmx wrote: post #22

Earlier quoted context omitted.

Yes but the if you want to do ahead-of-time type checking you need to run a tool like mypy, and none of those tools are as comprehensive or performant as Typescript. Also the ecosystem of libraries with type annotations is much smaller (Typescript has the first mover advantage, after all).

I literally said it supports exhaustive pattern matching which adds a level of safety and flexibility superior to that of type script. Many IDEs have real time type checking that highlights the errors so don't even have to run the external checker. Even if you don't use IDEs running the type checker is measured in seconds. Not far off from linters that most people will also use for TS. What you say about the librarie…

Maybe comprehensive is the wrong word to use. But Typescript is definitely more mature.

I really like to run these checks alongside black, pyupgrade etc. in my CI pipelines. And under the hood your IDE is just running the same "external" checker and parsing it's results to provide it's smarts.

As for performance, early mypy versions took literally minutes on a moderately sized FastAPI codebase I maintain. The team (which includes Guido himself!) literally created a new python native compiler (mypyc) to try and improve performance of the tool.

I like types in Python but it's early days compared to a more mature tool like Typescript. IMO it's pretty clear cut if you've spent significant time using both.

Re: Writing and linting Python at scale

#117

Post by Timon3 »

corethree wrote: post #113
Timon3 wrote: post #102

Earlier quoted context omitted.

> concat_tuples I specifically talked about arbitrary tuples. Your function only handles specific tuples. > Your second function you just broke out of the type system with Any. Give me a more exact, are you saying Any is a constrained type variable? I'm not sure what you're trying to say here. Yes, you can implement the function, I never claimed otherwise. I'm talking about the _type system_. > The thing is because t…

>I'm not sure what you're trying to say here. Yes, you can implement the function, I never claimed otherwise. I'm talking about the _type system_. The type constrains the definition. Look at the definition, there is ONLY one possible definition when converting this: Dict[str, T] -> Dict[str, Callable[[], T]] Literally. Try to think of another way to define the lambda, you can't. The more open the type the more constr…

> The type constrains the definition. Look at the definition, there is ONLY one possible definition when converting this:

Yes, and if Python had a better type system, I could write a more concrete definition. Why don't you understand that?

> I remarked on this in other replies. How will you define the return value of the concat and have it exactly type correct? It's not possible bro. You can't do much with tuples of arbitrary types. Not unless you have dependent types.

Of course it's possible, the type system even supports almost everything that's necessary. You can already define what I want with a single fixed-size tuple of arbitrary size, you just can't do it with multiple ones. Why do you have this strange fixation on calling this "wrong" and "impossible"?

Re: Writing and linting Python at scale

#118

Post by Timon3 »

corethree wrote: post #112
Timon3 wrote: post #101

Earlier quoted context omitted.

It absolutely isn't wrong, and it's strange that you have some aversion against it. Think of Tuple[int, ...] as a union of Tuples of all lengths containing only ints. It's a specific language feature, and it's not "wrong" in any way. And I have no idea why you claim that a list is "correct".

You're thinking fixed size arrays. It doesn't make sense to have tuples of arbitrary size. To make it arbitrary size you have to constrain it on a single generic parameter which makes it functionally equivalent to a fixed size array. You essentially just do this with an array or a list. Read my other reply to you to see the problem of taking functions with "generic" number of parameters. What are you going to do with…

> You're thinking fixed size arrays. It doesn't make sense to have tuples of arbitrary size.

My god. No, Python doesn't support "fixed size arrays", but it does support tuples of arbitrary size, no matter how often you claim that to be wrong.

> Read my other reply to you to see the problem of taking functions with "generic" number of parameters. What are you going to do with each parameter? What are the types of each parameter? You can't do anything other then breaking out of the type system. Basically you're filling those tuples with a bunch of "Any" types. So what's the point of the type system now? There is No point. Just like List[Any]

I could, for example, do the thing I've been writing dozens of replies asking for: concatenate two arbitrary tuples. It's a perfectly well-defined operation, and without an artificial limitation on unpacking of TypeVarTuples, it would already be possible.

It is already possible to do what you claim impossible with one arbitrary tuple. How can that be? You keep going on and on about how that doesn't work. Why does it?

Re: Writing and linting Python at scale

#119

Post by Timon3 »

corethree wrote: post #108
Timon3 wrote: post #105

Earlier quoted context omitted.

> This is wrong. Again, Arbitrary fixed-size tuples are equivalent to structs with an arbitrary amount of properties. Languages shouldn't do this, it destroys the nature of what a TUPLE is which is essentially just a struct with no names. Okay, that might be your personal feelings on the topic. But do you understand the concept of "generic functions"? Sometimes you have to apply generic transforms to data. Being able…

>Okay, that might be your personal feelings on the topic. But do you understand the concept of "generic functions"? Sometimes you have to apply generic transforms to data. Being able to correctly express your transformations in a type system isn't "wrong", it's useful. There's nothing like this in any type system I've seen. A struct with a generic amount of properties? Nonexistent. This isn't personal. This is the de…

> There's nothing like this in any type system I've seen. A struct with a generic amount of properties? Nonexistent. This isn't personal. This is the definition of a tuple. A tuple is a struct with no names. It is not a personal opinion.

I have literally shown you two type systems that have this feature. Why do you ignore them?

> You can have generic functions that operate on generic types but there's no such thing as a struct with generic amount of properties. Closest thing is a list.

Why are you acting like TypeVarTuples don't exist?

> This isn't an opinion. There's no such thing as tuple types of arbitrary length unless the implementer decides to get hand wavy with the definition of what a tuple is.

Again, why are you acting like TypeVarTuples don't exist?

> Arrays are the thing you want for variadic containers. For memory optimized languages like rust or C++ arrays are defined with a size. Array[5] is a different type then Array[3]

No, they are not, and I don't understand how you still don't get that. Python arrays don't carry any information about their length in their type, Python tuples do. They are not the same.

> but you can't define the function above where a function creates a new type that's dependent on the internal types of a and b.

Python literally already supports typing a function that creates a type that's dependent on the internal types of a and b. Why do you keep claiming it doesn't?

Re: Writing and linting Python at scale

#120

Post by Timon3 »

corethree wrote: post #109
Timon3 wrote: post #100

Earlier quoted context omitted.

That's confusing, considering Python has them with TypeVarTuples?

Yeah they're getting flexible with the definitions. The developers are committees of people many of which don't know type theory and introduce arbitrary concepts based off of misguided intuitions. It's the same with typescript I'm sure. You will note that this thing doesn't exist in haskell for tuples because haskell devs tend to be well versed with concept of what a tuple is.

So you keep going on and on about how this is wrong, and doesn't work, and doesn't exist, and it's impossible, because it isn't implemented in Haskell? Seriously?

This is one of the strangest and worst interactions I've had on this site, because I essentially made a comment that goes "the sky is blue, but it looks better at sunset when red" and you keep going "nu-uh! Impossible! Doesn't work! The sky is blue, nothing else makes sense!"

Post reply on HN