Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Writing and linting Python at scale

engineering.fb.com

Post reply on HN
Print view View original

131–140 of 160 posts

Re: Writing and linting Python at scale

#131

Post by Timon3 »

corethree wrote: post #127
Timon3 wrote: post #118

Earlier quoted context omitted.

> You're thinking fixed size arrays. It doesn't make sense to have tuples of arbitrary size. My god. No, Python doesn't support "fixed size arrays", but it does support tuples of arbitrary size, no matter how often you claim that to be wrong. > Read my other reply to you to see the problem of taking functions with "generic" number of parameters. What are you going to do with each parameter? What are the types of each…

>My god. No, Python doesn't support "fixed size arrays", but it does support tuples of arbitrary size, no matter how often you claim that to be wrong. My god I never claimed this. I'm just talking about programming in general. I'm trying to show you that what YOU are asking for is programming operations to happen at the type level. You're not seeing it correctly. >I could, for example, do the thing I've been writing…

The operator exists. I can use it today, without exiting the type system, both in Python (limited to only unpacking one tuple type) and Typescript (not limited). Why do you keep claiming this to be impossible, when it already exists right now?

It's also in no way a special case in TS, the type system supports many things you need dependent types for. Maybe read up on it before you claim it can't do these things?

Re: Writing and linting Python at scale

#132

Post by Timon3 »

corethree wrote: post #129
Timon3 wrote: post #117

Earlier quoted context omitted.

> The type constrains the definition. Look at the definition, there is ONLY one possible definition when converting this: Yes, and if Python had a better type system, I could write a more concrete definition. Why don't you understand that? > I remarked on this in other replies. How will you define the return value of the concat and have it exactly type correct? It's not possible bro. You can't do much with tuples of…

>Yes, and if Python had a better type system, I could write a more concrete definition. Why don't you understand that? I understand everything. It's you who doesn't understand. What I described is for ALL type systems in general. Not python types. >Of course it's possible, the type system even supports almost everything that's necessary. You can already define what I want with a single fixed-size tuple of arbitrary s…

> I understand everything. It's you who doesn't understand. What I described is for ALL type systems in general. Not python types.

Then why is what you're claiming to be impossible possible in Typescript? How can that be? Is the whole world wrong, aside from you?

> Look just prove it to me. Define the type signature for concat function that will concat two tuples with variadic parameters in TS.

You mean the one I already posted yesterday? https://www.typescriptlang.org/play?#code/C4TwDgpgBAglC8UDaA...

Or, if you prefer Python, run this:

  from typing import Tuple, TypeVar, TypeVarTuple, reveal_type

  T = TypeVar("T")
  Ts = TypeVarTuple("Ts")


  def concat(element: T, tup: Tuple[*Ts]) -> Tuple[T, *Ts]:
      return (element, *tup)

  element = "foo"
  tup = (True, 42)

  reveal_type(concat(element, tup))
You said it's impossible for the return type to be dependent on the input type. Why does Python disagree with you?

Re: Writing and linting Python at scale

#133

Post by Timon3 »

corethree wrote: post #130
Timon3 wrote: post #120

Earlier quoted context omitted.

So you keep going on and on about how this is wrong, and doesn't work, and doesn't exist, and it's impossible, because it isn't implemented in Haskell? Seriously? This is one of the strangest and worst interactions I've had on this site, because I essentially made a comment that goes "the sky is blue, but it looks better at sunset when red" and you keep going "nu-uh! Impossible! Doesn't work! The sky is blue, nothing…

>So you keep going on and on about how this is wrong, and doesn't work, and doesn't exist, and it's impossible, because it isn't implemented in Haskell? Seriously? Yeah seriously. You actually don't know what you're talking about. You're not convinced. I'm failing on that end, but this is real, you're not understanding me. Haskell one of the languages with the most advanced type systems in practical programming. Alge…

> Haskell one of the languages with the most advanced type systems in practical programming. Algebraic data types. You clearly haven't used it so of course you remain unconvinced.

Wrong.

> It seems this way to you because frankly, you're not very knowledgeable of what I'm talking about.

This wouldn't come across quite as bad if I didn't literally post an example long ago. If you're so knowledgeable about all this, why can't you acknowledge the examples I've already given? Why are you ignoring them?

Re: Writing and linting Python at scale

#134

Post by ltbarcly3 »

drcongo wrote: post #52
ltbarcly3 wrote: post #15

Earlier quoted context omitted.

Unfortunately ruff is very inconsistent and has lots of differences from the flake8 plugins it tries to emulate. Lots of rules are confused by irrelevant context so that it can miss lots of things it should find when the equivalent flake8 plugin still find them. It's automatic fixing of issues will happily introduce other issues that it doesn't find until the next run. I've tried pretty hard to use it and gave up, it…

We've gone all in on Ruff for several months now, across many projects, and not noticed any of these issues.

If you have a smaller codebase which has had consistent good practices you won't find many things anyway. I have a legacy codebase where ruff finds over 25,000 issues. Many times when I fix all of the issues for a given code (say UP031 just to pick an example) I can then run flake8 and find more valid examples of that same issue. On my desktop running flake8 takes about 7s, so there's literally no reason for me to bother with ruff if it is going to be so much less reliable, even if it is 1000x faster.

Ruff is ok I guess if you barely need it, but I have a medium sized (about 400kloc), quite old, and extremely poorly managed codebase that I support which powers a profitable business. Ruff just can't cope with it, at least as of a few months ago.

Re: Writing and linting Python at scale

#135

Post by insanitybit »

FreakLegion wrote: post #114
insanitybit wrote: post #77

Earlier quoted context omitted.

Does that work with recursive types? I have had mixed results with `from __future__ import annotations` personally, but I haven't written much Python in ~a year or so.

It's worked for many years, but you won't often see it used outside of class definitions because all of the other tools struggle with it (Pylint, Flake8, Pylance, etc. spit out some variation of an undefined variable error).

I don't think it has worked for years. I had used that same exact __annotations__ and my recollection was that recursive types still did not work.

Re: Writing and linting Python at scale

#136

Post by youerbt »

Timon3 wrote: post #123
youerbt wrote: post #90

Earlier quoted context omitted.

I'd call that function: polymorphic over tuple length.

That kind of works. I guess I keep thinking about it from the perspective of the type itself instead of the function that uses it.

It can also get confusing, because in some languages there is no general notion of being a tuple or n-tuple if you wish. Even if they are casually all called tuples.

For example in Haskell 2-tuple and 3-tuple are simply distinct types, as distinct as Int is from String. You can't speak to the type system about "all n-tuples".

Re: Writing and linting Python at scale

#138

Post by esyir »

ltbarcly3 wrote: post #15
wdroz wrote: post #6

I'm happy with Ruff[0], it's very fast. [0] -- https://github.com/astral-sh/ruff

Unfortunately ruff is very inconsistent and has lots of differences from the flake8 plugins it tries to emulate. Lots of rules are confused by irrelevant context so that it can miss lots of things it should find when the equivalent flake8 plugin still find them. It's automatic fixing of issues will happily introduce other issues that it doesn't find until the next run. I've tried pretty hard to use it and gave up, it…

Kinda funny, but my experience was quite the opposite there. When migrating our codebase from flake8 to ruff, I actually found a number of bugs in flake8 and flake 8 bugbear that were resolved by a more correct implementation in ruff

Re: Writing and linting Python at scale

#139

Post by appplication »

drchaim wrote: post #137

I dream with a unique tool that makes all formatting and linting and sorting and whatever. I don't care too much of the final style. I just prefer not to think too much about it. Ruff is close, let's see.

The fact that ruff is expanding with VC funding has me cautiously optimistic. They will hopefully be funded and incentivized to capture the existing fragmented ecosystem of Python dev support tooling. I’m sure their monetization will be more of the same cloud hosted perks etc we see from similar companies. Which is just fine with me, as it keeps their core offering OSS orthogonal to their value prop, which is usually convenience related rather than feature related.

I normally hate one-tool-to-rule-them-all ambitions, but when it comes to linting, type hints, auto formatting, import ordering, etc (basically anything you’d want to put in a pre-commit hook), all of that could have a much more coherent story if brought under one hood. Subverting pre-commit itself would also be great. The story around the virtual envs these hooks run in is incohesive with development best practices (define your dependencies in one and only one place), and author has made it aggressively clear on multiple occasions they are uninterested in fixing that.

Re: Writing and linting Python at scale

#140

Post by dragonwriter »

corethree wrote: post #128
dragonwriter wrote: post #115

Earlier quoted context omitted.

> To make it arbitrary size you have to constrain it on a single generic parameter Your information is out of date since Python 3.11: https://peps.python.org/pep-0646

Bro, Look at this: https://peps.python.org/pep-0646/#multiple-type-variable-tup... Does this not exactly mean: "To make it arbitrary size you have to constrain it on a single generic parameter"

> Bro, Look at this:

I’m not your bro and I've been discussing what that PEP does and does not allow in every post in this subthread.

> Does this not exactly mean: "To make it arbitrary size you have to constrain it on a single generic parameter"

No, it doesn't:

  Ts = TypeVarTuple(*Ts)

  def f(cond: Callable[[],bool], t: tuple[*Ts]) -> None | tuple[*Ts]: ...
The parameter t is an arbitrary-sized (and shaped) tuple, and the return value of f has the same shape as the parameter it is called with.

No multiple unpacking means you can't type things like concatenations of arbitrary-shaped tuples without losing type specificity (and resorting to just tuple), but it doesn't mean you can't type functions using arbitrary sized and shaped tuples outside of that restriction.

Post reply on HN