Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Writing and linting Python at scale

engineering.fb.com

Post reply on HN
Print view View original

71–80 of 160 posts

Re: Writing and linting Python at scale

#71

Post by dehrmann »

lagt_t wrote: post #5

But but but HN told me that python is only good for small scale and prototypes, the engineers at meta are wrong

Disclosure: Meta employee, but these views are my own, and not all the experiences are from Meta.

Type annotation have made Python much more scalable in terms of engineers and codebase size.

It still has other scale problems, especially if you actually need threads. One project I worked on managed Python worker tasks, and we resorted to subprocesses (within subprocesses!) because what we thought was IO-bound became CPU-bound, and workers started timing out on RPC calls. I also worked on a Python API service that scaled beautifully horizontally, but we had to manage extra logic for spinning up one worker per CPU.

At some point, you actually start caring about performance, but you're more likely to hit other issues before you care about the extra hardware cost.

Re: Writing and linting Python at scale

#72

Post by Timon3 »

Izkata wrote: post #68
Timon3 wrote: post #63

Earlier quoted context omitted.

> Tuples are already fixed size by nature, so adding a redundant "fixed-size" in that description was confusing. No, you can type variable-length tuples in Python. A variable int tuple, for example, can be typed as Tuple[int, ...]. You can't concatenate two variable-length tuples, which makes sense - where would the cutoff be? But you should absolutely be able to concatenate two fixed-size tuples, and it's very limit…

> A variable int tuple, for example, can be typed as Tuple[int, ...]. That's a type that matches tuples of any length, not a variable-length tuple. The size of a tuple can't be changed. A variable-length tuple doesn't even really make sense, what you'd want there is a list. > You can't concatenate two variable-length tuples, which makes sense - where would the cutoff be? But you should absolutely be able to concatena…

> That's a type that matches tuples of any length, not a variable-length tuple.

A tuple with a type that matches variable lengths of tuples is a variable-length tuple for that piece of code. You're free to show me some official definitions that proves this wording false, but until then it's useless nitpicking. Though you should probably take that up with Guido, who also calls them variable-length tuples: https://github.com/python/typing/issues/30

> This whole statement doesn't make sense. I'm assuming you're still talking about type definitions and not actually tuples.

The statement makes perfect sense, thank you. If you have trouble understanding my messages without me repeating the whole definition every time, maybe just skip them.

Re: Writing and linting Python at scale

#73

Post by Timon3 »

dazzlefruit wrote: post #69
Timon3 wrote: post #63

Earlier quoted context omitted.

> Tuples are already fixed size by nature, so adding a redundant "fixed-size" in that description was confusing. No, you can type variable-length tuples in Python. A variable int tuple, for example, can be typed as Tuple[int, ...]. You can't concatenate two variable-length tuples, which makes sense - where would the cutoff be? But you should absolutely be able to concatenate two fixed-size tuples, and it's very limit…

You mean that the tuples' size is statically known at the calling site, while your message could be interpreted as the size being statically known in the callee. I think this is clearer. The statement "arbitrary fixed-size tuples" sounded a bit like "an immutable mutable variable". It doesn't really say what's arbitrary about the tuples and in what context the size is fixed.

Considering the opposite is called a "variable-length tuple", and I wanted to express that I'm talking about arbitrary tuples with non-variable length, what wording would have made this clearer?

Re: Writing and linting Python at scale

#74

Post by dazzlefruit »

Timon3 wrote: post #70
dazzlefruit wrote: post #65

Earlier quoted context omitted.

"Arbitrary fixed-size tuples" probably don't have a widely accepted meaning :) I read it as "size known when compiling the function". The second example is cool. But I can't find a good practical use case for either example. If you have a collection that's both heterogeneous and whose [size/key set] is statically known, when would it make sense to apply such generic transformations to them? This sounds like you have…

> "Arbitrary fixed-size tuples" probably don't have a widely accepted meaning :) I read it as "size known when compiling the function". That is exactly what I'm talking about - the size of the tuples is known statically. > The second example is cool. But I can't find a good practical use case for either example. There are many interesting use cases in libraries, especially for some of the more esoteric features. Ever…

> But I have a use case for exactly this feature. Why should the language limit me? Why should I implement x functions that take different tuple lengths, with me having to choose the correct one for each use case, when I could write one function that does all?

If the PSF ran a poll for the most-wanted type checking features, I don't think this would come close to first. This sounds very niche. The people working on typing seemed very busy in the last few versions.

> Simple example - I have functions that return a Rust-like Result type, and I want to transform that into a different tuple-based format using a decorator. The transformation itself is static, but I can't write one function that handles it all, because Pythons type system is simply not developed enough. Something that would be incredibly easy in Typescript.

Returning errors as values isn't really how you're supposed to use Python though. And why is there a second tuple-based format that does the same thing?

It also smells slightly off that the rest of the code takes an object that's exactly similar to the first function's result, but with a transformation uniformly applied over the values. Shouldn't the first part's output and the second part's inputs both be clearly declared independently of each other? And then wouldn't it be an extremely niche case that both types are identical except for one transformation applied to all values? Is it worth the language complexity and a dedicated function?

Re: Writing and linting Python at scale

#75

Post by misnome »

ltbarcly3 wrote: post #15
wdroz wrote: post #6

I'm happy with Ruff[0], it's very fast. [0] -- https://github.com/astral-sh/ruff

Unfortunately ruff is very inconsistent and has lots of differences from the flake8 plugins it tries to emulate. Lots of rules are confused by irrelevant context so that it can miss lots of things it should find when the equivalent flake8 plugin still find them. It's automatic fixing of issues will happily introduce other issues that it doesn't find until the next run. I've tried pretty hard to use it and gave up, it…

> It's automatic fixing of issues will happily introduce other issues that it doesn't find until the next run.

I think a few of the "bad" auto fixes have been recently disabled by default, they now separate fixes into "safe" and "unsafe" (for example: it used to always change `x == True` to `x is True`, which broken common numpy selection patterns)

Re: Writing and linting Python at scale

#76

Post by dazzlefruit »

Timon3 wrote: post #73
dazzlefruit wrote: post #69

Earlier quoted context omitted.

You mean that the tuples' size is statically known at the calling site, while your message could be interpreted as the size being statically known in the callee. I think this is clearer. The statement "arbitrary fixed-size tuples" sounded a bit like "an immutable mutable variable". It doesn't really say what's arbitrary about the tuples and in what context the size is fixed.

Considering the opposite is called a "variable-length tuple", and I wanted to express that I'm talking about arbitrary tuples with non-variable length, what wording would have made this clearer?

What's missing is in which context the tuple's length is variable, and in which context it is fixed. You can have a tuple size fixed everywhere (because the callee sets it) or a tuple size fixed at each call (and propagated to the callee statically).

"Arbitrary" doesn't really help because it could refer to the elements' values, to their types, or to the tuple's length. Also "arbitrary" and "variable-length" sound like synonyms to me.

Guido might use some expressions in the context of Python steering discussions but that doesn't make them less obscure for the rest of us who read C++ docs every day instead.

Re: Writing and linting Python at scale

#77

Post by insanitybit »

npage97 wrote: post #66
insanitybit wrote: post #56

Earlier quoted context omitted.

Cool, guess they finally fixed this. Must've been in the last ~1 year, give or take. Of course, it relies on quoting your types, which is... a matter of taste, I suppose.

You can also use from __future__ import annotations so the quotes become unnecessary. https://peps.python.org/pep-0563/

Does that work with recursive types? I have had mixed results with `from __future__ import annotations` personally, but I haven't written much Python in ~a year or so.

Re: Writing and linting Python at scale

#78

Post by Timon3 »

dazzlefruit wrote: post #74
Timon3 wrote: post #70

Earlier quoted context omitted.

> "Arbitrary fixed-size tuples" probably don't have a widely accepted meaning :) I read it as "size known when compiling the function". That is exactly what I'm talking about - the size of the tuples is known statically. > The second example is cool. But I can't find a good practical use case for either example. There are many interesting use cases in libraries, especially for some of the more esoteric features. Ever…

> But I have a use case for exactly this feature. Why should the language limit me? Why should I implement x functions that take different tuple lengths, with me having to choose the correct one for each use case, when I could write one function that does all? If the PSF ran a poll for the most-wanted type checking features, I don't think this would come close to first. This sounds very niche. The people working on t…

> If the PSF ran a poll for the most-wanted type checking features, I don't think this would come close to first. This sounds very niche. The people working on typing seemed very busy in the last few versions.

Sure, but while it's not possible to type basic functions like ones that concatenate tuples, I can say that Pythons typing system is not superior to Typescripts.

> Returning errors as values isn't really how you're supposed to use Python though.

Okay, so how am I supposed to handle non-exceptional errors? Because using exceptions for that kind of thing absolutely isn't good practice.

> And why is there a second tuple-based format that does the same thing?

Legacy. Typescript allows me to do refactoring of things like these step by step and very easily. Python doesn't, because it's inflexible.

> It also smells slightly off that the rest of the code takes an object that's exactly similar to the first function's result, but with a transformation uniformly applied over the values. Shouldn't the first part's output and the second part's inputs both be clearly declared independently of each other? And then wouldn't it be an extremely niche case that both types are identical except for one transformation applied to all values? Is it worth the language complexity and a dedicated function?

Are we taking apart my code now or what? I can't stop the world and focus only on refactoring things to be neat and tidy for months on end. But I can improve individual parts, bit by bit - and in a language with a better typing system, I can do so way easier.

Re: Writing and linting Python at scale

#79

Post by chlorion »

Timon3 wrote: post #49
dazzlefruit wrote: post #44

Earlier quoted context omitted.

For the tuple example: from typing import TypeVar T, U, V, W = TypeVar('T'), TypeVar('U'), TypeVar('V'), TypeVar('W') def concatenate(a: tuple[T, U], b: tuple[V, W]) -> tuple[T, U, V, W]: return a + b For the generic type transformation example, I'm not sure what you mean: from typing import Any, Callable Transformer = Callable[[dict[str, Any]], dict[Callable, Any]] This seems to match your question but it's really w…

Your tuple example only works if both tuples have two elements. I specifically mentioned arbitrary fixed-size tuples (as in, tuples with an arbitrary non-variable length). Your generic type transformation example also doesn't come close to what Typescript does. The resulting dict will not have known keys based on the keys of the input dict. In Typescript I can write a function that takes an object with known keys, an…

The tuple thing requires variadic generics from my understanding.

I don't thing variadic generics support is supported in most statically typed languages. The only one I can think of right now that supports this is C++.

Re: Writing and linting Python at scale

#80

Post by Timon3 »

dazzlefruit wrote: post #76
Timon3 wrote: post #73

Earlier quoted context omitted.

Considering the opposite is called a "variable-length tuple", and I wanted to express that I'm talking about arbitrary tuples with non-variable length, what wording would have made this clearer?

What's missing is in which context the tuple's length is variable, and in which context it is fixed. You can have a tuple size fixed everywhere (because the callee sets it) or a tuple size fixed at each call (and propagated to the callee statically). "Arbitrary" doesn't really help because it could refer to the elements' values, to their types, or to the tuple's length. Also "arbitrary" and "variable-length" sound li…

> What's missing is in which context the tuple's length is variable, and in which context it is fixed.

Simple example: a function has a parameter whose type is "variable-length tuple of int". You can pass any tuple in that is known to have 0..n elements, all of type int. What would you have me call that, other than the name I've seen used in discussions on this feature?

> "Arbitrary" doesn't really help because it could refer to the elements' values, to their types, or to the tuple's length.

Read it as (arbitary (fixed-size tuples)). It was meant to forgo answers describing functions with known tuple sizes.

Post reply on HN