Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Writing and linting Python at scale

engineering.fb.com

Post reply on HN
Print view View original

61–70 of 160 posts

Re: Writing and linting Python at scale

#61

Post by Izkata »

devjab wrote: post #25
crabbone wrote: post #23

Earlier quoted context omitted.

I've been writing in Python for over ten years, in different roles, for wildly different projects (research, infra, Web, testing, education). I'm yet to find anything Python was good for. On engineering merits alone Python isn't best for anything, nor is it best for combinations of things. It's silly to think that any tool that works with Python does so because Python was the best language for the job, and they only…

I’m not really a fan of Python as such, but after a few decades in the industry, I’m beginning to think that being good at being bandaid is “better”. I can’t think of a single tech where we don’t have a bunch of duct tape (as we refer to it), not so much because we want to but because that’s just how things end up in the imperfect world of organisations. I value the techs that fit into this reality more than the ones…

Old proverb that I think applies here: A jack of all trades is a master of none, but oftentimes better than a master of one.

Re: Writing and linting Python at scale

#62

Post by Izkata »

Timon3 wrote: post #49
dazzlefruit wrote: post #44

Earlier quoted context omitted.

For the tuple example: from typing import TypeVar T, U, V, W = TypeVar('T'), TypeVar('U'), TypeVar('V'), TypeVar('W') def concatenate(a: tuple[T, U], b: tuple[V, W]) -> tuple[T, U, V, W]: return a + b For the generic type transformation example, I'm not sure what you mean: from typing import Any, Callable Transformer = Callable[[dict[str, Any]], dict[Callable, Any]] This seems to match your question but it's really w…

Your tuple example only works if both tuples have two elements. I specifically mentioned arbitrary fixed-size tuples (as in, tuples with an arbitrary non-variable length). Your generic type transformation example also doesn't come close to what Typescript does. The resulting dict will not have known keys based on the keys of the input dict. In Typescript I can write a function that takes an object with known keys, an…

Tuples are already fixed size by nature, so adding a redundant "fixed-size" in that description was confusing. I also thought you meant a predefined size like always 2-tuples or always 3-tuples.

Re: Writing and linting Python at scale

#63

Post by Timon3 »

Izkata wrote: post #62
Timon3 wrote: post #49

Earlier quoted context omitted.

Your tuple example only works if both tuples have two elements. I specifically mentioned arbitrary fixed-size tuples (as in, tuples with an arbitrary non-variable length). Your generic type transformation example also doesn't come close to what Typescript does. The resulting dict will not have known keys based on the keys of the input dict. In Typescript I can write a function that takes an object with known keys, an…

Tuples are already fixed size by nature, so adding a redundant "fixed-size" in that description was confusing. I also thought you meant a predefined size like always 2-tuples or always 3-tuples.

> Tuples are already fixed size by nature, so adding a redundant "fixed-size" in that description was confusing.

No, you can type variable-length tuples in Python. A variable int tuple, for example, can be typed as Tuple[int, ...].

You can't concatenate two variable-length tuples, which makes sense - where would the cutoff be? But you should absolutely be able to concatenate two fixed-size tuples, and it's very limiting that you can't.

Re: Writing and linting Python at scale

#64

Post by Izkata »

dazzlefruit wrote: post #47
crabbone wrote: post #31

Earlier quoted context omitted.

Surprise: all languages have types. Superior to TypeScript is neither a high bar, nor is this any kind of objective metric. I don't know why sum types are a blessing, also I don't know why pattern matching makes anything better. I can name a lot of problems with Python, and I'm sure that libraries isn't the only one. For example, for no reason, Python has multiple unrelated mechanisms to manage program state (object,…

Why is pathlib bad? Edit: I'm asking because pathlib is as good as a Python lib could be for me. Path manipulations are extremely clear and always safe. What more do you need?

I'm guessing verbosity? It also reads like they don't know why pathlib exists and assume they were created at the same time.

os.path came first, often works by poking the filesystem directly even when it doesn't seem like it needs to (vague memory, not completely certain), and I believe has os-specific quirks (so code won't necessarily work without changes).

pathlib was created later as a pure-python implementation that allows for manipulation paths without touching the filsystem except when you explicitly tell it to. Because it's also pure python I don't think it has any os-specific quirks either, but I haven't explored it in depth. Code should work across operating systems without changes.

I think I also remember at one point people talking about completely replacing os.path with pathlib, or at least gutting os.path to the essentials that wouldn't work as part of pathlib.

Re: Writing and linting Python at scale

#65

Post by dazzlefruit »

Timon3 wrote: post #49
dazzlefruit wrote: post #44

Earlier quoted context omitted.

For the tuple example: from typing import TypeVar T, U, V, W = TypeVar('T'), TypeVar('U'), TypeVar('V'), TypeVar('W') def concatenate(a: tuple[T, U], b: tuple[V, W]) -> tuple[T, U, V, W]: return a + b For the generic type transformation example, I'm not sure what you mean: from typing import Any, Callable Transformer = Callable[[dict[str, Any]], dict[Callable, Any]] This seems to match your question but it's really w…

Your tuple example only works if both tuples have two elements. I specifically mentioned arbitrary fixed-size tuples (as in, tuples with an arbitrary non-variable length). Your generic type transformation example also doesn't come close to what Typescript does. The resulting dict will not have known keys based on the keys of the input dict. In Typescript I can write a function that takes an object with known keys, an…

"Arbitrary fixed-size tuples" probably don't have a widely accepted meaning :) I read it as "size known when compiling the function".

The second example is cool. But I can't find a good practical use case for either example.

If you have a collection that's both heterogeneous and whose [size/key set] is statically known, when would it make sense to apply such generic transformations to them? This sounds like you have tuples or dataclasses where all elements have different meanings (since their types are fixed and different) _and_ you want to treat them like a generic collection _and_ you need the type checker to infer the result type.

The main use of tuples or dataclasses or `NamedTuples` is to pass or return values to/from functions without dealing with long lists of arguments. The elements aren't in the same category, it doesn't make sense to process them as one big collection, they mean different things.

(Also I think you made a mistake in your previous message, you wrote "an object with each key turned into a function" but it's the values that change types here.)

Re: Writing and linting Python at scale

#66

Post by npage97 »

insanitybit wrote: post #56
dazzlefruit wrote: post #46

Earlier quoted context omitted.

In VSCode (which uses PyRight/PyLance) and Python 3.10: JSONObject = None | str | int | bool | list["JSONObject"] | dict[str, "JSONObject"] # this type checks a: JSONObject = {"a": [1, 2, "7", True, {"false": None}]} # this doesn't type check b: JSONObject = {"a": [1, 2, "7", True, {"false": object()}]}

Cool, guess they finally fixed this. Must've been in the last ~1 year, give or take. Of course, it relies on quoting your types, which is... a matter of taste, I suppose.

You can also use from __future__ import annotations so the quotes become unnecessary. https://peps.python.org/pep-0563/

Re: Writing and linting Python at scale

#67

Post by theonething »

driggs wrote: post #8

It looks like the interesting feature of their tool Fixit 2 is that its lint rules know how to auto-apply themselves. I'm fine with an auto code formatter, an auto import organizer, but not sure how much I trust a linter to auto-apply "fixes".

Ruby's Rubocop linter has had this for a long time and it works great. I thought this would be the case for most of the mainstream languages.

Re: Writing and linting Python at scale

#68

Post by Izkata »

Timon3 wrote: post #63
Izkata wrote: post #62

Earlier quoted context omitted.

Tuples are already fixed size by nature, so adding a redundant "fixed-size" in that description was confusing. I also thought you meant a predefined size like always 2-tuples or always 3-tuples.

> Tuples are already fixed size by nature, so adding a redundant "fixed-size" in that description was confusing. No, you can type variable-length tuples in Python. A variable int tuple, for example, can be typed as Tuple[int, ...]. You can't concatenate two variable-length tuples, which makes sense - where would the cutoff be? But you should absolutely be able to concatenate two fixed-size tuples, and it's very limit…

> A variable int tuple, for example, can be typed as Tuple[int, ...].

That's a type that matches tuples of any length, not a variable-length tuple. The size of a tuple can't be changed. A variable-length tuple doesn't even really make sense, what you'd want there is a list.

> You can't concatenate two variable-length tuples, which makes sense - where would the cutoff be? But you should absolutely be able to concatenate two fixed-size tuples, and it's very limiting that you can't.

This whole statement doesn't make sense. I'm assuming you're still talking about type definitions and not actually tuples.

Re: Writing and linting Python at scale

#69

Post by dazzlefruit »

Timon3 wrote: post #63
Izkata wrote: post #62

Earlier quoted context omitted.

Tuples are already fixed size by nature, so adding a redundant "fixed-size" in that description was confusing. I also thought you meant a predefined size like always 2-tuples or always 3-tuples.

> Tuples are already fixed size by nature, so adding a redundant "fixed-size" in that description was confusing. No, you can type variable-length tuples in Python. A variable int tuple, for example, can be typed as Tuple[int, ...]. You can't concatenate two variable-length tuples, which makes sense - where would the cutoff be? But you should absolutely be able to concatenate two fixed-size tuples, and it's very limit…

You mean that the tuples' size is statically known at the calling site, while your message could be interpreted as the size being statically known in the callee.

I think this is clearer. The statement "arbitrary fixed-size tuples" sounded a bit like "an immutable mutable variable". It doesn't really say what's arbitrary about the tuples and in what context the size is fixed.

Re: Writing and linting Python at scale

#70

Post by Timon3 »

dazzlefruit wrote: post #65
Timon3 wrote: post #49

Earlier quoted context omitted.

Your tuple example only works if both tuples have two elements. I specifically mentioned arbitrary fixed-size tuples (as in, tuples with an arbitrary non-variable length). Your generic type transformation example also doesn't come close to what Typescript does. The resulting dict will not have known keys based on the keys of the input dict. In Typescript I can write a function that takes an object with known keys, an…

"Arbitrary fixed-size tuples" probably don't have a widely accepted meaning :) I read it as "size known when compiling the function". The second example is cool. But I can't find a good practical use case for either example. If you have a collection that's both heterogeneous and whose [size/key set] is statically known, when would it make sense to apply such generic transformations to them? This sounds like you have…

> "Arbitrary fixed-size tuples" probably don't have a widely accepted meaning :) I read it as "size known when compiling the function".

That is exactly what I'm talking about - the size of the tuples is known statically.

> The second example is cool. But I can't find a good practical use case for either example.

There are many interesting use cases in libraries, especially for some of the more esoteric features. Everything can be used for additional type safety.

> If you have a collection that's both heterogeneous and whose [size/key set] is statically known, when would it make sense to apply such generic transformations to them? This sounds like you have tuples or dataclasses where all elements have different meanings (since their types are fixed and different) _and_ you want to treat them like a generic collection _and_ you need the type checker to infer the result type.

Simple example - I have functions that return a Rust-like Result type, and I want to transform that into a different tuple-based format using a decorator. The transformation itself is static, but I can't write one function that handles it all, because Pythons type system is simply not developed enough. Something that would be incredibly easy in Typescript.

> The main use of tuples or dataclasses or `NamedTuples` is to pass or return values to/from functions without dealing with long lists of arguments. The elements aren't in the same category, it doesn't make sense to process them as one big collection, they mean different things.

But I have a use case for exactly this feature. Why should the language limit me? Why should I implement x functions that take different tuple lengths, with me having to choose the correct one for each use case, when I could write one function that does all?

> (Also I think you made a mistake in your previous message, you wrote "an object with each key turned into a function" but it's the values that change types here.)

Sure, though I could also literally write a Map that turns all object keys into functions.

Post reply on HN