Writing and linting Python at scale • Hacker News Board

Re: Writing and linting Python at scale

#31

Post by crabbone » Wed, Nov 22, 2023, 4:46 PM UTC

corethree wrote: post #21

lagt_t wrote: post #5

But but but HN told me that python is only good for small scale and prototypes, the engineers at meta are wrong

Python not only has types but it's type system is superior to typeScript. Get this python has sum types and exhaustive pattern matching exactly like rust or haskell. The only problem with python are the libraries are sometimes written with tricks that make static typing ineffective. Other than that it is really really good at scale. Better API then typescript imo which is really it's main competitor. edit: (rate limi…

Surprise: all languages have types.

Superior to TypeScript is neither a high bar, nor is this any kind of objective metric. I don't know why sum types are a blessing, also I don't know why pattern matching makes anything better.

I can name a lot of problems with Python, and I'm sure that libraries isn't the only one. For example, for no reason, Python has multiple unrelated mechanisms to manage program state (object, clojures, context managers). Or, here's another one: it has multiple public API for dealing with filesystem (i.e. os.path and pathlib), both of which are all sorts of bad, but just the fact that it has two, where one would do is bad.

It's not possible to make static typing ineffective. Static typing is a given, it happens in the virtue of you having a program. You may argue that it doesn't exist if unobserved in the same way how a tree that falls in the forest doesn't really fall if unnoticed. But, really, it's there, and it's there in every language. You cannot make it ineffective. It's like making centimeters ineffective -- I would struggle to imagine what that may possibly mean.

Re: Writing and linting Python at scale

#32

Post by myvoiceismypass » Wed, Nov 22, 2023, 4:47 PM UTC

crabbone wrote: post #24

jerf wrote: post #14

Earlier quoted context omitted.

I mean, as far as I'm concerned, everything Facebook has done has done nothing but reconfirm how unwise it is to build large-scale infrastructure on dynamic scripting languages. They have the resources to move heaven and earth to do what amounts to turning their dynamic scripting language back into static languages in everything but name, and they have the hole they've dug themselves into that justifies it. I have ne…

> dynamic scripting languages. Why keep repeating this nonsense? "Dynamic" or "scripting" aren't features of languages. When anyone says something like this, it's like talking about square chicken... (i.e. a category error). Obviously, you had some idea in your mind, and you wanted to communicate it somehow, but your readers will not know what it was unless you make an effort to analyze what you want to say and make…

Dynamic Typing is absolutely a feature of languages. Static Typing is, as well.

I was not confused by using these terms. I cannot imagine many people on HN were either.

Re: Writing and linting Python at scale

#33

Post by corethree » Wed, Nov 22, 2023, 4:48 PM UTC

seabrookmx wrote: post #22

corethree wrote: post #21

Earlier quoted context omitted.

Python not only has types but it's type system is superior to typeScript. Get this python has sum types and exhaustive pattern matching exactly like rust or haskell. The only problem with python are the libraries are sometimes written with tricks that make static typing ineffective. Other than that it is really really good at scale. Better API then typescript imo which is really it's main competitor. edit: (rate limi…

Yes but the if you want to do ahead-of-time type checking you need to run a tool like mypy, and none of those tools are as comprehensive or performant as Typescript. Also the ecosystem of libraries with type annotations is much smaller (Typescript has the first mover advantage, after all).

I literally said it supports exhaustive pattern matching which adds a level of safety and flexibility superior to that of type script.

Many IDEs have real time type checking that highlights the errors so don't even have to run the external checker.

Even if you don't use IDEs running the type checker is measured in seconds. Not far off from linters that most people will also use for TS.

What you say about the libraries is true though. But you can always place a type "shell" around those libraries such that your code is type safe.

The other main problem with the libraries in python is that a lot of people who use python are data scientists who haven't figured out why types are so great. Those guys are the main ones holding the libraries back.

Re: Writing and linting Python at scale

#34

Post by Timon3 » Wed, Nov 22, 2023, 4:51 PM UTC

corethree wrote: post #21

lagt_t wrote: post #5

But but but HN told me that python is only good for small scale and prototypes, the engineers at meta are wrong

Python not only has types but it's type system is superior to typeScript. Get this python has sum types and exhaustive pattern matching exactly like rust or haskell. The only problem with python are the libraries are sometimes written with tricks that make static typing ineffective. Other than that it is really really good at scale. Better API then typescript imo which is really it's main competitor. edit: (rate limi…

> Python not only has types but it's type system is superior to typeScript.

I strongly disagree. It might be better at some things, but it's much worse at others. Many functions can't be accurately typed (try, for example, to make a well-typed function that concatenates two arbitrary fixed-size tuples), and as far as I know generic type transformations can't be implemented (random example "this type takes a Dict[str, Any] and returns an object with each key turned into a function").

Re: Writing and linting Python at scale

#35

Post by lijok » Wed, Nov 22, 2023, 4:52 PM UTC

jerf wrote: post #26

crabbone wrote: post #24

Earlier quoted context omitted.

> dynamic scripting languages. Why keep repeating this nonsense? "Dynamic" or "scripting" aren't features of languages. When anyone says something like this, it's like talking about square chicken... (i.e. a category error). Obviously, you had some idea in your mind, and you wanted to communicate it somehow, but your readers will not know what it was unless you make an effort to analyze what you want to say and make…

Because it a common term used for a category of languages that everybody understands. If you have a problem with it, you can take it up with the aforementioned everybody. Personally I think it gets perilously close to the error that if you argue definitions enough you can change reality. It doesn't matter what labels we slap on clearly related languages Python/Perl/Javascript/Lua/PHP/Javascript, I consider it a mista…

> I consider it a mistake to build a large system with them in 2023

A mistake in what sense? What are the risks that you see with using these languages for large scale systems, and how do those risks balance around all the other risks that come with not using one of these languages and using a different one?

Re: Writing and linting Python at scale

#36

Post by atoav » Wed, Nov 22, 2023, 4:54 PM UTC

crabbone wrote: post #23

lagt_t wrote: post #5

But but but HN told me that python is only good for small scale and prototypes, the engineers at meta are wrong

I've been writing in Python for over ten years, in different roles, for wildly different projects (research, infra, Web, testing, education). I'm yet to find anything Python was good for. On engineering merits alone Python isn't best for anything, nor is it best for combinations of things. It's silly to think that any tool that works with Python does so because Python was the best language for the job, and they only…

This very subjective, but I think python is good at syntax. If I had to teach non-programmers programming it would be easily my number one choice. The included libraries are also pretty good.

Re: Writing and linting Python at scale

#37

Post by corethree » Wed, Nov 22, 2023, 4:59 PM UTC

crabbone wrote: post #31

corethree wrote: post #21

Earlier quoted context omitted.

Python not only has types but it's type system is superior to typeScript. Get this python has sum types and exhaustive pattern matching exactly like rust or haskell. The only problem with python are the libraries are sometimes written with tricks that make static typing ineffective. Other than that it is really really good at scale. Better API then typescript imo which is really it's main competitor. edit: (rate limi…

Surprise: all languages have types. Superior to TypeScript is neither a high bar, nor is this any kind of objective metric. I don't know why sum types are a blessing, also I don't know why pattern matching makes anything better. I can name a lot of problems with Python, and I'm sure that libraries isn't the only one. For example, for no reason, Python has multiple unrelated mechanisms to manage program state (object,…

>I don't know why sum types are a blessing, also I don't know why pattern matching makes anything better.

That's because you're inexperienced and haven't used them before. Try haskell or rust. This level of type safety actually reduces logical branching errors. And the key word is pattern matching.

Googling isn't going to give you the insight here imo you need the experience (probably a couple months). If you don't plan on getting it I suggest you ask a haskell developer or rust developer about why exhaustive pattern matching is such a great feature.

> Python has multiple unrelated mechanisms to manage program state

This is like a slightly bad for people who have ocd towards library organization. Two ways to do the same thing exists everywhere. Do you use looping or tco recursion? Why do some languages support both? Doesn't matter that much. This doesn't make a language horrible just makes it a bit bloated. Much worse is stuff like javascripts undefined value. Also TS has tons and tons of libraries that do the same thing. Why isn't that setting off your "bad" red flag instincts? Is it because they aren't in the std? So redundant commands outside of the std are ok but within the std.. bad bad bad? Have you actually hit a real problem related with this or is it just something that feels bad because of ocd?

>It's not possible to make static typing ineffective.

Categorically false. Python does have patterns and tricks which static type checkers can't catch. You struggle with meaning here because you failed to comprehend what I wrote and you're now arguing against a misinterpretation of my statements. Reread that part again, you definitely misunderstood.

Re: Writing and linting Python at scale

#38

Post by charliermarsh » Wed, Nov 22, 2023, 5:01 PM UTC

ltbarcly3 wrote: post #15

wdroz wrote: post #6

I'm happy with Ruff[0], it's very fast. [0] -- https://github.com/astral-sh/ruff

Unfortunately ruff is very inconsistent and has lots of differences from the flake8 plugins it tries to emulate. Lots of rules are confused by irrelevant context so that it can miss lots of things it should find when the equivalent flake8 plugin still find them. It's automatic fixing of issues will happily introduce other issues that it doesn't find until the next run. I've tried pretty hard to use it and gave up, it…

If you notice differences vs. the originating plugins, would definitely appreciate it if you could file an issue! We tend to be very responsive especially when it comes to matters of correctness.

Candidly, for Flake8 plugins, my experience is that Ruff tends to be more consistent, more robust in its inference and, at this point, more extensively tested than the original implementations -- both via our own testing and via the significant number of projects and companies that now use Ruff in production. (As compared to Pylint, though, we catch fewer issues, since Pylint does some type inference across files, which Ruff doesn't support yet.)

Ruff is also designed such that we will iteratively lint-and-fix until there are no more fixable issues, so if you've seen the linter introduce _new_ fixable diagnostics, that would be a bug too.

Regardless, thanks for giving it a try :)

Re: Writing and linting Python at scale

#39

Post by gostsamo » Wed, Nov 22, 2023, 6:02 PM UTC

chefandy wrote: post #17

jpc0 wrote: post #12

Earlier quoted context omitted.

https://engineering.fb.com/2023/08/07/developer-tools/fixit-... Just a test, for some reason your link isn't clickable for me.

Looks like they unintentionally (or intentionally for some reason I can't parse ATM?) triggered code formatting. If you have a line preceeded by two or more blank lines that is indented with two or more spaces then it'll formatting. Like This

unintentionally, sorry for that one.

Re: Writing and linting Python at scale

#40

Post by jerf » Wed, Nov 22, 2023, 6:20 PM UTC

lijok wrote: post #35

jerf wrote: post #26

Earlier quoted context omitted.

Because it a common term used for a category of languages that everybody understands. If you have a problem with it, you can take it up with the aforementioned everybody. Personally I think it gets perilously close to the error that if you argue definitions enough you can change reality. It doesn't matter what labels we slap on clearly related languages Python/Perl/Javascript/Lua/PHP/Javascript, I consider it a mista…

> I consider it a mistake to build a large system with them in 2023 A mistake in what sense? What are the risks that you see with using these languages for large scale systems, and how do those risks balance around all the other risks that come with not using one of these languages and using a different one?

I think I pretty much laid out the bulk of the argument. Facebook is moving heaven and earth to mitigate their choice of a dynamic language, even if it was the right choice at the time. You can go look at everything they've done, and everything Typescript has done, and all the gradual typing initiatives. I left plenty of threads to pull there.