Live data from Hacker News

Writing and linting Python at scale

engineering.fb.com

151–160 of 160 posts

Re: Writing and linting Python at scale

#151

Earlier quoted context omitted.

Surprise: all languages have types. Superior to TypeScript is neither a high bar, nor is this any kind of objective metric. I don't know why sum types are a blessing, also I don't know why pattern matching makes anything better. I can name a lot of problems with Python, and I'm sure that libraries isn't the only one. For example, for no reason, Python has multiple unrelated mechanisms to manage program state (object,…

Why is pathlib bad? Edit: I'm asking because pathlib is as good as a Python lib could be for me. Path manipulations are extremely clear and always safe. What more do you need?

It's broken just as os.path is. Python doesn't work well with file names in principle: it wants everything to be Unicode. That works for many, but if you want reliable code... you just have to throw all of that away.

Also, in case of pathlib, it adds no value on top of os.path of which it is a wrapper. Instead, it made the original library it wraps worse, because now os.path also needs to know about pathlib to be able to handle path fragments represented as pathlib instances.

All in all, it offers very little utility (a handful of shortcuts) vs increasing the size and memory footprint of "standard" library, complicating dispatch and therefore debugging... it's a bad trade.

Just not to get you confused. It's not an awful trade. It's not like the sky will fall down on you if you use it. It's just mostly worthless, with negligible downsides.

Re: Writing and linting Python at scale

#152

Earlier quoted context omitted.

Why is pathlib bad? Edit: I'm asking because pathlib is as good as a Python lib could be for me. Path manipulations are extremely clear and always safe. What more do you need?

It's broken just as os.path is. Python doesn't work well with file names in principle: it wants everything to be Unicode. That works for many, but if you want reliable code... you just have to throw all of that away. Also, in case of pathlib, it adds no value on top of os.path of which it is a wrapper. Instead, it made the original library it wraps worse, because now os.path also needs to know about pathlib to be abl…

> Python doesn't work well with file names in principle: it wants everything to be Unicode. That works for many, but if you want reliable code... you just have to throw all of that away.

Windows' APIs use UTF-16 and most file name encodings on Linux are UTF-8. How should Python handle this better?

> Also, in case of pathlib, it adds no value on top of os.path of which it is a wrapper.

Completely disagree. os.path is annoying to use. Treating paths as objects with methods and joining them with / makes my life much easier.

> increasing the size and memory footprint of "standard" library

By a ridiculous amount. pathlib is just another pure Python module with a bunch of simple functions and classes. [1]

> complicating dispatch and therefore debugging

You can simply declare and accept `Union[str, os.PathLike]` and convert the paths to whatever you want at the entrypoints, then use that in your own project. Where is the complexity? I've never seen this make debugging harder, it's just an additional type.

[1] https://github.com/python/cpython/blob/d9fc15222e96942e30ea8...

Re: Writing and linting Python at scale

#153

Earlier quoted context omitted.

Surprise: all languages have types. Superior to TypeScript is neither a high bar, nor is this any kind of objective metric. I don't know why sum types are a blessing, also I don't know why pattern matching makes anything better. I can name a lot of problems with Python, and I'm sure that libraries isn't the only one. For example, for no reason, Python has multiple unrelated mechanisms to manage program state (object,…

>I don't know why sum types are a blessing, also I don't know why pattern matching makes anything better. That's because you're inexperienced and haven't used them before. Try haskell or rust. This level of type safety actually reduces logical branching errors. And the key word is pattern matching. Googling isn't going to give you the insight here imo you need the experience (probably a couple months). If you don't p…

How much experience do you need and what kind? I don't have articles published in IEEE journals, but don't think you can get there in few months. But I can write simple proofs in eg. Coq or TLA+, so, I probably know a thing or two about types.

My work experience is measured in decades at this point. So, maybe you want to reflect on your ideas... it does take time to appreciate both the positive and the negative sides of any given type system. I don't think a few month will be enough, if you start from absolute blank slate. It's also silly to measure this in time, rather than effort. You probably never worked on complex problems, nor did you work on problems that require research, as opposite to copying from "best practices". This is where your conviction comes from, at least this is what it looks like.

Re: Writing and linting Python at scale

#154

Earlier quoted context omitted.

> dynamic scripting languages. Why keep repeating this nonsense? "Dynamic" or "scripting" aren't features of languages. When anyone says something like this, it's like talking about square chicken... (i.e. a category error). Obviously, you had some idea in your mind, and you wanted to communicate it somehow, but your readers will not know what it was unless you make an effort to analyze what you want to say and make…

> "Dynamic" or "scripting" aren't features of languages Surely dynamic typing is a language feature? I can't imagine what else someone would refer to with "dynamic".

> Surely dynamic typing is a language feature?

Languages are defined by their grammar. (or you can think about few other ways to define a language, s.a. a set of strings of some shape etc.) There's nothing static or dynamic about languages, just like there's nothing static or dynamic about integers or sausages.

In professional literature these words refer to the fact that some claims (or checks) about types of language expressions can be verified only at run time (when the actual value is known), or either at run time or prior to running the program. First is called "dynamic", second is "static".

Any programs can be checked prior to executing those programs. When there's an argument about "static" vs "dynamic", it is an argument about how useful static analysis can be. For example, in Unix Shell all types can be trivially inferred before executing a program: everything is a string, there aren't any other types. But this kind of analysis is also worthless because it doesn't help to check interesting properties of a program.

Non-professional users of this terminology tend to abuse it to mean something unrelated and inconsistent. You will meet a lot of claims that such and such language is "static" or "dynamic", even though, if you ask the person making that claim to explain what they actually mean by that, you'll discover that they don't really know what that is. Unfortunately, this "classification" is so common that a lot of people take it on faith, without even trying to examine it.

Here are some typical misuses of this terminology:

* If the language doesn't have type annotations, then it is a dynamic language. Of course, this isn't true because there are plenty of languages where type annotations are optional.

* If there isn't a compiler that performs static checks, then the language is "dynamic". Well, nobody so far wrote such a compiler, but that's not a proof it cannot exist. Also, of course, whether or not there is a compiler with w/e properties isn't relevant to the language itself.

* If the language has a mechanism to (automatically) reinterpret a value as belonging to unrelated types, then such language is "dynamic". Unfortunately, this makes virtually every useful language a "dynamic" language, which makes the distinction worthless.

You can probably think about other cases where this attempt at taxonomy fails.

----

To sum it up.

"Dynamic typing" is kind of like "West India" -- a result of confusion, a misunderstanding of the person using the term. The closest thing is "dynamic type checking". In turn, "dynamic type checking" isn't a property of a language, it's a phase of program analysis, specifically, when all values in that program have known types. Every program in any language can be analyzed at run time, which makes claims about some language being more "dynamic" than others nonsense.

Re: Writing and linting Python at scale

#155

Earlier quoted context omitted.

It's worked for many years, but you won't often see it used outside of class definitions because all of the other tools struggle with it (Pylint, Flake8, Pylance, etc. spit out some variation of an undefined variable error).

I don't think it has worked for years. I had used that same exact __annotations__ and my recollection was that recursive types still did not work.

It's worked for years. I've been using it the whole time.

Re: Writing and linting Python at scale

#156

Earlier quoted context omitted.

It's broken just as os.path is. Python doesn't work well with file names in principle: it wants everything to be Unicode. That works for many, but if you want reliable code... you just have to throw all of that away. Also, in case of pathlib, it adds no value on top of os.path of which it is a wrapper. Instead, it made the original library it wraps worse, because now os.path also needs to know about pathlib to be abl…

> Python doesn't work well with file names in principle: it wants everything to be Unicode. That works for many, but if you want reliable code... you just have to throw all of that away. Windows' APIs use UTF-16 and most file name encodings on Linux are UTF-8. How should Python handle this better? > Also, in case of pathlib, it adds no value on top of os.path of which it is a wrapper. Completely disagree. os.path is…

> most file name encodings

You just repeated what I said. Just read it again. You already know the answer, you just need to understand it.

> Completely disagree.

And make no worthwhile argument.

> You can simply declare and accept `Union[str, os.PathLike]`

What does this have to do with debugging? Do you know what "debugging" means?

Re: Writing and linting Python at scale

#157

Earlier quoted context omitted.

I've been writing in Python for over ten years, in different roles, for wildly different projects (research, infra, Web, testing, education). I'm yet to find anything Python was good for. On engineering merits alone Python isn't best for anything, nor is it best for combinations of things. It's silly to think that any tool that works with Python does so because Python was the best language for the job, and they only…

Python is always the second-best language for the job. Which makes it a great language to know.

Your claim is covered by this part of the post you replied to:

> nor is it best for combinations of things.

In other words, you are wrong. No, Python isn't second best, nor is it anywhere in the top-ten. It's worthless as a language, as in it's not worth paying attention to if you are interested in how you can solve programming problems by making better languages. It's worth knowing Python if you want a programming job, and this is where it shines. Anything else about it is either "meh" or just hands down awful.

Re: Writing and linting Python at scale

#158
post #25

Earlier quoted context omitted.

I've been writing in Python for over ten years, in different roles, for wildly different projects (research, infra, Web, testing, education). I'm yet to find anything Python was good for. On engineering merits alone Python isn't best for anything, nor is it best for combinations of things. It's silly to think that any tool that works with Python does so because Python was the best language for the job, and they only…

I’m not really a fan of Python as such, but after a few decades in the industry, I’m beginning to think that being good at being bandaid is “better”. I can’t think of a single tech where we don’t have a bunch of duct tape (as we refer to it), not so much because we want to but because that’s just how things end up in the imperfect world of organisations. I value the techs that fit into this reality more than the ones…

Your mistake is going from quantitative claims to categorical without evaluating the quantitative part first.

You say "everything is a band aid to some degree", and from that you conclude that "everything is equally bad or good". You conveniently forget the "to some degree" part to further your point.

It's similar to saying that all food has some amount of dust in it, so it shouldn't matter whether you just open the vacuum cleaner and eat the stuff collected on the drum, or if you order a meal at a fancy restaurant.

There's no evidence that Python "fits into reality" more than any other language. The thing that's going on for it is popularity. Popularity doesn't need to be rooted in technical merits, and in the case of Python it isn't. Finally, Python isn't unique in this sense. Even though the stage for programming language popularity pageant was set relatively recently, the participants learned to abuse the rules of the competition very quickly. There's a "rule" by a statistician whose name I cannot recall at this point which states that once people know the metric they are measured on, they will learn to game it. When we assess language popularity, we most assess how well the language authors or their community was able to game the metric rather than measuring any meaningful aspect of those languages.

Re: Writing and linting Python at scale

#159

Earlier quoted context omitted.

> increases productivity to not have to double check each and every instance of violating a lint error, At what cosmic speed should you be pumping out code for this to be a concern? Also, in C#/.NET, where programmers predominantly use MSVS, which is an atrocious editor with MSBuild, which is an atrocious build system, both hampering productivity... Also, plenty of linter errors are actual errors that need non-trivia…

> Also, plenty of linter errors are actual errors that need non-trivial fixing. These sound like they're not lints at all.

No, not really. It is perfectly possible to write unintentionally valid code, which would be caught by linter due to the author not following some convention.

Trivial and popular example of such code is assignment instead of comparison in the context of condition. Some languages allow for this to happen, but it's known as a possible mistake and so the conventions in such languages would discourage the use of assignment in the context of condition, even though it technically produces valid code.

Re: Writing and linting Python at scale

#160
post #6

I'm happy with Ruff[0], it's very fast. [0] -- https://github.com/astral-sh/ruff

Unfortunately ruff is very inconsistent and has lots of differences from the flake8 plugins it tries to emulate. Lots of rules are confused by irrelevant context so that it can miss lots of things it should find when the equivalent flake8 plugin still find them. It's automatic fixing of issues will happily introduce other issues that it doesn't find until the next run. I've tried pretty hard to use it and gave up, it…

[flagged]
Post reply on HN