Live data from Hacker News

Python's “disappointing” superpowers

lukeplant.me.uk

181–190 of 264 posts

Re: Python's “disappointing” superpowers

#181
post #172

Earlier quoted context omitted.

This is a false equivalence. You need all the help you can get maintaining large codebases and static typing is a huge help.

What did I make an equivalence between? Large codebases are hard, period. That is all I am saying. Having worked on 100k LOC systems in static and in dynamic, I will not claim either has a benefit. That is just hard. Static analysis is, of course, good. If that is static typing or otherwise. So is running the code making sure it does what you want when running. What is not a huge help, is a byzantine type hierarchy t…

Large codebases tend to be harder than smaller ones within one language. But language to language, static typing is such a significant help that it makes a large difference.

I’ve been writing in python for 18 years, and yet I can far more quickly get into a large typescript codebase than a python one of equivalent size.

Re: Python's “disappointing” superpowers

#182
post #172

Earlier quoted context omitted.

This is a false equivalence. You need all the help you can get maintaining large codebases and static typing is a huge help.

What did I make an equivalence between? Large codebases are hard, period. That is all I am saying. Having worked on 100k LOC systems in static and in dynamic, I will not claim either has a benefit. That is just hard. Static analysis is, of course, good. If that is static typing or otherwise. So is running the code making sure it does what you want when running. What is not a huge help, is a byzantine type hierarchy t…

"I will not claim either has benefit"

I bet that if you did a survey of developers who have worked on projects of this scale, 90% would disagree with your opinion.

Re: Python's “disappointing” superpowers

#183

Earlier quoted context omitted.

A lot of the orm style things are not really feasible in static-er languages without clunky approaches like codegen.

What part specifically? Java is (in)famous for its orms after all. Unless you want to count that as clunky codegen? I'd argue that all metaprogramming boils down to that though.

Almost all Java ORMs are dynamically typed in the ways that matter. The "clunky codegen" method would be something like JOOQ, that actually checks your schema.

Re: Python's “disappointing” superpowers

#184
post #172

Earlier quoted context omitted.

What did I make an equivalence between? Large codebases are hard, period. That is all I am saying. Having worked on 100k LOC systems in static and in dynamic, I will not claim either has a benefit. That is just hard. Static analysis is, of course, good. If that is static typing or otherwise. So is running the code making sure it does what you want when running. What is not a huge help, is a byzantine type hierarchy t…

Large codebases tend to be harder than smaller ones within one language. But language to language, static typing is such a significant help that it makes a large difference. I’ve been writing in python for 18 years, and yet I can far more quickly get into a large typescript codebase than a python one of equivalent size.

Amusingly, I hate python.

Static typing is a tool I like. I don't think we have evidence that it is an obvious win. And I've bounced off of several typescript code bases hard. To the point that I currently hate typescript. Despite being impressed with some of its capabilities.

Which is again to say ymmv. Bad code is bad with or without static types.

Re: Python's “disappointing” superpowers

#185
post #182
post #172

Earlier quoted context omitted.

What did I make an equivalence between? Large codebases are hard, period. That is all I am saying. Having worked on 100k LOC systems in static and in dynamic, I will not claim either has a benefit. That is just hard. Static analysis is, of course, good. If that is static typing or otherwise. So is running the code making sure it does what you want when running. What is not a huge help, is a byzantine type hierarchy t…

"I will not claim either has benefit" I bet that if you did a survey of developers who have worked on projects of this scale, 90% would disagree with your opinion.

Then do said survey? Find empirical evidence and share it.

My experience, sadly, is the louder a dev on the team is about either dynamic or static typing, the more likely that dev's code is something nobody else in the team wants to work with.

Re: Python's “disappointing” superpowers

#186
post #151

Earlier quoted context omitted.

This is nearly impossible to precisely study: - Because correlation != causation, for one thing: 2.5x the number of bugs in static "style" code bases (not sure what that means) does not mean that static type systems lead to more bugs. It could be that users of dynamically typed languages find themselves needing robust automated tests more often, because they can't rely on static typings. This would suggest that in a…

[flagged]

> Yeah but typing related bugs which is what static type checkers catch only account for 3% of bugs.

You're stating this as if it's fact and also universal, but I strongly doubt it. Even defining "type-related bug" is hard, because there's a ton of bugs that are not type-related but are much easier to avoid if you're using a good static type checker. For example, stringly-typed values, or switch statement exhaustiveness bugs. Some type checkers, including TypeScript, can eliminate almost all null reference errors when used with strict settings. Then to top it off, types add a bunch of static analysis capabilities you could not otherwise have: for example, you can find accidental dead code because you have an impossible condition that can be proven impossible by types, or more exactly find incorrect usages of library functions, and so forth. There's quite a range of problems and almost none of them are explicitly related to strong typing.

And let's say somehow, this is a universally correct number. That doesn't mean it's the reality faced in front of you. What if you already know the majority of your bugs, as logged today, could be prevented by a type checker?

> You are always going to get more bang for buck by simply writing more unit tests.

Good test coverage is a huge, non-trivial investment for any reasonably large, reasonably complicated codebase. Making all of your code testable in fact impacts how you write code, sometimes in ways that are actually similar to the differences you might make to account for static type systems too.

That said, you should do it! You should do both. Tests catch bugs that type checkers can't, but type checkers offer more than just that, and they catch them faster; typically before you hit save these days.

> Static typing is done primarily for performance and that requires a statically typed language.

I dunno why you're repeating this, people are objectively using static typing for other things. That makes your statement wrong on it's face, because what you're saying is not a matter of opinion, you're suggesting it's "primarily done for performance" and it just simply isn't. Given that TypeScript is one of the most popular programming languages right now, I'd argue this "primarily" categorization is just wrong.

But I think it goes deeper. Like, why does C do static typing? It does not actually need to much. In fact, in an early version of C, the type system was significantly weaker. Struct fields were global: you could access any struct field on any pointer. This is convenient and it doesn't disallow any correct code, but alas nobody is wishing for that to make a return.

I won't even bother getting into functional programming languages, but they exploit types far harder than any of the languages we've been discussing; it's just simply nothing to do with performance.

Maybe C has been trending in the "safer" direction for performance? No. Stakeholders have clearly been improving types specifically for the purpose of preventing classes of bugs observed in the real world. Literally just recently, a great article dropped about array types in Linux, for example:

https://people.kernel.org/kees/bounded-flexible-arrays-in-c

C++ and Rust use types for a lot of things. Some of them are performance, but it's not that simple either. Rust lifetimes are definitely about fast correctness, but it absolutely enforced "correctness" that you would not get with a global interpreter lock, because the principles of never having multiple mutable aliases simply makes sense given the machine model we have today.

So then TypeScript. Let's assume in aggregate it somehow prevents absolutely no bugs despite the obvious fact that it prevents entire classes of bugs that are common in idiomatic JS. Well, guess what? It still offers a ton. For example, it makes refactoring significantly easier. It's not even a discussion, refactoring old code is much easier when it's statically typed, even if it's ossified and weird and full of Chesterton fences. It's night and day. Typescript also offers excellent code intelligence: very accurate autocomplete, inline documentation, automated refactoring options, and yes, more static analysis options than just type checking; being able to statically analyze the code deeper opens up an explosion of linters and checkers for all sorts of things; checking for improper usage of React hooks, or common lodash mistakes.

> Hacking it into a scripting language is plain looking for trouble.

On the contrary, I wouldn't catch myself writing JS without TypeScript anymore. I consider the investment very small compared to the benefit.

In the real world, I often do not get the chance to start new projects from scratch, but instead wind up walking into existing messes. The last JS project I walked into wound up having thousands of errors per day that could've been prevented by static type checking, caused by hundreds of different bugs. And that's just what we could observe by adding metrics: there were still more bugs that were uncommon enough to not be caught initially. When you're dealing with an old ossified codebase like that, you sometimes wind up with more bugs going in than out a lot of the time. The only reliable way I've seen to reverse that trend is by incrementally adding more static analysis. Hope, after all, is not a strategy.

I don't believe there is an objective truth here, but I have strong doubts about the points presented here. The argument for static type checking is strong: it simply prevents classes of bugs, end of story. That's a very simple story and does not require any gymnastics to explain. The argument against static type checking feels very hand-wave-y at best. The fact that we keep landing on this weird point about how it is "primarily" used for performance especially, since I don't think that's true. I think you are mistaking the fact that dynamically typed languages are slow due to the inability of the compiler to optimize for statically typed languages choosing to be static simply for performance. I don't think it's even true for C.

Re: Python's “disappointing” superpowers

#187
post #174

Earlier quoted context omitted.

> JavaScript goes one step further though, and has no distinction between get-item accesses and attribute accesses. Can you explain a bit more about what you mean here? > Is it even possible to implement your own `MutableMapping` or `dataclasses` equivalent type in Python? Of course. With dataclasses that's what Python itself does; that's a pure Python module: https://github.com/python/cpython/blob/3.11/Lib/dataclass…

There is no difference between `object["hello"]` and `object.hello`. So in Typescript, all you need is `interface`, that's it. Everything (except primitives) is just an `interface` with keys or index signatures. > Of course. With dataclasses that's what Python itself does; that's a pure Python module: Sorry, I meant that you can't create an equivalent class that typechecks the same way. For example, when you define a…

> There is no difference between `object["hello"]` and `object.hello`.

Ah, ok. Technically I suppose one could unify these in Python; attribute accesses on objects are really dict lookups under the hood, and one could implement a list as a dict whose keys were restricted to be integers. But yes, Python chose not to go that route.

> I meant that you can't create an equivalent class that typechecks the same way.

Well, of course not. If you derive your own class direct from object, even if it's duck type compatible with some other class, it is not a subclass of that other class. That's just how Python's type system works.

> If you just alias `dataclasses`, you lose type support!

You mean you lose some functionality with third party type checking tools. Yes, that's true. (Although that particular limitation, at least, seems odd to me--see below.)

> The language simply can't express it.

No, the language does not have built-in tools that do the things you would like them to do with type checks and type annotations. And the third-party tools apparently don't cover these cases.

The "alias dataclass" case seems odd to me because a simple "is" check ("dataclass_alias is dataclass") should cover it--aliasing the object just means putting a reference to the same object in another namespace, and the "is" operator checks for object identity, not the name it has in any particular namespace.

The "dataclass_wrapper" case would be a good bit more complicated to check for, yes. Although type annotations ought to be able to help, since you can express "this function takes a class as an argument and returns a dataclass derived from that class" in Python type annotations.

Re: Python's “disappointing” superpowers

#188
post #185
post #182

Earlier quoted context omitted.

"I will not claim either has benefit" I bet that if you did a survey of developers who have worked on projects of this scale, 90% would disagree with your opinion.

Then do said survey? Find empirical evidence and share it. My experience, sadly, is the louder a dev on the team is about either dynamic or static typing, the more likely that dev's code is something nobody else in the team wants to work with.

Here is one sample point for your survey.

Static is better than dynamic for large projects, in my experience.

Re: Python's “disappointing” superpowers

#189

Earlier quoted context omitted.

> has on average 2.5x the number of bugs This claim is not supported by the linked article. In fact, the main claim (only 2% of bugs are type errors) of the linked article also does not make any sense. It is based on the assumption, that typing errors only ever cause TypeError, AttributeError, or NameError in Python, which is, ironically, false, because Python is a dynamic language. To give a concrete example, I went…

That not even an error, it's an usability improvement.

Firstly, usability bugs like this are still bugs.

Secondly, think how that issue surfaced in the first place. Somebody used TensorFlow in their Python code, and their Python code produced unexpected result and/or crashed. Judging by the fix, it is highly unlikely to have produced one of the errors from the above list. As a ML practitioner I can also tell you it likely means somebody had to spend a significant amount of time to get to the cause once they saw the output, because numerical miscalculations are very hard to debug in the first place.

Re: Python's “disappointing” superpowers

#190
post #158

I used to be a huge dynamic languages fan, with python being my favorite language over the majority of my career. Then I worked on a large python project of ~100k LOC with a team of ten. That's when I realized that writing code faster isn't the problem. Reading it is, making changes to code someone else wrote is, and refactoring across dozens of modules is a problem. Static languages help a lot with all three. I stil…

This is almost always framed in a way that shows large systems need the extra rigidity of some static systems. And that dynamic systems are only about rapid prototyping. I think that is a bit of a false framing. To wit, any system of 100k LOC will be hard to get into. Even harder to make changes in. There is no getting around that. None. Even worse if you have many entities flying about these 100k LOC. Each change to…

> any system of 100k LOC will be hard to get into

To get a gut feeling of the system, i.e., to understand the 10000 foot view of a 100k LOC system, is orders of magnitude easier with statically typed languages than dynamic languages.

And that is mostly because those 100k LOC are done by probably dozens of developers and that leads to different styles, approaches, conventions, etc.

A statically typed language would remove a lot of those headaches, because it does force a semblance of structure, that can be easy to reason with.

A language like Python is awesome for a micro-services architecture though.

Post reply on HN