Tests aren’t enough: Case study after adding type hints to urllib3 • Hacker News Board

Re: Tests aren’t enough: Case study after adding type hints to urllib3

#151

Post by acdha » Mon, Oct 18, 2021, 11:30 PM UTC

Jensson wrote: post #108

tetha wrote: post #97

Earlier quoted context omitted.

Hm, growing somewhat experienced, I find myself adapating an old quote more and more: Sufficiently advanced static typing is indistinguishable from dynamic typing. Now, I know, it's not true. It's entirely possible to build weird things in python that are provably impossible to typecheck statically. But modern language servers and their type inference capabilities in rust, terraform, or even straight up python are ve…

> Sufficiently advanced static typing is indistinguishable from dynamic typing. Static typing type checks are compile time, dynamic typing doesn't. I don't see how these two could be indistinguishable, in one you can't run the program with type errors, in the other you can.

This is true if you're focused on the compiler but if you include the adjacent question of what is reported to you in your editor while working it's a bit blurrier. If I write some Rust code and pass a string into something which expects an integer, I get a hard error preventing compilation. If I do the same thing in Python, however, and there's a prominent error displayed in my editor before I even run the code, how different is that from the perspective of anyone who isn't watching my screen? In either case the bug was caught before the code even ran and I likely have type-aware autocompletion to reduce the odds of making that mistake in the first place.

That's not to say that there aren't quite reasonable questions about how effective the different approaches are or how easy it is to fix the error, how complete the checks are, how deep the checks go, etc. A lot of how you feel about that is going to be subjective based on the languages you use and the quality of the code you work with — Rust has an advanced type system and great developer ergonomics providing unusually helpful error messages, Python has weaker typing but also a culture about simplicity which discourages some classes of bugs, Java has a lot of mushy-typed code where people got tired of language / compiler drawbacks and came up with ways to improve ergonomics at the expense of defeating the type checker, etc.

Re: Tests aren’t enough: Case study after adding type hints to urllib3

#152

Post by skrimp » Mon, Oct 18, 2021, 11:43 PM UTC

slaymaker1907 wrote: post #113

tetha wrote: post #97

Earlier quoted context omitted.

Hm, growing somewhat experienced, I find myself adapating an old quote more and more: Sufficiently advanced static typing is indistinguishable from dynamic typing. Now, I know, it's not true. It's entirely possible to build weird things in python that are provably impossible to typecheck statically. But modern language servers and their type inference capabilities in rust, terraform, or even straight up python are ve…

Please write a type for the following function: def compose(start, *args): def helper(x): for func in reversed(args): x = func(x) return start(x) return helper There is no mainstream typed language which can write a fully general type for the vararg compose function. TypeScript is probably the one that comes closest, but last I checked it still was unable to write a sufficiently powerful array type. You can write a t…

C++ is an extremely mainstream language that can write a fully general version of compose with variable arguments.

https://godbolt.org/z/h7n8Y7qf1

Like sure, you can't write out a type for the entire overload set. Overload sets don't have types, but functions do. However, I don't think you'd ever actually want to write out the type of the compose function. Instead, I think it would be more reasonable to request that every intermediate function call is type-checked with fully specified types. In C++ this is the case.

Re: Tests aren’t enough: Case study after adding type hints to urllib3

#153

Post by habitue » Mon, Oct 18, 2021, 11:55 PM UTC

pansa2 wrote: post #130

adrianmonk wrote: post #65

Earlier quoted context omitted.

Consider these 4 possible combinations for programming languages: (1) Low-level, static types (2) Low-level, dynamic types (3) High-level, static types (4) High-level, dynamic types For whatever reason, historically #1 and #4 have been most popular. C, C++, Pascal, Ada, and Java are #1. Python, JavaScript, Perl, and BASIC are #4. There haven't been a lot of #2 or #3 languages. Some #3 languages (TypeScript and Python…

Exactly. IMO we still don't have a good #3 language - in particular, all of the popular languages that can be compiled into native code are in category #1.

I'd say rust and swift are in the #3 category. Really, C++ is pretty damn high level, just not a sound type system. But it's static

Re: Tests aren’t enough: Case study after adding type hints to urllib3

#154

Post by tshaddox » Tue, Oct 19, 2021, 12:10 AM UTC

wvenable wrote: post #34

david422 wrote: post #19

I love static typing/type hints if for only 1 thing - code maintenance. Even code I wrote six months ago. Not having to dig through 6 functions deep to try to figure out whether "person" is a string, or an object, and if it's an object what attributes it has on it etc. is huge. And not to mention that some clever people decide - hey, if you pass a string I'll look up the person object - so you can pass an object or a…

The main argument for dynamic typing is speed in prototyping but I find that's opposite for me. I'm much more comfortable rapid prototyping and ripping stuff apart when I have a strongly static typed environment telling me what I just broke. Doing radical refactoring often involves just making those changes and then fixing all the IDE or compiler errors until it runs again.

I think the only reason dynamic typing can speed up prototyping is that it allows you to make certain type errors that you may never encounter at runtime while prototyping.

Re: Tests aren’t enough: Case study after adding type hints to urllib3

#155

Post by taeric » Tue, Oct 19, 2021, 12:19 AM UTC

munificent wrote: post #136

taeric wrote: post #74

Earlier quoted context omitted.

Quite the contrary, in my opinion. Lots of what makes various parts of the physical hard are in the infrastructure surrounding them. This is very similar to the complexity in software setups. Take game systems, as an example; far far more effort will be spent in the art and general asset management than is true for many business software setups. Which is why many of the business best practices haven't necessarily mov…

> Take game systems, as an example; far far more effort will be spent in the art and general asset management than is true for many business software setups. Which is why many of the business best practices haven't necessarily moved over to games. You're right that game development involves a lot of asset stuff that other business software doesn't have to worry about as much. (And, conversely, a lot of business softw…

I can't say with any authority on why practices are different between the different environments. So, to that end, I should have offered it as /a/ reason, as I don't think it is the sole one. I can't shake that it contributes, though. I mainly meant it is a counter to the implicit "devs at office job are too lazy to learn different ways."

For other examples, I would dip into major logistical simulations/optimizations. Which basically drop into linear algebra as soon as they can, where much of the idea of typing is basically thrown out, so that we can solve equations and constraints, with no real tracking of which is which at different locations. There is a tranlation in/out, but once in, things are effectively "matrix of values.

(As an amusing aside, I love that I get to message with the authors of books I'm reading on places like this. I have your Crafting Interpreters. Working through it at a glacial pace. Plan to pick up the other one next. Kudos and thanks!)

Re: Tests aren’t enough: Case study after adding type hints to urllib3

#156

Post by kkirsche » Tue, Oct 19, 2021, 12:26 AM UTC

richard_todd wrote: post #140

I think it's interesting that PEP 484 says ([1]): "the authors have no desire to ever make type hints mandatory, even by convention," while the opening of this article says "type hints have grown from a nice-to-have to an expectation for popular packages." Things don't always work out the way the PEP authors expect. [1]: https://www.python.org/dev/peps/pep-0484/

It’s the most demoralizing aspect as even just typing the standard library online documentation and examples (such as Emil autoname example) would be extremely valuable.

Re: Tests aren’t enough: Case study after adding type hints to urllib3

#157

Post by skrimp » Tue, Oct 19, 2021, 12:29 AM UTC

loup-vaillant wrote: post #126

slaymaker1907 wrote: post #113

Earlier quoted context omitted.

Please write a type for the following function: def compose(start, *args): def helper(x): for func in reversed(args): x = func(x) return start(x) return helper There is no mainstream typed language which can write a fully general type for the vararg compose function. TypeScript is probably the one that comes closest, but last I checked it still was unable to write a sufficiently powerful array type. You can write a t…

Vararg functions also have limited use. Especially considering that most of the time, your args will all have the same type, and therefore could just be passed in an array or similar. The one mainstream exception I know of is print functions, and we have* ways to statically check those. Your toy example, even generalised, has no practical use. If I can write this: compose(f, f1, f2, f3) Then I can write that instead…

This still technically reduces the generality of the given function since you are specifying that each function cannot have multiple overloads.

let f be a overload set matching the signatures {a -> b, i -> j} let g be a overload set matching the signatures {b -> c, j -> k}

compose(g, f) could be given a to return c or i to return k

Re: Tests aren’t enough: Case study after adding type hints to urllib3

#158

Post by tunesmith » Tue, Oct 19, 2021, 12:33 AM UTC

IshKebab wrote: post #98

tunesmith wrote: post #45

Earlier quoted context omitted.

I've been a types advocate for years, but it wasn't until working with Typescript that I started experiencing some of the downsides... To me, ideally, types are supposed to be a benefit not only in safety, but in understanding the intent of a piece of code more quickly. For an api or library interface, review the types to see what its intentions are. But there's something about the typescript type system, with all th…

Your criticism boils down to "it's possible to overcomplicate things". Sure if you completely remove static typing then you can't overcomplicate static typing. But is that really an argument against static typing?

No, and I am still in favor of static typing. But I also don't think it's purely up to team discipline. Something about typescript (or more aptly, javascript) incentivizes crazy typing more than other languages. Luckily, the last few years appears to have had more emphasis on designing languages to take these kinds of incentives into account, so I still think the future is bright.

Re: Tests aren’t enough: Case study after adding type hints to urllib3

#159

Post by tiew9Vii » Tue, Oct 19, 2021, 1:36 AM UTC

wvenable wrote: post #34

david422 wrote: post #19

I love static typing/type hints if for only 1 thing - code maintenance. Even code I wrote six months ago. Not having to dig through 6 functions deep to try to figure out whether "person" is a string, or an object, and if it's an object what attributes it has on it etc. is huge. And not to mention that some clever people decide - hey, if you pass a string I'll look up the person object - so you can pass an object or a…

The main argument for dynamic typing is speed in prototyping but I find that's opposite for me. I'm much more comfortable rapid prototyping and ripping stuff apart when I have a strongly static typed environment telling me what I just broke. Doing radical refactoring often involves just making those changes and then fixing all the IDE or compiler errors until it runs again.

It depends on the language but I find I'm also far more productive with strong types.

When I use a dynamic language I get no errors in dev, I need to run/invoke the program to see if it works. It may appear to work fine as I haven't executed a specific code path hence dynamic languages have extremely high test coverage. With dynamic languages I am delaying my feedback loop, I may get some visual output quicker but that doesn't mean my program is correct.

With a strongly typed language and utilizing types you use the compiler to guide you. The compiler says hey, this isn't correct, fix it, you go fix the error and recompile and repeat.

I've used Elm before and it's the only time I had a complex Javascript UI just compile and work first time. It's like a wow, did that just happen.

With Typescript it's not quite to the level of Elm but find my experience working with React etc far more productive. Typescript says hey, that's wrong, I expect ... you gave ..., you work through the errors and when it runs generally there's less silly mistakes than when I just use Javascript.

I'm learning Rust, the compiler error messages have greatly helped. When you compile it says hey, you tried to do ..., maybe you want ... instead. Not to sure what the suggestion is I try it and 9 times out of 10 it works, compiles, program runs.

With types you generally get better IDE auto complete support etc.

Now i'm using Python for my day job. My experience has been painful, discovering what arguments functions take, passing in wrong values, needing to run slow test suites, finding errors at runtime. Yes you can use type hints and I do but I find them far less reliable.

I guess I'm not a very good programmer so learnt to lean on a compiler to do the hard work for me, and if you have good type support you can lean on types more to get the compiler to help you more.

In Haskell I can write complex logic by writing out the types and ADT's. I've written whole programs with tests to verify the logic without writing a program. I find this incredible efficient for prototyping ideas, just write the types, the functions signitures etc etc. Once that is done you implement the functions, hit compile then boom, your shocked it just worked first time running.

Re: Tests aren’t enough: Case study after adding type hints to urllib3

#160

Post by iamevn » Tue, Oct 19, 2021, 1:47 AM UTC

slaymaker1907 wrote: post #113

tetha wrote: post #97

Earlier quoted context omitted.

Hm, growing somewhat experienced, I find myself adapating an old quote more and more: Sufficiently advanced static typing is indistinguishable from dynamic typing. Now, I know, it's not true. It's entirely possible to build weird things in python that are provably impossible to typecheck statically. But modern language servers and their type inference capabilities in rust, terraform, or even straight up python are ve…

Please write a type for the following function: def compose(start, *args): def helper(x): for func in reversed(args): x = func(x) return start(x) return helper There is no mainstream typed language which can write a fully general type for the vararg compose function. TypeScript is probably the one that comes closest, but last I checked it still was unable to write a sufficiently powerful array type. You can write a t…

I think you can get close implementing it as a macro in typed racket, expanding the type out based on how many arguments you give it. But then it's not a first class function until you expand it.

Found this implementation which also provides pre-expanded forms that are first class functions for specific lengths of arguments docs: https://docs.racket-lang.org/typed-compose/index.html implementation: https://git.marvid.fr/scolobb/typed-compose/src/branch/maste...