Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tests aren’t enough: Case study after adding type hints to urllib3

sethmlarson.dev

Post reply on HN
Print view View original

161–170 of 205 posts

Re: Tests aren’t enough: Case study after adding type hints to urllib3

#161

Post by lamontcg »

layer8 wrote: post #104
lamontcg wrote: post #100

Earlier quoted context omitted.

I'm okay with "var = new FooBarBazThingyWithALongName()" because I don't need to see the type name twice there. In an IDE you can get the type annotation from the IDE over every inferred var type, but I don't like requiring an IDE to see that information and like it showing up in 'less' as well.

I agree it's redundant if the type name occurs twice in the same statement. However, further evolution of the code often causes the instantiation to be moved elsewhere, and I wouldn't have confidence that the one doing that change then also changes `var` back to the type name. Instead, it would be nice to have syntax avoiding the duplication in the fashion of `FooBarBazThingyWithALongName thingy = new(...constructor…

Ideally linting tools on PRs show the refactored code as a violation, and it should be easy to rip a cleanup refactoring across the files before submitting a PR to avoid that.

Re: Tests aren’t enough: Case study after adding type hints to urllib3

#162

Post by fiddlerwoaroof »

loup-vaillant wrote: post #126
slaymaker1907 wrote: post #113

Earlier quoted context omitted.

Please write a type for the following function: def compose(start, *args): def helper(x): for func in reversed(args): x = func(x) return start(x) return helper There is no mainstream typed language which can write a fully general type for the vararg compose function. TypeScript is probably the one that comes closest, but last I checked it still was unable to write a sufficiently powerful array type. You can write a t…

Vararg functions also have limited use. Especially considering that most of the time, your args will all have the same type, and therefore could just be passed in an array or similar. The one mainstream exception I know of is print functions, and we have* ways to statically check those. Your toy example, even generalised, has no practical use. If I can write this: compose(f, f1, f2, f3) Then I can write that instead…

> Vararg functions also have limited use.

This is only because people are using statically typed language that place arbitrary restrictions on such functions and make them harder to use. In dynamically typed languages, vararg functions are widely used and enable patterns that are pretty nice.

Re: Tests aren’t enough: Case study after adding type hints to urllib3

#163

Post by fiddlerwoaroof »

sseagull wrote: post #47
kennywinker wrote: post #37

Earlier quoted context omitted.

I've seen essentially this code in so many organically grown codebases (when they grew up without types). It's usually close the the UI, because someone had to quickly add an alternate path to support some new user interaction function find_user(person) { if user is string { query_by_name(person) } else { query_by_name(person.name) } } and yeah, we all know it's kinda messy, but also that logic has to live somewhere…

I came very close to writing almost this exact code just the other day (except it was username or user id for me), but came to my senses. It's just so tempting in a dynamic language... In a static language, you either can't do it, have to really go out of your way to do it, or at least do function overloading (which is a bit cleaner)

Calling a function like this “ensureUser” is pretty idiomatic and useful, in lisp-style code bases. I think it’s a pattern related to “parse, don’t validate” in static-type lands: rather than _checking_ what the type is and throwing an error, you define a function that knows how to turn various representations of your type into its canonical shape.

Re: Tests aren’t enough: Case study after adding type hints to urllib3

#164

Post by lrobinovitch »

tabbott wrote: post #56

I agree with all the benefits of mypy cited in this article. For me, most important thing for the long-term health of a codebase is its readability/maintainability, and mypy static typing makes such a huge difference for that in large Python codebases. I'm really excited to see large libraries doing this migration. I'll add for folks thinking about this transition that we took a pretty different strategy for converti…

I really like this documented type ignore strategy and will start incorporating it in our codebase. Thanks for sharing.

Re: Tests aren’t enough: Case study after adding type hints to urllib3

#165

Post by kraf »

It would be really interesting to see some examples of the logic errors that were found that couldn't be found by tests. This seems to have been a very robust library. What kind of problems did you find? From what is mentioned in the article it really doesn't sound like the investment of hundreds of hours from multiple people has been actually worth it.

Re: Tests aren’t enough: Case study after adding type hints to urllib3

#166

Post by tehnub »

phoneperson wrote: post #138
tetha wrote: post #119

Earlier quoted context omitted.

As I said, there are infinite possible programs that cannot be type checked statically. This is derived from the halting theorem trivially - lambda(p) = if p.halts() then A() else B(). However, the intersection of programs I encounter in practice with the number of programs that can be statically checked is rather large.

You are probably thinking of Gödel's incompleteness theorems. I do not think that you can trivially derive it from the halting problem.

He just did derive it from the halting problem. In words: It's impossible to statically check the type of a function that takes a program P as its input, and returns the integer 1 as its output if P halts, and returns the string "1" if it does not halt. It would require solving the halting problem, which is undecidable.

Re: Tests aren’t enough: Case study after adding type hints to urllib3

#167

Post by pjmlp »

Jensson wrote: post #118
pjmlp wrote: post #109

Earlier quoted context omitted.

And at what level of abstraction does ISO C work on? "C Is Not a Low-level Language, Your computer is not a fast PDP-11." https://queue.acm.org/detail.cfm?id=3212479

That would be a good point if you actually could write the final code for CPU's, but you can't since the CPU internals do that for you. So from an application programmers perspective machine code is as low as it gets and C maps really well to machine code so C is a low level language.

Only if that machine is a 8 or 16 bit CPU, with code compiled as -O0.

And even then, it's impossible to write something like malloc() without using either an external Assembler or language extensions.

Re: Tests aren’t enough: Case study after adding type hints to urllib3

#168

Post by simion314 »

fiddlerwoaroof wrote: post #106
simion314 wrote: post #96

Earlier quoted context omitted.

>and people that enjoy the experience of iterating quickly in a dynamically typed language Programmers spend more time reading code then writing it. So I personally prefer the devs in the team will spend more time typing the code or use a bit more brain energy to think about types so later we can all read the code and understand it and edit faster. Dynamic works great for write-only scripts.

People always say this about reading code and it’s just never matched my experience working in either sort of codebase: one difference (comparing lisps and, say, Typescript or Java) is that lisps just have fewer lines to read. So, any assistance you get from the types is counteracted by having to read more code. But, additionally, I just don’t find it true to my experience that it’s easier to read and understand a dy…

Maybe it depends on what you work on.

From my experience things go like this

1 we have a simple problem I implement a simple elegant solution

2 some new feature is added, this means there are some special cases now, most of the time someone else in the team adds this new feature , so 4-5 different places are modified, functions need to get more parameters and some IFs are added in those 4-5 places

3 later a new feature is added again, a few more extra special cases again , the dev will again go add some more function parameters here and there , add more IFs but fails to find where all the places that might need to be modified are.

4 things are now a big mess, I have to fix it, and I now have to read all the code, my own old code that was changed with different exceptions and the other developers code. I spend a lot of time on reading stuff, understanding how stuff works now, understand why this new code does some stuff then I spend the time abstracting again a solution, abstracting all the special cases. After I have a solution in mind comes the refactoring, with static types or type hints is much easier to find where stuff is used so you know what to modify.

I am sure one some projects where maybe there is only 1 or few devs that all write quality code and there are no new requirements that need implemented ASAP the code could stay more readable but this is the exception unfortunately.

Re: Tests aren’t enough: Case study after adding type hints to urllib3

#169

Post by aaron695 »

taeric wrote: post #20
m12k wrote: post #11

I've only been in the industry for ~15 years, but it still feels like every year, some ecosystem discovers the value of something that another ecosystem has taken for granted for decades - type-checking, immutability, unidirectional data-flow, AOT-compilation, closures, pure functions, you name it. I'm glad we seem to be converging on a set of best practices as an industry, but sometimes I wish we were spending less…

I've been alive long enough to see that most things are useful, and all things are oversold. More, the nice easy things to build with major restrictions pretty much gets thrown out the window for complicated things that have constraints that most efforts don't have. This isn't just a software thing. Building a little shed outside? Would be silly to use the same rigor that goes into a high rise. Which would be crazy t…

> Building a little shed outside? Would be silly to use the same rigor that goes into a high rise.

This is a bad analogy. They both have 1000's of years of shared development (some high rises still use bricks), 100's in an industrial age and 20 years in the internet age. (Testing is only ~70 years old)

People today use shed technology that pre-internet builders would not have know about or not existed.

But that 1000 years also includes how to use workers to get the job done.

That's not going into the rigor required in many places to build a shed. Watch Still Mine (2012) https://www.imdb.com/title/tt2073086/ about a couple wanting to build a house themselves, it's not a shed, but fights to build sheds against local governments are less romantic. (And it's a good movie)

Re: Tests aren’t enough: Case study after adding type hints to urllib3

#170

Post by arwhatever »

fiddlerwoaroof wrote: post #69
arwhatever wrote: post #9

Earlier quoted context omitted.

Is it possible to elaborate in a comment? Honestly I probably wouldn’t take the time to read a lengthy article, but if there’s some elevator pitch then I’m all ears.

What makes CL/Clojure really work is that your editor (emacs usually, but there’s other options now) connects to the live program and has access to the entire runtime environment. So, you can do a lot of the things other languages need static types for via introspection (e.g. autocomplete: CL just asks the running program what functions are available that matches the current pattern and returns a list). Secondly, sin…

This is a really useful overview from which I gained some new insights. I appreciate you taking the time.
Post reply on HN