Open-sourcing MonkeyType – Let your Python code type-hint itself • Hacker News Board

Re: Open-sourcing MonkeyType – Let your Python code type-hint itself

#61

Post by WorldMaker » Thu, Dec 21, 2017, 10:46 PM UTC

xzel wrote: post #47

azhenley wrote: post #20

Earlier quoted context omitted.

Didn't Facebook do this for PHP (Hack) and Microsoft with JavaScript (TypeScript)? From a technical aspect, I do find these projects cool. I wonder if its more efficient for large companies to initially develop using dynamic languages then transition them with these optionally typed languages.

Facebook did even more with PHP with HHVM and other tooling. I'm not sure if JS was exactly the same for Microsoft as I saw it more as a cool internal project that got big vs something that eventually became necessary.

From what I read, several big teams at Microsoft were dogfooding Typescript immediately during development, even before it was ever externally released.

Re: Open-sourcing MonkeyType – Let your Python code type-hint itself

#62

Post by switchbak » Thu, Dec 21, 2017, 10:46 PM UTC

fiatjaf wrote: post #37

realbarack wrote: post #24

Earlier quoted context omitted.

I agree with you but I think it's pretty straightforward how this sort of thing happens: 1. Startup builds thing fast in dynamic language because they need to optimize for development speed and iteration, not maintainability or scalability. 2. Startup grows and continues to hire for expertise in the tech stack they are mostly already using. 3. Repeat for some years and some hundreds of engineers and you arrive at thi…

Isn't it faster to write everything in Go, for example, that has a compiler guiding you all the way and you rarely get runtime errors? I feel my developer time very much "optimized" when writing backends in Go than when I wrote then in Python (I also tried Node, which was a disaster).

Many in the Haskell community talk about being guided by their types, perhaps you're seeing a similar benefit?

Not sure why the negative vibes toward this comment, I think it's fairly sensible (of course, this is a very subjective subject).

Re: Open-sourcing MonkeyType – Let your Python code type-hint itself

#63

Post by imiric » Thu, Dec 21, 2017, 10:54 PM UTC

nawgszy wrote: post #33

maltalex wrote: post #13

Reading this as someone who writes mostly in statically typed languages, the whole exercise seems odd. Having so much dynamically typed code to maintain that you need to run production code using a separate tool just to figure out the types sounds just wrong. Why not use a statically typed language for such a large code-base? Is this done by purpose, or did they end up with a million lines of Python code and are look…

>Is this done by purpose, or did they end up with a million lines of Python code and are looking for ways to make the maintenance easier? Definitely the latter. I've seen this discussion a few times before, and it's always the same. Your initial developers are not looking down the road to the million lines of code milestone, they're just trying to make a product that might actually make some money here and now. I'm s…

To add to this, note that type hinting is quite a new feature in Python (introduced in v3.5, released in 2015), and this functionality simply wasn't available before. So any company heavily invested in Python today obviously wants to improve their runtime reliability, without having to rewrite parts of their stack.

Stricter typing goes a long way to achieve this, and gradual typing allows you to upgrade the code base at your own pace, which is great.

Consider this study[0] about TypeScript and Flow, which use the same approach for JavaScript, which found both able to detect ~15% of runtime bugs. So no wonder companies with large Python code bases would be the first to invest in this space.

Personally I feel this is a great addition to the language, and hope type checking becomes a first class citizen too, instead of being delegated to external tools like mypy[1] or pytype[2].

[0]: http://ttendency.cs.ucl.ac.uk/projects/type_study/

[1]: http://mypy-lang.org/

[2]: https://github.com/google/pytype

Re: Open-sourcing MonkeyType – Let your Python code type-hint itself

#64

Post by lambda » Thu, Dec 21, 2017, 10:59 PM UTC

maltalex wrote: post #13

Reading this as someone who writes mostly in statically typed languages, the whole exercise seems odd. Having so much dynamically typed code to maintain that you need to run production code using a separate tool just to figure out the types sounds just wrong. Why not use a statically typed language for such a large code-base? Is this done by purpose, or did they end up with a million lines of Python code and are look…

> Why not use a statically typed language for such a large code-base? Is this done by purpose, or did they end up with a million lines of Python code and are looking for ways to make the maintenance easier?

Being in a similar (though much smaller scale) situation ourselves, I suspect your latter suggestion is the answer; they wound up with a large amount of Python code due to expediency (Python is good at quickly getting things done), and are now finding the code base quite hard to maintain.

Something to remember is that as of 10 years ago, statically typed languages consisted of C (for some definition of "statically typed"), C++ (complicated and cumbersome, and didn't yet have widespread availability of "modern" C++ features), Java (cumbersome, heavyweight, lots of missing opportunities for abstraction back then), C# (heavily tied into the Microsoft ecosystem, not yet open source), and a bunch of weird academic languages that required tutorials about burritos to learn (ML, Haskell, etc).

While statically typed languages like C, C++, and Java were the mainstream languages of the 90s, C was too limited for a lot of people, C++ and Java too cumbersome to use, and so lightweight dynamically typed languages like Perl, Python, PHP, Ruby, and JavaScript picked up a lot of steam due to how much easier to pick up and more productive many programmers found themselves in those languages. But now we have large, fairly un-maintainable code bases in these languages, and people are realizing the value of static typing, in part due to the maintenance hassle and in part due to newer, more expressive and accessible statically typed languages being available (Elm, Rust, TypeScript, Scala, Go, Kotlin, as well as improvements to C++, Java, and C#).

But that leaves all of these old codebases, that are hard to maintain. Rather than doing a complete rewrite, adding static typing capabilities that can be applied to existing codebases is a way to make them more maintainable without spending all of the time of a complete rewrite and having everyone have to spend all of the time learning the new language while still maintaining the old codebase.

Re: Open-sourcing MonkeyType – Let your Python code type-hint itself

#65

Post by ageofwant » Thu, Dec 21, 2017, 11:05 PM UTC

kod wrote: post #60

scrollaway wrote: post #38

Earlier quoted context omitted.

I really don't get people. Instagram and Dropbox, through typing annotations in Python, are gradually improving a language that has codebases running globally, from YouTube to NASA. Clearly something is right with the situation when the incentives are aligned for a tech company to contribute back to the open source community in such fundamental ways. So why look for the mole and think "They should have done it differ…

No, it's like asking someone who spent a lot of time building an octagonal wheel and is now trying to shave down the corners... why didn't you use a circle to begin with.

They did not spend a lot of time building an octagonal wheel, they built a billion dollar company using Python. Now, when the code base has been proven, and the business rules solidified, they retrofit what they believe will make the code base easier to maintain.

Python is an excellent enabler of this kind of dynamic system evolution.

Re: Open-sourcing MonkeyType – Let your Python code type-hint itself

#66

Post by ageofwant » Thu, Dec 21, 2017, 11:11 PM UTC

lambda wrote: post #64

maltalex wrote: post #13

Reading this as someone who writes mostly in statically typed languages, the whole exercise seems odd. Having so much dynamically typed code to maintain that you need to run production code using a separate tool just to figure out the types sounds just wrong. Why not use a statically typed language for such a large code-base? Is this done by purpose, or did they end up with a million lines of Python code and are look…

> Why not use a statically typed language for such a large code-base? Is this done by purpose, or did they end up with a million lines of Python code and are looking for ways to make the maintenance easier? Being in a similar (though much smaller scale) situation ourselves, I suspect your latter suggestion is the answer; they wound up with a large amount of Python code due to expediency (Python is good at quickly get…

Also, these old code bases can now be fairly easily and optionally retrofitted using modern Python's support for type hints and tools like mypy and now MonkeyType. So people can continue using their existing code base and the vast Python ecosystem, with the added benefit of type checking where appropriate.

Re: Open-sourcing MonkeyType – Let your Python code type-hint itself

#67

Post by pfranz » Thu, Dec 21, 2017, 11:17 PM UTC

kod wrote: post #60

scrollaway wrote: post #38

Earlier quoted context omitted.

I really don't get people. Instagram and Dropbox, through typing annotations in Python, are gradually improving a language that has codebases running globally, from YouTube to NASA. Clearly something is right with the situation when the incentives are aligned for a tech company to contribute back to the open source community in such fundamental ways. So why look for the mole and think "They should have done it differ…

No, it's like asking someone who spent a lot of time building an octagonal wheel and is now trying to shave down the corners... why didn't you use a circle to begin with.

To me, this is a metaphor that might better explain it, "You need a different approach for getting your first million than your second." (I've heard it attributed to customers, revenue, personal income, etc). It sounds like Python (and features like dynamic typing) works very well at bootstrapping and developing. They're leveraging different features (more like static typing) once they get larger and more time is spent on maintenance (and rewriting everything isn't appealing [1]).

Honestly, your octagonal wheel metaphor works, too. Building the first car you spend a lot of time on octagonal (crude) wheels, but later spend a lot more money on round (precision) wheels. You could have gone bankrupt spending money originally on round wheels that were the wrong size.

[1] https://www.joelonsoftware.com/2000/04/06/things-you-should-...

Re: Open-sourcing MonkeyType – Let your Python code type-hint itself

#68

Post by nikhilalmeida » Thu, Dec 21, 2017, 11:18 PM UTC

maltalex wrote: post #13

Reading this as someone who writes mostly in statically typed languages, the whole exercise seems odd. Having so much dynamically typed code to maintain that you need to run production code using a separate tool just to figure out the types sounds just wrong. Why not use a statically typed language for such a large code-base? Is this done by purpose, or did they end up with a million lines of Python code and are look…

reasons (advantages) for using python (or any other dynamic lang.) when in startup mode is more than just typing. Similarly the advantages of continuing to use python outweigh the costs of doing this little "type-dance".

Re: Open-sourcing MonkeyType – Let your Python code type-hint itself

#69

Post by IshKebab » Thu, Dec 21, 2017, 11:22 PM UTC

nawgszy wrote: post #33

maltalex wrote: post #13

Reading this as someone who writes mostly in statically typed languages, the whole exercise seems odd. Having so much dynamically typed code to maintain that you need to run production code using a separate tool just to figure out the types sounds just wrong. Why not use a statically typed language for such a large code-base? Is this done by purpose, or did they end up with a million lines of Python code and are look…

>Is this done by purpose, or did they end up with a million lines of Python code and are looking for ways to make the maintenance easier? Definitely the latter. I've seen this discussion a few times before, and it's always the same. Your initial developers are not looking down the road to the million lines of code milestone, they're just trying to make a product that might actually make some money here and now. I'm s…

Navigating a large code base that is dynamically typed like Python is far more tedious than something like C++ or C#.

First you can't read what the types passed into and out of functions are. You have to find their usages to work it out. Second, you can't reliable do things like "find usages" or "go to definition" because of the dynamic typing.

Re: Open-sourcing MonkeyType – Let your Python code type-hint itself

#70

Post by CGamesPlay » Thu, Dec 21, 2017, 11:23 PM UTC

ambivalence wrote: post #56

CGamesPlay wrote: post #51

Why didn't you make the Python 3 type checking advisory instead? Like what Facebook did with Flow and Hack, why not make write a product that lets you statically analyze the types and will never itself cause runtime errors, and transition to types that way? What advantage does building this tool have? I understand that the thing I'm describing involves modifying the way Python 3 handles type annotations, but it doesn…

I'm not quite sure what you mean. Python has an external static checker for types, it's called mypy. Python's type annotations are in fact very similar to Flow and Hack in the sense that they provide gradual typing. The specification (see: PEP 484) describes that only annotated functions are type checked. Calls to non-annotated code are treated as accepting any type in arguments and returning the Any type (a special…

Sorry, I guess I jumped to the conclusion that you profiled in production because you had a mandatory type checker which would cause runtime errors if invalid types were passed. Let me try another question. Do you think that a project like MonkeyType would substantially aid adding flow annotations to a JavaScript project? If so, why is the value to the Python ecosystem higher than the value to the Flow/JavaScript ecosystem (i.e. why doesn't it exist for JS)?