Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Open-sourcing MonkeyType – Let your Python code type-hint itself

engineering.instagram.com

Post reply on HN
Print view View original

31–40 of 237 posts

Re: Open-sourcing MonkeyType – Let your Python code type-hint itself

#31

Post by vorotato »

azhenley wrote: post #20
maltalex wrote: post #13

Reading this as someone who writes mostly in statically typed languages, the whole exercise seems odd. Having so much dynamically typed code to maintain that you need to run production code using a separate tool just to figure out the types sounds just wrong. Why not use a statically typed language for such a large code-base? Is this done by purpose, or did they end up with a million lines of Python code and are look…

Didn't Facebook do this for PHP (Hack) and Microsoft with JavaScript (TypeScript)? From a technical aspect, I do find these projects cool. I wonder if its more efficient for large companies to initially develop using dynamic languages then transition them with these optionally typed languages.

Small projects scream dynamic types, but not every baby never grows up. Eventually the cognitive load becomes crippling and you're crying for static types.

Re: Open-sourcing MonkeyType – Let your Python code type-hint itself

#32

Post by hacker_9 »

maltalex wrote: post #13

Reading this as someone who writes mostly in statically typed languages, the whole exercise seems odd. Having so much dynamically typed code to maintain that you need to run production code using a separate tool just to figure out the types sounds just wrong. Why not use a statically typed language for such a large code-base? Is this done by purpose, or did they end up with a million lines of Python code and are look…

The evolution is fairly simple: python is very easy to setup, especially with django for working on the web, and for writing scripts quickly it work really well when everything is just an 'import' away.

Then as the product gets bigger, you'll hire python developers to keep up with the workload - and the best ones will be the ones who have committed their lives to Python. So you'll now end up with more and more python code.

Before you know it, you aren't writing small scripts anymore, but now you are writing quite large features, that take weeks and that require intimate knowledge of the code base so you don't keep backtracking and repeating yourself. But Python doesn't help you at this point, you traded static types for flexibility and now you have to pay the price.

At this point you're screwed, too many man hours spent on the codebase to redo it, so what to do? Well if you have the man power, build your own static type checker of course! I mean after all, if you have 100s of engineers who cares? You just throw more people at the problem until it goes away. Then wrap it up in a nice little package, and slap yourself on the back while you ride the instagram bubble.

Re: Open-sourcing MonkeyType – Let your Python code type-hint itself

#33

Post by nawgszy »

maltalex wrote: post #13

Reading this as someone who writes mostly in statically typed languages, the whole exercise seems odd. Having so much dynamically typed code to maintain that you need to run production code using a separate tool just to figure out the types sounds just wrong. Why not use a statically typed language for such a large code-base? Is this done by purpose, or did they end up with a million lines of Python code and are look…

>Is this done by purpose, or did they end up with a million lines of Python code and are looking for ways to make the maintenance easier?

Definitely the latter. I've seen this discussion a few times before, and it's always the same. Your initial developers are not looking down the road to the million lines of code milestone, they're just trying to make a product that might actually make some money here and now.

I'm sure Instagram was exactly that. They needed to handle images and some guy knew how to do it in Python. They wrote Python code, and then people liked Instagram. They eventually became a billion dollar company with millions of lines of code and no where along the road was there time to say "hey we need to refactor this whole thing". Or if that was said, management laughed and said "we need this feature".

So here is where you end up. The developers need to clean things up but they don't have time to clean it up by using a language, realistically, they probably don't know as well as the Python they wrote the millions of lines of code in.

Re: your last comment, navigating a million lines of any codebase is daunting, and especially more so if you aren't a developer in that language. I'm not sure what exactly "Python" has to do with that, besides that you're not a Python dev.

Re: Open-sourcing MonkeyType – Let your Python code type-hint itself

#34

Post by maxxxxx »

sbuccini wrote: post #12
JosephRedfern wrote: post #2

This sounds similar to Dropbox's PyAnnotate -- Guido van Rossum writes about it here: http://mypy-lang.blogspot.co.uk/2017/11/dropbox-releases-pya... Would be interesting to see how MonkeyType and PyAnnotate compare.

I'd also like to hear more about this -- both the feature set and the development process. It's interesting that two large engineering organizations responsible for some of the most popular applications on the planet spent hundreds of engineering hours building almost the same exact tool at around the exact same time. I'd also like to know how much quicker or better it could have been completed if it had been done ou…

"I'd also like to know how much quicker or better it could have been completed if it had been done out in the open. "

I assume they scratched their own itch so it was probably faster to do it on their own addressing their specific needs.

Re: Open-sourcing MonkeyType – Let your Python code type-hint itself

#35

Post by vasilipupkin »

newfoundglory wrote: post #30
vasilipupkin wrote: post #26

Earlier quoted context omitted.

I get that part, but for new projects, this perhaps is a hint to do things in Java?

80% of startups are going to die before their codebase reaches this size. They should probably not be making tech stack choices based on "what if we succeed beyond all likelihood".

two things come to mind: maybe those startups would die at a slower rate if their code was comprehensible to begin with ? but leaving that aside, for companies where survival is not an issue then, doesn't this indicate that using a dynamically typed language is not great?

Re: Open-sourcing MonkeyType – Let your Python code type-hint itself

#36

Post by [deleted] »

Kalium wrote: post #16
maltalex wrote: post #13

Reading this as someone who writes mostly in statically typed languages, the whole exercise seems odd. Having so much dynamically typed code to maintain that you need to run production code using a separate tool just to figure out the types sounds just wrong. Why not use a statically typed language for such a large code-base? Is this done by purpose, or did they end up with a million lines of Python code and are look…

It's rarely easy to re-implement large or complex codebases in a new language. That kind of major effort requires significant signoffs from leadership and a large effort. This tool? This tool takes an existing set of codebases and makes them safer. No major boiling of oceans required.

[deleted]

Re: Open-sourcing MonkeyType – Let your Python code type-hint itself

#37

Post by fiatjaf »

realbarack wrote: post #24
maltalex wrote: post #13

Reading this as someone who writes mostly in statically typed languages, the whole exercise seems odd. Having so much dynamically typed code to maintain that you need to run production code using a separate tool just to figure out the types sounds just wrong. Why not use a statically typed language for such a large code-base? Is this done by purpose, or did they end up with a million lines of Python code and are look…

I agree with you but I think it's pretty straightforward how this sort of thing happens: 1. Startup builds thing fast in dynamic language because they need to optimize for development speed and iteration, not maintainability or scalability. 2. Startup grows and continues to hire for expertise in the tech stack they are mostly already using. 3. Repeat for some years and some hundreds of engineers and you arrive at thi…

Isn't it faster to write everything in Go, for example, that has a compiler guiding you all the way and you rarely get runtime errors? I feel my developer time very much "optimized" when writing backends in Go than when I wrote then in Python (I also tried Node, which was a disaster).

Re: Open-sourcing MonkeyType – Let your Python code type-hint itself

#38

Post by scrollaway »

maltalex wrote: post #13

Reading this as someone who writes mostly in statically typed languages, the whole exercise seems odd. Having so much dynamically typed code to maintain that you need to run production code using a separate tool just to figure out the types sounds just wrong. Why not use a statically typed language for such a large code-base? Is this done by purpose, or did they end up with a million lines of Python code and are look…

I really don't get people. Instagram and Dropbox, through typing annotations in Python, are gradually improving a language that has codebases running globally, from YouTube to NASA.

Clearly something is right with the situation when the incentives are aligned for a tech company to contribute back to the open source community in such fundamental ways. So why look for the mole and think "They should have done it differently", when doing it differently has a high likelihood to mean not being as successful as they are today, and not having the occasion to contribute back?

It's like telling a successful charity "You should just take everyone's money and spend it on lamborghinis instead of wasting time building wells in africa".

Re: Open-sourcing MonkeyType – Let your Python code type-hint itself

#39

Post by bobwaycott »

maltalex wrote: post #13

Reading this as someone who writes mostly in statically typed languages, the whole exercise seems odd. Having so much dynamically typed code to maintain that you need to run production code using a separate tool just to figure out the types sounds just wrong. Why not use a statically typed language for such a large code-base? Is this done by purpose, or did they end up with a million lines of Python code and are look…

You appear to be ignoring who this coming from. Instagram is huge, and they built their service with Python. It isn’t coming from someone who is arguing this is how it should be done from the beginning. Hell, given the typical lifecycle of a startup, this situation occurs exactly because a dynamic language is chosen to speed getting a product to market and try to build a business around it.

Re: Open-sourcing MonkeyType – Let your Python code type-hint itself

#40

Post by fiatjaf »

scrollaway wrote: post #27
hacker_9 wrote: post #4

"At Instagram we have hundreds of engineers working on well over a million lines of Python 3." It always amazes me that some of the most popular products around are built with the worst technology choices. And now they had to build their own static type checker, which slows down random samples of real users, just to shore up the language's weaknesses? Outstanding.

Let's make this thread useful instead: Please enlighten the HN community at large: Tell us how you'd build instagram, what your tech choices would be and why Python is the "worst technology choice". (If this goes well maybe you'll get an email from Mike Krieger with the subject line "Here, you do this".)

Do you really think Instagram is a great work of art, that the Instagram codebase is exemplary?

I think it could be pure shit -- and indeed a very simple idea. They got big and famous, but it wasn't for dexterity in language choice or coding.

Post reply on HN