Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Open-sourcing MonkeyType – Let your Python code type-hint itself

engineering.instagram.com

Post reply on HN
Print view View original

191–200 of 237 posts

Re: Open-sourcing MonkeyType – Let your Python code type-hint itself

#191

Post by lmm »

andybak wrote: post #166
IshKebab wrote: post #69

Earlier quoted context omitted.

Navigating a large code base that is dynamically typed like Python is far more tedious than something like C++ or C#. First you can't read what the types passed into and out of functions are. You have to find their usages to work it out. Second, you can't reliable do things like "find usages" or "go to definition" because of the dynamic typing.

> Second, you can't reliable do things like "find usages" or "go to definition" because of the dynamic typing. In my experience PyCharm can do both correctly for the vast majority of cases.

Indeed; it only fails in the really fiddly cases where you need it the most.

Re: Open-sourcing MonkeyType – Let your Python code type-hint itself

#192

Post by lmm »

jimnotgym wrote: post #57
fiatjaf wrote: post #37

Earlier quoted context omitted.

Isn't it faster to write everything in Go, for example, that has a compiler guiding you all the way and you rarely get runtime errors? I feel my developer time very much "optimized" when writing backends in Go than when I wrote then in Python (I also tried Node, which was a disaster).

Go wasn't a serious option for Instagram though. According to Wikipedia Instagram launched in 2010, Go launched in 2009. That would have made them very early adopters, was the library support there back then like it was for Python? Just because there may be better tools now, should they scrap their working code that earned their fortune?

Well, OCaml, Haskell or maybe Scala would've been mature options in 2010, combining safety with language-level productivity that's comparable to Python, though the extent to which web frameworks available in those languages at that time were Django-equivalent is arguable. (Personally I would - and did - happily choose Scala with Wicket over Django in 2010)

In terms of what Instagram should do now, I'd say they should do what Facebook did: introduce thrift or similar, gradually move business logic into backend services written in more suitable languages, leaving the Python to eventually become just a thin web frontend. Retrofitting types onto code involves a lot of the same effort as rewriting it into a better language, and the rewards for the latter are higher, IME.

Re: Open-sourcing MonkeyType – Let your Python code type-hint itself

#193

Post by bonesss »

blub wrote: post #171
kbd wrote: post #169

Earlier quoted context omitted.

> This tool is the proof that Python has significant problems at scale... This does not prove your point. Annotating a large dynamically-typed codebase with type information is a large amount of work, regardless of the language. This tool makes that easier.

I'm bemused by your reply and curious to know why you think Dropbox and Instagram are working on static type analysers for their large Python code bases. Instagram at least gave us a hint: "we’re keen to make our code easier for new developers to read and understand, as well as more amenable to static analysis that shrinks the domain of possible bugs". It seems to me it's so difficult to manage such a code base, that…

> ... it seems prudent to avoid the said amount of work by picking another programming language for one's large-scale code base.

While I'm a type-adherent in my day-to-day (F# represent, wut wuuut), I think this isn't reflective of the chicken and egg dilemma for startups... There are an immense number of things they "should be" doing at scale that they can't do early because they're relying on their early product to scale. Time to market, and windows of opportunity, are critical to startups.

Any immature technical decision at that part of the lifecycle needs to be made not with an attempt to make perfect forever from the start, but rather with an eye to transitioning to smarter solutions aggressively as you scale up. The company may be 6 pivots away from success, so better to validate solutions in the market than prognosticate.

I don't accept the premise that a lack of typing significantly improves development velocity, per se, but language decisions are about ecosystems, key components, and local talent. Where these companies get into coo-coo land is not integrating those immature components into better systems as they're getting bigger. Next thing you know someone is writing a whole compiler chain for PHP in an attempt to reinvent a sane programming language, or trying to get Python to be Java.

Smart, modern, functional languages provide development velocity and pleasure equal to Python with fundamental type safety and guarantees. But in a world where just putting a button on a webpage means multiple dialects in multiple languages I think we should be ok mixing and matching on the backend to scale smartly.

Re: Open-sourcing MonkeyType – Let your Python code type-hint itself

#194

Post by lmm »

vasilipupkin wrote: post #26
nerfhammer wrote: post #21

Earlier quoted context omitted.

there's a difference between should be and should have been maybe it's easier to do this half-measure than rewrite your entire code base

I get that part, but for new projects, this perhaps is a hint to do things in Java?

New projects at large companies often are written in Java. New projects at startups aren't and shouldn't be - doubling future maintenance costs for the sake of a 20% reduction in development time now is a good trade, because 90% of startups fail, the important thing is to validate product/market fit as soon as possible.

(Though really you should just use Scala and get both Java-like safety and Python-like productivity)

Re: Open-sourcing MonkeyType – Let your Python code type-hint itself

#195

Post by bonesss »

blub wrote: post #174
techdragon wrote: post #172

Earlier quoted context omitted.

The reality borne out by the evidence [0][1] is that Python is at the very least perfectly suitable for the development of web stacks powering companies worth in excess of hundreds of million dollars. Putting aside concrete technical issues regarding the python runtime’s performance envelope (eg: startup time, FFI inter-op call time, etc) and memory footprint, there is no reason not to use Python. Again, concrete tec…

I'm not claiming that one can't be successful by using Python (or pretty much any programming language). Market success or user count are unfortunately not tightly correlated with technical excellence, as many of us have bitterly found out. Development speed, maintainability, error count are also important development issues. Unfortunately we don't have much data to judge, but the little that we have such as this art…

> ...we don't have much data to judge, but the little that we have such as this article indicate that dynamic typing has a non-negligible negative impact on the above.

In my own experience I see it on a spectrum: there are design decisions you can avoid by using dynamic typing that give you speed in the small that start to show their absence in the large. Savings on inputs (quicker coding), tend to evaporate as complexity grows because you're using so much time analysing outputs (the system), to determine behaviour. Every bit of up-front work saved gets amortised across issue after issue, and runtime testing during development, and increasing stagnation in high-level architecture.

For me the ideal is the type system of Haskell with the linguistic power of Haskell and the type inference of Haskell... only on a mainstream platform I can convince management to use.

Re: Open-sourcing MonkeyType – Let your Python code type-hint itself

#196

Post by walrus1066 »

tigershark wrote: post #135
c22 wrote: post #121

Earlier quoted context omitted.

The first language I learned (after Applesoft BASIC) was C. I wrote C for a long time. About 6 years ago I picked up Python. Today I find it much easier and much more pleasant to spin up a new idea in Python, to the point that it is my default choice for new projects with fuzzily defined goals. None of my ideas have become companies, but I could totally see just sticking with Python even past the point that it became…

Between c and Python there is an ocean of languages...

I'm with the OP on this, I've experience with Java, C#, Python & C++.

I'm a big fan of rapid prototyping with Python to map out problem domains, and once the domain has been mapped properly, rewriting in a statically typed language if necessary.

Python is much better for prototyping than the other langs I've used. Because the syntax is almost pseudocode, and the duck-typing makes a lot of design patterns and boilerplate obsolete, so I can dedicate my headspace to the problem at hand.

Right tool for the right job, as they say.

Re: Open-sourcing MonkeyType – Let your Python code type-hint itself

#197

Post by auxym »

joshuamorton wrote: post #141
BucketSort wrote: post #128

I come from a statically typed background (C++), but have been doing a lot of analytics in python in the past two years. It is frustrating not to have compile time guarantees when dealing with mathematical programs, because some things have to be a particular type (i.e. matrices of compatible dimensions). The result is a copious use of asserts, but it feels bad when you know that if you did this in a functional langu…

>(i.e. matrices of compatible dimensions) What language do you use where you can get these kinds of guarantees? As far as I know very few languages provide those kinds of dependent types statically.

I believe in the functional world, that sort of thing is implemented suing a feature called dependent types. It's not a very common feature, but Liquid Haskell implements it.

Re: Open-sourcing MonkeyType – Let your Python code type-hint itself

#198

Post by tigershark »

joshuamorton wrote: post #144
tigershark wrote: post #139

Earlier quoted context omitted.

Absolutely not. Python is not the technological superior choice for the majority of the applications. If you think so then your experience in different domains and application types must be very limited. The preconcept that dynamic languages are more productive is just an illusion because you can easily take shortcuts that will hamper your progress in the future. A proper typed language with HM type inference has the…

Empirically, there are a lot more million line python codebases than F# or haskell codebases, in fact I can name multiple million line python codebases, and 0 F# or haskell codebases. Given that, logic would indicate some sort of failure on the part of haskell and F#, or they would see wider adoption among the large codebases where they are so useful. Do you disagree?

Given how much smaller is the F# community and how much more you can crank in less lines of codes in F# I can believe it. Between C# and F# there is about an order of magnitude of difference in the LOCs for big projects and C# and Python are comparable from this metric.

Re: Open-sourcing MonkeyType – Let your Python code type-hint itself

#199

Post by moreless »

blub wrote: post #168
alexchamberlain wrote: post #54

Fantastic contribution back to the community; I look forward to trying it out. I must say this is the first time I've been disappointed with the quality of discussion on HN. For a community that promotes using the right tool for the job at the time , I would have thought people would be more open to the choices the early engineers made. I'm sure that Instagram are using a variety of tech across their stack.

Most people would expect software to crash, hang, be slow or somehow leak their personal information. That's normal behaviour for the products of the software industry. For a long time there have been efforts to ensure at least a degree of quality and robustness through processes, practices and verification tools. One such tool is a type system which allows encoding requirements and expectations that will be automati…

> This tool is the proof that Python has significant problems at scale, which is something the Python community has denied for a long time. They're still doing it in this thread, but the lesson looks pretty clear to me: if you plan on building large scale, don't use Python. Or PHP while we're at it (see Facebook).

Yeah, absolutely, don't do this! These are two examples of successful companies that did it and look at them now!

Being able to move fast and produce a winning product on time is much more important for startups. What does it matter if you used for a project, when it never went past 10 users because you were concentrating on wrong side aspect of your business? PHP is fine. Python is great. Use the tools that fit your problem and you know how to use, not the latest toy.

Re: Open-sourcing MonkeyType – Let your Python code type-hint itself

#200

Post by tigershark »

aub3bhat wrote: post #107
Cieplak wrote: post #41

Given that people have asked why not use a statically typed language, seems appropriate to mention that it's possible to write pythonic-looking C++: http://preshing.com/20141202/cpp-has-become-more-pythonic/ I have been using C++ a lot lately but really wish there were more tools for reflection at compile time, e.g., ability to iterate over all the members of a class. Other than that, I'm really loving C++17's auto t…

Please show me C++ equivalent of sorted([(k.weight, k.name) for k in somelist], reverse=True)

No idea about C++ but in C# seems quite more readable (and flexible)

    someList.OrderByDescending(x => x.weight).ThenByDescending(x => x.Name);
It's quite similar to expressing the concept in English, certainly more than using list comprehension in Python.

And how would you order it in Python by ascending on the first field and descending on the second using list comprehension?

Post reply on HN