Open-sourcing MonkeyType – Let your Python code type-hint itself • Hacker News Board

Re: Open-sourcing MonkeyType – Let your Python code type-hint itself

#171

Post by blub » Fri, Dec 22, 2017, 9:06 AM UTC

kbd wrote: post #169

blub wrote: post #168

Earlier quoted context omitted.

Most people would expect software to crash, hang, be slow or somehow leak their personal information. That's normal behaviour for the products of the software industry. For a long time there have been efforts to ensure at least a degree of quality and robustness through processes, practices and verification tools. One such tool is a type system which allows encoding requirements and expectations that will be automati…

> This tool is the proof that Python has significant problems at scale... This does not prove your point. Annotating a large dynamically-typed codebase with type information is a large amount of work, regardless of the language. This tool makes that easier.

I'm bemused by your reply and curious to know why you think Dropbox and Instagram are working on static type analysers for their large Python code bases. Instagram at least gave us a hint: "we’re keen to make our code easier for new developers to read and understand, as well as more amenable to static analysis that shrinks the domain of possible bugs".

It seems to me it's so difficult to manage such a code base, that they decided to do that "large amount of work" in addition to the large amount of work required to develop the necessary tools!

In which case it seems prudent to avoid the said amount of work by picking another programming language for one's large-scale code base.

Re: Open-sourcing MonkeyType – Let your Python code type-hint itself

#172

Post by techdragon » Fri, Dec 22, 2017, 9:06 AM UTC

blub wrote: post #168

alexchamberlain wrote: post #54

Fantastic contribution back to the community; I look forward to trying it out. I must say this is the first time I've been disappointed with the quality of discussion on HN. For a community that promotes using the right tool for the job at the time , I would have thought people would be more open to the choices the early engineers made. I'm sure that Instagram are using a variety of tech across their stack.

Most people would expect software to crash, hang, be slow or somehow leak their personal information. That's normal behaviour for the products of the software industry. For a long time there have been efforts to ensure at least a degree of quality and robustness through processes, practices and verification tools. One such tool is a type system which allows encoding requirements and expectations that will be automati…

The reality borne out by the evidence [0][1] is that Python is at the very least perfectly suitable for the development of web stacks powering companies worth in excess of hundreds of million dollars.

Putting aside concrete technical issues regarding the python runtime’s performance envelope (eg: startup time, FFI inter-op call time, etc) and memory footprint, there is no reason not to use Python. Again, concrete technical issues aside, Python will not be an issue until your business is big enough that having to push the performance of your product is a nice problem to have and you will probably have the money to spend solving it by either optimising your python or rewriting parts of your application, as all these large companies have done.

0 - https://www.linkedin.com/pulse/top-10-sites-built-django-fra... 1 - https://worldwebtech.weebly.com/blog/top-ten-most-popular-we...

Re: Open-sourcing MonkeyType – Let your Python code type-hint itself

#173

Post by tjoff » Fri, Dec 22, 2017, 9:14 AM UTC

joshuamorton wrote: post #164

tjoff wrote: post #163

Earlier quoted context omitted.

You don't put auto everywhere. You write out the type in 99% of cases and save auto for 100+ character templated types. There is no reason to to save the literally 0.2s (you can still spend that time reasoning about your code) it takes to write the type. It is better for yourself writing it and for readability to be explicit.

By everywhere, I mean where it's otherwise obvious: auto s = "Hello world"; for (auto c: s) { cout Those autos don't need to exist, they're completely inferable, otherwise you wouldn't use auto. It's not like you can use auto in function declarations, nor should you, I agree.

I always write std::string etc. in those cases. It is consistent and quicker to read and there is not tangible benefit to using auto.

Re: Open-sourcing MonkeyType – Let your Python code type-hint itself

#174

Post by blub » Fri, Dec 22, 2017, 9:24 AM UTC

techdragon wrote: post #172

blub wrote: post #168

Earlier quoted context omitted.

Most people would expect software to crash, hang, be slow or somehow leak their personal information. That's normal behaviour for the products of the software industry. For a long time there have been efforts to ensure at least a degree of quality and robustness through processes, practices and verification tools. One such tool is a type system which allows encoding requirements and expectations that will be automati…

The reality borne out by the evidence [0][1] is that Python is at the very least perfectly suitable for the development of web stacks powering companies worth in excess of hundreds of million dollars. Putting aside concrete technical issues regarding the python runtime’s performance envelope (eg: startup time, FFI inter-op call time, etc) and memory footprint, there is no reason not to use Python. Again, concrete tec…

I'm not claiming that one can't be successful by using Python (or pretty much any programming language). Market success or user count are unfortunately not tightly correlated with technical excellence, as many of us have bitterly found out.

Development speed, maintainability, error count are also important development issues. Unfortunately we don't have much data to judge, but the little that we have such as this article indicate that dynamic typing has a non-negligible negative impact on the above.

""having to push the performance of your product is a nice problem to have and you will probably have the money to spend solving it by either optimising your python or rewriting parts of your application, as all these large companies have done.""

Not considering the topic at all is negligent. I don't understand why you're so sure that the only possible outcomes are either not reaching scale or having the money to optimise or rewrite.

First of all, not even Facebook had the money to rewrite their PHP code base, so that's probably out of the question. And it's very well possible that one will reach scale and not have the money (or worse the time) to optimise, if optimising means inventing type checkers.

At least one should spend some time thinking about this topic and picking a language that is flexible and can scale at least somewhat. Having to stop writing production code in order to invent a Ruby static type checker or native compiler is decidedly not a good problem to have.

Re: Open-sourcing MonkeyType – Let your Python code type-hint itself

#175

Post by semi-extrinsic » Fri, Dec 22, 2017, 9:29 AM UTC

mehrdadn wrote: post #131

Waterluvian wrote: post #122

Earlier quoted context omitted.

That's kind of missing the point though. Languages are tools for different purposes. You may love chizels and lathes and saws but you wouldn't build a suspension bridge with wood. Just like I wouldn't build a cabinet with cement, steel, and rebar. Picking C++ over Python is like picking woodworking over metalworking. Python being slow is never an issue unless someone is insisting on using the wrong tool for the job.

> Python being slow is never an issue unless someone is insisting on using the wrong tool for the job. So you think, for example, Numba (and everything that uses it) is misguided?

Numba is like laminating wood to build structural beams - it will get you close to the performance of metal, for some applications, if you can accept the weight increase etc.

Numba could be seen as misguided from some points of view. E.g. when using Python for high performance scientific computing, you will typically be writing your computational kernels, I/O etc. in some compiled, superfast language (C/Fortran/CUDA/whatnot) and all the input handling/case setup/etc. in Python. If 1% of your compute time is spent Python and 99% is carefully optimized C, Numba is obviously pointless.

But that's for one application. Python is used for so many different things that you can't make blanket statements like this.

Re: Open-sourcing MonkeyType – Let your Python code type-hint itself

#176

Post by blub » Fri, Dec 22, 2017, 9:31 AM UTC

ubernostrum wrote: post #151

sidlls wrote: post #112

Earlier quoted context omitted.

There's nothing here that tunes down the dynamism. Hints aren't statically checked or enforced. It's still possible to pass in an empty list to an int-hinted var and, e.g. have `if not var` evaluate to True (rather than raise an Exception). Type hints allow external tools to check some things, but at this point you're basically imposing static types so why not use a language with the tooling and optimizations to take…

Ah yes, this old chestnut: "(language I don't like) is only suitable for teeny-tiny puny baby child's toy programs, and once you're not writing those anymore you must use a big strong grown-up language like all the other Real Programmers™ do!" The empirical evidence of reality is against you: there are successful large (in terms both of codebase and contributors/development team) projects in these awful terrible chil…

It doesn't correlate with success, but the choice of language does correlate with development speed, number of faults, maintainability, etc.

The interesting thing to note is that a language that's perfectly acceptable at the above at small or medium scale might turn into a hindrance at large-scale. An otherwise fast to develop in language like Python won't be so fast if every change has to be painstakingly reviewed and tested due to the complexity of interactions in the code base.

Using a type system to verify assumptions/requirements is not a recipe for success, but it can improve reliability.

Re: Open-sourcing MonkeyType – Let your Python code type-hint itself

#177

Post by troels » Fri, Dec 22, 2017, 9:43 AM UTC

muglug wrote: post #85

troels wrote: post #5

Interestingly, I made a similar tool for php some time ago. It was recently revived and is now in active development to bring it up to speed with recent developments in the language. https://github.com/troelskn/phpweaver

That is fascinating. I’m the author of a static analysis tool for PHP that can generate types, but clearly not at the same level as runtime analysis. I’ll use it on my company’s codebase and report back.

It would be great if you can give any feedback. There are quite a few rough edges currently.

Re: Open-sourcing MonkeyType – Let your Python code type-hint itself

#178

Post by nickm12 » Fri, Dec 22, 2017, 9:51 AM UTC

maltalex wrote: post #13

Reading this as someone who writes mostly in statically typed languages, the whole exercise seems odd. Having so much dynamically typed code to maintain that you need to run production code using a separate tool just to figure out the types sounds just wrong. Why not use a statically typed language for such a large code-base? Is this done by purpose, or did they end up with a million lines of Python code and are look…

It's important to realize that these tools just didn't exist 7-10 years ago when companies like Instagram and Dropbox were getting started.

Though type inference has been around in so-called "academic" languages for decades, it hit a tipping point in the last 10 years, to the point that every major dynamic language has static type checkers or dialects (like Typescript) that support static checking. Meanwhile, even traditional statically typed languages are growing stronger type checkers.

Re: Open-sourcing MonkeyType – Let your Python code type-hint itself

#179

Post by walrus1066 » Fri, Dec 22, 2017, 10:00 AM UTC

andybak wrote: post #166

IshKebab wrote: post #69

Earlier quoted context omitted.

Navigating a large code base that is dynamically typed like Python is far more tedious than something like C++ or C#. First you can't read what the types passed into and out of functions are. You have to find their usages to work it out. Second, you can't reliable do things like "find usages" or "go to definition" because of the dynamic typing.

> Second, you can't reliable do things like "find usages" or "go to definition" because of the dynamic typing. In my experience PyCharm can do both correctly for the vast majority of cases.

Still very limited, for example, if I have:

def some_func(foo):

   foo.run()

   ...

Find usages in the run() method will return dozens of results, the IDE can't help you any more, to find what 'foo' is at runtime.

Re: Open-sourcing MonkeyType – Let your Python code type-hint itself

#180

Post by bennofs » Fri, Dec 22, 2017, 10:03 AM UTC

Yxven wrote: post #115

maltalex wrote: post #13

Reading this as someone who writes mostly in statically typed languages, the whole exercise seems odd. Having so much dynamically typed code to maintain that you need to run production code using a separate tool just to figure out the types sounds just wrong. Why not use a statically typed language for such a large code-base? Is this done by purpose, or did they end up with a million lines of Python code and are look…

Personally, I would love to ditch python for a language with strong types and type inference, but what is the replacement for django? Where do I find a well-designed, well-documented, battle-tested framework that I can easily hire developers for? Currently, I think the nearest competitor is node with typescript, and I'd rather stick with python. Please tell me if I'm wrong.

Java? But that's another kind of hell