Live data from Hacker News

Python at Scale: Strict Modules

instagram-engineering.com

241–250 of 259 posts

Re: Python at Scale: Strict Modules

#241

Earlier quoted context omitted.

As far as I understand it, RPython isnt' really meant for actually writing programs in: > Do I have to rewrite my programs in RPython? > No, and you shouldn’t try. First and foremost, RPython is a language designed for writing interpreters. It is a restricted subset of Python. If your program is not an interpreter but tries to do “real things”, like use any part of the standard Python library or any 3rd-party library…

Hmm, still it might make a good foundation for the sort of thing you're talking about, eh? Cheers.

For sure!

Re: Python at Scale: Strict Modules

#242
post #229

Earlier quoted context omitted.

There's less conflict if you release the features in the same compiler of course, but it's definitely not "perfectly fine". My blog posts give some more color on that, but also see: https://discuss.ocaml.org/t/the-future-of-ppx/3766 (breakage) https://words.steveklabnik.com/an-overview-of-macros-in-rust (breakage) Also, OCaml has had at least 4 different metaprogramming systems -- ocamlmeta, camlp4, ppx, etc. Rust is…

There isn't anything specific of static type systems in those two links. Both ocaml and rust had powerful plugin systems that exposed the internal of the the compiler AST and they decided that they do not want to expose it as a stable interface. But exposing implementation details is not required to have a powerful metaprogramming environment. As far as I understand rust macros never did expose these internals and ha…

The point is that they all struggle with the design of metaprogramming. Why are there 4+ different systems in OCaml as "addons" whereas in Lisp and Python it's integrated in the language?

It's exactly analogous to types being integrated in OCaml, Rust, etc. whereas in Python, JS, Ruby, and PHP static typing it's an "add-on".

Scala also has a newer system https://scalameta.org/

C++ has new dramatic new proposals 20+ years after templates, addressing really basic use cases:

https://herbsutter.com/2017/07/26/metaclasses-thoughts-on-ge...

The point is that no language has gotten it right so they keep introducing new systems and breaking old ones.

C++ templates also have the mistake where type checking is done after template expansion. That's why you get terrible error messages.

If you're not convinced, that's OK, but try watching the talk by Yaron Minsky here:

http://www.oilshell.org/blog/2016/12/05.html

Re: Python at Scale: Strict Modules

#243

Earlier quoted context omitted.

These are all fair points. I really enjoy Python, but there are too many things I fight with on a regular basis that simply aren’t issues in Go. It could be so much better if (1) there was a better type system (mypy is unnecessarily shoehorned into the syntax and still very broken—can’t even express recursive types like JSON), (2) a good way to constrain the dynamism so performance could be improved, and (3) a better…

> type-system I agree, but if you for instance look at the TypeScript comparison sub-thread, you'll see that all the issues with both the syntax and implementation of the type-system are being aggressively resolved, and likely will all be so by 3.9. > Good way to constrain the dynamism so performance could be improved Couldn't agree more! > environment I find poetry a joy to use. If you want to bypass venvs all toget…

> I agree, but if you for instance look at the TypeScript comparison sub-thread, you'll see that all the issues with both the syntax and implementation of the type-system are being aggressively resolved, and likely will all be so by 3.9.

I'm happy to hear that; hopefully the efforts really do address these issues well.

> I find poetry a joy to use. If you want to bypass venvs all together, there's a lot of work to make that a reality, such as https://github.com/David-OConnor/pyflow.

I'll have to check those out, but one inherent problem is that even if these tools really do solve my pain points, adopting them means I'm leaving my org on a relatively small island, isolated from the Python community. If these really are the holy grail, why isn't the broader Python community adopting them? Please don't take this as me looking for something wrong--whatever Python build tool I use, I'll eventually need support and there's a lot to be said for having a thriving community that has almost always run into my exact problem before.

> Python in 3.5 added complete zip app support, which has improved this dramatically from my perspective. Extended by things like shiv https://github.com/linkedin/shiv make it fairly complete.

We're currently using this via pex. It mostly works, but we still run into problems occassionally (system dependencies, for example). Figuring out how to integrate these tools into the broader build process is another problem to solve--we're using `pants` which supports pex out of the box, but we're running into lots of bugs or other problems. I'll keep an eye on shiv.

> This is interesting to me. I prefer async/await in general, because it has become a standard across programming languages and I find it really easy to reason about. I also find channels to be too widely seen as a cure-all, when the only study so far has shown they actually led to an increase bug count. But I don't discount the value of real-parallelism, and am glad to see that Python has been pushing harder on that lately, with things like subshells that allow bypassing the GIL on a single thread.

My biggest issues with async/await are

(1) every package needs an async variant (async boto, async docker, etc etc). We work around this by running them in a thread pool executor, and I think that works, but I don't know if I'm holding the GIL unnecessarily and causing performance issues (fundamentally difficult to diagnose). This is roughly the "what color is my function" problem.

(2) it's really easy to starve the event loop by calling into something that transitively makes a sync call or otherwise just does a lot of CPU-heavy work. We've run into both kinds in production and they've been really hard to troubleshoot (because the requests that time out often aren't the ones that are actually causing the problems).

(3) dynamic typing means it's super easy to forget to await things. Tests should catch this, but we find ourselves writing tests _just_ to catch this (e.g., we now write tests for entrypoints that _just_ `await lib_function(params)`; we would normally not write tests for such simple functions, but now we have to). Static typing is the right way to solve this and mypy does, but mypy has too many other issues (at the moment) for our org.

One substantial criticism of goroutines is that they're less safe than async/await because you need to make sure the code you're running is threadsafe. I appreciate this criticism, but I think it's the right tradeoff for Go's performance aspirations (another great high-performance alternative is Rust's borrow checker, but that's the wrong tradeoff for Go's developer productivity aspirations).

Re: Python at Scale: Strict Modules

#244

Earlier quoted context omitted.

The problem here is that there is boilerplate at all. There shouldn't be. Boilerplate distracts from what is actually going on. I can generally identify code smells from the shape of python code (like, blur all the text so I can't read the words, and the shape of the blocks tells me everything I need), I can't do the same in go, because there's so much more indentation and visual stuff happening, and most of it (boil…

Like I said, I disagree with this. I suspect this is either because you're very experienced with Python and relatively inexperienced with Go, or perhaps you're simply an outlier. I think if you surveyed developers who are very experienced with Python and have at least a few months of experience with Go, you'll find people say that it's easier to identify issues in Go code--and I think this largely comes down to the r…

At this point I've written fairly little go code, but reviewed quite a bit. Among those I work with, my opinion seems to be shared.

> I think you'll find that visual structure is actually important.

I didn't say otherwise. What I did say is that go adds visual noise that isn't present in python. (and it is noise: the proposal to add try! shows that the error handling style is noisy. It can be basically entirely removed by an automated transformation). Actual pattern matching like rust has, or even what Google C++ has with StatusOr and [1] our nonsense RETURN_IF_ERROR macros are better than what go does, and just as explicit (actually often moreso, since its more difficult to forget an error condition)

[1]: https://github.com/protocolbuffers/protobuf/blob/d0f91c863ae...

Re: Python at Scale: Strict Modules

#245

Earlier quoted context omitted.

> type-system I agree, but if you for instance look at the TypeScript comparison sub-thread, you'll see that all the issues with both the syntax and implementation of the type-system are being aggressively resolved, and likely will all be so by 3.9. > Good way to constrain the dynamism so performance could be improved Couldn't agree more! > environment I find poetry a joy to use. If you want to bypass venvs all toget…

> I agree, but if you for instance look at the TypeScript comparison sub-thread, you'll see that all the issues with both the syntax and implementation of the type-system are being aggressively resolved, and likely will all be so by 3.9. I'm happy to hear that; hopefully the efforts really do address these issues well. > I find poetry a joy to use. If you want to bypass venvs all together, there's a lot of work to ma…

> I'll have to check those out, but one inherent problem is that even if these tools really do solve my pain points, adopting them means I'm leaving my org on a relatively small island, isolated from the Python community. If these really are the holy grail, why isn't the broader Python community adopting them? Please don't take this as me looking for something wrong--whatever Python build tool I use, I'll eventually need support and there's a lot to be said for having a thriving community that has almost always run into my exact problem before.

Only because they are so new. portray was built a few weeks ago and already has a thriving community building around it - but of course it's still a small drop of the whole ecosystem. Older tools I've built like isort, are now ingrained into the community: https://github.com/timothycrosley/isort, but that took years, even without major issues or complaints being present. It just takes people time to adopt new things.

Re: Python at Scale: Strict Modules

#246
post #189
post #147

Avoiding module side effects and making classes and modules immutable seem like two separate concerns

Not really. Mutation in general, and in modules in particular, inhibit a lot of reasoning about the code, and thus stops a whole lot of optimizations from being possible. Guile (a scheme dialect) recently got declarative modules for that reason, where a top level binding cannot change (i.e. you cannot set! a binding, but you can wrap in it a mutable container and change the contents of that container). This makes pro…

To quote the article (from.memory): "adding static modules is probably the single most important optimization guile can do in the near future".

The quote is probably wrong, but it is right in spirit.

Re: Python at Scale: Strict Modules

#247

Earlier quoted context omitted.

Afaik, typescript is pretty bad in terms of catching some basic errors. Types are not enforced. A caller can change sync function to async, breaking the functionality downstream.

Yes gradually typed languages are usually looser/more flexible about static typing. > A caller can change sync function to async, breaking the functionality downstream. I think you mean callee? Are you taking about using a returned result? Because most languages permit ignoring return values. If you want to check out promise use, check out https://tsetse.info/must-use-promises

Sorry, I did mean callee.

Consider

``` const myIntValue = f(); ```

This code will silently break when f changes from sync to async.

Re: Python at Scale: Strict Modules

#248
post #159

Earlier quoted context omitted.

On that note, I'd include Erlang. It's not gradually typed, per se, but you can have a fully dynamic language (no type specs, no Dialyzer), a completely optimistic static analyzer for inferring types and warning where it's inconsistent (Dialyzer runs), and then you can add specs where needed to tighten up and improve what Dialyzer can catch, to basically be a fully static language.

Dialazer is pretty bad and not even close to a static language.

If you fully spec out your code, it's actually quite close. In a project we did that in, the only type errors we encountered were ones that a static system would not have caught either (due to their being caused by incoming data that did not conform to our type expectations; for instance, deserializing JSON to a specific type).

Without specs, it will assume every type is 'any()', unless it has information to infer something more stringent. For instance, if it sees you add 5 to it somewhere, it will instead assume it is a number. Etc. Even if in practice it actually is a list of some kind (and so that addition of 5 will fail). Which, yes, ain't great. Hence why I said it was a gradual transition; it will catch provable errors (i.e., if you call append on that same variable as above, it will note that there is no type that allows both append, and + an integer, and error), but leave plenty of things uncaught that could have been caught had it known the type in question (via a type spec).

Re: Python at Scale: Strict Modules

#249

Earlier quoted context omitted.

On that note, I'd include Erlang. It's not gradually typed, per se, but you can have a fully dynamic language (no type specs, no Dialyzer), a completely optimistic static analyzer for inferring types and warning where it's inconsistent (Dialyzer runs), and then you can add specs where needed to tighten up and improve what Dialyzer can catch, to basically be a fully static language.

It's relatively easy, but not free, to do this. I find that the erlang (and elixir) guides seem to be a bit scant on best practices to achieve this level of discipline, for example, wrapping all gen_sever calls in module functions and presenting a well-defined API for the genserver module (and possibly, even linting for no naked genserver calls) is not really explained in this light. Similarly guidance is not provide…

Yeah; it requires some rigor to do. My point was simply that it _can_ be done, and while the effort is high, it does allow you to move from pure dynamic language, to highly defined type checking.

Re: Python at Scale: Strict Modules

#250
post #206
post #200

Earlier quoted context omitted.

Kotlin might become the default Java syntax, that's what the GP is saying. And I agree.

Java is Java. On what concerns the JVM, Kotlin hype will be over in a couple of years, and it will get as much use as Scala, Clojure, Beanshell, Groovy enjoy nowadays. Guest languages never get to own a platform, and with time all platform languages end up getting enough features that the large majority of developers never bother with extra tooling, debugging layer and idiomatic wrapper libraries of the guest languag…

> Guest languages never get to own a platform, and with time all platform languages end up getting enough features

They do up to the point where they differ philosophically. Java is never going to turn in to a Clojure, nor is it going to adopt the type of dynamic scripting features Groovy offers.

Kotlin and Scala are more at risk in that way.

Post reply on HN