Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Python at Scale: Strict Modules

instagram-engineering.com

Post reply on HN
Print view View original

151–160 of 259 posts

Re: Python at Scale: Strict Modules

#151

Post by weberc2 »

nine_k wrote: post #69
weberc2 wrote: post #47

Earlier quoted context omitted.

Sounds like Go. ;) This is a cheeky remark, but I use Python and Go, and Go very much feels like an improved Python in most ways. Especially when it comes to static analysis, build tooling, distribution, performance, etc. In particular, I love that there are no venvs, pipenvs, virtualenvs, pyenvs, wheels, eggs, setuptools, easy_installs, etc.

What Go adds in tooling and performance, it takes away in expressivity. What takes 3 lines in Python, takes 10-30 on Go.

Yeah, but typically only locally. Like using a for loop instead of a list comprehension, or handling errors. So more keystrokes, but in most cases not more complexity. In some cases (generic programming), Python really is more expressive, but those ~5% of cases aren’t worth the tooling/perf/maintainability tradeoffs most of the time.

Re: Python at Scale: Strict Modules

#152

Post by jbmsf »

I like the idea, but it feels a bit heavy handed outside of a very large team.

I think the first step here is to get away from the assumption that importing a module will have "interesting" side effects. This is not only a problem with Python...

I tend to create mini "dependency injection" frameworks that create a pattern for loading module code at some point well after import. This patterns tends to reduce to wrapping whatever code you have in the module in a function/closure instead of just running whenever.

Again, I like the idea of enforcing constraints with code, but I don't think it's a substitute for educating developers to avoid certain patterns and giving them infrastructure that makes the alternative easy.

Re: Python at Scale: Strict Modules

#153

Post by weberc2 »

ledauphin wrote: post #63
weberc2 wrote: post #47

Earlier quoted context omitted.

Sounds like Go. ;) This is a cheeky remark, but I use Python and Go, and Go very much feels like an improved Python in most ways. Especially when it comes to static analysis, build tooling, distribution, performance, etc. In particular, I love that there are no venvs, pipenvs, virtualenvs, pyenvs, wheels, eggs, setuptools, easy_installs, etc.

Go may "feel" like Python, but it's almost nothing like Python in actual practice. It's not dynamic (and doesn't even have generics), and its error handling is dramatically different.

It _is_ like Python in practice (I use both languages all the time). That’s largely why you see it used in many of the same places as Python. It has dynamic features by way of interface{}, which is every bit as “generic” as what Python has to offer. :) But yes, the error handling is different—values vs exceptions.

Re: Python at Scale: Strict Modules

#154

Post by nine_k »

weberc2 wrote: post #151
nine_k wrote: post #69

Earlier quoted context omitted.

What Go adds in tooling and performance, it takes away in expressivity. What takes 3 lines in Python, takes 10-30 on Go.

Yeah, but typically only locally. Like using a for loop instead of a list comprehension, or handling errors. So more keystrokes, but in most cases not more complexity. In some cases (generic programming), Python really is more expressive, but those ~5% of cases aren’t worth the tooling/perf/maintainability tradeoffs most of the time.

It depends.

Code reviewers glazing over copy-pasted boilerplate blocks can more easily lose track of the whole, and miss an error which is obvious when the whole is expressed in 10 lines.

There is some optimal range of expressive density for comfortable use by humans. APL or K likely above that level, and Go feels below it, not as low as COBOL, but still.

Re: Python at Scale: Strict Modules

#155

Post by timothycrosley »

weberc2 wrote: post #151
nine_k wrote: post #69

Earlier quoted context omitted.

What Go adds in tooling and performance, it takes away in expressivity. What takes 3 lines in Python, takes 10-30 on Go.

Yeah, but typically only locally. Like using a for loop instead of a list comprehension, or handling errors. So more keystrokes, but in most cases not more complexity. In some cases (generic programming), Python really is more expressive, but those ~5% of cases aren’t worth the tooling/perf/maintainability tradeoffs most of the time.

I'm curious what you see as the tooling benefits of go?

Re: Python at Scale: Strict Modules

#156

Post by timothycrosley »

carapace wrote: post #58
timothycrosley wrote: post #31

Earlier quoted context omitted.

I use Cython a lot! But mostly to speed up existing Python code, and build C-extensions faster. I don't see it as a strict subset of Python or a new language to build a community around. Nuitka I just started experimenting with to build standalone Python executable, and I really like the direction and roadmap they are following. In the end though both of these technologies seem like ways to somewhat speedup existing…

What about RPython? https://rpython.readthedocs.io/en/latest/rpython.html

As far as I understand it, RPython isnt' really meant for actually writing programs in:

> Do I have to rewrite my programs in RPython?

> No, and you shouldn’t try. First and foremost, RPython is a language designed for writing interpreters. It is a restricted subset of Python. If your program is not an interpreter but tries to do “real things”, like use any part of the standard Python library or any 3rd-party library, then it is not RPython to start with. You should only look at RPython if you try to write your own interpreter.

See: https://rpython.readthedocs.io/en/latest/faq.html

Re: Python at Scale: Strict Modules

#157

Post by dnautics »

lostcolony wrote: post #130
bcherny wrote: post #32

Earlier quoted context omitted.

> I wish there was a language that let you move gradually from one end to the other, exactly when you need to. This is precisely what gradually typed languages — like TypeScript, Flow, and typed Pythons — solve! I talked about this on Software Engineering Radio last week: https://www.se-radio.net/2019/10/episode-384-boris-cherny-on... .

On that note, I'd include Erlang. It's not gradually typed, per se, but you can have a fully dynamic language (no type specs, no Dialyzer), a completely optimistic static analyzer for inferring types and warning where it's inconsistent (Dialyzer runs), and then you can add specs where needed to tighten up and improve what Dialyzer can catch, to basically be a fully static language.

It's relatively easy, but not free, to do this. I find that the erlang (and elixir) guides seem to be a bit scant on best practices to achieve this level of discipline, for example, wrapping all gen_sever calls in module functions and presenting a well-defined API for the genserver module (and possibly, even linting for no naked genserver calls) is not really explained in this light. Similarly guidance is not provided for wrapping enum module calls (since that similarly destroys typing information)

Re: Python at Scale: Strict Modules

#158

Post by [deleted] »

miki123211 wrote: post #21

This is yet another example of the divide between wizarding and engineering[1]. When you're a small startup, what matters is the expressiveness of your language, and the ability do do a lot of things very very quickly. Type safety, performance, readability, those things don't matter. You're just a bunch of engineers who know the whole codebase inside out, you're pretty certain of what you're doing. In short, you're w…

[deleted]

Re: Python at Scale: Strict Modules

#159

Post by Thaxll »

lostcolony wrote: post #130
bcherny wrote: post #32

Earlier quoted context omitted.

> I wish there was a language that let you move gradually from one end to the other, exactly when you need to. This is precisely what gradually typed languages — like TypeScript, Flow, and typed Pythons — solve! I talked about this on Software Engineering Radio last week: https://www.se-radio.net/2019/10/episode-384-boris-cherny-on... .

On that note, I'd include Erlang. It's not gradually typed, per se, but you can have a fully dynamic language (no type specs, no Dialyzer), a completely optimistic static analyzer for inferring types and warning where it's inconsistent (Dialyzer runs), and then you can add specs where needed to tighten up and improve what Dialyzer can catch, to basically be a fully static language.

Dialazer is pretty bad and not even close to a static language.

Re: Python at Scale: Strict Modules

#160

Post by Thaxll »

coldtea wrote: post #79
TylerE wrote: post #39

Earlier quoted context omitted.

I asked for examples , not platitudes.

And yet you only offered a misinformed platitude in the form of a question. Google does all kind of new Java work (Golang contrary to myth, is just one of the languages Google uses for internal stuff, and niche at that), Amazon of course, most of Apple's backend services are Java, Twitter, AirBnB, Uber, LinkedIn, TripAdvisor, and tons of others use Java, and write green stuff in it all the time...

Uber back end is mostly Go.
Post reply on HN