Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Python at Scale: Strict Modules

instagram-engineering.com

Post reply on HN
Print view View original

61–70 of 259 posts

Re: Python at Scale: Strict Modules

#61

Post by ledauphin »

staticassertion wrote: post #34
timothycrosley wrote: post #18

More and more I want someone to create a new language that amounts to a strict subset of Python, with mypy built-in, and is compilable into machine code. Python has by far my favorite syntax, community, and in my experience leads to the greatest productivity. There just happens to be a lot of overly dynamic features, that aren't even used by most, but used just enough to hold back optimization and structural improvem…

https://github.com/python/mypy/tree/master/mypyc This may interest you. This is probably going to be the 'official' way to get what you're talking about.

yeah, I'm excited about this approach. but it's a long way from being a realistic approach for folks outside Dropbox, it seems.

Re: Python at Scale: Strict Modules

#62

Post by nine_k »

timothycrosley wrote: post #46
TylerE wrote: post #37

Earlier quoted context omitted.

I don't think there is any sort of long-term future in anything "Python". I think a successful modern language has to have the potential for efficient concurrency baked in, which isn't really possible without breaking compatibility, and the Python community would never survive another round like the 2->3 transition. (And I'm not convinced the community really survived that one either, given the amount of ongoing bitt…

> I think a successful modern language has to have the potential for efficient concurrency baked in I agree with this! > I don't think there is any sort of long-term future in anything "Python". I disagree with this :) I think Python has efficient IO concurrency built-in already with async, and I feel it is likely that it finds a way to work out CPU bound concurrency long-term, as projects like sub-shells with channe…

Efficient CPU concurrency needs either fine-grained locking (and removal of GIL), or data immutability.

Both seem rather hard to implement.

I predict that for CPU-intensive tasks, you'll keep using extensions in native code (like numpy or pytorch), or keep passing serialized objects through queues in multiprocessing setups.

Re: Python at Scale: Strict Modules

#63

Post by ledauphin »

weberc2 wrote: post #47
timothycrosley wrote: post #18

More and more I want someone to create a new language that amounts to a strict subset of Python, with mypy built-in, and is compilable into machine code. Python has by far my favorite syntax, community, and in my experience leads to the greatest productivity. There just happens to be a lot of overly dynamic features, that aren't even used by most, but used just enough to hold back optimization and structural improvem…

Sounds like Go. ;) This is a cheeky remark, but I use Python and Go, and Go very much feels like an improved Python in most ways. Especially when it comes to static analysis, build tooling, distribution, performance, etc. In particular, I love that there are no venvs, pipenvs, virtualenvs, pyenvs, wheels, eggs, setuptools, easy_installs, etc.

Go may "feel" like Python, but it's almost nothing like Python in actual practice. It's not dynamic (and doesn't even have generics), and its error handling is dramatically different.

Re: Python at Scale: Strict Modules

#65

Post by ledauphin »

rectangletangle wrote: post #4
ledauphin wrote: post #2

I love the idea, but it feels like just an idea at this point. I'd rather read about them releasing their 'compile-time' analyzer and revealing their measurements for how much startup time it saves. In our codebase, we have pretty strict developer-enforced rules about not doing I/O at the module level, usually through the use of simple "Lazy" wrappers for module-level objects. I'd be curious to know what other approa…

It is an interesting approach, though I feel like this could introduce some nasty unintended consequences given how dynamic and introspective Python can be (admittedly I haven't studied this particular implementation). I always treated this a bit like single underscore private functions/methods, i.e., follow a convention that produces code that's easy to reason about, even if it's not strictly enforced by the languag…

well, we also heavily use static typing, so you end up with something like

my_db_conn: Lazy[DbConn] = Lazy(lambda: make_db_conn(...))

and MyPy will tell you if you're doing something silly when you try to use it.

EDIT: After typing up this response and submitting I realize you were talking about their strict approach rather than ours. whoops :)

Re: Python at Scale: Strict Modules

#66

Post by jedberg »

TylerE wrote: post #23
miki123211 wrote: post #21

This is yet another example of the divide between wizarding and engineering[1]. When you're a small startup, what matters is the expressiveness of your language, and the ability do do a lot of things very very quickly. Type safety, performance, readability, those things don't matter. You're just a bunch of engineers who know the whole codebase inside out, you're pretty certain of what you're doing. In short, you're w…

Who is starting large-scale new projects in Java in 2019?

A big chuck of Netflix is written in Java, and they make new stuff in Java all the time.

Re: Python at Scale: Strict Modules

#67

Post by [deleted] »

nerdponx wrote: post #17
allan_s wrote: post #11

> This means that just by importing this module, we're mutating global state somewhere else. Yes, this ! That's why I hate Django and some flask app the most for, the fact that by importing a module, you're implicitly creating a database connection, and a lot of other magic stuff, which mean that now I can't import a constant defined in said module outside of `python manage.py` Also as said below in the article, sudd…

I much prefer frameworks/modules for which code is executed only once you invoke their "setup" function Django _does_ have a "setup" function. You can't import and use Django database connections outside of a running application without it. Flask also has a "run" method and does no i/o without it.

[deleted]

Re: Python at Scale: Strict Modules

#68

Post by hsaliak »

TylerE wrote: post #23
miki123211 wrote: post #21

This is yet another example of the divide between wizarding and engineering[1]. When you're a small startup, what matters is the expressiveness of your language, and the ability do do a lot of things very very quickly. Type safety, performance, readability, those things don't matter. You're just a bunch of engineers who know the whole codebase inside out, you're pretty certain of what you're doing. In short, you're w…

Who is starting large-scale new projects in Java in 2019?

Google

Re: Python at Scale: Strict Modules

#69

Post by nine_k »

weberc2 wrote: post #47
timothycrosley wrote: post #18

More and more I want someone to create a new language that amounts to a strict subset of Python, with mypy built-in, and is compilable into machine code. Python has by far my favorite syntax, community, and in my experience leads to the greatest productivity. There just happens to be a lot of overly dynamic features, that aren't even used by most, but used just enough to hold back optimization and structural improvem…

Sounds like Go. ;) This is a cheeky remark, but I use Python and Go, and Go very much feels like an improved Python in most ways. Especially when it comes to static analysis, build tooling, distribution, performance, etc. In particular, I love that there are no venvs, pipenvs, virtualenvs, pyenvs, wheels, eggs, setuptools, easy_installs, etc.

What Go adds in tooling and performance, it takes away in expressivity.

What takes 3 lines in Python, takes 10-30 on Go.

Re: Python at Scale: Strict Modules

#70

Post by yen223 »

nerdponx wrote: post #17
allan_s wrote: post #11

> This means that just by importing this module, we're mutating global state somewhere else. Yes, this ! That's why I hate Django and some flask app the most for, the fact that by importing a module, you're implicitly creating a database connection, and a lot of other magic stuff, which mean that now I can't import a constant defined in said module outside of `python manage.py` Also as said below in the article, sudd…

I much prefer frameworks/modules for which code is executed only once you invoke their "setup" function Django _does_ have a "setup" function. You can't import and use Django database connections outside of a running application without it. Flask also has a "run" method and does no i/o without it.

Without calling setup, you cannot import anything that touches Django models, like constants defined in a file that transitively imports a Django model.

In practice, this means that any script that depends indirectly on Django code will incur a lengthy startup cost (from having to call setup()), and will fail to run if there's no database connection, even if the script itself doesn't need the db.

Post reply on HN