Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Python at Scale: Strict Modules

instagram-engineering.com

Post reply on HN
Print view View original

11–20 of 259 posts

Re: Python at Scale: Strict Modules

#11

Post by allan_s »

> This means that just by importing this module, we're mutating global state somewhere else.

Yes, this !

That's why I hate Django and some flask app the most for, the fact that by importing a module, you're implicitly creating a database connection, and a lot of other magic stuff, which mean that now I can't import a constant defined in said module outside of `python manage.py`

Also as said below in the article, suddenly it's much harder to handle smoothly the "the database is momentary unavailable" (because someone has put the line starting the database connection in the global space of a module somewhere)

I much prefer frameworks/modules for which code is executed only once you invoke their "setup" function

Re: Python at Scale: Strict Modules

#12

Post by rectangletangle »

brenden2 wrote: post #9
b3orn wrote: post #7

Earlier quoted context omitted.

Python is strongly typed, just not static.

Python uses duck typing: https://en.wikipedia.org/wiki/Duck_typing I would categorize it as a subset of dynamic typing, and that's what Wikipedia says too.

Dynamic typing vs. static typing is on a different axis than strong vs. weak typing. Python is a strong dynamically typed language, with some "static lite" features introduced in Python 3.

Dynamic typing means that types can be changed arbitrarily at runtime, compared to statically typed languages which define all types at compile time.

Strong/weak means that type coercions rarely/never happen automatically. For instance JS has some interesting behavior enabled by weak typing `[] + [] -> ""`. Whereas Python rarely coerces things for you. The division operator in Python 2 was strongly typed, while they changed it to weak typing in Python 3 (inline with the practicality vs. purity convention).

Re: Python at Scale: Strict Modules

#13

Post by tummybug »

allan_s wrote: post #11

> This means that just by importing this module, we're mutating global state somewhere else. Yes, this ! That's why I hate Django and some flask app the most for, the fact that by importing a module, you're implicitly creating a database connection, and a lot of other magic stuff, which mean that now I can't import a constant defined in said module outside of `python manage.py` Also as said below in the article, sudd…

I'm not sure about django but flasks Application object has a before_first_request method which takes a function designed to do this type of initialization operations.

Re: Python at Scale: Strict Modules

#14

Post by tom_mellior »

brenden2 wrote: post #9
b3orn wrote: post #7

Earlier quoted context omitted.

Python is strongly typed, just not static.

Python uses duck typing: https://en.wikipedia.org/wiki/Duck_typing I would categorize it as a subset of dynamic typing, and that's what Wikipedia says too.

"strong typing": everything has a type and cannot be accessed at some other type; "static typing": everything's type can be determined statically (according to one definition)

In Python everything has a type, and you can't use a float as a list, for instance. It's correct to call it both strongly typed and dynamic, those are not antonyms.

Re: Python at Scale: Strict Modules

#15

Post by rectangletangle »

allan_s wrote: post #11

> This means that just by importing this module, we're mutating global state somewhere else. Yes, this ! That's why I hate Django and some flask app the most for, the fact that by importing a module, you're implicitly creating a database connection, and a lot of other magic stuff, which mean that now I can't import a constant defined in said module outside of `python manage.py` Also as said below in the article, sudd…

I'm a huge fan of Django, but I always felt that this was true. I wish there was more of a push to decouple parts of the framework. Keep the magic, but allow usage without it.

Re: Python at Scale: Strict Modules

#16

Post by ben509 »

> How do we know that the log_to_network or route functions are not safe to call at module level? We assume that anything imported from a non-strict module is unsafe, except for certain standard library functions that are known safe.

It's hard to know anything about the stdlib as it can be monkey patched, e.g. [1]

That said, you could solve this with diagnostics; calculate signatures of stdlib functions and classes to find any known safe ones that were patched. Run that check in your test suite to find problematic imports.

> If the utils module is strict, then we’d rely on the analysis of that module to tell us in turn whether log_to_network is safe.

I like this. It seems far more usable than proposals like adding const decorators.[2]

[1]: https://github.com/gevent/gevent/blob/master/src/gevent/monk...

[2]: https://github.com/python/typing/issues/242

Re: Python at Scale: Strict Modules

#17

Post by nerdponx »

allan_s wrote: post #11

> This means that just by importing this module, we're mutating global state somewhere else. Yes, this ! That's why I hate Django and some flask app the most for, the fact that by importing a module, you're implicitly creating a database connection, and a lot of other magic stuff, which mean that now I can't import a constant defined in said module outside of `python manage.py` Also as said below in the article, sudd…

I much prefer frameworks/modules for which code is executed only once you invoke their "setup" function

Django _does_ have a "setup" function. You can't import and use Django database connections outside of a running application without it.

Flask also has a "run" method and does no i/o without it.

Re: Python at Scale: Strict Modules

#18

Post by timothycrosley »

More and more I want someone to create a new language that amounts to a strict subset of Python, with mypy built-in, and is compilable into machine code. Python has by far my favorite syntax, community, and in my experience leads to the greatest productivity. There just happens to be a lot of overly dynamic features, that aren't even used by most, but used just enough to hold back optimization and structural improvement.

Re: Python at Scale: Strict Modules

#19

Post by TylerE »

timothycrosley wrote: post #18

More and more I want someone to create a new language that amounts to a strict subset of Python, with mypy built-in, and is compilable into machine code. Python has by far my favorite syntax, community, and in my experience leads to the greatest productivity. There just happens to be a lot of overly dynamic features, that aren't even used by most, but used just enough to hold back optimization and structural improvem…

nim

Re: Python at Scale: Strict Modules

#20

Post by ken »

ainar-g wrote: post #10
brenden2 wrote: post #5

It still blows my mind that people don't use strongly typed languages in the first place and spare themselves from all this future pain. My guess (based on my experiences) is that companies wind up in this position from having inexperienced people building early versions of products instead of hiring experienced engineers (who are usually more expensive).

It's a constant struggle against the current. Dynamically-typed languages are often “good enough for the time being”. I have the same issue explaining to our C/C++/Obj-C team why they should use static (Clang-Tidy, Infer, PVS-Studo) and dynamic (ASan, MSan, UBSan) analysis tools. They just keep giving me basically the same response of “I am a good programmer, and my code is good, and shame on you for even daring to t…

I'm confused. It should be easy to demonstrate the benefit, if there is one. Just show them the bugs!

For me, it's not "status anxiety". It's simply not worth the effort.

The last couple static analysis tools I ran on my programs, I spent a while getting the tool to not-crash (because even though the authors obviously had a static analysis tool themselves, they either didn't bother to run it on their own code, or it wasn't good enough to find actual issues). These tools flagged only a couple issues, and almost all of them were places where it couldn't really cause any problems, but the type system was not strong enough for me to prove why it couldn't go bad. So I spent a while sorting through false-positives.

I'm not going to spend hours with a tool to find only a couple (real) bugs, which no user has ever reported seeing, and which I've gotten no automated crash reports about. I have much better uses for my time.

Post reply on HN