Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Python at Scale: Strict Modules

instagram-engineering.com

Post reply on HN
Print view View original

41–50 of 259 posts

Re: Python at Scale: Strict Modules

#41

Post by zitterbewegung »

miki123211 wrote: post #21

This is yet another example of the divide between wizarding and engineering[1]. When you're a small startup, what matters is the expressiveness of your language, and the ability do do a lot of things very very quickly. Type safety, performance, readability, those things don't matter. You're just a bunch of engineers who know the whole codebase inside out, you're pretty certain of what you're doing. In short, you're w…

You can write fortran in any language. [1] https://blog.codinghorror.com/you-can-write-fortran-in-any-l... [2] https://queue.acm.org/detail.cfm?id=1039535

Re: Python at Scale: Strict Modules

#42

Post by dmix »

TylerE wrote: post #37
scrollaway wrote: post #30

Earlier quoted context omitted.

Python has some of my favourite syntax as well but I absolutely hate its annotations. TypeScript got typings right. I think the killer language will be typescript with access to both the python and JavaScript ecosystems. We'll see what that looks like. And of course if something changes the syntax, better anonymous functions will be the absolute first thing I would look for...

I don't think there is any sort of long-term future in anything "Python". I think a successful modern language has to have the potential for efficient concurrency baked in, which isn't really possible without breaking compatibility, and the Python community would never survive another round like the 2->3 transition. (And I'm not convinced the community really survived that one either, given the amount of ongoing bitt…

Data science/AI people breathed a whole lot of life into Python. I'm curious where it would be if they had went elsewhere.

Re: Python at Scale: Strict Modules

#43

Post by tahdig »

> ... many of whom are new to Python.

well, if you ask me to write language X, I would definitely make mistakes for the first couple of weeks/months/years, that is why you need code review, mentoring and education plans for your hires.

> Here’s another thing we often find developers doing at import time: fetching configuration from a network configuration source.

  MY_CONFIG = get_config_from_network_service()
I am pretty sure this an anti-pattern, if this code passed the code review, you should make your review process more strict.

  def myview(request):
    SomeClass.id = request.GET.get("id")
> Likely you’ve already spotted the problem

Well, yes, why would you do this? why would this pass code review? why do we we have linters and other checks for dynamic languages

> It works great for smaller teams on smaller codebases that can maintain good discipline around how to use it, and we should switch to a less dynamic language.

It seems we are here blaming python for shortcomings of a monolith also, instead of chunking out specific businesses modules to separate services/micro-services.

TO be honest the strict mode seems interesting, but I believe the problems they seem to be facing can be solved by a couple of changes to their pocess and code:

- everyone gets a mentor if they are not experienced in python or django

- code review atleast by two experienced python developers(does not count if you have coded for Java for 20 years)

- teams should try to move their logic outside the monolith(it sounds like they have a monolith)

- write CI tests to measure how much time it takes to import a file, if it takes more than T(line count * LINE_PROCESSING_THRESHOLD) you have to fix your code.

- prepare config and load it before running the actual server, no network call for getting config

All in all, python is suitable for big companies also, the thing is if don't care about the best practices, you would also have problems when you are a small startup, but in a big co it would make it impossible to move forward, trick is to independent of the company size follow best practices and have code review.

Re: Python at Scale: Strict Modules

#44

Post by scrollaway »

tahdig wrote: post #43

> ... many of whom are new to Python. well, if you ask me to write language X, I would definitely make mistakes for the first couple of weeks/months/years, that is why you need code review, mentoring and education plans for your hires. > Here’s another thing we often find developers doing at import time: fetching configuration from a network configuration source. MY_CONFIG = get_config_from_network_service() I am pre…

That's a long post to say "do more code review instead of investing into technical solutions to technical problems".

Clearly, Instagram's solution saves them time. That means faster code reviews which incidentally makes them more accurate. Your post doesn't really make sense.

Re: Python at Scale: Strict Modules

#45

Post by timothycrosley »

scrollaway wrote: post #40
timothycrosley wrote: post #36

Earlier quoted context omitted.

> TypeScript got typings right. I have not used TypeScript, but looking at it's documentation the syntax for type annotations look identical. Would you be willing to expand on why you think its approach is better / how it's different?

No importing basic types, using binary operators instead of awful things like Union and bracket accessors and what not, inline interfaces... Try it a bit. it truly is enjoyable. Fifteen years of python and I'm still enjoying TypeScript more. Mypy is limited by annotations having to be compatible python syntax.

> No importing basic type

This at least is being solved in Python: https://www.python.org/dev/peps/pep-0563/

> inline interfaces

Typed dicts in 3.8 look pretty similar at cursory glance https://www.python.org/dev/peps/pep-0589/

I'll definitely play around with it more!

Re: Python at Scale: Strict Modules

#46

Post by timothycrosley »

TylerE wrote: post #37
scrollaway wrote: post #30

Earlier quoted context omitted.

Python has some of my favourite syntax as well but I absolutely hate its annotations. TypeScript got typings right. I think the killer language will be typescript with access to both the python and JavaScript ecosystems. We'll see what that looks like. And of course if something changes the syntax, better anonymous functions will be the absolute first thing I would look for...

I don't think there is any sort of long-term future in anything "Python". I think a successful modern language has to have the potential for efficient concurrency baked in, which isn't really possible without breaking compatibility, and the Python community would never survive another round like the 2->3 transition. (And I'm not convinced the community really survived that one either, given the amount of ongoing bitt…

> I think a successful modern language has to have the potential for efficient concurrency baked in

I agree with this!

> I don't think there is any sort of long-term future in anything "Python".

I disagree with this :)

I think Python has efficient IO concurrency built-in already with async, and I feel it is likely that it finds a way to work out CPU bound concurrency long-term, as projects like sub-shells with channel communication demonstrate.

Re: Python at Scale: Strict Modules

#47

Post by weberc2 »

timothycrosley wrote: post #18

More and more I want someone to create a new language that amounts to a strict subset of Python, with mypy built-in, and is compilable into machine code. Python has by far my favorite syntax, community, and in my experience leads to the greatest productivity. There just happens to be a lot of overly dynamic features, that aren't even used by most, but used just enough to hold back optimization and structural improvem…

Sounds like Go. ;) This is a cheeky remark, but I use Python and Go, and Go very much feels like an improved Python in most ways. Especially when it comes to static analysis, build tooling, distribution, performance, etc. In particular, I love that there are no venvs, pipenvs, virtualenvs, pyenvs, wheels, eggs, setuptools, easy_installs, etc.

Re: Python at Scale: Strict Modules

#48

Post by carapace »

carapace wrote: post #28

> Instagram Server is a several-million-line Python monolith That's bananas. Nothing Instagram does requires that much code. Also, that much Python code means you're doing it wrong.

No, I'm seriously you guys.

Python is too expressive to require mega-LoC for that site.

You could implement an OS, relational DB, spreadsheet, and optimizing compiler all in less than that.

Re: Python at Scale: Strict Modules

#49

Post by scrollaway »

timothycrosley wrote: post #45
scrollaway wrote: post #40

Earlier quoted context omitted.

No importing basic types, using binary operators instead of awful things like Union and bracket accessors and what not, inline interfaces... Try it a bit. it truly is enjoyable. Fifteen years of python and I'm still enjoying TypeScript more. Mypy is limited by annotations having to be compatible python syntax.

> No importing basic type This at least is being solved in Python: https://www.python.org/dev/peps/pep-0563/ > inline interfaces Typed dicts in 3.8 look pretty similar at cursory glance https://www.python.org/dev/peps/pep-0589/ I'll definitely play around with it more!

Watch the difference: A variable that can be an object with two elements (one of them being a list of strings), or a tuple of exactly two strings.

    // typescript
    let t: {a: string[], b: number} | [string, string]

    # python 3.8
    from typing import TypedDict, Tuple, Union
    class SomeTypedDict(TypedDict):
        a: List[str]
        b: Union[float, int]

    t: Union[SomeTypedDict, Tuple[str, str]]
I had to google a bunch to figure out how to write the Python version, whereas the typescript one was completely natural to write. It takes one line and requires no imports. The interface is inlined. All of this also makes it more readable when you come across it eg. in an IDE tooltip.

Re: Python at Scale: Strict Modules

#50

Post by edraferi »

dmix wrote: post #42
TylerE wrote: post #37

Earlier quoted context omitted.

I don't think there is any sort of long-term future in anything "Python". I think a successful modern language has to have the potential for efficient concurrency baked in, which isn't really possible without breaking compatibility, and the Python community would never survive another round like the 2->3 transition. (And I'm not convinced the community really survived that one either, given the amount of ongoing bitt…

Data science/AI people breathed a whole lot of life into Python. I'm curious where it would be if they had went elsewhere.

R for the ecosystem, Julia for the performance & future proofing.
Post reply on HN