Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Python at Scale: Strict Modules

instagram-engineering.com

Post reply on HN
Print view View original

81–90 of 259 posts

Re: Python at Scale: Strict Modules

#81

Post by coldtea »

jedberg wrote: post #72

It's interesting to me that they are going down this path instead of the microservices path. This seems like something ripe for slowly breaking down into microservices. Someone made a change that took down production because of non-deterministic outcomes? How about break out whatever they were changing into it's own service? With proper fallbacks, breaking that part shouldn't take down all of production again. To be…

>Someone made a change that took down production because of non-deterministic outcomes? How about break out whatever they were changing into it's own service? With proper fallbacks, breaking that part shouldn't take down all of production again.

Yeah, now you'll have 10 interconnected services, 10x the complexity, and everything will have the ability to take down all of large parts of production, plus all the extra pain points of a distributed system...

Re: Python at Scale: Strict Modules

#82

Post by civicsquid »

jedberg wrote: post #72

It's interesting to me that they are going down this path instead of the microservices path. This seems like something ripe for slowly breaking down into microservices. Someone made a change that took down production because of non-deterministic outcomes? How about break out whatever they were changing into it's own service? With proper fallbacks, breaking that part shouldn't take down all of production again. To be…

Beyond increasing complexity, I think this also assumes a dependency graph that _can_ be broken down into microservices by the author/the author's team. From my experience a lot of things at this scale have such complex dependencies that unteasing those dependencies is difficult if not impossible without asking several teams to do something differently. And who knows how long that will take?

Re: Python at Scale: Strict Modules

#83

Post by orf »

carapace wrote: post #48
carapace wrote: post #28

> Instagram Server is a several-million-line Python monolith That's bananas. Nothing Instagram does requires that much code. Also, that much Python code means you're doing it wrong.

No, I'm seriously you guys. Python is too expressive to require mega-LoC for that site. You could implement an OS, relational DB, spreadsheet, and optimizing compiler all in less than that.

You have no idea about their codebase, the implementation details of their features nor how they counted the lines (comments included?). So stating that it’s dumb is beyond ridiculous.

You are right in that it’s certainly a high LoC count for Python, but still...

Re: Python at Scale: Strict Modules

#84

Post by konschubert »

I have a question about a detail in the article:

> But if we moved the log_to_network call out into the outer log_calls function, [...] this would no longer compile as a strict module.

My current understanding is that the log_calls method would NOT get executed during module load time!?!

Why would having a side effect in this function violate the intention of __strict__ ?

Re: Python at Scale: Strict Modules

#85

Post by scrollaway »

joshuamorton wrote: post #76
scrollaway wrote: post #49

Earlier quoted context omitted.

Watch the difference: A variable that can be an object with two elements (one of them being a list of strings), or a tuple of exactly two strings. // typescript let t: {a: string[], b: number} | [string, string] # python 3.8 from typing import TypedDict, Tuple, Union class SomeTypedDict(TypedDict): a: List[str] b: Union[float, int] t: Union[SomeTypedDict, Tuple[str, str]] I had to google a bunch to figure out how to…

Granted, in python, I'd call the use of a typed dict a smell. If you're able to spend the time creating the typed dict, just promote it to a dataclass. Using python ~3.9, this will look like @dataclasses.dataclass # or @attr.s class MyStruct: a: List[str] b: int|float t: MyStruct|Tuple[str,str] But MyStruct will be an actual object that can be manipulated as an object. And if you want to accept any object that fits t…

I'll give you that in controlled code the use of typed dicts would be a symptom of a code smell, but in less controlled environments where you're dealing with eg. JSON inputs, form inputs, SQL table results and so on … not so.

I'm also not onboard the "it's a code smell, it doesn't matter" train. IMO if python adopted the typescript typing syntax we'd all be better for it.

I also forgot to mention the atrocious typing syntax for functions. Once again, typescript is a lot more succint and readable.

Re: Python at Scale: Strict Modules

#86

Post by coldtea »

TylerE wrote: post #37
scrollaway wrote: post #30

Earlier quoted context omitted.

Python has some of my favourite syntax as well but I absolutely hate its annotations. TypeScript got typings right. I think the killer language will be typescript with access to both the python and JavaScript ecosystems. We'll see what that looks like. And of course if something changes the syntax, better anonymous functions will be the absolute first thing I would look for...

I don't think there is any sort of long-term future in anything "Python". I think a successful modern language has to have the potential for efficient concurrency baked in, which isn't really possible without breaking compatibility, and the Python community would never survive another round like the 2->3 transition. (And I'm not convinced the community really survived that one either, given the amount of ongoing bitt…

Err, did you notice that Python is in the top 3 language in TIOBE (up from 4), and named "Language of the year"?

What bizarro bubble do you live in?

Re: Python at Scale: Strict Modules

#87

Post by scrollaway »

konschubert wrote: post #84

I have a question about a detail in the article: > But if we moved the log_to_network call out into the outer log_calls function, [...] this would no longer compile as a strict module. My current understanding is that the log_calls method would NOT get executed during module load time!?! Why would having a side effect in this function violate the intention of __strict__ ?

> My current understanding is that the log_calls method would NOT get executed during module load time!?!

That's incorrect. log_calls gets executed on import because it's a decorator, so equivalent to `hello_world = log_calls(hello_world)` at the top-level (which does also get executed).

log_to_network in the _wrapped() definition doesn't get executed until hello_world gets called; but outside of the definition of _wrapped does get executed.

Re: Python at Scale: Strict Modules

#88

Post by carapace »

orf wrote: post #83
carapace wrote: post #48

Earlier quoted context omitted.

No, I'm seriously you guys. Python is too expressive to require mega-LoC for that site. You could implement an OS, relational DB, spreadsheet, and optimizing compiler all in less than that.

You have no idea about their codebase, the implementation details of their features nor how they counted the lines (comments included?). So stating that it’s dumb is beyond ridiculous. You are right in that it’s certainly a high LoC count for Python, but still...

I didn't say "dumb" I said "bananas".

And yes, knowing nothing else about their code base than A) It's in Python, and B) it's several million lines of code, I feel very confident that there is at least an order of magnitude too much of it. Instagram is just not doing anything that complicated.

(I should mention I specialize in maintaining and refactoring legacy Python code. I know what I'm talking about here.)

Re: Python at Scale: Strict Modules

#89

Post by coldtea »

edraferi wrote: post #50
dmix wrote: post #42

Earlier quoted context omitted.

Data science/AI people breathed a whole lot of life into Python. I'm curious where it would be if they had went elsewhere.

R for the ecosystem, Julia for the performance & future proofing.

What "future proofing"? It's not like Julia is "the future" it's just a contended, and it's not doing that good at that either...

Re: Python at Scale: Strict Modules

#90

Post by ken »

miki123211 wrote: post #21

This is yet another example of the divide between wizarding and engineering[1]. When you're a small startup, what matters is the expressiveness of your language, and the ability do do a lot of things very very quickly. Type safety, performance, readability, those things don't matter. You're just a bunch of engineers who know the whole codebase inside out, you're pretty certain of what you're doing. In short, you're w…

> When you're a small startup, what matters is the expressiveness of your language, and the ability do do a lot of things very very quickly. Type safety, performance, readability, those things don't matter.

I’ve never worked on a program so small that readability didn’t matter. I consider it a crucial ingredient of expressiveness and development speed.

Though your perspective could explain a few of the more atrocious code bases I’ve seen.

Post reply on HN