Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Python at Scale: Strict Modules

instagram-engineering.com

Post reply on HN
Print view View original

91–100 of 259 posts

Re: Python at Scale: Strict Modules

#91

Post by choiway »

miki123211 wrote: post #21

This is yet another example of the divide between wizarding and engineering[1]. When you're a small startup, what matters is the expressiveness of your language, and the ability do do a lot of things very very quickly. Type safety, performance, readability, those things don't matter. You're just a bunch of engineers who know the whole codebase inside out, you're pretty certain of what you're doing. In short, you're w…

I like the way you characterize this but what on your thoughts on why you can't be a wizard with a typed language? Seems to me that if you start with something like Go or Typescript you cover a decent middle ground of foregoing a lot of boilerplate while having code that you don't need to be a wizard to understand.

Re: Python at Scale: Strict Modules

#92

Post by typon »

TylerE wrote: post #23
miki123211 wrote: post #21

This is yet another example of the divide between wizarding and engineering[1]. When you're a small startup, what matters is the expressiveness of your language, and the ability do do a lot of things very very quickly. Type safety, performance, readability, those things don't matter. You're just a bunch of engineers who know the whole codebase inside out, you're pretty certain of what you're doing. In short, you're w…

Who is starting large-scale new projects in Java in 2019?

It's a good question. Why would you pick Java over Go or C++?

Re: Python at Scale: Strict Modules

#93

Post by [deleted] »

yen223 wrote: post #70
nerdponx wrote: post #17

Earlier quoted context omitted.

I much prefer frameworks/modules for which code is executed only once you invoke their "setup" function Django _does_ have a "setup" function. You can't import and use Django database connections outside of a running application without it. Flask also has a "run" method and does no i/o without it.

Without calling setup, you cannot import anything that touches Django models, like constants defined in a file that transitively imports a Django model. In practice, this means that any script that depends indirectly on Django code will incur a lengthy startup cost (from having to call setup()), and will fail to run if there's no database connection, even if the script itself doesn't need the db.

[deleted]

Re: Python at Scale: Strict Modules

#94

Post by jedberg »

civicsquid wrote: post #82
jedberg wrote: post #72

It's interesting to me that they are going down this path instead of the microservices path. This seems like something ripe for slowly breaking down into microservices. Someone made a change that took down production because of non-deterministic outcomes? How about break out whatever they were changing into it's own service? With proper fallbacks, breaking that part shouldn't take down all of production again. To be…

Beyond increasing complexity, I think this also assumes a dependency graph that _can_ be broken down into microservices by the author/the author's team. From my experience a lot of things at this scale have such complex dependencies that unteasing those dependencies is difficult if not impossible without asking several teams to do something differently. And who knows how long that will take?

That's why you do it slowly. You take a small part of the monolith and make a service that does the same thing. Then you replace the code in the monolith with a call to the service, while keeping track of how often it is called in the monolith.

As you keep moving along, some things that depend on that first service will start calling the new service directly, and some will still call it in the monolith. But your tracking will tell you how often and who is doing that, so you can find out why.

In the meantime, nothing will break, because the monolith is still a pass through proxy to your service.

Re: Python at Scale: Strict Modules

#95

Post by nerdponx »

heavenlyblue wrote: post #24
nerdponx wrote: post #17

Earlier quoted context omitted.

I much prefer frameworks/modules for which code is executed only once you invoke their "setup" function Django _does_ have a "setup" function. You can't import and use Django database connections outside of a running application without it. Flask also has a "run" method and does no i/o without it.

Every time I hear a comment alike parent's, it makes me think how many times a day I actually read a comment in the same fashion, but about something I actually know nothing about.

With credit to the original poster, they might be complaining about the fact that Django is a monolithic framework and you can't really use Django code without spinning up the i/o portion. Which is legitimate criticism, but frankly if that's what you need then you shouldn't be using Django.

Re: Python at Scale: Strict Modules

#96

Post by timothycrosley »

coldtea wrote: post #86
TylerE wrote: post #37

Earlier quoted context omitted.

I don't think there is any sort of long-term future in anything "Python". I think a successful modern language has to have the potential for efficient concurrency baked in, which isn't really possible without breaking compatibility, and the Python community would never survive another round like the 2->3 transition. (And I'm not convinced the community really survived that one either, given the amount of ongoing bitt…

Err, did you notice that Python is in the top 3 language in TIOBE (up from 4), and named "Language of the year"? What bizarro bubble do you live in?

Python is very healthy at the moment, and still growing! However, I think that makes it even more important as a community that we don't rest on our laurels and we fix the issues we do have. CPU concurrency is definitely one of those issues.

Re: Python at Scale: Strict Modules

#97

Post by jedberg »

coldtea wrote: post #81
jedberg wrote: post #72

It's interesting to me that they are going down this path instead of the microservices path. This seems like something ripe for slowly breaking down into microservices. Someone made a change that took down production because of non-deterministic outcomes? How about break out whatever they were changing into it's own service? With proper fallbacks, breaking that part shouldn't take down all of production again. To be…

> Someone made a change that took down production because of non-deterministic outcomes? How about break out whatever they were changing into it's own service? With proper fallbacks, breaking that part shouldn't take down all of production again. Yeah, now you'll have 10 interconnected services, 10x the complexity, and everything will have the ability to take down all of large parts of production, plus all the extra…

You won't have 10 times the complexity if you are taking a monolith and making each section services. You'll have to same dependency graph, it will just use the network to make calls between them instead of being local.

You'll have added complexity with the network calls, which is why I said it wouldn't be any less work, just different work.

Re: Python at Scale: Strict Modules

#98

Post by scrollaway »

carapace wrote: post #48
carapace wrote: post #28

> Instagram Server is a several-million-line Python monolith That's bananas. Nothing Instagram does requires that much code. Also, that much Python code means you're doing it wrong.

No, I'm seriously you guys. Python is too expressive to require mega-LoC for that site. You could implement an OS, relational DB, spreadsheet, and optimizing compiler all in less than that.

As orf said you have no idea about their codebase. And you have no idea what's included in that statement -- given that they talk about startup time, they most likely are taking into account the whole framework, a plethora of admin and analytics tools, lots of debugging / debug-only infrastructure, migrations, lots of tooling whose sole purpose is making it easier to work in large teams, etc…

(And for the record, Linux is ~37 million lines of actual code, Postgres ~2 million, and gcc ~8 million)

There's nothing absurd about one of the most visited websites on earth to be a couple million LOC.

Re: Python at Scale: Strict Modules

#99

Post by spraak »

bcherny wrote: post #32
miki123211 wrote: post #21

This is yet another example of the divide between wizarding and engineering[1]. When you're a small startup, what matters is the expressiveness of your language, and the ability do do a lot of things very very quickly. Type safety, performance, readability, those things don't matter. You're just a bunch of engineers who know the whole codebase inside out, you're pretty certain of what you're doing. In short, you're w…

> I wish there was a language that let you move gradually from one end to the other, exactly when you need to. This is precisely what gradually typed languages — like TypeScript, Flow, and typed Pythons — solve! I talked about this on Software Engineering Radio last week: https://www.se-radio.net/2019/10/episode-384-boris-cherny-on... .

Typescript was exactly what I was going to mention in reply

Re: Python at Scale: Strict Modules

#100

Post by cpeterso »

timothycrosley wrote: post #18

More and more I want someone to create a new language that amounts to a strict subset of Python, with mypy built-in, and is compilable into machine code. Python has by far my favorite syntax, community, and in my experience leads to the greatest productivity. There just happens to be a lot of overly dynamic features, that aren't even used by most, but used just enough to hold back optimization and structural improvem…

I like Python, but I often wonder how many developers use Python because they actually use dynamic language features versus just liking the languages' clean syntax and library ecosystem. I'm surprised languages that offer both REPL (for development) and AOT native compilation (for production), like OCaml, are not more popular. Evidence that syntax matters, I guess. :)

mypy and mypyc are interesting but their compile-time checks and optimizations are still hampered by Python's dynamic language semantics.

Post reply on HN