Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Python at Scale: Strict Modules

instagram-engineering.com

Post reply on HN
Print view View original

131–140 of 259 posts

Re: Python at Scale: Strict Modules

#131

Post by TylerE »

coldtea wrote: post #123
TylerE wrote: post #107

Earlier quoted context omitted.

Indicators like TIOBE are VERY lagging. Like 5-10 years

In my assessment, not really. They are surprisingly spot on (for the top spots and singling out contenders). Not to mention the language landscape is almost static at the top. Nobody's gonna came and take Python, Java, C, C++, JS, out in the next 10-15 years... Only a huge self-blunder, like the Perl 5 -> 6 transition, and only at much more volatile time (when paradigms change, e.g. when web dev changed from CGI, Per…

Well, for instance, TIOBE has Rust at #34, below such mainstays as ABAP, COBOL, and Scratch. It has Groovy above Ruby.

Re: Python at Scale: Strict Modules

#132

Post by depressedpanda »

ClippyIO wrote: post #101
carapace wrote: post #88

Earlier quoted context omitted.

I didn't say "dumb" I said "bananas". And yes, knowing nothing else about their code base than A) It's in Python, and B) it's several million lines of code, I feel very confident that there is at least an order of magnitude too much of it. Instagram is just not doing anything that complicated. (I should mention I specialize in maintaining and refactoring legacy Python code. I know what I'm talking about here.)

Features that are "not complicated" can actually very easily be "very complicated" at scale. Which Instagram does have. 500 million users, every single day.

Fewer LOC would actually benefit them at scale.

If you need several millions of lines of Python to do what Instagram server does, the code is bloated.

My bet is that they let too many Java devs loose on the code base, without experienced Python devs reviewing the commits and managing the deluge of unnecessary classes. I've seen it happen before.

Re: Python at Scale: Strict Modules

#133

Post by pron »

TylerE wrote: post #23
miki123211 wrote: post #21

This is yet another example of the divide between wizarding and engineering[1]. When you're a small startup, what matters is the expressiveness of your language, and the ability do do a lot of things very very quickly. Type safety, performance, readability, those things don't matter. You're just a bunch of engineers who know the whole codebase inside out, you're pretty certain of what you're doing. In short, you're w…

Who is starting large-scale new projects in Java in 2019?

Oh, just Apple, Amazon, Google, Netflix and likely every hospital, utility company, police force, military, or bank you depend on. And the people programming the robot swarms that pack your groceries (https://www.infoq.com/presentations/java-robot-swarms/).

Re: Python at Scale: Strict Modules

#134

Post by WatchDog »

typon wrote: post #92
TylerE wrote: post #23

Earlier quoted context omitted.

Who is starting large-scale new projects in Java in 2019?

It's a good question. Why would you pick Java over Go or C++?

Why would you pick Go or C++ over Java?

C++ is a hydra of complexity, sure it has it's place, but it's not nearly as productive as Java for your typical web application.

Go is almost the opposite, so simple it lacks features like generics. The last time I used Go it had fundamental usability issues around dependency management(although I think recent versions have improved on vendoring a little).

Re: Python at Scale: Strict Modules

#135

Post by pytester »

jedberg wrote: post #97
coldtea wrote: post #81

Earlier quoted context omitted.

> Someone made a change that took down production because of non-deterministic outcomes? How about break out whatever they were changing into it's own service? With proper fallbacks, breaking that part shouldn't take down all of production again. Yeah, now you'll have 10 interconnected services, 10x the complexity, and everything will have the ability to take down all of large parts of production, plus all the extra…

You won't have 10 times the complexity if you are taking a monolith and making each section services. You'll have to same dependency graph, it will just use the network to make calls between them instead of being local. You'll have added complexity with the network calls, which is why I said it wouldn't be any less work, just different work.

>it will just use the network to make calls between them

meaning that you get to throw network and server errors into the mix of things that can go wrong, and you get the fun of tracing failures back 3 hops to a server that decides to take too long to run a process one day and times out a connection downstream.

it's horrible debugging stuff like this.

Re: Python at Scale: Strict Modules

#136

Post by staticassertion »

ledauphin wrote: post #61
staticassertion wrote: post #34

Earlier quoted context omitted.

https://github.com/python/mypy/tree/master/mypyc This may interest you. This is probably going to be the 'official' way to get what you're talking about.

yeah, I'm excited about this approach. but it's a long way from being a realistic approach for folks outside Dropbox, it seems.

It's a long way from being a realistic approach for anyone, including Dropbox, really.

Re: Python at Scale: Strict Modules

#137

Post by carapace »

askafriend wrote: post #125
carapace wrote: post #88

Earlier quoted context omitted.

I didn't say "dumb" I said "bananas". And yes, knowing nothing else about their code base than A) It's in Python, and B) it's several million lines of code, I feel very confident that there is at least an order of magnitude too much of it. Instagram is just not doing anything that complicated. (I should mention I specialize in maintaining and refactoring legacy Python code. I know what I'm talking about here.)

Instagram does extremely complicated things. If you think it’s just a bunch of static photos in a feed, you’re wrong and it’d be clear you know nothing about the service.

> Instagram does extremely complicated things.

Name three?

Seriously, you're right, I've never used it. Wow me.

Re: Python at Scale: Strict Modules

#138

Post by pytester »

depressedpanda wrote: post #132
ClippyIO wrote: post #101

Earlier quoted context omitted.

Features that are "not complicated" can actually very easily be "very complicated" at scale. Which Instagram does have. 500 million users, every single day.

Fewer LOC would actually benefit them at scale. If you need several millions of lines of Python to do what Instagram server does, the code is bloated. My bet is that they let too many Java devs loose on the code base, without experienced Python devs reviewing the commits and managing the deluge of unnecessary classes. I've seen it happen before.

>If you need several millions of lines of Python to do what Instagram server does

I have this feeling that you're probably not all that aware of 95% of what their code actually does, and thus probably not in a position to make judgements as to whether their code base is truly bloated relative to what it does.

Re: Python at Scale: Strict Modules

#139

Post by jsmeaton »

timothycrosley wrote: post #18

More and more I want someone to create a new language that amounts to a strict subset of Python, with mypy built-in, and is compilable into machine code. Python has by far my favorite syntax, community, and in my experience leads to the greatest productivity. There just happens to be a lot of overly dynamic features, that aren't even used by most, but used just enough to hold back optimization and structural improvem…

For the sites I typically work on it’s very hard to give up the Django admin and all of the features it provides.

At the same time, I’d love a stronger type system to avoid a bunch of the pitfalls that the dynamism of python has.

So count me in.

Re: Python at Scale: Strict Modules

#140

Post by xzel »

ashishb wrote: post #122
paulddraper wrote: post #117

Earlier quoted context omitted.

TypeScript is perfectly this. (And other gradually typed solutions; TS is simply the most popular one.) You have the madness of thousand of developers flinging code at the universe due to the easiness of browsers, JS, and npm. This results in great speed, but not great quality. When your project/company now wants quality, you keep your code but transition to types. (In OSS space, Angular and Yarn projects have both d…

Afaik, typescript is pretty bad in terms of catching some basic errors. Types are not enforced. A caller can change sync function to async, breaking the functionality downstream.

You can decide if they're enforced or not. That is part of how it is gradual typing.
Post reply on HN