Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Python at Scale: Strict Modules

instagram-engineering.com

Post reply on HN
Print view View original

121–130 of 259 posts

Re: Python at Scale: Strict Modules

#121

Post by andybak »

miki123211 wrote: post #21

This is yet another example of the divide between wizarding and engineering[1]. When you're a small startup, what matters is the expressiveness of your language, and the ability do do a lot of things very very quickly. Type safety, performance, readability, those things don't matter. You're just a bunch of engineers who know the whole codebase inside out, you're pretty certain of what you're doing. In short, you're w…

And the worst thing is using languages/libraries/frameworks that presume everyone needs engineering when you need to wizard.

There's two many people that have swallowed SOLID whole and can no longer see good engineering as a trade-off against other factors.

For example, being strict about having the smallest possible public API and making most methods private protects me from future breakage that might never be an issue (I might never upgrade) but forces me copy/paste vast globs of your code into my own if I need access to something you didn't anticipate. (and that's assuming I have access to your source. Worst case is that I have to reimplement things that already exist in the code I'm interfacing with)

Python got this right. Private methods are a weak or strong hint that you might want to think twice before calling them. But you're the boss at the end of the day.

Re: Python at Scale: Strict Modules

#122

Post by ashishb »

paulddraper wrote: post #117
bcherny wrote: post #32

Earlier quoted context omitted.

> I wish there was a language that let you move gradually from one end to the other, exactly when you need to. This is precisely what gradually typed languages — like TypeScript, Flow, and typed Pythons — solve! I talked about this on Software Engineering Radio last week: https://www.se-radio.net/2019/10/episode-384-boris-cherny-on... .

TypeScript is perfectly this. (And other gradually typed solutions; TS is simply the most popular one.) You have the madness of thousand of developers flinging code at the universe due to the easiness of browsers, JS, and npm. This results in great speed, but not great quality. When your project/company now wants quality, you keep your code but transition to types. (In OSS space, Angular and Yarn projects have both d…

Afaik, typescript is pretty bad in terms of catching some basic errors. Types are not enforced. A caller can change sync function to async, breaking the functionality downstream.

Re: Python at Scale: Strict Modules

#123

Post by coldtea »

TylerE wrote: post #107
coldtea wrote: post #86

Earlier quoted context omitted.

Err, did you notice that Python is in the top 3 language in TIOBE (up from 4), and named "Language of the year"? What bizarro bubble do you live in?

Indicators like TIOBE are VERY lagging. Like 5-10 years

In my assessment, not really. They are surprisingly spot on (for the top spots and singling out contenders).

Not to mention the language landscape is almost static at the top. Nobody's gonna came and take Python, Java, C, C++, JS, out in the next 10-15 years...

Only a huge self-blunder, like the Perl 5 -> 6 transition, and only at much more volatile time (when paradigms change, e.g. when web dev changed from CGI, Perl 5 had already lost the web framework scene to PHP, Rails, Django and the like even before losing its main niche back then - admin work) can do any serious damage to a top language...

Anyway, let's check in 5 years...

Re: Python at Scale: Strict Modules

#124

Post by coldtea »

jedberg wrote: post #97
coldtea wrote: post #81

Earlier quoted context omitted.

> Someone made a change that took down production because of non-deterministic outcomes? How about break out whatever they were changing into it's own service? With proper fallbacks, breaking that part shouldn't take down all of production again. Yeah, now you'll have 10 interconnected services, 10x the complexity, and everything will have the ability to take down all of large parts of production, plus all the extra…

You won't have 10 times the complexity if you are taking a monolith and making each section services. You'll have to same dependency graph, it will just use the network to make calls between them instead of being local. You'll have added complexity with the network calls, which is why I said it wouldn't be any less work, just different work.

>You won't have 10 times the complexity if you are taking a monolith and making each section services. You'll have to same dependency graph, it will just use the network to make calls between them instead of being local.

Merely "use the network to make calls between them instead of being local" will add 10 times the complexity -- you suddenly have a distributed system, latency, delays, parts that can be on or off, de-centralized configuration (which can also get out of sync), and so on.

Re: Python at Scale: Strict Modules

#125

Post by askafriend »

carapace wrote: post #88
orf wrote: post #83

Earlier quoted context omitted.

You have no idea about their codebase, the implementation details of their features nor how they counted the lines (comments included?). So stating that it’s dumb is beyond ridiculous. You are right in that it’s certainly a high LoC count for Python, but still...

I didn't say "dumb" I said "bananas". And yes, knowing nothing else about their code base than A) It's in Python, and B) it's several million lines of code, I feel very confident that there is at least an order of magnitude too much of it. Instagram is just not doing anything that complicated. (I should mention I specialize in maintaining and refactoring legacy Python code. I know what I'm talking about here.)

Instagram does extremely complicated things.

If you think it’s just a bunch of static photos in a feed, you’re wrong and it’d be clear you know nothing about the service.

Re: Python at Scale: Strict Modules

#126

Post by pytester »

andybak wrote: post #121
miki123211 wrote: post #21

This is yet another example of the divide between wizarding and engineering[1]. When you're a small startup, what matters is the expressiveness of your language, and the ability do do a lot of things very very quickly. Type safety, performance, readability, those things don't matter. You're just a bunch of engineers who know the whole codebase inside out, you're pretty certain of what you're doing. In short, you're w…

And the worst thing is using languages/libraries/frameworks that presume everyone needs engineering when you need to wizard. There's two many people that have swallowed SOLID whole and can no longer see good engineering as a trade-off against other factors. For example, being strict about having the smallest possible public API and making most methods private protects me from future breakage that might never be an is…

>And the worst thing is using languages/libraries/frameworks that presume everyone needs engineering when you need to wizard.

I think this is why it's easy to point a thousand things built in python which people use every day (like instagram), while in, say, haskell, there are barely a handful (pandoc, facebook spam filter, etc.).

Re: Python at Scale: Strict Modules

#127

Post by victor106 »

GordonS wrote: post #75
TylerE wrote: post #35

Earlier quoted context omitted.

I said new projects. Not upgrades.

Not the OP, but I understood their comment to mean 'a rewrite in Java of a legacy system written in some other language'?

Exactly right. Java has lots to improve on but there seems to be unsubstantiated hate towards Java on HN which I find contrary to what happens in the real world. In the real world companies find hiring Java programmers relatively easy (maybe not the best and brightest) who can get a project off the ground easily

Re: Python at Scale: Strict Modules

#128

Post by lostcolony »

dunefox wrote: post #116
coldtea wrote: post #71

Earlier quoted context omitted.

Countless companies, huge and small -- from Apple and Amazon, to Google and your friendly local startup, plus all the enterprise world that's not a .NET shop... In what parallel universe is not Java immensely popular or not used for green projects?

Hopefully in every universe where Kotlin exists.

I can think of at least one universe where Kotlin exists, but most new JVM development is still in Java.

Re: Python at Scale: Strict Modules

#129

Post by gtirloni »

jedberg wrote: post #72

It's interesting to me that they are going down this path instead of the microservices path. This seems like something ripe for slowly breaking down into microservices. Someone made a change that took down production because of non-deterministic outcomes? How about break out whatever they were changing into it's own service? With proper fallbacks, breaking that part shouldn't take down all of production again. To be…

I think your comment makes perfect sense.

However, at their scale and with their engineering resources, I can only imagine an attitude of "we can make this work" (the monolith) is easier to justify. The same goes for the micro-services approach (except here you have to justify changing what has been working so far?)

I'd love to read more about the history behind this approach at Instagram.

Re: Python at Scale: Strict Modules

#130

Post by lostcolony »

bcherny wrote: post #32
miki123211 wrote: post #21

This is yet another example of the divide between wizarding and engineering[1]. When you're a small startup, what matters is the expressiveness of your language, and the ability do do a lot of things very very quickly. Type safety, performance, readability, those things don't matter. You're just a bunch of engineers who know the whole codebase inside out, you're pretty certain of what you're doing. In short, you're w…

> I wish there was a language that let you move gradually from one end to the other, exactly when you need to. This is precisely what gradually typed languages — like TypeScript, Flow, and typed Pythons — solve! I talked about this on Software Engineering Radio last week: https://www.se-radio.net/2019/10/episode-384-boris-cherny-on... .

On that note, I'd include Erlang. It's not gradually typed, per se, but you can have a fully dynamic language (no type specs, no Dialyzer), a completely optimistic static analyzer for inferring types and warning where it's inconsistent (Dialyzer runs), and then you can add specs where needed to tighten up and improve what Dialyzer can catch, to basically be a fully static language.
Post reply on HN