Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Python at Scale: Strict Modules

instagram-engineering.com

Post reply on HN
Print view View original

171–180 of 259 posts

Re: Python at Scale: Strict Modules

#171

Post by thelastbender12 »

scrollaway wrote: post #114
joshuamorton wrote: post #105

Earlier quoted context omitted.

> but in less controlled environments where you're dealing with eg. JSON inputs, form inputs, SQL table results and so on … not so. I more or less agree with this, but then again, IMO you should be isolating the less controlled code behind a controlled api. And the marginal value of converting `_ConvertQueryResultDictToQueryResult(qr: Dict[str, Any]) -> QueryResult` to something that uses a typeddict instead (which m…

Yeah I get what you're saying. And indeed. I seldom use dataclasses even though I should. Or namedtuples for that matter. It feels like the fact they're an import away makes them harder to use. Maybe both should be in the global namespace...

strongly agree. I love the extra clarity type annotations bring to the code, though going back to the start of the file every time I want to add an import is slightly dissuading.

Re: Python at Scale: Strict Modules

#172

Post by dgoldstein0 »

Sophistifunk wrote: post #115
cpeterso wrote: post #100

Earlier quoted context omitted.

I like Python, but I often wonder how many developers use Python because they actually use dynamic language features versus just liking the languages' clean syntax and library ecosystem. I'm surprised languages that offer both REPL (for development) and AOT native compilation (for production), like OCaml, are not more popular. Evidence that syntax matters, I guess. :) mypy and mypyc are interesting but their compile-…

I think a great deal of this sort of thing could be done by just doing some eval in a dynamic state before you stop the vm and compile its stable state, rather than the actual source code.

I think you missed part of the point of what the article was trying to say - or rather, what they hoped to do with this strict python. One of those things being some form of hot code loading. A snapshot of the state can't be incrementally rebuilt - it's very much all or nothing; whereas if we know or modules are side effect free, or at least some useful part of module loading is, we could cache that part and get faster start up times on incremental changes.

Re: Python at Scale: Strict Modules

#173

Post by rswail »

lern_too_spel wrote: post #78
TylerE wrote: post #23

Earlier quoted context omitted.

Who is starting large-scale new projects in Java in 2019?

For large scale projects, Java and C++ remain the go-to languages. I've seen a little bit of Go start to show up but no others. Other languages are used for libraries (Rust, C), only at certain employers (OCaml, Erlang), or for small-scale projects (nearly everything else).

On C++, inertia is a wonderful thing. Java has the benefits of extensive dependency injection and JVM/ecosystem tools that lower the risk of deployment of code. .NET also provides the controlled "managed code" environment of CLR.

Why any enterprise would use C++ for standard "business" or "large scale" programming makes no sense to me.

Enterprises want stability, not speed to market. Most of their infrastructure changes slowly (as in features deployed once or twice a year maybe). They have stable support mechanisms for this, including long and complex processes of approval.

Re: Python at Scale: Strict Modules

#174

Post by rswail »

adev_ wrote: post #162
WatchDog wrote: post #134

Earlier quoted context omitted.

Why would you pick Go or C++ over Java? C++ is a hydra of complexity, sure it has it's place, but it's not nearly as productive as Java for your typical web application. Go is almost the opposite, so simple it lacks features like generics. The last time I used Go it had fundamental usability issues around dependency management(although I think recent versions have improved on vendoring a little).

> C++ is a hydra of complexity, sure it has it's place, but it's not nearly as productive as Java for your typical web application. Modern C++ well is as productive ( probably even more productive ) as Java. The main issue with C++ is recruitment, C++ engineers are rare because C++ is barely teached.

C++ is barely taught in ProgrammerGenerationFactories because "modern" C++ still allows "old" C++ and makes it difficult to stop developers from doing that.

Just like MISRA tries to constrain C programmers from doing dumb things in the embedded world, "modern" C++ tries to the same in the business world. But there isn't an easy way to enforce it, especially when you're outsourcing to some code sweatshop.

Re: Python at Scale: Strict Modules

#176

Post by rswail »

carapace wrote: post #137
askafriend wrote: post #125

Earlier quoted context omitted.

Instagram does extremely complicated things. If you think it’s just a bunch of static photos in a feed, you’re wrong and it’d be clear you know nothing about the service.

> Instagram does extremely complicated things. Name three? Seriously, you're right, I've never used it. Wow me.

From a user's perspective, Instagram has: a) a way to post pictures/videos/sound recordings to a public feed. The pictures can include overlays of links to other users, to other posts, to song lyrics that play in sync with the music, etc etc. Users viewing their posts get the ability to comment/like/link, with automatic language detection and translation on demand.

b) a way to see how other users interact with their posts, allowing comments, seeing views and other analytics, monetizing etc etc

c) Provides advertisers with the ability to place stories (stories are a stream of short-lived (24 hours) video/audio posts that users see) or posts (that can be static/video/audio), with links to external sites, purchasing direct links ("Shop now"/"Buy this") etc

Instagram is much more than a stream of user images.

That doesn't include all the "back office" stuff like spam/reporting/censorship/language translation etc etc.

Re: Python at Scale: Strict Modules

#177

Post by adev_ »

rswail wrote: post #174
adev_ wrote: post #162

Earlier quoted context omitted.

> C++ is a hydra of complexity, sure it has it's place, but it's not nearly as productive as Java for your typical web application. Modern C++ well is as productive ( probably even more productive ) as Java. The main issue with C++ is recruitment, C++ engineers are rare because C++ is barely teached.

C++ is barely taught in ProgrammerGenerationFactories because "modern" C++ still allows "old" C++ and makes it difficult to stop developers from doing that. Just like MISRA tries to constrain C programmers from doing dumb things in the embedded world, "modern" C++ tries to the same in the business world. But there isn't an easy way to enforce it, especially when you're outsourcing to some code sweatshop.

> But there isn't an easy way to enforce it

Every mature enough language has a subset that you need to avoid. Including Java. This is precisely due to this kind of things that every company need to have coding guidelines and proper static analysis tools.

> especially when you're outsourcing to some code sweatshop.

If you outsource your dev to cheap, other side of the world, low quality engineers. Then you deserve your problem, in any language.

I worked in the past for a company (embedded programming) that had an entire team of expensive engineers in Luxembourg just to fix the stupidities of an other team of outsourced engineers in India.

Re: Python at Scale: Strict Modules

#178

Post by pjmlp »

cpeterso wrote: post #100
timothycrosley wrote: post #18

More and more I want someone to create a new language that amounts to a strict subset of Python, with mypy built-in, and is compilable into machine code. Python has by far my favorite syntax, community, and in my experience leads to the greatest productivity. There just happens to be a lot of overly dynamic features, that aren't even used by most, but used just enough to hold back optimization and structural improvem…

I like Python, but I often wonder how many developers use Python because they actually use dynamic language features versus just liking the languages' clean syntax and library ecosystem. I'm surprised languages that offer both REPL (for development) and AOT native compilation (for production), like OCaml, are not more popular. Evidence that syntax matters, I guess. :) mypy and mypyc are interesting but their compile-…

The dynamic language semantics of Lisp, Scheme, Smalltalk, JavaScript have not hampered the existence of good JIT/AOT compilers.

Smalltalk, for example you can completely change the structure of a class by sending a become: message.

What I think is missing is a bit of more PyPy love, and the Truffle and OpenJ9 Python support efforts.

Re: Python at Scale: Strict Modules

#179

Post by rswail »

carapace wrote: post #161
scrollaway wrote: post #143

Earlier quoted context omitted.

I'm not sure what good these kinds of bets are. All code could be made leaner and smaller. What's your point?

> I'm not sure what good these kinds of bets are. I'm just putting some numbers on my confidence level. > All code could be made leaner and smaller. What's your point? That Instagram's code could be made much leaner and smaller. I feel like I've been very clear about that.

And from Instagram's POV, the ROI on that would be much less than putting in the sort of belts and braces that the article talked about.

They don't have the time or space to engage in a massive technical debt reduction program, they're too busy destroying Snapchat and other competitors, reacting to TikTok, implementing an entirely new IGTV video service that provides their customers (ie advertisers and marketers) the equivalent of youtube within the Instagram universe, etc.

I'm sure that every large internet service's codebase out there could be made much leaner and smaller. The question is whether that is worth their while.

Re: Python at Scale: Strict Modules

#180

Post by pjmlp »

coldtea wrote: post #89
edraferi wrote: post #50

Earlier quoted context omitted.

R for the ecosystem, Julia for the performance & future proofing.

What "future proofing"? It's not like Julia is "the future" it's just a contended, and it's not doing that good at that either...

Its adoption at banks and life science research labs tells another story.

Naturally if the community finally gathers around PyPy, that might change.

Post reply on HN