Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Backend of Meta Threads is built with Python 3.10

twitter.com

Post reply on HN
Print view View original

281–290 of 458 posts

Re: Backend of Meta Threads is built with Python 3.10

#281

Post by Yoric »

VWWHFSfQ wrote: post #48
ambivalence wrote: post #22

Earlier quoted context omitted.

In short: static analysis, unit tests, staging environments, effective log aggregation. The first one is a big productivity booster as it shows you bugs before you commit them. Here's a post about this, not exactly new but still describing the general principles very well: https://instagram-engineering.com/static-analysis-at-scale-a...

I would say that those things don't really contribute to the actual runtime scalability of the application. Those things ensure that the code is at least correct for what it's supposed to do, not necessarily whether it's performant. Even the most perfectly-written Python code will still be dog-slow and require an enormous budget to actually run at scale. It's obviously fine to do that and there are many success stori…

For context, I was asking about scaling code complexity, not scaling performance.

That being said, my personal experience suggests that both are really hard in Python, and not independent.

Re: Backend of Meta Threads is built with Python 3.10

#282

Post by Yoric »

marcinzm wrote: post #40
Yoric wrote: post #18

Earlier quoted context omitted.

My few years of personal experience of using Python with even small-ish scale professional projects have been... underwhelming so far. I wonder how Meta (and Google, and all the others who write serious Python code) manage to keep things running – and I'm not talking (only) of performance, but largely of bugs. My personal suspicion is that the secret sauce is simply to have a sufficient budget to hire an army of code…

You can write crap code in any language. To avoid it you need to enforce some decent set of best practices for that specific language. That also means not using best practices from language X in language Y and then complaining that language Y is crap.

Yes. And one way to rephrase my question is: which best practices work for these Python mega-projects?

Re: Backend of Meta Threads is built with Python 3.10

#283

Post by Yoric »

hospitalJail wrote: post #41
Yoric wrote: post #18

Earlier quoted context omitted.

My few years of personal experience of using Python with even small-ish scale professional projects have been... underwhelming so far. I wonder how Meta (and Google, and all the others who write serious Python code) manage to keep things running – and I'm not talking (only) of performance, but largely of bugs. My personal suspicion is that the secret sauce is simply to have a sufficient budget to hire an army of code…

>My few years of personal experience of using Python with even small-ish scale professional projects have been... underwhelming so far You have years of experience but couldn't point to anything in particular? Heck, I like python and I can complain about dynamic typing issues in for loops, or that they are adding features like generators/decorators that make code more difficult to understand, which goes against the z…

Why would I need to point to anything in particular? I think that all Python developers know them already.

Re: Backend of Meta Threads is built with Python 3.10

#285

Post by [deleted] »

callamdelaney wrote: post #264
andrewstuart wrote: post #258

If you know you’ve got gigantic scale why would you not use a compiled language? Surely it makes massive difference to hardware utilization? I like Python and javascript but there’s perfectly fine tools around now for building systems that allow one server to do a lot more. They say you should not prematurely optimize. Equally you should not be prematurely unoptimised.

Because Zuck probably told his team to make this a few weeks ago to annoy Musk. Weird ego thing going on. Python is likely faster to push out an app like this than say rust or even go. Also why spend time optimising something that will be dead in a few months?

[deleted]

Re: Backend of Meta Threads is built with Python 3.10

#286

Post by ramesh31 »

c7DJTLrn wrote: post #275
pjmlp wrote: post #115

Earlier quoted context omitted.

You mean the company that had to create their two PHP implementations and a translator to C++, to actually scale?

Guess I'll just write my next project in Bash then, after all, if it's a good idea they will come right? Execution can wait until you've got a few hundred mil.

>Guess I'll just write my next project in Bash then, after all, if it's a good idea they will come right? Execution can wait until you've got a few hundred mil.

Yea actually. Bash/CGI could handle plenty. If thats your most comfortable language then go for it.

Re: Backend of Meta Threads is built with Python 3.10

#287

Post by reissbaker »

taopai wrote: post #60
kleton wrote: post #55

Really, all the performance intensive parts are in various c++ aggregator and recommendation type services. But the webserver is Django, yes.

Thanks for your comment. I'm really interested in this topic. How do you know the web server is Django? I searched but couldn't find this. Why would they use Django? I did some small projects in it but I assumed it wasn't very fast and wouldn't be suitable for a big app like this. I would like to know the pros and cons. Why didn't they build up something in C++ or Rust? Won't python limit the speed of responses despi…

I worked at IG on a previous iteration of Threads, with some of the engineers who wrote the current Threads app. It's Django!

(Heavily modded, run on a custom Python JIT, and using an extremely custom FB-developed database, also used for IG and FB.)

It's Django because IG was originally written in Django back in the day. FB's general approach to scaling is keep the interface generally similar, and slowly replace things behind the scenes as necessary — rather than doing big rewrites. It's seemed to work pretty well.

Ultimately the language used for the web server isn't a huge deal compared to the performance of the database, for these kinds of massive social apps. Plus, well, they do have the custom JIT — but that's very new, and when I first joined IG in 2019, we were running vanilla Python in production.

Re: Backend of Meta Threads is built with Python 3.10

#288

Post by ramesh31 »

pjmlp wrote: post #179
DoesntMatter22 wrote: post #142

Earlier quoted context omitted.

Lol. Do you think you are just going to use go and everything is going to scale to FB level size? Facebook was already at a scale larger than 99.99999 percent of sites before they had HHVM

Remember when Twitter was written in Ruby?

>Remember when Twitter was written in Ruby?

Ruby is the exception to the rule.

Friends don't let friends start new Ruby projects in 2023.

Re: Backend of Meta Threads is built with Python 3.10

#289

Post by czardoz »

whalesalad wrote: post #254
czardoz wrote: post #222

I worked on Cinder, and on the web server. Happy to answer technical questions if any :)

Who would benefit from Cinder? What kinds of workloads? The README is kinda lacking on this.

Cinder's feature set is highly optimized for IO bound web services that run under a forked-worker model.

For example: you start a main process, warm it up with a few requests, run the JIT compiler and then fork off worker processes to handle the main chunk of traffic.

As of now, it requires hand-tuning to get the best possible performance.

In terms of use cases, Cinder does the best when faced with "business logic" code (lots of inheritance, attribute lookups, method calls, etc). It can speed up numerical computations too, but you're probably better off using a library if that's the majority of the workload.

Re: Backend of Meta Threads is built with Python 3.10

#290

Post by czardoz »

ntonozzi wrote: post #245
czardoz wrote: post #222

I worked on Cinder, and on the web server. Happy to answer technical questions if any :)

What are the most impactful and likely features that you'd like to see merged into CPython?

I'd love to see lazy imports become part of the upstream at some point.
Post reply on HN