Backend of Meta Threads is built with Python 3.10 • Hacker News Board

Re: Backend of Meta Threads is built with Python 3.10

#291

Post by oaiey » Thu, Jul 06, 2023, 10:07 PM UTC

jajajajajajajaj wrote: post #269

andrewstuart wrote: post #258

If you know you’ve got gigantic scale why would you not use a compiled language? Surely it makes massive difference to hardware utilization? I like Python and javascript but there’s perfectly fine tools around now for building systems that allow one server to do a lot more. They say you should not prematurely optimize. Equally you should not be prematurely unoptimised.

If you want the real answer, it’s because Instagram was acquired and the startup used Django. That’s it.

Exactly. This is all over engineered thinking. They probably pulled the core Instagram engineering team and gave them a new task. They used what they had.

Re: Backend of Meta Threads is built with Python 3.10

#292

Post by czardoz » Thu, Jul 06, 2023, 10:07 PM UTC

vmsp wrote: post #244

czardoz wrote: post #222

I worked on Cinder, and on the web server. Happy to answer technical questions if any :)

Can you expand on the web server? Does it implement WSGI?

It has some elements of WSGI, but has expanded into its own thing which supports asyncio (this happened before ASGI was a thing).

Re: Backend of Meta Threads is built with Python 3.10

#293

Post by mrcwinn » Thu, Jul 06, 2023, 10:11 PM UTC

I’m building a tiny little product for trail runners and cyclists in Python 3.10 (Flask) at https://www.outandback.com. Python has been a lot of fun to pick up (I rely heavily on pandas and some ML libraries so it felt like the right stack).

Re: Backend of Meta Threads is built with Python 3.10

#294

Post by ericbarrett » Thu, Jul 06, 2023, 10:11 PM UTC

anayak wrote: post #131

seanp2k2 wrote: post #110

For the “Python isn’t fast enough for production backend” crowd from the same company that brought you the largest social network built on PHP + MySQL.

While that is technically correct, they haven't been running vanilla PHP for quite some time - it's jitted into x86 and run natively on the machine. It's also pretty extensively optimized. And the VM that does the jitting is all C++. Their MySQL deployment is also an internal fork that's been heavily modified for scalability (storage/compute separation, sharding, RocksDB based storage engine instead of InnoDB, Raft f…

Facebook scaled to ~1b daily actives on MySQL + InnoDB. There was lots of engineering work, like schema sharding (denormalization), automation, plenty of bug fixes and patches for MySQL (most or all contributed back to upstream, from what I remember), and of course a massive caching layer; plus throwing crazy hardware at the problem. Nonetheless the underlying engine was something any MySQL user or admin would have recognized. And we backed it all up, every day, in < 24 hours, using an unmodified mysqldump. (FB MySQL team 2009-2012)

Re: Backend of Meta Threads is built with Python 3.10

#295

Post by taopai » Thu, Jul 06, 2023, 10:18 PM UTC

reissbaker wrote: post #287

taopai wrote: post #60

Earlier quoted context omitted.

Thanks for your comment. I'm really interested in this topic. How do you know the web server is Django? I searched but couldn't find this. Why would they use Django? I did some small projects in it but I assumed it wasn't very fast and wouldn't be suitable for a big app like this. I would like to know the pros and cons. Why didn't they build up something in C++ or Rust? Won't python limit the speed of responses despi…

I worked at IG on a previous iteration of Threads, with some of the engineers who wrote the current Threads app. It's Django! (Heavily modded, run on a custom Python JIT, and using an extremely custom FB-developed database, also used for IG and FB.) It's Django because IG was originally written in Django back in the day. FB's general approach to scaling is keep the interface generally similar, and slowly replace thin…

Thank you, awesome answer! HN comunity is awesome.

Thanks also to every other answer I received for my question, I appreciated all of them.

Re: Backend of Meta Threads is built with Python 3.10

#296

Post by _fdsfs12123 » Thu, Jul 06, 2023, 10:20 PM UTC

thewataccount wrote: post #136

appplication wrote: post #99

Earlier quoted context omitted.

How is Django now? I’m a long time python dev (data eng space) but new-ish to web dev. I started a Flask project 2 years ago and found it to be pretty full of footguns, mixed messaging on best practices for scalable apps, and the ecosystem feels overbloated with vapor ware extensions. Is this just a Flask problem, or does Django have the same issues?

You simply can't beat Django's ORM for general stuff. It's too awesome. This alone makes it so hard to choose anything else. I know django doesn't have that "shiny factor" to it these days - but it's very reliable. > mixed messaging on best practices for scalable apps The WSGI stuff can be kinda confusing and is used across a lot of python frameworks including django and I think flask? My advice for "simple scaling"…

> Use celery immediately if you have any "long lived" tasks such as email

Hey, quick question from a relative newbie who is currently trying to solve this exact problem.

Besides Celery, what are good options for handling long-running requests with Django?

I see 3 options:

- Use Celery or django Q to offload processing to worker nodes (how do you deliver results from the worker node back to the FE client?)

- Use a library called django channels that I think supports all sorts of non-trivial use cases (jobs, websockets, long polling).

- Convert sync Django to use ASGI and async views and run it using uvicorn. This option is super convoluted based on this talk [0], because you have to ensure all middleware supports ASGI, and because the ORM is sync-only, so seems like very easy to shoot yourself in the foot.

The added complication, like I mentioned, is that my long-running requests need to return data back to the client in the browser. Not sure how to make it happen yet -- using a websocket connection, or long polling?

Sorry I am ambushing you randomly in the comments like this, but it sounds like you know Django well so maybe you have some insights.

---

[0] Async Django by Ivaylo Donchev https://www.youtube.com/watch?v=UJzjdJGS1BM

Re: Backend of Meta Threads is built with Python 3.10

#297

Post by epolanski » Thu, Jul 06, 2023, 10:21 PM UTC

ramesh31 wrote: post #288

pjmlp wrote: post #179

Earlier quoted context omitted.

Remember when Twitter was written in Ruby?

>Remember when Twitter was written in Ruby? Ruby is the exception to the rule. Friends don't let friends start new Ruby projects in 2023.

Ruby in 2023 is miles ahead it was 10 years ago.

Re: Backend of Meta Threads is built with Python 3.10

#298

Post by 9dev » Thu, Jul 06, 2023, 10:23 PM UTC

ericbarrett wrote: post #294

anayak wrote: post #131

Earlier quoted context omitted.

While that is technically correct, they haven't been running vanilla PHP for quite some time - it's jitted into x86 and run natively on the machine. It's also pretty extensively optimized. And the VM that does the jitting is all C++. Their MySQL deployment is also an internal fork that's been heavily modified for scalability (storage/compute separation, sharding, RocksDB based storage engine instead of InnoDB, Raft f…

Facebook scaled to ~1b daily actives on MySQL + InnoDB. There was lots of engineering work, like schema sharding (denormalization), automation, plenty of bug fixes and patches for MySQL (most or all contributed back to upstream, from what I remember), and of course a massive caching layer; plus throwing crazy hardware at the problem. Nonetheless the underlying engine was something any MySQL user or admin would have r…

> And we backed it all up, every day, in But… how? Considering all the transactions in flight, and everything? And did you ever Test disaster recovery with that setup?

I’ve worked on relatively big projects, but FAANG engineering is like some entirely different field of software engineering. Fascinating.

Re: Backend of Meta Threads is built with Python 3.10

#299

Post by epolanski » Thu, Jul 06, 2023, 10:26 PM UTC

Solvency wrote: post #271

jadtz wrote: post #221

Earlier quoted context omitted.

I think its pretty obvious they went with Instagram branding, as Facebook branding has been tarnished for a while.

I'm confused. It's Facebook by Meta. Instagram by Meta. Why isn't it Threads by Meta?

Tied to ig account.

Re: Backend of Meta Threads is built with Python 3.10

#300

Post by smrtinsert » Thu, Jul 06, 2023, 10:26 PM UTC

callamdelaney wrote: post #264

andrewstuart wrote: post #258

If you know you’ve got gigantic scale why would you not use a compiled language? Surely it makes massive difference to hardware utilization? I like Python and javascript but there’s perfectly fine tools around now for building systems that allow one server to do a lot more. They say you should not prematurely optimize. Equally you should not be prematurely unoptimised.

Because Zuck probably told his team to make this a few weeks ago to annoy Musk. Weird ego thing going on. Python is likely faster to push out an app like this than say rust or even go. Also why spend time optimising something that will be dead in a few months?

!remindme 2 months. After 30 million signups in a day I'm pretty sure we just saw the birth of a new Meta social network.