351–360 of 458 posts
For the “Python isn’t fast enough for production backend” crowd from the same company that brought you the largest social network built on PHP + MySQL.
the claim is usually more about how the maintenance costs are just way higher versus a statically typed language
Earlier quoted context omitted.
Facebook scaled to ~1b daily actives on MySQL + InnoDB. There was lots of engineering work, like schema sharding (denormalization), automation, plenty of bug fixes and patches for MySQL (most or all contributed back to upstream, from what I remember), and of course a massive caching layer; plus throwing crazy hardware at the problem. Nonetheless the underlying engine was something any MySQL user or admin would have r…
> And we backed it all up, every day, in But… how? Considering all the transactions in flight, and everything? And did you ever Test disaster recovery with that setup? I’ve worked on relatively big projects, but FAANG engineering is like some entirely different field of software engineering. Fascinating.
> Considering all the transactions in flight, and everything?
If I remember, we used --flush-logs --master-data=2 --single-transaction, giving it a consistent point-in-time dump of the schemas, with a recorded starting point for binlog replays, enabling point-in-time and up-to-the-minute restores. Nowadays you have GTIDs so these flags are obsolete (except --single-transaction).
--single-transaction does put extra load on the database—I think it was undo logs? it's been a minute—which caused some hair-pulling, and I believe they eventually moved to xtrabackup, before RocksDB. But binary backups were too space-hungry when I was there, so we made it work.
Another unexpected advantage of mysqldump vs. xtrabackup, besides size, was when a disk error caused silent data corruption on the underlying file system. Mysqldump often read from InnoDB's buffer cache, which still had the good data in it. Or if the bad block was paged back in, it wouldn't have a valid structure and mysqld would panic, so we knew we had to restore.
> And did you ever Test disaster recovery with that setup?
Yes! I wrote the first version of ORC. This blog post is from long after I left, but it's a good summary of how it worked: https://engineering.fb.com/2016/10/28/data-infrastructure/co...
It wasn't the best code (sorry Divij)—the main thing I'm proud of was the recursive acronym and the silly Warcraft theme. But it did the job.
Two things I remember about developing ORC:
1) The first version was an utter disaster. I was just learning Python, and I hit the global interpreter lock super hard, type-error crashes everywhere, etc. I ended up abandoning the project and restarting it a few months later, which became ORC. In the interim I did a few other Python projects and got somewhat better.
2) Another blocker the first version had was that the clients updated their status by doing SELECT...FOR UPDATE to a central table, en masse, which turns out to be really bad practice with MySQL. The database got lock-jammed, and I remember Domas Mituzas walking over to my desk demanding to know what I was doing. Hey, I never said I was a DBA! Anyway, that's why ORC ended up having the Warchief/Peon model—the Peons would push their status to the Warchief (or be polled, I forgot), so there was only a single writer to that table, requiring no hare-brained locking.
Earlier quoted context omitted.
> And we backed it all up, every day, in But… how? Considering all the transactions in flight, and everything? And did you ever Test disaster recovery with that setup? I’ve worked on relatively big projects, but FAANG engineering is like some entirely different field of software engineering. Fascinating.
(a) make each individual database small, but have a lot of them (b) There are lots of transactions in flight, but there is a well-ordered sequence of mutations (the binlog) that defines what has and has not been committed. So applying a backup means taking the full backup + replaying the binlogs. (c) testing can be done by just bringing up a slave from the backup and then comparing consistency with normal replicas.
Earlier quoted context omitted.
Facebook scaled to ~1b daily actives on MySQL + InnoDB. There was lots of engineering work, like schema sharding (denormalization), automation, plenty of bug fixes and patches for MySQL (most or all contributed back to upstream, from what I remember), and of course a massive caching layer; plus throwing crazy hardware at the problem. Nonetheless the underlying engine was something any MySQL user or admin would have r…
It sounds like you were involved in this. Since you were working there so long ago, would you be willing to write up a technical account of the things you did? I'd be interested in learning more about it. I figure the tech from 10 years ago is outdated enough that it wouldn't cause any issues if you made it public.
Earlier quoted context omitted.
My few years of personal experience of using Python with even small-ish scale professional projects have been... underwhelming so far. I wonder how Meta (and Google, and all the others who write serious Python code) manage to keep things running – and I'm not talking (only) of performance, but largely of bugs. My personal suspicion is that the secret sauce is simply to have a sufficient budget to hire an army of code…
>My few years of personal experience of using Python with even small-ish scale professional projects have been... underwhelming so far You have years of experience but couldn't point to anything in particular? Heck, I like python and I can complain about dynamic typing issues in for loops, or that they are adding features like generators/decorators that make code more difficult to understand, which goes against the z…
Those features are 20ish years old which made the word "adding" seem a little weird. Interestingly the Zen of Python itself is not much older than generators.
Earlier quoted context omitted.
I have ported code from Cinder to CPython . The fork has some optimizations that can be easily put in CPython and Facebook is open to port features. I’m not sure if Facebook wants to continually have a fork but CPython is open to have those features merged in if they make sense.
The Cinder team's longterm goal is to upstream as much as we can, and make the rest available as pip-installable extensions that anyone could theoretically install and use on CPython. The fewer internal changes and patches we need to maintain, the faster we can adopt upstream Python releases and all of the associated performance and tooling wins.
Earlier quoted context omitted.
Ehh, Django ORM always was one of the worst parts of Django.
True, but it's the best ORM currently available for any language. It might not be the fastest, but it is the one that's has the highest level of developer comfort. Using Django is probably the reason why I can stand using SQLAlchemy, it's way to complicated for everything I do and it's just not a nice an experience.
Django ORM is tolerable on small projects, and quickly gets in the way on anything a bit more complicated.
Earlier quoted context omitted.
> all the 100 users That’s off by multiple orders of magnitude, if you believe their reports of 30 million sign ups in under a day. https://www.theverge.com/2023/7/6/23786108/threads-internal-...
You mean sign-in's?