Enjoyed the write up, would be curious to see the twitter spend broken down by functionality given all the extra stuff they do. I imagine it's a non-linear relationship where the company has to burn more and more cash with every new feature (and esp things like Advertising which you need once your spend surpasses what a simple subscription can offer), more scale adds more complexity, bureaucracy and overhead (management, hr & recruiting, legal&accounting, etc). While it's likely there is waste (some of which is inevitable, see 'overhead' above) a super bareboens twitter can maybe run within one beefy machine but a 'real' twitter ends up needing millions + lots of people.
Production Twitter on one machine? 100Gbps NICs and NVMe are fast
221–230 of 500 posts
Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast
#222Earlier quoted context omitted.
Genuinely asking, why do you think Twitter needs 24 million vcpus to run? This is not apples to apples but Whatsapp is a product that entirely ran on 16 servers at the time of acquisition (1.5 billion users). It really begs the question why Twitter uses so much compute if there are companies that have operated significantly more efficiently. Twitter was unprofitable during acquisition and spent around half their reve…
They did not spend half their revenue on compute. It’s more like 20-25% for running data enters/staff for DCs. Check their earnings report. Whats app is not an applicable comparison because messages and videos are stored on the client device. Better to look at Pinterest and snap, which spend a lot on infra as well. The issue is storage, ads, and ML to name a few. For example, from 2015: “ Our Hadoop filesystems host…
Also Search (the article did says these wouldn’t fit to be fair but the discussion seems to be ignoring how much wouldn’t fit and why). Search is pretty expensive especially since to have it responsive you need the indexes to fit in memory—at least the Lucene variety, which at least in old YouTube videos Twitter used.
Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast
#223Earlier quoted context omitted.
Kubernetes just orchestrates containers. You can still run beefy machines and scale (if necessary) accordingly. If anything, Kubernetes allows you to save cost by going with a scalable number of small, inexpensive, fully utilized machines, vs one large, expensive, underused one.
I would wager that the majority of users of k8s do so on a cloud where they could provision VMs of the proper size to begin with. The utilization argument is specious.
> The utilization argument is specious.
It's not. Utilization is a key metric in capacity planning of large scalable apps.
Capacity is based upon max utilization. A scaled web app is does not have constant utilization. The parent I was responding to suggested running on one large/face instance. Ok... if you're capacity planning, are you planning for peak rps or min rps? Obviously peak. Peak times are always a fraction of your total server uptime. This means one big/fast server would be underutilized most of the time.
How do you expect to dynamically vertically scale in cloud to fit demand while using a single server? Re-provision another server (either smaller or larger), redeploy all apps to the server, and then route traffic? Great, you're doing kubernetes job by hand.
Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast
#224Good analysis. Obviously, this doesn't handle cases like redundancy and doesn't handle some of other critical workloads the company has. However, it does show how much real compute bloat these companies actually have - https://twitter.com/petrillic/status/1593686223717269504 where they use 24 million vcpus and spend 300 million a month on cloud.
On the other hand, Twitter does (or did) handle over 450 million monthly active users (based on stats websites), with a target for 315 monetizable daily active users (based on their earnings calls pre-privatization). Handling that amount of concurrency and beaming millions of tweets a day to home feeds and notifications is going to be logistically hard.
Anyone know if MAU includes API clients? Cuz there’s a lot of apps built on top of twitter that pull data right? Thats add a lot of traffic too.
Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast
#225I am both embarrassed and disappointed with the negativity this post has attracted. A litanny of "gotchas", where someone attempts to best the OP. What about x, y and z? It can't possibly scale. Twitter is so much more than this, etc. The OP isn't making the assertion that Twitter should replace their current system with a single large machine. The whole thread paints a picture of HN like it is full of a bunch of hal…
I must admit, as a bystander I'm torn here. I love that Tristan put out this post and made it so detailed with plenty of assumptions to cover. I also like to hear about possible issues and assumptions which the crowd calls out. Even naysayers can be helpful. I want both, but I don't want to crowd to go to far and kill the desire to produce this kind of content.
To me most interesting are factors I didn't consider in features I did cover. Next most interesting are features I didn't cover which are kinda core to Twitter being good, and also pose interesting performance problems, like the person who mentioned spam/abuse detection. After that are non-core features which pose interesting performance problems that are different from problems I already covered.
The comments that I think aren't contributing much are ones that mention features that I didn't cover but make no attempt to argue that they're actually hard to implement efficiently, or that assert that because I didn't implement something it isn't feasible to make as fast as I calculate, without arguing what would actually stop me from implementing something that efficient. Or ones who repeat that this isn't practical, which I say at length in the post.
Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast
#226Earlier quoted context omitted.
My friend mentioned this just before I published and I think that probably is the fastest largest thing you can get which would in some sense count as one machine. I haven't looked into it, but I wouldn't be surprised if they could get around the trickiest constraint, which is how many hard drives you can plug in to a non-mainframe machine for historical image storage. Definitely more expensive than just networking a…
Incidentally, a lot of people have argued that the massive datacenters used by e.g. AWS are effectively single large ("warehouse-scale") computers. In a way, it seems that the mainframe has been reinvented.
Oxide is basically building a minicomputer.
Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast
#227Earlier quoted context omitted.
But evidence is not necessarily proof.
Sure, but if someone accuses your house of having issues, and you retort that you've had it inspected by professionals, a reply of "Hah! That's evidence , not proof!" is just a bit smarmy.
A few weeks ago there was in incident[0] in Jersey, where some people called fire fighters one evening because they could smell gas, the fire fighters didn’t find any leaks, and the building literally blew up the next morning. Experts make mistakes, and failing to understand that evidence != proof can literally kill people. Sometimes, making the distinction is smarmy; other times, it’s just being sensible.
0. https://news.sky.com/story/amp/jersey-tower-explosion-questi...
Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast
#228It is all fun running on one machine until a capacitor leaks or something else goes south.
Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast
#229He will be up for surprise. HTTP with connection: keep-open can serve 100k req/sec. But that's for one client being served repeatedly over 1 connection. And this is the inflated number that's published in webserver benchmark tests. For more practical down to earth test, you need to measure performance w/o keep-alive. Request per second will drop to 12k / sec then. And that's for HTTP without encryption or ssl handsha…
And I would say real life Twitter involves mostly cell phone use where we see companies like Google try to push HTTP/3 to deal with head of line issues on lossy connections. Serving at the millions of hits per day on lossy networks is going to leave you with massive numbers of connections that have been abandoned but you don't know it yet. Or connections that are behaving like they are tar pitted and running at bits…
Vertical scaling doesn't have to be a single machine. You can do a lot with a half dozen machines split for different responsibilities, like we did in the 90's and 00's. Database, web servers, reverse proxy.
Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast
#230Earlier quoted context omitted.
I wonder how much of that is crypto spam bots replying to each other.
Can you detect and block those spam bots with less effort than it would take to process them?
That’s a not so simple calculation as comparing the raw request process time isn’t the complete picture. Spam bot content must be persistently stored, indexed, archived, etc so the long term cost is much more than the one off POST to create the entity.
You’d also have to quantify the improved user experience from seeing less spam v.s. inflated ad revenue for garbage views / content.