Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Production Twitter on one machine? 100Gbps NICs and NVMe are fast

thume.ca

Post reply on HN
Print view View original

171–180 of 500 posts

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#171

Post by peterhunt »

varunkmohan wrote: post #119
PragmaticPulp wrote: post #65

Earlier quoted context omitted.

> However, it does show how much real compute bloat these companies actually have No, it doesn’t. It’s a fun exercise in approaching Twitter as an academic exercise. It ignores all of the real-world functionality that makes it a business rather than a toy. A lot of complicated businesses are easy to prototype out if you discard all requirements other than the core feature. In the real world, more engineering work oft…

Genuinely asking, why do you think Twitter needs 24 million vcpus to run? This is not apples to apples but Whatsapp is a product that entirely ran on 16 servers at the time of acquisition (1.5 billion users). It really begs the question why Twitter uses so much compute if there are companies that have operated significantly more efficiently. Twitter was unprofitable during acquisition and spent around half their reve…

The real answer is twofold:

1. Lots of batch jobs. Sometimes it's unclear how much value they produce / whether they're still used.

2. Twitter probably made a mistake early on in taking a fanout-on-write approach to populate feeds. This is super expensive and necessitates a lot of additional infrastructure. There is a good video about it here: https://www.youtube.com/watch?v=WEgCjwyXvwc

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#172

Post by 867-5309 »

mr90210 wrote: post #115
trishume wrote: post #69

Earlier quoted context omitted.

Quote tweets I'd do as a reference and they'd basically have the cost of loading 2 tweets instead of one, so increasing the delivery rate by the fraction of tweets that are quote tweets. Hashtags are a search feature and basically need the same posting lists as for search, but if you only support hashtags the posting lists are smaller. I already have an estimate saying probably search wouldn't fit. But I think hashta…

[flagged]

>Absence of evidence is not evidence of absence.

>possible that they just didn't divulge

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#173

Post by mschuster91 »

trishume wrote: post #21
jasonhansel wrote: post #19

If you really wanted to run Twitter on one machine at any cost, wouldn't an IBM mainframe be much more practical? You can even run Linux on them now. The specs he cites would actually be fairly small for a mainframe, which can reach up to 40TB of memory. I'm not saying this is a good idea, but it seems better than what the OP proposes.

My friend mentioned this just before I published and I think that probably is the fastest largest thing you can get which would in some sense count as one machine. I haven't looked into it, but I wouldn't be surprised if they could get around the trickiest constraint, which is how many hard drives you can plug in to a non-mainframe machine for historical image storage. Definitely more expensive than just networking a…

> I haven't looked into it, but I wouldn't be surprised if they could get around the trickiest constraint, which is how many hard drives you can plug in to a non-mainframe machine for historical image storage.

Netapp is at something > 300TB storage per node IIRC, but in any case it would make more sense to use some cloud service. AWS EFS and S3 don't have any (practically reachable) limit in size.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#174

Post by lelandfe »

mr90210 wrote: post #115
trishume wrote: post #69

Earlier quoted context omitted.

Quote tweets I'd do as a reference and they'd basically have the cost of loading 2 tweets instead of one, so increasing the delivery rate by the fraction of tweets that are quote tweets. Hashtags are a search feature and basically need the same posting lists as for search, but if you only support hashtags the posting lists are smaller. I already have an estimate saying probably search wouldn't fit. But I think hashta…

[flagged]

If an inspector reviews your house and finds no issues, that is indeed evidence of absence.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#175

Post by kevingadd »

lazyasciiart wrote: post #85
jeffbee wrote: post #12

I like this kind of exercise. One thing I am not seeing is analytics, logs and so forth that as I understand it are significant portions of Twitter's production cost story.

If it's this cheap to run you don't need analytics because you don't need to monetize it, and if it's this simple you don't need logs because it'll all work correctly the first time!

"You don't need to monetize it" who's going to fund your Twitter-as-a-charity? What happens when the free money goes away? Businesses have to pay the bills eventually one way or another, you need to plan for that in advance

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#176

Post by seritools »

threeseed wrote: post #132
sethev wrote: post #23

John Carmack tweeted something that made me noodle on this too: >It is amusing to consider how much of the world you could serve something like Twitter to from a single beefy server if it really was just shuffling tweet sized buffers to network offload cards. Smart clients instead of web pages could make a very large difference. [1] Very interesting to see the idea worked out in more detail. [1] https://twitter.com/i…

> just shuffling tweet sized buffers to network offload cards Except that's not what it is doing at all. It assembles all the Tweets internally, applies an ML model to produce a finalised response to the user.

> if it really was [which it isn't]

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#177

Post by mapme »

varunkmohan wrote: post #119
PragmaticPulp wrote: post #65

Earlier quoted context omitted.

> However, it does show how much real compute bloat these companies actually have No, it doesn’t. It’s a fun exercise in approaching Twitter as an academic exercise. It ignores all of the real-world functionality that makes it a business rather than a toy. A lot of complicated businesses are easy to prototype out if you discard all requirements other than the core feature. In the real world, more engineering work oft…

Genuinely asking, why do you think Twitter needs 24 million vcpus to run? This is not apples to apples but Whatsapp is a product that entirely ran on 16 servers at the time of acquisition (1.5 billion users). It really begs the question why Twitter uses so much compute if there are companies that have operated significantly more efficiently. Twitter was unprofitable during acquisition and spent around half their reve…

They did not spend half their revenue on compute. It’s more like 20-25% for running data enters/staff for DCs. Check their earnings report.

Whats app is not an applicable comparison because messages and videos are stored on the client device. Better to look at Pinterest and snap, which spend a lot on infra as well.

The issue is storage, ads, and ML to name a few. For example, from 2015:

“ Our Hadoop filesystems host over 300PB of data on tens of thousands of servers. We scale HDFS by federating multiple namespaces.”

You can also see their hardware usage broken down by service as put in their blog.

https://blog.twitter.com/engineering/en_us/topics/infrastruc...

https://blog.twitter.com/engineering/en_us/a/2015/hadoop-fil....

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#178

Post by _zoltan_ »

cortesoft wrote: post #95
drewg123 wrote: post #42

How much bandwidth does Twitter use for images and videos? Less than 1.4Tb/s globally? If so, we could probably fit that onto a second machine. We can currently serve over 700Gb/s from a dual-socket Milan based server[1]. I'm still waiting for hardware, but assuming there are no new bottlenecks, that should directly scale up to 1.4Tb/s with Genoa and ConnectX-7, given the IO pathways are all at least twice the bandwi…

It is way more than 1.4TBs a second globally.

in this specific discussion it's very important to use Tb and TB correctly.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#179

Post by threeseed »

jiggawatts wrote: post #162

Something I've found a lot modern IT architects seem to ignore is "write amplification" or the equivalent effect for reads. If you have a 1 KB piece of data that you need to send to a customer, ideally that should require less than 1 KB of actual NIC traffic thanks to HTTP compression. If processing that 1 KB takes more than 1 KB of total NIC traffic within and out of your data centre, the you have some level of ampl…

Your point makes sense if you have no idea how Twitter works.

It needs to assemble tweets internally, sort them with an ML model, add in relevant ads and present a single response to the user because end-user latency matters.

And each of these systems eg. ads has their own features, complexities, development lifecycle and scaling requirements. And of course deploying them continuously without downtime. That is how you end up with disparate services and a lot of them for redundancy reasons.

I know you think you’re smarter than everyone at Twitter. But those who really know what they are doing have a lot more respect for the engineers who built this insanity. There are always good intentions.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#180

Post by sitkack »

I am both embarrassed and disappointed with the negativity this post has attracted.

A litanny of "gotchas", where someone attempts to best the OP. What about x, y and z? It can't possibly scale. Twitter is so much more than this, etc.

The OP isn't making the assertion that Twitter should replace their current system with a single large machine.

The whole thread paints a picture of HN like it is full of a bunch of half-educated, uncreative negative brats.

To the people that encourage a fun discussion, thank you! Great things are not built by people who only see how something cannot possibly work.

Post reply on HN