Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Production Twitter on one machine? 100Gbps NICs and NVMe are fast

thume.ca

Post reply on HN
Print view View original

61–70 of 500 posts

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#61

Post by fifilura »

andrewstuart wrote: post #24

Most projects I encounter these days instantly reach for kubernetes, containers and microservices or cloud functions. I find it much more appealing to just make the whole thing run on one fast machine. When you suggest this tend to people say "but scaling!", without understanding how much capacity there is in vertical. The thing most appealing about single server configs is the simplicity. The more simple a system ea…

I dont think k8s is what you are shooting at but the IPC that is required to run a set of microservices.

Like kafka.

My impression is that it is the serialisation that comes with each service-to-service communication that is really expensive.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#62

Post by habibur »

He will be up for surprise.

HTTP with connection: keep-open can serve 100k req/sec. But that's for one client being served repeatedly over 1 connection. And this is the inflated number that's published in webserver benchmark tests.

For more practical down to earth test, you need to measure performance w/o keep-alive. Request per second will drop to 12k / sec then.

And that's for HTTP without encryption or ssl handshake. Use HTTPS and watch it fall down to only 400 req / sec under load test [ without connection: keep-alive ].

That's what I observer.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#64

Post by jupp0r »

Thaxll wrote: post #54
andrewstuart wrote: post #24

Most projects I encounter these days instantly reach for kubernetes, containers and microservices or cloud functions. I find it much more appealing to just make the whole thing run on one fast machine. When you suggest this tend to people say "but scaling!", without understanding how much capacity there is in vertical. The thing most appealing about single server configs is the simplicity. The more simple a system ea…

Because OP example is very simplistic and left on the table very important details, you would base 250M on a single machine? What about backups, obervability, how do you update that stack without bringing down everything ... Also this is napkin maths, this could be off by 10 or 100x which would change everything. It's very simple to make a PoC on a very powerful machine, make it ready from production serving hunderd…

In addition, you should worry about what happens to your app if a hardware error, network problem or natural disaster makes your machine unavailable.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#65

Post by PragmaticPulp »

varunkmohan wrote: post #13

Good analysis. Obviously, this doesn't handle cases like redundancy and doesn't handle some of other critical workloads the company has. However, it does show how much real compute bloat these companies actually have - https://twitter.com/petrillic/status/1593686223717269504 where they use 24 million vcpus and spend 300 million a month on cloud.

> However, it does show how much real compute bloat these companies actually have

No, it doesn’t. It’s a fun exercise in approaching Twitter as an academic exercise. It ignores all of the real-world functionality that makes it a business rather than a toy.

A lot of complicated businesses are easy to prototype out if you discard all requirements other than the core feature. In the real world, more engineering work often goes to ancillary features that you never see as an end user.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#66

Post by toast0 »

trishume wrote: post #21
jasonhansel wrote: post #19

If you really wanted to run Twitter on one machine at any cost, wouldn't an IBM mainframe be much more practical? You can even run Linux on them now. The specs he cites would actually be fairly small for a mainframe, which can reach up to 40TB of memory. I'm not saying this is a good idea, but it seems better than what the OP proposes.

My friend mentioned this just before I published and I think that probably is the fastest largest thing you can get which would in some sense count as one machine. I haven't looked into it, but I wouldn't be surprised if they could get around the trickiest constraint, which is how many hard drives you can plug in to a non-mainframe machine for historical image storage. Definitely more expensive than just networking a…

> I wouldn't be surprised if they could get around the trickiest constraint, which is how many hard drives you can plug in to a non-mainframe machine for historical image storage.

Some commodity machines use external SAS to connect to more disk boxes. IMHO, there's not a real reason to keep images and tweets on the same server if you're going to need an external disk box anyway. Rather than getting a 4u server with a lot of disks and a 4u additional disk box, you may as well get 4u servers with a lot of disks each, use one for tweets and the other for images. Anyway, images are fairly easy to scale horizontally, there's not much simplicity gained by having them all in one host, like there is for tweets.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#67

Post by pikdum »

andrewstuart wrote: post #24

Most projects I encounter these days instantly reach for kubernetes, containers and microservices or cloud functions. I find it much more appealing to just make the whole thing run on one fast machine. When you suggest this tend to people say "but scaling!", without understanding how much capacity there is in vertical. The thing most appealing about single server configs is the simplicity. The more simple a system ea…

I go with k8s even on a single server nowadays, it just makes everything so much more convenient.

https://k3s.io/ makes it really easy to set up, too.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#68

Post by eddsh1994 »

jupp0r wrote: post #64
Thaxll wrote: post #54

Earlier quoted context omitted.

Because OP example is very simplistic and left on the table very important details, you would base 250M on a single machine? What about backups, obervability, how do you update that stack without bringing down everything ... Also this is napkin maths, this could be off by 10 or 100x which would change everything. It's very simple to make a PoC on a very powerful machine, make it ready from production serving hunderd…

In addition, you should worry about what happens to your app if a hardware error, network problem or natural disaster makes your machine unavailable.

Split the DB from the app and replicate with a load balancer?

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#69

Post by trishume »

BeefWellington wrote: post #40

I'm going to preface this criticism by saying that I think exercises like this are fun in an architectural/prototyping code-golf kinda way. However, I think the author critically under-guesses the sizes of things (even just for storage) by a reasonably substantial amount. e.g.: Quote tweets do not go against the size limit of the tweet field at Twitter. Likely they are embedding a tweet reference in some manner or ot…

Quote tweets I'd do as a reference and they'd basically have the cost of loading 2 tweets instead of one, so increasing the delivery rate by the fraction of tweets that are quote tweets.

Hashtags are a search feature and basically need the same posting lists as for search, but if you only support hashtags the posting lists are smaller. I already have an estimate saying probably search wouldn't fit. But I think hashtag-only search might fit, mainly because my impression is people doing hashtag searches are a small fraction of traffic nowadays so the main cost is disk, not sure though.

I did run the post by 5 ex-Twitter engineers and none of them said any of my estimates were super wrong, mainly just brought up additional features and things I didn't discuss (which I edited into the post before publishing). Still possible that they just didn't divulge or didn't know some number they knew that I estimated very wrong.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#70

Post by jupp0r »

vasco wrote: post #37
andrewstuart wrote: post #24

Most projects I encounter these days instantly reach for kubernetes, containers and microservices or cloud functions. I find it much more appealing to just make the whole thing run on one fast machine. When you suggest this tend to people say "but scaling!", without understanding how much capacity there is in vertical. The thing most appealing about single server configs is the simplicity. The more simple a system ea…

Kubernetes is useful if you have many teams working on things in parallel and you want them to deploy in similar ways to not have to reinvent the same wheel in 5 different ways by 5 different teams. If you don't have multiple teams, you don't need it.

It's also useful if you want your app to update without being down, which even a single team might want to do.
Post reply on HN