Live data from Hacker News

Production Twitter on one machine? 100Gbps NICs and NVMe are fast

thume.ca

331–340 of 500 posts

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#331
When I was working there I implemented my patent during a hack week (given a set of follows return the list of matching tweet ids, very similar to his prototype):

https://patents.google.com/patent/US20120136905A1/en (licensed under Innovators Patent Agreement, https://github.com/twitter/innovators-patent-agreement)

I could have definitely served all the chronological timeline requests on a normal server with lower latency that the 1.1 home timeline API. There are a bunch of numbers in the calculations that he is doing that are off but not by an order of magnitude. The big issue is that since I left back then Twitter has added ML ads, ML timeline and other features that make current Twitter much harder to fit on a machine than 2013 Twitter.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#332

Getting everything onto one machine works great until... it no longer fits on one machine. You add another feature and it requires a little bit more RAM, and another feature that needs a little bit more, and.. eventually it doesn't all fit. Now you have to go distributed. And your entire system architecture and all your development approaches are built around the assumptions of locality and cache line optimization an…

Weird criticism but o.k.

Edit: Unless I missed something, the author never argued that Twitter should be hosted on one machine and therefore criticizing the “fun stunt” like this makes no sense to me

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#333

Getting everything onto one machine works great until... it no longer fits on one machine. You add another feature and it requires a little bit more RAM, and another feature that needs a little bit more, and.. eventually it doesn't all fit. Now you have to go distributed. And your entire system architecture and all your development approaches are built around the assumptions of locality and cache line optimization an…

> You certainly could build server applications that way. But it feels like there's something fundamental to how service businesses operate that pushes away from that kind of hyperoptimized model.

That "fundamental" thing is the cultural expectation that SaaS offerings constantly grow in features, rather than in reliability or performance. As your example from the world of video games demonstrates, there is no industry-wide belief that things must be able to do ever-more, forever. It's really mostly SaaS and desktop software that has this weird and unreasonable culture around it. That's why your word processor can now send emails, and your email provider now does translations as well.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#334
post #333

Getting everything onto one machine works great until... it no longer fits on one machine. You add another feature and it requires a little bit more RAM, and another feature that needs a little bit more, and.. eventually it doesn't all fit. Now you have to go distributed. And your entire system architecture and all your development approaches are built around the assumptions of locality and cache line optimization an…

> You certainly could build server applications that way. But it feels like there's something fundamental to how service businesses operate that pushes away from that kind of hyperoptimized model. That "fundamental" thing is the cultural expectation that SaaS offerings constantly grow in features, rather than in reliability or performance. As your example from the world of video games demonstrates, there is no indust…

You're not taking into account data, you're only talking about features. What about when the data no longer fits on the one machine? Or processing the data exceeds the capacity of the machine?

Data growth through user growth or just normal day-to-day usage is expected.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#335
post #329
post #320

Earlier quoted context omitted.

Don't you need 3 numbers? Unless you believe in a flat earth ;). Also you need some slack space for metadata. Let's call it 100GB all in.

Lat long and assuming at ground gets you most of the way there.

"Add it all up, and the US has around 340 billion square feet of building stock[3]. This is about 12,200 square miles, or 0.00032% of US land area"

Judging by that, you need a negligible increase in the number of locations you can represent to handle everywhere stable and off the ground someone could be. Much less than one bit per person.

If you want to deal with people currently in airplanes then you could give them an extra couple bytes. It's less than a million people so it won't affect your total storage at all.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#336

Getting everything onto one machine works great until... it no longer fits on one machine. You add another feature and it requires a little bit more RAM, and another feature that needs a little bit more, and.. eventually it doesn't all fit. Now you have to go distributed. And your entire system architecture and all your development approaches are built around the assumptions of locality and cache line optimization an…

It's sort of strange you have to make these points, but as an industry we seem to have an extremely short memory.

Vertical scaling was absolutely the way most big applications were built up until well into the 90s. Companies like Oracle were really built on the fact that getting performance and reliability out of a single highly-contested massive server is hard but important if that's the way you're going. Linux became dominant primarily because horizontal scaling won that argument and it won it pretty much exactly because of:

1)what you said - you hit a hard cap on how big you can make your main server at which point you are really screwed. Scalability pain points become a hard wall.

2) when I say "server" I mean "servers" of course because you'd need an H/A failover, at which point you've eaten the cost of replication, handling failover etc and you may as well distribute

3) cost. Because hardware cost vs capability is exponential, as your requirements become bigger you pretty rapidly hit a point where lots of commodity hardware becomes cheaper for a given performance point than few big servers

So there's a reason that distributed systems on commodity hardware became the dominant architectural paradigm. It's not the only way to do it, but it's a reasonable default for many use cases. For a very high-throughput system like twitter it seems a very obvious choice.

Clearly there are costs to distribution, so if you can get away with a simpler architecture then as always Occam's razor applies. Also if you can easily distribute later then it probably makes sense to leave that option open and explore it when you need it rather than overcomplicate too early.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#338

Getting everything onto one machine works great until... it no longer fits on one machine. You add another feature and it requires a little bit more RAM, and another feature that needs a little bit more, and.. eventually it doesn't all fit. Now you have to go distributed. And your entire system architecture and all your development approaches are built around the assumptions of locality and cache line optimization an…

How about smaller projects that might be hugely successful, but will never be remotely close to a Twitter level of success?

It’s interesting to see how much can be done with a single machine, because most projects will never be this big.

Though there will still be other concerns like redundancy to deal with.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#339
No rate limiting, API data, quote tweets, view count, threads, likes, mentions, notifications, ads, video, images, account blocking (permanent or TTL), account muting (permanent or TTL), word filtering (permanent or TTL), moderation/reporting, user profile storage, or the fact that tweets that display show more than just the tweet itself. No mention that tweet activity all occurs concurrently and therefore the loading script is not at all a realistic estimate of real activity.

But sure, go ahead and take this as evidence that 10 people could build Twitter as I'm sure that's what will happen to this post. If that's true, why haven't they already done so? It should only take a couple weeks and one beefy machine, right?

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#340
post #63

In the coming years we will probably see a lot of complicated microservice architectures be replaced by well-designed and optimized Rust (and modern C++) monoliths that use simple replication to scale horizontally.

You don't even need Rust or highly optimised code. Just moving the existing code from "vCPUs" and networked storage to real CPUs with direct-attach NVME storage will be enough for most purposes. (btw you can do that now, just get yourself a beefy server at OVH/Hetzner and play around with it)
Post reply on HN