Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Production Twitter on one machine? 100Gbps NICs and NVMe are fast

thume.ca

Post reply on HN
Print view View original

21–30 of 500 posts

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#21

Post by trishume »

jasonhansel wrote: post #19

If you really wanted to run Twitter on one machine at any cost, wouldn't an IBM mainframe be much more practical? You can even run Linux on them now. The specs he cites would actually be fairly small for a mainframe, which can reach up to 40TB of memory. I'm not saying this is a good idea, but it seems better than what the OP proposes.

My friend mentioned this just before I published and I think that probably is the fastest largest thing you can get which would in some sense count as one machine. I haven't looked into it, but I wouldn't be surprised if they could get around the trickiest constraint, which is how many hard drives you can plug in to a non-mainframe machine for historical image storage. Definitely more expensive than just networking a few standard machines though.

I also bet that mainframes have software solutions to a lot of the multi-tenancy and fault tolerance challenges with running systems on one machine that I mention.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#23

Post by sethev »

John Carmack tweeted something that made me noodle on this too:

>It is amusing to consider how much of the world you could serve something like Twitter to from a single beefy server if it really was just shuffling tweet sized buffers to network offload cards. Smart clients instead of web pages could make a very large difference. [1]

Very interesting to see the idea worked out in more detail.

[1] https://twitter.com/id_aa_carmack/status/1350672098029694998

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#24

Post by andrewstuart »

Most projects I encounter these days instantly reach for kubernetes, containers and microservices or cloud functions.

I find it much more appealing to just make the whole thing run on one fast machine. When you suggest this tend to people say "but scaling!", without understanding how much capacity there is in vertical.

The thing most appealing about single server configs is the simplicity. The more simple a system easy, likely the more reliable and easy to understand.

The software thing most people are building these days can easily run lock stock and barrel on one machine.

I wrote a prototype for an in-memory message queue in Rust and ran it on the fastest EC2 instance I could and it was able to process nearly 8 million messages a second.

You could be forgiven for believing the only way to write software is is a giant kablooie of containers, microservices, cloud functions and kubernetes, because that's what the cloud vendors want you to do, and it's also because it seems to be the primary approach discussed. Every layer of such stuff add complexity, development, devops, maintenance, support, deployment, testing and (un)reliability. Single server systems can be dramatically mnore simple because you can trim is as close as possible down to just the code and the storage.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#25

Post by judge2020 »

andrewstuart wrote: post #24

Most projects I encounter these days instantly reach for kubernetes, containers and microservices or cloud functions. I find it much more appealing to just make the whole thing run on one fast machine. When you suggest this tend to people say "but scaling!", without understanding how much capacity there is in vertical. The thing most appealing about single server configs is the simplicity. The more simple a system ea…

Kubernetes and containers are a means to service architecture; It enabled scalability but does not require it. You should still be containerizing your applications to ensure a consistent environment, even if you only throw it in a docker-compose file on your production server.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#26

Post by andrewstuart »

judge2020 wrote: post #25
andrewstuart wrote: post #24

Most projects I encounter these days instantly reach for kubernetes, containers and microservices or cloud functions. I find it much more appealing to just make the whole thing run on one fast machine. When you suggest this tend to people say "but scaling!", without understanding how much capacity there is in vertical. The thing most appealing about single server configs is the simplicity. The more simple a system ea…

Kubernetes and containers are a means to service architecture; It enabled scalability but does not require it. You should still be containerizing your applications to ensure a consistent environment, even if you only throw it in a docker-compose file on your production server.

I don't even use containers - I aim primarily for simplicity and so far I have found I am able to build entire sophisticated systems without a single container. Containers I find make things much more complex.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#27

Post by traceroute66 »

britneybitch wrote: post #15

> colo cost + total server cost/(3 year) => $18,471/year Meanwhile the company I just left was spending more than this for dozens of kubernetes clusters on AWS before signing a single customer. Sometimes I wonder what I'm still doing in this industry.

> spending more than this for dozens of kubernetes clusters on AWS before signing a single customer

Yup.

Cloud is 21st century Nickel & Diming.

Sure it sounds cheap, everything is priced in small sounding cents per unit.

But then it very quickly becomes a compounding vicious circle ... a dozen different cloud tools, each charged at cents per unit, those units often being measured in increments of hours....next thing you know is your cloud bill has as many zeros on the end of it as the number of cloud services you are using. ;-)

And that's before we start talking about the data egress costs.

With colo you can start off with two 1/4 rack spaces at two different sites for resilience. You can get quite a lot of bang for your buck in a 1/4 rack with today's kit.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#28

Post by tluyben2 »

andrewstuart wrote: post #24

Most projects I encounter these days instantly reach for kubernetes, containers and microservices or cloud functions. I find it much more appealing to just make the whole thing run on one fast machine. When you suggest this tend to people say "but scaling!", without understanding how much capacity there is in vertical. The thing most appealing about single server configs is the simplicity. The more simple a system ea…

By far most indeed. And if you want failover, just run 2 two machines. More fun (for me) also than tying together all that complexity.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#29

Post by John23832 »

andrewstuart wrote: post #24

Most projects I encounter these days instantly reach for kubernetes, containers and microservices or cloud functions. I find it much more appealing to just make the whole thing run on one fast machine. When you suggest this tend to people say "but scaling!", without understanding how much capacity there is in vertical. The thing most appealing about single server configs is the simplicity. The more simple a system ea…

Kubernetes just orchestrates containers. You can still run beefy machines and scale (if necessary) accordingly.

If anything, Kubernetes allows you to save cost by going with a scalable number of small, inexpensive, fully utilized machines, vs one large, expensive, underused one.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#30

Post by Cyph0n »

trishume wrote: post #16
Cyph0n wrote: post #10

Earlier quoted context omitted.

My bad - not English, but ASCII. The assumed max tweet size is in bytes rather than (UTF) characters.

I specifically assumed a max tweet size based on the maximum number of UTF-8 bytes a tweet can contain (560), with a link to an analysis of that, and discussion of how you could optimize for the common case of tweets that contain way fewer UTF-8 bytes than that. Everything in my post assumes unicode.

Thanks for clarifying. I missed the max vs. average analysis because I was focused on the text. Still, as noted in the Rust code comment, the sample implementation doesn’t handle longer tweets.
Post reply on HN