Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Production Twitter on one machine? 100Gbps NICs and NVMe are fast

thume.ca

Post reply on HN
Print view View original

101–110 of 500 posts

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#101

Post by threeseed »

raverbashing wrote: post #75
Thaxll wrote: post #54

Earlier quoted context omitted.

Because OP example is very simplistic and left on the table very important details, you would base 250M on a single machine? What about backups, obervability, how do you update that stack without bringing down everything ... Also this is napkin maths, this could be off by 10 or 100x which would change everything. It's very simple to make a PoC on a very powerful machine, make it ready from production serving hunderd…

> What about backups Several ways of doing this without relying on k8s > observability This doesn't require k8s neither and it's more on your app. Systemd can restart systems by itself > how do you update that stack without bringing down everything That's probably where redundancy helps the most. I wouldn't run a big service without it (but again it found be at server level)

> but again it found be at server level

Can you educate us on how to have a resilient app with no-downtime updates at the server level.

Because if you're doing this via software then it's no different to Kubernetes.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#102

Post by sayrer »

trishume wrote: post #90
habibur wrote: post #62

He will be up for surprise. HTTP with connection: keep-open can serve 100k req/sec. But that's for one client being served repeatedly over 1 connection. And this is the inflated number that's published in webserver benchmark tests. For more practical down to earth test, you need to measure performance w/o keep-alive. Request per second will drop to 12k / sec then. And that's for HTTP without encryption or ssl handsha…

I agree most HTTP server benchmarks are highly misleading in that way, and mention in my post how disappointed I am at the lack of good benchmarks. I also agree that typical HTTP servers would fall over at much lower new connection loads. I'm talking about a hypothetical HTTPS server that used optimized kernel-bypass networking. Here's a kernel-bypass HTTP server benchmarked doing 50k new connections per core second…

Userspace networking is pretty common. The chair of the IETF even wrote one: https://github.com/NTAP/quant

"Quant uses the warpcore zero-copy userspace UDP/IP stack, which in addition to running on on top of the standard Socket API has support for the netmap fast packet I/O framework, as well as the Particle and RIOT IoT stacks. Quant hence supports traditional POSIX platforms (Linux, MacOS, FreeBSD, etc.) as well as embedded systems."

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#103

Post by samsquire »

I recommend this table of latency figures for any software engineer:

https://gist.github.com/jboner/2841832

Essentially IO is expensive except within a datacenter but even in a data center, you can do a lot of loop iterations in a hot loop in the time it takes to ask a server for something.

There is a whitepaper which talks about the raw throughput and performance of single core systems outperforming scalable systems. These should be required reading of those developing distributed systems.

http://www.frankmcsherry.org/assets/COST.pdf A summary: http://dsrg.pdos.csail.mit.edu/2016/06/26/scalability-cost/

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#105

Post by pixl97 »

habibur wrote: post #62

He will be up for surprise. HTTP with connection: keep-open can serve 100k req/sec. But that's for one client being served repeatedly over 1 connection. And this is the inflated number that's published in webserver benchmark tests. For more practical down to earth test, you need to measure performance w/o keep-alive. Request per second will drop to 12k / sec then. And that's for HTTP without encryption or ssl handsha…

And I would say real life Twitter involves mostly cell phone use where we see companies like Google try to push HTTP/3 to deal with head of line issues on lossy connections. Serving at the millions of hits per day on lossy networks is going to leave you with massive numbers of connections that have been abandoned but you don't know it yet. Or connections that are behaving like they are tar pitted and running at bits per second.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#106

Post by vbezhenar »

pikdum wrote: post #67
andrewstuart wrote: post #24

Most projects I encounter these days instantly reach for kubernetes, containers and microservices or cloud functions. I find it much more appealing to just make the whole thing run on one fast machine. When you suggest this tend to people say "but scaling!", without understanding how much capacity there is in vertical. The thing most appealing about single server configs is the simplicity. The more simple a system ea…

I go with k8s even on a single server nowadays, it just makes everything so much more convenient. https://k3s.io/ makes it really easy to set up, too.

I never tried k3s, but what's wrong with kubeadm? I think that's literally two commands to run single server k8s: kubeadm init and kubectl taint something.

The only thing bad about single server kubernetes is that it'll eat like 1-2 GB of RAM by itself. When you whole server could be 256 MB, that's a lot of wasted RAM.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#107

Post by judge2020 »

paulryanrogers wrote: post #86
TillE wrote: post #43

Earlier quoted context omitted.

Docker can be super simple. Like, if I want to run a Python service, that's just a few lines in a Dockerfile and a docker-compose.yml stub. Then I can trivially deploy that anywhere.

Simple to start. Yet it is more complex at run time, which can complicate (or simplify) debugging, depending on the problem.

Observability in production is where APM solutions like Datadog, Elastic, and Sentry come in; you can go from just logging errors all the way up to continuously profiling your application and beaming log files to them to correlate with metrics and database query timings.

If you're just doing a simple application, Sentry really is the way to go, while Datadog and ELK are agent-based and more intended for complex setups and big enterprises (especially in their pricing structure/infra costs).

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#108

Post by richwater »

traceroute66 wrote: post #27
britneybitch wrote: post #15

> colo cost + total server cost/(3 year) => $18,471/year Meanwhile the company I just left was spending more than this for dozens of kubernetes clusters on AWS before signing a single customer. Sometimes I wonder what I'm still doing in this industry.

> spending more than this for dozens of kubernetes clusters on AWS before signing a single customer Yup. Cloud is 21st century Nickel & Diming. Sure it sounds cheap, everything is priced in small sounding cents per unit. But then it very quickly becomes a compounding vicious circle ... a dozen different cloud tools, each charged at cents per unit, those units often being measured in increments of hours....next thing…

> You can get quite a lot of bang for your buck in a 1/4 rack with today's kit.

Until very recently, while money was still very cheap, the time overhead it would take to manage this just was not worth the cost savings.

Even with the market falling out from under VC, I think it still is a good tradeoff for many shops.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#109

Post by SilverBirch »

I think one of the under-estimated interesting points of twitter as a business is that this is the core. Yes, Twitter is 140 characters, it's got "300m users" which is probably 5m real heavy users. So yes, you could do a lot of "140 characters, a few tweets per person, few million users" on very little hardware. But that's why Twitters a shit business!

How much RAM did your advertising network need? Becuase that is what makes twitter a business! How are you building your advertiser profiles? Where are you accounting for fast roll out of a Snapchat/Instagram/BeReal/Tiktok equivalent? Oh look, your 140 characters just turned into a few hundreds megs of video that you're going to transcode 16 different ways for Qos. Ruh Roh!

How are your 1,000 engineers going to push their code to production on one machine?

Almost always the answer to "do more work" or "buy more machines" is "buy more machines".

All I'm saying is I'd change it to "Toy twitter on one machine" not Production.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#110

Post by trishume »

PragmaticPulp wrote: post #72

Very cool exercise. I enjoyed reading it. I see a lot of comments here assuming that this proves something about Twitter being inefficient. Before you jump to conclusions, take a look at the author’s code: https://github.com/trishume/twitterperf Notably absent are things like serving HTTP , not to even mention HTTPS. This was a fun exercise in algorithms, I/O, and benchmarking. It wasn’t actually imitating anything t…

Which I think I'm perfectly clear about in the blog post. The post is mostly about napkin math systems analysis, which does cover HTTP and HTTPS.

I'm now somewhat confident I could implement this if I tried, but it would take many years, the prototype and math is to check whether there's anything that would stop me if I tried and be a fun blog post about what systems are capable of.

I've worked on a team building a system to handle millions of messages per second per machine, and spending weeks doing math and building performance prototypes like this is exactly what we did before we built it for real.

Post reply on HN