Live data from Hacker News

Production Twitter on one machine? 100Gbps NICs and NVMe are fast

thume.ca

121–130 of 500 posts

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#121
post #93

Earlier quoted context omitted.

My friend mentioned this just before I published and I think that probably is the fastest largest thing you can get which would in some sense count as one machine. I haven't looked into it, but I wouldn't be surprised if they could get around the trickiest constraint, which is how many hard drives you can plug in to a non-mainframe machine for historical image storage. Definitely more expensive than just networking a…

It's a neat thought exercise, but wrong for so many reasons (there are probably like 100s). Some jump out: spam/abuse detection, ad relevance, open graph web previews, promoted tweets that don't appear in author timelines, blocks/mutes, etc. This program is what people think Twitter is, but there's a lot more to it. I think every big internet service uses user-space networking where required, so that part isn't new.

I think I'm pretty careful to say that this is a simplified version of Twitter. Of the features you list:

- spam detection: I agree this is a reasonably core feature and a good point. I think you could fit something here but you'd have to architect your entire spam detection approach around being able to fit, which is a pretty tricky constraint and probably would make it perform worse than a less constrained solution. Similar to ML timelines.

- ad relevance: Not a core feature if your costs are low enough. But see the ML estimates for how much throughput A100s have at dot producting ML embeddings.

- web previews: I'd do this by making it the client's responsibility. You'd lose trustworthiness though so users with hacked clients could make troll web previews, they can already do that for a site they control, but not a general site.

- blocks/mutes: Not a concern for the main timeline other than when using ML, when looking at replies will need to fetch blocks/mutes and filter. Whether this costs too much depends on how frequently people look at replies.

I'm fully aware that real Twitter has bajillions of features that I don't investigate, and you couldn't fit all of them on one machine. Many of them make up such a small fraction of load that you could still fit them. Others do indeed pose challenges, but ones similar to features I'd already discussed.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#122

> colo cost + total server cost/(3 year) => $18,471/year Meanwhile the company I just left was spending more than this for dozens of kubernetes clusters on AWS before signing a single customer. Sometimes I wonder what I'm still doing in this industry.

If they were a startup like you suggest then it's possible they were running on AWS credits.

You can get up to $100k and it's a big reason many startups go in that direction.

Also $20k is nothing when you factor in developer time etc.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#123
post #54

Most projects I encounter these days instantly reach for kubernetes, containers and microservices or cloud functions. I find it much more appealing to just make the whole thing run on one fast machine. When you suggest this tend to people say "but scaling!", without understanding how much capacity there is in vertical. The thing most appealing about single server configs is the simplicity. The more simple a system ea…

Because OP example is very simplistic and left on the table very important details, you would base 250M on a single machine? What about backups, obervability, how do you update that stack without bringing down everything ... Also this is napkin maths, this could be off by 10 or 100x which would change everything. It's very simple to make a PoC on a very powerful machine, make it ready from production serving hunderd…

It is a BoE system design. How is it off by 100x?

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#124
post #23

John Carmack tweeted something that made me noodle on this too: >It is amusing to consider how much of the world you could serve something like Twitter to from a single beefy server if it really was just shuffling tweet sized buffers to network offload cards. Smart clients instead of web pages could make a very large difference. [1] Very interesting to see the idea worked out in more detail. [1] https://twitter.com/i…

Isn't that what an OPA sorta kinda does.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#125

Earlier quoted context omitted.

That doesn't really seem like an example, since the whole thing doesn't run one machine. The database alone has multiple machines.

> That doesn't really seem like an example, since the whole thing doesn't run one machine. It is an example. It shows you how you can run a service that issues a few hundred million SSL certs a year off relatively few pieces of hardware, i.e. no need to go drinking the cloud Kool aid. There will never be a "perfect" example. The overall point here is demonstrating that the first answer to everything doesn't have to i…

> there is only one writer

And what happens if that writer goes down. Then the service just stops.

> buzzword-filled

I love how your buzzwords e.g. read replicas are okay but everyone else's are bad.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#126
post #114

Earlier quoted context omitted.

> That doesn't really seem like an example, since the whole thing doesn't run one machine. It is an example. It shows you how you can run a service that issues a few hundred million SSL certs a year off relatively few pieces of hardware, i.e. no need to go drinking the cloud Kool aid. There will never be a "perfect" example. The overall point here is demonstrating that the first answer to everything doesn't have to i…

Last I read, Hacker News was still running on one big machine. And still uses text files as its database.

Twitter runs ads and generates billions in revenue.

It can't just tolerate being down or having under-load issues like HN often is.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#127

Good analysis. Obviously, this doesn't handle cases like redundancy and doesn't handle some of other critical workloads the company has. However, it does show how much real compute bloat these companies actually have - https://twitter.com/petrillic/status/1593686223717269504 where they use 24 million vcpus and spend 300 million a month on cloud.

On the other hand, Twitter does (or did) handle over 450 million monthly active users (based on stats websites), with a target for 315 monetizable daily active users (based on their earnings calls pre-privatization). Handling that amount of concurrency and beaming millions of tweets a day to home feeds and notifications is going to be logistically hard.

Posted this on a comment above but systems like Whatsapp likely sent an insane amount of data as well but used only 16 servers over 1.5 billion users at time of acquisition. Modern NICs can handle millions of requests a second - I still feel there is a lot of excess here.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#128

Earlier quoted context omitted.

> You should still be containerizing your applications to ensure a consistent environment, even if you only throw it in a docker-compose file on your production server. I'll say that this is a good point, especially because if you don't use containers or a similar solution (even things like shipping VM images, for all I care), you'll end up with environment drift, unless your application is a statically compiled exec…

When it comes to Java, everything could be used as a directory installation. Like you need JDK, maven and tomcat? Download and extract it somewhere. Modify your current PATH to include java and that's about it. You can build big tar.gz instead of OCI container which will work just fine. So IMO it's perfectly possible to run Java applications without containers. You would need to think about network ports, about resou…

> When it comes to Java, everything could be used as a directory installation. Like you need JDK, maven and tomcat? Download and extract it somewhere. Modify your current PATH to include java and that's about it. You can build big tar.gz instead of OCI container which will work just fine.

I've seen something similar in projects previously, it never worked all that well.

While the idea of shipping one archive with everything is pretty good, people don't want to include the full JDK and Tomcat installs with each software delivery, unlike with containers, where you get some benefit out of layer re-use when they haven't changed, while having the confidence that what you tested is what you'll ship. Shipping 100 app versions with the same JDK + Tomcat version will mean reused layers instead of 100 copies in the archives. And if you don't ship everything together, but merely suggest that release X should run on JDK version Y, the possibility of someone not following those instructions at least once approaches 100% with every next release.

Furthermore, Tomcat typically will need custom configuration for the app server, as well as configuration for the actual apps. This means that you'd need to store the configuration in a bunch of separate files and then apply (copy) it on top of the newly delivered version. But you can't really do that directly, so you'd need to use something like Meld to compare whether the newly shipped default configuration doesn't include something that your old custom configuration doesn't (e.g. something new in web.xml or server.xml). The same applies to something like cacerts within your JDK install, if you haven't bothered to set up custom files separately.

Worse yet, if people aren't really disciplined about all of this, you'll end up with configuration drift over time - where your dev environment will have configuration A, your test environment will have configuration B (which will sort of be like A), and staging or prod will have something else. You'll be able to ignore some of those differences until everything will go horribly wrong one day, or maybe you'll get degraded performance but without a clear reason for it.

> So IMO it's perfectly possible to run Java applications without containers. You would need to think about network ports, about resource limits, but those are not hard things.

This is only viable/easy/not brittle when you have self-contained .jar files, which admittedly are pretty nice! Though if shipping JDK with each delivery isn't in the cards (for example, because of the space considerations), that's not safe either - I've seen performance degrade 10x because of a JDK patch release was different between two environments, all because of JDK being managed through the system packages.

Resource limits are generally doable, though Xms and Xmx lie to you, you'd need systemd slices or an equivalent for hard resource limits, which I haven't seen anyone seriously bother with, although they're at a risk of the entire server/VM becoming unresponsive should their process go rogue for whatever reason (e.g. CPU at 100%, which is arguably worse than OOM because of bad memory limits).

Ports are okay when you are actually in control of the software and nothing is hardcoded. Then again, another aspect is being able to run multiple versions of software at the same time (e.g. different MySQL/MariaDB releases for different services/projects on the same node), which most nix distributions are pretty bad at.

> And tomcat even provides zero-downtime upgrades, although it's not that easy to set up, but when it works, it does work.

I've seen this attempted, but it never worked properly - the codebases might not have been good, but those redeployments and integrating with Tomcat always lead to either memory leaks or odd cases of the app server breaking. That's why personally I actually enjoy the approach of killing the entire thing alongside the app and doing a restart (especially good with embedded Tomcat/Jetty/Undertow), using health checks for routing traffic instead.

I think doing these things at the app server level is generally just asking for headaches, though the idea of being able to do so is nice. Then again, I don't see servers like Payara (like GlassFish) in use anymore, so I guess Spring Boot with embedded Tomcat largely won, in combination with other tools.

> After I've got some experience with Kubernetes, I'd reach for it always because it's very simple and easy to use. But that requires to go through some learning curve, for sure.

I wouldn't claim that Kubernetes is simple if you need to run your own clusters, though projects like K3s, K0s and MicroK8s are admittedly pretty close.

> The best and unbeatable thing about containers is that there're plenty of ready ones. I have no idea how would I install postgres without apt. I guess I could download binaries (where?), put them somewhere, read docs, craft config file with data dir pointing to anotherwere and so on. That should be doable but that's time. I can docker run it in seconds and that's saved time. Another example is ingress-nginx + cert-manager. It would take hours if not days from me to craft set of scripts and configs to replicate thing which is available almost out of the box in k8s, well tested and just works.

This is definitely a benefit!

Though for my personal needs, I build most (funnily enough, excluding databases, but that's mostly because I'm lazy) of my own containers from a common Ubuntu base. Because of layer reuse, I don't even need tricks like copying files directly, but can use the OS package manager (though clean up package cache afterwards) and pretty approachable configuration methods: https://blog.kronis.dev/articles/using-ubuntu-as-the-base-fo...

In addition, my ingress is just a containerized instance of Apache running on my nodes, with Docker Swarm instead of Kubernetes: https://blog.kronis.dev/tutorials/how-and-why-to-use-apache-... In my case, the distinction between the web server running inside of a container and outside of a container is minimal, with the exception that Docker takes care of service discovery for me, which is delightfully simple.

I won't say that the ingress abstraction in Kubernetes isn't nice, though you can occasionally run into configurations which aren't as easy as they should be: e.g. configuring Apache/Nginx/Caddy/Traefik certs which has numerous tutorials and examples online vs trying to feed your wildcard TLS cert into a Traefik ingress, with all of the configuration so that your K3s cluster would use it as the default certificate for the apps you want to expose. Not that other ingresses aren't great (e.g. Nginx), it's just that you're buying into additional complexity and I've personally have also had cases where removing and re-adding it hangs because of some resource cleanup in Kubernetes failing to complete.

I guess what I'm saying is that it's nice to use containers for whatever the strong parts are (for example, the bit about being able to run things easily), though ideally without ending up with an abstraction that might eventually become leaky (e.g. using lots of Helm charts that have lots of complexity hiding under the hood). Just this week I had CI deploys starting to randomly fail because some of the cluster's certificates had expired and kubectl connections wouldn't work. A restart of the cluster systemd services helped make everything rotate, but that's another thing to think about, which otherwise wouldn't be a concern.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#129

Earlier quoted context omitted.

> However, it does show how much real compute bloat these companies actually have No, it doesn’t. It’s a fun exercise in approaching Twitter as an academic exercise. It ignores all of the real-world functionality that makes it a business rather than a toy. A lot of complicated businesses are easy to prototype out if you discard all requirements other than the core feature. In the real world, more engineering work oft…

Genuinely asking, why do you think Twitter needs 24 million vcpus to run? This is not apples to apples but Whatsapp is a product that entirely ran on 16 servers at the time of acquisition (1.5 billion users). It really begs the question why Twitter uses so much compute if there are companies that have operated significantly more efficiently. Twitter was unprofitable during acquisition and spent around half their reve…

Whatsapp doesn't do ranking.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#130

Most projects I encounter these days instantly reach for kubernetes, containers and microservices or cloud functions. I find it much more appealing to just make the whole thing run on one fast machine. When you suggest this tend to people say "but scaling!", without understanding how much capacity there is in vertical. The thing most appealing about single server configs is the simplicity. The more simple a system ea…

Kubernetes just orchestrates containers. You can still run beefy machines and scale (if necessary) accordingly. If anything, Kubernetes allows you to save cost by going with a scalable number of small, inexpensive, fully utilized machines, vs one large, expensive, underused one.

I would wager that the majority of users of k8s do so on a cloud where they could provision VMs of the proper size to begin with. The utilization argument is specious.
Post reply on HN