Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Production Twitter on one machine? 100Gbps NICs and NVMe are fast

thume.ca

Post reply on HN
Print view View original

91–100 of 500 posts

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#91

Post by fbdab103 »

jupp0r wrote: post #84
fbdab103 wrote: post #59

Earlier quoted context omitted.

I think you should always plan for failures, but modern enterprise hardware is quite reliable. I would even posit that if you stood up a brand new physical server today, it has a good chance of beating AWS uptime (well, not the AWS dashboard numbers) over a one year period.

"hardware is quite reliable" is not a valid strategy. Hardware fails with some non-zero probability. You need to have a plan in place what to do if that happens, taking into account service disruption, backups etc. Having a system in place that handles most of this gracefully (like kubernetes) is one way of having such a plan, there are others. Which one works best is dependent on your app, cost of downtime, your tea…

My first sentence was to always plan for failures.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#92

Post by pixl97 »

fbdab103 wrote: post #59
yodsanklai wrote: post #44

Earlier quoted context omitted.

> The thing most appealing about single server configs is the simplicity. The more simple a system easy, likely the more reliable and easy to understand. What if your unique machine crash?

I think you should always plan for failures, but modern enterprise hardware is quite reliable. I would even posit that if you stood up a brand new physical server today, it has a good chance of beating AWS uptime (well, not the AWS dashboard numbers) over a one year period.

It doesn't matter if hardware is 99.99% reliable if you're the .01% that day

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#93

Post by sayrer »

trishume wrote: post #21
jasonhansel wrote: post #19

If you really wanted to run Twitter on one machine at any cost, wouldn't an IBM mainframe be much more practical? You can even run Linux on them now. The specs he cites would actually be fairly small for a mainframe, which can reach up to 40TB of memory. I'm not saying this is a good idea, but it seems better than what the OP proposes.

My friend mentioned this just before I published and I think that probably is the fastest largest thing you can get which would in some sense count as one machine. I haven't looked into it, but I wouldn't be surprised if they could get around the trickiest constraint, which is how many hard drives you can plug in to a non-mainframe machine for historical image storage. Definitely more expensive than just networking a…

It's a neat thought exercise, but wrong for so many reasons (there are probably like 100s). Some jump out: spam/abuse detection, ad relevance, open graph web previews, promoted tweets that don't appear in author timelines, blocks/mutes, etc. This program is what people think Twitter is, but there's a lot more to it.

I think every big internet service uses user-space networking where required, so that part isn't new.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#94

Post by jupp0r »

eddsh1994 wrote: post #68
jupp0r wrote: post #64

Earlier quoted context omitted.

In addition, you should worry about what happens to your app if a hardware error, network problem or natural disaster makes your machine unavailable.

Split the DB from the app and replicate with a load balancer?

Now you have "Production Twitter on three machines". Follow the rabbit and you end up with something like what Twitter looks like today.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#95

Post by cortesoft »

drewg123 wrote: post #42

How much bandwidth does Twitter use for images and videos? Less than 1.4Tb/s globally? If so, we could probably fit that onto a second machine. We can currently serve over 700Gb/s from a dual-socket Milan based server[1]. I'm still waiting for hardware, but assuming there are no new bottlenecks, that should directly scale up to 1.4Tb/s with Genoa and ConnectX-7, given the IO pathways are all at least twice the bandwi…

It is way more than 1.4TBs a second globally.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#96

Post by vbezhenar »

KronisLV wrote: post #57
judge2020 wrote: post #25

Earlier quoted context omitted.

Kubernetes and containers are a means to service architecture; It enabled scalability but does not require it. You should still be containerizing your applications to ensure a consistent environment, even if you only throw it in a docker-compose file on your production server.

> You should still be containerizing your applications to ensure a consistent environment, even if you only throw it in a docker-compose file on your production server. I'll say that this is a good point, especially because if you don't use containers or a similar solution (even things like shipping VM images, for all I care), you'll end up with environment drift, unless your application is a statically compiled exec…

When it comes to Java, everything could be used as a directory installation. Like you need JDK, maven and tomcat? Download and extract it somewhere. Modify your current PATH to include java and that's about it. You can build big tar.gz instead of OCI container which will work just fine.

So IMO it's perfectly possible to run Java applications without containers. You would need to think about network ports, about resource limits, but those are not hard things.

And tomcat even provides zero-downtime upgrades, although it's not that easy to set up, but when it works, it does work.

After I've got some experience with Kubernetes, I'd reach for it always because it's very simple and easy to use. But that requires to go through some learning curve, for sure.

The best and unbeatable thing about containers is that there're plenty of ready ones. I have no idea how would I install postgres without apt. I guess I could download binaries (where?), put them somewhere, read docs, craft config file with data dir pointing to anotherwere and so on. That should be doable but that's time. I can docker run it in seconds and that's saved time. Another example is ingress-nginx + cert-manager. It would take hours if not days from me to craft set of scripts and configs to replicate thing which is available almost out of the box in k8s, well tested and just works.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#97

Post by kbumsik »

andrewstuart wrote: post #24

Most projects I encounter these days instantly reach for kubernetes, containers and microservices or cloud functions. I find it much more appealing to just make the whole thing run on one fast machine. When you suggest this tend to people say "but scaling!", without understanding how much capacity there is in vertical. The thing most appealing about single server configs is the simplicity. The more simple a system ea…

> The thing most appealing about single server configs is the simplicity.

In my experience this ended up with more complicated.

Those systems are typically developed by people who already left and are undocumented, and they become extremely difficult to figure out the config (packages, etc files... oh, where even the service files are located?) and almost impossible to reproduce.

It might be okay to leave it there, but when we need to modify or troubleshoot the system a nightmare begins...

Maybe I was just unlucky, but at least k8s configs are more organized and simpler than dealing with a whole custom configured Linux system.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#98

Post by jupp0r »

fbdab103 wrote: post #91
jupp0r wrote: post #84

Earlier quoted context omitted.

"hardware is quite reliable" is not a valid strategy. Hardware fails with some non-zero probability. You need to have a plan in place what to do if that happens, taking into account service disruption, backups etc. Having a system in place that handles most of this gracefully (like kubernetes) is one way of having such a plan, there are others. Which one works best is dependent on your app, cost of downtime, your tea…

My first sentence was to always plan for failures.

Yeah sorry it wasn't meant to dispute your argument, just as an addition.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#99

Post by kureikain »

Not to the extreme of fitting everything into one machine but I have explorer the idea of separate stateless workload into its own machine.

However, the stateless workload can still operate in a read-only manner if the stateful component failed.

I run an email forwarding service[1], and one of challenge is how can I ensure the email forwarding still work even if my primary database failed.

And I come up with a design that the app boot up, and load entire routing data from my postgres into its memory data structure, and persisted to local storage. So if postgres datbase failed, as long as I have an instance of those app(which I can run as many as I can), the system continue to work for existing customer.

The app use listen/notify to load new data from postgres into its memory.

Not exactly the same concept as the artcile, but the idea is that we try to design the system in a way where it can operate fully on a single machine. Another cool thing is that it easiser to test this, instead of loading data from Postgres, it can load from config files, so essentially the core biz logic is isolated into a single machine.

---

https://mailwip.com

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#100

Post by [deleted] »

andrewstuart wrote: post #24

Most projects I encounter these days instantly reach for kubernetes, containers and microservices or cloud functions. I find it much more appealing to just make the whole thing run on one fast machine. When you suggest this tend to people say "but scaling!", without understanding how much capacity there is in vertical. The thing most appealing about single server configs is the simplicity. The more simple a system ea…

[deleted]
Post reply on HN