Production Twitter on one machine? 100Gbps NICs and NVMe are fast

Post by toast0 » Sat, Jan 07, 2023, 10:13 PM UTC

ffssffss wrote: post #55

andrewstuart wrote: post #49

Earlier quoted context omitted.

Well you gotta have a backup strategy. I'm talking about the primary machine here, I assumed that would be obvious but maybe not. You build your failover strategy into your architecture - there's lots of ways to do it - I use Postgres so I would favor something based around log shipping.

And uptime is important, so you want to have that secondary running and ready, with a proxy in front of everything so you can switch as soon as you detect a failure. That's three hosts, plus your alerting has to be separate too, so that's four. Now, to orchestrate all this, we'll first get out Puppet...

Post by ignoramous » Sat, Jan 07, 2023, 10:15 PM UTC

varunkmohan wrote: post #119

PragmaticPulp wrote: post #65

Earlier quoted context omitted.

> However, it does show how much real compute bloat these companies actually have No, it doesn’t. It’s a fun exercise in approaching Twitter as an academic exercise. It ignores all of the real-world functionality that makes it a business rather than a toy. A lot of complicated businesses are easy to prototype out if you discard all requirements other than the core feature. In the real world, more engineering work oft…

Genuinely asking, why do you think Twitter needs 24 million vcpus to run? This is not apples to apples but Whatsapp is a product that entirely ran on 16 servers at the time of acquisition (1.5 billion users). It really begs the question why Twitter uses so much compute if there are companies that have operated significantly more efficiently. Twitter was unprofitable during acquisition and spent around half their reve…

Post by xwdv » Sat, Jan 07, 2023, 10:17 PM UTC

varunkmohan wrote: post #119

PragmaticPulp wrote: post #65

Earlier quoted context omitted.

> However, it does show how much real compute bloat these companies actually have No, it doesn’t. It’s a fun exercise in approaching Twitter as an academic exercise. It ignores all of the real-world functionality that makes it a business rather than a toy. A lot of complicated businesses are easy to prototype out if you discard all requirements other than the core feature. In the real world, more engineering work oft…

Genuinely asking, why do you think Twitter needs 24 million vcpus to run? This is not apples to apples but Whatsapp is a product that entirely ran on 16 servers at the time of acquisition (1.5 billion users). It really begs the question why Twitter uses so much compute if there are companies that have operated significantly more efficiently. Twitter was unprofitable during acquisition and spent around half their reve…

Post by sayrer » Sat, Jan 07, 2023, 10:19 PM UTC

trishume wrote: post #121

sayrer wrote: post #93

Earlier quoted context omitted.

It's a neat thought exercise, but wrong for so many reasons (there are probably like 100s). Some jump out: spam/abuse detection, ad relevance, open graph web previews, promoted tweets that don't appear in author timelines, blocks/mutes, etc. This program is what people think Twitter is, but there's a lot more to it. I think every big internet service uses user-space networking where required, so that part isn't new.

I think I'm pretty careful to say that this is a simplified version of Twitter. Of the features you list: - spam detection: I agree this is a reasonably core feature and a good point. I think you could fit something here but you'd have to architect your entire spam detection approach around being able to fit, which is a pretty tricky constraint and probably would make it perform worse than a less constrained solution…

Post by toast0 » Sat, Jan 07, 2023, 10:21 PM UTC

richwater wrote: post #108

traceroute66 wrote: post #27

Earlier quoted context omitted.

> spending more than this for dozens of kubernetes clusters on AWS before signing a single customer Yup. Cloud is 21st century Nickel & Diming. Sure it sounds cheap, everything is priced in small sounding cents per unit. But then it very quickly becomes a compounding vicious circle ... a dozen different cloud tools, each charged at cents per unit, those units often being measured in increments of hours....next thing…

> You can get quite a lot of bang for your buck in a 1/4 rack with today's kit. Until very recently, while money was still very cheap, the time overhead it would take to manage this just was not worth the cost savings. Even with the market falling out from under VC, I think it still is a good tradeoff for many shops.