111–120 of 500 posts
In the coming years we will probably see a lot of complicated microservice architectures be replaced by well-designed and optimized Rust (and modern C++) monoliths that use simple replication to scale horizontally.
I like this kind of exercise. One thing I am not seeing is analytics, logs and so forth that as I understand it are significant portions of Twitter's production cost story.
Anyone have a complete list of functional blocks that form Twitter? Beyond the obvious and what we see?
Over the last couple of months I've seen comments that summarise Twitter as a read-only service that doesn't have any real time posting requirements and similarly other comments that treat it as a write-only service with no real time read / fast search requirements.
Without _all_ the blocks even the simple surface level Twitter will have complexity people miss.
> colo cost + total server cost/(3 year) => $18,471/year Meanwhile the company I just left was spending more than this for dozens of kubernetes clusters on AWS before signing a single customer. Sometimes I wonder what I'm still doing in this industry.
Techies in tech industry are basically eating the rich .. A lot of buzzword to suck investment money in.
Earlier quoted context omitted.
That doesn't really seem like an example, since the whole thing doesn't run one machine. The database alone has multiple machines.
> That doesn't really seem like an example, since the whole thing doesn't run one machine. It is an example. It shows you how you can run a service that issues a few hundred million SSL certs a year off relatively few pieces of hardware, i.e. no need to go drinking the cloud Kool aid. There will never be a "perfect" example. The overall point here is demonstrating that the first answer to everything doesn't have to i…
I'm going to preface this criticism by saying that I think exercises like this are fun in an architectural/prototyping code-golf kinda way. However, I think the author critically under-guesses the sizes of things (even just for storage) by a reasonably substantial amount. e.g.: Quote tweets do not go against the size limit of the tweet field at Twitter. Likely they are embedding a tweet reference in some manner or ot…
Quote tweets I'd do as a reference and they'd basically have the cost of loading 2 tweets instead of one, so increasing the delivery rate by the fraction of tweets that are quote tweets. Hashtags are a search feature and basically need the same posting lists as for search, but if you only support hashtags the posting lists are smaller. I already have an estimate saying probably search wouldn't fit. But I think hashta…
Earlier quoted context omitted.
I go with k8s even on a single server nowadays, it just makes everything so much more convenient. https://k3s.io/ makes it really easy to set up, too.
I never tried k3s, but what's wrong with kubeadm? I think that's literally two commands to run single server k8s: kubeadm init and kubectl taint something. The only thing bad about single server kubernetes is that it'll eat like 1-2 GB of RAM by itself. When you whole server could be 256 MB, that's a lot of wasted RAM.
I think one of the under-estimated interesting points of twitter as a business is that this is the core. Yes, Twitter is 140 characters, it's got "300m users" which is probably 5m real heavy users. So yes, you could do a lot of "140 characters, a few tweets per person, few million users" on very little hardware. But that's why Twitters a shit business! How much RAM did your advertising network need? Becuase that is w…
Earlier quoted context omitted.
In addition, you should worry about what happens to your app if a hardware error, network problem or natural disaster makes your machine unavailable.
Split the DB from the app and replicate with a load balancer?
Sounds like you already have quite a number of different containers already.
Good analysis. Obviously, this doesn't handle cases like redundancy and doesn't handle some of other critical workloads the company has. However, it does show how much real compute bloat these companies actually have - https://twitter.com/petrillic/status/1593686223717269504 where they use 24 million vcpus and spend 300 million a month on cloud.
> However, it does show how much real compute bloat these companies actually have No, it doesn’t. It’s a fun exercise in approaching Twitter as an academic exercise. It ignores all of the real-world functionality that makes it a business rather than a toy. A lot of complicated businesses are easy to prototype out if you discard all requirements other than the core feature. In the real world, more engineering work oft…
This is not apples to apples but Whatsapp is a product that entirely ran on 16 servers at the time of acquisition (1.5 billion users). It really begs the question why Twitter uses so much compute if there are companies that have operated significantly more efficiently. Twitter was unprofitable during acquisition and spent around half their revenue on compute, maybe some of these features were not really necessary (but were just burning money)?
Earlier quoted context omitted.
Ok, we've put a question mark up there to make it more explorationy.
As the author, this sounds good to me! I'll probably even change the actual title to match. I originally was going to make it a question mark and the only reason I didn't is https://en.wikipedia.org/wiki/Betteridge%27s_law_of_headline... when I think the answer is probably "could probably be somewhat done" rather than "no".
Betteridge antiexamples are always welcome. I once tried to joke that Mr. Betteridge had "retired" and promptly got corrected about his employment status (https://news.ycombinator.com/item?id=10393754).