The title reminded me about this https://www.phoronix.com/news/Netflix-NUMA-FreeBSD-Optimized (2019) and this 2 years later: https://papers.freebsd.org/2021/eurobsdcon/gallatin-netflix-... (2021)
Production Twitter on one machine? 100Gbps NICs and NVMe are fast
11–20 of 500 posts
Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast
#12I like this kind of exercise. One thing I am not seeing is analytics, logs and so forth that as I understand it are significant portions of Twitter's production cost story.
Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast
#13Good analysis. Obviously, this doesn't handle cases like redundancy and doesn't handle some of other critical workloads the company has. However, it does show how much real compute bloat these companies actually have - https://twitter.com/petrillic/status/1593686223717269504 where they use 24 million vcpus and spend 300 million a month on cloud.
Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast
#14The title reminded me about this https://www.phoronix.com/news/Netflix-NUMA-FreeBSD-Optimized (2019) and this 2 years later: https://papers.freebsd.org/2021/eurobsdcon/gallatin-netflix-... (2021)
Latest version: http://nabstreamingsummit.com/wp-content/uploads/2022/05/202... (2022)
Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast
#15> colo cost + total server cost/(3 year) => $18,471/year
Meanwhile the company I just left was spending more than this for dozens of kubernetes clusters on AWS before signing a single customer. Sometimes I wonder what I'm still doing in this industry.
Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast
#16Earlier quoted context omitted.
What strikes you as English-only about it?
My bad - not English, but ASCII. The assumed max tweet size is in bytes rather than (UTF) characters.
I specifically assumed a max tweet size based on the maximum number of UTF-8 bytes a tweet can contain (560), with a link to an analysis of that, and discussion of how you could optimize for the common case of tweets that contain way fewer UTF-8 bytes than that. Everything in my post assumes unicode.
Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast
#17Earlier quoted context omitted.
What strikes you as English-only about it?
My bad - not English, but ASCII. The assumed max tweet size is in bytes rather than (UTF) characters.
That size in bytes is based on the max size in UTF-8 and UTF-16. Codepoints below U+1100 are counted as one "character" by twitter and will need at most 2 bytes. Codepoints above it are counted as two "characters" by twitter and will need at most 4 bytes. Therefore 560 bytes, and it supports all languages.
Side note, this is more pessimistic than it needs to be, if you're willing to transcode. The larger codepoints fit into 20-21 bits, and the smaller ones fit into 12-13 bits.
Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast
#18> colo cost + total server cost/(3 year) => $18,471/year Meanwhile the company I just left was spending more than this for dozens of kubernetes clusters on AWS before signing a single customer. Sometimes I wonder what I'm still doing in this industry.
Very often. I work with companies spending 10x as much on bad (bizarrely complex; indeed kubernetes, lambda, gateway, rds etc) setup and bad code on aws. Almost no traffic (b2b). Makes no sense at all.
Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast
#19If you really wanted to run Twitter on one machine at any cost, wouldn't an IBM mainframe be much more practical?
You can even run Linux on them now. The specs he cites would actually be fairly small for a mainframe, which can reach up to 40TB of memory.
I'm not saying this is a good idea, but it seems better than what the OP proposes.
Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast
#20> colo cost + total server cost/(3 year) => $18,471/year Meanwhile the company I just left was spending more than this for dozens of kubernetes clusters on AWS before signing a single customer. Sometimes I wonder what I'm still doing in this industry.
Techies in tech industry are basically eating the rich .. A lot of buzzword to suck investment money in.