Live data from Hacker News

Production Twitter on one machine? 100Gbps NICs and NVMe are fast

thume.ca

411–420 of 500 posts

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#411
post #260

Earlier quoted context omitted.

TFA, to me, touches about something I've wondered about a very long time ago... It was a little more than ten years ago for me. I realized that a hard disk could store a database of every human alive, including basic information, links (family relations) and maybe a photo. I still don't know what the implications are. Maybe we don't want to know, but it's not really that difficult to think about.

Is the storage really the complex part? Isn't gathering the actual information and avoiding errors (ex: I have a co-worker who's name is incorrectly spelled three different ways in prod services) the hard part?

Is the storage really the complex part? Isn't gathering the actual information and avoiding errors (...) the hard part?

Not for someone that works in consulting, or at least it wasn't. I remember that I had Access access to the production database that stored all the customers, present and pass. They wouldn't give me a password to that, but apparently they thought it was safe to enter the password without me looking so I could try some queries and test my last fixes.

Not sure if it still works the same, but I did some dumb query, left the computer on and, next day morning, a temporal file was in my %TEMP% with a lot of data of millions of persons worldwide. Had I be so inclined, with an external hard disk I could have started my homegrown NSA project.

Now think of this: how many times have you heard that the data of millions of customers were on sale after a data breach? Do you doubt that, let's say, China has every single person in the West on file?

Our own governments have us legally or semi-legally (exchange) anyway.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#412

Earlier quoted context omitted.

A Twitter-like service that fits on a single server could probably get by with the reduced revenue that comes with not offering obsessively fine-grained analytics and ad targeting.

I think about this often. Specifically, how much bloat exists in the world because individuals in our society are forced to justify their existence on a daily basis. Everyone has to be employed so it's better to keep adding more crap to products and make those products disposable in order to give people a job.

If your team is full of engineers whose specialty is running clusters with 1K+ machines, you'll get solutions that need clusters with 1K+ machines.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#413
This is one of the most interesting part of the whole post for me:

Through intense digging I found a researcher who left a notebook public including tweet counts from many years of Twitter’s 10% sampled “Decahose” API and discovered the surprising fact that tweet rate today is around the same as or lower than 2013! Tweet rate peaked in 2014 and then declined before reaching new peaks in the pandemic. Elon recently tweeted the same 500M/day number which matches the Decahose notebook and 2013 blog post, so this seems to be true! Twitter’s active users grew the whole time so I think this reflects a shift from a “posting about your life to your friends” platform to an algorithmic content-consumption platform.

So, the number of writes has been the same for a good long while.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#414

I'm going to preface this criticism by saying that I think exercises like this are fun in an architectural/prototyping code-golf kinda way. However, I think the author critically under-guesses the sizes of things (even just for storage) by a reasonably substantial amount. e.g.: Quote tweets do not go against the size limit of the tweet field at Twitter. Likely they are embedding a tweet reference in some manner or ot…

Isn’t a hashtag just another kind of user account — an account from which anyone can post? The data structures for the @BeefWellington timeline of tweets and the one for the #BeefWellington timeline of tweets could look roughly the same.

[deleted]

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#415
post #383

Getting everything onto one machine works great until... it no longer fits on one machine. You add another feature and it requires a little bit more RAM, and another feature that needs a little bit more, and.. eventually it doesn't all fit. Now you have to go distributed. And your entire system architecture and all your development approaches are built around the assumptions of locality and cache line optimization an…

Many of us will remember that Twitter in fact did start out with a monolithic database and had to rewrite a bunch of stuff when they couldn't make that work anymore. Of course they could fit a much larger dataset on one machine today. (But I will note the article is also assuming a chronological timeline by default, but that of course hasn't been true for years - the ranking Twitter does now is far more complex)

There's a story by Bryan Cantrill [1] about how he went to Twitter to help them understand why it would take 400 milliseconds of compute to process a request (I'll leave the reveal to Bryan). Scaling horizontally is probably necessary for something the size of Twitter, but that doesn't mean that we can half-ass the code and just throw more machines at the problem. If we write code with a bit more mechanical sympathy and avoid the latest non-proven fads, we can surely write software that is 10x faster and requires much less scaling.

[1] https://www.youtube.com/watch?v=LjFM8vw3pbU&t=3240s

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#416

Earlier quoted context omitted.

It's sort of strange you have to make these points, but as an industry we seem to have an extremely short memory. Vertical scaling was absolutely the way most big applications were built up until well into the 90s. Companies like Oracle were really built on the fact that getting performance and reliability out of a single highly-contested massive server is hard but important if that's the way you're going. Linux beca…

It's less "short memory" than the fact that you can be a "senior software engineer" after just 5 years or so experience. There is a significant cohort of (particularly web-tech) developers who were young children in the 90s, and whose professional careers started in the 2010s and have only ever known "the cloud", big-tech and big-tech tech (k8s, etc). It's a similar phenomenon to the observation that tech "innovation…

I'm a CS undergrad, do you have any recommendations for effectively combating this?

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#418

Getting everything onto one machine works great until... it no longer fits on one machine. You add another feature and it requires a little bit more RAM, and another feature that needs a little bit more, and.. eventually it doesn't all fit. Now you have to go distributed. And your entire system architecture and all your development approaches are built around the assumptions of locality and cache line optimization an…

Weird criticism but o.k. Edit: Unless I missed something, the author never argued that Twitter should be hosted on one machine and therefore criticizing the “fun stunt” like this makes no sense to me

There are certainly people reading this and nodding and thinking ‘yeah, this makes sense! Why don't we build services like this?’ And adding it to their mental list of arguments against microservices or whatever - and I wanted to make sure people hear the reasons why this kind of performance maximization tends not to be the norm.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#419

Getting everything onto one machine works great until... it no longer fits on one machine. You add another feature and it requires a little bit more RAM, and another feature that needs a little bit more, and.. eventually it doesn't all fit. Now you have to go distributed. And your entire system architecture and all your development approaches are built around the assumptions of locality and cache line optimization an…

How about smaller projects that might be hugely successful, but will never be remotely close to a Twitter level of success? It’s interesting to see how much can be done with a single machine, because most projects will never be this big. Though there will still be other concerns like redundancy to deal with.

Can you explain me what exactly meant by "success of Twitter"?

It's not sarcasm, I have twitter account but I never understood hype about twitter.

I see nothing in twitter from technical POW, closed twitter protocol looks very strange, they banned Trump, they were profitable in 2021, Elon Musk bought them for ~44 bln.

Maybe sellout of company with problems for such price is success.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#420

Getting everything onto one machine works great until... it no longer fits on one machine. You add another feature and it requires a little bit more RAM, and another feature that needs a little bit more, and.. eventually it doesn't all fit. Now you have to go distributed. And your entire system architecture and all your development approaches are built around the assumptions of locality and cache line optimization an…

Video games have had 3D water for decades before screen-space reflections, and many look serviceable decades later (Super Mario Sunshine looks great at 480p though dated at higher resolutions).

Curses. That undermines my argument completely. It wasn’t merely a random example of how you have to compromise on features to fit in the available hardware budget, but actually the premise upon which my entire argument rested.
Post reply on HN