Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Production Twitter on one machine? 100Gbps NICs and NVMe are fast

thume.ca

Post reply on HN
Print view View original

381–390 of 500 posts

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#381

Post by dmitriid »

nkellenicki wrote: post #349
p-e-w wrote: post #345

Earlier quoted context omitted.

If Twitter's data can fit on one machine, then the data of 99.99% of companies can. Not every product needs a billion users with Gigabytes of storage each. The assumption that if your startup's tech isn't scalable enough to become the next Google then it's the wrong tech is hilarious nonsense driven largely by ego fantasies.

It does not fit on one machine. Tweets alone generate petabytes of data a year, and other events are petabytes per day. https://blog.twitter.com/engineering/en_us/topics/infrastruc... https://ankush-chavan.medium.com/twitter-data-storage-and-pr...

> Tweets alone generate petabytes of data a year

Nope. It's not Tweets that generate that data. It's the insane amount of (mostly unnecessary) noise that gets thrown into the mix: analytics, logs, metrics, you name it.

Every time you scroll Twitter sends multiple events to the server. That alone will generate a large chunk of those petabytes.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#382

Post by brazzledazzle »

saagarjha wrote: post #369
Joeri wrote: post #358

Earlier quoted context omitted.

The thing is that hardware scales faster than humanity. When the internet boom happened there was no choice except to scale horizontally to reach a global audience, but as this article points out that assumption might no longer hold true for many services. It might make sense to return to vertically scaled highly reliable servers to achieve software simplicity and a lower overall cost. I’m always reminded of how stac…

Stack Overflow happens to go down fair maintenance a fair bit. Now, it’s not really a service that might need reliability like Twitter does*, but it’s important to keep in mind. *Some engineers may disagree

Also worth pointing out that Stack Overflow's Microsoft-centric architecture may also incentivize them to maximize vertical scaling to save on the licensing cost overhead horizontal scaling would incur for at least part of their stack.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#383

Post by vidarh »

jameshart wrote: post #311

Getting everything onto one machine works great until... it no longer fits on one machine. You add another feature and it requires a little bit more RAM, and another feature that needs a little bit more, and.. eventually it doesn't all fit. Now you have to go distributed. And your entire system architecture and all your development approaches are built around the assumptions of locality and cache line optimization an…

Many of us will remember that Twitter in fact did start out with a monolithic database and had to rewrite a bunch of stuff when they couldn't make that work anymore.

Of course they could fit a much larger dataset on one machine today.

(But I will note the article is also assuming a chronological timeline by default, but that of course hasn't been true for years - the ranking Twitter does now is far more complex)

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#384

Post by brazzledazzle »

threeseed wrote: post #283
to11mtm wrote: post #259

Earlier quoted context omitted.

Maybe it should be, though? I sometimes wonder how much value ML provides vs a proper sort function for anything but advertising.

It's an easy thought experiment. Do you think that the most successful web companies in the world with arguably the best people i.e. Amazon, Facebook, Instagram, TikTok, LinkedIn, Pinterest, Youtube, Netflix, Snapchat etc. have no idea what they are doing. That the highly complex, expensive and latency impacting recommendation systems could be replaced by trivial sorting. Or maybe they do work, do translate to increa…

A lot of people don't realize that Dunning-Kruger can catch any of us unaware. It's easy to look at a problem, think about the surface level challenges you'd have building it and come to the conclusion that you could do it better or simpler.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#385

Post by sgt »

Nextgrid wrote: post #344
thriftwy wrote: post #138

I remember Stack Overflow running on a single Windows Server box and mocking fellow LAMP developers with their propensity towards having dozens of VMs to same effect. That was some time ago, though.

I believe that's still the case: https://stackexchange.com/performance 9 web servers to serve the entire network. I wish more developers were aware of just how performant modern (non-cloud) hardware is.

SO only has two proxy servers, in which one is a failover? I bet a lot of people wouldn't believe that if you casually mentioned it.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#386

Post by threeseed »

Dylan16807 wrote: post #351
threeseed wrote: post #343

Earlier quoted context omitted.

It is trivial compared to the task of translating a user query into search results. It's akin to saying the magic behind OpenGPT is the dataset.

Keyword search is not that hard. Pagerank is not that hard. The comparable goal to the article is to be a search engine, not to fight google for best results.

> Keyword search is not that hard. Pagerank is not that hard.

George Hotz, is that you ?

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#387

Post by vidarh »

to11mtm wrote: post #259
seritools wrote: post #176

Earlier quoted context omitted.

> if it really was [which it isn't]

Maybe it should be, though? I sometimes wonder how much value ML provides vs a proper sort function for anything but advertising.

Try changing your ordering on Twitter to chronological and see how much you miss.

It's not just ads, it means the set of people you follow becomes extremely critical for your experience in a way that makes it far less engaging.

That may be good or bad for you as a user depending on what you want, but for Twitter having most people stick to the ML augmented timeline is essential to keep you hooked.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#388

Post by IshKebab »

kijin wrote: post #361
icedchai wrote: post #312

Earlier quoted context omitted.

True. You'd also save a ton of less operations and engineering staff. Running anything on a single server, however, is really a non starter for anything remotely serious. What do you do if you need to do an OS update? I suppose you could just never do those, like a former employer (1000+ day uptimes...)

Compare the cost of operating multiple servers, on one hand, with the lost revenue from having weekly or monthly maintenance windows during which you just put up a Fail Whale page. Most people overestimate the latter by a huge margin.

That's fine if your service is really local - you can do it at night. Not really an option for a global site. Imagine if Twitter went down for a few hours every month. People are addicted to Twitter. It might be at a critical time for an entire country (e.g. the Queen dies). Even worse you can't guarantee how long the upgrade will take.

You'd definitely need at least two servers. But I think you could surely just have simple master/slave replication and switch between them.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#389

Post by vidarh »

NavinF wrote: post #355
sayrer wrote: post #137

Earlier quoted context omitted.

"web previews: I'd do this by making it the client's responsibility." Actually a good example of how difficult the problem is. A very common attack is to switch a bit.ly link or something like that to a malicious destination. You would also DoS the hosts... as the Mastodon folks are discovering ( https://www.jwz.org/blog/2022/11/mastodon-stampede/ ) For blocks/mutes, you have to account for retweets and quotes, it's…

I'll go ahead and quote that blog post because they block HN users using the referer header. --- "Federation" now apparently means "DDoS yourself." Every time I do a new blog post, within a second I have over a thousand simultaneous hits of that URL on my web server from unique IPs. Load goes over 100, and mariadb stops responding. The server is basically unusable for 30 to 60 seconds until the stampede of Mastodons…

While that may be funny, the number of Mastodon instances is growing rapidly, to the point where it will need to eventually be dealt with (not least because hosting on a Pi or having a badly optimized setup both happens in real life). But more to this example, it shows passing preview responsibility to end user clients is a far bigger problem. Eg not many would be able to handle the onslaught of being linked to from a highly viral tweet if previews weren't cached.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#390

Post by brazzledazzle »

kijin wrote: post #367
habibur wrote: post #253

Earlier quoted context omitted.

Fascinating. Any special optimization you are using, or is it from off the shelf software and with standard configuration?

Sounds totally off-the-shelf. I have a basic LAMP server running on a 4-core VM on a laptop. I just threw ApacheBench at it (not the fastest benchmarking tool, either -- it eats up 1 core all by itself), and it handles 1200 req/s TLS with no keepalive, and 3400 req/s with keepalive. This stuff scales linearly with core count, so I wouldn't be surprised to see much higher numbers in real servers.

Is AB running on the same machine during the tests?
Post reply on HN