Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Production Twitter on one machine? 100Gbps NICs and NVMe are fast

thume.ca

Post reply on HN
Print view View original

371–380 of 500 posts

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#371

Post by saagarjha »

pixl97 wrote: post #105
habibur wrote: post #62

He will be up for surprise. HTTP with connection: keep-open can serve 100k req/sec. But that's for one client being served repeatedly over 1 connection. And this is the inflated number that's published in webserver benchmark tests. For more practical down to earth test, you need to measure performance w/o keep-alive. Request per second will drop to 12k / sec then. And that's for HTTP without encryption or ssl handsha…

And I would say real life Twitter involves mostly cell phone use where we see companies like Google try to push HTTP/3 to deal with head of line issues on lossy connections. Serving at the millions of hits per day on lossy networks is going to leave you with massive numbers of connections that have been abandoned but you don't know it yet. Or connections that are behaving like they are tar pitted and running at bits…

I don’t believe Twitter ever got around to rolling out HTTP/3 to their clients.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#372

Post by saagarjha »

judge2020 wrote: post #73
WJW wrote: post #53

Earlier quoted context omitted.

Is that 315 million monetizable DAUs? That sounds like a lot if the total is only 450 MAU. OTOH, 315k DAU seems like it wouldn't be enough to pay the bills.

There were some quarters with profit, some without; the past few years were mostly without IIRC. They were targeting 315 mDAUs for Q4 2023, but in the final earnings it was only 238 mDAUs. Actual MAU stats weren't public iirc but some random stats sites seemed to say 450m global MAUs, which likely includes people with no ad preferences or who only view NSFW content (which can't be shown next to (most?) ads). https://…

This includes users who aren’t shown ads, like people who are logged out or use third party clients.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#373

Post by saagarjha »

therealdrag0 wrote: post #224
judge2020 wrote: post #33

Earlier quoted context omitted.

On the other hand, Twitter does (or did) handle over 450 million monthly active users (based on stats websites), with a target for 315 monetizable daily active users (based on their earnings calls pre-privatization). Handling that amount of concurrency and beaming millions of tweets a day to home feeds and notifications is going to be logistically hard.

Anyone know if MAU includes API clients? Cuz there’s a lot of apps built on top of twitter that pull data right? Thats add a lot of traffic too.

Active users includes those, monetizable active users doesn’t.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#374

Post by saagarjha »

mgaunard wrote: post #203

All web and cloud technologies are inherently inefficient, and most programmers don't know networking or even how hardware works sufficiently well to optimize for high througput and low-latency. There was an article just yesterday about how Jane Street had developed an internal exchange way faster than any actual exchange by building it from the ground up, thinking about how the hardware works and how agents can inte…

> It wouldn't be surprising at all that you could build something equivalent to Twitter on just one beefy machine, maybe two for redundancy.

The blog post kind of gets a very cut-down version of Twitter running on a single machine. Actual Twitter absolutely would not work.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#376

Post by renonce »

untech wrote: post #286
moeny wrote: post #247

Earlier quoted context omitted.

's/(human|desktop)/smartphone/g'

Using two 32 bit numbers for coordinates, each record would take 8 bytes, which is 64 gigabytes for 8 billion population. Don’t think many smartphones have this RAM today.

You can build that phone. Phones made an engineering decision to put more battery rather than more RAM, and it's just a matter of putting more LPDDR5 chips onto the circuit board.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#377

Post by spullara »

steviesands wrote: post #356
spullara wrote: post #331

When I was working there I implemented my patent during a hack week (given a set of follows return the list of matching tweet ids, very similar to his prototype): https://patents.google.com/patent/US20120136905A1/en (licensed under Innovators Patent Agreement, https://github.com/twitter/innovators-patent-agreement ) I could have definitely served all the chronological timeline requests on a normal server with lower l…

A few thoughts. The first is, are we asking the wrong questions? Should it be, "If I spend 10m on hardware for predicting ads (storage/compute) that generates 25m in revenue, should I buy the hardware?". Sure, we can "minify" twitter, and it's a wonderful thought experiment, but it seems devoid of the context of revenue generation. The second is, it's interesting to understand social media industry wide infra cost pe…

The basic idea of the system was to scan a reverse chronologically ordered list of "user id, tweet id", filtering out any tweet whose user wasn't in the follow set (or sets in the case of scan sharing) until you retrieved enough tweets for the timeline request. There are a bunch of variants in the patent, but that is the basic idea. At the time, I estimated that Twitter was spending 80% of its CPU time in the DC doing thrift/json/html serialization/deserialization and mused about merging all the separate services into a single process. Lot's of opportunity for optimization.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#378

Post by tpetry »

kijin wrote: post #367
habibur wrote: post #253

Earlier quoted context omitted.

Fascinating. Any special optimization you are using, or is it from off the shelf software and with standard configuration?

Sounds totally off-the-shelf. I have a basic LAMP server running on a 4-core VM on a laptop. I just threw ApacheBench at it (not the fastest benchmarking tool, either -- it eats up 1 core all by itself), and it handles 1200 req/s TLS with no keepalive, and 3400 req/s with keepalive. This stuff scales linearly with core count, so I wouldn't be surprised to see much higher numbers in real servers.

Are these all new TLS connections? Because most benchmarks use TLS resumption which means the TLS handshake was only done once!

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#379

Post by anticristi »

wtallis wrote: post #255
vidarh wrote: post #237

Earlier quoted context omitted.

Biggest problem with this is the lack of considering analytics.twitter.com and ads.twitter.com. Twitter stores event data about everything that happens to a tweet, and lets you target ads with a lot of precision. While some of those writes may well be acceptable to lose, letting you write to caches, effectively you need to assume there are more analytics events triggering writes to something than there are tweet view…

A Twitter-like service that fits on a single server could probably get by with the reduced revenue that comes with not offering obsessively fine-grained analytics and ad targeting.

Not to mention that your business would end up being more profitable by avoiding GDPR fines. Most users in the EU would click "Reject all" anyway, so basically that code is not needed.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#380

Post by [deleted] »

jameshart wrote: post #311

Getting everything onto one machine works great until... it no longer fits on one machine. You add another feature and it requires a little bit more RAM, and another feature that needs a little bit more, and.. eventually it doesn't all fit. Now you have to go distributed. And your entire system architecture and all your development approaches are built around the assumptions of locality and cache line optimization an…

[deleted]
Post reply on HN