Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Production Twitter on one machine? 100Gbps NICs and NVMe are fast

thume.ca

Post reply on HN
Print view View original

441–450 of 500 posts

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#441

Post by ilyt »

pixl97 wrote: post #105
habibur wrote: post #62

He will be up for surprise. HTTP with connection: keep-open can serve 100k req/sec. But that's for one client being served repeatedly over 1 connection. And this is the inflated number that's published in webserver benchmark tests. For more practical down to earth test, you need to measure performance w/o keep-alive. Request per second will drop to 12k / sec then. And that's for HTTP without encryption or ssl handsha…

And I would say real life Twitter involves mostly cell phone use where we see companies like Google try to push HTTP/3 to deal with head of line issues on lossy connections. Serving at the millions of hits per day on lossy networks is going to leave you with massive numbers of connections that have been abandoned but you don't know it yet. Or connections that are behaving like they are tar pitted and running at bits…

That's so low overhead compared to everything else needed that it is near-irrelevant

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#442

Post by herval »

busymom0 wrote: post #266
mattbillenstein wrote: post #264

Earlier quoted context omitted.

It does strike me as a feature you'd typically serve out of some sort of search index since if you had to build search, you'd essentially get indexing of hashtags "for free"

You are probably right and I am wrong. I just looked at a tweet and clicking the hashtag takes to the search page with that hashtag typed in. Probably implemented similarly behind the scenes. Though hashtag most likely does an exact match search instead of fuzzy searching for regular words and phrases.

it does case matching (#hashTag === #hashtag === #HashTag) too

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#443

Post by kaba0 »

drowsspa wrote: post #220
mgaunard wrote: post #203

All web and cloud technologies are inherently inefficient, and most programmers don't know networking or even how hardware works sufficiently well to optimize for high througput and low-latency. There was an article just yesterday about how Jane Street had developed an internal exchange way faster than any actual exchange by building it from the ground up, thinking about how the hardware works and how agents can inte…

I'm always reminded of that tweet from @SwiftOnSecurity https://mobile.twitter.com/SwiftOnSecurity/status/1485822027... > Once you understand your computer has 16 cores running at 3GHz and yet doesn't boot up in .2 nanoseconds you understand everything they have taken from you. With their infinite VC money at their disposal, and with their programmers having 100 GHz machines with thousands of cores, 128 TB of RAM and…

Booting has fundamental hardware limits, though. Will that harddrive or whatever really reply all in 0.2 ns?

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#444

Post by ilyt »

jiggawatts wrote: post #156
trishume wrote: post #21

Earlier quoted context omitted.

My friend mentioned this just before I published and I think that probably is the fastest largest thing you can get which would in some sense count as one machine. I haven't looked into it, but I wouldn't be surprised if they could get around the trickiest constraint, which is how many hard drives you can plug in to a non-mainframe machine for historical image storage. Definitely more expensive than just networking a…

> which is how many hard drives you can plug in to a non-mainframe machine for historical image storage. You would be surprised . First off, SSDs are denser than hard drives now if you're willing to spend $$$. Second, "plug in" doesn't necessarily mean "in the chassis". You can expand storage with external disk arrays in all sorts of ways. Everything from external PCI-e cages to SAS disk arrays, fibre channel, NVMe-o…

> People really overestimate the data volume put out by a service like Twitter while simultaneously underestimating the bandwidth capability of a single server.

It's outright comical. Above we have people thinking somehow amount of TLS connections single server can handle is a problem, in service where there would be hundreds of thousands lines of code to generate the content served over it, all while using numbers from what seems like 10+ years old server hardware

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#445

Post by ilyt »

sayrer wrote: post #137
trishume wrote: post #121

Earlier quoted context omitted.

I think I'm pretty careful to say that this is a simplified version of Twitter. Of the features you list: - spam detection: I agree this is a reasonably core feature and a good point. I think you could fit something here but you'd have to architect your entire spam detection approach around being able to fit, which is a pretty tricky constraint and probably would make it perform worse than a less constrained solution…

"web previews: I'd do this by making it the client's responsibility." Actually a good example of how difficult the problem is. A very common attack is to switch a bit.ly link or something like that to a malicious destination. You would also DoS the hosts... as the Mastodon folks are discovering ( https://www.jwz.org/blog/2022/11/mastodon-stampede/ ) For blocks/mutes, you have to account for retweets and quotes, it's…

Well, Mastodon is criminally slow

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#446

Post by ilyt »

NavinF wrote: post #355
sayrer wrote: post #137

Earlier quoted context omitted.

"web previews: I'd do this by making it the client's responsibility." Actually a good example of how difficult the problem is. A very common attack is to switch a bit.ly link or something like that to a malicious destination. You would also DoS the hosts... as the Mastodon folks are discovering ( https://www.jwz.org/blog/2022/11/mastodon-stampede/ ) For blocks/mutes, you have to account for retweets and quotes, it's…

I'll go ahead and quote that blog post because they block HN users using the referer header. --- "Federation" now apparently means "DDoS yourself." Every time I do a new blog post, within a second I have over a thousand simultaneous hits of that URL on my web server from unique IPs. Load goes over 100, and mariadb stops responding. The server is basically unusable for 30 to 60 seconds until the stampede of Mastodons…

> I personally find this absolutely hilarious. Is that blog hosted on a Raspberry Pi or something? "Over a thousand" requests per second shouldn't even show up on the utilization graphs on a modern server.

Mastodon is written on Ruby on Rails. That should really answer all your questions about the problem but if you're unfamiliar Ruby is slow compared to any compiled language, Rails is slow compared to near-every framework on the planet and it isn't written that well either.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#447

Post by vidarh »

gnuvince wrote: post #415
vidarh wrote: post #383

Earlier quoted context omitted.

Many of us will remember that Twitter in fact did start out with a monolithic database and had to rewrite a bunch of stuff when they couldn't make that work anymore. Of course they could fit a much larger dataset on one machine today. (But I will note the article is also assuming a chronological timeline by default, but that of course hasn't been true for years - the ranking Twitter does now is far more complex)

There's a story by Bryan Cantrill [1] about how he went to Twitter to help them understand why it would take 400 milliseconds of compute to process a request (I'll leave the reveal to Bryan). Scaling horizontally is probably necessary for something the size of Twitter, but that doesn't mean that we can half-ass the code and just throw more machines at the problem. If we write code with a bit more mechanical sympathy…

Oh, absolutely. I think in terms of making people think about putting things in-process in RAM and being a bit more imaginative at looking at machines sizes as worth exploring is a good thing, and the article is interesting as a look at what you can do.

Many things can plausibly fit on a single machine irrespective of uptake (e.g. I worked on a system not long ago where even if we cornered the entire global market and the market expanded several magnitudes in size, our entire working set would still fit comfortable in memory).

But even when you need to scale, it absolutely can scale better if you're willing to not automatically resort to a standard database stack for everything for example.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#448

Post by vidarh »

Sebb767 wrote: post #431
glass3 wrote: post #399

Earlier quoted context omitted.

Twitter is not serious enough that there cannot be a timeout. With a second OS partition, the server can alternate between the working copy and a copy that is updated in a VM. For a free service, customers cannot complain even if there is a reboot every day and the service is down for a couple of minutes. Realistically, there would be a mirroring server to be prepared for hardware failures. One server can be restarte…

> With a second OS partition, the server can alternate between the working copy and a copy that is updated in a VM. For a free service, customers cannot complain even if there is a reboot every day and the service is down for a couple of minutes. Are you seriously suggesting that a service (the size of Twitter, no less) has an acceptable downtime of a few minutes a day? > Realistically, there would be a mirroring ser…

> has an acceptable downtime of a few minutes a day?

We kinda know the answer to this: Twitter was struggling with harm to its reputation for a long time because of regular Fail Whales. It absolutely was a huge problem for them at the time.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#449

Post by ilyt »

kaba0 wrote: post #443
drowsspa wrote: post #220

Earlier quoted context omitted.

I'm always reminded of that tweet from @SwiftOnSecurity https://mobile.twitter.com/SwiftOnSecurity/status/1485822027... > Once you understand your computer has 16 cores running at 3GHz and yet doesn't boot up in .2 nanoseconds you understand everything they have taken from you. With their infinite VC money at their disposal, and with their programmers having 100 GHz machines with thousands of cores, 128 TB of RAM and…

Booting has fundamental hardware limits, though. Will that harddrive or whatever really reply all in 0.2 ns?

And most machines don't even touch that limits. I remember some presentation about it [1] and actual limits aren't even touched, especially on servers.

From what I remember "hard" limits are CPU/DRAM-memory initialization and speed of read from flash chip storing, and sources of lag include firmware from add-on cards just being slow (if RAID controller takes 30 seconds to return, and firmware is not running initialization in parallel, that's your extra boot time. Or stuff out of left field like "IPMI controller logs stuff via serial so if you print too much text it slows down". Most BIOSes do things painfully parallel too.

* [1] https://www.youtube.com/watch?v=fTLsS_QZ8us

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#450

Post by steviesands »

spullara wrote: post #377
steviesands wrote: post #356

Earlier quoted context omitted.

A few thoughts. The first is, are we asking the wrong questions? Should it be, "If I spend 10m on hardware for predicting ads (storage/compute) that generates 25m in revenue, should I buy the hardware?". Sure, we can "minify" twitter, and it's a wonderful thought experiment, but it seems devoid of the context of revenue generation. The second is, it's interesting to understand social media industry wide infra cost pe…

The basic idea of the system was to scan a reverse chronologically ordered list of "user id, tweet id", filtering out any tweet whose user wasn't in the follow set (or sets in the case of scan sharing) until you retrieved enough tweets for the timeline request. There are a bunch of variants in the patent, but that is the basic idea. At the time, I estimated that Twitter was spending 80% of its CPU time in the DC doin…

Interesting, 80% seems a bit on the higher end nowadays though? For example, Google quantified this as the "datacenter tax" and through their cluster wide profiling tooling saw that it was 22-27% of all CPU cycles (still a huge amount). They go a different route and suggest hardware accelerators for common operations. Datacenter tax was defined as:

"The components that we included in the tax classification are: protocol buffer management, remote procedure calls (RPCs), hashing, compression, memory allocation and data movement."

https://static.googleusercontent.com/media/research.google.c...

Post reply on HN