Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Production Twitter on one machine? 100Gbps NICs and NVMe are fast

thume.ca

Post reply on HN
Print view View original

461–470 of 500 posts

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#461

Post by MichaelZuo »

aetimmes wrote: post #430

(Disclaimer: ex-Twitter SRE) > There’s a bunch of other basic features of Twitter like user timelines, DMs, likes and replies to a tweet, which I’m not investigating because I’m guessing they won’t be the bottlenecks. Each of these can, in fact, become their own bottlenecks. Likes in particular are tricky because they change the nature of the tweet struct (at least in the manner OP has implemented it) from WORM to wr…

What do you think about his interesting comment on the possibility of a mainframe?

"I also didn’t try to investigate configuring an IBM mainframe, which stands a chance of being the one type of “machine” where you might be able to attach enough storage to fit historical images."

It seems theoretically possible it could accomodate the entirety of Twitter in 'one machine'.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#462

Post by ilyt »

threeseed wrote: post #179
jiggawatts wrote: post #162

Something I've found a lot modern IT architects seem to ignore is "write amplification" or the equivalent effect for reads. If you have a 1 KB piece of data that you need to send to a customer, ideally that should require less than 1 KB of actual NIC traffic thanks to HTTP compression. If processing that 1 KB takes more than 1 KB of total NIC traffic within and out of your data centre, the you have some level of ampl…

Your point makes sense if you have no idea how Twitter works. It needs to assemble tweets internally, sort them with an ML model, add in relevant ads and present a single response to the user because end-user latency matters. And each of these systems eg. ads has their own features, complexities, development lifecycle and scaling requirements. And of course deploying them continuously without downtime. That is how yo…

> I know you think you’re smarter than everyone at Twitter. But those who really know what they are doing have a lot more respect for the engineers who built this insanity. There are always good intentions.

You ignored one possibility - that twitter engineers, or people managing them might be just incompetent and all of that might just be overly complex POS

There is that weird disgusting trend to assume just because company got big that means the tech choices were immaculate, and not everything else there is to successful companies.

You can make perfectly well doing company on totally mediocre product that hit the niche at right time

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#463

Post by dekhn »

sterlind wrote: post #151
dekhn wrote: post #136

Earlier quoted context omitted.

I wouldn't really agree with this since those machines don't share address spaces or directly attached busses. Better to say it's a warehouse-scale "service" provided by many machines which are aggregated in various ways.

I wonder though.. could you emulate a 20k-core VM with 100 terabytes of RAM on a DC? Ethernet is fast, you might be able to get in range of DRAM access with an RDMA setup. cache coherency would require some kind of crazy locking, but maybe you could do it with FPGAs attached to the RDMA controllers that implement something like Raft? it'd be kind of pointless and crash the second any machine in the cluster dies, but…

it's been done: https://www.bleepingcomputer.com/news/microsoft/monster-azur...

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#464

Post by HAL3000 »

aetimmes wrote: post #430

(Disclaimer: ex-Twitter SRE) > There’s a bunch of other basic features of Twitter like user timelines, DMs, likes and replies to a tweet, which I’m not investigating because I’m guessing they won’t be the bottlenecks. Each of these can, in fact, become their own bottlenecks. Likes in particular are tricky because they change the nature of the tweet struct (at least in the manner OP has implemented it) from WORM to wr…

> OP also overly focuses on throughput in his benchmarks

Because OP is a junior developer, he reads a lot of theory and blog posts, does a lot of research, but doesn't have much practical experience. Just look at his resume and what he wrote. As a result, most of what he write about is based on what he have read about senior developers doing in the companies he have worked for, perhaps he created some supporting software for core services but did not design or implemented the core, so he doesn't have firsthand experience. This is evident to anyone who has actually used DPDK (which is ridiculous proposal for Twitter like service in 2023 where you have XDP and io_uring, it's not HFT), designed and implemented high volume, low latency web services and knows where the bottleneck is in that kind of services from experience, theory will not give you that intuition and knowledge.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#465

Post by KingLancelot »

Dylan16807 wrote: post #322
KingLancelot wrote: post #231

Earlier quoted context omitted.

UTF-8 is 1-4 bytes per codepoint dude, not 1-2. 280 * 4 = 1120, not 560.

Twitter doesn't allow 280 of any codepoint. The more complicated ones count double against your limit.

Ah, so Twitter’s Unicode implementation is fucked.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#466

Post by gnuvince »

gnuvince wrote: post #415
vidarh wrote: post #383

Earlier quoted context omitted.

Many of us will remember that Twitter in fact did start out with a monolithic database and had to rewrite a bunch of stuff when they couldn't make that work anymore. Of course they could fit a much larger dataset on one machine today. (But I will note the article is also assuming a chronological timeline by default, but that of course hasn't been true for years - the ranking Twitter does now is far more complex)

There's a story by Bryan Cantrill [1] about how he went to Twitter to help them understand why it would take 400 milliseconds of compute to process a request (I'll leave the reveal to Bryan). Scaling horizontally is probably necessary for something the size of Twitter, but that doesn't mean that we can half-ass the code and just throw more machines at the problem. If we write code with a bit more mechanical sympathy…

Fixed timestamp: https://youtu.be/LjFM8vw3pbU?t=1613

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#467

Post by lossolo »

habibur wrote: post #253
lossolo wrote: post #152

Earlier quoted context omitted.

> Use HTTPS and watch it fall down to only 400 req / sec under load test [ without connection: keep-alive ]. I'm running about 2000 requests/s in one of my real-world production systems. All of the requests are without keep-alive and use TLS. They use about one core for TLS and HTTP processing.

Fascinating. Any special optimization you are using, or is it from off the shelf software and with standard configuration?

For offloading SSL I use haproxy with some custom settings and a few non standard kernel settings.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#468

Post by lossolo »

brazzledazzle wrote: post #391
lossolo wrote: post #152

Earlier quoted context omitted.

> Use HTTPS and watch it fall down to only 400 req / sec under load test [ without connection: keep-alive ]. I'm running about 2000 requests/s in one of my real-world production systems. All of the requests are without keep-alive and use TLS. They use about one core for TLS and HTTP processing.

Is this static or dynamic content? Are the simulated load test clients requesting the exact same pages/resources?

All dynamic content, all hitting data storage. There are no simulated clients, this is all real traffic from real clients, a lot of requests do writes, some do only reads.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#469

Post by lossolo »

tpetry wrote: post #378
kijin wrote: post #367

Earlier quoted context omitted.

Sounds totally off-the-shelf. I have a basic LAMP server running on a 4-core VM on a laptop. I just threw ApacheBench at it (not the fastest benchmarking tool, either -- it eats up 1 core all by itself), and it handles 1200 req/s TLS with no keepalive, and 3400 req/s with keepalive. This stuff scales linearly with core count, so I wouldn't be surprised to see much higher numbers in real servers.

Are these all new TLS connections? Because most benchmarks use TLS resumption which means the TLS handshake was only done once!

In my case, all of TLS connections are new and from real clients (not a benchmark/test)

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#470

Post by ilyt »

traceroute66 wrote: post #56
anamexis wrote: post #52

Earlier quoted context omitted.

That doesn't really seem like an example, since the whole thing doesn't run one machine. The database alone has multiple machines.

> That doesn't really seem like an example, since the whole thing doesn't run one machine. It is an example. It shows you how you can run a service that issues a few hundred million SSL certs a year off relatively few pieces of hardware, i.e. no need to go drinking the cloud Kool aid. There will never be a "perfect" example. The overall point here is demonstrating that the first answer to everything doesn't have to i…

> It is an example. It shows you how you can run a service that issues a few hundred million SSL certs a year off relatively few pieces of hardware, i.e. no need to go drinking the cloud Kool aid.

"100 million certs a year" is only like ~3 certs a second. Maybe twice in peak. That's not much. And you're doing same error, focusing on one core feature and ignoring everything around it

Post reply on HN