Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Production Twitter on one machine? 100Gbps NICs and NVMe are fast

thume.ca

Post reply on HN
Print view View original

421–430 of 500 posts

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#421

Post by kaba0 »

madeofpalk wrote: post #269
mrtksn wrote: post #193

Earlier quoted context omitted.

Twitter's "linked" tweets seems to be strangely unattached from context. What I mean is, Twitter seems to be processing data based on whatever it is in the tweet and doesn't maintain some grand coherent database. So I changed my Twitter handle and opened a new account with my original Twitter handle and to my surprise, I was receiving notifications of engagement with tweets my old account sent previously. I also hear…

Old stuff on twitter is weird. Tweets seem to eventually forget that You specifically liked a tweet, and will allow you to like a tweet again, and the like count will reflect you liking it twice.

Like count is almost surely just a locally incremented number, your like request will eventually get processed and will be dropped if it was already liked. It only has to be an eventually consistent value.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#422

Post by femiagbabiaka »

icedchai wrote: post #312
wtallis wrote: post #255

Earlier quoted context omitted.

A Twitter-like service that fits on a single server could probably get by with the reduced revenue that comes with not offering obsessively fine-grained analytics and ad targeting.

True. You'd also save a ton of less operations and engineering staff. Running anything on a single server, however, is really a non starter for anything remotely serious. What do you do if you need to do an OS update? I suppose you could just never do those, like a former employer (1000+ day uptimes...)

Twitter actually used to have downtime during certain updates IIRC. The state department asked them to skip one such maintenance in service of some Iranian protests in 2009. I highly doubt that the era of US ops in twitter has ended, so downtime is probably a nonstarter.

Source: https://foreignpolicy.com/2009/06/16/state-department-interv...

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#423

Post by kaba0 »

Joeri wrote: post #358
seanhunter wrote: post #336

Earlier quoted context omitted.

It's sort of strange you have to make these points, but as an industry we seem to have an extremely short memory. Vertical scaling was absolutely the way most big applications were built up until well into the 90s. Companies like Oracle were really built on the fact that getting performance and reliability out of a single highly-contested massive server is hard but important if that's the way you're going. Linux beca…

The thing is that hardware scales faster than humanity. When the internet boom happened there was no choice except to scale horizontally to reach a global audience, but as this article points out that assumption might no longer hold true for many services. It might make sense to return to vertically scaled highly reliable servers to achieve software simplicity and a lower overall cost. I’m always reminded of how stac…

Most, sure. Twitter? No way on Earth.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#424

Post by social_quotient »

teaearlgraycold wrote: post #309
vidarh wrote: post #237

Earlier quoted context omitted.

Biggest problem with this is the lack of considering analytics.twitter.com and ads.twitter.com. Twitter stores event data about everything that happens to a tweet, and lets you target ads with a lot of precision. While some of those writes may well be acceptable to lose, letting you write to caches, effectively you need to assume there are more analytics events triggering writes to something than there are tweet view…

Analytics data can easily be 90% of your data.

Phrased differently for effect. Analytics data can be 9x larger than core data.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#425

Post by jameshart »

Joeri wrote: post #358
seanhunter wrote: post #336

Earlier quoted context omitted.

It's sort of strange you have to make these points, but as an industry we seem to have an extremely short memory. Vertical scaling was absolutely the way most big applications were built up until well into the 90s. Companies like Oracle were really built on the fact that getting performance and reliability out of a single highly-contested massive server is hard but important if that's the way you're going. Linux beca…

The thing is that hardware scales faster than humanity. When the internet boom happened there was no choice except to scale horizontally to reach a global audience, but as this article points out that assumption might no longer hold true for many services. It might make sense to return to vertically scaled highly reliable servers to achieve software simplicity and a lower overall cost. I’m always reminded of how stac…

When the hardware scales, the tricks to wring maximum performance out of it change.

When you come back to build the new version of your game on the next gen console sure you can now add all those features but the processing pipelines are different now and the disk performance and memory to cache ratios have changed - getting your hyper optimized code to work in the new platform takes a ton of effort - so you either run it in some kind of emulation mode, sacrificing some of the performance for productivity, or you rewrite it.

Same happens with new generations of server hardware. Your clever hack to maximize NUMA locality of data to each core becomes a liability when the next hardware gen comes out and on-die caches are bigger. Decisions about what should use RAM get invalidated by faster SSDs.

Maybe you can build a service this way - hardware first. Pick a server platform for a couple of years; build to that capacity; ship; then start designing the next gen service to run on a new set of hardware?

To some extent this is how database or virtualization systems software is written. And if I’m not mistaken Twitter actually did develop their own database stack to optimally handle their particular data storage model, and I assume that was done pretty close to the metal.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#426

Post by madeofpalk »

kaba0 wrote: post #421
madeofpalk wrote: post #269

Earlier quoted context omitted.

Old stuff on twitter is weird. Tweets seem to eventually forget that You specifically liked a tweet, and will allow you to like a tweet again, and the like count will reflect you liking it twice.

Like count is almost surely just a locally incremented number, your like request will eventually get processed and will be dropped if it was already liked. It only has to be an eventually consistent value.

This is beyond optimistic updates (which twitter also has plenty of issues with - you'll reply to your tweet and it'll jump immediately to two replies, but on refresh just show your one).

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#427

Post by kaba0 »

gnuvince wrote: post #415
vidarh wrote: post #383

Earlier quoted context omitted.

Many of us will remember that Twitter in fact did start out with a monolithic database and had to rewrite a bunch of stuff when they couldn't make that work anymore. Of course they could fit a much larger dataset on one machine today. (But I will note the article is also assuming a chronological timeline by default, but that of course hasn't been true for years - the ranking Twitter does now is far more complex)

There's a story by Bryan Cantrill [1] about how he went to Twitter to help them understand why it would take 400 milliseconds of compute to process a request (I'll leave the reveal to Bryan). Scaling horizontally is probably necessary for something the size of Twitter, but that doesn't mean that we can half-ass the code and just throw more machines at the problem. If we write code with a bit more mechanical sympathy…

Twitter used to (not sure how much did that idiot fck it up) be quite great from this perspective from what I gathered. They had technical blog posts and are even featured in the famous data intensive book as an example for many scaling problems. Sure, they didn’t write it in assembly but instead use Java/Scala with Graal, but architecturally they had a sound system (plus what people routinely forget, the system should handle the general load, but it worths nothing if it fails at peaks, and twitter can easily hit a billion daily users during big global events)z

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#428

Post by kaba0 »

TacticalCoder wrote: post #244

TFA, to me, touches about something I've wondered about a very long time ago: what are the implications of CPU and storage growing at much faster rates than human population? Back in the 486 days you wouldn't be keeping, in RAM, data about every single human on earth (let's take "every single human on earth" as the maximum number of humans we'll offer our services to with on our hypothetical server). Nowadays keeping…

Hold it in memory? Sure. But actually working with it besides copying it around and doing some operation on a “row-by-row” basis? Not really.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#429

Post by knorker »

BeefWellington wrote: post #40

I'm going to preface this criticism by saying that I think exercises like this are fun in an architectural/prototyping code-golf kinda way. However, I think the author critically under-guesses the sizes of things (even just for storage) by a reasonably substantial amount. e.g.: Quote tweets do not go against the size limit of the tweet field at Twitter. Likely they are embedding a tweet reference in some manner or ot…

I would be surprised if indexes were not larger than the raw tweets.

Text search, hashtag index, some structured data for popular tweets, etc...

In order to deliver search results I would not be surprised if tweets are duplicated/denormalized, for quick search/lookup.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#430

Post by aetimmes »

(Disclaimer: ex-Twitter SRE)

> There’s a bunch of other basic features of Twitter like user timelines, DMs, likes and replies to a tweet, which I’m not investigating because I’m guessing they won’t be the bottlenecks.

Each of these can, in fact, become their own bottlenecks. Likes in particular are tricky because they change the nature of the tweet struct (at least in the manner OP has implemented it) from WORM to write-many, read-many, and once you do that, locking (even with futexes or fast atomics) becomes the constraining performance factor. Even with atomic increment instructions and a multi-threaded process model, many concurrent requests for the same piece of mutable data will begin to resemble serial accesses - and while your threads are waiting for their turn to increment the like counter by 1, traffic is piling up behind them in your network queues, which causes your throughput to plummet and your latency to skyrocket.

OP also overly focuses on throughput in his benchmarks, IMO. I'd be interested to see the p50/p99 latency of the requests graphed against throughput - as you approach the throughput limit of an RPC system, average and tail latency begin to increase sharply. Clients are going to have timeout thresholds, and if you can't serve the vast majority of traffic in under that threshold consistently (while accounting for the traffic patterns of viral tweets I mentioned above) then you're going to create your own thundering herd - except you won't have other machines to offload the traffic to.

Post reply on HN