Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Production Twitter on one machine? 100Gbps NICs and NVMe are fast

thume.ca

Post reply on HN
Print view View original

391–400 of 500 posts

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#391

Post by brazzledazzle »

lossolo wrote: post #152
habibur wrote: post #62

He will be up for surprise. HTTP with connection: keep-open can serve 100k req/sec. But that's for one client being served repeatedly over 1 connection. And this is the inflated number that's published in webserver benchmark tests. For more practical down to earth test, you need to measure performance w/o keep-alive. Request per second will drop to 12k / sec then. And that's for HTTP without encryption or ssl handsha…

> Use HTTPS and watch it fall down to only 400 req / sec under load test [ without connection: keep-alive ]. I'm running about 2000 requests/s in one of my real-world production systems. All of the requests are without keep-alive and use TLS. They use about one core for TLS and HTTP processing.

Is this static or dynamic content? Are the simulated load test clients requesting the exact same pages/resources?

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#392

Post by ksec »

trishume wrote: post #205
jiggawatts wrote: post #156

Earlier quoted context omitted.

> which is how many hard drives you can plug in to a non-mainframe machine for historical image storage. You would be surprised . First off, SSDs are denser than hard drives now if you're willing to spend $$$. Second, "plug in" doesn't necessarily mean "in the chassis". You can expand storage with external disk arrays in all sorts of ways. Everything from external PCI-e cages to SAS disk arrays, fibre channel, NVMe-o…

That's really cool! Each year of historical images I estimate at 2.8PB, so it would need to scale quite far to handle multiple years. How would you actually connect all those external drive chassis, is there some kind of chainable SAS or PCIe that can scale arbitrarily far? I consider NVMe-over-fabrics to be cheating and just using multiple machines and calling it one machine, but "one machine" is kinda an arbitrary…

It depends on how you think of "one machines". :) You can fit 1PB in 1U without something like NVMe-over-fabrics. So in a 4U unit gives you plenty of room.

We have Zen4c 128 Core with DDR5 now. We might get a 256 Core Zen6c with PCI-E 6.0 and DDR6 by 2026.

I really like these exercise of trying to shrink the amount of server needed, especially those on Web usage. And the mention of Mainframe. Which dont get enough credit for. I did something similar with Netflix 800Gbps's post. [1] Where they could serve every single user with less than 50 Racks by the end of this decade.

[1] https://news.ycombinator.com/item?id=33451430

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#393

Post by withinboredom »

doublepg23 wrote: post #273
narag wrote: post #260

Earlier quoted context omitted.

TFA, to me, touches about something I've wondered about a very long time ago... It was a little more than ten years ago for me. I realized that a hard disk could store a database of every human alive, including basic information, links (family relations) and maybe a photo. I still don't know what the implications are. Maybe we don't want to know, but it's not really that difficult to think about.

Is the storage really the complex part? Isn't gathering the actual information and avoiding errors (ex: I have a co-worker who's name is incorrectly spelled three different ways in prod services) the hard part?

Data hygiene is a problem everywhere. Most companies I worked at just throw out 'bad data' and file a bug. Occasionally, if the data is a secondary source, it will be recreated from a primary source (assuming the primary source is still available).

In the particular case, your coworker would be stored by some identifier (like an SSN or similar) and their name would be stored as "aliases" and allow multiple names. I have two nicknames that I answer to, depending on when in my life you met me, and my family calls me by my proper name. Online, I go by several handles depending on whether I want the reader to be able to figure out my real name. I even used to work somewhere where I was called by this handle (withinboredom) more than my real name.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#394

Post by vidarh »

varunkmohan wrote: post #119
PragmaticPulp wrote: post #65

Earlier quoted context omitted.

> However, it does show how much real compute bloat these companies actually have No, it doesn’t. It’s a fun exercise in approaching Twitter as an academic exercise. It ignores all of the real-world functionality that makes it a business rather than a toy. A lot of complicated businesses are easy to prototype out if you discard all requirements other than the core feature. In the real world, more engineering work oft…

Genuinely asking, why do you think Twitter needs 24 million vcpus to run? This is not apples to apples but Whatsapp is a product that entirely ran on 16 servers at the time of acquisition (1.5 billion users). It really begs the question why Twitter uses so much compute if there are companies that have operated significantly more efficiently. Twitter was unprofitable during acquisition and spent around half their reve…

Whatsapp is mostly message delivery from one person to a tiny group. It's absolutely trivial compared to Twitter.

In terms of what Twitter uses compute on, I'd guess analytics (Twitter measures "everything" for ad serving - go explore ads.twitter.com and analytics.twitter.com) and non-chronological timeline mixing both takes orders of magnitude more resources than the basic functionality.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#395

Post by kijin »

brazzledazzle wrote: post #390
kijin wrote: post #367

Earlier quoted context omitted.

Sounds totally off-the-shelf. I have a basic LAMP server running on a 4-core VM on a laptop. I just threw ApacheBench at it (not the fastest benchmarking tool, either -- it eats up 1 core all by itself), and it handles 1200 req/s TLS with no keepalive, and 3400 req/s with keepalive. This stuff scales linearly with core count, so I wouldn't be surprised to see much higher numbers in real servers.

Is AB running on the same machine during the tests?

Same machine, pegging a CPU core the whole time. So the server side only has 3 cores left to use.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#396

Post by kijin »

tpetry wrote: post #378
kijin wrote: post #367

Earlier quoted context omitted.

Sounds totally off-the-shelf. I have a basic LAMP server running on a 4-core VM on a laptop. I just threw ApacheBench at it (not the fastest benchmarking tool, either -- it eats up 1 core all by itself), and it handles 1200 req/s TLS with no keepalive, and 3400 req/s with keepalive. This stuff scales linearly with core count, so I wouldn't be surprised to see much higher numbers in real servers.

Are these all new TLS connections? Because most benchmarks use TLS resumption which means the TLS handshake was only done once!

AFAIK apachebench is so old it doesn't support TLS resumption, but I might be wrong.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#397

Post by withinboredom »

cellis wrote: post #320
untech wrote: post #286

Earlier quoted context omitted.

Using two 32 bit numbers for coordinates, each record would take 8 bytes, which is 64 gigabytes for 8 billion population. Don’t think many smartphones have this RAM today.

Don't you need 3 numbers? Unless you believe in a flat earth ;). Also you need some slack space for metadata. Let's call it 100GB all in.

GPS altitude is notoriously bad via consumer devices. It's better to store altitude as less than 2 bytes with the range from being a few km below sea level to the maximum altitude consumer devices will report (3,000 km -- though maybe less because there is also a speed limit at which they will stop reporting too so you can't just buy a GPS device and build a guided missile).

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#398

Post by kijin »

IshKebab wrote: post #388
kijin wrote: post #361

Earlier quoted context omitted.

Compare the cost of operating multiple servers, on one hand, with the lost revenue from having weekly or monthly maintenance windows during which you just put up a Fail Whale page. Most people overestimate the latter by a huge margin.

That's fine if your service is really local - you can do it at night. Not really an option for a global site. Imagine if Twitter went down for a few hours every month. People are addicted to Twitter. It might be at a critical time for an entire country (e.g. the Queen dies). Even worse you can't guarantee how long the upgrade will take. You'd definitely need at least two servers. But I think you could surely just hav…

Yeah, I was just responding to the "how about OS updates" part of the parent comment, for which scheduled downtime is a reasonable option. To protect your service from unscheduled downtime though, like a failing RAID array, you would need at least two servers.

Personally I wouldn't run a critical service on only one server, but two servers? Definitely doable. I actually have a service running on two servers in different DCs 700 miles apart. Zero downtime in 9 years. :)

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#399

Post by glass3 »

icedchai wrote: post #312
wtallis wrote: post #255

Earlier quoted context omitted.

A Twitter-like service that fits on a single server could probably get by with the reduced revenue that comes with not offering obsessively fine-grained analytics and ad targeting.

True. You'd also save a ton of less operations and engineering staff. Running anything on a single server, however, is really a non starter for anything remotely serious. What do you do if you need to do an OS update? I suppose you could just never do those, like a former employer (1000+ day uptimes...)

Twitter is not serious enough that there cannot be a timeout.

With a second OS partition, the server can alternate between the working copy and a copy that is updated in a VM. For a free service, customers cannot complain even if there is a reboot every day and the service is down for a couple of minutes.

Realistically, there would be a mirroring server to be prepared for hardware failures. One server can be restarted while the other is the main server.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#400

Post by vidarh »

Xeoncross wrote: post #181
BeefWellington wrote: post #40

I'm going to preface this criticism by saying that I think exercises like this are fun in an architectural/prototyping code-golf kinda way. However, I think the author critically under-guesses the sizes of things (even just for storage) by a reasonably substantial amount. e.g.: Quote tweets do not go against the size limit of the tweet field at Twitter. Likely they are embedding a tweet reference in some manner or ot…

> I think the author critically under-guesses the sizes of things (even just for storage) by a reasonably substantial amount. While true, and not to take away from the parent comment, I've noticed that the size of things is often partially the result of scaling out horizontally. Most companies I've worked at end up with a lot of duplicate records as each subsystem might want a copy or to cache a copy.

This is indeed a problem, and one of the reasons to be careful about knowing you can fit things in one machine before taking that approach, because the moment you're forced to move to a distributed model it's rarely one machine to two but one machine to a dozen coupled with a major rearchitecturing effort at just the wrong moment.

It's often fine to start without a fully decoupled system (net present value of the time and money needed to scale out might be far too high), but you need to know whether or not it's likely to come and what to look for so you can start preparing in time.

Post reply on HN