Live data from Hacker News

Production Twitter on one machine? 100Gbps NICs and NVMe are fast

thume.ca

361–370 of 500 posts

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#361

Earlier quoted context omitted.

A Twitter-like service that fits on a single server could probably get by with the reduced revenue that comes with not offering obsessively fine-grained analytics and ad targeting.

True. You'd also save a ton of less operations and engineering staff. Running anything on a single server, however, is really a non starter for anything remotely serious. What do you do if you need to do an OS update? I suppose you could just never do those, like a former employer (1000+ day uptimes...)

Compare the cost of operating multiple servers, on one hand, with the lost revenue from having weekly or monthly maintenance windows during which you just put up a Fail Whale page. Most people overestimate the latter by a huge margin.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#363
post #62

He will be up for surprise. HTTP with connection: keep-open can serve 100k req/sec. But that's for one client being served repeatedly over 1 connection. And this is the inflated number that's published in webserver benchmark tests. For more practical down to earth test, you need to measure performance w/o keep-alive. Request per second will drop to 12k / sec then. And that's for HTTP without encryption or ssl handsha…

[deleted]

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#365
I think many people in this thread are making the mistake of ignoring evolutionary factors in system engineering. If a system doesn't need to adopt or change, lots of things can be much more efficient, easier and simpler, likely the order of 10x~100x. But you gotta appreciate that we're all paid because we need to swap wheels on running trains (or even engines in flying airplanes). A large fraction of demand for redundancy, introspection, abstraction and generalization comes from this.

Why do we want to apply ML at the cost of a significant fleet cost increase? Because it can make the overall system consistently perform against external changes via generalization, thus the system can evolve more cheaply. Why do we want to implement a complex logging layer although it doesn't bring direct gains on system performance? Because you need to inspect the system to understand its behavior and find out where it needs to change. The list can go on and I can give you hundreds of reasons why we need all these apparently unnecessary complexities and overheads can be important for systems' longevity.

I don't deny the existence of accidental complexities (probably Twitter can become 2~3x simpler and cheaper given sufficient eng resource and time), but in many cases you probably won't be able to confidently say if some overheads are accidental or essential since system engineering is essentially a highly predictive/speculative activity. To make this happen, you gotta have a precise understanding of how the system "currently works" to make a good bet rather than re-imagination of the system with your own wish list of how the system "should work". There's a certain value on the latter option, but it's usually more constructive to build an alternative rather than complaining about the existing system. This post is great since the author actually tried to build something to prove its possibility, this knowledge could turn out to be valuable for other Twitter alternatives later on.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#366
post #214

Earlier quoted context omitted.

I didn't get the impression that this would duplicate the entire functionality of Twitter, just what amounts to the MVP functionality. If you are only talking about the MVP it's at least somewhat plausible with a lot of careful engineering and highly efficient data manipulation. Adding images, videos, other large attachments, rich search, and all the advertising and billing and analytics stuff would blow this out of…

I didn't get the impression that this would duplicate the entire functionality of Twitter, just what amounts to the MVP functionality. If you are only talking about the MVP it's at least somewhat plausible with a lot of careful engineering and highly efficient data manipulation. I agree mostly. Where I differ in that I would argue that hashtags were THE thing that Twitter is most known for but that could be a perspec…

FWIW my opinion of hashtags (relatively new to Twitter) is that they’re only used by brands and mostly cringe people

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#367

Earlier quoted context omitted.

> Use HTTPS and watch it fall down to only 400 req / sec under load test [ without connection: keep-alive ]. I'm running about 2000 requests/s in one of my real-world production systems. All of the requests are without keep-alive and use TLS. They use about one core for TLS and HTTP processing.

Fascinating. Any special optimization you are using, or is it from off the shelf software and with standard configuration?

Sounds totally off-the-shelf.

I have a basic LAMP server running on a 4-core VM on a laptop. I just threw ApacheBench at it (not the fastest benchmarking tool, either -- it eats up 1 core all by itself), and it handles 1200 req/s TLS with no keepalive, and 3400 req/s with keepalive. This stuff scales linearly with core count, so I wouldn't be surprised to see much higher numbers in real servers.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#368

Earlier quoted context omitted.

Isn’t a hashtag just another kind of user account — an account from which anyone can post? The data structures for the @BeefWellington timeline of tweets and the one for the #BeefWellington timeline of tweets could look roughly the same.

Hashtags aren't like user accounts, no - they're strings that are part of a tweet. In theory, a separate data structure shouldn't be needed since you can just search the full text of tweets, but in practice, I don't know how that scales for the number of all-time tweets.

I think he was talking about how hashtags are implemented.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#369
post #358

Earlier quoted context omitted.

It's sort of strange you have to make these points, but as an industry we seem to have an extremely short memory. Vertical scaling was absolutely the way most big applications were built up until well into the 90s. Companies like Oracle were really built on the fact that getting performance and reliability out of a single highly-contested massive server is hard but important if that's the way you're going. Linux beca…

The thing is that hardware scales faster than humanity. When the internet boom happened there was no choice except to scale horizontally to reach a global audience, but as this article points out that assumption might no longer hold true for many services. It might make sense to return to vertically scaled highly reliable servers to achieve software simplicity and a lower overall cost. I’m always reminded of how stac…

Stack Overflow happens to go down fair maintenance a fair bit. Now, it’s not really a service that might need reliability like Twitter does*, but it’s important to keep in mind.

*Some engineers may disagree

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#370

Earlier quoted context omitted.

Maybe they are considering that we shouldn't build and optimize our society solely for the purposes of maximizing revenue. Would it be better to live in a world where Twitter (for example) existed because it is a useful thing and not because it might make lots of money?

Doesn't it lose a lot of money and its usefulness is directly correlated with its current massive usage.

It ran about break even until very recently.
Post reply on HN