Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Production Twitter on one machine? 100Gbps NICs and NVMe are fast

thume.ca

Post reply on HN
Print view View original

431–440 of 500 posts

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#431

Post by Sebb767 »

glass3 wrote: post #399
icedchai wrote: post #312

Earlier quoted context omitted.

True. You'd also save a ton of less operations and engineering staff. Running anything on a single server, however, is really a non starter for anything remotely serious. What do you do if you need to do an OS update? I suppose you could just never do those, like a former employer (1000+ day uptimes...)

Twitter is not serious enough that there cannot be a timeout. With a second OS partition, the server can alternate between the working copy and a copy that is updated in a VM. For a free service, customers cannot complain even if there is a reboot every day and the service is down for a couple of minutes. Realistically, there would be a mirroring server to be prepared for hardware failures. One server can be restarte…

> With a second OS partition, the server can alternate between the working copy and a copy that is updated in a VM. For a free service, customers cannot complain even if there is a reboot every day and the service is down for a couple of minutes.

Are you seriously suggesting that a service (the size of Twitter, no less) has an acceptable downtime of a few minutes a day?

> Realistically, there would be a mirroring server to be prepared for hardware failures. One server can be restarted while the other is the main server.

But for that mirroring, you need to replicate disk writes, databases, backups etc.. This additional load would easily bring the server to a point where a single server would no longer suffice, even an insanely spec'ed one.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#432

Post by Sebb767 »

icedchai wrote: post #312
wtallis wrote: post #255

Earlier quoted context omitted.

A Twitter-like service that fits on a single server could probably get by with the reduced revenue that comes with not offering obsessively fine-grained analytics and ad targeting.

True. You'd also save a ton of less operations and engineering staff. Running anything on a single server, however, is really a non starter for anything remotely serious. What do you do if you need to do an OS update? I suppose you could just never do those, like a former employer (1000+ day uptimes...)

> You'd also save a ton of less operations and engineering staff.

You absolutely would not. The cost of having developers put out extremely optimized code (due to the scaling limits) and cuddle that single server to never, ever fail easily eclipses the cost of a having a multiple servers by a few orders of magnitude.

EDIT: To the downvoters, I'd really love to see the calculation on how engineering time would be cheaper than buying a second server in any reasonable timeframe.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#433

Post by mcqueenjordan »

Fun thought experiment! I can't help but be reminded of the Good Will Hunting quote, though:

SEAN: So if I asked you about art you’d probably give me the skinny on every art book ever written. Michelangelo? You know a lot about him. Life’s work, political aspirations, him and the pope, sexual orientation, the whole works, right? But I bet you can’t tell me what it smells like in the Sistine Chapel. You’ve never actually stood there and looked up at that beautiful ceiling. Seen that.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#434

Post by ilyt »

cellis wrote: post #320
untech wrote: post #286

Earlier quoted context omitted.

Using two 32 bit numbers for coordinates, each record would take 8 bytes, which is 64 gigabytes for 8 billion population. Don’t think many smartphones have this RAM today.

Don't you need 3 numbers? Unless you believe in a flat earth ;). Also you need some slack space for metadata. Let's call it 100GB all in.

Most people don't live in caves or in the sky city, 2 is enough.

And in places you care (multi-floor buildings) you aren't getting GPS signal inside anyway...

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#435

Post by ilyt »

Dylan16807 wrote: post #351
threeseed wrote: post #343

Earlier quoted context omitted.

It is trivial compared to the task of translating a user query into search results. It's akin to saying the magic behind OpenGPT is the dataset.

Keyword search is not that hard. Pagerank is not that hard. The comparable goal to the article is to be a search engine, not to fight google for best results.

I think you severely underestimate the effort search engines have to put to filter out all the SEO garbage and come up with some sensible results

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#437

Post by wrigby »

britannio wrote: post #416
popcorncowboy wrote: post #410

Earlier quoted context omitted.

It's less "short memory" than the fact that you can be a "senior software engineer" after just 5 years or so experience. There is a significant cohort of (particularly web-tech) developers who were young children in the 90s, and whose professional careers started in the 2010s and have only ever known "the cloud", big-tech and big-tech tech (k8s, etc). It's a similar phenomenon to the observation that tech "innovation…

I'm a CS undergrad, do you have any recommendations for effectively combating this?

No approach is a silver bullet here, but what I've found effective is to seek out friendships with / mentorship from senior and staff-level engineers. They'll have tons of war stories from 10-20 years ago, and may even have some snarky opinions on what "new technologies" are just re-inventions of something that the industry had already solved decades ago.

As a current undergrad, you can also look to your professors for this (especially those with industry experience before they went into teaching).

After graduation, this may mean working at a company _with_ those older engineers, as opposed to a 5-20 person startup with a homogeneous group of 20-somethings.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#438

Post by saalweachter »

TacticalCoder wrote: post #244

TFA, to me, touches about something I've wondered about a very long time ago: what are the implications of CPU and storage growing at much faster rates than human population? Back in the 486 days you wouldn't be keeping, in RAM, data about every single human on earth (let's take "every single human on earth" as the maximum number of humans we'll offer our services to with on our hypothetical server). Nowadays keeping…

A bacteria has order of 50 billion atoms. (Eukaryotic human cells, 100 trillion.)

That's getting to the point you could store 20 bytes per atom in a terabyte.

(The big bottleneck is that you need picosecond resolution simulation steps and to cover minutes to see a protein fold.)

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#439

Post by ilyt »

summerlight wrote: post #365

I think many people in this thread are making the mistake of ignoring evolutionary factors in system engineering. If a system doesn't need to adopt or change, lots of things can be much more efficient, easier and simpler, likely the order of 10x~100x. But you gotta appreciate that we're all paid because we need to swap wheels on running trains (or even engines in flying airplanes). A large fraction of demand for redu…

> A large fraction of demand for redundancy, introspection, abstraction and generalization comes from this.

Sure, you need to invest into it but those are things you can reuse for every app and feature you build.

And those are not the reason why those systems are so complex, those are just ways to keep complex systems running and manageable. In most they also do not stand in the way of making system better but help in it.

They need to exist because the architecture of system grew organically from smaller system over and over again and big restructurization was deemed not worth it. It's "just have a bunch more hardware and engineers" vs "we're not delivering features and we might not get rewrite right".

And every time you throw money at the problem the problem becomes a bigger problem and potential benefits from "getting it right" are also getting bigger. But nobody wants to be herald that tells management "we 're going to spend 6-12 months" for somethinkg that have few years of pay-off

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#440

Post by ilyt »

habibur wrote: post #62

He will be up for surprise. HTTP with connection: keep-open can serve 100k req/sec. But that's for one client being served repeatedly over 1 connection. And this is the inflated number that's published in webserver benchmark tests. For more practical down to earth test, you need to measure performance w/o keep-alive. Request per second will drop to 12k / sec then. And that's for HTTP without encryption or ssl handsha…

using what ? That numbers are on low side even for my old desktop
Post reply on HN