Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Use one big server

specbranch.com

Post reply on HN
Print view View original

111–120 of 601 posts

Re: Use one big server

#111

Post by reillyse »

rubiquity wrote: post #39
reillyse wrote: post #12

Nope. Multiple small servers. 1) you need to get over the hump and build in multiple servers into your architecture from the get go (the author says you need two servers minimum), so really we are talking about two big servers. 2) having multiple small servers allows us to spread our service into different availability zones 3) multiple small servers allows us to do rolling deploys without bringing down our entire se…

The line of thinking you follow is what is plaguing this industry with too much complexity and simultaneously throwing away incredible CPU and PCIe performance gains in favor of using the network. Any technical decisions about how many instances to have and how they should be spread out needs to start as a business decision and end in crisp numbers about recovery point/time objections, and yet somehow that nearly nev…

Ok, so to your points.

"It depends" is the correct answer to the question, but the least informative.

One Big Server or multiple small servers? It depends.

It always depends. There are many workloads where one big server is the perfect size. There are many workloads where many small servers are the perfect solution.

What my point is, is that the ideas put forward in the article are flawed for the vast majority of use cases.

I'm saying that multiple small servers are a better solution on a number of different axis.

For 1) "One Server (Plus a Backup) is Usually Plenty" Now I need some kind of remote storage streaming system and some kind of manual recovery, am I going to fail over to the backup (and so it needs to be as big as my "One server" or will I need to manually recover from my backup?

2) Yes it depends on your availability goals, but you get this as a side effect of having more than one small instance

3) Maybe I was ambiguous here. I don't just mean rolling deploys of code. I also mean changing the server code, restarting, upgrading and changing out the server. What happens when you migrate to a new server (when you scale up by purchasing a different box). Now we have a manual process that doesn't get executed very often and is bound to cause downtime.

4) Now we have "Use one Big Server - and a bunch of small ones"

I'm going to add a final point on reliability. By far the biggest risk factor for reliability is me the engineer. I'm responsible for bringing down my own infra way more than any software bug or hardware issue. The probability of me messing up everything when there is one server that everything depends on is much much higher, speaking from experience.

So. Like I said, I could have said "It depends" but instead I tried to give a response that was someway illuminating and helpful, especially given the strong opinions expressed in the article.

I'll give a little color with the current setup for a site I run.

moustachecoffeeclub.com runs on ECS

I have 2 on-demand instances and 3 spot instances

One tiny instance running my caches (redis, memcache) One "permanent" small instance running my web server

Two small spot instances running web server One small spot instance running background jobs

small being about 3 GB and 1024 CPU units

And an RDS instance with backup about $67 / month

All in I'm well under $200 per month including database.

So you can do multiple small servers inexpensively.

Another aspect is that I appreciate being able to go on vacation for a couple of weeks, go camping or take a plane flight without worrying if my one server is going to fall over when I'm away and my site is going to be down for a week. In a big company maybe there is someone paid to monitor this, but with a small company I could come back to a smoking hulk of a company and that wouldn't be fun.

Re: Use one big server

#112

Post by jvanderbot »

No thanks. I have a few hobby sites, a personal vanity page, and some basic CPU expensive services that I use.

Moving to Aws server-less has saved me so much headache with system updates, certificate management, archival and backup, networking, and so much more. Not to mention with my low-but-spikey load, my breakeven is a long way off.

Re: Use one big server

#113

Post by dghlsakjg »

bilekas wrote: post #103
gregmac wrote: post #8

> However, cloud providers have often had global outages in the past, and there is no reason to assume that cloud datacenters will be down any less often than your individual servers. A nice thing about being in a big provider is when they go down a massive portion of the internet goes down, and it makes news headlines. Users are much less likely to complain about your service being down when it's clear you're just c…

I can't tell if this is a good thing or a bad thing though! Imagine the clout of saying : "we stayed online while AWS died"

Depends on how technical your customer base is. Even as a developer I would tend not to ascribe too much signal to that message. All it tells me is that you don't use AWS.

"We stayed online when GCP, AWS, and Azure go down" is a different story. On the other hand, if those three go down simultaneously, I suspect the state of the world will be such that I'm not worried about the internet.

Re: Use one big server

#114

Post by bob1029 »

rubiquity wrote: post #39
reillyse wrote: post #12

Nope. Multiple small servers. 1) you need to get over the hump and build in multiple servers into your architecture from the get go (the author says you need two servers minimum), so really we are talking about two big servers. 2) having multiple small servers allows us to spread our service into different availability zones 3) multiple small servers allows us to do rolling deploys without bringing down our entire se…

The line of thinking you follow is what is plaguing this industry with too much complexity and simultaneously throwing away incredible CPU and PCIe performance gains in favor of using the network. Any technical decisions about how many instances to have and how they should be spread out needs to start as a business decision and end in crisp numbers about recovery point/time objections, and yet somehow that nearly nev…

> simultaneously throwing away incredible CPU and PCIe performance gains

We really need to double down on this point. I worry that some developers believe they can defeat the laws of physics with clever protocols.

The amount of time it takes to round trip the network in the same datacenter is roughly 100,000 to 1,000,000 nanoseconds.

The amount of time it takes to round trip L1 cache is around half a nanosecond.

A trip down PCIe isn't much worse, relatively speaking. Maybe hundreds of nanoseconds.

Lots of assumptions and hand waving here, but L1 cache can be around 1,000,000x faster than going across the network. SIX orders of magnitude of performance are instantly sacrificed to the gods of basic physics the moment you decide to spread that SQLite instance across US-EAST-1. Sure, it might not wind up a million times slower on a relative basis, but you'll never get access to those zeroes again.

Re: Use one big server

#116

Post by wahnfrieden »

I've started augmenting one big server with iCloud (CloudKit) storage, specifically syncing local Realm DBs to the user's own iCloud storage. Which means I can avoid taking custody of PII/problematic data, can include non-custodial privacy in product value/marketing, and means I can charge enough of a premium for the one big server to keep it affordable. I know how to scale servers in and out, so I feel the value of avoiding all that complexity. This is a business approach that leans into that, with a way to keep the business growing with domain complexity/scope/adoption (iCloud storage, probably other good APIs like this to work with along similar lines).

Re: Use one big server

#117

Post by [deleted] »

SoftTalker wrote: post #57
Nextgrid wrote: post #17

Earlier quoted context omitted.

> you need to get over the hump and build in multiple servers into your architecture from the get go (the author says you need two servers minimum), so really we are talking about two big servers. Managing a handful of big servers can be done manually if needed - it's not pretty but it works and people have been doing it just fine before the cloud came along. If you intentionally plan on having dozens/hundreds of sma…

I manage hundreds of servers, and use Ansible. It's simple and it gets the job done. I tried to install Kubernetes on a cluster and couldn't get it to work. I mean I know it works, obviously, but I could not figure it out and decided to stay with what works for me.

[deleted]

Re: Use one big server

#118

Post by nostrebored »

secabeen wrote: post #58
nostrebored wrote: post #33

Earlier quoted context omitted.

But most companies aren't startups. Many companies are established, growing businesses with a need to be able to easily implement new initiatives and products. The benefits of cloud for LE are completely different. I'm happy to break down why, but I addressed the smb and mid-enterprise space here because most large enterprises already know they shouldn't run on a single rack.

> I addressed the smb and mid-enterprise space here because most large enterprises already know they shouldn't run on a single rack. This is a straw man. No one, anywhere in this thread or in the OPs original article proposed a single-rack solution. From the OP: > Running a primary and a backup server is usually enough, keeping them in different datacenters.

This is just a complete lack of engagement with the post. Most LE’s know they shouldn’t run a two rack setup either. That is not the size or layout of any LE that I’ve interacted with. The closest is a bank in the developing world that had a few racks split across data centers in the same city and was desperately trying to move away given power instability in the country.

Re: Use one big server

#119

Post by Nextgrid »

bilekas wrote: post #103
gregmac wrote: post #8

> However, cloud providers have often had global outages in the past, and there is no reason to assume that cloud datacenters will be down any less often than your individual servers. A nice thing about being in a big provider is when they go down a massive portion of the internet goes down, and it makes news headlines. Users are much less likely to complain about your service being down when it's clear you're just c…

I can't tell if this is a good thing or a bad thing though! Imagine the clout of saying : "we stayed online while AWS died"

HN implicitly gets this clout - it became the real status page of most of the internet.

Re: Use one big server

#120

Post by phkahler »

treis wrote: post #53
tjoff wrote: post #50

Earlier quoted context omitted.

This seems like a recently popular exaggeration, I'd wager no one but a select few in the HN-bubble actually cares. You will primarily be judged by how much of an inconvenience the outage was to every individual. The best you can hope for is that the local ISP gets the blame, but honestly. It can't be more than a rounding error in the end.

I think it's more of a shield against upper management. AWS going down is treated like an act of god rendering everyone blameless. But if it's your one big server that goes down then it's your fault.

>> AWS going down is treated like an act of god rendering everyone blameless.

Someone decided to use AWS, so there is blame to go around. I'm not saying if that blame is warranted or not, just that it sounds like a valid thing to say for people who want to blame someone.

Post reply on HN