Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tell HN: AWS appears to be down again

news.ycombinator.com

Post reply on HN
Print view View original

361–370 of 646 posts

Re: Tell HN: AWS appears to be down again

#361

Post by tluyben2 »

JCM9 wrote: post #303

AWS didn’t “go down”. They had an outage in one AZ, which is why there are multiple AZs in each region. If your app went down then you should be blaming your developers on this one, not AWS. Those having issues are discovering gaps in their HA designs. Obviously it’s not good for an AZ to go down but it does happen and why any production workload should be architected to have seamless failover and recover to other AZ…

> People commenting that servers shouldn’t go down ect don’t understand how true HA architectures work. You should expect and build for stuff to fail like this. Otherwise it’s like complaining that you lost data because a disk failed. Disks fail… build architecture where that won’t take you down.

Is that comparison fair? If you have 2 raid-5 mirrored raid 5 boxes in your room and all disks fail at the same time, you should complain. And that won't happen. These entire datacenter failures should be anticipated, but to expect them is a bit too easy I think. There are plenty of hosters who don't have this stuff even once for the last decade in their only datacenter. I do not find it strange to expect or even demand that level but to protect yourself if it happens in any case if that fits your specific project and budget.

Edit; OK meant that raid-5 remark in the same context as the hosting; it can and does happen but it shouldn't; you should plan for a contingency but expect it goes far. We never had it (1000s of hard-drive, decades of hosting, millions of sites) and so we plan for it with backups; if it happens it will take some downtime but it costs next to nothing over time to do that. If we expected it, we would need to take far different measures. And we had less downtime in a decade than aws AZ had in the past months. I have a problem with the word 'expect'.

Re: Tell HN: AWS appears to be down again

#362

Post by dijit »

praseodym wrote: post #359
dijit wrote: post #302

Earlier quoted context omitted.

EDIT: I misunderstood you were talking about power feeds, the normal case is the run "48% as if it's 100%" (because of power spikes, but also most types of transformers run more efficiently under specific levels of load (40-60). Normally this is factored into the Rack you buy from a hardware provider, they will tell you that you have 10A or 16A on each feed, if you exceed that: it will work, but you are overloading t…

OP is talking about the DC power feed, not a single server PSU.

You don't get fed DC power, you get fed AC power.

But, point taken: yes your power feed should be running at Mostly this is outsourced to the datacenter provider; they'll give you a per side rating. (usually 10A or 16A) which also matches the cooling profile of the cabinet.

Re: Tell HN: AWS appears to be down again

#363

Post by nostrebored »

Retric wrote: post #313
lmilcin wrote: post #275

Earlier quoted context omitted.

Think about it this way: 1) Can you make your on prem infrastructure go down less than Amazon's? 2) Is it worth it? In my experience most people grossly underestimate how expensive it is to create reliable infrastructure and at the same time overestimate how important it is for their services to run uninterrupted. -- EDIT: I am not arguing you shouldn't build your more reliable infrastructure. AWS is just a point on…

It really spends on how reliable you need to be. Don’t forget you get downtime from both AWS and your own issues so even 4 9’s is off the table with pure AWS. If you need to be more reliable than AWS you need to run a hybrid inside and outside of AWS which means most of the advantages of running on AWS goes away.

Very untrue. Many businesses with 4 9 SLAs are all in on AWS. It requires active/active setups though!

Re: Tell HN: AWS appears to be down again

#364

Post by tzs »

laumars wrote: post #330
Nextgrid wrote: post #285

Earlier quoted context omitted.

> Can you make your on prem infrastructure go down less than Amazon's? Obviously depends on what you need, but for a small to medium web app that needs a load-balancer, a few app servers, a database and a cache, yes absolutely - all of these have been solved problems for over a decade and aren't rocket science to install & maintain. > Is it worth it? I'd argue that the "worth" would be less about immunity to occasion…

I have run high availability (HA) systems in prem and your statement vastly understates the difficulty and expense. You need multiple physical links in running to different ISPs because builders working on properties further down the street could accidentally cut through your fibre. Or the ISP themselves could suffer an outage. You need a back up generator and to be a short distance away from a petrol station so you…

> You need a back up generator and to be a short distance away from a petrol station so you can refuel quickly and regularly when suffering from longer durations of power outages.

I don't see why the petrol station needs to be a short distance away. Unless the plan is to walk to the petrol station and back (which should not be the plan[1]), anyplace within reasonable driving distance should do.

[1] long duration electrical outages will often take out everything a short distance away, and the petrol stations usually have electric pumps.

Re: Tell HN: AWS appears to be down again

#365

Post by manquer »

doublerabbit wrote: post #248
ryanbrunner wrote: post #193

Earlier quoted context omitted.

I think most people on here are coming from the perspective of startups, which scale out of a single server setup pretty quickly. At a bare minimum, most will have dedicated purpose-built servers like Redis or a DB, and often there's separate background workers, or a load balancer with a couple of web servers. When your server requirements get into needing 5-6 servers (not at all atypical for a startup in their first…

> running your own stuff becomes more of a challenge pretty quickly. Factor in 2-3x growth a year, and the challenges just mount. What challenges are you thinking of? You buy a full-rack in colocation and then just buy servers/hardware when required. If a company has the budget for AWS or some other cloud provider then they would have a budget for colocation; which in long term is cheaper. I see no additional challen…

Long term is unknown to the startup , they may fail or pivot .

Buying upfront hardware is not feasible even if I had the cash(which most don't), I don't know if the company would last that long or would be doing things that require x servers .

What you are saying is similar to saying may be it is cheaper to buy the building /floor instead of renting space for office. - most small biz cannot afford do that, or expect their business to change (fail/take off) in the time frame ROI would come to take that commitment.

This is all assuming that a the startup has skill in setting up and managing physical servers and there is no opportunity costs( delayed features) on doing so, both are not a given.

small companies ( and poor people) typically don't buy low quality stuff or buy into rent seeking business models because they are dumb it is usually because they cannot afford to do long term thinking.

Re: Tell HN: AWS appears to be down again

#366

Post by mbesto »

Spooky23 wrote: post #316
mbesto wrote: post #228

Earlier quoted context omitted.

> Not at this rate. Source? Has there ever been an industry wide survey that compares availability from "insert average colo/data center operations" with the cloud ones? And I'm not talking about "we have 12 SREs who are based in Cupertino and are all paid top dollar to support a colo"...I'm talking average .

Running a multi-tenant datacenter or hyper scale cloud datacenter is a different business than running a datacenter. The myth of the cost of running facilities on HN is insane - it’s like saying you can’t drive a car unless you hire a formula 1 driver. I worked through the ranks at a large enterprise that ran a “big” datacenter for a decade. The facilities team was about 6 people, average salary around $90k. I can on…

> The facilities team was about 6 people, average salary around $90k.

Ok so $540k salaries + benefits, so ~$700k. Then you have transaction costs:

- Annual salary increases

- Any cost associated with people leaving (severance, hiring, recruiters, HR, HR systems)

- Systems that run in the data center (logging, monitoring, etc.)

- Procurement costs with changing costs in hardware (silicon shortages, etc.)

- Security compliance overhead and associated risks

- Finance resources required to capitalize and manage asset allocation

- etc. etc.

Versus

- Click a button and voila it works.

- Hire way less engineers to manage the system administrative portion

> If the answer is automatically “AWS “, you’re making a decision to burn dollars for convenience.

100% AGREE. The answer is always "it depends", but just like people are saying "just put in the cloud", the opposite of "well it worked for us using a data center" isn't that simple.

Re: Tell HN: AWS appears to be down again

#367

Post by joana035 »

uvdn7 wrote: post #331
quantumfissure wrote: post #261

Me: Hesitation at last job moving absolutely everything (including backups) to AWS because if it goes down it's a problem I'm a firm believer in some kind of physical/easily accessible backup. Coworkers: "You're an f'n idiot. Amazon and Facebook don't go down, you're holding us back!" Me: leaves cause that treatment was the final straw Amazon and Facebook both go down within a month of each other, and supposedly they…

You could have just showed them historical data of both companies being unavailable for extended amount of time. What happened in the past few months is not new.

"just", as if you never had to argument against aws fanboys...

Re: Tell HN: AWS appears to be down again

#368

Post by laumars »

Nextgrid wrote: post #333
laumars wrote: post #330

Earlier quoted context omitted.

I have run high availability (HA) systems in prem and your statement vastly understates the difficulty and expense. You need multiple physical links in running to different ISPs because builders working on properties further down the street could accidentally cut through your fibre. Or the ISP themselves could suffer an outage. You need a back up generator and to be a short distance away from a petrol station so you…

That's true for on-prem infrastructure, but is all already handled for you if you rent servers from hosting providers such as OVH/Hetzner or even rent colocation space in an existing DC, and is still cheaper than the cloud equivalent (and as we saw recently, actually more reliable as well).

But then you’re still reliant on those hosting providers not fscking up; just like with cloud providers. Literally the same complaint the GP was making about AWS applies for OVH et al too.

In fact I used to run some hobby projects in OVH (as an aside, I really liked their services) so I’m aware that they have their own failures too.

Re: Tell HN: AWS appears to be down again

#369

Post by electroly »

kingsloi wrote: post #273

Of all the AWS outage, my team and I have dodged them all, except this one. 3 instances down and unavailable > Due to this degradation your instance could already be unreachable >:(

FWIW I don't think that message has anything to do with this outage. I think it's just a coincidence that you got some degraded hosts. They didn't send out emails like that for this AZ outage (nor would I expect them to -- that email is for when host machines die).

Re: Tell HN: AWS appears to be down again

#370

Post by ocdtrekkie »

lmilcin wrote: post #275
quantumfissure wrote: post #261

Me: Hesitation at last job moving absolutely everything (including backups) to AWS because if it goes down it's a problem I'm a firm believer in some kind of physical/easily accessible backup. Coworkers: "You're an f'n idiot. Amazon and Facebook don't go down, you're holding us back!" Me: leaves cause that treatment was the final straw Amazon and Facebook both go down within a month of each other, and supposedly they…

Think about it this way: 1) Can you make your on prem infrastructure go down less than Amazon's? 2) Is it worth it? In my experience most people grossly underestimate how expensive it is to create reliable infrastructure and at the same time overestimate how important it is for their services to run uninterrupted. -- EDIT: I am not arguing you shouldn't build your more reliable infrastructure. AWS is just a point on…

My on prem infrastructure goes down drastically less than Amazon's.

...My home Internet even is scoring better than Amazon right now, in fact. Yours probably is too.

Post reply on HN