Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tell HN: AWS appears to be down again

news.ycombinator.com

Post reply on HN
Print view View original

461–470 of 646 posts

Re: Tell HN: AWS appears to be down again

#461

Post by numbsafari »

thraxil wrote: post #441
numbsafari wrote: post #406

Earlier quoted context omitted.

S3 isn't perfect. Read the fine print. I've had buckets and objects disappear into the ether. It is exceedingly rare, but it's not impossible. Offline/alt-cloud backups are probably a lot cheaper than you think, and will win you points during any audit.

> Offline/alt-cloud backups are probably a lot cheaper than you think, and will win you points during any audit. With the caveat that you're going to have to implement all your access controls, monitoring and compliance mechanisms on those alternate backups. No point winning points during an audit for having backups outside AWS if you lose even more points for "backups weren't properly secured against unauthorized ac…

Well, obviously, it goes without saying.

But none of that changes the fact that you shouldn't put all your eggs in one basket.

Re: Tell HN: AWS appears to be down again

#462

Post by silon42 »

aeonflux wrote: post #311
kalleth wrote: post #277

Earlier quoted context omitted.

I'd be surprised if they needed backups for a few hours of downtime with (reportedly) complete recovery where no data was corrupted. There are industries where this would be required, and it's possible I guess, but neither of these downtime events were "data loss" events, just availability events for short-ish periods of time that wouldn't - for me - result in activating our DR plans. I must admit that I do always tr…

I once had to argue that we still do need backup even though S3 has redundancy. They laughed when I mentioned a possible lock-up from AWS (even due to a mistake or whatever). I asked what if we delete data from app by mistake? They told me we need to be careful not to do that. I guess I am getting more and more tired of arrogant 25 years old programmers with 1-2 years in industry and no experience.

Next time also mention that it might be a problem to get a consistent back of microservices...

Re: Tell HN: AWS appears to be down again

#463

Post by strunz »

JCM9 wrote: post #303

AWS didn’t “go down”. They had an outage in one AZ, which is why there are multiple AZs in each region. If your app went down then you should be blaming your developers on this one, not AWS. Those having issues are discovering gaps in their HA designs. Obviously it’s not good for an AZ to go down but it does happen and why any production workload should be architected to have seamless failover and recover to other AZ…

Except many AWS services still route through us-east-1 anyway, which is why they have had huge outages recently. AWS isn't as redundant as people think it is.

Re: Tell HN: AWS appears to be down again

#464

Post by manquer »

RA_Fisher wrote: post #448
ncallaway wrote: post #372

Earlier quoted context omitted.

> No point in investing in S3 and then doing it again yourself. I mean that's just obviously wrong, though. There is a point. > Either you don't trust the cloud and you can run NAS or equivalent (with s3 APIs easily today) much cheaper or trust them to keep your data safe and available. What if you trust the cloud 90%, and you trust yourself 90%, and you think it's likely that the failure cases between the two are li…

True with two independent servers at 90% each, that’s 0.1^2 = 1% chance both fail— so redundancy can add a lot of reliability.

Only if they are truly independent of each other.

You and AWS are using similar chips similar hard disks even with similar failure rates.

If you both use same hardware from say batch both can defects and fail at similar times.or you use the same file systems, that say corrupts both your backups.

90% is not a magic number , you need to know AWS supply chains and practices thoroughly and keep yours different enough not to have same risks as AWS does for your system to have independent probability of failures.

Re: Tell HN: AWS appears to be down again

#465

Post by laumars »

vinay_ys wrote: post #454
laumars wrote: post #377

Earlier quoted context omitted.

Because there are laws on what you can and cannot fill with fuel. So you may find you have to make smaller but more frequent visits. Also buying fuel for a petrol station is going to be more expensive than having a commercial tanker refill it. So ideally you wouldn’t be making large top ups from the local petrol station except under exceptional outages. As for wider power outages affecting the fuel pumps, I suspect t…

A typical multi-MW power-hungry high-tech facility (datacenters, manufacturing, hospitals etc) will have large underground fuel storage tanks big enough to run the full load on generators for couple of days and they are continuously kept refilled via fuel tanker trucks through contracts with bulk fuel distributors. They usually have an SLA of a 40KL tanker in 4 hour notice. In case of advance warning of heavy-rains/f…

indeed but that wasn’t the type of facilities that the GP was talking about when they said running web services were a solved problem.

If you do move to an established data centre then you’re back to my earlier point that you’re still then dependant on their services instead of having ownership to fix all the problems yourself (which was the original argument the GP made in favour of switching away from the cloud).

Re: Tell HN: AWS appears to be down again

#466

Post by smiths1999 »

aeonflux wrote: post #311
kalleth wrote: post #277

Earlier quoted context omitted.

I'd be surprised if they needed backups for a few hours of downtime with (reportedly) complete recovery where no data was corrupted. There are industries where this would be required, and it's possible I guess, but neither of these downtime events were "data loss" events, just availability events for short-ish periods of time that wouldn't - for me - result in activating our DR plans. I must admit that I do always tr…

I once had to argue that we still do need backup even though S3 has redundancy. They laughed when I mentioned a possible lock-up from AWS (even due to a mistake or whatever). I asked what if we delete data from app by mistake? They told me we need to be careful not to do that. I guess I am getting more and more tired of arrogant 25 years old programmers with 1-2 years in industry and no experience.

Maybe they are getting tired of arrogant older programmers assuming they cannot possibly be wrong. God forbid a 25 year old might actually have a good idea (and I am far removed from my 20s).

Maybe having S3 redundancy wasn't the most important thing to be tackled? Does your company really need that complexity? Are you so big and such an important service that you cannot possibly risk going down or losing data?

Re: Tell HN: AWS appears to be down again

#469

Post by sgarland »

jsmith99 wrote: post #340
NovemberWhiskey wrote: post #314

Earlier quoted context omitted.

I think it's predicated on a misunderstanding of what "fail-safe" actually means. For example, in railway signaling, drivers are trained to interpret a signal with no light as the most restrictive aspect (e.g. "danger"). That way, any failure of a bulb in a colored light signal, or a failure of the signal as a whole, results in a safe outcome (albeit that the train might be delayed while the driver calls up the signa…

Or nuclear reactors that fail safe by dropping all the control rods into the core to stop all activity. The reactor may be permanently ruined after that (with a cost of hundreds of millions or billions to revert) but there will be no risk of meltdown.

Sort of. A failsafe reactor design [can] include[s] things like:

* Negative temperature coefficient of reactivity: as temperature increases, the neutron flux is reduced, which both makes it more controllable, and tends to prevent runaway reactions.

* Negative void coefficient of reactivity: as voids (steam pockets) increase, the neutron is reduced.

* Control rods constructed solely of neutron adsorbent. The RBMK reactor (Chernobyl) in particular used graphite followers (tips), which _increased_ reactivity initially when being lowered.

It's also worth noting that nuclear reactors are designed to be operated within certain limits. The RBMK reactor would have been fine had it been operated as designed.

Source: was a nuclear reactor operator on a submarine.

Re: Tell HN: AWS appears to be down again

#470

Post by laumars »

vinay_ys wrote: post #424
laumars wrote: post #330

Earlier quoted context omitted.

I have run high availability (HA) systems in prem and your statement vastly understates the difficulty and expense. You need multiple physical links in running to different ISPs because builders working on properties further down the street could accidentally cut through your fibre. Or the ISP themselves could suffer an outage. You need a back up generator and to be a short distance away from a petrol station so you…

It is much easier than you think. There are well-defined standards and trained trades people and whole host of companies who make great products and provide after-sales services to do it. Every major financial services, telecom and high-precision manufacturing companies run their infra this way. It is definitely less niche than rolling your own encryption.

My analogy wasn’t saying it’s niche. It was comparing the difficulty. And yes, there are trained people (I’m one of them ;) ) but that doesn’t make it easy, cheap, nor even less error prone than using cloud services. Which was my point.

Also the reasons those companies usually run their own infra is historically down to legislation more than preference. At least that’s been the case with almost all of the companies I’ve built on prem HA systems for.

Post reply on HN