Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tell HN: AWS appears to be down again

news.ycombinator.com

Post reply on HN
Print view View original

411–420 of 646 posts

Re: Tell HN: AWS appears to be down again

#411

Post by manquer »

ncallaway wrote: post #372
manquer wrote: post #342

Earlier quoted context omitted.

S3 and (others) have version history that can be enabled. If you have to take care of availablity and redundancy and delete protection and backups then why pay the premium S3 is charging ? Either you don't trust the cloud and you can run NAS or equivalent (with s3 APIs easily today) much cheaper or trust them to keep your data safe and available. No point in investing in S3 and then doing it again yourself.

> No point in investing in S3 and then doing it again yourself. I mean that's just obviously wrong, though. There is a point. > Either you don't trust the cloud and you can run NAS or equivalent (with s3 APIs easily today) much cheaper or trust them to keep your data safe and available. What if you trust the cloud 90%, and you trust yourself 90%, and you think it's likely that the failure cases between the two are li…

My point is if your redundancy is better than AWS then why pay for them ? If it not they why invest in your own?.

You can argue that you protect against different threats than AWS does . So far I have not seen a meaningful argument of threats a on Prem protects differently than the cloud that you need both.

Say for example your solution is to put all your data backups on the moon then it makes sense to do both, AWS does not protect against threat to planet wide issues.

However if you are both protecting against exact same risks having just provider redundancy only protects against events like AWS goes down for days /months or goes bankrupt.

All business decisions have some risk , provider redundancy does not seem a risk to mitigate for the cost it would mean for most businesses I have seen.

Even Amazon.com or Google apps host on their own cloud and not use multi cloud after all, their regular businesses are much bigger than their cloud biz , they would still risk those to stick to their cloud/services only.

Re: Tell HN: AWS appears to be down again

#412

Post by snth »

captn3m0 wrote: post #32

4:35 AM PST We are investigating increased EC2 launched failures and networking connectivity issues for some instances in a single Availability Zone (USE1-AZ4) in the US-EAST-1 Region. Other Availability Zones within the US-EAST-1 Region are not affected by this issue. via https://stop.lying.cloud/

What is this website? Is there an "about" or something? What is it doing differently from the official AWS status page?

Re: Tell HN: AWS appears to be down again

#413

Post by tluyben2 »

dylan604 wrote: post #384
tluyben2 wrote: post #361

Earlier quoted context omitted.

> People commenting that servers shouldn’t go down ect don’t understand how true HA architectures work. You should expect and build for stuff to fail like this. Otherwise it’s like complaining that you lost data because a disk failed. Disks fail… build architecture where that won’t take you down. Is that comparison fair? If you have 2 raid-5 mirrored raid 5 boxes in your room and all disks fail at the same time, you…

>And that won't happen HA! I had received new 16-bay chasis and all of the drives needed plus cold spares for each chasis. Set them up and started the RAID-5 init on a Friday. Left them running in the rack over the weekend. Returned on Monday to find multiple drives in each chasis had failed. Even with dedicated one of the 16 drives as a hot swap, the volumes would all have failed in an unrecoverable manner. All driv…

Guess I was lucky, we ran a lot of these over the decades when things were far more unreliable than now and never experienced anything like that. Manufacturing issues, sure, but we always had everything we bought run on stress for 48 hours and see if that killed it, if it didn't, it didn't usually break anymore (I have many of the machines from mid to end 2000s still and they don't have diskfailures now while they ran for many years).

Re: Tell HN: AWS appears to be down again

#414

Post by AtNightWeCode »

tluyben2 wrote: post #361
JCM9 wrote: post #303

AWS didn’t “go down”. They had an outage in one AZ, which is why there are multiple AZs in each region. If your app went down then you should be blaming your developers on this one, not AWS. Those having issues are discovering gaps in their HA designs. Obviously it’s not good for an AZ to go down but it does happen and why any production workload should be architected to have seamless failover and recover to other AZ…

> People commenting that servers shouldn’t go down ect don’t understand how true HA architectures work. You should expect and build for stuff to fail like this. Otherwise it’s like complaining that you lost data because a disk failed. Disks fail… build architecture where that won’t take you down. Is that comparison fair? If you have 2 raid-5 mirrored raid 5 boxes in your room and all disks fail at the same time, you…

"Won't happen". The 40,000 hours of runtime bug did happen. I would recommend people to take backups and store them offline or at least isolated from the main storage.

Re: Tell HN: AWS appears to be down again

#415

Post by [deleted] »

dragonwriter wrote: post #347
Nextgrid wrote: post #322

Earlier quoted context omitted.

> Human capital side would disagree with that I think I hear this argument a lot, but every startup I've been involved with had a full-time DevOps engineer wrangling Terraform & YAML files - that same engineer can be assigned to manage the bare-metal infrastructure.

> I hear this argument a lot, but every startup I've been involved with had a full-time DevOps engineer wrangling Terraform & YAML files - that same engineer can be assigned to manage the bare-metal infrastructure. Bare metal infrastructure requires a lot more management at any given scale. I mean, you can run stuff that lets you do part of the management the same as cloud resources, but you also have to then manage…

[deleted]

Re: Tell HN: AWS appears to be down again

#416

Post by Tempest1981 »

dylan604 wrote: post #384
tluyben2 wrote: post #361

Earlier quoted context omitted.

> People commenting that servers shouldn’t go down ect don’t understand how true HA architectures work. You should expect and build for stuff to fail like this. Otherwise it’s like complaining that you lost data because a disk failed. Disks fail… build architecture where that won’t take you down. Is that comparison fair? If you have 2 raid-5 mirrored raid 5 boxes in your room and all disks fail at the same time, you…

>And that won't happen HA! I had received new 16-bay chasis and all of the drives needed plus cold spares for each chasis. Set them up and started the RAID-5 init on a Friday. Left them running in the rack over the weekend. Returned on Monday to find multiple drives in each chasis had failed. Even with dedicated one of the 16 drives as a hot swap, the volumes would all have failed in an unrecoverable manner. All driv…

Would like to know the manufacturer and model.

Re: Tell HN: AWS appears to be down again

#417

Post by jeremyjh »

manquer wrote: post #342
aeonflux wrote: post #311

Earlier quoted context omitted.

I once had to argue that we still do need backup even though S3 has redundancy. They laughed when I mentioned a possible lock-up from AWS (even due to a mistake or whatever). I asked what if we delete data from app by mistake? They told me we need to be careful not to do that. I guess I am getting more and more tired of arrogant 25 years old programmers with 1-2 years in industry and no experience.

S3 and (others) have version history that can be enabled. If you have to take care of availablity and redundancy and delete protection and backups then why pay the premium S3 is charging ? Either you don't trust the cloud and you can run NAS or equivalent (with s3 APIs easily today) much cheaper or trust them to keep your data safe and available. No point in investing in S3 and then doing it again yourself.

There are completely independent risks that you are dealing with here. If you are a small company there is a non-insignificant risk that your cloud account will be closed and it will be impossible to find out why or to fix it in a timely matter. There have been several that were only fixed after being escalated to the front page of Hacker News, and we haven't heard about the ones that didn't get enough upvotes to get our attention and were never fixed.

Also, what we saw on Dec 7th was that the complexity of Amazon's infrastructure introduces risks of downtime that simply cannot be fully mitigated by Amazon, or by any other single provider. More redundancy introduces more complexity at both the micro level and macro level.

It doesn't really cost that much to at least store replicated data in an independent cloud, particularly a low-cost one like Digital Ocean.

Re: Tell HN: AWS appears to be down again

#418

Post by Bluecobra »

Godel_unicode wrote: post #335
Bluecobra wrote: post #238

Earlier quoted context omitted.

> What am I missing? My guess is that they cheaped out in having redundant PSUs to get you to use multiple availability zones. (More zones = more revenue) Even a single PSU shouldn’t be an issue if they plugged in an ATS switch though.

Unless the ATS breaks, which happens.

For sure, in my context I meant a ATS in single rack/cabinet. If that went bad the blast radius would be contained to a single cabinet. But yeah, anything can and will happen. At another place I worked at, a site UPS took down an entire server room. It was pretty nice Eaton system but there was some event that fried the whole thing. Eaton had to send an specialist to investigate the matter as those events are pretty rare.

Re: Tell HN: AWS appears to be down again

#419

Post by Spooky23 »

mbesto wrote: post #366
Spooky23 wrote: post #316

Earlier quoted context omitted.

Running a multi-tenant datacenter or hyper scale cloud datacenter is a different business than running a datacenter. The myth of the cost of running facilities on HN is insane - it’s like saying you can’t drive a car unless you hire a formula 1 driver. I worked through the ranks at a large enterprise that ran a “big” datacenter for a decade. The facilities team was about 6 people, average salary around $90k. I can on…

> The facilities team was about 6 people, average salary around $90k. Ok so $540k salaries + benefits, so ~$700k. Then you have transaction costs: - Annual salary increases - Any cost associated with people leaving (severance, hiring, recruiters, HR, HR systems) - Systems that run in the data center (logging, monitoring, etc.) - Procurement costs with changing costs in hardware (silicon shortages, etc.) - Security co…

Say all of those costs are $2,000,000, and you have 25,000 billable endpoints in the datacenter… you’re looking at less than $0.01/hour for that overhead on a unit basis.

Obviously, there’s a huge capital investment component too that has to be incorporated. Those costs may be really high if you’re in a growth phase as you need to overbuy capacity.

Just to be clear, I’m not arguing that on-prem is magically cheap. :) But it has its place too!

Re: Tell HN: AWS appears to be down again

#420

Post by tluyben2 »

AtNightWeCode wrote: post #414
tluyben2 wrote: post #361

Earlier quoted context omitted.

> People commenting that servers shouldn’t go down ect don’t understand how true HA architectures work. You should expect and build for stuff to fail like this. Otherwise it’s like complaining that you lost data because a disk failed. Disks fail… build architecture where that won’t take you down. Is that comparison fair? If you have 2 raid-5 mirrored raid 5 boxes in your room and all disks fail at the same time, you…

"Won't happen". The 40,000 hours of runtime bug did happen. I would recommend people to take backups and store them offline or at least isolated from the main storage.

Sure, I plan for it but I do not expect it. And it never happened for me over decades. But I did plan for it, just not in the way the parent said and it did cost me far less.
Post reply on HN