Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tell HN: AWS appears to be down again

news.ycombinator.com

Post reply on HN
Print view View original

451–460 of 646 posts

Re: Tell HN: AWS appears to be down again

#451

Post by tluyben2 »

dylan604 wrote: post #433
Tempest1981 wrote: post #416

Earlier quoted context omitted.

Would like to know the manufacturer and model.

Sorry, this was back in 2006-2007 time frame. I have no idea on model numbers as that's just not information I ever cared to commit to memory.

My time frame is about since end 90s to now. I saw more failures in general in the olden days. I have quite a lot of rented servers at traditional hosters or Colo that have not been down outside kernel security updates for 10+ years. No hardware issues. I am now swapping them out for new servers which cost less for more, but hardware wise they have been pounder hard for 10 years without issues. All hotswap raid5 drives, so when broken, they or us just fixed it without downtime.

Re: Tell HN: AWS appears to be down again

#452

Post by pkulak »

darkstar999 wrote: post #446
pkulak wrote: post #425

I used to think it was silly to have your own hardware (like a NAS) in your house. What makes you think you can do it better than AWS? Santa is bringing me a Synology in three days.

Why not both? I just got a Synology NAS and it makes cloud sync dead simple. Now the most important things are on my PC, mirrored on 2 drives in my NAS, and on AWS S3 (or any other cloud storage).

Oh yeah. My plan is to migrate everything to the NAS, then have that back up to Glacier and/or Rsync.net. By S3, do you mean Glacier?

Re: Tell HN: AWS appears to be down again

#453

Post by xwdv »

quantumfissure wrote: post #261

Me: Hesitation at last job moving absolutely everything (including backups) to AWS because if it goes down it's a problem I'm a firm believer in some kind of physical/easily accessible backup. Coworkers: "You're an f'n idiot. Amazon and Facebook don't go down, you're holding us back!" Me: leaves cause that treatment was the final straw Amazon and Facebook both go down within a month of each other, and supposedly they…

You’re still in the wrong, don’t be so smug. These few downtimes are no big deal in the grand scheme of things, and your proposed solution would have been more work and headaches for little to no realizable gains, and not to mention the cybersecurity ramifications. Quite frankly, they are probably glad that you’re gone and not around to gloat about every trivial bit of downtime.

Re: Tell HN: AWS appears to be down again

#454

Post by vinay_ys »

laumars wrote: post #377
tzs wrote: post #364

Earlier quoted context omitted.

> You need a back up generator and to be a short distance away from a petrol station so you can refuel quickly and regularly when suffering from longer durations of power outages. I don't see why the petrol station needs to be a short distance away. Unless the plan is to walk to the petrol station and back (which should not be the plan[1]), anyplace within reasonable driving distance should do. [1] long duration elec…

Because there are laws on what you can and cannot fill with fuel. So you may find you have to make smaller but more frequent visits. Also buying fuel for a petrol station is going to be more expensive than having a commercial tanker refill it. So ideally you wouldn’t be making large top ups from the local petrol station except under exceptional outages. As for wider power outages affecting the fuel pumps, I suspect t…

A typical multi-MW power-hungry high-tech facility (datacenters, manufacturing, hospitals etc) will have large underground fuel storage tanks big enough to run the full load on generators for couple of days and they are continuously kept refilled via fuel tanker trucks through contracts with bulk fuel distributors. They usually have an SLA of a 40KL tanker in 4 hour notice. In case of advance warning of heavy-rains/floods or other natural disasters that can disrupt road networks, they can have more fuel trucks situated close-by as stand-by. Depending on your contract, you may have priority over other customers in the area. These are fairly standard practices.

Re: Tell HN: AWS appears to be down again

#455

Post by manquer »

lambic wrote: post #422
manquer wrote: post #388

Earlier quoted context omitted.

Yes threat models is obvious qualifier, if you have a business that requires backup on the moon if there asteroid collision then by all means got for it.[1] For most companies what AWS.or Azure offers is more than adequate. An internal bad actor with that level of privileged access can delete your local backups or external one can all things you he can do to AWS he can likely do easier to your company storage DC too.…

We have AWS backups, "offsite" backups on another cloud provider, and air-gapped backups in a disconnected hard drive in a safe. The extra expense outlay for the 2 additional backups is approximately $50/month, so it's not going to break the bank.

Egress from aws is not cheap.

At $50/month scale a lot of things are possible. Most companies cannot store their data in a hard disk in a safe. If you can, then cloud is a convenience not a necessity for you. I.e. you are perfectly fine running your storage stack for the most part.

My company is not very big(100ish employees) and we pay $200k+ for AWS in just storage and AWS is not even out primary cloud. If we have to do what you have, it is probably in bandwidth costs alone another $500k. Add running costs in another cloud and recurring bandwidth for transfers , retrieval from Glacier for older data on top of that.[1]

Over 3 years that would be easily $1-$1.5 million in net new expenses for us scale.

No sane business is going to sign off on +3x storage costs on a risk that cannot be easily modeled[2] and costs that cannot be priced into the product, just so one sysadmin can sleep better at night.

[1]your hard disk in a safe third component is not sensible discussion point at reasonable scale.

[2] this would be probability of data loss with AWS * business cost of losing that data > cost of secondary system.

Or probability of data availablity event(like now) * business cost of that > cost of an active secondary system .

For almost no business in the world the either equation would be valid.

For example even the cost is 100B dollars in revenue with 6 nines of durability the expected loss would be only $10,000 (100B * 0.000001) a secondary system is much costlier than that.

Re: Tell HN: AWS appears to be down again

#456

Post by dookahku »

quantumfissure wrote: post #261

Me: Hesitation at last job moving absolutely everything (including backups) to AWS because if it goes down it's a problem I'm a firm believer in some kind of physical/easily accessible backup. Coworkers: "You're an f'n idiot. Amazon and Facebook don't go down, you're holding us back!" Me: leaves cause that treatment was the final straw Amazon and Facebook both go down within a month of each other, and supposedly they…

Send Your former colleagues a group email asking how it is

Re: Tell HN: AWS appears to be down again

#457

Post by chrisandchris »

manquer wrote: post #388
kalleth wrote: post #375

Earlier quoted context omitted.

In most startups? You're mostly correct. But you still have some risks here, yes, with a super low probability, but a company-killing impact. In some industries - banking, finance, anything regulated, or really (I'd argue) anywhere where losing all of your data is company killing - you will need a disaster recovery strategy in place. The risks requiring non-AWS backups are things like: - A failed payment goes unnotic…

Yes threat models is obvious qualifier, if you have a business that requires backup on the moon if there asteroid collision then by all means got for it.[1] For most companies what AWS.or Azure offers is more than adequate. An internal bad actor with that level of privileged access can delete your local backups or external one can all things you he can do to AWS he can likely do easier to your company storage DC too.…

Nope. 3-2-1 strategy. 3 Backups, 2 Medias, 1 Offsite. Now try to delete files from the media in my safe. Only I have a key.

Sure, your threat model may vary. But relying on cloud only for your backup is simply not enough. If you split access for your AWS backup and your DC backup to two different people, you mitigated your thread model. If you only have 1 backup location, that's going to be very hard.

Re: Tell HN: AWS appears to be down again

#458

Post by [deleted] »

mnordhoff wrote: post #232
benedikt wrote: post #189

Earlier quoted context omitted.

you can find out which zone is mapped to use1-az4 for your account with awscli: aws ec2 describe-availability-zones | jq -r '.AvailabilityZones[] | select(.ZoneId == "use1-az4") | .ZoneName'

Or if you open the EC2 console (it's up this time!) and scroll down to the bottom. https://console.aws.amazon.com/ec2/v2/home?region=us-east-1#... : (Edit: I hope I didn't sound sarcastic. I don't open random console pages and scroll all the way down to check for new features. Some people will have noticed, some won't.)

[deleted]

Re: Tell HN: AWS appears to be down again

#459

Post by laumars »

Nextgrid wrote: post #426
laumars wrote: post #368

Earlier quoted context omitted.

But then you’re still reliant on those hosting providers not fscking up; just like with cloud providers. Literally the same complaint the GP was making about AWS applies for OVH et al too. In fact I used to run some hobby projects in OVH (as an aside, I really liked their services) so I’m aware that they have their own failures too.

Old-school hosting providers have a lot less moving parts than cloud providers. They have their outages, but they’re usually less frequent.

Are they though? Let’s look at what the recent AWS outages have been: a single region (but AWS makes multi-region easy). The biggest impact to most people is the AWS console, something that one seldom actually needs given AWS is API driven. If the same type of outage happened on OVH then you’d lose KVM to your physical servers. But you seldom need those either.

The Azure outage was just AD service but you can roll your own there if you wanted.

Plus if you want to talk about SaaS then OVH et al have their own SaaS too. In fact the difference between OVH and AWS is more about scale than it is about reliability (with AWS you can buy hardware and rack it in AWS just like with OVH too).

Or maybe by “old skool” you mean the few independent hosts that don’t offer SaaS. However they’re usually pretty small fry and this outages are less likely to be reported. Whereas any AWS service going down is massive news.

I’m not a cloud-fanboy by any means (I actually find AWS the least enjoyable to manage from a purely superficial perspective) but I’ve worked across a number of different hosting providers as well as building out HA systems on prem and the anti-cloud sentiment here really misses the pragmatic reality of things.

Re: Tell HN: AWS appears to be down again

#460

Post by cameronh90 »

kalleth wrote: post #375
manquer wrote: post #342

Earlier quoted context omitted.

S3 and (others) have version history that can be enabled. If you have to take care of availablity and redundancy and delete protection and backups then why pay the premium S3 is charging ? Either you don't trust the cloud and you can run NAS or equivalent (with s3 APIs easily today) much cheaper or trust them to keep your data safe and available. No point in investing in S3 and then doing it again yourself.

In most startups? You're mostly correct. But you still have some risks here, yes, with a super low probability, but a company-killing impact. In some industries - banking, finance, anything regulated, or really (I'd argue) anywhere where losing all of your data is company killing - you will need a disaster recovery strategy in place. The risks requiring non-AWS backups are things like: - A failed payment goes unnotic…

Just to add that S3 supports a compliance object lock that can't even be overridden by the root user. Also AWS doesn't delete your account or data until 90 days after the account is closed.

Between these two protections, it's pretty hard to lose data from S3 if you really want to keep it. I would guess they are better protections than you could achieve in your own self managed DC.

I'm guessing AWS has some clause in their contract that means they can refuse to deal with you or even return any of your data if they feel like it. Not sure if that's ever happened, but still worth considering it.

Post reply on HN