Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tell HN: AWS appears to be down again

news.ycombinator.com

Post reply on HN
Print view View original

491–500 of 646 posts

Re: Tell HN: AWS appears to be down again

#491

Post by jrockway »

laumars wrote: post #330
Nextgrid wrote: post #285

Earlier quoted context omitted.

> Can you make your on prem infrastructure go down less than Amazon's? Obviously depends on what you need, but for a small to medium web app that needs a load-balancer, a few app servers, a database and a cache, yes absolutely - all of these have been solved problems for over a decade and aren't rocket science to install & maintain. > Is it worth it? I'd argue that the "worth" would be less about immunity to occasion…

I have run high availability (HA) systems in prem and your statement vastly understates the difficulty and expense. You need multiple physical links in running to different ISPs because builders working on properties further down the street could accidentally cut through your fibre. Or the ISP themselves could suffer an outage. You need a back up generator and to be a short distance away from a petrol station so you…

> You need multiple physical links in running to different ISPs because builders working on properties further down the street could accidentally cut through your fibre.

At my last job we provided redundant paths (including entry to your building) as an add-on service. So you might not need two ISPs if you're only worried about fiber cuts. You could still be worried about things like "we think all Juniper routers in the world will die at the exact same instant", in which case you need to make sure you pick an ISP that uses Cisco equipment. And of course, it's possible that your ISP pushes a bad route and breaks the entirety of their link to the rest of the Internet.

Re: Tell HN: AWS appears to be down again

#492

Post by JackFr »

chrisandchris wrote: post #457
manquer wrote: post #388

Earlier quoted context omitted.

Yes threat models is obvious qualifier, if you have a business that requires backup on the moon if there asteroid collision then by all means got for it.[1] For most companies what AWS.or Azure offers is more than adequate. An internal bad actor with that level of privileged access can delete your local backups or external one can all things you he can do to AWS he can likely do easier to your company storage DC too.…

Nope. 3-2-1 strategy. 3 Backups, 2 Medias, 1 Offsite. Now try to delete files from the media in my safe. Only I have a key. Sure, your threat model may vary. But relying on cloud only for your backup is simply not enough. If you split access for your AWS backup and your DC backup to two different people, you mitigated your thread model. If you only have 1 backup location, that's going to be very hard.

Not sure I agree about the usefulness of different media.

Having had to restore databases from tapes and removable drives for a compliance/legal incident, we had a failure rate of >50% on the tapes and about 33% for the removable drives.

I came away not trusting any backup that wasn’t on line.

Re: Tell HN: AWS appears to be down again

#493

Post by hinkley »

numbsafari wrote: post #402
quantumfissure wrote: post #261

Me: Hesitation at last job moving absolutely everything (including backups) to AWS because if it goes down it's a problem I'm a firm believer in some kind of physical/easily accessible backup. Coworkers: "You're an f'n idiot. Amazon and Facebook don't go down, you're holding us back!" Me: leaves cause that treatment was the final straw Amazon and Facebook both go down within a month of each other, and supposedly they…

Today's gentle reminder that there are things other than network or service outages that can and do occur that might necessitate an outside backup. What happens if AWS or [insert other megacloud] decides your account needs to be nuked from orbit due to a hack or some other confusion? We almost had this happen over the summer because of a problem with our bank's ability to process ACH payments. Very frustrating experi…

I would make a friendly wager that AWS user IDs don't contain check digits, let alone bullet proof ones (simple check digits don't guard against transposition errors). And that somewhere, someone can manually enter an account to delete, and that one of us will eventually have an account numbered XXX1234 and some idiot with account XXX1243 will legitimately earn an account deletion, but we'll be the ones who wake up to bad news.

Re: Tell HN: AWS appears to be down again

#494

Post by nightpool »

Nextgrid wrote: post #333
laumars wrote: post #330

Earlier quoted context omitted.

I have run high availability (HA) systems in prem and your statement vastly understates the difficulty and expense. You need multiple physical links in running to different ISPs because builders working on properties further down the street could accidentally cut through your fibre. Or the ISP themselves could suffer an outage. You need a back up generator and to be a short distance away from a petrol station so you…

That's true for on-prem infrastructure, but is all already handled for you if you rent servers from hosting providers such as OVH/Hetzner or even rent colocation space in an existing DC, and is still cheaper than the cloud equivalent (and as we saw recently, actually more reliable as well).

I've had way more networking and availability failures from Hetzner this year alone then I've ever seen from AWS. They regularly replace their networking switches without any redundancy, leaving entire DCs offline for hours. They're okay for hobby projects, but I would never host a business-critical site with them

Re: Tell HN: AWS appears to be down again

#495

Post by f6v »

Spooky23 wrote: post #316
mbesto wrote: post #228

Earlier quoted context omitted.

> Not at this rate. Source? Has there ever been an industry wide survey that compares availability from "insert average colo/data center operations" with the cloud ones? And I'm not talking about "we have 12 SREs who are based in Cupertino and are all paid top dollar to support a colo"...I'm talking average .

Running a multi-tenant datacenter or hyper scale cloud datacenter is a different business than running a datacenter. The myth of the cost of running facilities on HN is insane - it’s like saying you can’t drive a car unless you hire a formula 1 driver. I worked through the ranks at a large enterprise that ran a “big” datacenter for a decade. The facilities team was about 6 people, average salary around $90k. I can on…

That’s not what parent was asking about. The question is if a company of certain size is more likely to suffer from an outage on AWS compared to own hardware.

I’ve been deploying to AWS for years and can’t remember and outage on their side in my region. But this is anecdotal and doesn’t necessarily reflect the statistics.

Re: Tell HN: AWS appears to be down again

#496

Post by jmartrican »

thedougd wrote: post #432
jmartrican wrote: post #310

Earlier quoted context omitted.

Seems like multi-cloud solution might be the way to go.

I doubt it. The complexity of multi-cloud will also give you downtime. Most of the folks impacted by cloud outages do not have highly available systems in place. Perhaps, for their business, the cost doesn't justify the outcome. If you need high uptime for instances, build your system to be highly available and leverage the fault domain constructs your provider offers (placement groups, availability zones, regions, l…

Agreed it is hard for those reason you specified.

To do it, first I would not use any cloud features that cannot be easily setup in another cloud. So no lambdas. Just k8s clusters, maybe DBs if they can be setup to backup between clouds. I was able to migrate from AWS k8s to DO K8S very easily.... just pointed my k8s configs to the new cluster (plus configuring the DO load balancers).

In my case, I need the dynamic DNS (havnt looked into it yet), auto-scaling is already setup with k8s, and the DB backups between DBs (next project).

Re: Tell HN: AWS appears to be down again

#497

Post by laumars »

bcrosby95 wrote: post #490
laumars wrote: post #459

Earlier quoted context omitted.

Are they though? Let’s look at what the recent AWS outages have been: a single region (but AWS makes multi-region easy). The biggest impact to most people is the AWS console, something that one seldom actually needs given AWS is API driven. If the same type of outage happened on OVH then you’d lose KVM to your physical servers. But you seldom need those either. The Azure outage was just AD service but you can roll yo…

Most people using AWS aren't using multi-region, as evidenced by the wide array of problems on the internet when a region goes down. I would also argue many aren't even using multiple availability zones, as evidenced by the wide array of problems on the internet when a single AZ goes down. I think you're vastly over-estimating how most companies are using AWS, and are substituting your own requirements for theirs. Wh…

I fully believe some people are doing AWS wrong. But you cannot compare the worst offenders in AWS against the best defenders of deploying on prem - it’s just not a fair like for like comparison comparing the worst against the best.

Hence why I compare doing HA in AWS correctly vs doing HA on prem correctly.

Re: Tell HN: AWS appears to be down again

#498

Post by gime_tree_fiddy »

swid wrote: post #393
aeonflux wrote: post #311

Earlier quoted context omitted.

I once had to argue that we still do need backup even though S3 has redundancy. They laughed when I mentioned a possible lock-up from AWS (even due to a mistake or whatever). I asked what if we delete data from app by mistake? They told me we need to be careful not to do that. I guess I am getting more and more tired of arrogant 25 years old programmers with 1-2 years in industry and no experience.

One thing you should absolutely not count on, but might be a course of actions for large clients, is to contact support and ask them to restore accidentally / maliciously deleted files. I would never use this as part of the backup and restore plan; but I was lucky when a bunch of customer files were deleted due to a bug in a release. Something like 100k files were deleted from Google Storage without us having backup.…

This is not a reliable mechanism btw. There will be times when they won't be able to restore the data for you. Their product has options to avoid this situation like object versioning.

Re: Tell HN: AWS appears to be down again

#499

Post by exdsq »

smiths1999 wrote: post #466
aeonflux wrote: post #311

Earlier quoted context omitted.

I once had to argue that we still do need backup even though S3 has redundancy. They laughed when I mentioned a possible lock-up from AWS (even due to a mistake or whatever). I asked what if we delete data from app by mistake? They told me we need to be careful not to do that. I guess I am getting more and more tired of arrogant 25 years old programmers with 1-2 years in industry and no experience.

Maybe they are getting tired of arrogant older programmers assuming they cannot possibly be wrong. God forbid a 25 year old might actually have a good idea (and I am far removed from my 20s). Maybe having S3 redundancy wasn't the most important thing to be tackled? Does your company really need that complexity? Are you so big and such an important service that you cannot possibly risk going down or losing data?

I’d love to know what someone works on when the risk of losing data is not worth one or two days engineering work.
Post reply on HN