Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tell HN: AWS appears to be down again

news.ycombinator.com

Post reply on HN
Print view View original

601–610 of 646 posts

Re: Tell HN: AWS appears to be down again

#601

Post by Talanes »

2-718-281-828 wrote: post #225
itsoktocry wrote: post #216

Earlier quoted context omitted.

> is that just playing with words? It conveys reality, that "fail-safe" isn't literal, as if anyone believed that.

I mean it has to be play with words or tongue in cheek simply b/c the assumption of a fail-safe system failing is already contradictory. So you cannot say anything smart about that beyond - there are no fail-safe systems that fail.

https://en.wikipedia.org/wiki/Gare_de_Lyon_rail_accident

Fail safes do fail. Often due to severe user error.

Re: Tell HN: AWS appears to be down again

#602

Post by throwaway81523 »

ffritz wrote: post #472
BiteCode_dev wrote: post #351

Earlier quoted context omitted.

1 - For the last 10 years, servers have been beasts. You have a lot of cores, plenty of HD and RAM. Servers are less expensives than devs. Scaling vertically can go VERY far. 2 - Caching is life. We have 3 layers of caching: cloudflare, varnish, and redis. Most things don't need to be real time. A lot of things can be a month old and the user doesn't care. User need immediate feedback to be happy, but not necessary f…

This was super interesting to read, thank you very much. Regarding the ffmpeg parameters and formats in general: Do you use newer formats too, like AV1 and the like?

The main One Weird Trick that I remember from messing with ffmpeg is that for in-browser viewing, I had to generate a lot more mp4 reference frames than the default. This was because of deficiency in the HTML5 view extensions in browers a few years ago. I don't know if that is still an issue. The extra frames increased bandwidth consumption, but surprisingly, only slightly rather than big bloat.

Re: Tell HN: AWS appears to be down again

#603

Post by noogle »

Spivak wrote: post #550
dijit wrote: post #531

Earlier quoted context omitted.

Then don’t host anything, don’t do software and don’t pretend to be “the future”.

This makes no sense. This has nothing to do with the tech and more to do with every team's natural push and pull with build over buy. It's completely pointless to respond to someone who didn't get their DoorDash order with "see this is why you should just make food at home." It completely ignores the reason someone chose to order takeout in the first place.

But if someone chose DoorDash over a smaller delivery service because "DoorDash is big and reliable so the x5-10 cost is worth it", and then DoorDash fails repeatedly, it makes sense to ask "is DoorDash really better than using the smaller vendor?"

Re: Tell HN: AWS appears to be down again

#604

Post by ClumsyPilot »

notyourday wrote: post #591
ClumsyPilot wrote: post #521

Earlier quoted context omitted.

And so is development time of any distributed software system, and training time required to operate it correctly

> And so is development time of any distributed software system, and training time required to operate it correctly Software is much easier than hardware. If you are to start a project today in this kind of hardware, you will be operating it in 2029, without changes.

"Software is much easier than hardware. If you are to start a project today in this kind of hardware, you will be operating it in 2029, without changes."

I don't think this makes sense, you are using the three statements "Software is cheaper", "Software takes less time" and "software is easier" as if they all mean the same thing, and proving one means proving all of them.

Hardware takes a long time, okay, that does not mean it's expensive. Building a hydroelectric dam takes 20 years, but it provides the cheapest source of electricity that ever existed. Ships can take a decade from order to delivery, they are the cheapest mode of transport.

Re: Tell HN: AWS appears to be down again

#605

Post by bcrosby95 »

acdha wrote: post #394
boudin wrote: post #357

Earlier quoted context omitted.

Issues are across all us-east 1, not one AZ. Load balancers are not doing well at all. The only way in this case to avoid an outage is to be cross regions or cross cloud which is quite more complex to handle and require more resources to do well. And I hope that nobody is listening your blaming and pointing fingers advice, that's the worst way to solve anything. It's AWS job to ensure that things are reliable, that t…

Some load balancers may be having issues but I have multiple busy workloads showing no issues all morning. One big challenge can be that some people reporting multi-AZ issues are shifting traffic and competing with everyone else, while workloads which were already running in the other AZs were fine. It can be really hard to accurately tell how much the problems you’re seeing generalize to everyone else. I do agree th…

But if AWS can't handle the shifting traffic in response to an AZ going down then at that point you're just gambling on whether or not you're in the lucky AZ.

Re: Tell HN: AWS appears to be down again

#606

Post by codeduck »

mentat wrote: post #599
codeduck wrote: post #378

Earlier quoted context omitted.

We've had alerts for packet loss and had issues in recovering region-spanning services (both AWS and 3rd party). Yes, some of these we should be better at handling ourselves, but... it's all very well to say "expect to lose an AZ" but during this outage it's not been physically possible to remove the broken AZ instances from multi-AZ services because we cannot physically get them to respond to or acknowledge commands…

You don't have health checks?

How are health checks supposed to help when you can't do anything?

Re: Tell HN: AWS appears to be down again

#607

Post by JshWright »

vinay_ys wrote: post #217
aledalgrande wrote: post #133

If you haven't seen yet, news is it was a power loss: > 5:01 AM PST We can confirm a loss of power within a single data center within a single Availability Zone (USE1-AZ4) in the US-EAST-1 Region. This is affecting availability and connectivity to EC2 instances that are part of the affected data center within the affected Availability Zone. We are also experiencing elevated RunInstance API error rates for launches wi…

This is quite interesting as they claim their datacenter design does better than Uptime's Tier3+ design requirements which require redundant power supply paths. [ https://aws.amazon.com/compliance/uptimeinstitute/ ]. I really hope they publish a thorough RCA for this incident.

> I really hope they publish a thorough RCA for this incident.

We're still waiting on the RCA for last week's us-west outage...

Re: Tell HN: AWS appears to be down again

#608

Post by ernsheong »

RONROC wrote: post #154

The prevailing wisdom throughout the last couple of years was: “ditch your on-prem infrastructure and migrate to a major cloud provider” And its starting to seem like it could be something like: “ditch your on-prem infrastructure and spin up your own managed cloud” This is probably untenable for larger orgs where convenience gets the blank check treatment, but for smaller operations that can’t realize that value at s…

Google Cloud seems to be doing much better, at least recently. There's also Azure. AWS seems to have placed growth above everything else at customers' expense.

Re: Tell HN: AWS appears to be down again

#609

Post by ramraj07 »

smiths1999 wrote: post #570
ramraj07 wrote: post #484

Earlier quoted context omitted.

You really chose to die on “backups are for old people” as a hill?

I'm not sure how you got "backups are for old people" from my post. My point is that there are two sides to this. Perhaps the data being stored on S3 data _was_ backup data and this engineer was proposing replicating the backup data to GCP. That's probably not the highest priority for most companies. Maybe the OP was right and the other engineers were wrong. Who knows. In my experience, the kind of person that argues…

I’ve most definitely been in numerous places where arrogant 25 year olds with CS degrees but not smart enough to make it to FAAnG think they know what they are talking about when they don’t. Not every 25yo is an idiot, but many especially in tech think they are smarter than they are because they’re paid these obscene amounts of money.

Re: Tell HN: AWS appears to be down again

#610

Post by manquer »

miked85 wrote: post #511
manquer wrote: post #411

Earlier quoted context omitted.

My point is if your redundancy is better than AWS then why pay for them ? If it not they why invest in your own?. You can argue that you protect against different threats than AWS does . So far I have not seen a meaningful argument of threats a on Prem protects differently than the cloud that you need both . Say for example your solution is to put all your data backups on the moon then it makes sense to do both, AWS…

> Even Amazon.com or Google apps host on their own cloud and not use multi cloud after all, their regular businesses are much bigger than their cloud biz This is probably true with Google, but AWS contributes > 50% of Amazon's operating income. [1] [1] https://www.techradar.com/news/aws-is-now-a-bigger-part-of-a...

Interesting, no wonder AWS head became Amazon CEO.

Their retail/e-commerce side is less profitable than AWS but the absolute revenue is still massive and the risk of losing that a chunk of that revenue(and income) due to tech issues is still enormous risk for Amazon .

Post reply on HN