Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tell HN: AWS appears to be down again

news.ycombinator.com

Post reply on HN
Print view View original

421–430 of 497 posts

Re: Tell HN: AWS appears to be down again

#421

Post by jorblumesea »

tomlagier wrote: post #347

I wonder if AWS will make more or less money from these outages? Will large players flee because of excessive instability? Or will smaller players go from single-AZ to more expensive multi-AZ? My guess is that no-one will leave and lots of single-AZ tenants who should be multi-AZ will use this as the impetus to do it. Honestly, having events like this is probably good for the overall resilience of distributed systems…

No one just "moves off" AWS. Once your apps are spaghetti coded with lambdas, buckets and all sorts of stuff, it's basically impossible to get off. More than likely, as you noticed, it will increase spending since multi-AZ/multi-region will become the norm.

Re: Tell HN: AWS appears to be down again

#423

Post by Nextgrid »

owlbynight wrote: post #402
api wrote: post #242

Earlier quoted context omitted.

... because you need to contact a server 1000 miles away to issue badges in your building. This cloud-for-everything-even-local-devices thing is both hilarious and sad. I wonder if anyone had trouble doing their dishes or laundry today, because I'm sure someone thought dish washers and washing machines needed cloud.

Yes, everyone but you is wrong. Many logical people have decided to abstract away their soul-crushing anxieties and legal gray area during outages to incredibly stable and well-staffed cloud infrastructure providers. If you and your team are better at taking care of hardware than an entire building full of highly paid engineering specialists, then that's cool for you, but also, no you're not. That's not to say you're…

> but also, no you're not.

If you plan to replicate all of AWS I'd agree with you. But if all you need is a handful of servers, you could end up with better uptime doing it in-house just because you don't have all the moving parts that make AWS tick, reducing the chance for something to go wrong.

My bare-metal servers stayed up during both of the recent outages, not because I'm some kind of genius that's better than the AWS engineers but just because it's a dead simple stack that has zero moving parts and my project doesn't require anything more complex.

Re: Tell HN: AWS appears to be down again

#424

Post by orourkek »

reaperducer wrote: post #368
ec109685 wrote: post #285

Earlier quoted context omitted.

I don't know if you can say an on-premise badge hosting service would be more reliable than the cloud.

Taking badges out of the cloud reduces points of failure by several orders of magnitude. Cloud-based badges make sense if you have locations with small staffs and no HR people or managers. Like if you're controlling access to a microwave tower on the top of a mountain. But badges-in-the-cloud for an office building full of people who are being supervised by supposedly trusted managers, and all of whom has been vetted…

> Taking badges out of the cloud reduces points of failure by several orders of magnitude.

I'm not convinced that's true, or at least certainly not an order of magnitude. Wouldn't a badge system hosted on-prem also need a user management system (database), a hosted management interface, have a dependency on the LAN, and need most of the same hardware? Such a system would also need to be running on a local server(s), which introduces points of failure around power continuity/surges, physical security, ongoing maintenance, etc.

Re: Tell HN: AWS appears to be down again

#425

Post by user3939382 »

owlbynight wrote: post #402
api wrote: post #242

Earlier quoted context omitted.

... because you need to contact a server 1000 miles away to issue badges in your building. This cloud-for-everything-even-local-devices thing is both hilarious and sad. I wonder if anyone had trouble doing their dishes or laundry today, because I'm sure someone thought dish washers and washing machines needed cloud.

Yes, everyone but you is wrong. Many logical people have decided to abstract away their soul-crushing anxieties and legal gray area during outages to incredibly stable and well-staffed cloud infrastructure providers. If you and your team are better at taking care of hardware than an entire building full of highly paid engineering specialists, then that's cool for you, but also, no you're not. That's not to say you're…

> If you and your team are better at taking care of hardware than an entire building full of highly paid engineering specialists

The trade offs aren't quite that simple. Those specialists are necessary because they're building and maintaining infrastructure that's extremely complex since it has a crazy scale and has to be all things to all people. When you're running in-house, your infrastructure is simpler because it's custom tailored to your specific requirements and scale.

There are tradeoffs that make cloud vs local make sense in different contexts and there's no one right answer.

Re: Tell HN: AWS appears to be down again

#426

Post by darepublic »

chasd00 wrote: post #237
tyingq wrote: post #85

Earlier quoted context omitted.

They did add an update, faster than last time: "7:42 AM PST We are investigating Internet connectivity issues to the US-WEST-2 Region." https://status.aws.amazon.com/ Edit: They added US-WEST-1: "7:52 AM PST We are investigating Internet connectivity issues to the US-WEST-1 Region." Edit: Found root case, maybe? "8:01 AM PST We have identified the root cause of the Internet connectivity to the US-WEST-1 Region and ha…

someone tripped over the fiber run i bet. Or, a cleaning person unplugged a router to plugin a vacuum (that actually happened but to a minicomputer iirc)

Unfortunately the vacuum, a shiny IoT connected appliance, didn't work because AWS was down

Re: Tell HN: AWS appears to be down again

#427

Post by Nexxxeh »

CoastalCoder wrote: post #281

Asking as a non-cloud-developer: why would Crunchyroll's recovery [0] lag so much behind AWS's recovery [1]? [0] https://downdetector.com/status/crunchyroll/ [1] https://downdetector.com/status/aws-amazon-web-services/

Crunchyroll seems to barely work at the best of times, and when it does, it's still a mess.

All sorts of issues still unresolved for years, including the ridiculously annoying "Finishes playing season English sub, autoplays first season of German dub, which then gets stuck". Still no profiles (nerfing their super-premium offering). Auto-resume points are unreliable, the Android app is hot garbage at dealing with network disruption...

I can only imagine their back-end is mostly Visual Basic running on a single AWS-powered VM.

Re: Tell HN: AWS appears to be down again

#428

Post by Nextgrid »

yawnxyz wrote: post #197
qeternity wrote: post #137

Earlier quoted context omitted.

> I'm curious — what does Hacker News run on? It seems impervious to any kind of downtime... On a dirty, disgusting dedicated server.

> On a dirty, disgusting dedicated server. I'm adding "reliable" into that mix. Too bad they're too expensive and hard to setup for side projects, but HN is probably one of the most stable site I frequently visit, and I don't even think about it.

I disagree that they're expensive. Expensive to own maybe, but you can rent them on a monthly basis from something like Hetzner or OVH for a fraction of the cost of AWS (especially when you include bandwidth which is free and unmetered in this case) and they handle hardware maintenance for you.

Hard to setup is relative. It all depends on what you're doing and how much reliability you need. For a side project or a dev server you can just start with Debian, stick to packaged software (most language runtimes and services such as Postgres or Redis are available) as much as possible and call it a day. You can even enable auto-updates on such a stable distro.

The knowledge you'll gain by dealing with bare-metal is also going to be useful in the cloud even in container environments.

Post reply on HN