Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS down again?

aws.amazon.com

Post reply on HN
Print view View original

151–160 of 257 posts

Re: AWS down again?

#151

Post by jboggan »

clavicat wrote: post #35

Anyone feel like Big Tech outages are happening more frequently recently? In recent months, we’ve seen Amazon Web Services, Facebook, Gmail, and Twitter go down. Are we at the point where the people who maintain the infrastructure are now completely different than the ones who built it, and are struggling to keep it running because they don’t understand it as well?

As the layers of the stack increase in depth the probability that any one piece breaks the system approaches one. There will be a Singularity but it will be a horizontal asymptote instead of a vertical one.

Re: AWS down again?

#152

Post by reidfromhome »

[deleted] wrote: post #125
koprulusector wrote: post #84

I expect better of the community here. All it takes is a chance to take a cheap shot at one of the “big boys” and then all of a sudden the weasels come scampering out of the wood work. Seriously, those commenting “oh boy! Time to rethink this whole cloud thing!” You’re either so new to this stuff to have no experience to remember the days before cloud, you’re trolling because you’re high on nostalgia remembering the…

[deleted]

There never was an outage. It was just the marketing home page for aws.amazon.com that was down for about 30 minutes and everyone had a knee jerk reaction to “haha AWS down went down again!!” Everything that mattered works fine.

Re: AWS down again?

#153

Post by ksec »

technobabbler wrote: post #104
koprulusector wrote: post #84

I expect better of the community here. All it takes is a chance to take a cheap shot at one of the “big boys” and then all of a sudden the weasels come scampering out of the wood work. Seriously, those commenting “oh boy! Time to rethink this whole cloud thing!” You’re either so new to this stuff to have no experience to remember the days before cloud, you’re trolling because you’re high on nostalgia remembering the…

Y'know, debates about cloud vs in-house uptime aside, there's one thing I'm really grateful to the cloud vendors for: making downtime somebody else's problem. Rather than a late-night panicked run to the smoking server, now we can just shrug and wait a few hours and it fixes itself. Most websites aren't that critical, and it's nice not having to lose sleep over devops issues.

>making downtime somebody else's problem.

It is even more, making downtime's responsibility somebody else's problem. You can now simply point to AWS is down, AWS is slow, AWS is causing error, and there is nothing we can do about it :). And as long as management knows everyone is having the same problem they are perfectly fine with it.

Re: AWS down again?

#154

Post by El_RIDO »

41209 wrote: post #67
PedroBatista wrote: post #43

Only the most informed and rational companies will see how and why most of this "in the cloud" thing is a bad idea. Most will accept it as a fact of life and continue to pay for it both directly and indirectly, as long as there's cheap money going around the cloud business can't do wrong. But it's hilarious to see people indulging in byzantine "World scale" resilient systems that depend on a single vendor.

Would you rather have to fix your own data center, or wait 4 hours. AWS works 99% of the time,plus it's someone else's problem

I absolutely prefer to have the option to go into a datacenter in a hurry and actually fix stuff and be in charge, then be stuck with having to wait an indefinite amount of time, twiddle my thumbs, apologize to customers and hope for the best.

While I considered myself a decent Windows NT admin, 20 years ago, the reason I went all in on Linux and FLOSS software at the turn of the century was because I dreaded the powerlessness that these proprietary solutions gave me, when they failed. You'd call the vendors, pored through logs, finding obscure, undocumented error codes, etc. With FLOSS and self-hosting youve got all the information at your fingertips. And if you encountered bugs and you can dig into the sources, patch them, re-compile and fix things - and share them with others and feel that you're contributing to our profession.

When I got the chance to do cloud projects over the last 5 - 10 years, I always took these opportunities, hoping to ensure that I keep up with the tech. At first I was hopeful to offload the boring ops tasks, take our config management to the next level and automate even more. With every platform I got to work with, AWS, Azure and GCP so far, we kept finding bugs in their APIs, outdated or otherwise incorrect documentation and very unpredictable performance, unless you actually can run stacks at scale to average it out (as in more then just 10 - 20 instances or a larger clustered SaaS of the cloud vendor). Many times we also encountered undocumented limits that required requesting support and waiting for approval by the cloud vendor, to get even their mid-sized resources allocated. It all works very nicely on the free-tier-eligible, smallest instances and services, if as slow and high latency as is to be expected, but as soon as you actually need some decently sized storage, compute or bandwidth, it becomes quickly more expensive than what you can put in two or three datacenters for redundancy yourself, if you look at the yearly costs. So far none of the PoCs I was involved with ever got approved long term. They mostly end up as reference implementations for our customers or show case material for the corporate blog. :-(

No thanks, I don't want to go back to feel that powerless as I did on closed systems ever again. Luckily, although many seem to think that cloud is the only option to run at a global scale, you can still provide lots of valuable services on the internet using robust hardware, housed in well connected datacenters.

Re: AWS down again?

#155

Post by gerbilly »

gtirloni wrote: post #124
gerbilly wrote: post #117

Earlier quoted context omitted.

Well at least you said please, however: I have built and run my own infrastructure from the ground up, and I've been made to transition to the cloud. The experience hasn't been great. It may simply be sour 'grapes' because, after all the expertise a whole generation has built up learning UNIX, and all the internet protocols (DNS, ARP, Email, reading RFCs, networking, routing) we get told that all that old stuff is ju…

Your own lack of knowledge about how to make the cloud work properly doesn't mean it's completely useless. The "old school" knowledge is still very useful in building and troubleshooting cloud-based infrastructure. You're creating a false dichotomy.

False dichotomy, if you say so.

I haven't found it too helpful when dealing with AWS and the serverless trend, whose popularity is really just based on price and economics, not technical superiority.

Serverless turns every simple system into a distributed system with the number of failure modes now multiplied by ten.

That sounds like fun.

I do know how to use serverless for the record, I just think it's an overhyped, overpriced waste of time.

Re: AWS down again?

#156

Post by armchairhacker »

clavicat wrote: post #35

Anyone feel like Big Tech outages are happening more frequently recently? In recent months, we’ve seen Amazon Web Services, Facebook, Gmail, and Twitter go down. Are we at the point where the people who maintain the infrastructure are now completely different than the ones who built it, and are struggling to keep it running because they don’t understand it as well?

maybe we actually have a case for decentralized “web3” networks. At least ones which really need reliability, but can’t can use whatever e.g. airplanes use (telehealth? online exams?)

Of course you need to make sure the network is actually decentralized (see Solana). Or maybe you can just rely on Amazon and GCP and Azure, because surely they won’t all fail at the same time.

Re: AWS down again?

#157

Post by reidfromhome »

bierjunge wrote: post #5
raffraffraff wrote: post #2

I've been getting the "We're sorry!" error for at least 15 minutes. No idea about regions or anything, I was just about to look up their docs about moving accounts between orgs etc. Anyone affected? Edit: Seems to be flapping between the 'sorry' error and a blank page. Thoughts and prayers with the SREs, if they call 'em that over there.

Same here. And as always, AWS status page [0] is completely green and useless. [0] https://status.aws.amazon.com/

The only thing that was down was the marketing home page at aws.amazon.com, which is not a “service” and is not on the SHD. All of the actual services are fine, hence the green.

Re: AWS down again?

#158

Post by zdragnar »

gerbilly wrote: post #117
koprulusector wrote: post #84

I expect better of the community here. All it takes is a chance to take a cheap shot at one of the “big boys” and then all of a sudden the weasels come scampering out of the wood work. Seriously, those commenting “oh boy! Time to rethink this whole cloud thing!” You’re either so new to this stuff to have no experience to remember the days before cloud, you’re trolling because you’re high on nostalgia remembering the…

Well at least you said please, however: I have built and run my own infrastructure from the ground up, and I've been made to transition to the cloud. The experience hasn't been great. It may simply be sour 'grapes' because, after all the expertise a whole generation has built up learning UNIX, and all the internet protocols (DNS, ARP, Email, reading RFCs, networking, routing) we get told that all that old stuff is ju…

> Now that because of recent events and we have a chance to be heard

Is there anything about this particular incident that is new that contributes to your position?

Maybe we run in different tech circles, but I feel like the "on-prem vs. cloud" has been litigated fairly extensively here and elsewhere. In fact, as you said yourself:

"Many of us have been taking shots at those 'big boys' since the start of this trend."

Re: AWS down again?

#159

Post by Consultant32452 »

gerbilly wrote: post #117
koprulusector wrote: post #84

I expect better of the community here. All it takes is a chance to take a cheap shot at one of the “big boys” and then all of a sudden the weasels come scampering out of the wood work. Seriously, those commenting “oh boy! Time to rethink this whole cloud thing!” You’re either so new to this stuff to have no experience to remember the days before cloud, you’re trolling because you’re high on nostalgia remembering the…

Well at least you said please, however: I have built and run my own infrastructure from the ground up, and I've been made to transition to the cloud. The experience hasn't been great. It may simply be sour 'grapes' because, after all the expertise a whole generation has built up learning UNIX, and all the internet protocols (DNS, ARP, Email, reading RFCs, networking, routing) we get told that all that old stuff is ju…

I see this as akin to how factory work has changed with modernization. In a modern highly automated factory you need a much smaller number of highly specialized engineers to maintain the robots. In the analogy, these fewer highly specialized engineers are analogous to the ones who need to know understand TCP/IP in depth and now run AWS. The rest of the workers, now replaced by robots, can move on to different productive work.

Re: AWS down again?

#160

Post by hnlmorg »

julianlam wrote: post #132
gerbilly wrote: post #117

Earlier quoted context omitted.

Well at least you said please, however: I have built and run my own infrastructure from the ground up, and I've been made to transition to the cloud. The experience hasn't been great. It may simply be sour 'grapes' because, after all the expertise a whole generation has built up learning UNIX, and all the internet protocols (DNS, ARP, Email, reading RFCs, networking, routing) we get told that all that old stuff is ju…

That's an interesting position to take, because I imagine if a team of developers decided to invent the internet in a vacuum today, it'd be a hell of a lot more secure than the "let's hope nobody uses this protocol maliciously" attitude prevalent in the early days of the internet. Not that that is a bad thing, but just something to think about.

In the early days of the internet it was a closed network of academic and government properties. Nobody at that time would have guess it would grow into even the 80s style internet, let alone what we have now.
Post reply on HN