Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS down again?

aws.amazon.com

Post reply on HN
Print view View original

191–200 of 257 posts

Re: AWS down again?

#191

Post by iso1631 »

gerbilly wrote: post #117
koprulusector wrote: post #84

I expect better of the community here. All it takes is a chance to take a cheap shot at one of the “big boys” and then all of a sudden the weasels come scampering out of the wood work. Seriously, those commenting “oh boy! Time to rethink this whole cloud thing!” You’re either so new to this stuff to have no experience to remember the days before cloud, you’re trolling because you’re high on nostalgia remembering the…

Well at least you said please, however: I have built and run my own infrastructure from the ground up, and I've been made to transition to the cloud. The experience hasn't been great. It may simply be sour 'grapes' because, after all the expertise a whole generation has built up learning UNIX, and all the internet protocols (DNS, ARP, Email, reading RFCs, networking, routing) we get told that all that old stuff is ju…

I use "cloud" services, but I ensure that my systems continue even if they fail. If AWS is down, maybe I can't do some analysis on historic logs until it comes back, that's a known failure

I look at the components of a system and think "what happens if this is turned off // breaks in an unusual way // goes slow", and ensure that the predicted effects are known and acceptable based on the likelihood of failure.

That's the same whether it's an AWS managed DNS service, storage bucket, or a raspberry pi on my desk. As a systems engineer I know what that component does, what happens when it doesn't "do", and ensure the business knows how to work around it when it breaks.

If your business can't cope with an AWS outage (even if it's not as efficient) then you've got problems.

Plan for failure, and it doesn't take you by surprise.

Re: AWS down again?

#192

Post by pwinnski »

gerbilly wrote: post #117
koprulusector wrote: post #84

I expect better of the community here. All it takes is a chance to take a cheap shot at one of the “big boys” and then all of a sudden the weasels come scampering out of the wood work. Seriously, those commenting “oh boy! Time to rethink this whole cloud thing!” You’re either so new to this stuff to have no experience to remember the days before cloud, you’re trolling because you’re high on nostalgia remembering the…

Well at least you said please, however: I have built and run my own infrastructure from the ground up, and I've been made to transition to the cloud. The experience hasn't been great. It may simply be sour 'grapes' because, after all the expertise a whole generation has built up learning UNIX, and all the internet protocols (DNS, ARP, Email, reading RFCs, networking, routing) we get told that all that old stuff is ju…

> The current generation couldn't invent the internet. You know why, cos they would never have the patience to spec it out like the old timers did. Go read a few RFCs and try to imagine a scrum team today putting as much thought into an up front design.

This is a staggering bit of revisionism. As someone who was around at the time, I remember that many RFCs were written based on already-working code. They had some advantages: nobody much cared what they were doing, so they didn't have to answer to multiple levels of management, and they clearly gave almost no thought to security from bad actors, but if you think there aren't people today--including at those very cloud providers you disdain--doing work at least as well-thought-out as those early pioneers, you haven't been paying any attention at all.

I'll stay off your lawn, but maybe take off those rose-colored glasses and stop pretending the past was rosy.

Re: AWS down again?

#194

Post by gregmac »

grey-area wrote: post #147
oxfordmale wrote: post #116

Earlier quoted context omitted.

Two questions I have regarding your in-house hardware: 1. How easy can I access your physical servers ? 2. What happens if there is a catastrophic failure, for example local power outage or a major flooding 3. How secure is your server? Are you regularly patching your operation systems 4. If I want to run a project that requires double the capacity of your current hardware for a specific project, how long is it going…

There are many approaches which don't depend on AWS, and not all of them mean hosting your own physical servers, and they certainly don't mean you don't have an off-site backup policy. There are well-understood answers to all your questions, they are not too difficult, they just cost money - some businesses choose not to spend that money, some weigh cost-benefit and go for AWS, some decide to go for in-house servers,…

> they just cost money - some businesses choose not to spend that money, some weight cost-benefit and go for AWS, some weigh cost benefit and decide to go for in house, some go for hosted Virtual Servers, some go for Serverless.

I'd also say some choose not to spend the money, but fail to consider the cost of that choice.

For example: Doing old-school manual deployments that require herculean efforts to update at off-hours on the weekends burns people out, and makes it hard to attract new talent. In other words, you've made the decision to spend more money on finding and retaining people. But it's definitely way cheaper to pay for colocating a single Dell server you bought 3 years ago than what you'd spend in the same time on AWS.

And if your hardware never dies, paying for the redundancy might seem silly. A lot like paying for fire insurance despite the fact your house has never even burned down.

Re: AWS down again?

#195

Post by _hyn3 »

Spivak wrote: post #161
gerbilly wrote: post #117

Earlier quoted context omitted.

Well at least you said please, however: I have built and run my own infrastructure from the ground up, and I've been made to transition to the cloud. The experience hasn't been great. It may simply be sour 'grapes' because, after all the expertise a whole generation has built up learning UNIX, and all the internet protocols (DNS, ARP, Email, reading RFCs, networking, routing) we get told that all that old stuff is ju…

(Assumedly) much much younger person here who has spent the majority of her career on prem, has lifted multiple shops to AWS, and right now works full time on an all AWS stack. Your experience with UNIX isn't worthless, but it's worthless to anyone who's working "further up the stack" than you. If you feel like your skills are degrading then you need to find a job somewhere that's actually building infra that shops w…

> You're confusing people building products with people building infrastructure

Even though I'm on the side of the "UNIX graybeards" here, this is a super-great point. We do need to recognize that it is a great time to be building networked applications, precisely because the younger ones these days don't need to understand TCP/IP or anything else related to infrastructure.

I confess that I got caught up in the hype as well and built quite a few "multi-AZ" apps that I thought would help me get to five nines that much faster. (for non-cloud folks, that's 99.999% availability, which was something to pursue before the cloud.)

Of course, when those abstractions break, those same younger ones are completely helpless, and my single-server apps have been running non-stop for years at traditional providers except for a few minutes for reboots following updates. I've never had a multi-hour outage, especially one that's completely out of my control where I can only point a finger at AWS and say, "sorry, it's not my fault."

Re: AWS down again?

#196

Post by tuldia »

emptybottle wrote: post #110
jeremyjh wrote: post #54

Earlier quoted context omitted.

Do you think people who manage their own data-centers never have outages?

Running your own data center is on the far other end of the spectrum. There are numerous options between solely using AWS and building a data center.

this

Re: AWS down again?

#197

Post by technobabbler »

tuldia wrote: post #165
ksec wrote: post #153

Earlier quoted context omitted.

>making downtime somebody else's problem. It is even more, making downtime's responsibility somebody else's problem. You can now simply point to AWS is down, AWS is slow, AWS is causing error, and there is nothing we can do about it :). And as long as management knows everyone is having the same problem they are perfectly fine with it.

> You can now simply point to AWS is down Heck, if that was even possible... "everything is green" dashboards... :-)

Obviously, the answer is to pay for a cloud dashboard-of-dashboards that show orange lights when AWS's is falsely green.

"We need this new fancy dashboard to monitor our other dashboards in one place, more accurately!"

Re: AWS down again?

#198

Post by technobabbler »

londons_explore wrote: post #127
technobabbler wrote: post #104

Earlier quoted context omitted.

Y'know, debates about cloud vs in-house uptime aside, there's one thing I'm really grateful to the cloud vendors for: making downtime somebody else's problem. Rather than a late-night panicked run to the smoking server, now we can just shrug and wait a few hours and it fixes itself. Most websites aren't that critical, and it's nice not having to lose sleep over devops issues.

I find most cloud users still have to panic at 3am when there is a cloud outage because usually there are knobs to tweak in their application to mitigate the outage. For example, do a region failover, turn off some feature that depends on dynamodb, or push an emergency release to make the application server handle spurious 403 errors from an Amazon backend. Since Amazon failure modes are so varied, it's impossible to…

If you're going closer-to-the-metal cloud, then yeah. I think that's part of the allure of newer, more abstracted cloud hosts like Vercel, Netlify, Gatsby, etc.

They take care of all the infrastructure for ya, and if something breaks, you just twiddle your thumbs while they twiddle the dials.

Of course this sort of laziness is not a valid approach if you're a huge enterprise or running some live-saving project that needs seventy nines of uptime, but for non-critical websites, it's a great sigh of relief...

Re: AWS down again?

#199

Post by rstupek »

y4mi wrote: post #181
rstupek wrote: post #101

Earlier quoted context omitted.

That is true. But hope a drive by ddos doesn't pick you that day!

How do the attackers ddos an inhouse deployment that cannot be accessed from the internet without joining a VPN? They can try to saturate the VPN host maybe, but that's going to be challenging considering that it's going to be limited to connection requests without valid credentials. and these are likely set to be ignored on multiple failed attempts through fail2ban or similar tooling

You have a connection to the internet with an IP address do you not? That connection can easily be saturated with traffic and fail2ban would not have a chance to do anything.

Re: AWS down again?

#200

Post by SQueeeeeL »

strken wrote: post #175
gerbilly wrote: post #117

Earlier quoted context omitted.

Well at least you said please, however: I have built and run my own infrastructure from the ground up, and I've been made to transition to the cloud. The experience hasn't been great. It may simply be sour 'grapes' because, after all the expertise a whole generation has built up learning UNIX, and all the internet protocols (DNS, ARP, Email, reading RFCs, networking, routing) we get told that all that old stuff is ju…

This conflates two very different things: cut-throat KPI-driven development practices at companies like Amazon, and the entire current generation of software developers. You're taking problems caused by the environment and attributing them to a moral deficiency in the people, which is neither fair nor helpful.

Environment (in general) is what determines skill set, not the other way around. If the interview process makes you focus on leetcode skills, and your manager focuses on LOCs produced and hitting story milestones and less on spending time integrating with the team and learning about legacy codebase; it makes sense those who came out of this environment would be less prepared to tackle certain kinds of problems.
Post reply on HN