Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS down again?

aws.amazon.com

Post reply on HN
Print view View original

121–130 of 257 posts

Re: AWS down again?

#121

Post by carlmr »

api wrote: post #93
y4mi wrote: post #89

Earlier quoted context omitted.

> Even if you can manage more uptime on your own than through the cloud (which I doubt) Getting higher uptime is super easy for smallish inhouse deployments. You just don't install any updates and let the server run, trusting your VPN to shield you from possible security issues. The maintenance burden is the reason why people often prefer the cloud services, not the uptime. Because maintaining the instance with updat…

This is particularly true with modern fully solid state hardware. Uptimes of decades are likely possible if you don't mess with anything. We run some bare metal servers. They just never go down. Solid continuous pings for years as monitored from elsewhere on the Internet. That's because they're just boxes on a rack somewhere running an OS and some steady-state services (ZeroTier roots). Simplicity is more robust than…

>Many companies spend enough on AWS to buy an entire rack of hardware at a different data center every month and pay 2-3 employees to manage it. That hardware would be up to 100X as fast and powerful as what they rent at AWS and bandwidth would be almost free. That's a really distorted market. The amortized costs should not be this different.

I think the issue here is that it's not zero-cost to switch. Your processes will adapt to some implicit assumptions that aren't true outside AWS, Azure, or whatever vendor you locked yourself into.

If we somehow managed to have a completely standardized interface here, the market would be more competitive.

Re: AWS down again?

#122

Post by christophilus »

technobabbler wrote: post #104
koprulusector wrote: post #84

I expect better of the community here. All it takes is a chance to take a cheap shot at one of the “big boys” and then all of a sudden the weasels come scampering out of the wood work. Seriously, those commenting “oh boy! Time to rethink this whole cloud thing!” You’re either so new to this stuff to have no experience to remember the days before cloud, you’re trolling because you’re high on nostalgia remembering the…

Y'know, debates about cloud vs in-house uptime aside, there's one thing I'm really grateful to the cloud vendors for: making downtime somebody else's problem. Rather than a late-night panicked run to the smoking server, now we can just shrug and wait a few hours and it fixes itself. Most websites aren't that critical, and it's nice not having to lose sleep over devops issues.

I mostly agree, but this is a double-edged sword. I recently had a 6 hour outage due to lots and lots of waiting on a fix from our managed database provider. Had we been running the DB ourselves, I would have had the permissions to monitor the DB and caught the issue early (it was a pretty trivial thing to monitor). If we had gone down, I would have been able to run my restore within an hour, rather than 6. And lastly, self-hosted performance was much better than it's been with a managed DB.

Re: AWS down again?

#123

Post by tuldia »

SCHiM wrote: post #21

I think this adds some momentum to the pendulum swinging back the other way. Maybe cloud teams can patch your services better than your in-house team can (See 2 critical issues in Azure the last 3 months, _caused_ by MS itself). Maybe the cloud has a higher uptime than your on-premise infrastructure (see the AWS, Azure outages). Make sure to compare the actual outage time v.s. the stats doctored by various political…

The learned helplessness in the cloud is stupefying, so many outages and downtime that could have been avoided by a competent admin.

Re: AWS down again?

#124

Post by gtirloni »

gerbilly wrote: post #117
koprulusector wrote: post #84

I expect better of the community here. All it takes is a chance to take a cheap shot at one of the “big boys” and then all of a sudden the weasels come scampering out of the wood work. Seriously, those commenting “oh boy! Time to rethink this whole cloud thing!” You’re either so new to this stuff to have no experience to remember the days before cloud, you’re trolling because you’re high on nostalgia remembering the…

Well at least you said please, however: I have built and run my own infrastructure from the ground up, and I've been made to transition to the cloud. The experience hasn't been great. It may simply be sour 'grapes' because, after all the expertise a whole generation has built up learning UNIX, and all the internet protocols (DNS, ARP, Email, reading RFCs, networking, routing) we get told that all that old stuff is ju…

Your own lack of knowledge about how to make the cloud work properly doesn't mean it's completely useless. The "old school" knowledge is still very useful in building and troubleshooting cloud-based infrastructure. You're creating a false dichotomy.

Re: AWS down again?

#125

Post by [deleted] »

koprulusector wrote: post #84

I expect better of the community here. All it takes is a chance to take a cheap shot at one of the “big boys” and then all of a sudden the weasels come scampering out of the wood work. Seriously, those commenting “oh boy! Time to rethink this whole cloud thing!” You’re either so new to this stuff to have no experience to remember the days before cloud, you’re trolling because you’re high on nostalgia remembering the…

[deleted]

Re: AWS down again?

#126

Post by spamizbad »

SCHiM wrote: post #21

I think this adds some momentum to the pendulum swinging back the other way. Maybe cloud teams can patch your services better than your in-house team can (See 2 critical issues in Azure the last 3 months, _caused_ by MS itself). Maybe the cloud has a higher uptime than your on-premise infrastructure (see the AWS, Azure outages). Make sure to compare the actual outage time v.s. the stats doctored by various political…

One problem facing on-prem orgs today is the sad state of commodity server hardware; poor quality control, buggy firmware (and vendors who won’t help), BMCs with massive attack surfaces (that have already claimed one VPN provider), and commodity network switches that can push lots of packets but aren’t terribly flexible for people who are pushing more challenging payloads around their DC (video, etc).

"Hyperscalers" like Amazon, Microsoft, Facebook and Google build their own hardware and are able to avoid many of these problems. Unfortunately, none of this stuff is available off the shelf to mere mortals.

There’s a startup trying to fix this problem (Oxide) which I think is launching their racks next year. Will be interesting to see what happens.

Re: AWS down again?

#127

Post by londons_explore »

technobabbler wrote: post #104
koprulusector wrote: post #84

I expect better of the community here. All it takes is a chance to take a cheap shot at one of the “big boys” and then all of a sudden the weasels come scampering out of the wood work. Seriously, those commenting “oh boy! Time to rethink this whole cloud thing!” You’re either so new to this stuff to have no experience to remember the days before cloud, you’re trolling because you’re high on nostalgia remembering the…

Y'know, debates about cloud vs in-house uptime aside, there's one thing I'm really grateful to the cloud vendors for: making downtime somebody else's problem. Rather than a late-night panicked run to the smoking server, now we can just shrug and wait a few hours and it fixes itself. Most websites aren't that critical, and it's nice not having to lose sleep over devops issues.

I find most cloud users still have to panic at 3am when there is a cloud outage because usually there are knobs to tweak in their application to mitigate the outage. For example, do a region failover, turn off some feature that depends on dynamodb, or push an emergency release to make the application server handle spurious 403 errors from an Amazon backend.

Since Amazon failure modes are so varied, it's impossible to make the above tweaks fully automated. There'll always be a new way the cloud can misbehave, and often your service can at least maintain partial service by being nimble enough.

Re: AWS down again?

#128

Post by mac-chaffee »

marginalia_nu wrote: post #64
jeremyjh wrote: post #54

Earlier quoted context omitted.

Do you think people who manage their own data-centers never have outages?

When they do, they at least have the ability to do something about it.

My last outage was a power outage on a sunny day that lasted slightly longer than our battery backups. Without economies of scale, installing/maintaining extra generators for a small self-managed data center is prohibitively expensive.

There's always a service provider somewhere in the chain who can drop the ball.

Re: AWS down again?

#129

Post by ignoramous »

koprulusector wrote: post #84

I expect better of the community here. All it takes is a chance to take a cheap shot at one of the “big boys” and then all of a sudden the weasels come scampering out of the wood work. Seriously, those commenting “oh boy! Time to rethink this whole cloud thing!” You’re either so new to this stuff to have no experience to remember the days before cloud, you’re trolling because you’re high on nostalgia remembering the…

> oh boy! Time to rethink this whole cloud thing!

The problem isn't the cloud, the problem is all of the web properties going down all at once, exacerbated by the fact that AWS charges a premium for any viable cross-region / multi-cloud architecture (given its relatively high egress fees). This is discounting the inter-dependence among AWS services themselves, some of which aren't multi-region (afaik).

Re: AWS down again?

#130

Post by netsec_burn »

desdiv wrote: post #90
ff7c11 wrote: post #55

${jndi:ldap://AWS Down}?

For those out of the loop: this is a reference to the Log4j RCE story that's also currently on the HN frontpage. Amazon uses a lot of Java, so the two stories might be actually connected somehow. [0] https://news.ycombinator.com/item?id=29504755

Spoiler: They are. AWS and Amazon both used Log4j exploitable from the homepage (/) of both sites through headers.

Source: me

Post reply on HN