Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS down again?

aws.amazon.com

Post reply on HN
Print view View original

201–210 of 257 posts

Re: AWS down again?

#201

Post by phailhaus »

gerbilly wrote: post #117
koprulusector wrote: post #84

I expect better of the community here. All it takes is a chance to take a cheap shot at one of the “big boys” and then all of a sudden the weasels come scampering out of the wood work. Seriously, those commenting “oh boy! Time to rethink this whole cloud thing!” You’re either so new to this stuff to have no experience to remember the days before cloud, you’re trolling because you’re high on nostalgia remembering the…

Well at least you said please, however: I have built and run my own infrastructure from the ground up, and I've been made to transition to the cloud. The experience hasn't been great. It may simply be sour 'grapes' because, after all the expertise a whole generation has built up learning UNIX, and all the internet protocols (DNS, ARP, Email, reading RFCs, networking, routing) we get told that all that old stuff is ju…

I mean, because you're wrong? Even with the recent outages, AWS still has far higher uptime and support than anything that someone could cobble together in their own small company. That's the advantage of using cloud infrastructure financed by billions of dollars.

> The current generation couldn't invent the internet.

Come on now, this is an overdone, lame argument and I can't believe you're seriously suggesting this. Do you also lament the fact that kids these days can't bind their own books? The point of tools is to be built upon, not to sit around marveling at your own genius. If you build a good tool, the folks that come after you don't have to think about it. That's how you make progress.

Re: AWS down again?

#202

Post by redwood »

naikrovek wrote: post #118
redwood wrote: post #6

Still no post mortem from Monday's right? We're all still in the dark?

> Still it's usually a month or more after a large outage to see the full breakdown on what happened. people expecting to see it the same week are kinda not living in reality.

Fair I guess I would have expected more in the way of official comms though. Most of what I've seen is based on deduction and speculation rather than from the horse's mouth

Re: AWS down again?

#203

Post by gerbilly »

tinalumfoil wrote: post #56
SCHiM wrote: post #21

I think this adds some momentum to the pendulum swinging back the other way. Maybe cloud teams can patch your services better than your in-house team can (See 2 critical issues in Azure the last 3 months, _caused_ by MS itself). Maybe the cloud has a higher uptime than your on-premise infrastructure (see the AWS, Azure outages). Make sure to compare the actual outage time v.s. the stats doctored by various political…

Even if you can manage more uptime on your own than through the cloud (which I doubt), being on the cloud means downtime is correlated with downtime of other services. That's usually a good thing. Your customers will be more understanding if your outage is part if a wider outage that makes national news. Any services you integrate with are likely down too. If two services with 99% uncorrelated uptime together drops t…

> being on the cloud means downtime is correlated with downtime of other services. That's usually a good thing.

A long time ago, I used to circulate snarky little emails at work.

One of them was responding to this very concept.

My managers were throwing out our working and mature UNIX servers (implementing DNS, Mail, and other services) in favour of NT.

The new system crashed a lot, we had some security breaches, but at least it was 'industry standard.'

Managements' response was that with the UNIX stuff we had no one to pin our outages on. Now we could blame Microsoft, and call their support line.

I circulated an email making fun of this justification, which promoted a fictional product called 'Blame Studio' which would help you map out the blame path for any of your products or services.

It would help to make sure that none of the blame ever landed on you, but rather was always redirected onto some other company.

Re: AWS down again?

#204

Post by technobabbler »

gerbilly wrote: post #117
koprulusector wrote: post #84

I expect better of the community here. All it takes is a chance to take a cheap shot at one of the “big boys” and then all of a sudden the weasels come scampering out of the wood work. Seriously, those commenting “oh boy! Time to rethink this whole cloud thing!” You’re either so new to this stuff to have no experience to remember the days before cloud, you’re trolling because you’re high on nostalgia remembering the…

Well at least you said please, however: I have built and run my own infrastructure from the ground up, and I've been made to transition to the cloud. The experience hasn't been great. It may simply be sour 'grapes' because, after all the expertise a whole generation has built up learning UNIX, and all the internet protocols (DNS, ARP, Email, reading RFCs, networking, routing) we get told that all that old stuff is ju…

IMHO these are entirely different skillsets, and a division-of-labor question rather than some sort of insurmountable generation gap. It's not like infrastructural know-how isn't relevant anymore; they just became CDN engineers or DevOps or senior scaling or reliability engineers. Their jobs are no easier than before, especially when you have to consider network traversals across layers of virtualized/containerized services across multiple data centers owned by disparate parties and maintained by different vendors.

Virtualization aside, we haven't abandoned basic infrastructure, but centralized it in the hands of a few huge, expert providers. IMO this is a good thing, and was both necessary and natural as the Web grew to offer more and more opportunities to more new professionals. In detaching HTML from HTTP from ARP, etc. we gave rise to entire new professions like full-time UX (which arguably the Old Guard was never good at beyond a small audience of academics and engineers), or various flavors of front-end developer, or serverless ecosystems.

The Web and associated technologies advanced so quickly it was impractical for a single IT or network department to know all of it anymore, and some of the newer webapps wouldn't have been possible if that same team or company had to also manage all of their own basic infrastructure like it was the 90s still.

Now you can be a front-end only shop, or a UX consultant, or a network engineer who never has to touch HTML, or, or, or... maybe big enterprises always had and could always have all of those in-house, but the division of labor has been a huge boon for small businesses and startups and nonprofits, who just don't have the same resources.

As someone who grew up configuring zmodem and running BBSes and having to (mis)configure NetBIOS all the time, I am so, so glad I never have to worry about OSI layers and such ever again. It's boring to me, and the experts at it are SO much better at it, might as well let them handle it. Especially when the cost of that outsourcing is often like sanity. The division of concerns lets you focus on the things you're either interested in and/or good at.

Our professionals haven't gotten worse. The stack has gotten much deeper.

Re: AWS down again?

#205

Post by doublerabbit »

koprulusector wrote: post #84

I expect better of the community here. All it takes is a chance to take a cheap shot at one of the “big boys” and then all of a sudden the weasels come scampering out of the wood work. Seriously, those commenting “oh boy! Time to rethink this whole cloud thing!” You’re either so new to this stuff to have no experience to remember the days before cloud, you’re trolling because you’re high on nostalgia remembering the…

Serverless is just buzzword for a container on a virtual machine on a server. The Cloud is just a buzzword for a company with a data centre selling virtual servers from other big servers.

> You’re either so new to this stuff to have no experience to remember the days before cloud

I do, and it was much more peaceful.

Everything you can do on "in the cloud" you can do in colocation. Which in the long-run is cheaper, more secure* and it's yours! Including the data.

There are caveats, network, component failure. Investment in to these and you can have a pretty king setup.

The cloud enabled magnitude of email spam, brute-forcing, botnets, security vulnerabilities and much more. Operators are lazy don't want to combat it. Providers are bias, then again you can say that about any business.

People flock to the cloud like it's the greatest thing, when all your buying in to is a expensive price-plan for a company who will happily knock you off their service if you somehow brush up the wrong way and then charge you for a closed account.

I do laugh when something goes wrong for FANG. Partly, I'm cynical and want to see the world burn, but it's also these companies exploit their userbase, their staff and the environments resources. When Facebook locked themselves out of their own offices due to the BGP issue, now that's funny.

My colocation costs are:

$5000 covers for a three-year 1Gbit 2u server racking space in two different DCs. Where I have full-control, as many services as I desire, allowed to host what I desire and where the internet space is actually mine. It may be a small cube of internet but I know it's my network, my traffic.

Cloud is whitewash for me, I won't buy in to it. It has it purposes and if your happy with it, fine. But for me Colo for life.

Re: AWS down again?

#206

Post by 41209 »

El_RIDO wrote: post #154
41209 wrote: post #67

Earlier quoted context omitted.

Would you rather have to fix your own data center, or wait 4 hours. AWS works 99% of the time,plus it's someone else's problem

I absolutely prefer to have the option to go into a datacenter in a hurry and actually fix stuff and be in charge, then be stuck with having to wait an indefinite amount of time, twiddle my thumbs, apologize to customers and hope for the best. While I considered myself a decent Windows NT admin, 20 years ago, the reason I went all in on Linux and FLOSS software at the turn of the century was because I dreaded the pow…

You may be correct, for some very specific use cases. No matter how much corporate clients want to holler, a few hours of downtime isn't going to hurt anyone.

I know I'd rather someone else do it, then having to drive 2 hours away to a data center at 4:00 in the morning. But I don't know exactly what you're working on , I definitely can't imagine some use cases where a few hours of down time is just unacceptable. I know I wouldn't want to run a logistics firm with servers that go down all the time.

Re: AWS down again?

#207

Post by bushbaba »

x86_64Ubuntu wrote: post #87
acdha wrote: post #47

Earlier quoted context omitted.

This morning I’d bet more on a rushed log4j patch. They use it heavily.

It's kind of terrible that a logging library can lead to such downtime.

When you need to force update all of prod asap. Yes it is terrible. The fact log4j even does remote network calls is crazy

Re: AWS down again?

#208

Post by papito »

willcipriano wrote: post #111
papito wrote: post #57

Earlier quoted context omitted.

Even on HN there are some former AWS employees who talk about how its all stitched together and flying on a wing a prayer. Apparently the on-call is just a traumatizing experience. It will take real damage to revenue for the management to pay that debt off.

Is there a case of an organization ever paying off "tech debt" (I refuse the term, I call it incomplete software)? I've only ever seen it snowball until the product falls into the sea and they start again fresh.

You never pay it off. It's like "the war on crime" or "the war on drugs". Don't call it a war if there is no clear win. You can never win the war, but you MUST win the battles in order to not lose it.

Re: AWS down again?

#209

Post by xtracto »

willcipriano wrote: post #111
papito wrote: post #57

Earlier quoted context omitted.

Even on HN there are some former AWS employees who talk about how its all stitched together and flying on a wing a prayer. Apparently the on-call is just a traumatizing experience. It will take real damage to revenue for the management to pay that debt off.

Is there a case of an organization ever paying off "tech debt" (I refuse the term, I call it incomplete software)? I've only ever seen it snowball until the product falls into the sea and they start again fresh.

I loved a quote someone else wrote here some time ago: "Hackers are just tech debt collectors" [1]. If ever there has been a true quote, it is this.

[1] https://news.ycombinator.com/item?id=29039611

Post reply on HN