Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tell HN: AWS appears to be down again

news.ycombinator.com

Post reply on HN
Print view View original

341–350 of 497 posts

Re: Tell HN: AWS appears to be down again

#341

Post by nefitty »

cloverich wrote: post #257
tzs wrote: post #210

Earlier quoted context omitted.

Many years ago I stood at the window of my comfortable apartment, watching wind and cold rain rage outside. I thought about my cave men ancestors who during such a storm if they needed water would have to go out and get it, getting themselves soaked. If I wanted water, the tap in the kitchen would give it to me, in a nice controlled fashion. If I did feel like having water rain down upon me, my shower would do that,…

At a very general level once you move past subsistence farming you become reliant on society to provide your needs. And in turn provide some value that can only come from spending your time on things other than farming. And that is I suppose how civilization advances. Its kind of funny to work backwards though, because even subsistence farmers are reliant on society for protection -- they are farmers not soldiers aft…

There's an interesting misconception that humans developed agricultural societies because they achieved better outcomes as individuals. Research shows that hunter-gatherers were healthy and better nourished than humans in early agricultural settlements.

What's probably closer to truth is that many humans were forced to join farming communities. Stronger individuals or tribes probably enslaved others, and then forced them to build and produce.

The patterns of inequity and the march toward hyper-specialization we still see today make sense in that context.

As a tangent, if anyone is interested in that "cavemanness" deep in our DNA, check out the idea of primitive camping. That was my first experience camping, and I expected an idealized tv-ad experience. The trip was not framed as "primitive camping" to me.

I was dealing with intense burnout, stress, ADHD symptoms, immune problems, trouble sleeping... And I was thrown into the desert in the summer with a tent and some beer. It fucking sucked sooo bad. It fucking sucked sooo bad that I forgot every stupid problem I had, because I spent the entire time in survival mode. Setting up camp. Hauling equipment up and down dunes. Staying hydrated in the 100f+ heat. Making food. Making sure my wife and friends were ok. Strategizing how to defend our camp from bugs and psychos.

I really have not had such an existentially-dense experience as that one. And no, I didn't take any mushrooms, as the rest of the group did. I wanted to be lookout. Maybe I come from a long line of hyperaware sentries.

Re: Tell HN: AWS appears to be down again

#342

Post by dijit »

opportune wrote: post #310
gitfan86 wrote: post #200

Earlier quoted context omitted.

I do contracting dev work and my specialty is being able to drill down into any part of the engineering assets, ops, sec, dev. People think someone like me is slow and expensive until they have a problem that no one else wants to touch.

You know how to debug kernel issues? You know how AWS virtualization works and how to diagnose a problem with the AWS networking stack?

These are not as hard things as you make them sound. They are the things traditional sysadmins spent time understanding.

But I am also incredulous by the parent.

I wouldn’t be able to diagnose traces on a motherboard or a defective (but partially functioning) CPU.

I wouldn’t be able to diagnose irregular voltage conditions or drop offs.

The amount of stuff that I know I couldn’t diagnose is absurdly high, but the amount I don’t even know that I can’t diagnose is higher still.

And my job, like the parents, is to drill down and spend time in specifics.

Re: Tell HN: AWS appears to be down again

#343

Post by r3trohack3r »

earthboundkid wrote: post #195

HOST THE GODDAMN STATUS PAGE ON AZURE FOR FUCKS SAKE. There is zero excuse for this shit. Be professional. Acknowledge reality. It is logically impossible to run your own status page. Trying to do so just wastes everyone else on the internet's time when you have an outage.

I don’t understand, are folks looking at a different status page than me?

This morning we saw some weird behavior in us-west-2, our traffic just _vanished_. I thought: there is no way this is us.

Went to https://status.aws.amazon.com/

Top of the board showed “Internet Connectivity Issues (Oregon)”

And that was that. The board worked exactly as it should - it immediately explained my missing traffic and kept me up-to-date with the status of the outage on their side.

Re: Tell HN: AWS appears to be down again

#344

Post by mikestew »

pier25 wrote: post #299
hotpotamus wrote: post #218

Earlier quoted context omitted.

Not to mention that when the power goes out, the illusion fades fairly quickly. Learned that lesson myself in the Texas snowstorm early this year.

Absolutely. My wife and I lived for a year in an off-the-grid cabin in some mountains in Mexico. We had solar panels and a generator we used only when absolutely necessary. We were never without power, but we lived with the constant anxiety of optimizing our energy consumption. Some stuff we could only do during the day and at night we only used devices with batteries. For a couple of weeks we didn't have running wat…

Absolutely. My wife and I lived for a year in an off-the-grid cabin in some mountains in Mexico.

I think everyone ought to do a week in an RV with no connections to utilities. Not to take away from your story, but a similar scenario comes up when we "dry camp" (no water or electrical connections): resources are not unlimited. We have solar panels, big-ass inverter and big-ass battery to go with it. But if we want lights at night, best not run that 1100W microwave for too long, because the panels won't keep up and the battery isn't that big. We have a built-in generator, but unlike most RV owners, we are loathe to use it. It's almost like a game, and if that generator fires up then we've lost.

You want to let the water run while you brush your teeth? Go right ahead, our water tank is plenty big...oh, wait, but the holding tanks aren't. Shut that tap off before there's dirty water coming up through the shower. Speaking of showers, use the outside shower, as the holding tanks won't hold enough for your 30 minute, piping-hot shower.

Point of it all is that it one quickly learns that it all has to come from somewhere, and it has to go somewhere after you've dirtied it. I'd like to think that it has made the both of us more conscious of our usage.

Re: Tell HN: AWS appears to be down again

#345

Post by fishtoaster »

tinco wrote: post #311
QuiiBz wrote: post #43

I tried to monitor services status using https://stop.lying.cloud , but they are also hosted to AWS, and down too.

Now that they're back up they're not reporting any problems, how is it supposed to work? It looks like it is just repeating the status reported on the Amazon status page.

It is. It's just the AWS status page run through a transformation function to:

1. Remove all the thousand green services that no one cares about when looking at AWS status

2. Upgrade all yellows to reds because Amazon refuses to list anything as "down" no matter how bad the outage is.

3. Insert a snarky legend

Re: Tell HN: AWS appears to be down again

#346

Post by zikduruqe »

derekp7 wrote: post #231
grumple wrote: post #199

Earlier quoted context omitted.

But you could probably devise a scheme by which you feed all of your students an assortment of berries and figure out which ones are safe based on which students get sick or die.

You are close. You first rub the berry on your skin (or leaf, or whatever). Wait 24 hours to see if a rash develops. Then you taste it, wait another 24 hours. Then you eat one, and see if you get sick after another 24 hours. Now you can eat several, and build up from there. Yes, that is a lot of time to go hungry and testing just one item. And then you still don't know what actually gives you nutrition vs just not ki…

That is the Universal Edibility Test and gets repeated ad nauseam in all the survival circles. You would miss out on some fine choice foods if you did that. Stinging Nettles (Urtica dioica) is one of them. Pokeweed (Phytolacca americana) too.

Source - used to teach these skills before it was cool to be a "survivalist" on TV and social media.

Re: Tell HN: AWS appears to be down again

#347

Post by tomlagier »

I wonder if AWS will make more or less money from these outages?

Will large players flee because of excessive instability? Or will smaller players go from single-AZ to more expensive multi-AZ?

My guess is that no-one will leave and lots of single-AZ tenants who should be multi-AZ will use this as the impetus to do it.

Honestly, having events like this is probably good for the overall resilience of distributed systems. It's like an immune system, you don't usually fail in the same way repeatedly.

Re: Tell HN: AWS appears to be down again

#348

Post by FooHentai »

swiftcoder wrote: post #232
lordnacho wrote: post #183

Earlier quoted context omitted.

Surely we are way past the point where someone knows how the whole thing works, all the way down. I doubt even a very skilled engineer would know how his own machine works all the way down. What I think happens mostly is the skilled dev can use his experience to know where to investigate and where to look for solutions. The question is organisational. Might it be that certain orgs have gotten so convoluted that they…

> I doubt even a very skilled engineer would know how his own machine works all the way down Knowing how it works, and being able to build a new one, are also two very different problems. For example, there are plenty of Computer Science folks who learned how to design chips (layout the circuits, write the microcode, etc) - but you need a whole extra background in EE and Physics to be able to fab said chip...

I mean you can (and I have) etch your own circuit boards, but that's obviously not at the scale you need for anything other than primitive processing (70s 8-bit at MHz scale), and even then you're just looking at the next layer down (how to make your own chemical wash and get copper onto a board) as a barrier if we're truly talking about 'from scratch'.

We really depend on three things - knowledge (stored collectively and in various media eg books), materials (tools and manufactured precursor goods, available via active supply chains or existing stores), and most importantly having our basic needs met trivially so that all our time is not sucked up addressing them.

A scenario where someone has a 'wasteland' to pick over for their basic needs, knowledge and materials looks quite different to a return to primitive living where what nature provides is all their is to work with. 'if you want to bake an apple pie, first you must invent the universe' or however it goes...

Then of course there's the question of why someone would have any interest in obtaining computing power were either of those scenarios to occur. Much like the 'how do we warn future civilisations about our nuclear waste' problem perhaps it is acceptable to not bother, they'll figure it out again on their own eventually given enough time.

This stuff is fun to think about at 6am when hay fever is preventing my sleep :)

Re: Tell HN: AWS appears to be down again

#349

Post by fishtoaster »

saganus wrote: post #134
QuiiBz wrote: post #43

I tried to monitor services status using https://stop.lying.cloud , but they are also hosted to AWS, and down too.

How does this service work? It seems to have all the look and feel of AWS, and somehow has more up to date info than the official AWS status page?

It's the same info - it just changes all blues to yellows and all yellows to reds. :)

Re: Tell HN: AWS appears to be down again

#350

Post by gitfan86 »

opportune wrote: post #310
gitfan86 wrote: post #200

Earlier quoted context omitted.

I do contracting dev work and my specialty is being able to drill down into any part of the engineering assets, ops, sec, dev. People think someone like me is slow and expensive until they have a problem that no one else wants to touch.

You know how to debug kernel issues? You know how AWS virtualization works and how to diagnose a problem with the AWS networking stack?

You know how to debug kernel issues? Yes

You know how AWS virtualization works and how to diagnose a problem with the AWS networking stack? Yes, and yes assuming that everything AWS is responsible for is operating within spec. Obviously, I don't have access to their switches, and cannot see anything at layer 1 or 2.

Post reply on HN