Live data from Hacker News

Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

aws.amazon.com

221–230 of 410 posts

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#221
post #49

> Customers accessing Amazon S3 and DynamoDB were not impacted by this event. We've seen plenty of S3 errors during that period. Kind of undermines credibility of this report.

Do you use VPC endpoints for S3? The next sentence explained failures I observed with S3: "However, access to Amazon S3 buckets and DynamoDB tables via VPC Endpoints was impaired during this event."

I could not modify file properties in S3, uploading new or modified files was spotty, and AWS Console GUI access was broken as well. Was that because of VPC endpoints?

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#222

Earlier quoted context omitted.

Do you use VPC endpoints for S3? The next sentence explained failures I observed with S3: "However, access to Amazon S3 buckets and DynamoDB tables via VPC Endpoints was impaired during this event."

I could not modify file properties in S3, uploading new or modified files was spotty, and AWS Console GUI access was broken as well. Was that because of VPC endpoints?

"Customers also experienced login failures to the AWS Console in the impacted region during the event"

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#223

Earlier quoted context omitted.

I worked at Amazon. While my boss was on vacation I took over for him in the "Launch readiness" meeting for our team's component of our project. Basically, you go to this meeting with the big decision makers and business people once a week and tell them what your status is on deliverables. You are supposed to sum up your status as "Green/Yellow/Red" and then write (or update last week's document) to explain your stat…

your story reminded me of the Challenger disaster and the "see no evil" bureaucratic shenanigans about the O-rings failing to seal in cold weather. "How dare you threaten our launch readiness go/no-go?!"

Was Challenger the one where they buried the issue in a hundred-slide-long PowerPoint? Or was that the other shuttle?

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#224

Earlier quoted context omitted.

reality most of the real world successful projects are mostly untested because that's not actually a high ROI endeavor. it kills me to realize that mediocre code you can hack all over to do unnatural things is generally higher value in phase I than the same code done well in twice the time.

This attitude is why modern software is a continuing controlled flight into terrain.

Finger wagging has also failed to produce any solutions.

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#225
post #53

My company uses AWS. We had significant degradation for many of their APIs for over six hours, having a substantive impact on our business. The entire time their outage board was solid green. We were in touch with their support people and knew it was bad but were under NDA not to discuss it with anyone. Of course problems and outages are going to happen, but saying they have five nines (99.999) uptime as measured by…

This. We're under NDA too on internal support. Our customers know we use AWS and they go and check the AWS status dashboards and tell us there's nothing wrong so the inevitable vitriol is always directed at us which we then have to defend.

I guess you have to hope that every outage that impacts you is big enough to make the news.

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#226
post #174

"Amazon Secure Token Service (STS) experienced elevated latencies" I was getting 503 "service unavailable" from STS during the outage most of the time I tried calling it. I guess by "elevated latency", they mean from anyone with retry logic that would keep trying after many consecutive attempts?

I suppose all outages are just elevated latency. Has anyone ever had an outage and said "fuck it, we're going out of business" and never came back up? That's the only true outage ;)

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#228

My company uses AWS. We had significant degradation for many of their APIs for over six hours, having a substantive impact on our business. The entire time their outage board was solid green. We were in touch with their support people and knew it was bad but were under NDA not to discuss it with anyone. Of course problems and outages are going to happen, but saying they have five nines (99.999) uptime as measured by…

I worked at Amazon. While my boss was on vacation I took over for him in the "Launch readiness" meeting for our team's component of our project. Basically, you go to this meeting with the big decision makers and business people once a week and tell them what your status is on deliverables. You are supposed to sum up your status as "Green/Yellow/Red" and then write (or update last week's document) to explain your stat…

I worked at an Amazon air-shipping warehouse for a couple years, and hearing this confirms my suspicions about the management there. Lower management (supervisors, people actually in the building) were very aware of problems, but the people who ran the building lived out of state, so they only actually went to the building on very rare occasions.

Equipment was constantly breaking down, in ways that ranged from inconvenient to potentially dangerous. Seemingly basic design decisions, like the shape of chutes, were screwed up in mind-boggling ways (they put a right-angle corner partway down each chute, which caused packages to get stuck in the chutes constantly). We were short on equipment almost every day; things like poles to help us un-jam packages were in short supply, even though we could move hundreds of thousands of packages a day. On top of all this, the facility opened with half its sorting equipment, and despite promises that we'd be able to add the rest of the equipment in the summer, during Amazon's slow season...it took them two years to even get started.

And all the while, they demanded ever-increasing package quotas. At first, 120,000 packages/day was enough to raise eyebrows--we broke records on a daily basis in our first holiday rush--but then, they started wanting 200,000, then 400,000. Eventually it came out that the building wouldn't even be breaking even until it hit something like 500,000.

As we scaled up, things got even worse. None of the improvements that workers suggested to management were used, to my knowledge, even simple things like adding an indicator light to freight elevators.

Meanwhile, it eventually became clear that there wasn't enough space to store cargo containers in the building. 737s and the like store packages mostly in these giant curved cargo containers, and we needed them to be locked in place while working around/in them...except that, surprise, the people planning the building hadn't planned any holding areas for containers that weren't in use! We ended up sticking them in the middle of the work area.

Which pissed off the upper management when they visited. Their decision? Stop doing it. Are we getting more storage space for the cans? No. Are we getting more workers on the airplane ramp so we can put these cans outside faster? No. But we're not allowed to store those cans in the middle of the work area anymore, even if there aren't any open stations with working locks. Oh, by the way, the locking mechanisms that hold the cans in place started to break down, and to my knowledge they never actually fixed any of the locks. (A guy from their safety team claims they've fixed like 80 or 90 of the stations since the building opened, but none of the broken locks I've seen were fixed in the 2 years I worked there.)

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#229

"At 7:30 AM PST, an automated activity to scale capacity of one of the AWS services hosted in the main AWS network triggered an unexpected behavior from a large number of clients inside the internal network. This resulted in a large surge of connection activity that overwhelmed the networking devices between the internal network and the main AWS network, resulting in delays for communication between these networks. T…

Yes and other cloud providers (Google, Microsoft) probably have similar. Besides special network equipment, they use PCIe accelerator/coprocessors on their hypervisors to offload all non-VM activity (Nitro instances)

They also recently announced Graviton ARM CPUs

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#230

Earlier quoted context omitted.

It's 2006, you work for an 'online book store' that's experimenting with this cloud thing. Are you going to build a whole new network involving multi-million dollar networking appliances?

Developing EC2 did involve building a whole new network. It was a new service, built from the ground up to be a public product.

Amazon was famously frugal during that era.
Post reply on HN