Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS Restored My Account: The Human Who Made the Difference

seuros.com

Post reply on HN
Print view View original

51–60 of 95 posts

Re: AWS Restored My Account: The Human Who Made the Difference

#51

Post by colmmacc »

Timshel wrote: post #47
colmmacc wrote: post #43

Earlier quoted context omitted.

Without prejudging the COE; it won't surprise anyone to learn that there are bad actors out there who try "every trick in the book" to have accounts that they don't pay for, and lying to customer support is absolutely one of those tricks, as is trying to be creative with changing payment instruments. In these cases, it's also really important that customer support stick to a script and can't be abused as part of soci…

I love the irony that an issue caused by a failing automation was solved due to human escalation but let's not try to improve the escalation process but add more automation ...

We do both; automation never forgets.

Re: AWS Restored My Account: The Human Who Made the Difference

#52

Post by cnst »

Keep in mind, just 2 days ago, AWS has provided the following statement to Tom's Hardware:

https://www.tomshardware.com/software/cloud-storage/aws-accu...

https://archive.is/0b3Hc

> Update: August 5 7:30am (ET): In a statement, an AWS spokesperson told Tom's Hardware "We always strive to work with customers to resolve account issues and provided an advance warning of the potential account suspension. The account was suspended as part of AWS’s standard security protocols for accounts that fail the required verification, and it is incorrect to claim this was because of a system error or accident."

This shows a bigger part of this problem.

When these mistakes do happen, they're invariably treated as standard operating procedures.

They're NEVER treated as errors.

It would appear that the entire support personnel chain and PR literally have no escalation path to treat any of these things as errors.

Instead, they simply double-down that it's NOT an error that the accounts was terminated on an insufficient notice over bogus claims and broken policies.

Re: AWS Restored My Account: The Human Who Made the Difference

#53

Post by dylan604 »

pavel_lishin wrote: post #18
dylan604 wrote: post #15

Earlier quoted context omitted.

Offsite does not have to mean another cloud solution. If you have a copy of data on a thumbdrive or other external media, you can leave it at your parent's house (or other family member). You can keep a copy of it in a safety deposit box. If it's at a family member's house, it's just an excuse to visit on a semi-regular schedule as you swap out with an updated version.

Yep. My backups - personal ones, not business - live in three places: 1. A hard drive in a fire safe. 2. An S3 bucket, mediated by Wasabi. 3. My friend's server that lives at his house half a continent away. It would be nice to have a fourth location that's a physical hard-drive that lives outside of my house, but close enough to drive to for pick-up, but it would mean either paying for a safety deposit box as you me…

> My friend's server that lives at his house half a continent away.

I figure that if a disaster that takes out my house and someone across town at the same time, I probably won't be worrying about restoring data. Across continent would only be viable with a server like you mentioned essentially buddyCloud.

Re: AWS Restored My Account: The Human Who Made the Difference

#54

Post by eqvinox »

ke4qqq wrote: post #36
eqvinox wrote: post #11

> "people in positions of leadership, such as my boss, are aware of your blog post and I’ve been tasked with finding out what I can" Translation: "someone noticed it trending on HN, decided it was bad publicity, and that they should do something about it" Implication: what mattered was the bad publicity, not the poor support infrastructure. The latter won't change, and the next person with similar problems will get t…

As the cited ‘boss’ I’ll say the publicity wasn’t the concern. The concern was that someone wanted to use our services and we had made that so frustrating that they were writing blog posts about how it had gone wrong. The various teams (anti-fraud and support) are investigating how we failed this customer so we can improve and hopefully keep this from happening again. (This is the ‘Correction of Error’ process that’s…

> The concern was that someone wanted to use our services and we had made that so frustrating that they were writing blog posts about how it had gone wrong.

I'm concerned that you're being very unspecific talking about "our services" and "it" going wrong.

What went wrong here is AWS not spending enough money on humans in the support teams. And of course this is a neverending balancing act between profitability and usability. Like any other profit vs. usability consideration, the curve probably has a knee somewhere when the service becomes too unusable and too many people flee to the competition.

And it seems current economic wisdom is that that knee in the curve is pretty far on the "bad support" side of the scale.

Which is to say, the cynic in me doesn't believe you'll be making any changes, mostly because that knee in the curve is in fact pretty far on the "bad support" side, and economics compels you to exploit that.

Re: AWS Restored My Account: The Human Who Made the Difference

#55

Post by CamperBob2 »

colmmacc wrote: post #43
CamperBob2 wrote: post #32

Earlier quoted context omitted.

So, the key is somehow finding a way to bring problems like this to the group's attention, it sounds like. HN lamentations are great when they work, but they don't exactly scale. At least one layer of human support needs to have the ability -- not just the ability, but the obligation! -- to escalate to your team when a customer service problem occurs that doesn't fit a pattern of known/active scams and they are unabl…

Without prejudging the COE; it won't surprise anyone to learn that there are bad actors out there who try "every trick in the book" to have accounts that they don't pay for, and lying to customer support is absolutely one of those tricks, as is trying to be creative with changing payment instruments. In these cases, it's also really important that customer support stick to a script and can't be abused as part of soci…

Right, obviously you can't give the level-1 schlubs the keys to the kingdom, but they need to be able to escalate. What you're doing now, trapping customers in a maze of no-reply dead ends, isn't OK. It's never a good long-term play to let bad actors drive your business model. (Well, all right, maybe PayPal has to do that, but you don't.)

One obvious approach would be to charge for access to human support. I'll bet the OP would happily have paid $50 to talk to someone with both the ability and inclination to escalate the issue. In rare instances such as this one where the problem really is on your end, the $50 would be refunded.

Re: AWS Restored My Account: The Human Who Made the Difference

#56

Post by gamblor956 »

seuros wrote: post #24
qualeed wrote: post #12

Awesome that the author got their stuff back. But... > My data is back. Not because of viral pressure. Not because of bad PR. [...] > “I am devastated to read on your blog about the deletion of your AWS data. I did want to reach out to let you know that people in positions of leadership, such as my boss, are aware of your blog post and I’ve been tasked with finding out what I can, and to at least prevent this from ha…

The AWS employee actually contacted me before my blog post even reached three digits in views. So no, it wasn’t PR-driven in the reactive sense. But here’s what I learned from this experience: If you are stuck in a room full of deaf people, stop screaming, just open the door and go find someone who can hear you. The 20 days of pain I went through, it wasn’t because AWS couldnt fix it. It’s because I believed that one…

It took someone from outside the ticketing system to actually listen and say: Wait. This makes no sense.

Which only happened because of your blog post. In other words, the effort to prevent bad PR led to them fixing your problem immediately, while 20 days of doing things the "right" way yielded absolutely no results.

This actually makes the problem you've described even worse: it indicates that AWS has absolutely no qualms about failing to properly support the majority of its customers.

The proper thing for them to do was not to have a human "outside the system" fix your problem. It was for them to fix the system so that the system could have fixed your problem.

That being said: Azure is so much worse than AWS. Even bad PR won't push them to fix things.

Re: AWS Restored My Account: The Human Who Made the Difference

#57

Post by belter »

seuros wrote: post #24
qualeed wrote: post #12

Awesome that the author got their stuff back. But... > My data is back. Not because of viral pressure. Not because of bad PR. [...] > “I am devastated to read on your blog about the deletion of your AWS data. I did want to reach out to let you know that people in positions of leadership, such as my boss, are aware of your blog post and I’ve been tasked with finding out what I can, and to at least prevent this from ha…

The AWS employee actually contacted me before my blog post even reached three digits in views. So no, it wasn’t PR-driven in the reactive sense. But here’s what I learned from this experience: If you are stuck in a room full of deaf people, stop screaming, just open the door and go find someone who can hear you. The 20 days of pain I went through, it wasn’t because AWS couldnt fix it. It’s because I believed that one…

AWS Support absolutely fumbled the incident, but what you should have learned from the experience, and the majority of others commenting here is: Running a business critical workload in one AWS account is a self-inflicted single point of failure. Using separate accounts for prod/dev/test (and a break-glass account) is one of #1 security best practices:

“SEC01-BP01 Separate workloads using accounts.” - https://docs.aws.amazon.com/wellarchitected/latest/security-...

Keep resources out of the payer/management accounts. Consolidated billing is fine, but the management account should stay empty.

"Best practices for the management account" - https://docs.aws.amazon.com/organizations/latest/userguide/o...

Enable cross-account backups. Copy snapshots or AWS Backup vaults to a second account so Support lockouts don’t equal data loss.

"Creating backup copies across AWS accounts" - https://docs.aws.amazon.com/aws-backup/latest/devguide/creat...

Populate Billing, Security, and Ops alternate contacts. AWS Support escalates to those addresses when the primary inbox is dead. "Update the alternate contacts for your AWS account" - https://docs.aws.amazon.com/accounts/latest/reference/manage...

Follow the multi-account white-paper for long-term org design. It is not optional reading. "Organizing Your AWS Environment Using Multiple Accounts" - https://docs.aws.amazon.com/whitepapers/latest/organizing-yo...

Maybe get some training?

https://aws.amazon.com/training/classroom/

Re: AWS Restored My Account: The Human Who Made the Difference

#58

Post by [deleted] »

qualeed wrote: post #12

Awesome that the author got their stuff back. But... > My data is back. Not because of viral pressure. Not because of bad PR. [...] > “I am devastated to read on your blog about the deletion of your AWS data. I did want to reach out to let you know that people in positions of leadership, such as my boss, are aware of your blog post and I’ve been tasked with finding out what I can, and to at least prevent this from ha…

[deleted]

Re: AWS Restored My Account: The Human Who Made the Difference

#59

Post by [deleted] »

qualeed wrote: post #37
bigstrat2003 wrote: post #30

Earlier quoted context omitted.

I think the blog made a difference, yes - but that doesn't mean it was just a PR move by Amazon. It's perfectly possible that the Amazon employee who contacted the author truly does care and wanted to help because of that. It's fair to say that without the blog post this issue wouldn't have been noticed or fixed, but anything past that is really just speculating about people's motives.

> but that doesn't mean it was just a PR move by Amazon Being a PR move isn't inherently a bad thing. The bad thing is the lack of support when PR isn't at risk. > It's fair to say that without the blog post this issue wouldn't have been noticed or fixed, but anything past that is really just speculating about people's motives. My only (minor) issue with the blog post is starting by saying "Not because of PR" when th…

[deleted]
Post reply on HN