Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS Restored My Account: The Human Who Made the Difference

seuros.com

Post reply on HN
Print view View original

31–40 of 95 posts

Re: AWS Restored My Account: The Human Who Made the Difference

#31

Post by qualeed »

seuros wrote: post #24
qualeed wrote: post #12

Awesome that the author got their stuff back. But... > My data is back. Not because of viral pressure. Not because of bad PR. [...] > “I am devastated to read on your blog about the deletion of your AWS data. I did want to reach out to let you know that people in positions of leadership, such as my boss, are aware of your blog post and I’ve been tasked with finding out what I can, and to at least prevent this from ha…

The AWS employee actually contacted me before my blog post even reached three digits in views. So no, it wasn’t PR-driven in the reactive sense. But here’s what I learned from this experience: If you are stuck in a room full of deaf people, stop screaming, just open the door and go find someone who can hear you. The 20 days of pain I went through, it wasn’t because AWS couldnt fix it. It’s because I believed that one…

>So no, it wasn’t PR-driven in the reactive sense.

At my small business, we proactively monitor blogs and forums for mentions of our company name so that we can head off problems before they become big. I'm extremely confident that is what happened here.

It was PR-driven in the proactive sense. Which is still PR-driven. (which, by the way, I have no problem with! the problem is the shitty support when it isn't PR-driven)

Regardless, I 100% feel your pain with dealing with support agents that won't break script, and I am legitimately happy that you both got to reach someone that was high enough up the ladder to act human and that they were able to restore your data.

Re: AWS Restored My Account: The Human Who Made the Difference

#32

Post by CamperBob2 »

colmmacc wrote: post #20
eqvinox wrote: post #11

> "people in positions of leadership, such as my boss, are aware of your blog post and I’ve been tasked with finding out what I can" Translation: "someone noticed it trending on HN, decided it was bad publicity, and that they should do something about it" Implication: what mattered was the bad publicity, not the poor support infrastructure. The latter won't change, and the next person with similar problems will get t…

Every week at AWS we have an account protection meeting; it's got the teams who handle fraud and abuse detection, compromised accounts (e.g. when a customer has a security issue on their side), non-payment, forgotten creds, as well as our support team. You'll find the most junior members of those teams, all the way up to VPs, in the same meeting diving into the nitty gritty. Every week. Disabling a legitimate in-use…

So, the key is somehow finding a way to bring problems like this to the group's attention, it sounds like. HN lamentations are great when they work, but they don't exactly scale.

At least one layer of human support needs to have the ability -- not just the ability, but the obligation! -- to escalate to your team when a customer service problem occurs that doesn't fit a pattern of known/active scams and they are unable to help the customer themselves. Sounds like that's not currently the case.

Re: AWS Restored My Account: The Human Who Made the Difference

#34

Post by igleria »

mkl wrote: post #17
igleria wrote: post #10

I really want to send a virtual hug to Tarus and you. Reminds me of one of the few reasons I'm still in this industry... solving real problems that real people have.

*Tarus. Auto-incorrect?

Ugh, I should be better at keeping names read on other browser tabs in my meat volatile memory

Re: AWS Restored My Account: The Human Who Made the Difference

#35

Post by seuros »

electroly wrote: post #14

Lesson learned: If you have important workloads in AWS, but don't spend enough to get a dedicated account rep, make sure you have some sort of online presence that you can use later to gain access to the real support group.

That might be a 'lesson', but it’s like saying:

"If you want your paperwork processed in Morocco, make sure you know someone at the commune, and ideally have tea with their cousin."

Yes, it works, but it shouldn’t be the system.

What happened with AWS isn’t a clever survival tip, it’s proof that without an account manager, you are just noise in a ticket queue, unless you bring social proof or online visibility.

This should have never come down to 'who you know' or 'how loud you can go online'.

It a big luck that i'm speaking in english and have online presence, what if i was ranting in French, Arabic, or even Darija in Facebook. Tarus will have never noticed.

Re: AWS Restored My Account: The Human Who Made the Difference

#36

Post by ke4qqq »

eqvinox wrote: post #11

> "people in positions of leadership, such as my boss, are aware of your blog post and I’ve been tasked with finding out what I can" Translation: "someone noticed it trending on HN, decided it was bad publicity, and that they should do something about it" Implication: what mattered was the bad publicity, not the poor support infrastructure. The latter won't change, and the next person with similar problems will get t…

As the cited ‘boss’ I’ll say the publicity wasn’t the concern. The concern was that someone wanted to use our services and we had made that so frustrating that they were writing blog posts about how it had gone wrong.

The various teams (anti-fraud and support) are investigating how we failed this customer so we can improve and hopefully keep this from happening again. (This is the ‘Correction of Error’ process that’s being worked on. And CoE’s aren’t a punitive ‘blame session’ - it’s figuring out how a problem happened and how we can fix or avoid it systemically going forward).

To be fair, the publicity did mean that multiple people were flagging this and driving escalations around it.

Re: AWS Restored My Account: The Human Who Made the Difference

#37

Post by qualeed »

bigstrat2003 wrote: post #30
qualeed wrote: post #12

Awesome that the author got their stuff back. But... > My data is back. Not because of viral pressure. Not because of bad PR. [...] > “I am devastated to read on your blog about the deletion of your AWS data. I did want to reach out to let you know that people in positions of leadership, such as my boss, are aware of your blog post and I’ve been tasked with finding out what I can, and to at least prevent this from ha…

I think the blog made a difference, yes - but that doesn't mean it was just a PR move by Amazon. It's perfectly possible that the Amazon employee who contacted the author truly does care and wanted to help because of that. It's fair to say that without the blog post this issue wouldn't have been noticed or fixed, but anything past that is really just speculating about people's motives.

>but that doesn't mean it was just a PR move by Amazon

Being a PR move isn't inherently a bad thing.

The bad thing is the lack of support when PR isn't at risk.

>It's fair to say that without the blog post this issue wouldn't have been noticed or fixed, but anything past that is really just speculating about people's motives.

My only (minor) issue with the blog post is starting by saying "Not because of PR" when the opening email from the human at amazon was "saw your blog". I think it is evident that Tarus Balog did indeed actually care!

Re: AWS Restored My Account: The Human Who Made the Difference

#38

Post by electroly »

seuros wrote: post #35
electroly wrote: post #14

Lesson learned: If you have important workloads in AWS, but don't spend enough to get a dedicated account rep, make sure you have some sort of online presence that you can use later to gain access to the real support group.

That might be a 'lesson', but it’s like saying: "If you want your paperwork processed in Morocco, make sure you know someone at the commune, and ideally have tea with their cousin." Yes, it works, but it shouldn’t be the system. What happened with AWS isn’t a clever survival tip, it’s proof that without an account manager, you are just noise in a ticket queue, unless you bring social proof or online visibility. This…

We seem to be saying exactly the same thing. I agree, strongly, with everything you just said here. AWS has a support problem if this was necessary, and I'm not personally prepared with an online presence if it happened to me. I'll simply be screwed and will have to recreate a new account from backups. It's something for me to think about. I can't fix AWS--I can only change what I do in response.

I recently opened a DigitalOcean account and it was locked for a few days after I had moved workloads in. They took four days to unlock the account, and for my trouble they continued to charge me for my resources during the time the account was locked when I couldn't log in to delete them. I didn't have any recourse at all. They did issue a credit because I asked nicely, but if they said no, that would have been it.

Re: AWS Restored My Account: The Human Who Made the Difference

#39

Post by colmmacc »

lotsofpulp wrote: post #29
colmmacc wrote: post #20

Earlier quoted context omitted.

Every week at AWS we have an account protection meeting; it's got the teams who handle fraud and abuse detection, compromised accounts (e.g. when a customer has a security issue on their side), non-payment, forgotten creds, as well as our support team. You'll find the most junior members of those teams, all the way up to VPs, in the same meeting diving into the nitty gritty. Every week. Disabling a legitimate in-use…

It might not be for you, but it is an acceptable cost of doing business for whoever is high enough up the chain that is setting the budget such that sufficient customer support is not available. And disabling an in use account was not the issue here. There not being a way to get the account re enabled is the issue.

In this case, I am high enough in that chain.

Re: AWS Restored My Account: The Human Who Made the Difference

#40

Post by qualeed »

boogieknite wrote: post #28
qualeed wrote: post #12

Awesome that the author got their stuff back. But... > My data is back. Not because of viral pressure. Not because of bad PR. [...] > “I am devastated to read on your blog about the deletion of your AWS data. I did want to reach out to let you know that people in positions of leadership, such as my boss, are aware of your blog post and I’ve been tasked with finding out what I can, and to at least prevent this from ha…

much much much smaller scale but i accidentally signed up for some AWS certificate service i didnt understand or use and took a $700 bill on the chin customer service was great and refunded my money without me blogging about it. we messaged back and forth about what i was trying to do and what i thought i was signing up for. i think it helped to have a long history of tiny aws instances because they mentioned reviewi…

Money is typically much easier to deal with than data or account restoration. With that said, it's nice to hear a good support story every now and then! People don't usually take the time to write a blog post (or comment, etc.) when they have a quick and productive conversation with a support agent, so all we end up hearing are horror stories.
Post reply on HN