AWS Restored My Account: The Human Who Made the Difference • Hacker News Board

Re: AWS Restored My Account: The Human Who Made the Difference

#41

Post by seuros » Thu, Aug 07, 2025, 4:46 PM UTC

qualeed wrote: post #31

seuros wrote: post #24

Earlier quoted context omitted.

The AWS employee actually contacted me before my blog post even reached three digits in views. So no, it wasn’t PR-driven in the reactive sense. But here’s what I learned from this experience: If you are stuck in a room full of deaf people, stop screaming, just open the door and go find someone who can hear you. The 20 days of pain I went through, it wasn’t because AWS couldnt fix it. It’s because I believed that one…

> So no, it wasn’t PR-driven in the reactive sense. At my small business, we proactively monitor blogs and forums for mentions of our company name so that we can head off problems before they become big. I'm extremely confident that is what happened here. It was PR-driven in the proactive sense. Which is still PR-driven. (which, by the way, I have no problem with! the problem is the shitty support when it isn't PR-dr…

Thank you for your concern, and I appreciate the nuance in your take.

Yes, it is totally possible that AWS monitors blogs and forums for early damage control, like your company does.

But we shouldn’t paint it like I was bailed out by some algorithmic PR radar and nothing else.

Let’s not fall into the “Fuk the police” style of thinking where every action is assumed to be manipulation. Tarus didn’t reach out like a Scientology agent demanding I take the post down or warning me of consequences.

He came with empathy, internal leverage, and actually made things move.

When before i read Tarus email, i wrote in Slack to Nate Berkopec (puma maintainer): `Hi. AWS destroyed me, i'm going to take a big break .`

Then his email reset my cortisol levels to acceptable level.

Most importantly, this incident triggered a CoE (Correction of Error) process inside AWS.

That means internal systems and defaults are being reviewed, and that’s more than I expected. We’re getting a real update, that will affect cases like mine in the future.

So yeah, it may have started in the visibility layer, but what matters is that someone human got involved, and actual change is now happening.

Re: AWS Restored My Account: The Human Who Made the Difference

#42

Post by Timshel » Thu, Aug 07, 2025, 4:48 PM UTC

colmmacc wrote: post #20

eqvinox wrote: post #11

> "people in positions of leadership, such as my boss, are aware of your blog post and I’ve been tasked with finding out what I can" Translation: "someone noticed it trending on HN, decided it was bad publicity, and that they should do something about it" Implication: what mattered was the bad publicity, not the poor support infrastructure. The latter won't change, and the next person with similar problems will get t…

Every week at AWS we have an account protection meeting; it's got the teams who handle fraud and abuse detection, compromised accounts (e.g. when a customer has a security issue on their side), non-payment, forgotten creds, as well as our support team. You'll find the most junior members of those teams, all the way up to VPs, in the same meeting diving into the nitty gritty. Every week. Disabling a legitimate in-use…

> Disabling a legitimate in-use account is one of our absolute nightmares

Might be your nightmare but at the same time there is no way for your customers to report it or your own support agents to escalate that something wrong might have happened and someone should look again ...

Re: AWS Restored My Account: The Human Who Made the Difference

#43

Post by colmmacc » Thu, Aug 07, 2025, 4:50 PM UTC

CamperBob2 wrote: post #32

colmmacc wrote: post #20

Earlier quoted context omitted.

Every week at AWS we have an account protection meeting; it's got the teams who handle fraud and abuse detection, compromised accounts (e.g. when a customer has a security issue on their side), non-payment, forgotten creds, as well as our support team. You'll find the most junior members of those teams, all the way up to VPs, in the same meeting diving into the nitty gritty. Every week. Disabling a legitimate in-use…

So, the key is somehow finding a way to bring problems like this to the group's attention, it sounds like. HN lamentations are great when they work, but they don't exactly scale. At least one layer of human support needs to have the ability -- not just the ability, but the obligation! -- to escalate to your team when a customer service problem occurs that doesn't fit a pattern of known/active scams and they are unabl…

Without prejudging the COE; it won't surprise anyone to learn that there are bad actors out there who try "every trick in the book" to have accounts that they don't pay for, and lying to customer support is absolutely one of those tricks, as is trying to be creative with changing payment instruments.

In these cases, it's also really important that customer support stick to a script and can't be abused as part of social engineering, hijacking, or fraud check bypass. "No we can't reset your account" is a very important protection too. I agree that there is an obligation to escalation, but I suspect the focus of the COE will be on how we could have detected this without human judgement. There's got to be a way.

Re: AWS Restored My Account: The Human Who Made the Difference

#44

Post by anonymars » Thu, Aug 07, 2025, 4:50 PM UTC

electroly wrote: post #14

Lesson learned: If you have important workloads in AWS, but don't spend enough to get a dedicated account rep, make sure you have some sort of online presence that you can use later to gain access to the real support group.

Primary lesson learned should be NEVER HAVE ONE POINT OF FAILURE. If all your data is on one account (regardless of where) and nowhere else that is ONE POINT OF FAILURE

Re: AWS Restored My Account: The Human Who Made the Difference

#45

Post by anonymars » Thu, Aug 07, 2025, 4:53 PM UTC

diegocg wrote: post #33

I would say that the lesson here is that cross-vendor replication is more important than intra-vendor replication. It is clear that technology can (largely) avoid data losses, but there will always be humans at charge

Nitpick: true replication is high-availability, not disaster-recovery (i.e. not a backup)

If wrong data gets deleted, and that gets replicated, now you simply have two copies of bad data.

Re: AWS Restored My Account: The Human Who Made the Difference

#46

Post by qualeed » Thu, Aug 07, 2025, 4:54 PM UTC

seuros wrote: post #41

qualeed wrote: post #31

Earlier quoted context omitted.

> So no, it wasn’t PR-driven in the reactive sense. At my small business, we proactively monitor blogs and forums for mentions of our company name so that we can head off problems before they become big. I'm extremely confident that is what happened here. It was PR-driven in the proactive sense. Which is still PR-driven. (which, by the way, I have no problem with! the problem is the shitty support when it isn't PR-dr…

Thank you for your concern, and I appreciate the nuance in your take. Yes, it is totally possible that AWS monitors blogs and forums for early damage control, like your company does. But we shouldn’t paint it like I was bailed out by some algorithmic PR radar and nothing else. Let’s not fall into the “Fu k the police” style of thinking where every action is assumed to be manipulation. Tarus didn’t reach out like a Sc…

>But we shouldn’t paint it like I was bailed out by some algorithmic PR radar and nothing else.

>[...] assumed to be manipulation

I think you're reading way more negativity into "PR" than I'm intending (which is no negativity).

It's very clear Tarus is a caring person who really did empathize with your situation and did their best to rectify the situation. It's not a bad thing that your issue may (most likely) have been brought to his attention because of "PR radar" or whatever.

The bad part, on Amazon and other similar companies, is how they typically respond when a potential PR hit isn't on the line. Which, as I'm sure you know because you experienced it prior to posting your blog, is often a brick wall.

The overwhelming issue is that you often require some sort of threat of damage to their PR to be assisted. That doesn't make the PR itself a bad thing. And that fact implies nothing about the individuals like Tarus who care. Often the lowly tier 1 support empathizes, they just aren't allowed to do anything or say anything.

Re: AWS Restored My Account: The Human Who Made the Difference

#47

Post by Timshel » Thu, Aug 07, 2025, 5:01 PM UTC

colmmacc wrote: post #43

CamperBob2 wrote: post #32

Earlier quoted context omitted.

So, the key is somehow finding a way to bring problems like this to the group's attention, it sounds like. HN lamentations are great when they work, but they don't exactly scale. At least one layer of human support needs to have the ability -- not just the ability, but the obligation! -- to escalate to your team when a customer service problem occurs that doesn't fit a pattern of known/active scams and they are unabl…

Without prejudging the COE; it won't surprise anyone to learn that there are bad actors out there who try "every trick in the book" to have accounts that they don't pay for, and lying to customer support is absolutely one of those tricks, as is trying to be creative with changing payment instruments. In these cases, it's also really important that customer support stick to a script and can't be abused as part of soci…

I love the irony that an issue caused by a failing automation was solved due to human escalation but let's not try to improve the escalation process but add more automation ...

Re: AWS Restored My Account: The Human Who Made the Difference

#48

Post by bink » Thu, Aug 07, 2025, 5:03 PM UTC

mox1 wrote: post #23

electroly wrote: post #14

Lesson learned: If you have important workloads in AWS, but don't spend enough to get a dedicated account rep, make sure you have some sort of online presence that you can use later to gain access to the real support group.

...or ensure you have backups of data in a non-AWS location?

With modern cloud computing is that enough? Can you migrate AuroraDBs, Lambdas, Kinesis configs, IAM policies, S3 bucket configs, Opensearch configs to another cloud platform easily? I suppose if you're comfortable going back to AWS after they randomly delete all your data then the remote backups will be helpful, but not so much if you plan to migrate to another provider.

Re: AWS Restored My Account: The Human Who Made the Difference

#49

Post by bink » Thu, Aug 07, 2025, 5:07 PM UTC

Traubenfuchs wrote: post #25

jacquesm wrote: post #16

One of the questions we always ask of cloud hosted companies what their plan is in case they ever lose their cloud account. Typically this is met with incredulity: nobody has ever lost their AWS account and all it's data? Right? Right??? Well, not normally, no. But it does happen. Not often enough to be a meaningful statistical issue, but if it were to happen to you then a little forethought can turn a complete disas…

We have a slack channel with more than 20 external @amazon.com engineers and salespeople that instantly respond to all our queries, sometimes proactively inform us of stuff or offer advice... Doesn't everyone has this?

I think you'll find this depends on how much you spend with Amazon. Most accounts (by number) don't have this.

Re: AWS Restored My Account: The Human Who Made the Difference

#50

Post by seuros » Thu, Aug 07, 2025, 5:07 PM UTC

Author here: Let’s be clear on backups:

Yes, I had backups everywhere. Across providers, in different countries. But I built a system tied to my AWS account number, my instances, my IDs, my workflows.

When that account went down, all those “other” backups were just dead noise encrypted forever. Bringing them up to the story only invites the 'just use your other backups' fallback, and ignores the real fragility of centralized dependencies.

It is like this: the UK still maintains BBC Radio 4’s Analogue Emergency Broadcast—a signal so vital that if it’s cut, UK nuclear submarines and missile silos automatically trigger retaliation. No questions asked. That's how much stakes they place on a reliable signal.

If your primary analogue link fails, the world ends. That's precisely how I felt when AWS pulled my account, because I’d tied my critical system to a single point of failure. If the account was just Read only, i will waited because i could have access to my data and rotated keys.

AWS is the apex cloud provider on the planet. This isnt about redundancy or best practices.

it's about how much trust and infrastructure we willingly lend to one system.

Remember that if BBC Radio 4 signal get to fail for some reasons, the world will get nuked, only cockroaches will survive… and your RDS and EC2 billing fees.