Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS deleted my 10-year account and all data without warning

seuros.com

Post reply on HN
Print view View original

121–130 of 173 posts

Re: AWS deleted my 10-year account and all data without warning

#121

Post by seuros »

davidhyde wrote: post #79
gblargg wrote: post #50

Earlier quoted context omitted.

The author doesn't grasp what putting all your eggs into one basket means: > Before anyone says “you put all your eggs in one basket,” let me be clear: I didn’t. I put them in one provider, with what should have been bulletproof redundancy: That's one basket. A single point of failure. "But it should have been impossible to fail!" Backups are to handle the "impossible" failure (in reality nothing is 100% reliable).

This one time, traveling through Asia, a simple merchant transaction triggered a fraud alarm on my card. The default for my bank at the time was to cancel my card automatically. This was before the days where cards could become unblocked. I had to travel to another city to pick up a new card in 10 working ways. This was a Mastercard credit card. I thought I was smart traveling with both a mastercard and a Visa card.…

My post is not about backup strategy, it’s about what happens when the infrastructure itself becomes hostile, and support throw you from one team to another. AWS didn't just delete files.

They gaslit me for 20 days while violating their own stated policies.

Re: AWS deleted my 10-year account and all data without warning

#122

Post by seuros »

Tohsig wrote: post #77
gblargg wrote: post #50

Earlier quoted context omitted.

The author doesn't grasp what putting all your eggs into one basket means: > Before anyone says “you put all your eggs in one basket,” let me be clear: I didn’t. I put them in one provider, with what should have been bulletproof redundancy: That's one basket. A single point of failure. "But it should have been impossible to fail!" Backups are to handle the "impossible" failure (in reality nothing is 100% reliable).

Fully agree. It's the same reason why you wouldn't put a repo on Github and then mirror that repo to Github . At a minimum any good mirror would be on a different provider, and ideally (if we get real picky) on a completely different cloud service.

When you put a repo in Github, everybody that forked or clone that repo become a Mirror.

Re: AWS deleted my 10-year account and all data without warning

#123

Post by seuros »

yongjik wrote: post #81
blargey wrote: post #18

Earlier quoted context omitted.

The lede buried under that lede is that (according to an insider?) some AWS employee accidentally wiped everything immediately (contrary to typical practice in such situations of retaining data while things get sorted out), leading to a chain of brushing-off / covering-up percolating through whatever support chain the OP was talking to.

I find that story really hard to believe. In a company the size of Amazon, I can't imagine a rogue employee running a tool that can willy-nilly wipe out customers' data without several levels of manager approval. Besides, what would be the potential benefit of such a hypothetical script? The author mentions "a bill under $200," so that's the upper limit on how much it costs AWS to keep the author's whole data. If I w…

The idea of a rogue AWS team running a deletion script without oversight should sound ridiculous. At a company that size of AWS, you will expect guardrails, audits, approvals.

But here is the thing: no one from AWS has given me an official explanation. Not during the 20-day support hell, not after termination, not even when I asked directly: “Does my data still exist?” Just a slow drip of templated replies, evasions, and contradictions.

An AWS insider did reach out claiming it was an internal test gone wrong, triggered by a misused --dry flag and targeting At a company that size, you'd expect guardrails, audits, approvals low-activity accounts.

According to them, the team ran it without proper approval. Maybe it is true. Maybe they were trying to warn me. Maybe its a trap to get me to throw baseless accusations and discredit myself.

I'm not presenting that theory as fact. I don’t know what happened behind the wall.

What I do know is:

- My account was terminated without following AWS’s own 90-day retention policy

- I had a valid payment method on file

- Support stonewalled every direct question for 20 days

- No answers were provided, even post-mortem

Re: AWS deleted my 10-year account and all data without warning

#124

Post by seuros »

hyperman1 wrote: post #55

The java command line theory seems strange to me. Java has no standard command line parser in the JDK (Standard library). Apache commons cli probably comes closest to a standard, and they support --gnu-style-long-options just like everybody else. The jvm(runtime) has some long non-POSIX options, but that's not very relevant here.

Yeah, it's fishy. I never claimed the Java theory is confirmed, just that it’s what an insider told me after the fact.

They said a dry-run flag was passed in the --gnu-style form, but the internal tool expected -dry, and since Java has no native CLI parser, it just ignored it and ran for real.

Supposedly the dev team was used to Python-style CLIs, and this got through without proper testing.

Re: AWS deleted my 10-year account and all data without warning

#125

Post by PartiallyTyped »

akerl_ wrote: post #19
PartiallyTyped wrote: post #10

Earlier quoted context omitted.

That is wrong in every way possible. Each service is isolated, has its own control plane, and often even split into multiple cells. Source: I worked there. https://docs.aws.amazon.com/wellarchitected/latest/reducing-...

You’ve missed the point by leaning into your pedantry. The point is that from the perspective of vendor/platform risk (things like account closure, billing disputes, etc), all of AWS is the same basket, even if you use multiple regions and services.

I dismissed the claim that they share a single control panel across services because it is false.

I dismissed that claim alone and talked about nothing else. The comment I replied to added nothing to the conversation except an unsubstantiated and plainly false remark.

> Wild that people don't realize that these "separate" systems in AWS all share things like the same control plane.

This comment does not say anything about all being in one basket. It only claims that all services run the same control plane which is so far from truth it must have been made from somebody utterly unaware of how big cloud teams operate.

Seems that this community has lost the ability to reason and converse and would rather be outraged.

Re: AWS deleted my 10-year account and all data without warning

#126

Post by PartiallyTyped »

stefan_ wrote: post #28
PartiallyTyped wrote: post #10

Earlier quoted context omitted.

That is wrong in every way possible. Each service is isolated, has its own control plane, and often even split into multiple cells. Source: I worked there. https://docs.aws.amazon.com/wellarchitected/latest/reducing-...

Do all these cells and control planes run different software built by different teams? I mean, sure every Ford Pinto is strictly it's own vehicle, but each will predictably burst into flames when you impinge its fuel tank and I don't wanna travel on a company operating an all Pinto fleet.

Yes. And parts of each control plane are written by different teams and each component is isolated.

Re: AWS deleted my 10-year account and all data without warning

#127

Post by PartiallyTyped »

ImPostingOnHN wrote: post #36
senderista wrote: post #24

Earlier quoted context omitted.

Not sure why you’re being downvoted, this is 100% correct from someone else who worked there.

> Not sure why you’re being downvoted If you're not sure why someone is being downvoted, there's a good chance there is a misunderstanding somewhere, and a good way for you to understand is to wait for people to comment, and then read those comments. Alternatively (and in a more general sense), if you want to take a more active role in your learning, a good way to learn is to ask questions, and then read the answers.

There is no misunderstanding involved. HN is often too emotional and will downvote and upvote whatever supports their beliefs or outrage du jour.

Re: AWS deleted my 10-year account and all data without warning

#129

Post by throwaway413121 »

seuros wrote: post #119
spiralcoaster wrote: post #37

The amount of self-aggrandizing and lack of self awareness tells me this author is doing to do all of this again. This post could be summed up with "I should have had backups. Lesson learned", but instead they deflect to whining about how their local desktop is a mess and they NEED to store everything remotely to stay organized. They're going to dazzle you with all of their hardened bunker this, and multiple escape r…

I did have backups. Multi-region. Redundant. I followed AWS’s own best practices to the letter. The only failure I didn’t plan for? AWS becoming the failure. The provider nuking everything in violation of their own retention policies. That’s not a backup problem, that is a provider trust problem. The reason i did not kept a local copy, was that i formatted my computer after a hardware failure, after the nurse dropped…

As someone who has lost data myself, i'm really sorry this happened to you. I refrained from commenting on your article originally, but you seem stuck in a mental state blaming AWS for deleting your "backups" that you established with "best practices".

But you need to be aware that you never had backups in the way most sysadmins mean. If i need a friend to take care of a loved one while i'm away, and my backup plan is having the same person take them care of them but in a different house or with a different haircut, that's no backup plan: that's bus factor = 1.

Backups mean having a second (or third, etc) copy of your data stored with a 3rd party. Backup assumes you have an original copy of the entirety of the data to begin with.

From this point, and i'm sorry it bit you like this, but you never followed any good sysadmin practices about backups and disaster recovery. I have no idea what AWS best practices say, but trusting a single actor (whether hardware manufacturer or services provider) with all your data has always been against the 3-2-1 golden rule of backups and what happened to you was inevitable.

Blame AWS all you want, but Google does exactly the same thing all the time, deleting 15-years-old accounts with all associated data with no recourse. Some of us thought the cloud was safe and had all their "cross-region" backups burn in flames in OVH Strasbourg.

We could never trust cloud companies, and some of us never did. I never trusted AWS with my data, and i'm sorry you made that mistake, but you may also take the opportunity to learn how to handle backups properly in the future and never trust a single egg basket, or whatever metaphor is more appropriate.

Good luck to you in the future!

Re: AWS deleted my 10-year account and all data without warning

#130

Post by throwaway413121 »

seuros wrote: post #120
mcv wrote: post #109

Earlier quoted context omitted.

Yeah, at some point the article says: > I’d done everything right. Vault encryption keys stored separately from my main infrastructure. Defense in depth. Zero trust architecture. The works. Did you? Is putting all your eggs in one basket "defense in depth"? Is total trust in AWS "zero trust architecture"? I'm not defending AWS here; they fully deserve all the fallout they can get from this, and I do feel for the dev…

I dont disagree with your broader point—centralizing everything in one provider is a systemic risk. The architecture was built assuming infrastructure within AWS might fail. What I didn’t plan for was the provider itself turning hostile, skipping their own retention policy, and treating verification as a deletion trigger.

> The architecture was built assuming infrastructure within AWS might fail.

From what i gather it was not. Or did you have a strategy for a 0-warning complete AWS service closure? Just imagine AWS closing their doors from one day to the next due to economic losses, or due to judicial inquiries into their illegal practices: were you really prepared for their failure?

The cloud was never data living in tiny rain droplets and swimming across the earth to our clients. The cloud was always somebody else's computer(s) that they control, and we don't. I'm sorry you learnt that lesson the hard way.

Post reply on HN