I’ve just received a budget alert for $25 million! I also created an urgent ticket but haven’t heard back.
AWS: Inaccurate Estimated Billing Data – $1.7 billion
771–780 of 793 posts
Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion
#772Probably the closest I've ever been to getting a heart attack. Normally <$1 per month, and now suddenly $284,006,266,443.74. Whatever the bug is on their end, this is unforgivable.
Same here. I got an email with a bill of $233 million and an estimated $433 million until the end of the month. I panicked and nuked my entire setup (which wasn't used that much, anyway, the alert threshold was $1) - I really wonder how many people did the same. It's been 2 hours and I still haven't fully calmed down.
I did the same! I was trembling all day!
Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion
#773My estimate was only $21M (vs ~$0.01 average bill). Wish I had checked status sooner and saved myself the panic! My process went: verify email is not phishing (it was), login to console and check dashboard (same amount), attempt to understand cost (cost management kept contradicting itself), try to log support ticket and only on that part did I notice the status notification. At least I can breathe again now!
same process, same near heart attack - especially some days ago I had malicious activity on my platform and thought that now they actually found something. Apparently you can trigger an Action (e.g. prevent uploads) when the billing alert triggers, but then my platform wouldn't work anymore, just because AWS had an issue. Also insane that Amazon still hasn't send an email to clarify.
News to me! Since my usage is just a hobby app I’m definitely implementing this action response - thank you
Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion
#774Earlier quoted context omitted.
That job description feels so far beyond parody that I could scarcely believe it until opening the link! What a world.
It gets worst: "Senior Software Development Manager, AWS Global Bill Generation" https://www.amazon.jobs/de/jobs/10471948/senior-software-dev... "We're transforming from monthly batch processing and manual war rooms to continuous billing, autonomous agents, and self-healing infrastructure. We believe operational burden is a technical problem, not a staffing problem" This looks clearly...a staffing problem...
> This looks clearly...a staffing problem...
And if that weren't enough, the "manual war rooms" alone should be setting off alarm bells.
Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion
#775It's crazy enough this will be fixed soon. Years ago I found an actual hidden error in my bill. (This was early 2010s). The system was calculating the EC2 reservation savings incorrectly for some of my servers. I was crunching all their detailed usage data on a regular basis in an 18 tab spreadsheet and couldn't get it to fully reconcile. I spent months trying to track down the discrepancy. Once I found it, I had to…
A couple of my coworkers think I’m nuts for watching cost explorer so closely but 1. The time it takes to look and notice costs that don’t make sense easily pays for itself, and then some (in my experience). I doubt you spent $7k of your time tracking this down, and you probably noticed optimization opportunities that saved you even more 2. I hate the idea of wasting money on buying Jeff Bezos a bigger yacht
Do you just watch for trends? Curious to know your process to verify if the cost is right or wrong.
Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion
#776I got a 20K bill once and it was actually drafted from my bank account. It took me a couple of months and involving the office of the AG of my state to get the issue resolved and get my money back. Since then I never touched any AWS product, moved my small stuff to Azure. It’s been years since AWS have these issues with billing, you can find the stories online, students billed 60K for a compromised account launching…
Right, and good luck getting a correct bill from Azure. And when you are finally fed up, it will take months to close your Azure account.
Exactly. It seems like a step down in every way.
Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion
#777Earlier quoted context omitted.
Not even tests but just some basic anomaly detection lol. Like maybe if the bill amounts increase by like 10M% there should be someone that looks into it
While I didn't work on AWS, I did intern on the retail side of Amazon, and there's definitely this sort of monitoring in place. Surely somebody was paged. And even if not, this is "just" the cost explorer estimations, not what is ending up on folk's bills. I learned about https://en.wikipedia.org/wiki/2011_T%C5%8Dhoku_earthquake_an... > from alarms like this, as sales in Japan almost entirely stopped. I've been told…
> there's definitely this sort of monitoring in place. Surely somebody was paged.
Well you’re half right. Either there wasn’t monitoring for this or if there was monitoring in place that’s not what caught this, because the page originated from a customer support ticket.
Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion
#778Woke up to a billing alarm email. Thought I had leaked my AWS keys accidentally and somehow run up 437 billion dollars of charges. Joke's on them though, I don't have 437 billion dollars Anyways I didn't need coffee. That produced an adrenaline release unlike any I've experienced before. Thanks AWS
huge irl laugh at “i don’t have 437 billion dollars”
“Best I can do for you is $3.75 and a cup of coffee.”
Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion
#779[flagged]
Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion
#780[flagged]