This is just the cloud area, what if Amazon starts vibe charging regular customers because of some bug? Accounts that are directly linked with regular people's payment methods?
AWS: Inaccurate Estimated Billing Data – $1.7 billion
391–400 of 793 posts
Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion
#392Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion
#393Question: Why does AWS need to roll back estimated bills to a "last known good" state? I get wanting to do that for ACTUAL billing mistakes, but for estimates, they're just that -- approximations. I guess it's fine for predictive purposes to store estimates so they can be compared to actual usage and optimized. But why would AWS bind the values of present estimates to the estimates made earlier in the month. The calculation should always be:
1. Current month's usage * applicable rates; + 2. Estimated future usage for the month * applicable rates.
And Item 1 obviously requires proper data persistence, but Item 2 is just a projection. If they don't have Item 1 correct, AWS's whole system is in question, but I don't think that's the issue. I'm going to guess now -- looking forward to reading the root cause analysis -- that the problem is that someone decided to get too fancy with the estimates, and built a latent requirement that all prior estimates for the month must be available to compute the current estimate. Without estimates working, no estimates are available, and some denominator in an averaging or smoothing or normalizing function goes to 0; then everyone's estimated bill explodes without bound (subject to floating-point arithmetic) resulting in trillion-dollar estimates.
Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion
#394Earlier quoted context omitted.
Can you not set spending limits in AWS?
No you can't. Spending limits imply realtime billing backend flows and they also imply deleting all your data so that you don't pay for storage.
> and they also imply deleting all your data so that you don't pay for storage.
Not necessarily. They could imply that your storage becomes inaccessible immediately, but only gets deleted after some time period (say, 1 month). What spending limits do depends on the implementation.
Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion
#395I am running a niche SaaS with around 20 users per day on AWS.
I too was shocked when I saw the $1.7billion bill, instead of the usual $1.5billion.
Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion
#396Earlier quoted context omitted.
Wanna bet the description of this job post will be updated by the end of the day? "Software Development Engineer II, AWS Invoicing" https://www.amazon.jobs/de/jobs/10428480/software-developmen... "...Our platforms are powered by generative AI, large language models, knowledge graphs, and agentic architectures that dynamically compose specialized agents based on context. We apply these capabilities across three reinfo…
That job description feels so far beyond parody that I could scarcely believe it until opening the link! What a world.
> enabling domain experts to review in hours what previously took weeks.
This is a gold-mine. They need to get sued heavily for this incompetence.
Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion
#397Pff rookie numbers, mine was 375 billion.
Small potato's sir, my bill > GDP of Switzerland. A cool $1.2T
Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion
#398Several comments here talk about "nearly" having a heart attack. But I wonder: since it's happened to so many people, chances are someone had a heart attack for real. Can they legally be made responsible for that?
When someone says "I nearly had a heart attack," it's _highly_ unlikely that they actually nearly had a heart attack. I don't think the chances are good that anyone actually had a heart attack.
While I agree that this phrase is most commonly spoken in a figurative sense, folks with marginal heart health have a much greater than 0% chance of having something of this nature trigger an actual heart attack.
Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion
#399Who else had LinkedIn posts about this flashing before your eyes?
Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion
#400It's crazy enough this will be fixed soon. Years ago I found an actual hidden error in my bill. (This was early 2010s). The system was calculating the EC2 reservation savings incorrectly for some of my servers. I was crunching all their detailed usage data on a regular basis in an 18 tab spreadsheet and couldn't get it to fully reconcile. I spent months trying to track down the discrepancy. Once I found it, I had to…
A couple of my coworkers think I’m nuts for watching cost explorer so closely but
1. The time it takes to look and notice costs that don’t make sense easily pays for itself, and then some (in my experience). I doubt you spent $7k of your time tracking this down, and you probably noticed optimization opportunities that saved you even more
2. I hate the idea of wasting money on buying Jeff Bezos a bigger yacht