Earlier quoted context omitted.
Update as of 7:53am PDT: "The rollback of a recent change did not resolve the issue and we are continuing to investigate multiple mitigation paths. Estimated bill updates remain paused."
>Estimated bill updates remain paused Wait what if someones actually getting usage spiked
AWS: Inaccurate Estimated Billing Data – $1.7 billion
461–470 of 793 posts
Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion
#462I got a 20K bill once and it was actually drafted from my bank account. It took me a couple of months and involving the office of the AG of my state to get the issue resolved and get my money back. Since then I never touched any AWS product, moved my small stuff to Azure. It’s been years since AWS have these issues with billing, you can find the stories online, students billed 60K for a compromised account launching…
Service provider lesson #1: Never ever ever enable auto-pay! The convenience (and even the savings, if applicable) aren't worth the risk of the service provider autonomously slurping up all your money.
Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion
#463They sent 3 warnings to my email, ok, I understand bugs happen (probably vibe-coded). But they didn't even send any notification that it's a bug. Going to leave AWS after that.
Somehow I highly doubt anyone will leave AWS over this unless their use of AWS is way more low-complexity than the average account. People make similar pronouncements after every us-east-1 outage makes the news, but I feel like AWS would be going out of business by now if people followed through. It reminds me of airlines, where after a particularly grueling irregular ops experience, a few dozen people file off the p…
Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion
#464Apparently what used to be `GB of storage consumed` is confused with `Bytes of storage consumed`, leading to a cool off by 2*30 error. > You're right to question my calculation. The MCP server failed to connect when I tried to look up the field definition. I guessed instead of validating. This is on me. But look at all the revenue!
Just today I gave my local agent a CSV which listed a bunch files with of human readable size units and asked it to count rows in each GB range. Sounds simple enough but it completely miscalculated, because it parsed MB as GB for some reason. In hindsight it would've be quicker just to do it in Excel or something.
Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion
#465Earlier quoted context omitted.
I heard this false justification already in 2007, in spite of many customers asking for it. Incidentaly, smaller competitors solved this issue decades ago, while the big cloud decided it is more convenient never to implement it.
Big cloud didn't want to rewrite its billing systems from scratch to please its smallest customers.
Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion
#466Ive dealt with this error at AWS. It’s a unit error. In my case we _meant_ to charge like 5¢/GB, but missed the unit (GB), and then the billing system defaults to bytes. 5¢ per Byte of data transferred meant some customers were seeing MM bills within hours. Got paged by support around 2am, had it fixed and amendments issues by 3-4am, apology emails shortly after. Services emit metering values that arent directly tied…
1: "Oh, well, this is not a mundane detail, Michael!" https://www.youtube.com/watch?v=3fGHaVn5rGo
Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion
#467Earlier quoted context omitted.
Wanna bet the description of this job post will be updated by the end of the day? "Software Development Engineer II, AWS Invoicing" https://www.amazon.jobs/de/jobs/10428480/software-developmen... "...Our platforms are powered by generative AI, large language models, knowledge graphs, and agentic architectures that dynamically compose specialized agents based on context. We apply these capabilities across three reinfo…
Wow: In this role you will: - Design and build agentic AI systems that analyze, generate, and validate... - Build agentic architectures that compose specialized AI agents dynamically... - Build AI-driven continuous validation frameworks powered by agentic workflows and large language models that autonomously manage... This is invoicing? If ever there was a domain that was purely deterministic, you'd hope it was invoi…
Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion
#468Question: Why does AWS need to roll back estimated bills to a "last known good" state? I get wanting to do that for ACTUAL billing mistakes, but for estimates, they're just that -- approximations. I guess it's fine for predictive purposes to store estimates so they can be compared to actual usage and optimized. But why would AWS bind the values of present estimates to the estimates made earlier in the month. The calc…
“Rolling back” estimated bills is reprocessing the historic metering data by an older or newer pricing plan version. As i mentioned in another comment someone will have messed up a metering type vale (eg GB/B). Thats why theyll need a few hours to redrive the metering data.
Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion
#469Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion
#470Earlier quoted context omitted.
Just today I gave my local agent a CSV which listed a bunch files with of human readable size units and asked it to count rows in each GB range. Sounds simple enough but it completely miscalculated, because it parsed MB as GB for some reason. In hindsight it would've be quicker just to do it in Excel or something.
That is literally what Excel is for. Why didn't you use that first of all?