Fast and loose with billing data. Welcome to the new Amazon.
AWS: Inaccurate Estimated Billing Data – $1.7 billion
291–300 of 793 posts
Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion
#292I got 3 consecutive emails warning that my budget crossed its $18 threshold. Opened it up: cost was 78 million. Thought it was a phishing attempt, logged into my actual account, and... still 78 million. EMOTIONAL DAMAGE.
Wanna bet the description of this job post will be updated by the end of the day? "Software Development Engineer II, AWS Invoicing" https://www.amazon.jobs/de/jobs/10428480/software-developmen... "...Our platforms are powered by generative AI, large language models, knowledge graphs, and agentic architectures that dynamically compose specialized agents based on context. We apply these capabilities across three reinfo…
That job description feels so far beyond parody that I could scarcely believe it until opening the link! What a world.
Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion
#293Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion
#294Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion
#295Probably the closest I've ever been to getting a heart attack. Normally <$1 per month, and now suddenly $284,006,266,443.74. Whatever the bug is on their end, this is unforgivable.
Yeah, this one is bad because it’s off by so much I’m shocked it wasn’t caught by tests, alerts about unusual changes in the billing system, or even accounting. Like surely the P&L reports look all kinds of wrong right now, they have to be showing like 6M% profit margins and revenue measured in quadrillions.
I’m also a little surprised this didn’t trip a circuit breaker. For something as non-real-time as billing, I’m surprised they don’t have an automated kill switch that pauses the billing system and fires a page if variance in bills spikes. Naively some kind of “if the standard deviation of customer bills for this year changes by more than 50%, pause the billing system”. At that number of customers, those numbers should be pretty stable beyond internal billing changes they could normalize for.
Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion
#296Earlier quoted context omitted.
Wanna bet the description of this job post will be updated by the end of the day? "Software Development Engineer II, AWS Invoicing" https://www.amazon.jobs/de/jobs/10428480/software-developmen... "...Our platforms are powered by generative AI, large language models, knowledge graphs, and agentic architectures that dynamically compose specialized agents based on context. We apply these capabilities across three reinfo…
That job description feels so far beyond parody that I could scarcely believe it until opening the link! What a world.
It gets worst:
"Senior Software Development Manager, AWS Global Bill Generation" https://www.amazon.jobs/de/jobs/10471948/senior-software-dev...
"We're transforming from monthly batch processing and manual war rooms to continuous billing, autonomous agents, and self-healing infrastructure. We believe operational burden is a technical problem, not a staffing problem"
This looks clearly...a staffing problem...
Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion
#297Have even seen a $9.2 trillion for a friend.
Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion
#298Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion
#299Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion
#300How do we know if our bills were ever right if this made it into production?
That's the neat part, you don't!
Well, they publish unit prices for everything, so you could just get to counting. Whenever I've had to do cost estimates, you estimate how much AWS resources you need and then times that by the unit price.