Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS: Inaccurate Estimated Billing Data – $1.7 billion

news.ycombinator.com

Post reply on HN
Print view View original

451–460 of 793 posts

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#451

Post by quickthrowman »

root-parent wrote: post #296
sebmellen wrote: post #292

Earlier quoted context omitted.

That job description feels so far beyond parody that I could scarcely believe it until opening the link! What a world.

It gets worst: "Senior Software Development Manager, AWS Global Bill Generation" https://www.amazon.jobs/de/jobs/10471948/senior-software-dev... "We're transforming from monthly batch processing and manual war rooms to continuous billing, autonomous agents, and self-healing infrastructure. We believe operational burden is a technical problem, not a staffing problem" This looks clearly...a staffing problem...

They need to fire whoever is running AP and AP software development. Vibe invoicing is ridiculous for anyone to do, let alone Amazon.

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#452

Post by mxuribe »

qrios wrote: post #256

As someone who usually works with data analysis, the distribution of the numbers strikes me as odd. Almost all of them have one number that appears four times, and one or two that appear three times. And overall, there are an unusually small number (0–9) of digits that appear at all. Maybe it's not just vibe-coded, maybe the numbers themselves are being hallucinated by an LLM.

Its the LLMs talking to each other in secret code: random-looking numbers! They've achieved sentience!

Look at them up there, just plotting with each other! :-)

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#453

Post by donavanm »

wglass wrote: post #350

It's crazy enough this will be fixed soon. Years ago I found an actual hidden error in my bill. (This was early 2010s). The system was calculating the EC2 reservation savings incorrectly for some of my servers. I was crunching all their detailed usage data on a regular basis in an 18 tab spreadsheet and couldn't get it to fully reconcile. I spent months trying to track down the discrepancy. Once I found it, I had to…

> After 14 months I got a $7,000 refund. I was told it had to be approved by the head of AWS.

$7,000 of credits is no problem. At that time a friendly neighborhood PM or director could issue the credit without much oversight.

Your problem is the time period. Amending a bill in the same cycle is EZ. Fixing the previous cycle is a PITA but pretty common. Issuing amendments for the previous financial _years_ would be a huuuuge PITA going through finance etc.

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#454

Post by el_memorioso »

xp84 wrote: post #370
roskoalexey wrote: post #148

They sent 3 warnings to my email, ok, I understand bugs happen (probably vibe-coded). But they didn't even send any notification that it's a bug. Going to leave AWS after that.

Somehow I highly doubt anyone will leave AWS over this unless their use of AWS is way more low-complexity than the average account. People make similar pronouncements after every us-east-1 outage makes the news, but I feel like AWS would be going out of business by now if people followed through. It reminds me of airlines, where after a particularly grueling irregular ops experience, a few dozen people file off the p…

Airlines are all subject to a lot of the same factors, but there are unequivocally better and worse performers in terms of on-time arrivals, by a lot. Take a look at the Air Travel Consumer report for details.

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#455

Post by positr0n »

drew870mitchell wrote: post #285
tedggh wrote: post #239

I got a 20K bill once and it was actually drafted from my bank account. It took me a couple of months and involving the office of the AG of my state to get the issue resolved and get my money back. Since then I never touched any AWS product, moved my small stuff to Azure. It’s been years since AWS have these issues with billing, you can find the stories online, students billed 60K for a compromised account launching…

AWS is basically a utility. I think it's inevitable that their carelessness around billing will end up with them being regulated like one.

I can't think of another regulated utility that doesn't provide service to (essentially) all humans directly in their homes.

Everyone knows what water and electricity are, the vast majority couldn't explain what service AWS provides.

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#457

Post by dymk »

noisy_boy wrote: post #194
Hamuko wrote: post #178

Earlier quoted context omitted.

Reminds me of the time a Robinhood user killed themselves after Robinhood erroneously showed their account balance at -$730k. https://www.bbc.com/news/world-us-canada-55990461

In that case, they settled with Robinhood and FINRA fined Robinhood 70 million dollars. https://edition.cnn.com/2021/07/01/business/robinhood-lawsui...

"They" being the family; Robinhood couldn't settle with the guy who was wronged because he, ya know, killed himself first

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#458

Post by jeffrallen »

tedggh wrote: post #239

I got a 20K bill once and it was actually drafted from my bank account. It took me a couple of months and involving the office of the AG of my state to get the issue resolved and get my money back. Since then I never touched any AWS product, moved my small stuff to Azure. It’s been years since AWS have these issues with billing, you can find the stories online, students billed 60K for a compromised account launching…

Right, and good luck getting a correct bill from Azure. And when you are finally fed up, it will take months to close your Azure account.

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#460

Post by [deleted] »

TrickyRick wrote: post #383
everforward wrote: post #295

Earlier quoted context omitted.

Yeah, this one is bad because it’s off by so much I’m shocked it wasn’t caught by tests, alerts about unusual changes in the billing system, or even accounting. Like surely the P&L reports look all kinds of wrong right now, they have to be showing like 6M% profit margins and revenue measured in quadrillions. I’m also a little surprised this didn’t trip a circuit breaker. For something as non-real-time as billing, I’m…

If I were to guess this bug is in the "display" part of the system which is probably distinct from the "actually take money from the customer" part of the system. One can imagine they have gates on the "actually take money" part, especially for a large bill like ours which was ~$300b or about 2.5x AWS' 2025 revenue... In one month. Surely if we had actually accumulated that bill they would be the ones with the proble…

[deleted]
Post reply on HN