Live data from Hacker News

AWS: Inaccurate Estimated Billing Data – $1.7 billion

news.ycombinator.com

481–490 of 793 posts

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#481

Ive dealt with this error at AWS. It’s a unit error. In my case we _meant_ to charge like 5¢/GB, but missed the unit (GB), and then the billing system defaults to bytes. 5¢ per Byte of data transferred meant some customers were seeing MM bills within hours. Got paged by support around 2am, had it fixed and amendments issues by 3-4am, apology emails shortly after. Services emit metering values that arent directly tied…

This is why I always fail loud rather than pick a stupid default.

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#483

From their status page >The second path involves rolling back a recent change to the billing computation subsystem. Want to bet AI code was involved?

Do people not write tests anymore? How is 1 + 1 = 2 hard to test? Vibe coded implementation paired with vibe coded tests made to fit the implementation?

Reminds me of the decimal point scene in Office Space.

https://www.youtube.com/watch?v=3fGHaVn5rGo

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#484

Ive dealt with this error at AWS. It’s a unit error. In my case we _meant_ to charge like 5¢/GB, but missed the unit (GB), and then the billing system defaults to bytes. 5¢ per Byte of data transferred meant some customers were seeing MM bills within hours. Got paged by support around 2am, had it fixed and amendments issues by 3-4am, apology emails shortly after. Services emit metering values that arent directly tied…

"I must've put a decimal point in the wrong place or something. I always mess up some mundane detail."

Unit mistakes happen all the time, which is why you should be using your units library religiously and still being vigilant even then.

Worst case I've found was off by 15 orders of magnitude.

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#486

One can almost smell the vibes. This is peanuts compared to a major cybersecurity catastrophe that’s surely in the making. To give credit to the technology and the people using it - and I’m not being facetious - it’s actually incredible that at the current levels of usage the unprecedented catastrophic event has not yet happened.

Vibes, son. Nothing else in the world smells like that ... I love the smell of Vibes in the morning.

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#487

From their status page >The second path involves rolling back a recent change to the billing computation subsystem. Want to bet AI code was involved?

10 layers of approvals to get a reasonable change implemented, but this shit gets pushed to production instantly. AI is driving leaders insane - they actually believe they can stop operating with humans and take all of the money. Remember your pay is transactional. Never give them more than written in your contract, and there is no shame in playing dirty politics to advance within.

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#490
post #350

It's crazy enough this will be fixed soon. Years ago I found an actual hidden error in my bill. (This was early 2010s). The system was calculating the EC2 reservation savings incorrectly for some of my servers. I was crunching all their detailed usage data on a regular basis in an 18 tab spreadsheet and couldn't get it to fully reconcile. I spent months trying to track down the discrepancy. Once I found it, I had to…

A couple of my coworkers think I’m nuts for watching cost explorer so closely but 1. The time it takes to look and notice costs that don’t make sense easily pays for itself, and then some (in my experience). I doubt you spent $7k of your time tracking this down, and you probably noticed optimization opportunities that saved you even more 2. I hate the idea of wasting money on buying Jeff Bezos a bigger yacht

> 2. I hate the idea of wasting money on buying Jeff Bezos a bigger yacht

Then you aren't using AWS. At least half of all the money you give to Amazon is yacht money.

Post reply on HN