Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS: Inaccurate Estimated Billing Data – $1.7 billion

news.ycombinator.com

Post reply on HN
Print view View original

641–650 of 793 posts

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#641

Post by bas »

01284a7e wrote: post #466
donavanm wrote: post #449

Ive dealt with this error at AWS. It’s a unit error. In my case we _meant_ to charge like 5¢/GB, but missed the unit (GB), and then the billing system defaults to bytes. 5¢ per Byte of data transferred meant some customers were seeing MM bills within hours. Got paged by support around 2am, had it fixed and amendments issues by 3-4am, apology emails shortly after. Services emit metering values that arent directly tied…

No tests? Just mess up some mundane detail [1] and voila! Wake-up calls and heart attacks for 100,000s of administrators? 1: "Oh, well, this is not a mundane detail, Michael!" https://www.youtube.com/watch?v=3fGHaVn5rGo

Who needs tests when you have vibes?

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#642

Post by andai »

01284a7e wrote: post #466
donavanm wrote: post #449

Ive dealt with this error at AWS. It’s a unit error. In my case we _meant_ to charge like 5¢/GB, but missed the unit (GB), and then the billing system defaults to bytes. 5¢ per Byte of data transferred meant some customers were seeing MM bills within hours. Got paged by support around 2am, had it fixed and amendments issues by 3-4am, apology emails shortly after. Services emit metering values that arent directly tied…

No tests? Just mess up some mundane detail [1] and voila! Wake-up calls and heart attacks for 100,000s of administrators? 1: "Oh, well, this is not a mundane detail, Michael!" https://www.youtube.com/watch?v=3fGHaVn5rGo

>no tests?

Earlier this week my slopservant implemented several comprehensive changes to a codebase.

It also wrote extensive tests to verify the correctness of the changes.

A few days later I was working on something else and realized, everything had been implemented backwards, in a way that was nonsensical and also completely pointless.

The many tests it had written were just confirming the LLM's idea of correctness, which turned out to be... completely incorrect.

I laughed when I realized, if I had been using Rust, or indeed, formal verification, that wouldn't have helped at all — it would have just written a mathematical proof, proving the correctness of the wrong thing!

Not sure what lesson to take from that (except read the damn diffs, obviously — it was a hobby project okay ;), but it seems like the more reliable this stuff gets, the more we expect it to work properly, the more risky it becomes.

https://en.wikipedia.org/wiki/Normalization_of_deviance

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#645

Post by VulgarExigency »

Izkata wrote: post #559
VulgarExigency wrote: post #289

Earlier quoted context omitted.

The user is asking me to calculate how much money they should charge their customer. The values they've given me are 0.45, 1.67, and 2.50. This is 2.50 + 1.67 + 0.45 = 4.62, but it could be any other number. Perhaps we should be concatenating the numbers instead. Wait! The . could also mean multiplication. 0 . 45 . 1. 67 . 2 . 50 = 3015000. But wouldn't multiplying by 0 zero it out? That can't be right, we wouldn't b…

So uh did you type that out or generate it somewhere? Number felt high so I wanted to double check and I only get 301500.

I typed it. I'm afraid my biological neural network has been trained by reading too many chains-of-thought. I might have added an extra 0 by accident, I didn't double check, but that just makes it more AI-like, really.

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#646

Post by nullorempty »

The latest status update says things will be back to normal for everyone on the 19th.

This spells for me a busy on-call and work over the weekend.

It sucks!

You think there will be lessons learnt from mistakes?

Not a fucking chance! Business as usual come monday!

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#648

Post by AlotOfReading »

reddalo wrote: post #604
fc417fc802 wrote: post #586

Earlier quoted context omitted.

Claiming something isn't standard because it isn't based on SI is entirely circular in the case of weights and measures. That said I wish the US would bite the bullet and make the switch. Mandating dual labeling on everything would be a great start. Then in 20 years we could narrow it back down to one.

You just need to start teaching kids the metric system. Then when they'll grow up the switch will almost magically happen by itself.

All of my science classes in the rural US from middle school onward were taught entirely in metric, and it had been that way for decades prior. Obviously I'm an adult now, and metric still isn't universal.

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#650

Post by yawaramin »

vasco wrote: post #625
01284a7e wrote: post #466

Earlier quoted context omitted.

No tests? Just mess up some mundane detail [1] and voila! Wake-up calls and heart attacks for 100,000s of administrators? 1: "Oh, well, this is not a mundane detail, Michael!" https://www.youtube.com/watch?v=3fGHaVn5rGo

I'm sorry but anyone that sees a multi million or billion dollar bill on an account that does nowhere near that should not be scared. It's obviously a mistake. Stories like this have happened with banks in my country. Check your account and you have billions in there. Guess what happened to those that withdrew money? The judge told them any reasonable person would know this is a bug. Had to give it back. Same thing h…

The problem with an anomalous AWS bill is that I don't know if it's because of a bug on their end or because of a goof on my end, eg did I accidentally leave on some giant compute for a month or something. With a billion in my bank account I at least know that I definitely didn't put it in there.
Post reply on HN