Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS: Inaccurate Estimated Billing Data – $1.7 billion

news.ycombinator.com

Post reply on HN
Print view View original

631–640 of 793 posts

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#631

Post by SlightlyLeftPad »

01284a7e wrote: post #466
donavanm wrote: post #449

Ive dealt with this error at AWS. It’s a unit error. In my case we _meant_ to charge like 5¢/GB, but missed the unit (GB), and then the billing system defaults to bytes. 5¢ per Byte of data transferred meant some customers were seeing MM bills within hours. Got paged by support around 2am, had it fixed and amendments issues by 3-4am, apology emails shortly after. Services emit metering values that arent directly tied…

No tests? Just mess up some mundane detail [1] and voila! Wake-up calls and heart attacks for 100,000s of administrators? 1: "Oh, well, this is not a mundane detail, Michael!" https://www.youtube.com/watch?v=3fGHaVn5rGo

It’s fractions of a penny Peter.

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#632

Post by shermantanktop »

sscaryterry wrote: post #563
donavanm wrote: post #449

Ive dealt with this error at AWS. It’s a unit error. In my case we _meant_ to charge like 5¢/GB, but missed the unit (GB), and then the billing system defaults to bytes. 5¢ per Byte of data transferred meant some customers were seeing MM bills within hours. Got paged by support around 2am, had it fixed and amendments issues by 3-4am, apology emails shortly after. Services emit metering values that arent directly tied…

This isn't a flippant comment. Imagine though, being presented with this. Imagine having some underlying health problem (e.g. cardiovascular). Do not be surprised if real people actually die from this mistake, from the anxiety, the surprise, the helplessness.

But someone that susceptible is likely going to have a bad reaction to many possible unexpected things. How would they react to a minor traffic accident? a family member getting hurt? a letter from tax authorities asking questions?

Having that serious an underlying health problem means everyday life represents risk for you. I don't think that means everyone else has to behave differently wrt (in this case wrt to billing mistakes) to keep you healthy.

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#633

Post by DrewADesign »

qeternity wrote: post #626
CobrastanJorji wrote: post #589

Earlier quoted context omitted.

There will have been tests, but there will have been missing end-to-end tests. Test 1 will verify that the new system/product emits billing entries in some expected way ("We did 100 bytes of operations and we see we called the billing system for 100 bytes of stuff, yay, test pass"). Test 2 will be in the billing system ("We provide an incoming bill for SKU#12345 for 100 gigabyte-units and we see it costs $17, yay, te…

But these aren't the right services where the test should be, right? There's another service that says "ok we take the 100 bytes from A, and we take the $17 SKU from B, and this should equal $X". It's the third service that multiplies these things that failed. Where are the tests for that?

To me this sounds like a human-or-LLM-driven error. There must be a pretty limited set of factors that determine a pricing unit: I’m not really sure how a deterministic system could do that infrequently enough to not be a bigger story. Maybe a reeeeaaaallly rare race condition or something like that? To me this smells like having enough manual work involved in the process to fuck something up, but not nearly enough eyes on it to notice.

I’m totally guessing though.

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#634

Post by dclowd9901 »

akdev1l wrote: post #474
01284a7e wrote: post #466

Earlier quoted context omitted.

No tests? Just mess up some mundane detail [1] and voila! Wake-up calls and heart attacks for 100,000s of administrators? 1: "Oh, well, this is not a mundane detail, Michael!" https://www.youtube.com/watch?v=3fGHaVn5rGo

Not even tests but just some basic anomaly detection lol. Like maybe if the bill amounts increase by like 10M% there should be someone that looks into it

Clearly some folks here have never worked at Amazon before. It's genuinely terrifying to see what most of the internet runs on.

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#636

Post by simmerup »

vasco wrote: post #625
01284a7e wrote: post #466

Earlier quoted context omitted.

No tests? Just mess up some mundane detail [1] and voila! Wake-up calls and heart attacks for 100,000s of administrators? 1: "Oh, well, this is not a mundane detail, Michael!" https://www.youtube.com/watch?v=3fGHaVn5rGo

I'm sorry but anyone that sees a multi million or billion dollar bill on an account that does nowhere near that should not be scared. It's obviously a mistake. Stories like this have happened with banks in my country. Check your account and you have billions in there. Guess what happened to those that withdrew money? The judge told them any reasonable person would know this is a bug. Had to give it back. Same thing h…

I’d be concerned as it throws into doubt their entire accounting mechanism

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#637

Post by shawkinaw »

yuchen20 wrote: post #62

I got 3 consecutive emails warning that my budget crossed its $18 threshold. Opened it up: cost was 78 million. Thought it was a phishing attempt, logged into my actual account, and... still 78 million. EMOTIONAL DAMAGE.

Mine was just over $1 billion. I thought the email was a phish at first and almost trashed it, but it seemed otherwise very real, so I logged in to AWS just to check. Then I almost had a heart attack when I looked at my billing estimate. Eventually I saw a banner on the support page, but my question is why wasn’t there a banner on the page SAYING I OWE A BILLION DOLLARS?!

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#638

Post by skissane »

CobrastanJorji wrote: post #589
01284a7e wrote: post #466

Earlier quoted context omitted.

No tests? Just mess up some mundane detail [1] and voila! Wake-up calls and heart attacks for 100,000s of administrators? 1: "Oh, well, this is not a mundane detail, Michael!" https://www.youtube.com/watch?v=3fGHaVn5rGo

There will have been tests, but there will have been missing end-to-end tests. Test 1 will verify that the new system/product emits billing entries in some expected way ("We did 100 bytes of operations and we see we called the billing system for 100 bytes of stuff, yay, test pass"). Test 2 will be in the billing system ("We provide an incoming bill for SKU#12345 for 100 gigabyte-units and we see it costs $17, yay, te…

> Somebody will have said at some point "we should actually have the tests charge money" and somebody else will have said "well we can't have the tests actually charge money, that's a legal/accounting problem, it might even be a crime" and then nobody would have asked what the next best thing was.

If you support multicurrency billing, then have the tests bill a test customer in XTS (ISO 4217 code for test currency).

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#639

Post by [deleted] »

qurren wrote: post #622
awakeasleep wrote: post #609

Earlier quoted context omitted.

Having been the manager writing those reports, you can only practically find causes that are within a single team’s ability to resolve. If you find a problem like this thread’s hypothetical, the process stops being an annoyance just to line level managers, and something that directors and vice presidents need to handle by changing strategic priorities within their organizations. That entails a real loss of face for t…

From the manager's side, you're absolutely correct. The SEV looks bad on you, and is a headache to document. From the perspective of people you manage, it's a very different picture. We (I say "we", because I was an IC) sit under you, and every year at performance review time you're effectively required to put some percentage of us in the "LE" bucket. Never mind that we could theoretically all HV3+ if you measure by…

[deleted]

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#640

Post by db48x »

fron wrote: post #98

Woke up to a billing alarm email. Thought I had leaked my AWS keys accidentally and somehow run up 437 billion dollars of charges. Joke's on them though, I don't have 437 billion dollars Anyways I didn't need coffee. That produced an adrenaline release unlike any I've experienced before. Thanks AWS

I was trying to go to sleep when I got mine. It was certainly above the $16 threshold I had set!
Post reply on HN