Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS: Inaccurate Estimated Billing Data – $1.7 billion

news.ycombinator.com

Post reply on HN
Print view View original

691–700 of 793 posts

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#691

Post by raverbashing »

skissane wrote: post #638
CobrastanJorji wrote: post #589

Earlier quoted context omitted.

There will have been tests, but there will have been missing end-to-end tests. Test 1 will verify that the new system/product emits billing entries in some expected way ("We did 100 bytes of operations and we see we called the billing system for 100 bytes of stuff, yay, test pass"). Test 2 will be in the billing system ("We provide an incoming bill for SKU#12345 for 100 gigabyte-units and we see it costs $17, yay, te…

> Somebody will have said at some point "we should actually have the tests charge money" and somebody else will have said "well we can't have the tests actually charge money, that's a legal/accounting problem, it might even be a crime" and then nobody would have asked what the next best thing was. If you support multicurrency billing, then have the tests bill a test customer in XTS (ISO 4217 code for test currency).

You certainly don't want your service caring about a test currency.

Nobody is pricing their services in test currency

They are either pricing in $ or in "units" and then converting upstream

This is an even bigger foot gun than what's happening here

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#692

Post by andai »

antihipocrat wrote: post #687
andai wrote: post #642

Earlier quoted context omitted.

>no tests? Earlier this week my slopservant implemented several comprehensive changes to a codebase. It also wrote extensive tests to verify the correctness of the changes. A few days later I was working on something else and realized, everything had been implemented backwards, in a way that was nonsensical and also completely pointless. The many tests it had written were just confirming the LLM's idea of correctness…

I'm struggling with this on a hobby project now. I'm torn between continuing the fast pace of development and taking a pause and checking the entire codebase. The more features I add now will likely make the inevitable refactor much harder, but adding new features is so easy that I want want extend this illusion of productivity just a little bit longer!

Yeah my logic was, we're doing so many refactors that it doesn't make sense for me to start memorizing how things work until things settle down a little.

The argument in favour of "read each delta" (aside from catching them doing stupid shit) is to keep your mental model synced. But I already didn't understand my own code before AI, so that's moot too!

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#693

Post by hsbauauvhabzb »

01284a7e wrote: post #466
donavanm wrote: post #449

Ive dealt with this error at AWS. It’s a unit error. In my case we _meant_ to charge like 5¢/GB, but missed the unit (GB), and then the billing system defaults to bytes. 5¢ per Byte of data transferred meant some customers were seeing MM bills within hours. Got paged by support around 2am, had it fixed and amendments issues by 3-4am, apology emails shortly after. Services emit metering values that arent directly tied…

No tests? Just mess up some mundane detail [1] and voila! Wake-up calls and heart attacks for 100,000s of administrators? 1: "Oh, well, this is not a mundane detail, Michael!" https://www.youtube.com/watch?v=3fGHaVn5rGo

What could a banana possibly cost Michael, ten dollars?

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#694

Post by hsbauauvhabzb »

chasd00 wrote: post #561
27183 wrote: post #535

Earlier quoted context omitted.

We have a pretty strong existence proof... the thing happened in production. Unless they have some means to override a failing test and scp broken shit to prod, there wasn't a test.

why would a test setting unit to Bytes fail and not MB, KB, or GB, and so on? That's like trying create a unit test for email opt-in, both true and false are valid values. It's up to the user to select the right one.

Why have a default at all? Explicit is better than implicit.

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#695

Post by znsnsksjiaja »

vasco wrote: post #625
01284a7e wrote: post #466

Earlier quoted context omitted.

No tests? Just mess up some mundane detail [1] and voila! Wake-up calls and heart attacks for 100,000s of administrators? 1: "Oh, well, this is not a mundane detail, Michael!" https://www.youtube.com/watch?v=3fGHaVn5rGo

I'm sorry but anyone that sees a multi million or billion dollar bill on an account that does nowhere near that should not be scared. It's obviously a mistake. Stories like this have happened with banks in my country. Check your account and you have billions in there. Guess what happened to those that withdrew money? The judge told them any reasonable person would know this is a bug. Had to give it back. Same thing h…

If I experience a bug when transferring money (switching a digit or some such) they’ll shrug their shoulders and say there is nothing they can do. No amount of proof will push them to increment and decrement some integers over there.

They transfer money to me. Their mistake: there is nothing I can do. Rules for me but not for thee?

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#696

Post by soniclettuce »

27183 wrote: post #593
chasd00 wrote: post #561

Earlier quoted context omitted.

why would a test setting unit to Bytes fail and not MB, KB, or GB, and so on? That's like trying create a unit test for email opt-in, both true and false are valid values. It's up to the user to select the right one.

I'm not quite following your objection.. I'd expect a test that checks the multiplier is correct would detect orders of magnitude discrepancy. So if you're billing $x/byte you'd write a test for the billing thing that checks that, given y bytes, the bill is x*y. [edit] This may need to be an integration test to be effective, there is a certain peril to mocking that could bite you here. But that's fine, we have the te…

I think they're saying that their understanding of the hypothetical error is that somebody more or less configured a price as 5¢/byte instead of 5¢/GB. And if that's the case, that's not really a thing testing will catch. 5¢/byte is as "valid" a price as 5¢/GB.

There's a lot of hypotheticals in there, so who knows if testing could have caught this specific problem. But I see what they're saying in principle.

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#697

Post by mulmen »

DrewADesign wrote: post #633
qeternity wrote: post #626

Earlier quoted context omitted.

But these aren't the right services where the test should be, right? There's another service that says "ok we take the 100 bytes from A, and we take the $17 SKU from B, and this should equal $X". It's the third service that multiplies these things that failed. Where are the tests for that?

To me this sounds like a human-or-LLM-driven error. There must be a pretty limited set of factors that determine a pricing unit: I’m not really sure how a deterministic system could do that infrequently enough to not be a bigger story. Maybe a reeeeaaaallly rare race condition or something like that? To me this smells like having enough manual work involved in the process to fuck something up, but not nearly enough e…

Not everything is an LLM boogeyman. Sometimes a bug is just a bug.

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#698

Post by mlinhares »

donavanm wrote: post #449

Ive dealt with this error at AWS. It’s a unit error. In my case we _meant_ to charge like 5¢/GB, but missed the unit (GB), and then the billing system defaults to bytes. 5¢ per Byte of data transferred meant some customers were seeing MM bills within hours. Got paged by support around 2am, had it fixed and amendments issues by 3-4am, apology emails shortly after. Services emit metering values that arent directly tied…

The last thing a billing system should have is defaults, this is such an insane thing.

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#699

Post by ngsevers »

ngsevers wrote: post #679

After 10 years of building on AWS infra I still had no idea what anything would cost. At some point it just became clear to me that they intentionally designed it this way and that complexity is the feature. I switched to digital ocean last year and really haven't had to think about infrastructure again.

Just a note I thought i'd add, they actually have your estimated next months bill on the upper right hand corner of the page at all times.

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#700

Post by thewhitetulip »

mlinhares wrote: post #698
donavanm wrote: post #449

Ive dealt with this error at AWS. It’s a unit error. In my case we _meant_ to charge like 5¢/GB, but missed the unit (GB), and then the billing system defaults to bytes. 5¢ per Byte of data transferred meant some customers were seeing MM bills within hours. Got paged by support around 2am, had it fixed and amendments issues by 3-4am, apology emails shortly after. Services emit metering values that arent directly tied…

The last thing a billing system should have is defaults, this is such an insane thing.

Hey man, vibe coders gonna vine code. No wait. Loop enginering.

We don't need to do stupid things like review or plan or strategize. Put everything into LLM and let the black box take over everything! Yay! LLMs!!!!

Post reply on HN