What if the error wasn't so obvious? How many people have been overcharged and never knew?
AWS: Inaccurate Estimated Billing Data – $1.7 billion
721–730 of 793 posts
Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion
#722Earlier quoted context omitted.
You certainly don't want your service caring about a test currency. Nobody is pricing their services in test currency They are either pricing in $ or in "units" and then converting upstream This is an even bigger foot gun than what's happening here
AWS supports billing in multiple currencies: https://repost.aws/knowledge-center/supported-aws-currencies So how is it a "foot gun" to add XTS as an additional currency, for internal use only? I presume AWS sets its prices in USD, and then converts them to the other currencies using the relevant exchange rate - the services themselves don't know about UAE dirham, but the billing system does. So XTS just becomes anoth…
> AWS supports billing in multiple currencies
Of course. I'm not saying that it doesn't.
Read it again
I'm saying, the conversion to (any) currency needs to be done downstream of the service (to a general billing service). The service needs to bill "credits".
> presume AWS sets its prices in USD, and then converts them to the other currencies using the relevant exchange rate
Yes that's what I wrote in the first post
Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion
#723Earlier quoted context omitted.
To me this sounds like a human-or-LLM-driven error. There must be a pretty limited set of factors that determine a pricing unit: I’m not really sure how a deterministic system could do that infrequently enough to not be a bigger story. Maybe a reeeeaaaallly rare race condition or something like that? To me this smells like having enough manual work involved in the process to fuck something up, but not nearly enough e…
> human-or-LLM-driven error. In a computer system, dont those categories cover pretty much everything except a meteor strike?
Occasionally you have faulty hardware. Most commonly RAM.
Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion
#724My alert thresholds are $8 and $10 because I only use aws for some backups to S3. It was reporting my actual spend as $425 billion, and forecast at $777 billion.
I thought it was a bad phishing email first, when I saw it in the console I just laughed. I killed my access keys anyway before I noticed the incident on their status page.
Checked this morning and it's all back to normal.
I'm not sure I've seen anyone with a higher actual / forecast from this. Can anyway top mine?
Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion
#725Earlier quoted context omitted.
"I must've put a decimal point in the wrong place or something. I always mess up some mundane detail."
Multiple rockets exploded and space missions failed because of the imperial vs. metric BS, and those mistakes were made by people all with PhDs or equivalences. This is still pretty mild in comparison haha. Units and datetime will always be the bane of any professions ...
names and money movement also come to mind.
Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion
#726Ive dealt with this error at AWS. It’s a unit error. In my case we _meant_ to charge like 5¢/GB, but missed the unit (GB), and then the billing system defaults to bytes. 5¢ per Byte of data transferred meant some customers were seeing MM bills within hours. Got paged by support around 2am, had it fixed and amendments issues by 3-4am, apology emails shortly after. Services emit metering values that arent directly tied…
Are you able to just re-run the data for the timeframe, or is "fixing it" a more manual process? And what observability are you working with into the process?
Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion
#727Earlier quoted context omitted.
You overestimate how much people give shits at big techs like Amazon. When literally everything is driven with sticks instead of carrots, the work culture does not invite employees to proactively care about product quality. You'd be better off letting the heart attacks happen and take the 3am on-call and be the hero instead. It would be good promo doc material, and being a hero is extremely good insurance against get…
Speaking from my experience at Amazon this is not the case. Any customer impact like this would necessitate a COE (correction of errors) report, which means a list of required action items to prevent such issues from happening again, which typically suck up at least man-month of labor. Not to mention the report itself, which has to be written by a manager. In fact, there are regular AWS-wide meetings where L10 techni…
> Any customer impact like this would necessitate a COE (correction of errors) report, which means a list of required action items to prevent such issues from happening again
And this surely works out great and completely prevents any issue, and this "inaccurate estimated billing data" issue this submission is about is both the first and last time this issue ever happens at Amazon?
Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion
#728Earlier quoted context omitted.
Maybe they are former AWS employees for a reason and now want things to go better than they were at AWS.
AWS has had this reputation for over a decade. Every former AWS (including poached not fired) has relayed to me a verson of this. Every once in a while (~1:12) you get one that sounds like a Mormon missionary praising how its not that way and AWS is perfect.
> Every once in a while (~1:12) you get one that sounds like a Mormon missionary praising how its not that way and AWS is perfect.
The strange thing is that I only come across those missionaries online and never in person (although I did go to a AWS event once [never again] and met a bunch of them, so seemingly they are actually real, to my surprise).
Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion
#729I got a bill for about R900 for just having a look at some of their services?
I will never use them again, they are huge so probably dont really care about small users
I have actually replaced most of what they do with Raspberry Pi's using cloudflare tunnels
Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion
#730Earlier quoted context omitted.
AWS supports billing in multiple currencies: https://repost.aws/knowledge-center/supported-aws-currencies So how is it a "foot gun" to add XTS as an additional currency, for internal use only? I presume AWS sets its prices in USD, and then converts them to the other currencies using the relevant exchange rate - the services themselves don't know about UAE dirham, but the billing system does. So XTS just becomes anoth…
> AWS supports billing in multiple currencies Of course. I'm not saying that it doesn't. Read it again I'm saying, the conversion to (any) currency needs to be done downstream of the service (to a general billing service). The service needs to bill "credits". > presume AWS sets its prices in USD, and then converts them to the other currencies using the relevant exchange rate Yes that's what I wrote in the first post
> The service needs to bill "credits"
When talking about e2e tests, it matters where the e's are. If they are at the public interface of "the service", then, indeed, you'd test against "credits".
But ideally, there'd be e2e tests that test the consumers' experience, where the 'e's are the web interface, emails, CLI and other things consumers click on, read, enter etc. This is where a "test currency" makes sense.