Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS: Inaccurate Estimated Billing Data – $1.7 billion

news.ycombinator.com

Post reply on HN
Print view View original

601–610 of 793 posts

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#601

Post by anvuong »

Twirrim wrote: post #575
donavanm wrote: post #449

Ive dealt with this error at AWS. It’s a unit error. In my case we _meant_ to charge like 5¢/GB, but missed the unit (GB), and then the billing system defaults to bytes. 5¢ per Byte of data transferred meant some customers were seeing MM bills within hours. Got paged by support around 2am, had it fixed and amendments issues by 3-4am, apology emails shortly after. Services emit metering values that arent directly tied…

I wonder if AWS billing still uses CSV files for passing data around. IIRC it was one of my first on-calls at AWS over a decade ago now, and I got a page early evening because some stuff we did with billing records broke because some "smart" engineer thought it'd be a great idea to put an experimental record in with a description something like "I wonder what happens if I put, a comma in this field", into the product…

That was just a massive operational failure, not the fault of any single engineer. No change, except hotfixes, should be able to land on prod unless it has at least go through test, staging, and at the scale of Amazon, shadow testing.

Engineers will do what engineers will always want to do, they want to see how things break, and sometimes they manage to fix it.

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#602

Post by anvuong »

pudgywalsh wrote: post #471
donavanm wrote: post #449

Ive dealt with this error at AWS. It’s a unit error. In my case we _meant_ to charge like 5¢/GB, but missed the unit (GB), and then the billing system defaults to bytes. 5¢ per Byte of data transferred meant some customers were seeing MM bills within hours. Got paged by support around 2am, had it fixed and amendments issues by 3-4am, apology emails shortly after. Services emit metering values that arent directly tied…

"I must've put a decimal point in the wrong place or something. I always mess up some mundane detail."

Multiple rockets exploded and space missions failed because of the imperial vs. metric BS, and those mistakes were made by people all with PhDs or equivalences. This is still pretty mild in comparison haha.

Units and datetime will always be the bane of any professions ...

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#603

Post by el1s7 »

akdev1l wrote: post #474
01284a7e wrote: post #466

Earlier quoted context omitted.

No tests? Just mess up some mundane detail [1] and voila! Wake-up calls and heart attacks for 100,000s of administrators? 1: "Oh, well, this is not a mundane detail, Michael!" https://www.youtube.com/watch?v=3fGHaVn5rGo

Not even tests but just some basic anomaly detection lol. Like maybe if the bill amounts increase by like 10M% there should be someone that looks into it

I think billing is the only thing AWS doesn't really care about optimizing or putting enough tests to avoid anomalies lol.

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#604

Post by reddalo »

fc417fc802 wrote: post #586
blemasle wrote: post #553

Earlier quoted context omitted.

I didn't know the imperial system was named "standard". Funny, cause its everything but standard both internationally and its definitions (which are not standard as based on SI)

Claiming something isn't standard because it isn't based on SI is entirely circular in the case of weights and measures. That said I wish the US would bite the bullet and make the switch. Mandating dual labeling on everything would be a great start. Then in 20 years we could narrow it back down to one.

You just need to start teaching kids the metric system. Then when they'll grow up the switch will almost magically happen by itself.

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#605

Post by sierra1011 »

I got an email this morning.

> You requested that we alert you when the actual cost associated with your Monthly AWS budget exceeds $2.00 for the current month. The month actual cost associated with this budget is $646,677,805.51.

Current usage: $1.70.

I called it a rounding error and figured it would wash out within a couple of days...

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#606

Post by fragmede »

27183 wrote: post #551
nullorempty wrote: post #548

Earlier quoted context omitted.

Technically, there could be a test. It could just be wrong!

If a tree falls in the forest and nobody hears it... [edit] Testing your tests, like testing your backups, is a good idea

Yes, test the negative case as well. eg if you get the system setup so you can log in, also make sure you get permission denied for bad login info.

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#607

Post by fragmede »

hotstickyballs wrote: post #584
Twirrim wrote: post #575

Earlier quoted context omitted.

I wonder if AWS billing still uses CSV files for passing data around. IIRC it was one of my first on-calls at AWS over a decade ago now, and I got a page early evening because some stuff we did with billing records broke because some "smart" engineer thought it'd be a great idea to put an experimental record in with a description something like "I wonder what happens if I put, a comma in this field", into the product…

CSV files are widely known to be used by the most frugal companies so of course it is.

Have you seen what RDS costs‽ AWS couldn't possibly afford that!

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#608

Post by 0cf8612b2e1e »

inigyou wrote: post #410
SAI_Peregrinus wrote: post #394

Earlier quoted context omitted.

> and they also imply deleting all your data so that you don't pay for storage. Not necessarily. They could imply that your storage becomes inaccessible immediately, but only gets deleted after some time period (say, 1 month). What spending limits do depends on the implementation.

That's even more work to implement. And now you store files on a second account that pays for only one day a month to not get deleted.

No wiggle room to come up with a workable solution. Let’s go shopping instead.

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#609

Post by awakeasleep »

mightyham wrote: post #549
qurren wrote: post #479

Earlier quoted context omitted.

You overestimate how much people give shits at big techs like Amazon. When literally everything is driven with sticks instead of carrots, the work culture does not invite employees to proactively care about product quality. You'd be better off letting the heart attacks happen and take the 3am on-call and be the hero instead. It would be good promo doc material, and being a hero is extremely good insurance against get…

Speaking from my experience at Amazon this is not the case. Any customer impact like this would necessitate a COE (correction of errors) report, which means a list of required action items to prevent such issues from happening again, which typically suck up at least man-month of labor. Not to mention the report itself, which has to be written by a manager. In fact, there are regular AWS-wide meetings where L10 techni…

Having been the manager writing those reports, you can only practically find causes that are within a single team’s ability to resolve.

If you find a problem like this thread’s hypothetical, the process stops being an annoyance just to line level managers, and something that directors and vice presidents need to handle by changing strategic priorities within their organizations.

That entails a real loss of face for them, and because they are the ones who actually run the show, it would will only happen if you have one that is naïve or a masochist. In either case that moves them out of management.

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#610

Post by fragmede »

blitzar wrote: post #376
root-parent wrote: post #268

Earlier quoted context omitted.

Wanna bet the description of this job post will be updated by the end of the day? "Software Development Engineer II, AWS Invoicing" https://www.amazon.jobs/de/jobs/10428480/software-developmen... "...Our platforms are powered by generative AI, large language models, knowledge graphs, and agentic architectures that dynamically compose specialized agents based on context. We apply these capabilities across three reinfo…

> 194,400.00 USD annually Fuck it, im in.

That's base, don't forget bonus and stock.
Post reply on HN