Ive dealt with this error at AWS. It’s a unit error. In my case we _meant_ to charge like 5¢/GB, but missed the unit (GB), and then the billing system defaults to bytes. 5¢ per Byte of data transferred meant some customers were seeing MM bills within hours. Got paged by support around 2am, had it fixed and amendments issues by 3-4am, apology emails shortly after. Services emit metering values that arent directly tied…
Imagine a programming language that has physical measurement unit support so this could have never happened.
AWS: Inaccurate Estimated Billing Data – $1.7 billion
661–670 of 793 posts
Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion
#662Ive dealt with this error at AWS. It’s a unit error. In my case we _meant_ to charge like 5¢/GB, but missed the unit (GB), and then the billing system defaults to bytes. 5¢ per Byte of data transferred meant some customers were seeing MM bills within hours. Got paged by support around 2am, had it fixed and amendments issues by 3-4am, apology emails shortly after. Services emit metering values that arent directly tied…
I wonder if AWS billing still uses CSV files for passing data around. IIRC it was one of my first on-calls at AWS over a decade ago now, and I got a page early evening because some stuff we did with billing records broke because some "smart" engineer thought it'd be a great idea to put an experimental record in with a description something like "I wonder what happens if I put, a comma in this field", into the product…
It was the same old kona files for metering until at least a ‘23-24 when I last did metering/pricing work. “Luckily” the size of the EC2 pricing plan was so stupidly large that it was forcing improvements to how those were serialized, and how private pricing plans were created/assigned to different payer ids.
Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion
#663Earlier quoted context omitted.
This isn't a flippant comment. Imagine though, being presented with this. Imagine having some underlying health problem (e.g. cardiovascular). Do not be surprised if real people actually die from this mistake, from the anxiety, the surprise, the helplessness.
actually $17B isn't so bad because it's so obviously an error and not something you can fatfinger. if you expected it to be $17 and it was $17,000 you'd be much more worried as that looks like a plausible error on your part
Yeah that's what I was thinking. And if someone sent me a bill for 17 billion I would just tell them I don't have it and never will, sorry.
Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion
#664Earlier quoted context omitted.
You overestimate how much people give shits at big techs like Amazon. When literally everything is driven with sticks instead of carrots, the work culture does not invite employees to proactively care about product quality. You'd be better off letting the heart attacks happen and take the 3am on-call and be the hero instead. It would be good promo doc material, and being a hero is extremely good insurance against get…
Speaking from my experience at Amazon this is not the case. Any customer impact like this would necessitate a COE (correction of errors) report, which means a list of required action items to prevent such issues from happening again, which typically suck up at least man-month of labor. Not to mention the report itself, which has to be written by a manager. In fact, there are regular AWS-wide meetings where L10 techni…
> > everything is driven with sticks instead of carrots
> this is not the case
> [describes all sticks and no carrots]
Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion
#665Earlier quoted context omitted.
Unit mistakes happen all the time, which is why you should be using your units library religiously and still being vigilant even then. Worst case I've found was off by 15 orders of magnitude.
One of my personal rules is to ALWAYS suffix the variables and fields with unit names: `timeout_ms` or `rate_kbps`, NOT `timeout` or `rate`. Unless the variable type already constrains it (e.g. Duration in Go).
This also helps downstream a lot.
Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion
#666Urgent? Not for you. This is like that old joke “If you owe a bank thousands, you have a problem; owe a bank a billion, the bank has a problem”
Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion
#667Earlier quoted context omitted.
I hope they send out some free credits at least. I imagine quite a few people got a real fucking scare today. They haven't even sent out any corrections yet.
This is probably going to push me to completely close a couple of AWS accounts I setup when doing training courses so I could get certified (mandatory requirement from my work). I'm not currently running anything and have no plans to at the moment. I've always had a mild dread that I'll suddenly get a bill for more than $0.00. If AWS can goof in a way that causes obviously massive bills (like today), what's to say th…
Excellent point!
I remember seeing a story shared by AMS RDS on linked-in that basically said... "Now with predictable pricing"
Huh?
Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion
#668Several comments here talk about "nearly" having a heart attack. But I wonder: since it's happened to so many people, chances are someone had a heart attack for real. Can they legally be made responsible for that?
Not a lawyer but if you can sue for emotional distress for similar situations I would imagine there is an angle here
Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion
#669Earlier quoted context omitted.
Speaking from my experience at Amazon this is not the case. Any customer impact like this would necessitate a COE (correction of errors) report, which means a list of required action items to prevent such issues from happening again, which typically suck up at least man-month of labor. Not to mention the report itself, which has to be written by a manager. In fact, there are regular AWS-wide meetings where L10 techni…
Amazon is heterogeneous. So much so, that positive anecdotes and negative anecdotes are near worthless without specifying the org. Depending on if you're a cost cutting team, fixed expense team or organization, if you're a revenue driving team, or if you're a core team, or the very many other splits you can come up about the relationship between the expense/balance sheets and the team itself...there are very very dif…
This was very much my experience, having worked in two different sub-organizations at AWS, and on several different services, in two different countries.
There's just extreme variation in the quality of the management, the quality of the engineers, the operational/development role split, the on-call schedule, and the development and testing methodology.
Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion
#670Earlier quoted context omitted.
You overestimate how much people give shits at big techs like Amazon. When literally everything is driven with sticks instead of carrots, the work culture does not invite employees to proactively care about product quality. You'd be better off letting the heart attacks happen and take the 3am on-call and be the hero instead. It would be good promo doc material, and being a hero is extremely good insurance against get…
Speaking from my experience at Amazon this is not the case. Any customer impact like this would necessitate a COE (correction of errors) report, which means a list of required action items to prevent such issues from happening again, which typically suck up at least man-month of labor. Not to mention the report itself, which has to be written by a manager. In fact, there are regular AWS-wide meetings where L10 techni…
> COEs are such a huge annoyance for teams that they create a strong incentive to be proactive in preventing issues like this from happening.
Absolutely not my experience at AWS.
All the teams I was on treated them as "not a big deal", kind of a non-punitive exercise in technical writing, and the COE was always assigned to be written by an engineer who was not involved in causing the COE.
Also, the kinds of issues that did or didn't lead to COEs appeared to be largely random. I was considered to be an extremely good operational trouble-shooter on the team where I spent most of my time at AWS, and I was never able to predict what an L7-8 manager would decide was COE-worthy.