Live data from Hacker News

AWS: Inaccurate Estimated Billing Data – $1.7 billion

news.ycombinator.com

621–630 of 793 posts

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#621
post #492

Earlier quoted context omitted.

Are you speaking from experience or simply making things up? I know a fair number of former AWS engineers and managers. None of them think like this.

I am former AWS and this is pretty accurate. The other factor to add here is that, with some exceptions, the whole company feels like a Rube Goldberg machine and very few people care about what happens outside their cog (because they’re not incentivized to do so).

Rube Goldberg machine attached with used bubble gum and somehow the bubblegum was chewed in all the wrong ways

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#622

Earlier quoted context omitted.

Speaking from my experience at Amazon this is not the case. Any customer impact like this would necessitate a COE (correction of errors) report, which means a list of required action items to prevent such issues from happening again, which typically suck up at least man-month of labor. Not to mention the report itself, which has to be written by a manager. In fact, there are regular AWS-wide meetings where L10 techni…

Having been the manager writing those reports, you can only practically find causes that are within a single team’s ability to resolve. If you find a problem like this thread’s hypothetical, the process stops being an annoyance just to line level managers, and something that directors and vice presidents need to handle by changing strategic priorities within their organizations. That entails a real loss of face for t…

From the manager's side, you're absolutely correct. The SEV looks bad on you, and is a headache to document.

From the perspective of people you manage, it's a very different picture.

We (I say "we", because I was an IC) sit under you, and every year at performance review time you're effectively required to put some percentage of us in the "LE" bucket. Never mind that we could theoretically all HV3+ if you measure by "normal" peoples' standards, your manager isn't going to let you mark all of us as HV3 at the performance meetings. I know this, because I've been there as well at those meetings where truly high performing people were downrated to fit a distribution.

So what happens? When I see a peer's critical lurking bug, I have no incentive to fix it for the sake of prevention. If I fix it quietly so that it never surfaces, it looks like I haven't done any work for the week, or have done un-impactful work, and I get the stick from you. Preventing fires doesn't look like work, to non-technical eyes.

The only "safe" way to play this game of "survivor" is to let that bug surface eventually, then when the SEV comes up, I jump in and fix it, earning your approval, skip approval, VP approval, as well as potentially the other person gets the stick from you, because you have to give the stick to someone anyway, you get a reason to stick it on them. At least it's not getting stuck on me.

I'm sorry if this comes off as shocking to you, but it really shouldn't; the incentive structure is NOT set up for teamwork, plain and simple. If "putting customers first" is a value, then it absolutely needs to start from systematic changes of how people are managed.

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#623

You know, at one time Amazon was grouped in with all of the other big tech companies as a member of FAANG, as if the company has a culture of solid engineering at its core like Google or Apple. It doesn't anymore, if it ever did. Read the shareholder letters over the last 5 years like this one ( https://www.aboutamazon.com/news/company-news/amazon-ceo-and... ). Move fast. Embrace AI in every corner of the company. La…

> other times it's AWS regions going dark for hours.

They have the best uptime of all major cloud providers.

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#624
post #617

Earlier quoted context omitted.

This isn't a flippant comment. Imagine though, being presented with this. Imagine having some underlying health problem (e.g. cardiovascular). Do not be surprised if real people actually die from this mistake, from the anxiety, the surprise, the helplessness.

its so far out as to be obviously not real a smaller error by say, just one or two orders of magnitude are much more believable as a reader

I've seen people have panic attacks over much smaller amounts.

Just because you've not seen it or cannot fathom it happening in your world doesn't mean it doesn't happen.

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#625

Ive dealt with this error at AWS. It’s a unit error. In my case we _meant_ to charge like 5¢/GB, but missed the unit (GB), and then the billing system defaults to bytes. 5¢ per Byte of data transferred meant some customers were seeing MM bills within hours. Got paged by support around 2am, had it fixed and amendments issues by 3-4am, apology emails shortly after. Services emit metering values that arent directly tied…

No tests? Just mess up some mundane detail [1] and voila! Wake-up calls and heart attacks for 100,000s of administrators? 1: "Oh, well, this is not a mundane detail, Michael!" https://www.youtube.com/watch?v=3fGHaVn5rGo

I'm sorry but anyone that sees a multi million or billion dollar bill on an account that does nowhere near that should not be scared. It's obviously a mistake. Stories like this have happened with banks in my country. Check your account and you have billions in there. Guess what happened to those that withdrew money? The judge told them any reasonable person would know this is a bug. Had to give it back. Same thing here, any reasonable person doesn't get scared.

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#626

Earlier quoted context omitted.

No tests? Just mess up some mundane detail [1] and voila! Wake-up calls and heart attacks for 100,000s of administrators? 1: "Oh, well, this is not a mundane detail, Michael!" https://www.youtube.com/watch?v=3fGHaVn5rGo

There will have been tests, but there will have been missing end-to-end tests. Test 1 will verify that the new system/product emits billing entries in some expected way ("We did 100 bytes of operations and we see we called the billing system for 100 bytes of stuff, yay, test pass"). Test 2 will be in the billing system ("We provide an incoming bill for SKU#12345 for 100 gigabyte-units and we see it costs $17, yay, te…

But these aren't the right services where the test should be, right?

There's another service that says "ok we take the 100 bytes from A, and we take the $17 SKU from B, and this should equal $X".

It's the third service that multiplies these things that failed. Where are the tests for that?

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#627

Earlier quoted context omitted.

I wonder if AWS billing still uses CSV files for passing data around. IIRC it was one of my first on-calls at AWS over a decade ago now, and I got a page early evening because some stuff we did with billing records broke because some "smart" engineer thought it'd be a great idea to put an experimental record in with a description something like "I wonder what happens if I put, a comma in this field", into the product…

That was just a massive operational failure, not the fault of any single engineer. No change, except hotfixes, should be able to land on prod unless it has at least go through test, staging, and at the scale of Amazon, shadow testing. Engineers will do what engineers will always want to do, they want to see how things break, and sometimes they manage to fix it.

Kinda want to push back on that. It's not like engineers are separate from operations, not really. A lot of operations is just what engineers have been socialised to do.

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#628

Earlier quoted context omitted.

Wanna bet the description of this job post will be updated by the end of the day? "Software Development Engineer II, AWS Invoicing" https://www.amazon.jobs/de/jobs/10428480/software-developmen... "...Our platforms are powered by generative AI, large language models, knowledge graphs, and agentic architectures that dynamically compose specialized agents based on context. We apply these capabilities across three reinfo…

Wow: In this role you will: - Design and build agentic AI systems that analyze, generate, and validate... - Build agentic architectures that compose specialized AI agents dynamically... - Build AI-driven continuous validation frameworks powered by agentic workflows and large language models that autonomously manage... This is invoicing? If ever there was a domain that was purely deterministic, you'd hope it was invoi…

When I was at AWS, they famously required an extensive "CoE", correction of errors, or post-mortem, in an instance of over-charging a customer $0.26.

The idea is that if we can make small billing mistakes like that, we can make large billing mistakes, and need to invest in the correctness of the systems powering billing.

I have great respect for the engineering culture within AWS during those times. I am glad to have left before seeing it degrade and decline.

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#629

Earlier quoted context omitted.

Big cloud didn't want to rewrite its billing systems from scratch to please its smallest customers.

With AI it should take like a weekend.

I think they already tried rewriting billing with AI. Very smartly they only tried rewriting the estimator first. This post is about the outcome of it.

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#630
post #616
post #535

Earlier quoted context omitted.

We have a pretty strong existence proof... the thing happened in production. Unless they have some means to override a failing test and scp broken shit to prod, there wasn't a test.

missing canaries more likely? insufficient tests that dont assert on the right things? the existence of a test doesnt mean it catches the right thing based on the description, id bet the COE action item will be to do a migration that enforces units are passed at the billing service level theres no good reason for the billing service to make up its own units.

Yeah types and APIs that are difficult to misuse are way better than any regression test, canary, or what have you. Systems that are correct by construction always beat "you're holding it wrong".
Post reply on HN