AWS: Inaccurate Estimated Billing Data – $1.7 billion • Hacker News Board

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#671

Post by dlenski » Sat, Jul 18, 2026, 2:39 AM UTC

donavanm wrote: post #449

Ive dealt with this error at AWS. It’s a unit error. In my case we _meant_ to charge like 5¢/GB, but missed the unit (GB), and then the billing system defaults to bytes. 5¢ per Byte of data transferred meant some customers were seeing MM bills within hours. Got paged by support around 2am, had it fixed and amendments issues by 3-4am, apology emails shortly after. Services emit metering values that arent directly tied…

> Services emit metering values that arent directly tied to prices.

Yep. The metering ("Kona") is separated from the billing to such degree that it's basically impossible to find anyone at the company who understands both.

I remember having to work on some metering code, and trying to figure out whether it would result in correct billing that matched our service's documentation. I was basically told off by a more senior engineer for wasting my time on something that was completely tangential and outside of the "engineering" domain entirely.

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#672

Post by RandomThrow321 » Sat, Jul 18, 2026, 2:58 AM UTC

dlenski wrote: post #670

mightyham wrote: post #549

Earlier quoted context omitted.

Speaking from my experience at Amazon this is not the case. Any customer impact like this would necessitate a COE (correction of errors) report, which means a list of required action items to prevent such issues from happening again, which typically suck up at least man-month of labor. Not to mention the report itself, which has to be written by a manager. In fact, there are regular AWS-wide meetings where L10 techni…

> COEs are such a huge annoyance for teams that they create a strong incentive to be proactive in preventing issues like this from happening. Absolutely not my experience at AWS. All the teams I was on treated them as "not a big deal", kind of a non-punitive exercise in technical writing, and the COE was always assigned to be written by an engineer who was not involved in causing the COE. Also, the kinds of issues th…

Obviously experiences vary greatly, but I was on one of the largest AWS orgs and they were quite punitive. People would demand them for perceived slights, then assign reviewers that were tough or on their side. Many of my friends had different experiences, though.

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#673

Post by kjs3 » Sat, Jul 18, 2026, 3:04 AM UTC

jerf wrote: post #660

yuchen20 wrote: post #62

I got 3 consecutive emails warning that my budget crossed its $18 threshold. Opened it up: cost was 78 million. Thought it was a phishing attempt, logged into my actual account, and... still 78 million. EMOTIONAL DAMAGE.

The real fun comes later when the US Federal Government taxes us on the forgiven debt as income.

The finance fun noone knows about. I knew someone who did this as revenge. Sent some asshole a fake invoice for US$stupidmoney for a couple of months which said asshole ignored, then sent a certified mail 'I forgive asshole US$stupidmoney', and copied the IRS. IRS went to collect on the unpaid taxes, and if you've never been there, the IRS does not play when it comes to that sort of money. I don't know how it ended, but there were multiple lawyers involved, so it was expensive no matter what.

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#674

Post by dlenski » Sat, Jul 18, 2026, 3:07 AM UTC

RandomThrow321 wrote: post #672

dlenski wrote: post #670

Earlier quoted context omitted.

> COEs are such a huge annoyance for teams that they create a strong incentive to be proactive in preventing issues like this from happening. Absolutely not my experience at AWS. All the teams I was on treated them as "not a big deal", kind of a non-punitive exercise in technical writing, and the COE was always assigned to be written by an engineer who was not involved in causing the COE. Also, the kinds of issues th…

Obviously experiences vary greatly, but I was on one of the largest AWS orgs and they were quite punitive. People would demand them for perceived slights, then assign reviewers that were tough or on their side. Many of my friends had different experiences, though.

Hmm. I'm curious about which org that was.

I spent the slight majority of my time at AWS in RDS.

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#675

Post by kjs3 » Sat, Jul 18, 2026, 3:18 AM UTC

ainiriand wrote: post #275

sscaryterry wrote: post #41

Vibe coding billing systems is a top-notch idea :)

Hey what do you think about vibe coding weapon systems? Do you want to be my cofounder?

How about vibe coded health care systems.

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#676

Post by 27183 » Sat, Jul 18, 2026, 3:19 AM UTC

fragmede wrote: post #606

27183 wrote: post #551

Earlier quoted context omitted.

If a tree falls in the forest and nobody hears it... [edit] Testing your tests, like testing your backups, is a good idea

Yes, test the negative case as well. eg if you get the system setup so you can log in, also make sure you get permission denied for bad login info.

Yeah, negative tests are more important than the "test your tests" thing. Negative tests are themselves regression tests. Testing that a test works is something you do at dev time, it doesn't really live in CI. Negative tests, as well as positive tests, establish invariants on the code under test. They're effectively permanent, immortalized in the CI suite. Testing that the test actually works is a one time thing at the time the test is written. Permute the code under test, check that the test fails in the expected way, done.

You just have to trust your colleagues won't edit the tests in such a way that vandalizes the invariant. That's why you can never trust an LLM to edit a test. They're notorious for tweaking tests such that they pass.

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#677

Post by kjs3 » Sat, Jul 18, 2026, 3:20 AM UTC

bcrosby95 wrote: post #463

xp84 wrote: post #370

Earlier quoted context omitted.

Somehow I highly doubt anyone will leave AWS over this unless their use of AWS is way more low-complexity than the average account. People make similar pronouncements after every us-east-1 outage makes the news, but I feel like AWS would be going out of business by now if people followed through. It reminds me of airlines, where after a particularly grueling irregular ops experience, a few dozen people file off the p…

No, AWS won't go out of business, afterall, people still use IBM mainframes.

Because IBM mainframes give the right answers.

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#678

Post by monkpit » Sat, Jul 18, 2026, 3:38 AM UTC

cyberax wrote: post #659

AlotOfReading wrote: post #484

Earlier quoted context omitted.

Unit mistakes happen all the time, which is why you should be using your units library religiously and still being vigilant even then. Worst case I've found was off by 15 orders of magnitude.

One of my personal rules is to ALWAYS suffix the variables and fields with unit names: `timeout_ms` or `rate_kbps`, NOT `timeout` or `rate`. Unless the variable type already constrains it (e.g. Duration in Go).

Until someone comes along and changes it without updating the name, or the value comes from a remote service and they change without you changing your name… etc.

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#679

Post by ngsevers » Sat, Jul 18, 2026, 3:43 AM UTC

After 10 years of building on AWS infra I still had no idea what anything would cost. At some point it just became clear to me that they intentionally designed it this way and that complexity is the feature. I switched to digital ocean last year and really haven't had to think about infrastructure again.

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#680

Post by handoflixue » Sat, Jul 18, 2026, 3:47 AM UTC

minitoar wrote: post #235

handoflixue wrote: post #174

Earlier quoted context omitted.

Realtime billing seems entirely within the abilities of AWS. "Limits except for Storage" seems even easier - I don't think I've ever heard of a storage-based billing story, although I'm sure one or two exist

What is a storage-based billing story?

Like kgwgk already said - people post the occasional horror story where they run up a huge AWS bill, either due to a run-away process or getting hacked. It seems to almost always be about Compute, not Storage.