Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS: Inaccurate Estimated Billing Data – $1.7 billion

news.ycombinator.com

Post reply on HN
Print view View original

591–600 of 793 posts

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#591

Post by fc417fc802 »

londons_explore wrote: post #557
michaelmrose wrote: post #514

Earlier quoted context omitted.

The way you describe requires somehow counting every bit but somehow discarding most which is obviously nonsense. This seems statistically invalid insofar as it will tend to overbill potentially by a lot on the minority of cases. Don't you know how much of the pipe is occupied by a given customers code at any given time or what data is being sent

You have to do it when the customer list is too big to keep a counter per customer.

A probabilistic counter per customer is also a counter per customer. Still, probabilistic billing is an amusing thought though.

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#592

Post by thewebguyd »

zero_k wrote: post #576
cliglot wrote: post #361

[dead]

Wow, so true! Domain expertise is apparently not a requirement. At all. Wow.

Of course not, domain expertise is expensive no matter what the field is, and to the big companies like Amazon the whole point of GenAI is to get rid of those pesky expensive humans and replace them with cheaper humans.

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#593

Post by 27183 »

chasd00 wrote: post #561
27183 wrote: post #535

Earlier quoted context omitted.

We have a pretty strong existence proof... the thing happened in production. Unless they have some means to override a failing test and scp broken shit to prod, there wasn't a test.

why would a test setting unit to Bytes fail and not MB, KB, or GB, and so on? That's like trying create a unit test for email opt-in, both true and false are valid values. It's up to the user to select the right one.

I'm not quite following your objection.. I'd expect a test that checks the multiplier is correct would detect orders of magnitude discrepancy. So if you're billing $x/byte you'd write a test for the billing thing that checks that, given y bytes, the bill is x*y.

[edit] This may need to be an integration test to be effective, there is a certain peril to mocking that could bite you here. But that's fine, we have the technology.

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#594

Post by csomar »

dawnerd wrote: post #269
tedggh wrote: post #239

I got a 20K bill once and it was actually drafted from my bank account. It took me a couple of months and involving the office of the AG of my state to get the issue resolved and get my money back. Since then I never touched any AWS product, moved my small stuff to Azure. It’s been years since AWS have these issues with billing, you can find the stories online, students billed 60K for a compromised account launching…

That’s why you always use a spend limited card with variable cost providers.

Most debit/prepaid cards will get rejected. Credit Cards technically have a limit but they really don't. It's an open line to your finances.

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#595

Post by skywhopper »

curun1r wrote: post #467
ibejoeb wrote: post #308

Earlier quoted context omitted.

Wow: In this role you will: - Design and build agentic AI systems that analyze, generate, and validate... - Build agentic architectures that compose specialized AI agents dynamically... - Build AI-driven continuous validation frameworks powered by agentic workflows and large language models that autonomously manage... This is invoicing? If ever there was a domain that was purely deterministic, you'd hope it was invoi…

I’m not so sure about that. I can see a real rationale for creating sanity checks using AI to more quickly/proactively catch pathological billing issues before they become HN nightmare stories. They wouldn’t replace billing code, but there are many ways that stupid customer mistakes can cause real costs to Amazon that either have to be refunded and absorbed by Amazon or paid by the customer causing a negative opinion…

This is exactly the sort of thing that’s not possible, though. An AI will not be able to detect a “lambda loop” because it will look exactly like a “successful lambda rollout”. This sort of watchdog would just as likely shut down the wrong things and make AWS feel a lot less safe.

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#596

Post by fc417fc802 »

mito88 wrote: post #546
amelius wrote: post #516

Earlier quoted context omitted.

Big numbers can lead to stress which can lead to all kinds of disorders.

small numbers too.... :)

We should just go with numbers in general in order to play it safe. The new guidance from legal is absolutely no numbers on invoices for liability reasons. Similar to the removal of actual math from math class we're going to let the experts figure out how to implement it.

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#598

Post by rconti »

jayanmn wrote: post #355
yuchen20 wrote: post #62

I got 3 consecutive emails warning that my budget crossed its $18 threshold. Opened it up: cost was 78 million. Thought it was a phishing attempt, logged into my actual account, and... still 78 million. EMOTIONAL DAMAGE.

Enterprise account . We got - 3trillion and change

same. over 2t in one day.

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#599

Post by dabinat »

euio757 wrote: post #404
lelandfe wrote: post #379

This just hit global news: https://www.theguardian.com/technology/2026/jul/17/amazon-we... > Amazon Web Services customers receive bills for up to $1.5tn after global glitch

> One UK man whose bill is usually less than £1 says he ‘almost had a heart attack’ when he saw £5.8bn invoice That sucks, some people will get legit panic attacks and worse over this, especially for the smaller, more believable numbers in the 50k-500k range. Hope they recover and sue for medical bill costs, emotional damage etc. And like one reddit user suggests, everyone affected should write to their representativ…

It says a lot about AWS that people believed these estimates were real. Amazon does not have good safeguards to prevent astronomical bills.

If someone gets access to your account they can just buy a 3-year reserved instance u7in-24tb.224xlarge and it will add almost $2m to your bill.

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#600

Post by qurren »

thewebguyd wrote: post #588
qurren wrote: post #479

Earlier quoted context omitted.

You overestimate how much people give shits at big techs like Amazon. When literally everything is driven with sticks instead of carrots, the work culture does not invite employees to proactively care about product quality. You'd be better off letting the heart attacks happen and take the 3am on-call and be the hero instead. It would be good promo doc material, and being a hero is extremely good insurance against get…

> take the 3am on-call and be the hero instead Ah yes, the good old ITism "Everything's good, what are we even paying you for?" followed by "Everything's on fire, what are we even paying you for?" I moved out of it largely for that reason, am now an infrastructure/IT project manager, quite refreshing actually.

The trick to surviving under such management is to jump in and put out other peoples' fires but not spend time preventing them even if you know how to.
Post reply on HN