Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS: Inaccurate Estimated Billing Data – $1.7 billion

news.ycombinator.com

Post reply on HN
Print view View original

381–390 of 793 posts

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#381

Post by myself248 »

dawnerd wrote: post #269
tedggh wrote: post #239

I got a 20K bill once and it was actually drafted from my bank account. It took me a couple of months and involving the office of the AG of my state to get the issue resolved and get my money back. Since then I never touched any AWS product, moved my small stuff to Azure. It’s been years since AWS have these issues with billing, you can find the stories online, students billed 60K for a compromised account launching…

That’s why you always use a spend limited card with variable cost providers.

Or just own your own hardware. Spend a few bucks at Microcenter, build a machine, and there's simply no mechanism by which they could decide later that you should actually pay 100x more, and then magically suck it out of your bank account.

None of this can happen unless you first cede control.

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#383

Post by TrickyRick »

everforward wrote: post #295
pqvst wrote: post #9

Probably the closest I've ever been to getting a heart attack. Normally <$1 per month, and now suddenly $284,006,266,443.74. Whatever the bug is on their end, this is unforgivable.

Yeah, this one is bad because it’s off by so much I’m shocked it wasn’t caught by tests, alerts about unusual changes in the billing system, or even accounting. Like surely the P&L reports look all kinds of wrong right now, they have to be showing like 6M% profit margins and revenue measured in quadrillions. I’m also a little surprised this didn’t trip a circuit breaker. For something as non-real-time as billing, I’m…

If I were to guess this bug is in the "display" part of the system which is probably distinct from the "actually take money from the customer" part of the system. One can imagine they have gates on the "actually take money" part, especially for a large bill like ours which was ~$300b or about 2.5x AWS' 2025 revenue... In one month. Surely if we had actually accumulated that bill they would be the ones with the problems when we can't pay it.

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#384

Post by sokoloff »

whatever1 wrote: post #366

Is it even possible to audit the cloud pricing? They just give us a number and we pay.

On AWS, you can enable CUR (cost and usage reporting) and get detailed, line-item billing figures that you can audit.

And naturally, companies like Cloudability [now Apptio] and others have sprung up to do parts of this for you [at a fee, of course...]

https://docs.aws.amazon.com/cur/latest/userguide/what-is-cur...

I'm sure other cloud vendors have similar functionality (because they need this on the back end to do their own billing anyway).

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#385

Post by galonk »

berkes wrote: post #286
qrios wrote: post #256

As someone who usually works with data analysis, the distribution of the numbers strikes me as odd. Almost all of them have one number that appears four times, and one or two that appear three times. And overall, there are an unusually small number (0–9) of digits that appear at all. Maybe it's not just vibe-coded, maybe the numbers themselves are being hallucinated by an LLM.

> Almost all of them have one number that appears four times, and one or two that appear three times To me that looked suspiciously like string-handling in a weakly typed language. Like when you do `"100" + 1` in JavaScript, or `int("100" * 2)` in Python. I've seen my share of such bugs in PHP, Python, Ruby, JavaScript. In production. Obviously not as simple as the examples, but subtle, like when a library update cha…

Pedantic as hell but `"100" * 2` in Python (= `"100100"` for those who don't know) isn't really typing, it's operator overloading. Any language with that could implement the same questionable design decision.

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#386

Post by masafej536 »

Polizeiposaune wrote: post #358
pcarmichael wrote: post #30

https://health.aws.amazon.com/health/status "Operational issue - AWS Billing Console (Global) Service - AWS Billing Console Severity Impacted - Inaccurate Estimated Billing Data"

Update as of 7:53am PDT: "The rollback of a recent change did not resolve the issue and we are continuing to investigate multiple mitigation paths. Estimated bill updates remain paused."

>Estimated bill updates remain paused

Wait what if someones actually getting usage spiked

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#387

Post by ButlerianJihad »

tedggh wrote: post #239

I got a 20K bill once and it was actually drafted from my bank account. It took me a couple of months and involving the office of the AG of my state to get the issue resolved and get my money back. Since then I never touched any AWS product, moved my small stuff to Azure. It’s been years since AWS have these issues with billing, you can find the stories online, students billed 60K for a compromised account launching…

For a while I had a portion of my "homelab" on AWS. I was an educator in a classroom where the students were learning cloud stuff, and the instructor was encouraging the students to stand-up cloud environments for learning, so I figured that I would do the same.

I used AWS' free tier, of course, and I enjoyed the initial setup in EC2, and I did a LAMP-stack MediaWiki installation. It wasn't too difficult, but two things sent me away forever.

1. It was impossible, or at least highly labor-intensive, in this modern era to adequately secure an ordinary Linux system running Internet-facing services. I put fail2ban and I filtered a lot of ports, and still spammers attacked me on Layer 7.

2. It was impossible, actually impossible, to limit or cap my cloud expenses in any billing cycle. Sure, run free-tier all I want. Sure, come in within the limits almost every month. But if I configured one thing wrong, or one thing went runaway, I'd have a sizable bill that I couldn't dispute. And even worse, those "runaways" weren't necessarily things in my sphere of control, but could be triggered by basically anyone coming in and using my VPC resources, especially egress network traffic.

So I closed out my cloud account, and I developed a lot of sympathy for businesses and corps that now are forced to run "in the cloud" rather than on-prem or their own machine rooms, but now they have no way to control expenses.

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#388

Post by daft_pink »

Maybe it’s one of those absurd situations where canceling a service doesn’t actually stop the charges. Instead, they quietly begin billing you for some random add-on that was bundled with the original service. You never knew it existed, never knew it had to be canceled separately, and now you’re paying full price for a completely pointless ghost service because the only thing it was tied to has already been canceled.

It sounds ridiculous, but something very similar happened to me with Amazon WorkSpaces. During the WorkSpaces setup, an AWS Active Directory (Directory Service) instance was provisioned as part of the deployment. When I later canceled WorkSpaces, I had no idea the Directory Service had to be deleted separately. I kept getting billed for it, and it ultimately cost more per month than the WorkSpace itself had.

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#389

Post by idiotsecant »

jayanmn wrote: post #355
yuchen20 wrote: post #62

I got 3 consecutive emails warning that my budget crossed its $18 threshold. Opened it up: cost was 78 million. Thought it was a phishing attempt, logged into my actual account, and... still 78 million. EMOTIONAL DAMAGE.

Enterprise account . We got - 3trillion and change

Quick do your IPO before the books update

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#390

Post by idiotsecant »

VulgarExigency wrote: post #289
lukaslueg wrote: post #17

Apparently what used to be `GB of storage consumed` is confused with `Bytes of storage consumed`, leading to a cool off by 2*30 error. > You're right to question my calculation. The MCP server failed to connect when I tried to look up the field definition. I guessed instead of validating. This is on me. But look at all the revenue!

The user is asking me to calculate how much money they should charge their customer. The values they've given me are 0.45, 1.67, and 2.50. This is 2.50 + 1.67 + 0.45 = 4.62, but it could be any other number. Perhaps we should be concatenating the numbers instead. Wait! The . could also mean multiplication. 0 . 45 . 1. 67 . 2 . 50 = 3015000. But wouldn't multiplying by 0 zero it out? That can't be right, we wouldn't b…

Would be funny if it wasn't so close to true
Post reply on HN