Live data from Hacker News

AWS: Inaccurate Estimated Billing Data – $1.7 billion

news.ycombinator.com

461–470 of 793 posts

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#461

Earlier quoted context omitted.

Update as of 7:53am PDT: "The rollback of a recent change did not resolve the issue and we are continuing to investigate multiple mitigation paths. Estimated bill updates remain paused."

>Estimated bill updates remain paused Wait what if someones actually getting usage spiked

Hackers rejoice!

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#462
post #239

I got a 20K bill once and it was actually drafted from my bank account. It took me a couple of months and involving the office of the AG of my state to get the issue resolved and get my money back. Since then I never touched any AWS product, moved my small stuff to Azure. It’s been years since AWS have these issues with billing, you can find the stories online, students billed 60K for a compromised account launching…

> I got a 20K bill once and it was actually drafted from my bank account.

Service provider lesson #1: Never ever ever enable auto-pay! The convenience (and even the savings, if applicable) aren't worth the risk of the service provider autonomously slurping up all your money.

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#463
post #370

They sent 3 warnings to my email, ok, I understand bugs happen (probably vibe-coded). But they didn't even send any notification that it's a bug. Going to leave AWS after that.

Somehow I highly doubt anyone will leave AWS over this unless their use of AWS is way more low-complexity than the average account. People make similar pronouncements after every us-east-1 outage makes the news, but I feel like AWS would be going out of business by now if people followed through. It reminds me of airlines, where after a particularly grueling irregular ops experience, a few dozen people file off the p…

No, AWS won't go out of business, afterall, people still use IBM mainframes.

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#464

Apparently what used to be `GB of storage consumed` is confused with `Bytes of storage consumed`, leading to a cool off by 2*30 error. > You're right to question my calculation. The MCP server failed to connect when I tried to look up the field definition. I guessed instead of validating. This is on me. But look at all the revenue!

Just today I gave my local agent a CSV which listed a bunch files with of human readable size units and asked it to count rows in each GB range. Sounds simple enough but it completely miscalculated, because it parsed MB as GB for some reason. In hindsight it would've be quicker just to do it in Excel or something.

That is literally what Excel is for. Why didn't you use that first of all?

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#465

Earlier quoted context omitted.

I heard this false justification already in 2007, in spite of many customers asking for it. Incidentaly, smaller competitors solved this issue decades ago, while the big cloud decided it is more convenient never to implement it.

Big cloud didn't want to rewrite its billing systems from scratch to please its smallest customers.

With AI it should take like a weekend.

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#466

Ive dealt with this error at AWS. It’s a unit error. In my case we _meant_ to charge like 5¢/GB, but missed the unit (GB), and then the billing system defaults to bytes. 5¢ per Byte of data transferred meant some customers were seeing MM bills within hours. Got paged by support around 2am, had it fixed and amendments issues by 3-4am, apology emails shortly after. Services emit metering values that arent directly tied…

No tests? Just mess up some mundane detail [1] and voila! Wake-up calls and heart attacks for 100,000s of administrators?

1: "Oh, well, this is not a mundane detail, Michael!" https://www.youtube.com/watch?v=3fGHaVn5rGo

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#467

Earlier quoted context omitted.

Wanna bet the description of this job post will be updated by the end of the day? "Software Development Engineer II, AWS Invoicing" https://www.amazon.jobs/de/jobs/10428480/software-developmen... "...Our platforms are powered by generative AI, large language models, knowledge graphs, and agentic architectures that dynamically compose specialized agents based on context. We apply these capabilities across three reinfo…

Wow: In this role you will: - Design and build agentic AI systems that analyze, generate, and validate... - Build agentic architectures that compose specialized AI agents dynamically... - Build AI-driven continuous validation frameworks powered by agentic workflows and large language models that autonomously manage... This is invoicing? If ever there was a domain that was purely deterministic, you'd hope it was invoi…

I’m not so sure about that. I can see a real rationale for creating sanity checks using AI to more quickly/proactively catch pathological billing issues before they become HN nightmare stories. They wouldn’t replace billing code, but there are many ways that stupid customer mistakes can cause real costs to Amazon that either have to be refunded and absorbed by Amazon or paid by the customer causing a negative opinion of AWS. If a billing AI watching costs in realtime could detect, say, a lambda loop in the first 10 min and either alert the customer or kill it, that would make AWS feel a lot safer to use. Enumerating these conditions and fixing them individually is a task that Amazon has proven incapable of achieving. An AI watchdog layer might be the perfect shortcut to addressing all of these problems at once. Because it’s well-trodden territory that AWS has so many multi-thousand dollar foot guns that make it really scary to use as a hobbyist or small business on a tight budget.

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#468

Question: Why does AWS need to roll back estimated bills to a "last known good" state? I get wanting to do that for ACTUAL billing mistakes, but for estimates, they're just that -- approximations. I guess it's fine for predictive purposes to store estimates so they can be compared to actual usage and optimized. But why would AWS bind the values of present estimates to the estimates made earlier in the month. The calc…

Thats not really how estimates work. The actual metering data is ingested in near real time. The metering * pricing plan is processed within a few hours; thats what youre seeing for “estimated spend” IIRC. The actual billing accumulation is done later, at the end of the cycle, because pricing has cross service discounts, price tranches, credits tied to total spend, etc.

“Rolling back” estimated bills is reprocessing the historic metering data by an older or newer pricing plan version. As i mentioned in another comment someone will have messed up a metering type vale (eg GB/B). Thats why theyll need a few hours to redrive the metering data.

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#470
post #464

Earlier quoted context omitted.

Just today I gave my local agent a CSV which listed a bunch files with of human readable size units and asked it to count rows in each GB range. Sounds simple enough but it completely miscalculated, because it parsed MB as GB for some reason. In hindsight it would've be quicker just to do it in Excel or something.

That is literally what Excel is for. Why didn't you use that first of all?

When all you have is a hammer, but the hammer looks more like a swiss knife
Post reply on HN