Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS: Inaccurate Estimated Billing Data – $1.7 billion

news.ycombinator.com

Post reply on HN
Print view View original

441–450 of 793 posts

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#441

Post by everforward »

inigyou wrote: post #413
everforward wrote: post #315

Earlier quoted context omitted.

Storage-based billing is huge, unless you mean something other than “places that make you pay for storage separately”. Also many places I’ve worked, storage is a huge part of the spend but that depends a lot on what you do. e-commerce doesn’t use a ton of it, but if you handle user-generated content or do any kind of training (LLM, computer vision, etc) then you can very much end up in a place where storage becomes a…

Storage-based billing stories . When an account is hijacked it's always for compute, not storage.

Oh, I also don’t think I’ve ever seen that but I’m not surprised. Even if you could steal a huge amount of storage, filling it with data would take ages and the cat and mouse game of moving the data as hacks get uncovered would be untenable.

I have seen things get hacked for bandwidth, back in the days before you could rent a gbps uplink from the cloud for $0.12. Some scene release groups would hack into universities or companies to do the initial seeding over their super fast links. It used storage, but that wasn’t really the goal.

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#443

Post by dymk »

01284a7e wrote: post #372
yuchen20 wrote: post #62

I got 3 consecutive emails warning that my budget crossed its $18 threshold. Opened it up: cost was 78 million. Thought it was a phishing attempt, logged into my actual account, and... still 78 million. EMOTIONAL DAMAGE.

Yes, I am taking legal action, no doubt.

…for emotional damage?

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#444

Post by poly2it »

lukaslueg wrote: post #17

Apparently what used to be `GB of storage consumed` is confused with `Bytes of storage consumed`, leading to a cool off by 2*30 error. > You're right to question my calculation. The MCP server failed to connect when I tried to look up the field definition. I guessed instead of validating. This is on me. But look at all the revenue!

This error could be fixed with better typing. If you compute on GiB in a billing system, make sure it can only ever be mutated with a GiB type!

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#445

Post by mxuribe »

ainiriand wrote: post #275
sscaryterry wrote: post #41

Vibe coding billing systems is a top-notch idea :)

Hey what do you think about vibe coding weapon systems? Do you want to be my cofounder?

We retro-fitted a Terminator T100 model with the brain of the latest LLM models, and then gave'em 2 shotguns...and, you'll never guess what happened next!

Well, actually i guess you can guess what happens next! lol :-D

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#447

Post by londons_explore »

ibejoeb wrote: post #308
root-parent wrote: post #268

Earlier quoted context omitted.

Wanna bet the description of this job post will be updated by the end of the day? "Software Development Engineer II, AWS Invoicing" https://www.amazon.jobs/de/jobs/10428480/software-developmen... "...Our platforms are powered by generative AI, large language models, knowledge graphs, and agentic architectures that dynamically compose specialized agents based on context. We apply these capabilities across three reinfo…

Wow: In this role you will: - Design and build agentic AI systems that analyze, generate, and validate... - Build agentic architectures that compose specialized AI agents dynamically... - Build AI-driven continuous validation frameworks powered by agentic workflows and large language models that autonomously manage... This is invoicing? If ever there was a domain that was purely deterministic, you'd hope it was invoi…

Probably not actually. Transferring one kilobyte across a network link has such a low value that the billing costs of aggregating it cost more than the revenue.

So instead you take a probabilistic approach - charge the user for a megabyte of data transfer 0.1% of the time, and bill nothing 99.9% of the time.

Now the typical cost is the same, the users bill is probably accurate to the cent, but you have divided the number of billing records by 1000.

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#448

Post by dlev_pika »

lelandfe wrote: post #379

This just hit global news: https://www.theguardian.com/technology/2026/jul/17/amazon-we... > Amazon Web Services customers receive bills for up to $1.5tn after global glitch

1.5 trillion? Those are rookie numbers.

How about $5,544,640,717,404.09?

That was in my inbox this morning lmao

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#449

Post by donavanm »

Ive dealt with this error at AWS. It’s a unit error. In my case we _meant_ to charge like 5¢/GB, but missed the unit (GB), and then the billing system defaults to bytes. 5¢ per Byte of data transferred meant some customers were seeing MM bills within hours. Got paged by support around 2am, had it fixed and amendments issues by 3-4am, apology emails shortly after.

Services emit metering values that arent directly tied to prices. Every SKU/line item is defined in a “pricing plan”, with a unit type, regions, and price per unit. The metering records are joined to a pricing plan based on account id, region, sku, etc. mess up the unit type in the pricing plan and the metering data conversion doesnt work, and you get crazy bills.

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#450

Post by jayd16 »

chanux wrote: post #426
root-parent wrote: post #277

Earlier quoted context omitted.

>> Or just a distracted dev And a distracted tester? And a distracted pipeline of regression tests? No, the truth is way worst...

What if there's only half a dev and a swarm of agents after the layoffs?

> only half a dev

That's one way to cut staff.

Post reply on HN