Live data from Hacker News

AWS: Inaccurate Estimated Billing Data – $1.7 billion

news.ycombinator.com

551–560 of 793 posts

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#551
post #535

Earlier quoted context omitted.

We have a pretty strong existence proof... the thing happened in production. Unless they have some means to override a failing test and scp broken shit to prod, there wasn't a test.

Technically, there could be a test. It could just be wrong!

If a tree falls in the forest and nobody hears it...

[edit] Testing your tests, like testing your backups, is a good idea

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#552

Earlier quoted context omitted.

Wow: In this role you will: - Design and build agentic AI systems that analyze, generate, and validate... - Build agentic architectures that compose specialized AI agents dynamically... - Build AI-driven continuous validation frameworks powered by agentic workflows and large language models that autonomously manage... This is invoicing? If ever there was a domain that was purely deterministic, you'd hope it was invoi…

Probably not actually. Transferring one kilobyte across a network link has such a low value that the billing costs of aggregating it cost more than the revenue. So instead you take a probabilistic approach - charge the user for a megabyte of data transfer 0.1% of the time, and bill nothing 99.9% of the time. Now the typical cost is the same, the users bill is probably accurate to the cent, but you have divided the nu…

I don't know how cloud services count usage, but this is certainly not true for telco. I manage several fleets of hundreds/thousands of SIM cards (mostly IoT/M2M applications), and almost every provider counts the data traffic per byte. Different business and use case, I know, but still.

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#553
post #525

Earlier quoted context omitted.

Unit mistakes happen all the time, which is why you should be using your units library religiously and still being vigilant even then. Worst case I've found was off by 15 orders of magnitude.

One of the Mars landers famously failed due to unit conversion errors from metric to standard.

I didn't know the imperial system was named "standard". Funny, cause its everything but standard both internationally and its definitions (which are not standard as based on SI)

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#554

Earlier quoted context omitted.

There's a certain obligation on both sides of a contract to pay attention. If you're not watching your billing, and then try to claim overcharging a year later, you'll get a lot less satisfaction even from regulators or judges than if you notice it when (or soon after) it happens.

Cable bills are extremely complicated on purpose and people are taxed for time attention and intelligence. The employees and company have an obligation not to exploit this even if the issue is only discovered after the fact. You don't get to export any of the responsibility to your customer. They don't prepare the bill and it's not their job to find your fuck ups

No argument, but fuck-ups happen, and get fixed more quickly and easily when people are paying attention.

I once got a monthly water bill for ~$35,000 at a residential, single-family home. Good thing I was paying attention and looked at the bill before the auto-pay bank draft hit.

Someone had misread the meter.

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#556

Earlier quoted context omitted.

A couple of my coworkers think I’m nuts for watching cost explorer so closely but 1. The time it takes to look and notice costs that don’t make sense easily pays for itself, and then some (in my experience). I doubt you spent $7k of your time tracking this down, and you probably noticed optimization opportunities that saved you even more 2. I hate the idea of wasting money on buying Jeff Bezos a bigger yacht

> 2. I hate the idea of wasting money on buying Jeff Bezos a bigger yacht Then you aren't using AWS. At least half of all the money you give to Amazon is yacht money.

Unfortunately not a choice at my organization

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#557

Earlier quoted context omitted.

Probably not actually. Transferring one kilobyte across a network link has such a low value that the billing costs of aggregating it cost more than the revenue. So instead you take a probabilistic approach - charge the user for a megabyte of data transfer 0.1% of the time, and bill nothing 99.9% of the time. Now the typical cost is the same, the users bill is probably accurate to the cent, but you have divided the nu…

The way you describe requires somehow counting every bit but somehow discarding most which is obviously nonsense. This seems statistically invalid insofar as it will tend to overbill potentially by a lot on the minority of cases. Don't you know how much of the pipe is occupied by a given customers code at any given time or what data is being sent

You have to do it when the customer list is too big to keep a counter per customer.

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#558
post #350

It's crazy enough this will be fixed soon. Years ago I found an actual hidden error in my bill. (This was early 2010s). The system was calculating the EC2 reservation savings incorrectly for some of my servers. I was crunching all their detailed usage data on a regular basis in an 18 tab spreadsheet and couldn't get it to fully reconcile. I spent months trying to track down the discrepancy. Once I found it, I had to…

> After 14 months I got a $7,000 refund. I was told it had to be approved by the head of AWS. $7,000 of credits is no problem. At that time a friendly neighborhood PM or director could issue the credit without much oversight. Your problem is the time period. Amending a bill in the same cycle is EZ. Fixing the previous cycle is a PITA but pretty common. Issuing amendments for the previous financial _years_ would be a…

Banks and financial institutions are the same. If they haven’t issued you a monthly/quarterly statement yet, they can just apologize and tell you the numbers are wrong please wait for the statement. But it is a major issue if an actual statement has the wrong numbers.

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#559

Apparently what used to be `GB of storage consumed` is confused with `Bytes of storage consumed`, leading to a cool off by 2*30 error. > You're right to question my calculation. The MCP server failed to connect when I tried to look up the field definition. I guessed instead of validating. This is on me. But look at all the revenue!

The user is asking me to calculate how much money they should charge their customer. The values they've given me are 0.45, 1.67, and 2.50. This is 2.50 + 1.67 + 0.45 = 4.62, but it could be any other number. Perhaps we should be concatenating the numbers instead. Wait! The . could also mean multiplication. 0 . 45 . 1. 67 . 2 . 50 = 3015000. But wouldn't multiplying by 0 zero it out? That can't be right, we wouldn't b…

So uh did you type that out or generate it somewhere?

Number felt high so I wanted to double check and I only get 301500.

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#560

One can almost smell the vibes. This is peanuts compared to a major cybersecurity catastrophe that’s surely in the making. To give credit to the technology and the people using it - and I’m not being facetious - it’s actually incredible that at the current levels of usage the unprecedented catastrophic event has not yet happened.

Clearing LLMs out of our business infrastructure is going to be a massive undertaking. Though I have a tech background, I work in commercial real estate. We are recently seeing new levels of idiocy from the employees, including real estate brokers with zero tech knowledge "coding" solutions to find sites for clients and blindly trusting the output (which I came to find out was complete bullshit), as well as some who…

>Clearing LLMs out of our business infrastructure is...n't going to happen.

We've given Moloch a new form, and it ain't going away.

Post reply on HN