AWS: Inaccurate Estimated Billing Data – $1.7 billion • Hacker News Board

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#781

Post by skjfjnflw » Sun, Jul 19, 2026, 4:00 PM UTC

mightyham wrote: post #549

qurren wrote: post #479

Earlier quoted context omitted.

You overestimate how much people give shits at big techs like Amazon. When literally everything is driven with sticks instead of carrots, the work culture does not invite employees to proactively care about product quality. You'd be better off letting the heart attacks happen and take the 3am on-call and be the hero instead. It would be good promo doc material, and being a hero is extremely good insurance against get…

Speaking from my experience at Amazon this is not the case. Any customer impact like this would necessitate a COE (correction of errors) report, which means a list of required action items to prevent such issues from happening again, which typically suck up at least man-month of labor. Not to mention the report itself, which has to be written by a manager. In fact, there are regular AWS-wide meetings where L10 techni…

[dead]

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#782

Post by thayne » Mon, Jul 20, 2026, 1:29 AM UTC

CobrastanJorji wrote: post #589

01284a7e wrote: post #466

Earlier quoted context omitted.

No tests? Just mess up some mundane detail [1] and voila! Wake-up calls and heart attacks for 100,000s of administrators? 1: "Oh, well, this is not a mundane detail, Michael!" https://www.youtube.com/watch?v=3fGHaVn5rGo

There will have been tests, but there will have been missing end-to-end tests. Test 1 will verify that the new system/product emits billing entries in some expected way ("We did 100 bytes of operations and we see we called the billing system for 100 bytes of stuff, yay, test pass"). Test 2 will be in the billing system ("We provide an incoming bill for SKU#12345 for 100 gigabyte-units and we see it costs $17, yay, te…

> well we can't have the tests actually charge money, that's a legal/accounting problem, it might even be a crime

You don't have a non-production environment where you can have billing that doesn't involve real money?

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#783

Post by Twirrim » Mon, Jul 20, 2026, 2:21 AM UTC

donavanm wrote: post #662

Twirrim wrote: post #575

Earlier quoted context omitted.

I wonder if AWS billing still uses CSV files for passing data around. IIRC it was one of my first on-calls at AWS over a decade ago now, and I got a page early evening because some stuff we did with billing records broke because some "smart" engineer thought it'd be a great idea to put an experimental record in with a description something like "I wonder what happens if I put, a comma in this field", into the product…

It was the same old kona files for metering until at least a ‘23-24 when I last did metering/pricing work. “Luckily” the size of the EC2 pricing plan was so stupidly large that it was forcing improvements to how those were serialized, and how private pricing plans were created/assigned to different payer ids.

It's one of those "I hate it, but I get it" situations. Migrating metering over to a whole new format on AWS's scale is a scarily large undertaking.

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#784

Post by picofarad » Mon, Jul 20, 2026, 1:48 PM UTC

spydum wrote: post #746

nonameiguess wrote: post #740

Earlier quoted context omitted.

This reminds me of a discussion a few months back from a BSD maintainer who had done a lot of volunteer work for AWS over the years. I think he might have even been the person who alerted them to the insecurity of IMDSv1. There was a sense that AWS might have had great talented developers at the time, but they clearly didn't really understand the domain of running a hypervisor service exposed to the public. This feel…

I have bad news if you think banks are the pinnacle of cyber security practices... Maybe some rare few are, but by and large they generally stink. They tend to give that impression because they hire tons of auditors and panjandrum, to hassle their suppliers, but internally they are winging it like the rest of us.

Does panjandrum mean chaotic rocketry, flahy but worthless in the end?

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#785

Post by picofarad » Mon, Jul 20, 2026, 1:56 PM UTC

shermantanktop wrote: post #632

sscaryterry wrote: post #563

Earlier quoted context omitted.

This isn't a flippant comment. Imagine though, being presented with this. Imagine having some underlying health problem (e.g. cardiovascular). Do not be surprised if real people actually die from this mistake, from the anxiety, the surprise, the helplessness.

But someone that susceptible is likely going to have a bad reaction to many possible unexpected things. How would they react to a minor traffic accident? a family member getting hurt? a letter from tax authorities asking questions? Having that serious an underlying health problem means everyday life represents risk for you. I don't think that means everyone else has to behave differently wrt (in this case wrt to bill…

Yeah someone thinking they screwed up to the tune of tens of mlllions of dollars (or billions, someone said 2^30 error); this is different to an auto accident or something. Most people can't even fathom millions or billions of dollars.

I'm stressed thinking about it, like from a theoretical standpoint.

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#786

Post by pedrosousa » Mon, Jul 20, 2026, 2:01 PM UTC

binaryturtle wrote: post #146

fron wrote: post #98

Woke up to a billing alarm email. Thought I had leaked my AWS keys accidentally and somehow run up 437 billion dollars of charges. Joke's on them though, I don't have 437 billion dollars Anyways I didn't need coffee. That produced an adrenaline release unlike any I've experienced before. Thanks AWS

I wonder how many people may have gotten an actual heart infarct because of that. There may be a person out there that may be dead as a result. It's entirely irresponsible of Amazon to even display such values to the user.

I legit had a small panic attack due to this.

I'm not a devops but we are running some s3 buckets. When I saw the number I couldn't even comprehend and thought we had been hacked.

I feel like this is such a bad mistake from AWS and we will never be compensated for it.

Put me off having anything on AWS from now on.

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#787

Post by picofarad » Mon, Jul 20, 2026, 2:07 PM UTC

AlotOfReading wrote: post #648

reddalo wrote: post #604

Earlier quoted context omitted.

You just need to start teaching kids the metric system. Then when they'll grow up the switch will almost magically happen by itself.

All of my science classes in the rural US from middle school onward were taught entirely in metric, and it had been that way for decades prior. Obviously I'm an adult now, and metric still isn't universal.

The cost is too high. You can't have mph and kmh signs on the same road. So you gotta switch all the signs at once. Mile markers, exit numbers (are miles), distance to signs, all have to change.

I did the math on a comment elsewhere. Just google how many miles of interstate there are, and quadruple it, thats not even approaching how many signs you'd need for just the interstate. Signs run $25-ish, plus installation, which can cost >$1000. At the very least the km markers will need to be installed.

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#788

Post by picofarad » Mon, Jul 20, 2026, 2:21 PM UTC

yunnpp wrote: post #472

idiotsecant wrote: post #390

Earlier quoted context omitted.

Would be funny if it wasn't so close to true

'My absurd statement doesn't sound right, so the "opposite" (assuming it's well-defined and unique) must be true' is peak LLM logic. You can tell it was trained on Reddit commentary.

Was it proof by contrapositive?

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#789

Post by SAI_Peregrinus » Mon, Jul 20, 2026, 3:32 PM UTC

inigyou wrote: post #410

SAI_Peregrinus wrote: post #394

Earlier quoted context omitted.

> and they also imply deleting all your data so that you don't pay for storage. Not necessarily. They could imply that your storage becomes inaccessible immediately, but only gets deleted after some time period (say, 1 month). What spending limits do depends on the implementation.

That's even more work to implement. And now you store files on a second account that pays for only one day a month to not get deleted.

How would that work? If you only pay for one day, your access (read and write) gets cut off after the first month. Then if you don't pay the bill for the entire grace period, your data gets deleted.

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#790

Post by myself248 » Mon, Jul 20, 2026, 6:52 PM UTC

dawnerd wrote: post #768

csomar wrote: post #594

Earlier quoted context omitted.

Most debit/prepaid cards will get rejected. Credit Cards technically have a limit but they really don't. It's an open line to your finances.

I’ve never had a limited card get rejected. Granted I use ones from privacy.com. That’s also what I have hooked up to my aws / amazon.

They are also not all that limited. I used a Privacy.com card for a New York Times trial subscription, which of course I could not cancel online, so I just let it expire. A few days past the renewal date, I saw a $waytoomuch charge on my card.

I wrote to Privacy, and this was their reply:

> Jun 1, 2022, 5:21 PM EDT

> I've been reviewing your dispute and wanted to touch base with you to explain what happened.

> It appears that the disputed charge is a "force post" by the merchant. This happens when a merchant cannot collect funds for a transaction after repeated attempts and completes the transaction without an authorization — it's literally an unauthorized transaction that's against payment card network rules. It's a pretty sneaky move used by some merchants, and unfortunately, it's not something Privacy can block.

> We are refunding the transaction to the original funding source used for this purchase. You should see this refund reflected in your bank statement in the next 5 business days. However, please note that since this merchant uses force posts, they will continue to do this monthly until the subscription is cancelled. Please contact them to have the subscription cancelled.

> Let me know if you have any questions or concerns!