Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS: Inaccurate Estimated Billing Data – $1.7 billion

news.ycombinator.com

Post reply on HN
Print view View original

541–550 of 793 posts

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#541

Post by TJSomething »

ibejoeb wrote: post #308
root-parent wrote: post #268

Earlier quoted context omitted.

Wanna bet the description of this job post will be updated by the end of the day? "Software Development Engineer II, AWS Invoicing" https://www.amazon.jobs/de/jobs/10428480/software-developmen... "...Our platforms are powered by generative AI, large language models, knowledge graphs, and agentic architectures that dynamically compose specialized agents based on context. We apply these capabilities across three reinfo…

Wow: In this role you will: - Design and build agentic AI systems that analyze, generate, and validate... - Build agentic architectures that compose specialized AI agents dynamically... - Build AI-driven continuous validation frameworks powered by agentic workflows and large language models that autonomously manage... This is invoicing? If ever there was a domain that was purely deterministic, you'd hope it was invoi…

This is like half of all job listings I've read recently. And it's a decent amount of fintech that's like this.

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#542

Post by mcpherrinm »

akdev1l wrote: post #474
01284a7e wrote: post #466

Earlier quoted context omitted.

No tests? Just mess up some mundane detail [1] and voila! Wake-up calls and heart attacks for 100,000s of administrators? 1: "Oh, well, this is not a mundane detail, Michael!" https://www.youtube.com/watch?v=3fGHaVn5rGo

Not even tests but just some basic anomaly detection lol. Like maybe if the bill amounts increase by like 10M% there should be someone that looks into it

While I didn't work on AWS, I did intern on the retail side of Amazon, and there's definitely this sort of monitoring in place. Surely somebody was paged. And even if not, this is "just" the cost explorer estimations, not what is ending up on folk's bills.

I learned about https://en.wikipedia.org/wiki/2011_T%C5%8Dhoku_earthquake_an...> from alarms like this, as sales in Japan almost entirely stopped.

I've been told a tale of another incident where some customer ran some huge cpu-intensive workload that didn't do any networking. It caused various alarms to fire because it "looked like" a part of the network was idle (potentially indicating some sort of networking failure)

It's generally (in the broad sense) easy to add alarms for things going wrong, but in my experience anomaly detectors are just as likely to fire from other weird things like that happening.

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#543

Post by golem14 »

gleenn wrote: post #525
AlotOfReading wrote: post #484

Earlier quoted context omitted.

Unit mistakes happen all the time, which is why you should be using your units library religiously and still being vigilant even then. Worst case I've found was off by 15 orders of magnitude.

One of the Mars landers famously failed due to unit conversion errors from metric to standard.

Wasn't it (also?) the Ariane V flight in 1996? Oh, NVM, that was an overflow error.

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#544

Post by fullstop »

marksk wrote: post #34

logged in this morning to find a bill of $595 Billion... heart rate went through the roof... then I noticed the open issue, phew! nice one guys... you got me there... But with AWS costs rising anyway (not by that much but OK), I'm probably not the only one to start reconsidering their cloud strategy. I think this might have just pushed me over the edge.

Even if it was 595 billion, that sounds like their problem.

"If you owe your bank manager a thousand pounds, you are at his mercy. If you owe him a million pounds, he is at yours."

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#545

Post by michaelmrose »

SoftTalker wrote: post #528
michaelmrose wrote: post #493

Earlier quoted context omitted.

Reminds me of working for a cable company and being told that even if we screwed up and stole from the customer the look back period was only a few months and if we found an error from before that we weren't supposed to correct it.

There's a certain obligation on both sides of a contract to pay attention. If you're not watching your billing, and then try to claim overcharging a year later, you'll get a lot less satisfaction even from regulators or judges than if you notice it when (or soon after) it happens.

Cable bills are extremely complicated on purpose and people are taxed for time attention and intelligence.

The employees and company have an obligation not to exploit this even if the issue is only discovered after the fact.

You don't get to export any of the responsibility to your customer. They don't prepare the bill and it's not their job to find your fuck ups

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#546

Post by mito88 »

amelius wrote: post #516
bot403 wrote: post #504

Earlier quoted context omitted.

Why? What's the damages? They showed you a wrong number, then later acknowledged it and fixed it. Just because the number was "very big" to you doesn't mean you were actually aggrieved in some way.

Big numbers can lead to stress which can lead to all kinds of disorders.

small numbers too....

:)

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#547

Post by pixl97 »

anvuong wrote: post #436
root-parent wrote: post #277

Earlier quoted context omitted.

>> Or just a distracted dev And a distracted tester? And a distracted pipeline of regression tests? No, the truth is way worst...

Yep, the truth is nobody cares when people start submitting dozens of PRs a day with a bunch of AI-generated code reviews attached to it, all saying everything looks good. I'm witnessing this happening at my workplace right now: Sr/Staff uses Claude to generate 10 pages of design document, Jr uses Claude/Cursor to generate a humongous commit based on this document and create a PR, then bunch of automated AI-based cod…

> the truth is nobody cares

The number of errors I've seen over the last 30 years seems to say humans not caring is as much of a deal AI use. It's easy to blame AI for humans being lazy, but I do think it comes naturally to us.

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#548

Post by nullorempty »

27183 wrote: post #535
mvdtnz wrote: post #511

Earlier quoted context omitted.

Why would you think there are "no tests"?

We have a pretty strong existence proof... the thing happened in production. Unless they have some means to override a failing test and scp broken shit to prod, there wasn't a test.

Technically, there could be a test. It could just be wrong!

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#549

Post by mightyham »

qurren wrote: post #479
akdev1l wrote: post #474

Earlier quoted context omitted.

Not even tests but just some basic anomaly detection lol. Like maybe if the bill amounts increase by like 10M% there should be someone that looks into it

You overestimate how much people give shits at big techs like Amazon. When literally everything is driven with sticks instead of carrots, the work culture does not invite employees to proactively care about product quality. You'd be better off letting the heart attacks happen and take the 3am on-call and be the hero instead. It would be good promo doc material, and being a hero is extremely good insurance against get…

Speaking from my experience at Amazon this is not the case. Any customer impact like this would necessitate a COE (correction of errors) report, which means a list of required action items to prevent such issues from happening again, which typically suck up at least man-month of labor. Not to mention the report itself, which has to be written by a manager.

In fact, there are regular AWS-wide meetings where L10 technical staff will randomly pick and review reports from across the organization. Getting picked for one of these is not a fun experience.

COEs are such a huge annoyance for teams that they create a strong incentive to be proactive in preventing issues like this from happening. One of the rules when it comes to writing COEs is that they are not the fault of individuals but processes; but in reality, no one wants to be the cause of one.

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#550

Post by Sohcahtoa82 »

galonk wrote: post #385
berkes wrote: post #286

Earlier quoted context omitted.

> Almost all of them have one number that appears four times, and one or two that appear three times To me that looked suspiciously like string-handling in a weakly typed language. Like when you do `"100" + 1` in JavaScript, or `int("100" * 2)` in Python. I've seen my share of such bugs in PHP, Python, Ruby, JavaScript. In production. Obviously not as simple as the examples, but subtle, like when a library update cha…

Pedantic as hell but `"100" * 2` in Python (= `"100100"` for those who don't know) isn't really typing, it's operator overloading. Any language with that could implement the same questionable design decision.

And as much as I love Python, being able to multiple a string by an integer doesn't make sense when adding an integer to a string is a TypeError.

Being able to repeat a string is fine, but it should be a str.repeat() function, not an operator overload like that.

Post reply on HN