Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS: Inaccurate Estimated Billing Data – $1.7 billion

news.ycombinator.com

Post reply on HN
Print view View original

701–710 of 793 posts

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#701

Post by rtpg »

CobrastanJorji wrote: post #589
01284a7e wrote: post #466

Earlier quoted context omitted.

No tests? Just mess up some mundane detail [1] and voila! Wake-up calls and heart attacks for 100,000s of administrators? 1: "Oh, well, this is not a mundane detail, Michael!" https://www.youtube.com/watch?v=3fGHaVn5rGo

There will have been tests, but there will have been missing end-to-end tests. Test 1 will verify that the new system/product emits billing entries in some expected way ("We did 100 bytes of operations and we see we called the billing system for 100 bytes of stuff, yay, test pass"). Test 2 will be in the billing system ("We provide an incoming bill for SKU#12345 for 100 gigabyte-units and we see it costs $17, yay, te…

the cynical take is of course that the tests were written by Claude and not validated.

Unfortunately "automatic generation" of unit tests have made a big part of the value of unit tests ("I am telling you what I expect here") disappear.

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#702

Post by rtpg »

andai wrote: post #642
01284a7e wrote: post #466

Earlier quoted context omitted.

No tests? Just mess up some mundane detail [1] and voila! Wake-up calls and heart attacks for 100,000s of administrators? 1: "Oh, well, this is not a mundane detail, Michael!" https://www.youtube.com/watch?v=3fGHaVn5rGo

>no tests? Earlier this week my slopservant implemented several comprehensive changes to a codebase. It also wrote extensive tests to verify the correctness of the changes. A few days later I was working on something else and realized, everything had been implemented backwards, in a way that was nonsensical and also completely pointless. The many tests it had written were just confirming the LLM's idea of correctness…

write the tests yourself. Or at least the sketch of the test (to get filled in later). Make a commit of what you do so you can then look at the changes.

If you write the tests the agents have an easier time writing the code!

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#703

Post by imhoguy »

ButlerianJihad wrote: post #387
tedggh wrote: post #239

I got a 20K bill once and it was actually drafted from my bank account. It took me a couple of months and involving the office of the AG of my state to get the issue resolved and get my money back. Since then I never touched any AWS product, moved my small stuff to Azure. It’s been years since AWS have these issues with billing, you can find the stories online, students billed 60K for a compromised account launching…

For a while I had a portion of my "homelab" on AWS. I was an educator in a classroom where the students were learning cloud stuff, and the instructor was encouraging the students to stand-up cloud environments for learning, so I figured that I would do the same. I used AWS' free tier, of course, and I enjoyed the initial setup in EC2, and I did a LAMP-stack MediaWiki installation. It wasn't too difficult, but two thi…

This is why traditional bare metal and VPS rental wins for any kind of private or small business use.

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#704

Post by thewhitetulip »

bcrosby95 wrote: post #465
inigyou wrote: post #409

Earlier quoted context omitted.

Big cloud didn't want to rewrite its billing systems from scratch to please its smallest customers.

With AI it should take like a weekend.

Weekend is too long bro! With AI they can weite 50 implementations in a single prompt. They clearly are using tooo many humans and that's why they're lagging

More AI will 100% fix the issue lol

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#705

Post by dev_l1x_be »

yawaramin wrote: post #661
dev_l1x_be wrote: post #583

Earlier quoted context omitted.

Imagine a programming language that has physical measurement unit support so this could have never happened.

Like F#...? https://learn.microsoft.com/en-us/dotnet/fsharp/language-ref...

Exactly. There are other languages that can do that or worst case you can implement it yourself (not sure how hard that is).

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#706

Post by scott_w »

rtpg wrote: post #701
CobrastanJorji wrote: post #589

Earlier quoted context omitted.

There will have been tests, but there will have been missing end-to-end tests. Test 1 will verify that the new system/product emits billing entries in some expected way ("We did 100 bytes of operations and we see we called the billing system for 100 bytes of stuff, yay, test pass"). Test 2 will be in the billing system ("We provide an incoming bill for SKU#12345 for 100 gigabyte-units and we see it costs $17, yay, te…

the cynical take is of course that the tests were written by Claude and not validated. Unfortunately "automatic generation" of unit tests have made a big part of the value of unit tests ("I am telling you what I expect here") disappear.

Nah, this is not uncommon in AWS. I came across a similar bug 10 years ago when playing with Kinesis and nearly had a heart attack at a $2m billing alert! Turns out it was just 5c

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#707

Post by dsifry »

Just got an alert that we would owe $12 Billion. BILLION. After getting me out of bed and a breathless 15 minutes of investigation, I found: https://health.aws.amazon.com/health/status#billingconsole_1... which notes this is an ongoing issue on Amazon's Global Billing Console.

Thanks for the heart attack, Amazon!

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#708

Post by lenkite »

pixl97 wrote: post #547
anvuong wrote: post #436

Earlier quoted context omitted.

Yep, the truth is nobody cares when people start submitting dozens of PRs a day with a bunch of AI-generated code reviews attached to it, all saying everything looks good. I'm witnessing this happening at my workplace right now: Sr/Staff uses Claude to generate 10 pages of design document, Jr uses Claude/Cursor to generate a humongous commit based on this document and create a PR, then bunch of automated AI-based cod…

> the truth is nobody cares The number of errors I've seen over the last 30 years seems to say humans not caring is as much of a deal AI use. It's easy to blame AI for humans being lazy, but I do think it comes naturally to us.

Once you lose ownership you lose interest. AI has supercharged that. It a huge matter of scale now.

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#709

Post by robocat »

inigyou wrote: post #490
steve_adams_86 wrote: post #400

Earlier quoted context omitted.

A couple of my coworkers think I’m nuts for watching cost explorer so closely but 1. The time it takes to look and notice costs that don’t make sense easily pays for itself, and then some (in my experience). I doubt you spent $7k of your time tracking this down, and you probably noticed optimization opportunities that saved you even more 2. I hate the idea of wasting money on buying Jeff Bezos a bigger yacht

> 2. I hate the idea of wasting money on buying Jeff Bezos a bigger yacht Then you aren't using AWS. At least half of all the money you give to Amazon is yacht money.

> yacht money

Huh? Apparently Amazon has never paid dividends, has done much less than 1% buybacks, even as CEO Bezos was getting less than an engineer in total comp (but did get loans or something).

Apparently Bezos owns about 9% of common stock - no dual class shite.

So your yacht spending sounds like a hallucination: if I'm missing something, please correct me.

Amazon's weight: about 3.7–3.8% of the S&P 500. So he isn't even that overweighted (especially as the market doesn't like owners selling down - bad optics).

Bezos has spent about 28 billion for Blue Origin by selling Amazon stock - not using shell games. He has comparitively spent a few percent of that amount on superyachts.

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#710

Post by swiftcoder »

RandomThrow321 wrote: post #672
dlenski wrote: post #670

Earlier quoted context omitted.

> COEs are such a huge annoyance for teams that they create a strong incentive to be proactive in preventing issues like this from happening. Absolutely not my experience at AWS. All the teams I was on treated them as "not a big deal", kind of a non-punitive exercise in technical writing, and the COE was always assigned to be written by an engineer who was not involved in causing the COE. Also, the kinds of issues th…

Obviously experiences vary greatly, but I was on one of the largest AWS orgs and they were quite punitive. People would demand them for perceived slights, then assign reviewers that were tough or on their side. Many of my friends had different experiences, though.

Some orgs also use them to extract concessions from dependencies further up the chain. Upstream service won't fix an issue that's been causing problems for us? "unintentionally" let it become a SEV, so that we can send the CoE up the chain and get Jassy to drop the hammer on that team...
Post reply on HN